Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date
input {
generator {
lines => ['2011-04-19T03:44:01.103Z testlog1',
'2011-04-19T03:44:02.035Z testlog2',
'2011-04-19T03:44:03.654Z testlog3',
'2011-04-19T03:44:03.654Z testlog3']
count => 1
}
}
filter {
dissect {
mapping => {
"message" => "%{ts} %{msg}"
}
}
date {
match => [ "ts", "ISO8601" ]
}
fingerprint {
source => "message"
target => "[@metadata][fingerprint]"
method => "MD5"
key => "test"
}
ruby {
code => "event.set('@metadata[prefix]', event.get('@timestamp').to_i.to_s(16))"
}
}
output {
elasticsearch {
document_id => "%{[@metadata][prefix]}%{[@metadata][fingerprint]}"
}
stdout { codec => rubydebug { metadata => true }}
}
@jeremyforan

This comment has been minimized.

Copy link

commented May 22, 2019

I get the following Error when attempting to use the code
[2019-05-22T13:56:55,964][ERROR][logstash.filters.ruby ] Ruby exception occurred: Invalid FieldReference: '@metadata[prefix]'

adding brackets to the set metadata variable seems to resolve the issue:
ruby { code => 'event.set("[@metadata][prefix]", event.get("@timestamp").to_i.to_s(16))' }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.