Skip to content

Instantly share code, notes, and snippets.

@cdahlqvist
Last active July 3, 2020 03:50
Show Gist options
  • Save cdahlqvist/8ff8fb8e56c43414396edf60e7944144 to your computer and use it in GitHub Desktop.
Save cdahlqvist/8ff8fb8e56c43414396edf60e7944144 to your computer and use it in GitHub Desktop.
Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date
input {
generator {
lines => ['2011-04-19T03:44:01.103Z testlog1',
'2011-04-19T03:44:02.035Z testlog2',
'2011-04-19T03:44:03.654Z testlog3',
'2011-04-19T03:44:03.654Z testlog3']
count => 1
}
}
filter {
dissect {
mapping => {
"message" => "%{ts} %{msg}"
}
}
date {
match => [ "ts", "ISO8601" ]
}
fingerprint {
source => "message"
target => "[@metadata][fingerprint]"
method => "MD5"
key => "test"
}
ruby {
code => "event.set('@metadata[prefix]', event.get('@timestamp').to_i.to_s(16))"
}
}
output {
elasticsearch {
document_id => "%{[@metadata][prefix]}%{[@metadata][fingerprint]}"
}
stdout { codec => rubydebug { metadata => true }}
}
@jeremyforan
Copy link

I get the following Error when attempting to use the code
[2019-05-22T13:56:55,964][ERROR][logstash.filters.ruby ] Ruby exception occurred: Invalid FieldReference: '@metadata[prefix]'

adding brackets to the set metadata variable seems to resolve the issue:
ruby { code => 'event.set("[@metadata][prefix]", event.get("@timestamp").to_i.to_s(16))' }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment