Skip to content

Instantly share code, notes, and snippets.

@thikade
Created April 5, 2015 11:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save thikade/e731ce9c6183d3d82630 to your computer and use it in GitHub Desktop.
Save thikade/e731ce9c6183d3d82630 to your computer and use it in GitHub Desktop.
logstash: apache access log example using fingerprint filter - how to create unified event ids that can be re-imported/updated into Elasticsearch
input {
file {
type => "apache"
start_position => "beginning"
# path => ["/var/log/secure", "/var/log/messages"]
path => ["/var/log/access.log.201*"]
exclude => ["*.gz"]
}
}
filter {
fingerprint {
"key" => "0123"
"method" => "SHA1"
}
if [type] == "apache" {
grok {
match => [ "message", "%{COMBINEDAPACHELOG} %{NUMBER:duration:int}" ]
match => [ "message", "%{COMBINEDAPACHELOG}" ]
add_tag => [ "parsed" ]
add_tag => [ "index" ]
}
mutate {
convert => {
"response" => "integer"
"bytes" => "integer"
}
}
ruby {
# convert Apache responsetime from usec into ms
code => "event['duration'] = (event['duration'] / 1000) if event['duration']"
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
add_tag => [ "dated" ]
}
}
}
output {
# stdout { codec => rubydebug }
# output grok failures into custom error log
if "_grokparsefailure" in [tags] {
stdout { codec => rubydebug }
file {
message_format => "%{message}"
path => "/tmp/logstash_grok_errors.log"
}
}
# only output "index" events to ES
if "index" in [tags] {
elasticsearch { document_id => "%{fingerprint}" }
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment