Skip to content

Instantly share code, notes, and snippets.

@teebu
Forked from maxkandler/logstash.conf
Last active February 6, 2019 10:08
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save teebu/57317bbcf3c30f73b5535060484e83a0 to your computer and use it in GitHub Desktop.
Save teebu/57317bbcf3c30f73b5535060484e83a0 to your computer and use it in GitHub Desktop.
Grok filter for Cloudfront Logs to be used with Logstash & ElasticSearch
filter {
grok {
match => [
"message", "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}[\t]%{TIME:time}[\t](?<x_edge_location>\b[\w\-]+\b)[\t](?:%{NUMBER:sc_bytes}|-)[\t]%{IPORHOST:clientip}[\t]%{WORD:cs_method}[\t]%{HOSTNAME:cs_host}[\t]%{NOTSPACE:cs_uri_stem}[\t]%{NUMBER:sc_status}[\t]%{GREEDYDATA:referrer}[\t]%{GREEDYDATA:agent}[\t]%{GREEDYDATA:cs_uri_query}[\t]%{GREEDYDATA:cookies}[\t]%{WORD:x_edge_result_type}[\t]%{NOTSPACE:x_edge_request_id}[\t]%{HOSTNAME:x_host_header}[\t]%{GREEDYDATA:cs_protocol}[\t]%{INT:cs_bytes}[\t]%{GREEDYDATA:time_taken}[\t]%{GREEDYDATA:x_forwarded_for}[\t]%{GREEDYDATA:ssl_protocol}[\t]%{GREEDYDATA:ssl_cipher}[\t]%{GREEDYDATA:x_edge_response_result_type}[\t]%{GREEDYDATA:cs_protocol_version}"
]
}
geoip {
source => "c_ip"
}
mutate {
add_field => ["listener_timestamp", "%{year}-%{month}-%{day} %{time}"]
convert => {
"[geoip][coordinates]" => "float"
"sc_bytes" => "integer"
"cs_bytes" => "integer"
"time_taken" => "float"
}
}
date {
match => ["listener_timestamp", "yyyy-MM-dd HH:mm:ss"]
}
}
@teebu
Copy link
Author

teebu commented Nov 1, 2017

x_edge_location can contain dash, WORD group doesn't allow for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment