Skip to content

Instantly share code, notes, and snippets.

@pcolazurdo
Last active December 2, 2016 09:20
Show Gist options
  • Save pcolazurdo/4538923 to your computer and use it in GitHub Desktop.
Save pcolazurdo/4538923 to your computer and use it in GitHub Desktop.
LogStash configuration Sample
java -cp . logstash.runner agent -f logstash.conf -- web --backend elasticsearch:///?local
//Add this line to patterns/grok-patterns
TELEFENGINX %{IPORHOST:site} %{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBE:httpversion})?|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwardedfor}
input {
file {
type => nginx_web
path => ["/var/log/nginx/*"]
exclude => ["*.gz"]
}
}
filter {
grok {
type => "nginx_web"
patterns_dir => [ "/tmp/logstash/patterns" ]
pattern => "%{TELEFENGINX}"
}
}
output {
elasticsearch { embedded => true }
}
/// The following is the sample nginx.conf that defines the log used by the grok pattern defined above
log_format combined_with_server '$http_host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
@igaskin
Copy link

igaskin commented Aug 10, 2016

@KIVagant I encountered the same problem when running this gork. There appears to be a typo, and the "R" was forgotten. It should read: HTTP/%{NUMBER:httpversion}

@justusbluemer
Copy link

This does not seem to work anymore. This is the error I get when running with this configuration:

[2016-12-02T10:16:56,423][ERROR][logstash.agent ] fetched an invalid config {:config=>"input {\n file {\n type => nginx_web\n path => [\"/Users/justusblumer/Downloads/access_logs_curved/access-curved.log.*\"]\n exclude => [\"*.gz\"]\n }\n}\n\nfilter {\n grok {\n type => \"nginx_web\"\n patterns_dir => [ \"/tmp/logstash/patterns\" ]\n pattern => \"%{TELEFENGINX}\"\n }\n}\n\noutput {\n elasticsearch { embedded => true }\n}\n", :reason=>"The setting typein plugingrokis obsolete and is no longer available. You can achieve this same behavior with the new conditionals, like:if [type] == "sometype" { grok { ... } }. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment