LogStash configuration Sample

Execute logstash

java -cp . logstash.runner agent -f logstash.conf -- web --backend elasticsearch:///?local

grok-patterns

//Add this line to patterns/grok-patterns

TELEFENGINX %{IPORHOST:site} %{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBE:httpversion})?|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwardedfor}

logstash.conf

input {
  file {
    type => nginx_web
    path => ["/var/log/nginx/*"]
    exclude => ["*.gz"]
  }
}

filter {
  grok {
    type => "nginx_web"
    patterns_dir => [ "/tmp/logstash/patterns" ]
    pattern => "%{TELEFENGINX}"
  }
}

output {
  elasticsearch { embedded => true }
}

sample nginx.conf

/// The following is the sample nginx.conf that defines the log used by the grok pattern defined above


log_format  combined_with_server  '$http_host $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

Thanks, @pcolazurdo.

For everyone, who trying to find instructions similar to placed above: yet another examples for nginx log parsing with filebeat, logstash and grok.
https://www.devops.zone/solr-elasticsearch/shipping-your-nginx-logs-to-elasticsearch-using-logstash/
https://www.digitalocean.com/community/tutorials/adding-logstash-filters-to-improve-centralized-logging/
http://logz.io/blog/nginx-log-analysis/

If you have Filebeat, you should add your filter{} section between input and output sections in /etc/logstash/conf.d/beats.conf

@pcolazurdo, you have error in this part of the pattern: HTTP/%{NUMBE:httpversion} Expected: NUMBE_R_

@KIVagant I encountered the same problem when running this gork. There appears to be a typo, and the "R" was forgotten. It should read: HTTP/%{NUMBER:httpversion}

This does not seem to work anymore. This is the error I get when running with this configuration:

[2016-12-02T10:16:56,423][ERROR][logstash.agent ] fetched an invalid config {:config=>"input {\n file {\n type => nginx_web\n path => [\"/Users/justusblumer/Downloads/access_logs_curved/access-curved.log.*\"]\n exclude => [\"*.gz\"]\n }\n}\n\nfilter {\n grok {\n type => \"nginx_web\"\n patterns_dir => [ \"/tmp/logstash/patterns\" ]\n pattern => \"%{TELEFENGINX}\"\n }\n}\n\noutput {\n elasticsearch { embedded => true }\n}\n", :reason=>"The setting typein plugingrokis obsolete and is no longer available. You can achieve this same behavior with the new conditionals, like:if [type] == "sometype" { grok { ... } }. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}