LogStash configuration Sample

Execute logstash

java -cp . logstash.runner agent -f logstash.conf -- web --backend elasticsearch:///?local

grok-patterns

//Add this line to patterns/grok-patterns

TELEFENGINX %{IPORHOST:site} %{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBE:httpversion})?|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwardedfor}

logstash.conf

input {
  file {
    type => nginx_web
    path => ["/var/log/nginx/*"]
    exclude => ["*.gz"]
  }
}

filter {
  grok {
    type => "nginx_web"
    patterns_dir => [ "/tmp/logstash/patterns" ]
    pattern => "%{TELEFENGINX}"
  }
}

output {
  elasticsearch { embedded => true }
}

sample nginx.conf

/// The following is the sample nginx.conf that defines the log used by the grok pattern defined above


log_format  combined_with_server  '$http_host $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

This does not seem to work anymore. This is the error I get when running with this configuration:

[2016-12-02T10:16:56,423][ERROR][logstash.agent ] fetched an invalid config {:config=>"input {\n file {\n type => nginx_web\n path => [\"/Users/justusblumer/Downloads/access_logs_curved/access-curved.log.*\"]\n exclude => [\"*.gz\"]\n }\n}\n\nfilter {\n grok {\n type => \"nginx_web\"\n patterns_dir => [ \"/tmp/logstash/patterns\" ]\n pattern => \"%{TELEFENGINX}\"\n }\n}\n\noutput {\n elasticsearch { embedded => true }\n}\n", :reason=>"The setting typein plugingrokis obsolete and is no longer available. You can achieve this same behavior with the new conditionals, like:if [type] == "sometype" { grok { ... } }. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}