Skip to content

Instantly share code, notes, and snippets.

@dblessing
dblessing / gist:4530592
Created January 14, 2013 14:58
Nginx grok parsing
--Grok pattern--
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|-)\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) \"(?:%{URI:referrer}|-)\" %{QS:agent} (?:%{NUMBER:request_time}|-) (?:%{NUMBER:upstream_response_time}|-)
--Example failed log lines--
10.0.0.63 - - [14/Jan/2013:08:51:33 -0600] "GET /my-app/login?ticket=ST-483-pcqXUW2YnfWJdRMezMji-cas HTTP/1.1" 302 0 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)" 0.105 0.105 .
10.123.19.153 - - [14/Jan/2013:08:51:24 -0600] "GET /my-other-app/edit?phone=555-555-5555&email=fakemail%40yahoo.com&isFromBasicSearch=true HTTP/1.1" 200 4991 "http://mydomain.org/my-other-app/add" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .N
lurch__ joined the chat room.
[08:59am] lurch__: banana
lurch__ left the chat room.
[08:59am] asg: a drive-by fruiting
input {
lumberjack {
port => 6782
ssl_certificate => "/etc/lumberjack/lumberjack.crt"
ssl_key => "/etc/lumberjack/lumberjack.key"
type => "lumberjack"
}
}
/usr/lib/jvm/java-1.7.0-sun/bin/java -Xmx2560m -Xms2560m -jar logstash-1.1.9-monolithic.jar agent --config /etc/logstash/conf.d/ --log /var/log/logstash/logstash.log --grok-patterns-path /opt/logstash/patterns
Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (ENOENT) No such file or directory - /etc/lumberjack/lumberjack.crt
at org.jruby.RubyFile.initialize(org/jruby/RubyFile.java:333)
at org.jruby.RubyIO.open(org/jruby/RubyIO.java:1179)
at org.jruby.RubyKernel.open(org/jruby/RubyKernel.java:333)
at org.jruby.RubyIO.read(org/jruby/RubyIO.java:3611)
at org.jruby.RubyIO.read(org/jruby/RubyIO.java:3708)
at RUBY.initialize(jar:file:/opt/logstash/logstash-1.1.9-monolithic.jar!/gems/jls-lumberjack-0.0.7/lib/lumberjack/server.rb:39)
at RUBY.register(file:/opt/logstash/logstash-1.1.9-monolithic.jar!/logstash/inputs/lumberjack.rb:35)
at RUBY.run_input(file:/opt/logstash/logstash-1.1.9-monolithic.jar!/logstash/agent.rb:707)
at org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)
if $drop {
validate_bool($drop)
$opt_drop = " drop => ${drop}\n"
}
"logstash" : {
"template" : "logstash-*",
"order" : 0,
"settings" : {
"index.query.default_field" : "@message",
"index.number_of_replicas" : "1",
"index.number_of_shards" : "5",
"index.store.compress.tv" : "true",
"index.store.compress.stored" : "true"
},
"templates" : {
"logstash" : {
"template" : "logstash-*",
"order" : 0,
"settings" : {
"index.number_of_replicas" : "1",
"index.query.default_field" : "@message",
"index.store.compress.tv" : "true",
"index.store.compress.stored" : "true",
"index.number_of_shards" : "5"
''{ "@timestamp": "$time_iso8601", "@message": "$server_name - $remote_addr - $remote_user [$time_local] $request $status", "@type": "nginx" , "@fields": { "server_name": "$server_name", "remote_addr": "$remote_addr", "remote_user": "$remote_user", "request": "$request", "request_method": "$request_method", "status": "$status", "body_bytes_sent": "$body_bytes_sent", "http_referer": "$http_referer", "http_user_agent": "$http_user_agent", "request_time": "$request_time", "upstream_response_time": "$upstream_response_time" } }'';'
class::param1: 'foo'
class::param2: 'bar'
-------
class:
param1: 'foo'
param2: 'bar'