Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Example log4j logstash configuration
# one way w/o web interface:
# java -jar lib/playbooks/servers/files/logstash-1.1.1-monolithic.jar agent -f lib/playbooks/servers/files/logstash.conf
# another with a web interface(http://localhost:9292) :
# java -jar lib/playbooks/servers/files/logstash-1.1.1-monolithic.jar agent -f lib/playbooks/servers/files/logstash.conf -- web --backend 'elasticsearch:///?local'
# nc localhost 3333 < logfile
#
# When you are all done, you can delete the 'data' folder to delete the database.
input {
tcp {
type => "log4j"
port => 3333
}
stdin {
type => "log4j"
}
}
filter {
grok {
type => "log4j"
pattern => "%{LOGLEVEL:severity}\s+%{WORD:category} *-* *%{GNDATE:timestamp} \[\w+\[%{GREEDYDATA:thread},.*\]\] %{GREEDYDATA:message}"
patterns_dir => "lib/playbooks/servers/files/patterns"
add_tag => "log4j"
}
date {
type => "log4j"
timestamp => "MM-dd-yyyy hh:mm:ss.SSS a Z"
exclude_tags => "_grokparsefailure"
}
# Catches normal space indented type things, probably could be removed b/c the other multiline should do everythign we need
multiline {
type => "log4j"
tags => ["_grokparsefailure"] # exclude anything we already handled
pattern => ".*"
what => "previous"
add_tag => "notgrok"
}
}
output {
elasticsearch { embedded => true }
}
@OhadR
Copy link

OhadR commented Jun 4, 2015

the new versions of logstash accept log4j as input, so the grok filter is not needed anymore here. for example:
input {
log4j {
type => "log4j"
}
...
}

@farrukhnajmi
Copy link

farrukhnajmi commented Aug 21, 2015

The log4j input only works when events is sent over a TCP socket from a Log4j SocketAppender. If you are just trying to import an existing log file then you need the input as described here along with the filter.

Note above filter does not work with latest logstash 1.5.3 and I am struggling to find a fix for following error:

Unknown setting 'timestamp' for date {:level=>:error}

If project owner could help identify the changes needed that would be terrific.

@ZillaG
Copy link

ZillaG commented Jun 21, 2016

How do you manage the spaces between patterns? IOW, how do you make grok so it knows that sometimes there is one space between log severity pattern and category pattern, and sometimes there are two spaces?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment