Example log4j logstash configuration

gistfile1.rb

# one way w/o web interface:
# java -jar lib/playbooks/servers/files/logstash-1.1.1-monolithic.jar agent -f lib/playbooks/servers/files/logstash.conf
# another with a web interface(http://localhost:9292) :
# java -jar lib/playbooks/servers/files/logstash-1.1.1-monolithic.jar agent -f lib/playbooks/servers/files/logstash.conf -- web --backend 'elasticsearch:///?local'
# nc localhost 3333 < logfile
#
# When you are all done, you can delete the 'data' folder to delete the database.

input {
  tcp {
    type => "log4j"
    port => 3333
  }
  stdin {
    type => "log4j"
  }
}

filter {
  grok {
    type => "log4j"
    pattern => "%{LOGLEVEL:severity}\s+%{WORD:category} *-* *%{GNDATE:timestamp} \[\w+\[%{GREEDYDATA:thread},.*\]\] %{GREEDYDATA:message}"
    patterns_dir => "lib/playbooks/servers/files/patterns"
    add_tag => "log4j"
  }

  date {
    type => "log4j"
    timestamp => "MM-dd-yyyy hh:mm:ss.SSS a Z"
    exclude_tags => "_grokparsefailure"
  }

  # Catches normal space indented type things, probably could be removed b/c the other multiline should do everythign we need
  multiline {
    type => "log4j"
    tags => ["_grokparsefailure"] # exclude anything we already handled
    pattern => ".*"
    what => "previous"
    add_tag => "notgrok"
  }

}

output {
  elasticsearch { embedded => true }
}

The log4j input only works when events is sent over a TCP socket from a Log4j SocketAppender. If you are just trying to import an existing log file then you need the input as described here along with the filter.

Note above filter does not work with latest logstash 1.5.3 and I am struggling to find a fix for following error:

Unknown setting 'timestamp' for date {:level=>:error}

If project owner could help identify the changes needed that would be terrific.

How do you manage the spaces between patterns? IOW, how do you make grok so it knows that sometimes there is one space between log severity pattern and category pattern, and sometimes there are two spaces?