Skip to content

Instantly share code, notes, and snippets.

@louiszuckerman
Created November 7, 2011 22:18
Show Gist options
  • Save louiszuckerman/1346387 to your computer and use it in GitHub Desktop.
Save louiszuckerman/1346387 to your computer and use it in GitHub Desktop.
Logstash parser for ModSecurity/CRS entries in the Apache ErrorLog
Logstash Configuration...
input {
file {
format => "plain"
path => "/var/log/apache2/*error.log"
type => "apacheerror"
}
}
filter {
grok {
type => "apacheerror"
pattern => [ "%{MODSECAPACHEERROR}", "%{GENERICAPACHEERROR}" ]
patterns_dir => "/path/to/patterns/file/below"
}
date {
type => "apacheerror"
timestamp => "EEE MMM dd HH:mm:ss yyyy"
}
}
Grok patterns...
APACHEERRORTIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
APACHEERRORPREFIX \[%{APACHEERRORTIME:timestamp}\] \[%{NOTSPACE:apacheseverity}\] \[client %{IPORHOST:sourcehost}\]
GENERICAPACHEERROR %{APACHEERRORPREFIX} %{GREEDYDATA:message}
MODSECPREFIX %{APACHEERRORPREFIX} ModSecurity: %{NOTSPACE:modsecseverity}\. %{GREEDYDATA:modsecmessage}
MODSECRULEFILE \[file %{QUOTEDSTRING:rulefile}\]
MODSECRULELINE \[line %{QUOTEDSTRING:ruleline}\]
MODSECMATCHOFFSET \[offset %{QUOTEDSTRING:matchoffset}\]
MODSECRULEID \[id %{QUOTEDSTRING:ruleid}\]
MODSECRULEREV \[rev %{QUOTEDSTRING:rulerev}\]
MODSECRULEMSG \[msg %{QUOTEDSTRING:rulemessage}\]
MODSECRULEDATA \[data %{QUOTEDSTRING:ruledata}\]
MODSECRULESEVERITY \[severity %{QUOTEDSTRING:ruleseverity}\]
MODSECRULETAGS (?:\[tag %{QUOTEDSTRING:ruletag0}\] )?(?:\[tag %{QUOTEDSTRING:ruletag1}\] )?(?:\[tag %{QUOTEDSTRING:ruletag2}\] )?(?:\[tag %{QUOTEDSTRING:ruletag3}\] )?(?:\[tag %{QUOTEDSTRING:ruletag4}\] )?(?:\[tag %{QUOTEDSTRING:ruletag5}\] )?(?:\[tag %{QUOTEDSTRING:ruletag6}\] )?(?:\[tag %{QUOTEDSTRING:ruletag7}\] )?(?:\[tag %{QUOTEDSTRING:ruletag8}\] )?(?:\[tag %{QUOTEDSTRING:ruletag9}\] )?(?:\[tag %{QUOTEDSTRING}\] )*
MODSECHOSTNAME \[hostname %{QUOTEDSTRING:targethost}\]
MODSECURI \[uri %{QUOTEDSTRING:targeturi}\]
MODSECUID \[unique_id %{QUOTEDSTRING:uniqueid}\]
MODSECAPACHEERROR %{MODSECPREFIX} %{MODSECRULEFILE} %{MODSECRULELINE} (?:%{MODSECMATCHOFFSET} )?(?:%{MODSECRULEID} )?(?:%{MODSECRULEREV} )?(?:%{MODSECRULEMSG} )?(?:%{MODSECRULEDATA} )?(?:%{MODSECRULESEVERITY} )?%{MODSECRULETAGS}%{MODSECHOSTNAME} %{MODSECURI} %{MODSECUID}
@Maks3w
Copy link

Maks3w commented Dec 21, 2013

There is a bug in the date filter. Apache error uses milliseconds and the timestamp format should be:

EEE MMM dd HH:mm:ss.SSSSSS yyyy

If not logstash throws a warning because cannot parse the date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment