-
-
Save martinseener/5247292 to your computer and use it in GitHub Desktop.
filter { | |
grok { | |
pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{YEAR}): (?:%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): (?<messagebody>(?:id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" tcpflags=\"%{DATA:utm_ulogd_tcpflags}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" reason=\"%{DATA:utm_ips_reason}\" group=\"%{INT:utm_ips_group}\" srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" sid=\"%{INT:utm_ips_sid}\" class=\"%{DATA:utm_ips_class}\" priority=\"%{INT:utm_ips_priority}\" generator=\"%{INT:utm_ips_generator}\" msgid=\"%{INT:utm_ips_msgid}\"|\"%{DATA:utm_pluto_vpnname}\"\[%{INT}\] %{IP:utm_pluto_vpnremoteip} #%{INT}: %{GREEDYDATA:utm_pluto_message}|%{GREEDYDATA}))'] | |
type => "sophosutm" | |
} | |
} |
Is there an updated version of this logic pattern available? I'm on version 9.313-3 of sophos UTM and also receiving a "grokparsefailure"
BUMP
Also getting the "grokparsefailure" with Sophos UTM 9.3
Has anyone found a working grok/pattern?
BUMP BUMP! This would be great!
UTM 9.353-4
I wrote this logstash filter for my UTM home setup. It does grab the date, uses the KV filter to get all the key value pairs. I use the logging subsystem to set the "type" of event. E.g. firewall logs = ulogd, proxy logs = httpproxy... On my UTM, I simply add remote syslog to the remote logging settings and point at my logstash machine. (I used port 5140) The only other "special" parsing I added were url_domain and protocol for the proxy logs. This works out of the box for: packet filter, proxy, end point web protection and ips (These are all log formats that use the DATE, subsystem, key=value format.
Hope this helps some people. any other kv style logs should work too - and it wouldn't be too hard to add other log types like dhcpd or named.
Bob.
I am new to GIT so I have to figure out how to post the code here... but here are my UTM settings...
input {
tcp {
port => 5140
}
udp {
port => 5140
}
}
filter {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}[%{NUMBER:syslog_pid}]: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
kv {
source => "syslog_message"
}
mutate {
replace => [ "type", "%{syslog_program}" ]
remove_field => [ "syslog_message", "syslog_timestamp" ]
}
if [type] == "httpproxy" {
grok { match => { "url" => "(?<protocol>https?)://%{IPORHOST:url_domain}/" } }
}
} # end of filter
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "utm-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
OK - I guess I create a fork to properly share my code bbybblank/gist:19ccbfdbf7d56fd747c1
_grokparsefailure for Firmware-Version: 9.310