Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Grok Sophos UTM 9.x Pattern (for logstash) (Simple)
filter {
grok {
pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{YEAR}): (?:%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): (?<messagebody>(?:id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" tcpflags=\"%{DATA:utm_ulogd_tcpflags}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" reason=\"%{DATA:utm_ips_reason}\" group=\"%{INT:utm_ips_group}\" srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" sid=\"%{INT:utm_ips_sid}\" class=\"%{DATA:utm_ips_class}\" priority=\"%{INT:utm_ips_priority}\" generator=\"%{INT:utm_ips_generator}\" msgid=\"%{INT:utm_ips_msgid}\"|\"%{DATA:utm_pluto_vpnname}\"\[%{INT}\] %{IP:utm_pluto_vpnremoteip} #%{INT}: %{GREEDYDATA:utm_pluto_message}|%{GREEDYDATA}))']
type => "sophosutm"
}
}
@martinseener
Copy link
Author

martinseener commented May 6, 2014

Updated Gist (Rev11) for th new log format for UTM 9.111-7

@driegel
Copy link

driegel commented Apr 13, 2015

Well, with today's versions of Logstash it is also possible to use 'kv' for parsing the lines.
However, I feel it is necessary to parse 'date and time' and 'host' ... with 'grok', then find the 'id=' token with if and then use 'kv'.

@breetzel
Copy link

breetzel commented May 20, 2015

_grokparsefailure for Firmware-Version: 9.310

@ericdude101
Copy link

ericdude101 commented Jun 27, 2015

Is there an updated version of this logic pattern available? I'm on version 9.313-3 of sophos UTM and also receiving a "grokparsefailure"

@ericdude101
Copy link

ericdude101 commented Sep 23, 2015

BUMP

@dhardison
Copy link

dhardison commented Sep 24, 2015

Also getting the "grokparsefailure" with Sophos UTM 9.3

Has anyone found a working grok/pattern?

@Braggy85
Copy link

Braggy85 commented Oct 6, 2015

BUMP BUMP! This would be great!

@bbybblank
Copy link

bbybblank commented Jan 30, 2016

UTM 9.353-4

I wrote this logstash filter for my UTM home setup. It does grab the date, uses the KV filter to get all the key value pairs. I use the logging subsystem to set the "type" of event. E.g. firewall logs = ulogd, proxy logs = httpproxy... On my UTM, I simply add remote syslog to the remote logging settings and point at my logstash machine. (I used port 5140) The only other "special" parsing I added were url_domain and protocol for the proxy logs. This works out of the box for: packet filter, proxy, end point web protection and ips (These are all log formats that use the DATE, subsystem, key=value format.

Hope this helps some people. any other kv style logs should work too - and it wouldn't be too hard to add other log types like dhcpd or named.

Bob.

I am new to GIT so I have to figure out how to post the code here... but here are my UTM settings...
utm settings

@bbybblank
Copy link

bbybblank commented Jan 30, 2016

input {
tcp {
port => 5140
}
udp {
port => 5140
}
}

filter {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}[%{NUMBER:syslog_pid}]: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
kv {
source => "syslog_message"
}
mutate {
replace => [ "type", "%{syslog_program}" ]
remove_field => [ "syslog_message", "syslog_timestamp" ]
}

if [type] == "httpproxy" {
    grok { match => { "url" => "(?<protocol>https?)://%{IPORHOST:url_domain}/" } }

}

} # end of filter

output {

elasticsearch {
    hosts => ["localhost:9200"]
    index => "utm-%{+YYYY.MM.dd}"

}

stdout { codec => rubydebug }

}

@bbybblank
Copy link

bbybblank commented Jan 30, 2016

OK - I guess I create a fork to properly share my code bbybblank/gist:19ccbfdbf7d56fd747c1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment