Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Grok Sophos UTM 9.x Pattern (for logstash) (Simple)
filter {
grok {
pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{YEAR}): (?:%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): (?<messagebody>(?:id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" tcpflags=\"%{DATA:utm_ulogd_tcpflags}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" reason=\"%{DATA:utm_ips_reason}\" group=\"%{INT:utm_ips_group}\" srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" sid=\"%{INT:utm_ips_sid}\" class=\"%{DATA:utm_ips_class}\" priority=\"%{INT:utm_ips_priority}\" generator=\"%{INT:utm_ips_generator}\" msgid=\"%{INT:utm_ips_msgid}\"|\"%{DATA:utm_pluto_vpnname}\"\[%{INT}\] %{IP:utm_pluto_vpnremoteip} #%{INT}: %{GREEDYDATA:utm_pluto_message}|%{GREEDYDATA}))']
type => "sophosutm"
}
}
@martinseener
Copy link
Author

martinseener commented Oct 2, 2013

Extremly enhanced Grok Pattern for Sophos UTM ulogd and pluto log messages in various formats and separated the messagebody from the header

@martinseener
Copy link
Author

martinseener commented May 6, 2014

Updated Gist (Rev11) for th new log format for UTM 9.111-7

@driegel
Copy link

driegel commented Apr 13, 2015

Well, with today's versions of Logstash it is also possible to use 'kv' for parsing the lines.
However, I feel it is necessary to parse 'date and time' and 'host' ... with 'grok', then find the 'id=' token with if and then use 'kv'.

@breetzel
Copy link

breetzel commented May 20, 2015

_grokparsefailure for Firmware-Version: 9.310

@ericdude101
Copy link

ericdude101 commented Jun 27, 2015

Is there an updated version of this logic pattern available? I'm on version 9.313-3 of sophos UTM and also receiving a "grokparsefailure"

@ericdude101
Copy link

ericdude101 commented Sep 23, 2015

BUMP

@dhardison
Copy link

dhardison commented Sep 24, 2015

Also getting the "grokparsefailure" with Sophos UTM 9.3

Has anyone found a working grok/pattern?

@Braggy85
Copy link

Braggy85 commented Oct 6, 2015

BUMP BUMP! This would be great!

@bbybblank
Copy link

bbybblank commented Jan 30, 2016

UTM 9.353-4

I wrote this logstash filter for my UTM home setup. It does grab the date, uses the KV filter to get all the key value pairs. I use the logging subsystem to set the "type" of event. E.g. firewall logs = ulogd, proxy logs = httpproxy... On my UTM, I simply add remote syslog to the remote logging settings and point at my logstash machine. (I used port 5140) The only other "special" parsing I added were url_domain and protocol for the proxy logs. This works out of the box for: packet filter, proxy, end point web protection and ips (These are all log formats that use the DATE, subsystem, key=value format.

Hope this helps some people. any other kv style logs should work too - and it wouldn't be too hard to add other log types like dhcpd or named.

Bob.

I am new to GIT so I have to figure out how to post the code here... but here are my UTM settings...
utm settings

@bbybblank
Copy link

bbybblank commented Jan 30, 2016

input {
tcp {
port => 5140
}
udp {
port => 5140
}
}

filter {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}[%{NUMBER:syslog_pid}]: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
kv {
source => "syslog_message"
}
mutate {
replace => [ "type", "%{syslog_program}" ]
remove_field => [ "syslog_message", "syslog_timestamp" ]
}

if [type] == "httpproxy" {
    grok { match => { "url" => "(?<protocol>https?)://%{IPORHOST:url_domain}/" } }

}

} # end of filter

output {

elasticsearch {
    hosts => ["localhost:9200"]
    index => "utm-%{+YYYY.MM.dd}"

}

stdout { codec => rubydebug }

}

@bbybblank
Copy link

bbybblank commented Jan 30, 2016

OK - I guess I create a fork to properly share my code bbybblank/gist:19ccbfdbf7d56fd747c1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment