Instantly share code, notes, and snippets.

Embed
What would you like to do?
pfsense2-2.grok
# GROK match pattern for logstash.conf filter: %{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}
# GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):
# GROK Patterns for pfSense 2.2 Logging Format
#
# Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)
# Edited 14 Feb 2015 by E. Paul
#
# Usage: Use with following GROK match pattern
#
# %{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}
LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
IP_SPECIFIC_DATA (%{IPv4_SPECIFIC_DATA}|%{IPv6_SPECIFIC_DATA})
IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),
IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
PROTOCOL_DATA (%{TCP_DATA}|%{UDP_DATA}|%{ICMP_DATA})
TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})
UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})
ICMP_DATA (%{ICMP_TYPE}%{ICMP_RESPONSE})
ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
ICMP_RESPONSE (%{ICMP_ECHO_REQ_REPLY}|%{ICMP_UNREACHPORT}| %{ICMP_UNREACHPROTO}|%{ICMP_UNREACHABLE}|%{ICMP_NEED_FLAG}|%{ICMP_TSTAMP}|%{ICMP_TSTAMP_REPLY})
ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})
ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})
ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})
ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})
ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})
ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})
ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})
@bradvido

This comment has been minimized.

bradvido commented Jun 24, 2015

Nice work! Is there any plan to include ICMPv6 in this?

@rudibroekhuizen

This comment has been minimized.

rudibroekhuizen commented Sep 19, 2015

Very nice!

Please take a look at my fork at https://gist.github.com/rudibroekhuizen/bb1fb0182fb860a1cfda. I've added a pattern for the pfsync protocol, which is used to sync two pfsense firewalls.

One line changed:
PROTOCOL_DATA (%{TCP_DATA}|%{UDP_DATA}|%{ICMP_DATA}|%{PFSYNC_DATA})

One line added:
PFSYNC_DATA (datalength=%{INT:data_length})

An pfsync event looks like this:
9,16777216,,1000000103,em0,match,block,in,4,0x10,,255,8086,0,DF,240,pfsync,636,192.168.1.1,192.168.1.2,datalength=616

@Shifter2600

This comment has been minimized.

Shifter2600 commented Oct 4, 2015

I have tried a few tweaks but for th elife of me can not get fields to come in as INT or NUMBER.
I have been focusing on this line
IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
I have tried
IP_DATA (%{NUMBER:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
IP_DATA (%{NUMBER:length:int}),(%{IP:src_ip}),(%{IP:dest_ip}),

But logstash always recognizes it as String anyone else having this issue.
Majority of fields are string the only number ones are the
geoip.latitude,
geoip.area_code,
geoip.longitude,
geoip.dma_code,
_score
fields.

@bradvido

This comment has been minimized.

bradvido commented Jan 1, 2016

I noticed that ack_number is not always present, so I had to change it to (%{INT:ack_number})?
See my fork

@arodd

This comment has been minimized.

arodd commented Mar 29, 2016

I added optional anchor/ecn fields in my fork after reviewing parse failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment