Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
pfsense2-2.grok
# GROK match pattern for logstash.conf filter: %{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}
# GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):
# GROK Patterns for pfSense 2.2 Logging Format
#
# Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)
# Edited 14 Feb 2015 by E. Paul
#
# Usage: Use with following GROK match pattern
#
# %{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}
LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
IP_SPECIFIC_DATA (%{IPv4_SPECIFIC_DATA}|%{IPv6_SPECIFIC_DATA})
IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),
IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
PROTOCOL_DATA (%{TCP_DATA}|%{UDP_DATA}|%{ICMP_DATA})
TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})
UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})
ICMP_DATA (%{ICMP_TYPE}%{ICMP_RESPONSE})
ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
ICMP_RESPONSE (%{ICMP_ECHO_REQ_REPLY}|%{ICMP_UNREACHPORT}| %{ICMP_UNREACHPROTO}|%{ICMP_UNREACHABLE}|%{ICMP_NEED_FLAG}|%{ICMP_TSTAMP}|%{ICMP_TSTAMP_REPLY})
ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})
ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})
ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})
ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})
ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})
ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})
ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})
@bradvido

This comment has been minimized.

Copy link

commented Jun 24, 2015

Nice work! Is there any plan to include ICMPv6 in this?

@rudibroekhuizen

This comment has been minimized.

Copy link

commented Sep 19, 2015

Very nice!

Please take a look at my fork at https://gist.github.com/rudibroekhuizen/bb1fb0182fb860a1cfda. I've added a pattern for the pfsync protocol, which is used to sync two pfsense firewalls.

One line changed:
PROTOCOL_DATA (%{TCP_DATA}|%{UDP_DATA}|%{ICMP_DATA}|%{PFSYNC_DATA})

One line added:
PFSYNC_DATA (datalength=%{INT:data_length})

An pfsync event looks like this:
9,16777216,,1000000103,em0,match,block,in,4,0x10,,255,8086,0,DF,240,pfsync,636,192.168.1.1,192.168.1.2,datalength=616

@Shifter2600

This comment has been minimized.

Copy link

commented Oct 4, 2015

I have tried a few tweaks but for th elife of me can not get fields to come in as INT or NUMBER.
I have been focusing on this line
IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
I have tried
IP_DATA (%{NUMBER:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
IP_DATA (%{NUMBER:length:int}),(%{IP:src_ip}),(%{IP:dest_ip}),

But logstash always recognizes it as String anyone else having this issue.
Majority of fields are string the only number ones are the
geoip.latitude,
geoip.area_code,
geoip.longitude,
geoip.dma_code,
_score
fields.

@bradvido

This comment has been minimized.

Copy link

commented Jan 1, 2016

I noticed that ack_number is not always present, so I had to change it to (%{INT:ack_number})?
See my fork

@arodd

This comment has been minimized.

Copy link

commented Mar 29, 2016

I added optional anchor/ecn fields in my fork after reviewing parse failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.