Skip to content

Instantly share code, notes, and snippets.

@sakalajuraj
Last active August 1, 2020 00:48
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save sakalajuraj/6339942 to your computer and use it in GitHub Desktop.
Save sakalajuraj/6339942 to your computer and use it in GitHub Desktop.
Logstash filter for squid log
# Content of the file /etc/logstash/conf.d/squid.conf
# Squid logs MAC address of the source host
# Logs are sending by syslog-ng raw without standard log headers
input {
tcp {
host => "xxx.xxx.xxx.xxx"
port => xxxx
type => SQUID
tags => [SQUID]
}
}
filter {
if [type] == "SQUID" {
grok {
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:request_msec:float} %{IPORHOST:src_ip} %{COMMONMAC:src_mac} %{WORD}/%{NUMBER:response_status:int} %{NUMBER:response_size:int} %{WORD:http_method} (%{URIPROTO:http_proto}://)?%{IPORHOST2:dst_host}(?::%{POSINT:port})?(?:%{NOTSPACE:uri_param})? %{USERNAME:user} %{WORD}/(%{IPORHOST:dst_ip}|-) %{GREEDYDATA:content_type}" ]
}
date {
match => [ "timestamp", "UNIX" ]
remove_field => [ "timestamp" ]
}
if [dst_ip] {
geoip {
source => "dst_ip"
target => "dst_geoip"
database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
fields => [ "country_code2", "country_name" ]
}
}
}
}
output {
if [type] == "SQUID" {
elasticsearch {
flush_size => 1000
protocol => "transport"
host => "xxx.xxx.xxx.xxx"
cluster => "xxxxx"
index => "logstash-squid-%{+YYYY.MM.dd}"
}
}
}
@RuBiCK
Copy link

RuBiCK commented Jan 12, 2016

I'm trying to use your filter but I get the following error:
pattern %{IPORHOST2:dst_host} not defined"}

It could be related with the version of logstash-patterns-core?

@xzxpurple2017
Copy link

Should be %{IPORHOST}

Try this instead:

%{NUMBER:timestamp}%{SPACE}%{NUMBER:request_msec:float} %{IPORHOST:src_ip} %{WORD}/%{NUMBER:response_status:int} %{NUMBER:response_size} %{WORD:http_method} (%{URIPROTO:http_proto}://)?%{IPORHOST:dst_host}(?::%{POSINT:port})?(?:%{NOTSPACE:uri_param})? %{USERNAME:user} %{WORD}/(%{IPORHOST:dst_ip}|-)%{GREEDYDATA:content_type}

@genevera
Copy link

genevera commented Dec 5, 2016

@r57shell - thanks; the original squid grok pattern wasn't working for me either but yours does!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment