Skip to content

Instantly share code, notes, and snippets.

@hrchu

hrchu/gist:0cc5e783387f5453f528

Last active April 23, 2019 09:48
Show Gist options
  • Save hrchu/0cc5e783387f5453f528 to your computer and use it in GitHub Desktop.
Save hrchu/0cc5e783387f5453f528 to your computer and use it in GitHub Desktop.
Download ZIP
logstash-grok-pattern-squid
Raw
gistfile1.txt
# Follows the squid format in default:
# logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt
# http://www.squid-cache.org/Doc/config/logformat/
%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{USERNAME:user} %{WORD:squid_hierarchy_status}/%{IPORHOST:server_ip_or_peer_name} (?<mime_content_type>\S+\/\S+)
@ebuildy
Copy link

ebuildy commented Aug 4, 2015

This not working for https. (URI should be NOTSPACE):

1438694263.977 7356 172.17.1.161 TCP_MISS/200 0 CONNECT graph.facebook.com:443 - HIER_DIRECT/127.0.0.1 -

@ebuildy
Copy link

ebuildy commented Aug 4, 2015 

%{POSINT:timestamp}.%{POSINT:timestamp_ms}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{NOTSPACE:request_url} %{NOTSPACE:user} %{WORD:squid}/%{IP:server_ip} %{NOTSPACE:content_type}

@reticentnz
Copy link

I still get some failures on HTTPS urls with your pattern ebuildy. One example is:

1446816499.069 648 1.1.1.1 TCP_MISS/200 2415 CONNECT ent-shasta-rrs.symantec.com:443 - FIRSTUP_PARENT/185.46.212.34 -

@hiskia
Copy link

hiskia commented Nov 9, 2015

%{POSINT:timestamp}.%{POSINT:timestamp_ms}\s+%{NUMBER:response_time}\s+%{IPORHOST:user} %{WORD:result}/%{NUMBER:status_codes} %{NUMBER:transfer_size} %{WORD:request_method} (?=%{NOTSPACE:request_url})((?:(http://)%{IPORHOST:domain}(?::%{POSINT:port})?)(?:|%{URIPATH:url_path})(?:|%{URIPARAM:url_querystring})) (?:-|%{NOTSPACE:client_identity}) %{WORD:peering_code}/(?:-|%{NOTSPACE:peerhost}) (?:-|%{NOTSPACE:content_type})

@rafaelrb
Copy link

rafaelrb commented Sep 2, 2016

This one works even if squid is sending access logs to a remote syslog server.

(%{DATA:head}:)?\s+%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{NOTSPACE:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:transfer_size} %{NOTSPACE:http_method} (%{URIPROTO:proto}://)?%{NOTSPACE:request_url}(:%{INT:tcp_port})?(/%{NOTSPACE:url_path})?\s+%{NOTSPACE:client_identity}\s+%{NOTSPACE:peer_code}/%{NOTSPACE:peerhost}\s+%{NOTSPACE:content_type}

@brandond
Copy link

brandond commented Sep 19, 2016 •
edited
Loading

Here's one that properly breaks the request URI into scheme, host, port, and path. NOTSPACE can't be used for the host since it's greedy and will prevent the port and path from matching.

%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{NOTSPACE:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:transfer_size} %{NOTSPACE:http_method} (%{URIPROTO:url_scheme}://)?(?<url_host>\S+?)(:%{INT:url_port})?(/%{NOTSPACE:url_path})?\s+%{NOTSPACE:client_identity}\s+%{NOTSPACE:peer_code}/%{NOTSPACE:peerhost}\s+%{NOTSPACE:content_type}

@PinkIAndBlue
Copy link

Dear friends,
In my environment I have squid on pfsense 2.4.3 and I have problems to see application and url on squid logs in my logstash. Can you help me with this configuration? Anyone know how to send this complete log to logstash, can you send me the step by step? that's my sample log message. I don´t see the apllication
message:<166>Sep 18 08:57:17 (squid-1): 1537271837.019 119300 10.32.211.43 TCP_TUNNEL/200 424889 CONNECT www.reclameaqui.com.br:443 joaojbm HIER_DIRECT/23.4.77.209 -

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment