Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
A Logstash Grok filter for IIS (W3C default fields + bytes sent)
filter{
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"]
}
}
@esoterydactyl

This comment has been minimized.

Copy link

@esoterydactyl esoterydactyl commented Aug 7, 2015

thanks!

@ptheodorou1987

This comment has been minimized.

Copy link

@ptheodorou1987 ptheodorou1987 commented Jun 23, 2016

Very helpful! Thank you

@bradvido

This comment has been minimized.

Copy link

@bradvido bradvido commented Nov 17, 2016

bitchin

@dhdanno

This comment has been minimized.

Copy link

@dhdanno dhdanno commented May 4, 2017

Time saver

@DougSchmidt-AI

This comment has been minimized.

Copy link

@DougSchmidt-AI DougSchmidt-AI commented Dec 6, 2017

This is a great start.

But be forewarned that different IIS versions have different different default W3C log configurations, so your IIS logs might not quite match. If you are aggregating IIS logs with different configurations, consider using an array of patterns in the match predicate.

grok {
  match => { "message" => [
	"%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken:int}",
	"%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{NOTSPACE:computername} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:protocol} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{IPORHOST:cshost} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytessent:int} %{NUMBER:bytesrecvd:int} %{NUMBER:timetaken:int}"
  ] }
}
date {
  match => [ "log_timestamp", "ISO8601" ]
  target => "@timestamp"
}
geoip {
  source => "clienthost"
}
@RobertoFlores

This comment has been minimized.

Copy link

@RobertoFlores RobertoFlores commented Jan 1, 2018

Ok, i end here because i was triying to config logstash for IIS version 8.5, here my final config
filter{
grok {
match => ["message","%{DATE:date} %{TIME:time} %{IPORHOST:s-ip} %{WORD:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken:int}"]
}
}

@pete-leese

This comment has been minimized.

Copy link

@pete-leese pete-leese commented Mar 23, 2018

Hey @RobertoFlores - here is my Grok query but for some reason it cannot find a match when I have the brackets in the Referrer and user agent name - I cant see anything different from what you are doing above though?

%{TIMESTAMP_ISO8601:logtime} %{WORD:s-sitename} %{WORD:s-computername} %{IPORHOST:s-ip} %{WORD:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs-version} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{IPORHOST:cs-host} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:c-win32-status} %{NUMBER:sc-bytes} %{NUMBER:cs-bytes} %{NUMBER:time-taken}

Example log item:

2018-02-02 00:01:32 W3SVC1 UKAPPSVR 172.18.131.173 GET /123/I/Home/PLMonstants - 80 Joe+Bloggs 172.18.17.185 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.1;+Trident/7.0;+rv:11.0)+like+Gecko https://blahblah.co.uk/theappname/live/app/thingy localhost 200 0 0 3393 2644 90

was using http://grokconstructor.appspot.com/do/match to validate?

Any ideas what I could be doing wrong?

@CDonoghue94

This comment has been minimized.

Copy link

@CDonoghue94 CDonoghue94 commented Apr 2, 2018

Take out the brackets for cs(User-Agent) and cs(Referer), change to something like cs-User-Agent

@pete-leese

This comment has been minimized.

Copy link

@pete-leese pete-leese commented Jun 13, 2018

Not that easy I’m afraid. We have over 200 servers which are also feeding into splunk.

Are there any ways of handling this scenario by configuring grok to ignore the brackets but still match?

Cheers

Pete

@hebertviana

This comment has been minimized.

Copy link

@hebertviana hebertviana commented May 15, 2020

Hi

in version 10 of iis with all fields set, you can use this filter here.

%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-Query} %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}

@pshell-coder

This comment has been minimized.

Copy link

@pshell-coder pshell-coder commented Jul 28, 2020

Hi

in version 10 of iis with all fields set, you can use this filter here.

%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-Query} %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}

This works fine but sometimes I see grok parsing is failing. This is because either Cookie or CS-Referer sometimes they are blank or sometimes those fields contains string.
Any idea @hebertviana ??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment