Skip to content

Instantly share code, notes, and snippets.

@thejuan

thejuan/IIS Logstash Grok

Created February 21, 2014 18:16
Show Gist options
  • Star 14 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save thejuan/9139946 to your computer and use it in GitHub Desktop.
Save thejuan/9139946 to your computer and use it in GitHub Desktop.
Download ZIP
A Logstash Grok filter for IIS (W3C default fields + bytes sent)
Raw
IIS Logstash Grok
filter{
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"]
}
}
@esoterydactyl
Copy link

thanks!

@ptheodorou1987
Copy link

Very helpful! Thank you

@bradvido
Copy link

bitchin

@danktec
Copy link

danktec commented May 4, 2017

Time saver

@DougSchmidt-AI
Copy link

This is a great start.

But be forewarned that different IIS versions have different different default W3C log configurations, so your IIS logs might not quite match. If you are aggregating IIS logs with different configurations, consider using an array of patterns in the match predicate.

grok {
  match => { "message" => [
	"%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken:int}",
	"%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{NOTSPACE:computername} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:protocol} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{IPORHOST:cshost} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytessent:int} %{NUMBER:bytesrecvd:int} %{NUMBER:timetaken:int}"
  ] }
}
date {
  match => [ "log_timestamp", "ISO8601" ]
  target => "@timestamp"
}
geoip {
  source => "clienthost"
}

@RobertoFlores
Copy link

Ok, i end here because i was triying to config logstash for IIS version 8.5, here my final config
filter{
grok {
match => ["message","%{DATE:date} %{TIME:time} %{IPORHOST:s-ip} %{WORD:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken:int}"]
}
}

@pete-leese
Copy link

Hey @RobertoFlores - here is my Grok query but for some reason it cannot find a match when I have the brackets in the Referrer and user agent name - I cant see anything different from what you are doing above though?

%{TIMESTAMP_ISO8601:logtime} %{WORD:s-sitename} %{WORD:s-computername} %{IPORHOST:s-ip} %{WORD:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs-version} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{IPORHOST:cs-host} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:c-win32-status} %{NUMBER:sc-bytes} %{NUMBER:cs-bytes} %{NUMBER:time-taken}

Example log item:

2018-02-02 00:01:32 W3SVC1 UKAPPSVR 172.18.131.173 GET /123/I/Home/PLMonstants - 80 Joe+Bloggs 172.18.17.185 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.1;+Trident/7.0;+rv:11.0)+like+Gecko https://blahblah.co.uk/theappname/live/app/thingy localhost 200 0 0 3393 2644 90

was using http://grokconstructor.appspot.com/do/match to validate?

Any ideas what I could be doing wrong?

@CDonoghue94
Copy link

Take out the brackets for cs(User-Agent) and cs(Referer), change to something like cs-User-Agent

@pete-leese
Copy link

Not that easy I’m afraid. We have over 200 servers which are also feeding into splunk.

Are there any ways of handling this scenario by configuring grok to ignore the brackets but still match?

Cheers

Pete

@hebertviana
Copy link

Hi

in version 10 of iis with all fields set, you can use this filter here.

%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-Query} %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}

@pshell-coder
Copy link

pshell-coder commented Jul 28, 2020 •
edited

Hi

in version 10 of iis with all fields set, you can use this filter here.

%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-Query} %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}

This works fine but sometimes I see grok parsing is failing. This is because either Cookie or CS-Referer sometimes they are blank or sometimes those fields contains string.
Any idea @hebertviana ??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment