Instantly share code, notes, and snippets.

Embed
What would you like to do?
Easy IIS log file format specification for goaccess.
goaccess -f u_ex150629.log --log-format "$(cat u_ex150629.log | ./goiisformat.sh)" --date-format '%Y-%m-%d' --time-format '%H:%M:%S'
#!/usr/bin/env sh
while read line; do
if [[ $line == \#Fields:* ]]; then
line=${line/\#Fields: /}
line=${line/date/%d}
line=${line/time/%t}
line=${line/s-sitename/%^}
line=${line/s-computername/%^}
line=${line/s-ip/%^}
line=${line/cs-method/%m}
line=${line/cs-uri-stem/%U}
line=${line/cs-uri-query/%^}
line=${line/s-port/%^}
line=${line/cs-username/%^}
line=${line/c-ip/%h}
line=${line/cs-version/%H}
line=${line/cs(User-Agent)/%u}
line=${line/cs(Cookie)/%^}
line=${line/cs(Referer)/%R}
line=${line/cs-host/%^}
line=${line/sc-status/%s}
line=${line/sc-substatus/%^}
line=${line/sc-win32-status/%^}
line=${line/sc-bytes/%b}
line=${line/cs-bytes/%^}
line=${line/time-taken/%L}
echo $line
exit;
fi
done
@felquis

This comment has been minimized.

felquis commented Apr 5, 2016

My goiisformat.sh became

#!/usr/bin/env sh

while read line; do
  if [[ $line == \#Fields:* ]]; then
    line=${line/\#Fields: /}
    line=${line/date/%d}
    line=${line/time/%t}
    line=${line/s-sitename/%^}
    line=${line/cs-method/%m}
    line=${line/cs-uri-stem/%U}
    line=${line/cs-uri-query/%^}
    line=${line/s-port/%^}
    line=${line/cs-username/%^}
    line=${line/c-ip/%h}
    line=${line/cs(User-Agent)/%u}
    line=${line/cs(Cookie)/%^}
    line=${line/cs(Referer)/%R}
    line=${line/cs-host/%^}
    line=${line/sc-status/%s}
    line=${line/sc-substatus/%^}
    line=${line/sc-win32-status/%^}
    line=${line/sc-bytes/%b}
    line=${line/cs-bytes/%^}
    line=${line/time-taken/%L}
    echo $line
    exit;
  fi
done

And I had to remove the --time-format value to make it work.. based on this comment allinurl/goaccess#233 (comment)

@elproducto

This comment has been minimized.

elproducto commented Jan 14, 2017

Thanks to you both for your contribution to help process IIS Log files in GoAccess. You have inspired me to contribute as well. I am using the following syntax to process multiple IIS file. However all my logs prior to 2016 are throwing error when using the this syntax, error stating content does not match specifier.

I decided to combine my logs into a single W3C file using Microsoft Log Parser, however to process the file I needed to modify the goiisformat script to work with file outputted by the Parser. I have shared the modifications in a forked script gomspformat.sh . The gomspformat script will provide specifiers for W3C file generated by Microsoft Log Parser. I used the Log parser to combine my IIS logs into a single W3C file, then used GoAccess and the gomspformat to process this file.

I wonder if this method would work to process multiple Server IIS logs into a single report.

Multiple IIS Logs via GoAccess and any Soheilpro's GoIISFormat Script

cat  u_ex* | goaccess --log-format "$(cat u_ex170114.log | ~/goiisformat.sh)" --date-format '%Y-%m-%d' --time-format '%H:%M:%S'

In this example a random file was selected for the string "cat u_ex170114.log" to be piped to the script
Just as an FYI you will need to ensure that the goiisformat.sh script created is not in DOS format. Dos formatted files will throw an error message with "command not found". I used Nano inside of Cygwin to make the goiishformat.sh script file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment