A little script to parse http server logs with Piwik (piwik.org) script. Log files are in a given directory with sub-directories for each website present in Piwik. Each website has an idsite in Piwik and this idsite is passed to parser script via a piwik.conf file present in website directory

README.md

Script parsing logs to Piwik analytics

Create web site

1. Create website in Piwik and get its idsite
2. Create a directory for this website in /home/analytics/website_logs/ and create in it a directory parsed

For example.org, create /home/analytics/website_logs/example.org dir and /home/analytics/website_logs/example.org/parsed

3. Write a file piwik.conf in this directory with almost this line: idsite={{idsite}}

Parse logs

1. Puts log files in directory of required website and have fun

A cron launch parse_analytic_logs script. This script parse logs file through the script provided by Piwik with given parameter idsite. source for script provided by Piwik : http://piwik.org/docs/log-analytics-tool-how-to

parse_analytic_logs_to_piwik

#! /bin/bash

# =======================================================================
#            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
#                    Version 2, December 2004
#
# Copyright © 2014 Sebastien Ballarin <sebastien.ballarin@gmail.com>
#
# Everyone is permitted to copy and distribute verbatim or modified
# copies of this license document, and changing it is allowed as long
# as the name is changed.
#
#            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
#   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
#
#             0. You just DO WHAT THE FUCK YOU WANT TO.
# =======================================================================



for site in /home/analytics/website_logs/*
do
  if [ -d $site ]; then
    echo "$site analytic logs"
    
    if [ -f $site/piwik.conf ]; then
      piwik_id=$(cat $site/piwik.conf | grep idsite | sed 's/idsite=//')

      for file in $site/*.log
      do
        if [ -f $file ]; then
          filename=$(basename $file .log)
          echo "parsing file : $file for site with piwik ID $piwik_id ..."
          python /var/www/piwik/misc/log-analytics/import_logs.py --url=http://analytics.sebthemonster.com --idsite=$piwik_id --enable-http-errors --enable-http-redirects $file > /home/analytics/logs/$filename.parsing.log && mv $file $site/parsed
        fi
      done
    fi 
  fi
done