Web Server Log File Analysis

The PiwikServer stats are very good at tracking people who consent to be tracked, however they don't track bots, and abusive people who don't want to be tracked. The difference in the traffic is quite stark, for example on 26th June 2013 the javascript based Piwik stats reported:

• 1106 visits
• 3314 page views

Where as the raw log files showed ten times as many page views:

• 7155 visits
• 32688 page views

Therefore to get a handle on what the web servers are doing, as opposed to what realy people are doing on the site, we need some tools other than Piwik.

logstalgia

This allows a realtime display of log files, install logstalgia on your local machine, for example:

sudo aptitude install logstalgia

And then pipe the logs into it via ssh, for example these are the commands to see a real time display from the 3 servers:

ssh puffin.webarch.net sudo tail -f /var/log/nginx/awstats.log | logstalgia --sync
ssh parrot.webarch.net sudo tail -f /home/*/logs/access.log | logstalgia --sync
ssh penguin.webarch.net sudo tail -f /var/log/nginx/*.access.log | logstalgia --sync

For more info see ​https://code.google.com/p/logstalgia/ and the videos here ​https://www.youtube.com/user/Logstalgia

goaccess

To get an overview of a log file you can use goaccess of the server to load a specific log fole, for example on puffin:

goaccess -f /var/log/nginx/awstats.log

This is good to displaying totals like this: