Web Server Log File Analysis

The PiwikServer stats are very good at tracking people who consent to be tracked, however they don't track bots, and abusive people who don't want to be tracked. The difference in the traffic is quite stark, for example on 26th June 2013 the javascript based Piwik stats reported:

• 1106 visits
• 3314 page views

Where as the raw log files showed ten times as many page views:

• 7155 visits
• 32688 page views

Therefore to get a handle on what the web servers are doing, as opposed to what really people are doing on the site, we need some tools other than Piwik, this page was created on ticket:555.

webalizer

PenguinServer is set up to get a copy of the Nginx awstats.log file very day via logrotate, see AwStatsInstall#Copyingthelogstopenguin and on Penguin there is a puffin user account which has this crontab:

05 07 * * * /usr/local/bin/puffin-webalizer

Which runs /usr/local/bin/puffin-webalizer which contains:

#!/bin/bash

DATE=$(date "+%Y-%m-%d")
LOG_FILE=/home/puffin/nginx/puffin-nginx-$DATE.log
STATS_DIR=/web/penguin.transitionnetwork.org/www/webalizer/puffin

cd $STATS_DIR
webalizer -p -n transitionnetwork.org -o $STATS_DIR $LOG_FILE

These logs are password protected and available at ​https://penguin.transitionnetwork.org/webalizer/puffin/

logstalgia

This allows a realtime display of log files, install logstalgia on your local machine, for example:

sudo aptitude install logstalgia

And then pipe the logs into it via ssh, for example these are the commands to see a real time display from the 3 servers:

ssh puffin.webarch.net sudo tail -f /var/log/nginx/awstats.log | logstalgia --sync
ssh parrot.webarch.net sudo tail -f /home/*/logs/access.log | logstalgia --sync
ssh penguin.webarch.net sudo tail -f /var/log/nginx/*.access.log | logstalgia --sync

The following screen shot doesn't do it justice, the last two numbers of the IP address have been removed from this image:

For more info see ​https://code.google.com/p/logstalgia/ and the videos here ​https://www.youtube.com/user/Logstalgia

goaccess

To get an overview of a log file you can use goaccess of the server to load a specific log file, for example on puffin, this is the current log:

goaccess -f /var/log/nginx/awstats.log

And this is yesterdays:

goaccess -f /var/log/nginx/awstats.log.1

This displays totals like this:

When we upgrade to Wheezy, see ticket:535 we should set up Goaccess to generate a HTML / email report per day.

For more information see the goaccess web site at ​http://goaccess.prosoftcorp.com/