[[PageOutline(2-5, Table of Contents, floated)]]
= AWStats =

== Copying the logs to penguin ==

The nginx logs on wiki:PuffinServer are copied to wiki:PenguinServer using scp, in {{{/etc/logrotate.d/nginx}}} the {{{nginx-logs}}} script is run just before the logs are rotated:

{{{
        prerotate
                /usr/local/bin/nginx-logs
                /usr/local/bin/50x-errors chris@webarchitects.co.uk
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
}}}

This script contains:

{{{
#!/bin/bash

DATE=$(date "+%Y-%m-%d")
LOG_FILE=/var/log/nginx/access.log
REMOTE_FILE=puffin-nginx-$DATE.log

scp $LOG_FILE penguin:nginx/$REMOTE_FILE
}}}

It depends on the {{{/root/.ssh/config}}} file on puffin containing:

{{{
Host penguin
  Hostname penguin.webarch.net
  User puffin
}}}

And the puffin root users public key being on penguin.

On penguin the {{{/etc/ssh/sshd_config}}} file contains:

{{{
AllowGroups sudo sshaccess
}}}

And the puffin user on penguin was created with no password and in the sshaccess group:

{{{
adduser --disabled-password --ingroup sshaccess puffin
}}}

Then the ssh public key was copied to {{{/home/puffin/.ssh/authorized_keys}}} on penguin and this was added at the start so it can only be used from puffin's IP address:

{{{
from="puffin.webarch.net" ssh-rsa AAAA...
}}}

The log files that are created on penguin are in {{{/home/puffin/nginx/}}} and have a file name based on the date they are created, eg:

{{{
-rw-r----- 1 puffin puffin  11M Jun 22 06:25 puffin-nginx-2013-06-22.log
}}}

== Processing the logs ==

We don't want to keep IP addresses in the log files -- we want to remove them so the stats can be made public, and the perl script http://wiki.opennicproject.org/Tier2ConfigObfuscatingLogs looks like it will do the job:

{{{
#! /usr/bin/perl
#
# blurAddys.pl - Obfuscate IP addresses in a file
#
# cat some.log | blurAddys.pl > some_blurred.log
#
#####################################################################
use strict;

while(<STDIN>)
{
	s/\d{1,3}(\.|-)\d{1,3}(\.|-)\d{1,3}(\.|-)\d{1,3}/XX$1XX$2XX$3XX/g;
	#s/([0-9A-Fa-f]{4}:[0-9A-Fa-f:]+:[0-9A-Fa-f]{1,4})([^:0-9A-Fa-f])/XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX$2/g;
	print $_;
}
}}}

The regular expression for ipv6 addresses was matching the date so that has been commented out. 

So the above perl script was saved to {{{/usr/local/bin/blurAddys.pl}}}

== Awstats install ==

Deb installed:

{{{
aptitude search awstats

The following NEW packages will be installed:
  awstats libnet-xwhois-perl{a}
}}}