wiki:PiwikServer

Version 37 (modified by chris, 4 years ago) (diff)

--

Piwik Web Stats

The server is available at:

https://stats.transitionnetwork.org/

It is currently running on wiki:PenguinServer#stats.transitionnetwork.org via Nginx, previously there was a live site on Quince and a dev site on Kiwi using Apache.

See the GeoIP install notes, ticket:470#comment:33 and the setup of the Mediawiki plugin ticket:470#comment:32 and the Nginx install ticket:470#comment:13 and cronjob ticket:470#comment:13 and the Nginx config ticket:470#comment:34

Issues

Updates

To update Piwik, backup the config file:

sudo -i
cd /web/stats.transitionnetwork.org/
cp piwik/config/config.ini.php .

There is an optional task of switching off the recording of stats, but it's not clear that the sites are busy enough to make this necessary, see ticket:418.

Download, extract, and update the database:

wget "http://piwik.org/latest.zip"
unzip latest.zip
cp config.ini.php piwik/config/
php /web/stats.transitionnetwork.org/piwik/index.php -- "module=CoreUpdater" 

Plugins

Do Not Track

Upgraded to version 0.3 on ticket:414

Penguin Server

See wiki:PenguinServer#stats.transitionnetwork.org and see ticket:470#comment:13 for the notes on the migration from Apache to Nginx. See also the initial research comments on the migration to Nginx, ticket:357#comment:14 ticket:357#comment:28.

Quince Server

The live install at https://stats.transitionnetwork.org/ on the NewLiveServer, the install was done on ticket:160.

Install

cd /web
cd stats.transitionnetwork.org/
wget http://piwik.org/latest.zip
unzip latest.zip
cd piwik
chown -R www-data:www-data tmp/
chown -R www-data:www-data config/
chown -R www-data:www-data themes/

Apache

Create stats.transitionnetwork.org.conf in cd /etc/apache2/sites-available/:

<VirtualHost *:80>
  ServerName stats.transitionnetwork.org
  ServerAlias stats.transitionnetwork.org.webarch.net
  ServerSignature Off
  UseCanonicalName On

  ErrorLog /var/log/apache2/stats.transitionnetwork.org_error.log
  ErrorLog /var/log/apache2/error.log
  SetEnvIf Remote_Addr "127\.0\.0\.1" loopback
  SetEnvIf Remote_Addr "kiwi\.transitionnetwork\.org" loopback
  CustomLog /var/log/apache2/stats.transitionnetwork.org_access.log combined env=!loopback
  CustomLog /var/log/apache2/access.log combined env=!loopback

  Redirect / https://stats.transitionnetwork.org/
</VirtualHost>

<VirtualHost *:443>
  ServerName stats.transitionnetwork.org
  ServerAlias stats.transitionnetwork.org.webarch.net
  ServerSignature Off
  UseCanonicalName On

  LogLevel warn
  ErrorLog /var/log/apache2/stats.transitionnetwork.org_ssl_error.log
  ErrorLog /var/log/apache2/error.log
  SetEnvIf Remote_Addr "127\.0\.0\.1" loopback
  SetEnvIf Remote_Addr "kiwi\.transitionnetwork\.org" loopback
  CustomLog /var/log/apache2/stats.transitionnetwork.org_ssl_access.log combined env=!loopback
  CustomLog /var/log/apache2/access.log combined env=!loopback

  SSLEngine on
  SSLCipherSuite HIGH
  SSLProtocol all -SSLv2
  SSLCertificateFile      /etc/ssl/transitionnetwork.org/transitionnetwork.org.pem
  SSLCertificateChainFile /etc/ssl/transitionnetwork.org/gandi.pem

  DocumentRoot "/web/stats.transitionnetwork.org/piwik"
  <Directory "/web/stats.transitionnetwork.org/piwik">
    DirectoryIndex index.php
    Options -Indexes
    AddType 'image/x-icon' .ico
    AddHandler php5-script .php
    AddType text/html .php
    SSLOptions +StdEnvVars
    AllowOverride None
    order allow,deny
    allow from all
  </Directory>
  <Directory "/web/stats.transitionnetwork.org/piwik/tmp">
    Options Indexes
    RemoveHandler .php
    AllowOverride None
    order allow,deny
    allow from all
  </Directory>
  <Location /config>
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
  </Location>

  <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
  </FilesMatch>
  <Directory /usr/lib/cgi-bin>
    SSLOptions +StdEnvVars
  </Directory>

</VirtualHost>

Enable it and restart:

/etc/apache2/sites-enabled
ln -s ../sites-available/stats.transitionnetwork.org.conf 35-stats.transitionnetwork.org.conf
apache2ctl configtest
apache2ctl restart

MySQL

As MySQL root user:

CREATE USER 'stats'@'localhost' IDENTIFIED BY '********';
GRANT USAGE ON * . * TO 'stats'@'localhost' IDENTIFIED BY '********' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
CREATE DATABASE IF NOT EXISTS `stats` ;
GRANT ALL PRIVILEGES ON `stats` . * TO 'stats'@'localhost';
FLUSH PRIVILEGES;

Web Install

MySQL settings:

  • database server: 127.0.0.1
  • login: stats
  • password:
  • database name: stats
  • table prefix: (blank - we don't need one, the default is piwik_)
  • adapter: PDO_MYSQL

Super User settings:

  • super user login: webproject
  • password:
  • email: webproject (at) transitionnetwork.org

Suggested JS code:

<!-- Piwik -->
<script type="text/javascript">
var pkBaseURL = (("https:" == document.location.protocol) ? "https://stats.transitionnetwork.org/" : "http://stats.transitionnetwork.org/");
document.write(unescape("%3Cscript src='" + pkBaseURL + "piwik.js' type='text/javascript'%3E%3C/script%3E"));
</script><script type="text/javascript">
try {
var piwikTracker = Piwik.getTracker(pkBaseURL + "piwik.php", 1);
piwikTracker.trackPageView();
piwikTracker.enableLinkTracking();
} catch( err ) {}
</script><noscript><p><img src="http://stats.transitionnetwork.org/piwik.php?idsite=1" style="border:0" alt="" /></p></noscript>
<!-- End Piwik Tracking Code -->

Suggested image code:

<!-- Piwik Image Tracker -->
<img src="https://stats.transitionnetwork.org/piwik.php?idsite=1&rec=1" style="border:0" alt="" />
<!-- End Piwik -->

Configure

In the General Settings page:

  • Allow Piwik archiving to trigger when reports are viewed from the browser: no
  • Reports for today (or any other Date Range including today) will be processed at most every: 1800

Set up cron:

crontab -u www-data -e

5 * * * * /web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh > /dev/null

Test it:

su www-data -c "sh /web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh"

This generated this error:

ALERT - script tried to increase memory_limit to 134217728 bytes which is above the allowed value (attacker 'REMOTE_ADDR not set', file '/web/stats.transitionnetwork.org/piwik/core/Piwik.php', line 947)

Which can be ignored because in /etc/php5/apache2/php.ini it's set to double that already, 256M, so I don't understand why it tried to raise it.

Things added to /web/stats.transitionnetwork.org/piwik/config/config.ini.php, taken from global.ini.php:

[General]
force_ssl_login = 1
feedback_email_address = "webproject (at) transitionnetwork.org"
assume_secure_protocol = 1

GA Import

First install Python 2.6 from source:

aptitude build-dep python2.5 
cd /usr/local/src
wget http://www.python.org/ftp/python/2.6.7/Python-2.6.7.tgz
tar -zxvf Python-2.6.7.tgz
cd Python-2.6.7
./configure --with-threads --enable-shared
make
make install 
ln -s /usr/local/lib/libpython2.6.so* /usr/lib/
cd /usr/local/src
wget 'http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e'
tar zxvf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
python setup.py install
cd /usr/local/src
wget "https://gdata-python-client.googlecode.com/files/gdata-2.0.14.tar.gz" --no-check-certificate
tar -zxvf gdata-2.0.14.tar.gz
cd gdata-2.0.14
python setup.py install
cd /usr/local/src
aptitude purge python-mysqldb
wget "http://downloads.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz"
tar -zxvf MySQL-python-1.2.3.tar.gz 
cd MySQL-python-1.2.3
vim site.cfg
 mysql_config = /usr/bin/mysql_config
aptitude install libmysqlclient-dev
python setup.py build
python setup.py install

Then the GA code and import:

cd /web/stats.transitionnetwork.org/
wget http://clearcode.cc/google2piwik-1.2.5.tgz
tar -zxvf google2piwik-1.2.5.tgz
cd google2piwik-1.2.5
cp google2piwik.conf.sample google2piwik.conf

Edit google2piwik.conf:

## Google Account Credentials
# If you don't know your site table_id type:
#                       ./google2piwik.py -p
[google]
user_login = transitiontownsnetwork [at] gmail.com
user_pass  = XXX
table_id   = ga:XXX

# MySQL Piwik Database configuration
[mysql]
db = stats
host = localhost
port = 3306
user = stats
passwd = XXX
table_prefix =

# Set export date range
[export]
start = 2008-01-01
end   = 2011-06-29

## Piwik configuration of site.
# For default Piwik installation with one website, site_id should be set to 1
[piwik]
site_id  = 1
site_url = http://transitionnetwork.org

Test and backup db:

./google2piwik.py -p
./google2piwik.py -c
mysql_backup

Then change the timezone to UTC for the site on https://stats.transitionnetwork.org/ run it in screen:

screen
./google2piwik.py --verbose=2

Kiwi Server

The Piwik server that is to be installed on the DevelopmentServer, the install was done on ticket:160

Config

In /web/piwik.transitionnetwork.org/piwik/config

assume_secure_protocol = 1

Install

Site: https://piwik.transitionnetwork.org/

Download

cd /web/
mkdir piwik.transitionnetwork.org
cd piwik.transitionnetwork.org/
wget http://piwik.org/latest.zip
unzip latest.zip

Apache

Then set up apache, see /etc/apache2/sites-available/piwik.transitionnetwork.org and run the web installer and:

cd /web/piwik.transitionnetwork.org/piwik/
chown -R www-data:www-data tmp/
chown -R www-data:www-data config/

MySQL

Then a MySQL user and database was added using phpMyAdmin which ran the following SQL:

CREATE USER 'piwik'@'localhost' IDENTIFIED BY '********';
GRANT USAGE ON * . * TO 'piwik'@'localhost' IDENTIFIED BY '********' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
CREATE DATABASE IF NOT EXISTS `piwik` ;
GRANT ALL PRIVILEGES ON `piwik` . * TO 'piwik'@'localhost';

Tracking code

The installer suggested this JS tracking code, see the docs at http://piwik.org/docs/javascript-tracking/ :

<!-- Piwik -->
<script type="text/javascript">
var pkBaseURL = (("https:" == document.location.protocol) ? "https://piwik.transitionnetwork.org.webarch.net/" : "http://piwik.transitionnetwork.org.webarch.net/");
document.write(unescape("%3Cscript src='" + pkBaseURL + "piwik.js' type='text/javascript'%3E%3C/script%3E"));
</script><script type="text/javascript">
try {
var piwikTracker = Piwik.getTracker(pkBaseURL + "piwik.php", 1);
piwikTracker.trackPageView();
piwikTracker.enableLinkTracking();
} catch( err ) {}
</script><noscript><p><img src="http://piwik.transitionnetwork.org.webarch.net/piwik.php?idsite=1" style="border:0" alt="" /></p></noscript>
<!-- End Piwik Tracking Code -->

And this image tracking code:

<!-- Piwik Image Tracker -->
<img src="https://piwik.transitionnetwork.org.webarch.net/piwik.php?idsite=1&rec=1" style="border:0" alt="" />
<!-- End Piwik -->

The following parameters can also be passed to the image URL:

  • rec - (required) The parameter &rec=1 is required to force the request to be recorded
  • idsite - (required) Defines the Website ID being tracked
  • action_name - Defines the custom Page Title for this page view
  • idgoal - The request will trigger the given Goal
  • urlref - The Referrer URL: must be set to the referrer URL used before landing on the page containing the Image tracker. For example, in PHP this value is accessible via

$_SERVER['HTTP_REFERER']

  • revenue - Used with idgoal, defines the custom revenue for this conversion

Customise the logo:

cd /web/piwik.transitionnetwork.org/piwik
chown -R www-data:www-data themes/

Cron

Set up the crontab to automatically archive the reports:

crontab -u www-data -e

5 * * * * /web/piwik.transitionnetwork.org/piwik/misc/cron/archive.sh > /dev/null

Python

For the Google Analytics data import we need Python 2.6 and we are running with Python 2.5.2, so following this a source compile was done on the dev server:

aptitude build-dep python2.5 
cd /usr/local/src
wget http://www.python.org/ftp/python/2.6.7/Python-2.6.7.tgz
tar -zxvf Python-2.6.7.tgz
cd Python-2.6.7
./configure --with-threads --enable-shared
make
make install 
ln -s /usr/local/lib/libpython2.6.so* /usr/lib/
cd /usr/local/src
wget 'http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e'
tar zxvf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
python setup.py install
cd /usr/local/src
wget "https://gdata-python-client.googlecode.com/files/gdata-2.0.14.tar.gz" --no-check-certificate
tar -zxvf gdata-2.0.14.tar.gz
cd gdata-2.0.14
python setup.py install
aptitude purge python-mysqldb
wget "http://downloads.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz"
tar -zxvf MySQL-python-1.2.3.tar.gz 
cd MySQL-python-1.2.3
vim site.cfg
 mysql_config = /usr/bin/mysql_config
aptitude install libmysqlclient-dev
python setup.py build
python setup.py install

GA Import

See python notes above and Google2Piwik.

cd /web/piwik.transitionnetwork.org/
wget http://clearcode.cc/google2piwik-1.2.5.tgz
tar -zxvf google2piwik-1.2.5.tgz
cd google2piwik-1.2.5
cp google2piwik.conf.sample google2piwik.conf

Edit google2piwik.conf:

## Google Account Credentials
# If you don't know your site table_id type:
#                       ./google2piwik.py -p
[google]
user_login = XXX@gmail.com
user_pass  = XXX
table_id   = ga:XXX

# MySQL Piwik Database configuration
[mysql]
db = piwik
host = localhost
port = 3306
user = piwik
passwd = XXX
table_prefix = 

# Set export date range
[export]
start = 2011-05-01
end   = 2011-06-29

## Piwik configuration of site.
# For default Piwik installation with one website, site_id should be set to 1
[piwik]
site_id  = 1
site_url = http://www.transitionnetwork.org

Get the table_id and test and backup db:

./google2piwik.py -p
./google2piwik.py -c
mysql_backup

Piwik, for the website you are importing data to, change the timezone to UTC and run the script:

./google2piwik.py --verbose=2

Requirements

The requirements are:

  • PHP version 5.1.3 or greater, We highly recommend using at least PHP 5.3 as it is much more memory efficient than previous PHP versions, we have: PHP 5.2.6-1+lenny10 with Suhosin-Patch 0.9.6.2 (cli), when the server is migrated to squeeze we will get a new php, it can wait till then.
  • MySQL version 4.1 or greater, we have 5.0.51a-24+lenny5-log
  • (enabled by default) PHP extension pdo and pdo_mysql, weh have these already installed and they are configured via /etc/php5/apache2/conf.d/pdo.ini and /etc/php5/apache2/conf.d/pdo_mysql.ini
  • PHP GD extension, installed already
  • Increase your PHP memory_limit value in your php.ini config file to be at least memory_limit = 128M, we have 256M already in /etc/php5/apache2/php.ini