Transition Technology: Ticket #393: Upgrade Piwiki to 1.7

status changed; cc set

chris — Sat, 18 Feb 2012 11:25:09 GMT

cc laura, ed added
status changed from new to assigned

chris — Mon, 12 Mar 2012 12:38:41 GMT

This upgrade was done at the same time as ticket:394 and the time spent on this task is recorded on that ticket.

Following the notes for the manual upgrade, http://piwik.org/update/ and doing it initially on kiwi, https://piwik.transitionnetwork.org/

cd /web/piwik.transitionnetwork.org
cp piwik/config/config.ini.php config.ini.php.2012-03-12.bak
wget http://piwik.org/latest.zip -O piwik.1.7.1.zip
mv piwik piwik.bak
unzip piwik.1.7.1.zip
cp config.ini.php.2012-03-12.bak piwik/config/config.ini.php
chown -R www-data:www-data /web/piwik.transitionnetwork.org/piwik/themes
chown -R www-data:www-data /web/piwik.transitionnetwork.org/piwik/tmp
chown -R www-data:www-data /web/piwik.transitionnetwork.org/piwik/config
php /web/piwik.transitionnetwork.org/piwik/index.php -- "module=CoreUpdater"

On the live server the additional step to disable Piwik Tracking, http://piwik.org/faq/how-to/#faq_111 should be done and also the new version should be extracted to /tmp and set up before it replaces the existing install.

The live server update is still to be done but the dev site has been upgraded without issues.

hours, totalhours changed

chris — Tue, 13 Mar 2012 13:45:47 GMT

hours changed from 0.0 to 1.8
totalhours changed from 0.0 to 1.8

Upgrade on quince, first edit /web/stats.transitionnetwork.org/piwik/config/config.ini.php and add the following to the [Tracker] section, as per http://piwik.org/faq/how-to/#faq_111

record_statistics = 0

Then:

cd /web/stats.transitionnetwork.org
mkdir piwik.1.7.1
cd piwik.1.7.1
wget http://piwik.org/latest.zip -O piwik.1.7.1.zip
unzip piwik.1.7.1.zip
cd piwik
cp /web/stats.transitionnetwork.org/piwik/config/config.ini.php config/
chown -R www-data:www-data themes
chown -R www-data:www-data tmp
chown -R www-data:www-data config
cd /web/stats.transitionnetwork.org/
mv piwik piwik.bak ; mv piwik.1.7.1/piwik .
php /web/stats.transitionnetwork.org/piwik/index.php -- "module=CoreUpdater"
  PHP Fatal error:  Exception thrown without a stack frame in Unknown on line 0
  Fatal error: Exception thrown without a stack frame in Unknown on line 0
cd piwik
chown -R www-data:www-data tmp

Then this error in the web interface:

Unable to load plugin 'DoNotTrack' because '/web/stats.transitionnetwork.org/piwik/plugins/DoNotTrack/DoNotTrack.php' couldn't be found.

Fixed by:

cd /web/stats.transitionnetwork.org/
rsync -av piwik.bak/plugins/DoNotTrack/ piwik/plugins/
php /web/stats.transitionnetwork.org/piwik/index.php -- "module=CoreUpdater"

Then the web interface has the following error:

Fatal error: Class 'Piwik_Db_Schema_Myisam' not found in /web/stats.transitionnetwork.org/piwik/core/Db/Schema.php on line 139

The only reference I could find to this error message doesn't help, http://forum.piwik.org/read.php?2,85650 -- perhaps !InnoDB isn't supported, see http://dev.piwik.org/trac/ticket/2593 and ticket:396

So switch the piwik database to Myisam:

mysql -e "show tables in stats;" | tail --lines=+2 | xargs -i echo "ALTER TABLE \`{}\` ENGINE=MYISAM;" > stats_
cat stats_alter.sql | mysql stats

Check that all tables are converted:

echo "SHOW TABLE STATUS;" | mysql stats | grep -i INNODB

Run the cronjob:

su - www-data
/web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh

There is still the same error in the admin interface, not sure what to try next, I haven't been able to raise a ticket on the piwik trac but I have posted to their forum:

http://forum.piwik.org/read.php?2,85650,page=1#msg-87181

hours, totalhours changed

chris — Tue, 13 Mar 2012 14:19:23 GMT

hours changed from 0.0 to 0.05
totalhours changed from 1.8 to 1.85

Matt suggested uploading one php file again, http://forum.piwik.org/read.php?2,85650,page=1#msg-85657 I tried this:

cd /web/stats.transitionnetwork.org/piwik/core/Db
mv Schema Schema.bak
svn co http://dev.piwik.org/svn/trunk/core/Db/Schema/
  A    Schema/Myisam.php
  Checked out revision 6040.

It didn't solve the problem.

hours, totalhours changed

chris — Tue, 13 Mar 2012 14:30:28 GMT

hours changed from 0.0 to 0.1
totalhours changed from 1.85 to 1.95

I cleared the APC opcode Cache and restarted apache and tried running the cron job again, this time it generated an error:

su - www-data
/web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh
Starting Piwik reports archiving...
Archiving period = day for idsite = 1...
PHP Fatal error:  Call to undefined method Piwik_Config::getInstance() in /web/stats.transitionnetwork.org/piwi
/Schema/Myisam.php on line 437
Fatal error: Call to undefined method Piwik_Config::getInstance() in /web/stats.transitionnetwork.org/piwik/cor
ma/Myisam.php on line 437

I switched back to the old Schema/Myisam.php and the cron job runs without errors.

hours, status, totalhours changed; resolution set

chris — Wed, 14 Mar 2012 11:58:11 GMT

hours changed from 0.0 to 0.2
status changed from assigned to closed
resolution set to fixed
totalhours changed from 1.95 to 2.15

The suggestion on the forum, which seemed crazy to me:

I had to remove APC completely and reinstall it. After that, everything was okay.

http://forum.piwik.org/read.php?2,85650,page=1#msg-87189

Worked!

cd /etc/php5/conf.d/
cp apc.ini apc.ini.bak
dpkg -r php-apc
aptitude install php-apc
cp apc.ini.bak apc.ini
apache2ctl restart

So /web/stats.transitionnetwork.org/piwik/config/config.ini.php was edited to remove record_statistics = 0 and docs updated wiki:NewLiveServer#piwik to contain a link to this ticket which will now be closed.

Sorry that almost a days worth of stats have been lost during this upgrade.

hours, status, totalhours changed; resolution deleted

chris — Wed, 14 Mar 2012 12:25:36 GMT

hours changed from 0.0 to 0.15
status changed from closed to reopened
resolution fixed deleted
totalhours changed from 2.15 to 2.3

Sorry I have had to reopen this, it's not solved after all.

Following it up at the piwik trac, bug #3048 http://dev.piwik.org/trac/ticket/3048

hours, totalhours changed

chris — Wed, 14 Mar 2012 14:07:08 GMT

hours changed from 0.0 to 0.27
totalhours changed from 2.3 to 2.57

The ticket was closed on the piwik trac: http://dev.piwik.org/trac/ticket/3049#comment:1

I don't have permissions to reopen it, I guess if needs be I can open a new ticket to ask that the first ticket be reopened http://dev.piwik.org/trac/ticket/3049#comment:2

hours, totalhours changed

chris — Thu, 15 Mar 2012 11:32:48 GMT

hours changed from 0.0 to 0.2
totalhours changed from 2.57 to 2.77

Updated suggestion from the developers http://dev.piwik.org/trac/ticket/3049#comment:3

cd /web/stats.transitionnetwork.org/piwik/core/Db/Schema
rm Myisam.php
wget http://dev.piwik.org/trac/export/5924/trunk/core/Db/Schema/Myisam.php
apache2ctl restart

This didn't help, I have added a comment to the ticket on the piwik trac: http://dev.piwik.org/trac/ticket/3049#comment:4

hours, totalhours changed

chris — Tue, 20 Mar 2012 14:51:00 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.77 to 2.87

Posted another reply to the ticket on the piwik forum http://dev.piwik.org/trac/ticket/3049#comment:6

chris — Wed, 21 Mar 2012 10:44:03 GMT

Following the suggestion here http://dev.piwik.org/trac/ticket/3049#comment:9 to do a reinstall, first edit the [Tracker] section of /web/stats.transitionnetwork.org/piwik/config/config.ini.php to turn tracking off:

record_statistics = 0

Create a new directory and download the latest version:

cd /web/stats.transitionnetwork.org
mkdir piwik.1.7.1
cd piwik.1.7.1
wget http://piwik.org/latest.zip
unzip latest.zip
rm How\ to\ install\ Piwik.html
rm latest.zip
mv piwik/* .

Set up apache to serve this new install at https://newstats.transitionnetwork.org.uk/ config file /etc/apache2/sites-available/newstats.transitionnetwork.org.conf.

Piwik suggests changing some ownerships:

chown -R www-data:www-data /web/stats.transitionnetwork.org/piwik.1.7.1
chmod -R 0755 /web/stats.transitionnetwork.org/piwik.1.7.1/tmp
chmod -R 0755 /web/stats.transitionnetwork.org/piwik.1.7.1/tmp/templates_c/
chmod -R 0755 /web/stats.transitionnetwork.org/piwik.1.7.1/tmp/cache/
chmod -R 0755 /web/stats.transitionnetwork.org/piwik.1.7.1/tmp/assets/
chmod -R 0755 /web/stats.transitionnetwork.org/piwik.1.7.1/tmp/tcpdf/

Create a new database:

mysql> CREATE USER 'newstats'@'localhost' IDENTIFIED BY 'xxxxxx';
mysql> CREATE DATABASE IF NOT EXISTS `newstats`;
mysql> GRANT ALL PRIVILEGES ON `newstats`.* TO 'newstats'@'localhost';
mysql> FLUSH PRIVILEGES;

Complete the install using the web interface and the newsite has the same error:

Fatal error: Class 'Piwik_Db_Schema_Myisam' not found in /web/stats.transitionnetwork.org/piwik.1.7.1/core/Db/Schema.php on line 139

So perhaps the problem is with APC?

hours, totalhours changed

chris — Wed, 21 Mar 2012 10:55:29 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.87 to 2.97

Replying to chris:

So perhaps the problem is with APC?

Actually I really don't know, the class can't be found in the file, this grep produces no results:

grep "Piwik_Db_Schema_Myisam" /web/stats.transitionnetwork.org/piwik.1.7.1/core/Db/Schema.php

I think the problem must be with Piwik.

Another comment has been added to the other ticket: http://dev.piwik.org/trac/ticket/3049#comment:10

hours, totalhours changed

chris — Wed, 21 Mar 2012 12:32:36 GMT

hours changed from 0.0 to 0.25
totalhours changed from 2.97 to 3.22

Following the suggestions here http://dev.piwik.org/trac/ticket/3049#comment:11 but these didn't solve the problem, see http://dev.piwik.org/trac/ticket/3049#comment:12

hours, totalhours changed

chris — Wed, 21 Mar 2012 13:26:17 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.22 to 3.32

Debugging suggestion here http://dev.piwik.org/trac/ticket/3049#comment:13 tried, results here: http://dev.piwik.org/trac/ticket/3049#comment:14

hours, totalhours changed

chris — Wed, 21 Mar 2012 14:07:49 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.32 to 3.42

Piwik developers have asked for ssh access to help solve this problem, http://dev.piwik.org/trac/ticket/3049#comment:16

I have emailed the list to check if this is OK.

hours, totalhours changed

chris — Mon, 26 Mar 2012 09:53:36 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.42 to 3.52

The Piwik developers haven't been in touch, I have asked them for either a ssh or gpg public key so I can set them up with access to the server.

I have added another comment to the ticket on the piwik trac site http://dev.piwik.org/trac/ticket/3049#comment:19

hours, totalhours changed

chris — Thu, 29 Mar 2012 12:06:51 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.52 to 3.62

Still not heard from them, have left another comment http://dev.piwik.org/trac/ticket/3049#comment:20

hours, totalhours changed

chris — Sat, 31 Mar 2012 10:45:12 GMT

hours changed from 0.0 to 0.2
totalhours changed from 3.62 to 3.82

I have shutdown the test site I installed for the Piwki developers and posted another comment asking for the bug to be reopened: http://dev.piwik.org/trac/ticket/3049#comment:21

hours, totalhours changed

chris — Thu, 05 Apr 2012 10:23:41 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.82 to 3.92

I have opened a new ticket asking that the first ticked be reopened:

http://dev.piwik.org/trac/ticket/3092

hours, totalhours changed

chris — Mon, 16 Apr 2012 13:18:24 GMT

hours changed from 0.0 to 2.0
totalhours changed from 3.92 to 5.92

This is now fixed!

Read on for details...

One on the piwik developers found the problem was with these APC settings in /etc/php5/apache2/conf.d/apc.ini:

apc.include_once_override = 1
apc.mmap_file_mask = /tmp/apc.XXXXXX

When these were changed to:

apc.include_once_override = 0
apc.mmap_file_mask = /dev/zero

The problem went away, the piwik ticket, http://dev.piwik.org/trac/ticket/3049#comment:27 and forum thread have been updated, http://forum.piwik.org/read.php?2,85650,page=1#msg-88217

The documentation has been updated here wiki:NewLiveServer#apc

Before the Piwik developer came to the rescue I tried the suggestions here for clearing the APC cache, but none helped:

http://stackoverflow.com/questions/911158/how-to-clear-apc-cache-entries

There is a password protected cache clearing script here:

https://quince.transitionnetwork.org/info/apc-clear-cache.php

And you can see the results here:

https://quince.transitionnetwork.org/info/apc.php

I'm not convinced that this has been deleting all the files from the cache as this graph didn't drop to zero:

https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/php_apc_files.html

status changed; resolution set

chris — Mon, 16 Apr 2012 13:18:41 GMT

status changed from reopened to closed
resolution set to fixed