Transition Technology: Ticket #726: Upgrade to Piwik 2.2.1

attachment set

chris — Tue, 06 May 2014 11:38:23 GMT

attachment set to piwik_06052014.png

hours, totalhours changed

chris — Tue, 06 May 2014 11:40:52 GMT

hours changed from 0.0 to 0.4
totalhours changed from 0.0 to 0.4

Following the wiki:PiwikServer#Updates notes, set record_statistics to 0 in /web/stats.transitionnetwork.org/piwik/config/config.ini.php and then backup the config file:

cd /web/stats.transitionnetwork.org/
cp piwik/config/config.ini.php .

Download, extract, and update the database and restart php-fpm:

export PIWIK=2.1.1
wget "http://builds.piwik.org/piwik-$PIWIK.tar.gz"
  2014-05-06 12:26:58 ERROR 404: Not Found.

So:

wget "http://builds.piwik.org/latest.tar.gz" -O "piwik-$PIWIK.tar.gz"
tar -zxvf piwik-$PIWIK.tar.gz
cp config.ini.php piwik/config/
chown -R www-data:www-data piwik/
php /web/stats.transitionnetwork.org/piwik/console core:update
  Everything is already up to date.
/etc/init.d/php5-fpm restart

Switch recording of stats back on by setting record_statistics to 1 in /web/stats.transitionnetwork.org/piwik/config/config.ini.php:

[Tracker]
record_statistics = 1

That has gone OK however there are two issues:

The "All Websites" page is blank.
There have been no recent stats recorded since 18th April, see the following graph.

I don't know the cause of these two issues.

attachment set

chris — Tue, 06 May 2014 12:35:47 GMT

attachment set to piwik_06052014_2.png

attachment set

chris — Tue, 06 May 2014 12:38:00 GMT

attachment set to piwik_06052014_3.png

hours, totalhours changed

chris — Tue, 06 May 2014 12:43:25 GMT

hours changed from 0.0 to 1.0
totalhours changed from 0.4 to 1.4

The last Piwik update does coincide with the last upgrade, see ticket:720 (Upgrade to Piwik 2.2.0) closed by chris at 9:13am on 17th April 2014:

https://tech.transitionnetwork.org/trac/timeline?from=04%2F18%2F14&daysback=5

We are not the only ones with the blank all websites page:

http://dev.piwik.org/trac/ticket/5038

So I have posted a comment on the Piwik Trac:

http://dev.piwik.org/trac/ticket/5038#comment:5

For the lack of stats there is this ticket:

http://dev.piwik.org/trac/ticket/5015

This is our current crontab:

5 * * * * /web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh > /dev/null

This has been commented out and /etc/cron.d/piwik-archive was created containing:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log

The log file was created and chowned:

touch /var/log/piwik-archive.log
chown root:www-data /var/log/piwik-archive.log
chmod 664 /var/log/piwik-archive.log

And the script was run manually:

su - www-data
/web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/

And the output looks good:

INFO CoreConsole[2014-05-06 12:30:53] [3511c] ---------------------------
INFO CoreConsole[2014-05-06 12:30:53] [3511c] INIT
INFO CoreConsole[2014-05-06 12:30:53] [3511c] Piwik is installed at: http://stats.transitionnetwork.org/index.php
INFO CoreConsole[2014-05-06 12:30:53] [3511c] Running Piwik 2.2.1 as Super User
INFO CoreConsole[2014-05-06 12:30:54] [3511c] ---------------------------
INFO CoreConsole[2014-05-06 12:30:54] [3511c] NOTES
INFO CoreConsole[2014-05-06 12:30:54] [3511c] - Reports for today will be processed at most every 1800 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-05-06 12:30:54] [3511c] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-05-06 12:30:55] [3511c] - Will process 9 websites with new visits since 7 days 0 hours , IDs: 1, 2, 3, 4, 5, 7, 8, 9, 11
INFO CoreConsole[2014-05-06 12:30:55] [3511c] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 12
INFO CoreConsole[2014-05-06 12:30:55] [3511c] - Will process 2 other websites because the last time they were archived was on a different day (in the website's timezone) , IDs: 6, 10
INFO CoreConsole[2014-05-06 12:30:55] [3511c] Will ignore websites and help finish a previous started queue instead. IDs: 7, 8, 9, 11, 12, 6, 10
INFO CoreConsole[2014-05-06 12:30:55] [3511c] ---------------------------
INFO CoreConsole[2014-05-06 12:30:55] [3511c] START
INFO CoreConsole[2014-05-06 12:30:55] [3511c] Starting Piwik reports archiving...
INFO CoreConsole[2014-05-06 12:31:05] [3511c] Archived website id = 7, period = day, 1222 visits in last 52 days, 18 visits today, Time elapsed: 9.728s
INFO CoreConsole[2014-05-06 12:31:14] [3511c] Archived website id = 7, period = week, 6996 visits in last 260 weeks, 47 visits this week, Time elapsed: 8.675s
INFO CoreConsole[2014-05-06 12:31:19] [3511c] Archived website id = 7, period = month, 6996 visits in last 52 months, 104 visits this month, Time elapsed: 5.821s
INFO CoreConsole[2014-05-06 12:31:22] [3511c] Archived website id = 7, period = year, 6996 visits in last 7 years, 2665 visits this year, Time elapsed: 2.313s
INFO CoreConsole[2014-05-06 12:31:22] [3511c] Archived website id = 7, 4 API requests, Time elapsed: 26.543s [1/7 done]
INFO CoreConsole[2014-05-06 12:31:26] [3511c] Archived website id = 8, period = day, 34 visits in last 52 days, 0 visits today, Time elapsed: 3.794s

And we now have the stats again:

And the docs have been updated with the crontab setup, wiki:PiwikServer#Cron

The "All Websites" page being blank is still an issue.

hours, totalhours changed

chris — Tue, 06 May 2014 13:59:00 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.4 to 1.65

Replying to chris:

The "All Websites" page being blank is still an issue.

Have opened a new ticket for this:

http://dev.piwik.org/trac/ticket/5102

Following the suggestion from Matt, http://dev.piwik.org/trac/ticket/5038#comment:6

chris — Wed, 07 May 2014 09:13:08 GMT

tsteur at http://dev.piwik.org/trac/ticket/5102#comment:2 has asked:

Is it maybe possible to provide us a login for your Piwik instance? View permission should be enough. This would help us debugging this issue. If so, please send us an email to hello@… and leave a comment here in case it goes into the spam folder

Ed - is this OK?

ed — Wed, 07 May 2014 09:23:14 GMT

fine by me

hours, totalhours changed

chris — Thu, 08 May 2014 11:58:53 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.65 to 1.9

Replying to ed:

fine by me

Thanks Ed, have done:

http://dev.piwik.org/trac/ticket/5102#comment:4

hours, status, totalhours changed; resolution set

chris — Thu, 08 May 2014 22:42:03 GMT

hours changed from 0.0 to 1.0
status changed from new to closed
resolution set to fixed
totalhours changed from 1.9 to 2.9

The reply, http://dev.piwik.org/trac/ticket/5102#comment:5

Files to those HTML files were blocked by the webserver:

/plugins/MultiSites/angularjs/dashboard/dashboard.html /plugins/CoreHome/angularjs/siteselector/siteselector.html?cb=8c07843a982258284632854e69fa9ffd /plugins/Feedback/angularjs/ratefeature/ratefeature.html?cb=8c07843a982258284632854e69fa9ffd

Please whitelist *.html files within the plugin folder.

And the culprit is this part of the Nginx config:

## Disallow access to several helper files.
        location ~* \.(?:bat|html?|git|ini|sh|svn[^.]*|txt|tpl|xml)$ {
        return 404;
}

And digging through the tickets here I found ticket:357#comment:28

Piwki has been setup, following http://wiki.nginx.org/Piwik which is based on https://github.com/perusio/piwik-nginx

And at https://github.com/perusio/piwik-nginx/blob/master/apps/piwik/piwik.conf we have:

    ## Disallow access to several helper files.
    location ~* \.(?:bat|html?|git|ini|sh|svn[^.]*|txt|tpl|xml)$ {
        return 404;
    }

It also turns out that we are not the first to hit this issue:

So html?| has been removed from the regex and it's been followed up via off-list emails with the Piwik devs and on their ticket http://dev.piwik.org/trac/ticket/5102#comment:6.

Everything is working now.

I have also deleted their account.