Transition Technology: Ticket #768: Piwik Archive Cron Error

hours, totalhours changed

chris — Fri, 01 Aug 2014 17:37:22 GMT

hours changed from 0.0 to 0.35
totalhours changed from 0.0 to 0.35

The PiwikServer crontab, which was run as user www-data was commented out a while ago:

# http://piwik.org/docs/setup-auto-archiving/
#5 * * * * /web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh > /dev/null

And now we have /etc/cron.d/piwik-archive which contains:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log

The documentation for this is:

http://piwik.org/docs/setup-auto-archiving/#linuxunix-how-to-set-up-a-crontab-to-automatically-archive-the-reports

I have spent some time searching for any reported bugs regarding this issue, it appears to be intermittent. Haven't found an answer yet.

description changed

chris — Fri, 01 Aug 2014 17:37:46 GMT

description modified (diff)

chris — Fri, 01 Aug 2014 18:13:53 GMT

The description of this ticket was edited to remove the token_auth string, I have since changed my password for Piwik and this has reset the token_auth, this can be checked at a MySQL level, eg:

select login from user where token_auth="XXXX";

hours, totalhours changed

chris — Wed, 03 Sep 2014 10:58:54 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.35 to 0.6

Replying to chris:

And now we have /etc/cron.d/piwik-archive which contains:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log

I have changed that to:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log

So logs are appended rather than clobbered and also added log rotation by creating /etc/logrotate.d/piwik which contains:

/var/log/piwik-archive.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        notifempty
}

I have spent some time searching for any reported bugs regarding this issue, it appears to be intermittent. Haven't found an answer yet.

Recent emails have this error:

From: root@penguin.webarch.net (Cron Daemon)
Date: Thu, 28 Aug 2014 01:13:07 +0100 (BST)
To: root@localhost
Subject: Cron <www-data@penguin> /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log
ERROR CoreConsole[2014-08-28 00:10:38] [69fb6] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-28 00:10:38] [69fb6] Empty or invalid response '' for website id 1, Time elapsed: 7.769s, skipping
ERROR CoreConsole[2014-08-28 00:10:53] [69fb6] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=7&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-28 00:10:53] [69fb6] Empty or invalid response '' for website id 7, Time elapsed: 15.540s, skipping
ERROR CoreConsole[2014-08-28 00:13:07] [69fb6] 4 total errors during this script execution, please investigate and try and fix these errors.

So I have doubled the memory_limit in /etc/php5/fpm/php.ini:

;memory_limit = 128M
memory_limit = 256M

And in restarted php5-fpm.

This might have fixed it, will check tomorrow.

hours, totalhours changed

chris — Thu, 04 Sep 2014 09:19:07 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.6 to 0.7

No errors since the changes yesterday, but we now have logs to used to check if there have been errors.

hours, totalhours changed

chris — Fri, 05 Sep 2014 09:47:18 GMT

hours changed from 0.0 to 0.3
totalhours changed from 0.7 to 1.0

There was another error last night:

Date: Fri,  5 Sep 2014 01:11:30 +0100 (BST)
To: root@localhost
Subject: Cron <www-data@penguin> /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log
ERROR CoreConsole[2014-09-05 00:08:46] [65cbf] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-09-05 00:08:46] [65cbf] Empty or invalid response '' for website id 1, Time elapsed: 8.091s, skipping
ERROR CoreConsole[2014-09-05 00:11:30] [65cbf] 2 total errors during this script execution, please investigate and try and fix these errors.

The above error in the logs:

ERROR CoreConsole[2014-09-05 00:08:46] [65cbf] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
INFO CoreConsole[2014-09-05 00:11:30] [65cbf] Error: Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
INFO CoreConsole[2014-09-05 00:11:30] [65cbf] Error: Empty or invalid response '' for website id 1, Time elapsed: 8.091s, skipping

It's worth nothing that most of these errors have been around 1:10am BST (00:10 UTC, the server is on UTC), perhaps this is the same time that the backups are being generated and and answer is to amend the crontab so it doesn't run at this time.

So following the suggestion here I have edited /etc/cron.d/piwik-archive so it won't run at midnight or 1am:

MAILTO="root@localhost"
#5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log
5 2-23 * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log

Hopefully that will solve this issue.