Piwik Archive Cron Error

[chris]

Have been getting these emails from PiwikServer:

From: root@penguin.webarch.net (Cron Daemon)                                                                                                 
Date: Fri,  1 Aug 2014 14:06:48 +0100 (BST)                                                                                                  
To: root@localhost                                                                                                                           
Subject: Cron <www-data@penguin> /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log                                                                                                                  
                                                                                                                                             
ERROR CoreConsole[2014-08-01 13:05:18] [3e5ac] Got invalid response from API request:                                                        
+http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=week&date=last2&format=php&token_auth=XXXXXXXXXXXX&trigger=archivephp. Response was ' <div style='word-wrap: break-word; border: 3px solid red; padding:4px; width:70%;      
+background-color:#FFFF96;'>         <strong>There is an error. Please report the message (Piwik 2.4.1)         and full backtrace in the <a 
+href='?module=Proxy&action=redirect&url=http://forum.piwik.org' target='_blank'>Piwik forums</a> (please do a Search first as it might have 
+been reported already!).<br /><br/>         Warning:</strong>                                                                               
+<em>file_get_contents(http://api.piwik.org/1.0/getLatestVersion/?piwik_version=2.4.1&php_version=5.4.4-14%2Bdeb7u12&url=https%3A%2F%2Fstats.
+transitionnetwork.org%2Fweb%2Fstats.transitionnetwork.org%2Fpiwik%2Fconsole&trigger=API&timezone=Europe%2FLondon): failed to open stream:   
+HTTP requ                                                                                                                                   
 est fail                                                                                                                                    
 ed! </em> in <strong>/web/stats.transitionnetwork.org/piwik/core/Http.php</strong> on line <strong>406</strong> <br /><br />Backtrace       
+--&gt;<div style="font-family:Courier;font-size:10pt"><br /> #0  Piwik\Error::errorHandler(...) called at [:]<br /> #1                      
+file_get_contents(...) called at [/web/stats.transitionnetwork.org/piwik/core/Http.php:406]<br /> #2  Piwik\Http::sendHttpRequestBy(...)    
+called at [/web/stats.transitionnetwork.org/piwik/core/Http.php:94]<br /> #3  Piwik\Http::sendHttpRequest(...) called at                    
+[/web/stats.transitionnetwork.org/piwik/core/UpdateCheck.php:72]<br /> #4  Piwik\UpdateCheck::check(...) called at                          
+[/web/stats.transitionnetwork.org/piwik/plugins/CoreUpdater/CoreUpdater.php:142]<br /> #5                                                   
+Piwik\Plugins\CoreUpdater\CoreUpdater->updateCheck(...) called at [:]<br /> #6  call_user_func_array(...) called at                         
+[/web/stats.transitionnetwork.org/piwik/core/EventDispatcher.php:98]<br /> #7  Piwik\EventDispatcher->postEvent(...) called at              
+[/web/stats.transitionnetwor                                                                                                                
 k.org/pi                                                                                                                                    
 wik/core/Piwik.php:766]<br /> #8  Piwik\Piwik::postEvent(...) called at                                                                     
+[/web/stats.transitionnetwork.org/piwik/core/FrontController.php:391]<br /> #9  Piwik\FrontController->init(...) called at                  
+[/web/stats.transitionnetwork.org/piwik/core/dispatch.php:33]<br /> #10  require_once(...) called at                                        
+[/web/stats.transitionnetwork.org/piwik/index.php:47]<br /> #11  require_once(...) called at                                                
+[/web/stats.transitionnetwork.org/piwik/core/CliMulti/RequestCommand.php:53]<br /> #12  Piwik\CliMulti\RequestCommand->execute(...) called  
+at [/web/stats.transitionnetwork.org/piwik/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:252]<br /> #13              
+Symfony\Component\Console\Command\Command->run(...) called at                                                                               
+[/web/stats.transitionnetwork.org/piwik/vendor/symfony/console/Symfony/Component/Console/Application.php:887]<br /> #14                     
+Symfony\Component\Console\Application->doRunCommand(...) called at                                                                          
+[/web/stats.transitionnetwork.org/piwik/vendor/symfony/console/Symfony/Component/Co                                                         
 nsole/Ap                                                                                                                                    
 plication.php:193]<br /> #15  Symfony\Component\Console\Application->doRun(...) called at                                                   
+[/web/stats.transitionnetwork.org/piwik/core/Console.php:64]<br /> #16  Piwik\Console->doRun(...) called at                                 
+[/web/stats.transitionnetwork.org/piwik/vendor/symfony/console/Symfony/Component/Console/Application.php:124]<br /> #17                     
+Symfony\Component\Console\Application->run(...) called at [/web/stats.transitionnetwork.org/piwik/console:31]<br /> </div><br />            
+</pre></div><br />  <div style='word-wrap: break-word; border: 3px solid red; padding:4px; width:70%; background-color:#FFFF96;'>           
+<strong>There is an error. Please report the message (Piwik 2.4.1)         and full backtrace in the <a                                     
+href='?module=Proxy&action=redirect&url=http://forum.piwik.org' target='_blank'>Piwik forums</a> (please do a Search first as it might have 
+been reported already!).<br /><br/>         Warning:</strong>                                                                               
+<em>file_get_contents(http://api.piwik.org/1.0/getLatestVersion/?piwik_version=2.4.1&php_version                                            
 =5.4.4-1                           

[chris]

The PiwikServer crontab, which was run as user www-data was commented out a while ago:

# http://piwik.org/docs/setup-auto-archiving/
#5 * * * * /web/stats.transitionnetwork.org/piwik/misc/cron/archive.sh > /dev/null

And now we have /etc/cron.d/piwik-archive which contains:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log

The documentation for this is:

• ​http://piwik.org/docs/setup-auto-archiving/#linuxunix-how-to-set-up-a-crontab-to-automatically-archive-the-reports

I have spent some time searching for any reported bugs regarding this issue, it appears to be intermittent. Haven't found an answer yet.

[chris]

The description of this ticket was edited to remove the token_auth string, I have since changed my password for Piwik and this has reset the token_auth, this can be checked at a MySQL level, eg:

select login from user where token_auth="XXXX";

[chris]

Replying to chris:

And now we have /etc/cron.d/piwik-archive which contains:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log

I have changed that to:

MAILTO="root@localhost"
5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log

So logs are appended rather than clobbered and also added log rotation by creating /etc/logrotate.d/piwik which contains:

/var/log/piwik-archive.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        notifempty
}

I have spent some time searching for any reported bugs regarding this issue, it appears to be intermittent. Haven't found an answer yet.

Recent emails have this error:

From: root@penguin.webarch.net (Cron Daemon)
Date: Thu, 28 Aug 2014 01:13:07 +0100 (BST)
To: root@localhost
Subject: Cron <www-data@penguin> /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ > /var/log/piwik-archive.log                                                                                                                     

ERROR CoreConsole[2014-08-28 00:10:38] [69fb6] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-28 00:10:38] [69fb6] Empty or invalid response '' for website id 1, Time elapsed: 7.769s, skipping
ERROR CoreConsole[2014-08-28 00:10:53] [69fb6] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=7&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-28 00:10:53] [69fb6] Empty or invalid response '' for website id 7, Time elapsed: 15.540s, skipping
ERROR CoreConsole[2014-08-28 00:13:07] [69fb6] 4 total errors during this script execution, please investigate and try and fix these errors.

So I have doubled the memory_limit in /etc/php5/fpm/php.ini:

;memory_limit = 128M
memory_limit = 256M

And in restarted php5-fpm.

This might have fixed it, will check tomorrow.

[chris]

No errors since the changes yesterday, but we now have logs to used to check if there have been errors.

[chris]

There was another error last night:

Date: Fri,  5 Sep 2014 01:11:30 +0100 (BST)
To: root@localhost
Subject: Cron <www-data@penguin> /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log

ERROR CoreConsole[2014-09-05 00:08:46] [65cbf] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.                       
ERROR CoreConsole[2014-09-05 00:08:46] [65cbf] Empty or invalid response '' for website id 1, Time elapsed: 8.091s, skipping
ERROR CoreConsole[2014-09-05 00:11:30] [65cbf] 2 total errors during this script execution, please investigate and try and fix these errors. 

The above error in the logs:

ERROR CoreConsole[2014-09-05 00:08:46] [65cbf] Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.

INFO CoreConsole[2014-09-05 00:11:30] [65cbf] Error: Got invalid response from API request: http://stats.transitionnetwork.org/index.php?module=API&method=API.get&idSite=1&period=day&date=last2&format=php&token_auth=XXXX&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
INFO CoreConsole[2014-09-05 00:11:30] [65cbf] Error: Empty or invalid response '' for website id 1, Time elapsed: 8.091s, skipping

It's worth nothing that most of these errors have been around 1:10am BST (00:10 UTC, the server is on UTC), perhaps this is the same time that the backups are being generated and and answer is to amend the crontab so it doesn't run at this time.

So following ​the suggestion here I have edited /etc/cron.d/piwik-archive so it won't run at midnight or 1am:

MAILTO="root@localhost"
#5 * * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log
5 2-23 * * * www-data /web/stats.transitionnetwork.org/piwik/console core:archive --url=http://stats.transitionnetwork.org/ >> /var/log/piwik-archive.log

Hopefully that will solve this issue.