Transition Technology: Ticket #645: APC Tuning on Parrot and Penguin

chris — Tue, 10 Dec 2013 17:59:51 GMT

The number of alerts I'm getting from APC on wiki:PenguinServer is getting a bit annoying, every time the purge rate goes over 10% I get sent a email every 5mins:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/php_apc_purge.html

The APC reference docs are here:

http://php.net/manual/en/apc.configuration.php

hours, totalhours changed

chris — Wed, 11 Dec 2013 11:38:31 GMT

hours changed from 0.0 to 0.15
totalhours changed from 0.0 to 0.15

Quick fix for the warning emails for now, have edited /usr/share/munin/plugins/php_apc_ on penguin and changed this line:

purge_rate.warning 70

It was set to 10, see https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/php_apc_purge.html

I have also tweaked the cache sizes.

APC graphs for both servers can be viewed here:

https://penguin.transitionnetwork.org/munin/php-apc-day.html

More fine tuning can wait till another month.

attachment set

chris — Sun, 15 Dec 2013 13:09:20 GMT

attachment set to penguin-2013-12-15_php_apc_purge-week.png

Penguin APC Purge Rate 15th Dec 2013

hours, totalhours changed

chris — Sun, 15 Dec 2013 13:13:35 GMT

hours changed from 0.0 to 0.15
totalhours changed from 0.15 to 0.3

I have started getting email alerts every 5 mins from Munin due to the APC PHP purge rate hitting 70%:

So I have edited /usr/share/munin/plugins/php_apc_ and changed this:

purge_rate.warning 90

And restarted munin-node.

chris — Thu, 19 Dec 2013 10:19:29 GMT

Some warning thresholds and cache sizes were tweaked while the Piwik upgrade was running, ticket:660.

hours, totalhours changed

chris — Thu, 19 Dec 2013 12:09:43 GMT

hours changed from 0.0 to 0.4
totalhours changed from 0.3 to 0.7

Tweaked settings in /etc/php5/mods-available/apc.ini on wiki:PenguinServer:

apc.shm_size="96M"
apc.ttl=0
apc.user_ttl=7200
apc.gc_ttl=600
apc.max_file_size="2M"

I have also added a link to the APC stats from https://penguin.transitionnetwork.org/

These are the settings we have for /etc/php5/mods-available/apc.ini on wiki:ParrotServer:

apc.shm_size="180M"
apc.ttl=0
apc.user_ttl=7200
apc.gc_ttl = 600
apc.max_file_size="2M"

I have added a apache rule to make the APC info only available via HTTPS, in /etc/apache2/conf.d/webarch.conf:

RewriteRule ^/apc.php(.*) https://%{SERVER_NAME}/apc.php$1 [R,L]

And I have copies the Trac .htpasswd file to parrot so it can be used to protect access to https://parrot.webarch.net/apc.php and there is a now link to this from the front page for the server https://parrot.webarch.net/

The warning thresholds for the graphs here have also been tweaked:

https://penguin.transitionnetwork.org/munin/php-apc-day.html

description changed

chris — Thu, 19 Dec 2013 12:16:51 GMT

description modified (diff)

status changed

chris — Thu, 19 Dec 2013 12:16:59 GMT

status changed from new to accepted

hours, totalhours changed

chris — Thu, 19 Dec 2013 13:02:17 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.7 to 0.8

wiki:ParrotServer APC settings in /etc/php5/mods-available/apc.ini were tewaked again based on the stats:

apc.shm_size="256M"
apc.ttl=0
apc.user_ttl=7200
apc.gc_ttl = 600
apc.max_file_size="3M"

attachment set

chris — Thu, 16 Jan 2014 14:43:33 GMT

attachment set to parrot_2014-01-16_php_apc_purge-week.png

hours, totalhours changed

chris — Thu, 16 Jan 2014 14:49:17 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.8 to 1.05

I'm getting an alert every 5 mins because the wiki:PenguinServer APC purge rate has hit 90%:

I'm not sure what should be done to best address this, for now I have edited /usr/share/munin/plugins/php_apc_ to increase the warning threshold to 99%:

purge_rate.warning 99

hours, totalhours changed

chris — Thu, 16 Jan 2014 14:56:51 GMT

hours changed from 0.0 to 0.15
totalhours changed from 1.05 to 1.2

Looing at the APC info at https://penguin.transitionnetwork.org/info/apc.php the memory usage is quite high:

Free: 14.4 MBytes (15.0%)
Used: 81.6 MBytes (85.0%)

So I have increased the available memory from 96MB to 128MB by editing /etc/php5/mods-available/apc.ini

apc.shm_size="128M"

And restarting php-fpm.

I have also added a link to this ticket from wiki:PenguinServer#APCStatsandPHPinfo

hours, totalhours changed

chris — Tue, 18 Feb 2014 09:28:49 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.2 to 1.45

wiki:PenguinServer purge rate is at 100% again and I'm getting alerts every 5 mins, I have done some research and looks at the stats and my best guess is that changing apc.ttl to 24 hours might help (it was at 0), so this is the updated /etc/php5/mods-available/apc.ini:

extension=apc.so
apc.shm_size="128M"
apc.ttl=86400
apc.user_ttl=7200
apc.gc_ttl=600
apc.max_file_size="3M"

Restart:

/etc/init.d/php5-fpm restart

chris — Wed, 02 Apr 2014 09:54:08 GMT

On wiki:PenguinServer the purge rate still tends to 100%, this isn't resolved, see:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/php_apc_purge.html