Ticket #645 (accepted maintenance)

Opened 3 years ago

Last modified 3 years ago

APC Tuning on Parrot and Penguin

Reported by: chris Owned by: chris
Priority: minor Milestone: Maintenance
Component: Dev server Keywords:
Cc: ed Estimated Number of Hours: 0.0
Add Hours to Ticket: 0 Billable?: yes
Total Hours: 1.45

Description (last modified by chris) (diff)

As part of the upgrade from Squeeze to Wheezy, see ticket:535, Munin graphs for APC were added:

There is also more APC info here:

The documentation for the variables which can be set in /etc/php5/mods-available/apc.ini can be found here:

This monitoring is generating quite a high volume of warnings about fragmentation and purges and this ticket has been created to try to sort this issue out.

Attachments

penguin-2013-12-15_php_apc_purge-week.png (22.0 KB) - added by chris 3 years ago.
Penguin APC Purge Rate 15th Dec 2013
parrot_2014-01-16_php_apc_purge-week.png (18.4 KB) - added by chris 3 years ago.

Change History

comment:1 Changed 3 years ago by chris

The number of alerts I'm getting from APC on wiki:PenguinServer is getting a bit annoying, every time the purge rate goes over 10% I get sent a email every 5mins:

The APC reference docs are here:

comment:2 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.15
  • Total Hours changed from 0.0 to 0.15

Quick fix for the warning emails for now, have edited /usr/share/munin/plugins/php_apc_ on penguin and changed this line:

purge_rate.warning 70

It was set to 10, see https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/php_apc_purge.html

I have also tweaked the cache sizes.

APC graphs for both servers can be viewed here:

More fine tuning can wait till another month.

Changed 3 years ago by chris

Penguin APC Purge Rate 15th Dec 2013

comment:3 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.15
  • Total Hours changed from 0.15 to 0.3

I have started getting email alerts every 5 mins from Munin due to the APC PHP purge rate hitting 70%:

Penguin APC Purge Rate 15th Dec 2013

So I have edited /usr/share/munin/plugins/php_apc_ and changed this:

purge_rate.warning 90

And restarted munin-node.

comment:4 Changed 3 years ago by chris

Some warning thresholds and cache sizes were tweaked while the Piwik upgrade was running, ticket:660.

comment:5 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.4
  • Total Hours changed from 0.3 to 0.7

Tweaked settings in /etc/php5/mods-available/apc.ini on wiki:PenguinServer:

apc.shm_size="96M"
apc.ttl=0
apc.user_ttl=7200
apc.gc_ttl=600
apc.max_file_size="2M"

I have also added a link to the APC stats from https://penguin.transitionnetwork.org/

These are the settings we have for /etc/php5/mods-available/apc.ini on wiki:ParrotServer:

apc.shm_size="180M"
apc.ttl=0
apc.user_ttl=7200
apc.gc_ttl = 600
apc.max_file_size="2M"

I have added a apache rule to make the APC info only available via HTTPS, in /etc/apache2/conf.d/webarch.conf:

RewriteRule ^/apc.php(.*) https://%{SERVER_NAME}/apc.php$1 [R,L]

And I have copies the Trac .htpasswd file to parrot so it can be used to protect access to https://parrot.webarch.net/apc.php and there is a now link to this from the front page for the server https://parrot.webarch.net/

The warning thresholds for the graphs here have also been tweaked:

Last edited 3 years ago by chris (previous) (diff)

comment:6 Changed 3 years ago by chris

  • Description modified (diff)

comment:7 Changed 3 years ago by chris

  • Status changed from new to accepted

comment:8 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.1
  • Total Hours changed from 0.7 to 0.8

wiki:ParrotServer APC settings in /etc/php5/mods-available/apc.ini were tewaked again based on the stats:

apc.shm_size="256M"
apc.ttl=0
apc.user_ttl=7200
apc.gc_ttl = 600
apc.max_file_size="3M"

Changed 3 years ago by chris

comment:9 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.25
  • Total Hours changed from 0.8 to 1.05

I'm getting an alert every 5 mins because the wiki:PenguinServer APC purge rate has hit 90%:


I'm not sure what should be done to best address this, for now I have edited /usr/share/munin/plugins/php_apc_ to increase the warning threshold to 99%:

purge_rate.warning 99

comment:10 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.15
  • Total Hours changed from 1.05 to 1.2

Looing at the APC info at https://penguin.transitionnetwork.org/info/apc.php the memory usage is quite high:

  • Free: 14.4 MBytes (15.0%)
  • Used: 81.6 MBytes (85.0%)

So I have increased the available memory from 96MB to 128MB by editing /etc/php5/mods-available/apc.ini

apc.shm_size="128M"

And restarting php-fpm.

I have also added a link to this ticket from wiki:PenguinServer#APCStatsandPHPinfo

comment:11 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.25
  • Total Hours changed from 1.2 to 1.45

wiki:PenguinServer purge rate is at 100% again and I'm getting alerts every 5 mins, I have done some research and looks at the stats and my best guess is that changing apc.ttl to 24 hours might help (it was at 0), so this is the updated /etc/php5/mods-available/apc.ini:

extension=apc.so
apc.shm_size="128M"
apc.ttl=86400
apc.user_ttl=7200
apc.gc_ttl=600
apc.max_file_size="3M"

Restart:

/etc/init.d/php5-fpm restart
Note: See TracTickets for help on using tickets.