Transition Technology: Ticket #397: Live server RAM and disk upgrade

hours, totalhours changed

chris — Thu, 23 Feb 2012 15:54:04 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.0 to 0.5

status changed; milestone set

chris — Thu, 23 Feb 2012 15:54:30 GMT

status changed from new to accepted
milestone set to PSE

hours, totalhours changed

chris — Thu, 23 Feb 2012 17:05:15 GMT

hours changed from 0.0 to 0.3
totalhours changed from 0.5 to 0.8

The reboot has been done, it did take a couple of mins to come up as a filesystem check was done.

The new disk for MySQL had to be commented out as the machine wouldn't boot with them available, this was the error:

xm create quince.cfg
Using config file "./quince.cfg".
Error: Device 51715 (vbd) could not be connected.
Device /dev/FastDisk/quince-var-lib-mysql is mounted in the privileged domain,
and so cannot be mounted by a guest.

I will investigate this and solve it and then we will need another quick reboot to make the disk available.

I haven't yet done any application configuration updates to make use of the additional RAM.

When munin next updates there will be a step on the memory graph: https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/memory.html

There were a few warnings in the boot log file which should probably be investigated:

Feb 23 16:57:07 quince apache2: PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, null given in
+/web/transitionnetwork.org/www/includes/database.mysql.inc on line 388
Feb 23 16:57:07 quince apache2: PHP Warning:  mysql_query() expects parameter 2 to be resource, null given in
+/web/transitionnetwork.org/www/includes/database.mysql.inc on line 121
Feb 23 16:57:07 quince apache2: PHP Warning:  mysql_errno() expects parameter 1 to be resource, null given in
+/web/transitionnetwork.org/www/includes/database.mysql.inc on line 142

hours, totalhours changed

chris — Thu, 23 Feb 2012 17:11:13 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.8 to 0.9

Replying to chris:

The new disk for MySQL had to be commented out as the machine wouldn't boot with them available, this was the error:
xm create quince.cfg
Using config file "./quince.cfg".
Error: Device 51715 (vbd) could not be connected.
Device /dev/FastDisk/quince-var-lib-mysql is mounted in the privileged domain,
and so cannot be mounted by a guest.
I will investigate this and solve it and then we will need another quick reboot to make the disk available.

OK, this was simple, I unmounted the disk in Dom0 and rebooted quince again.

The new disk, /dev/xvda3 is now available to be configured and used.

The new RAM now appears on the munin stats: https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/memory.html

hours, totalhours changed

chris — Fri, 24 Feb 2012 14:24:55 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.9 to 1.4

The memory available to varnish was set to 256M, I have doubled this to 512M, see wiki:NewLiveServer#varnish

The memory available to APC was set to 128M, I have doubled this to 256M, see wiki:NewLiveServer#apc

The memory available to memcache was set to 128M, I have doubled this to 256M, see wiki:NewLiveServer#apc

The max number of apache processes was set to 18, I have increased this to 25, see wiki:NewLiveServer#apache

When the apache maxclients was set to 18 the max memory useage was around 2.12GB, with the increase to 25 maxclients there is the potential for the apache memory use to reach around 3GB.

If all these changes are added up they come to over 1GB of RAM but I don't expect all of them to use up all available RAM, but it's something that I'll keep an eye on via the Munin stats https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/index.html

Some further adjustments might well be needed.

hours, totalhours changed

chris — Fri, 24 Feb 2012 18:34:02 GMT

hours changed from 0.0 to 0.2
totalhours changed from 1.4 to 1.6

Replying to chris:

The memory available to memcache was set to 128M, I have doubled this to 256M, see wiki:NewLiveServer#apc

Of course that should have been wiki:NewLiveServer#memcache

Based on a good look at the Munin stats for Memcache items https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/index.html I think this value in /etc/memcached.conf was limiting how many things were stored:

# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024

Or I have increased it to 4096 and updated the docs wiki:NewLiveServer#memcache

# Limit the number of simultaneous incoming connections. The daemon default is 1024
-c 4096

chris — Fri, 24 Feb 2012 23:30:12 GMT

Replying to chris:

I think this value in /etc/memcached.conf was limiting how many things were stored:

No it wasn't, this isn't an issue, my mistake.

We need to let memcache run for some days to see how much it will now store with the extra RAM allocation.

chris — Sat, 25 Feb 2012 12:03:01 GMT

We don't have any munin stats for APC, I will install one of these to get some:

http://tjstein.com/2010/05/apc-plugin-for-munin/ https://code.google.com/p/munin-php-apc/

hours, totalhours changed

chris — Mon, 27 Feb 2012 13:37:16 GMT

hours changed from 0.0 to 0.61
totalhours changed from 1.6 to 2.21

Replying to chris:

We don't have any munin stats for APC, I will install one of these to get some:

http://tjstein.com/2010/05/apc-plugin-for-munin/ https://code.google.com/p/munin-php-apc/

I have installed the munin-php-apc one on kiwi:

wget https://munin-php-apc.googlecode.com/files/munin_plugin_php_apc-0.1.zip
unzip munin_plugin_php_apc-0.1.zip
cd php_apc
cp apc_info.php /web/kiwi.webarch.net/www/
cp php_apc_ /usr/share/munin/plugins/
cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/php_apc_ php_apc_files
ln -s /usr/share/munin/plugins/php_apc_ php_apc_fragmentation
ln -s /usr/share/munin/plugins/php_apc_ php_apc_hit_miss
ln -s /usr/share/munin/plugins/php_apc_ php_apc_purge
ln -s /usr/share/munin/plugins/php_apc_ php_apc_rates
ln -s /usr/share/munin/plugins/php_apc_ php_apc_usage

The DocumentRoot for http://wiki.transitionnetwork.org/ was pointing to /web/transitionnetwork.org.webarch.net/www where there was all sorts lying around (an old Drupal install by the looks of it) so to make things tidy I created /web/kiwi.webarch.net/www/ and moved the files served at http://kiwi.transitionnetwork.org/ | http://kiwi.webarch.net/ there and edited the apache config at /etc/apache2/sites-available/kiwi.webarch.net.

The following was added to etc/munin/plugin-conf.d/munin-node:

[php_apc_*]
user root
env.url http://localhost/apc_info.php?auto

The stats this is generating are here: https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#php-apc

This seems to be working well, I'll now set this up on the live server.

hours, totalhours changed

chris — Mon, 27 Feb 2012 13:59:56 GMT

hours changed from 0.0 to 0.3
totalhours changed from 2.21 to 2.51

php-apc has been installed on the live server, stats here: https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/index.html#php-apc

The only difference with the dev server install was to avoid varnish serving up stale info from the php script it was bypassed by using the back-end apache port, in /etc/munin/plugin-conf.d/munin-node:

[php_apc_*]
user root
env.url http://127.0.0.1:8080/apc_info.php?auto

The docs have been updated here wiki:NewLiveServer#munin and here wiki:DevelopmentServer#Munin

Regarding memory usage on the live server, Varnish has used all the extra RAM it was given already, clearly it could use more, memcache and apc haven't yet used the additional RAM they were given. Apache hasn't had a need to use it's additional RAM yet.

If we are to get more RAM then the first application that should get some more is Varnish.

I'll leave this ticket open so we can add comments about memory use and use it to inform a decisions about getting additional RAM.

hours, totalhours changed

chris — Wed, 29 Feb 2012 13:46:44 GMT

hours changed from 0.0 to 0.2
totalhours changed from 2.51 to 2.71

Replying to chris:

Regarding memory usage on the live server, Varnish has used all the extra RAM it was given already, clearly it could use more, memcache and apc haven't yet used the additional RAM they were given. Apache hasn't had a need to use it's additional RAM yet.

If we are to get more RAM then the first application that should get some more is Varnish.

The above still stands, memcached has now started evicting things from it's cache https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/memcached_evictions/index.html -- I think it could use more memory, it's currently set to have 256M.

APC is using 167M of it's allocated 256M, this settings seems OK, https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/php_apc_usage.html

Varnish could use more RAM but I don't think any more should be allocated unless more is added physically.

status, description changed; resolution set

chris — Fri, 25 Jan 2013 16:37:49 GMT

status changed from accepted to closed
resolution set to fixed
description modified (diff)

The Drupal sites are being migrated to wiki:PuffinServer and all the other sites have been migrated to wiki:PenguinServer, so this ticket is no longer needed.