Live server RAM and disk upgrade

[chris]

The live server is all set to have the additional 1GB of RAM added to it, it just needs a reboot at an appropriate time for this to become available. Once it's available configurations changes will be needed to Apache, Varnish etc to enable applications to make use of it.

In addition to upgrading the RAM we have made a new 15GB partition available to the virtual machine (it'll only appear after a reboot) with the intention of using this for the !MySQL databases. This partition is on a pair of mirrored SCSI disks, the rest of the virtual machine is on a pair of SATA disks.

A quick test of the new partition disk speeds:

 hdparm -tT /dev/FastDisk/quince-var-lib-mysql 

/dev/FastDisk/quince-var-lib-mysql:
 Timing cached reads:   3626 MB in  2.00 seconds = 1815.54 MB/sec
 Timing buffered disk reads:  408 MB in  3.01 seconds = 135.60 MB/sec

Compared to the SATA disks:

hdparm -tT /dev/xvda2
 
/dev/xvda2:
 Timing cached reads:   1998 MB in  2.00 seconds = 999.00 MB/sec
 Timing buffered disk reads:  92 MB in  3.03 seconds =  30.34 MB/sec

The faster partition and the switch to !InodeDB, see ticket:396 should, hopefully, result in dramatic performance improvement for !MySQL.

[chris]

The reboot has been done, it did take a couple of mins to come up as a filesystem check was done.

The new disk for MySQL had to be commented out as the machine wouldn't boot with them available, this was the error:

xm create quince.cfg 
Using config file "./quince.cfg".
Error: Device 51715 (vbd) could not be connected.
Device /dev/FastDisk/quince-var-lib-mysql is mounted in the privileged domain,
and so cannot be mounted by a guest.

I will investigate this and solve it and then we will need another quick reboot to make the disk available.

I haven't yet done any application configuration updates to make use of the additional RAM.

When munin next updates there will be a step on the memory graph: ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/memory.html

There were a few warnings in the boot log file which should probably be investigated:

Feb 23 16:57:07 quince apache2: PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, null given in
+/web/transitionnetwork.org/www/includes/database.mysql.inc on line 388
Feb 23 16:57:07 quince apache2: PHP Warning:  mysql_query() expects parameter 2 to be resource, null given in
+/web/transitionnetwork.org/www/includes/database.mysql.inc on line 121
Feb 23 16:57:07 quince apache2: PHP Warning:  mysql_errno() expects parameter 1 to be resource, null given in
+/web/transitionnetwork.org/www/includes/database.mysql.inc on line 142

[chris]

Replying to chris:

The new disk for MySQL had to be commented out as the machine wouldn't boot with them available, this was the error:

xm create quince.cfg 
Using config file "./quince.cfg".
Error: Device 51715 (vbd) could not be connected.
Device /dev/FastDisk/quince-var-lib-mysql is mounted in the privileged domain,
and so cannot be mounted by a guest.

I will investigate this and solve it and then we will need another quick reboot to make the disk available.

OK, this was simple, I unmounted the disk in Dom0 and rebooted quince again.

The new disk, /dev/xvda3 is now available to be configured and used.

The new RAM now appears on the munin stats: ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/memory.html

[chris]

The memory available to varnish was set to 256M, I have doubled this to 512M, see wiki:NewLiveServer#varnish

The memory available to APC was set to 128M, I have doubled this to 256M, see wiki:NewLiveServer#apc

The memory available to memcache was set to 128M, I have doubled this to 256M, see wiki:NewLiveServer#apc

The max number of apache processes was set to 18, I have increased this to 25, see wiki:NewLiveServer#apache

When the apache maxclients was set to 18 the max memory useage was around 2.12GB, with the increase to 25 maxclients there is the potential for the apache memory use to reach around 3GB.

If all these changes are added up they come to over 1GB of RAM but I don't expect all of them to use up all available RAM, but it's something that I'll keep an eye on via the Munin stats ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/index.html

Some further adjustments might well be needed.

[chris]

Replying to chris:

The memory available to memcache was set to 128M, I have doubled this to 256M, see wiki:NewLiveServer#apc

Of course that should have been wiki:NewLiveServer#memcache

Based on a good look at the Munin stats for Memcache items ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/index.html I think this value in /etc/memcached.conf was limiting how many things were stored:

# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024

Or I have increased it to 4096 and updated the docs wiki:NewLiveServer#memcache

# Limit the number of simultaneous incoming connections. The daemon default is 1024
-c 4096

[chris]

Replying to chris:

I think this value in /etc/memcached.conf was limiting how many things were stored:

No it wasn't, this isn't an issue, my mistake.

We need to let memcache run for some days to see how much it will now store with the extra RAM allocation.

[chris]

We don't have any munin stats for APC, I will install one of these to get some:

​http://tjstein.com/2010/05/apc-plugin-for-munin/
​https://code.google.com/p/munin-php-apc/

[chris]

Replying to chris:

We don't have any munin stats for APC, I will install one of these to get some:

​http://tjstein.com/2010/05/apc-plugin-for-munin/
​https://code.google.com/p/munin-php-apc/

I have installed the munin-php-apc one on kiwi:

wget https://munin-php-apc.googlecode.com/files/munin_plugin_php_apc-0.1.zip
unzip munin_plugin_php_apc-0.1.zip
cd php_apc
cp apc_info.php /web/kiwi.webarch.net/www/
cp php_apc_ /usr/share/munin/plugins/
cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/php_apc_ php_apc_files
ln -s /usr/share/munin/plugins/php_apc_ php_apc_fragmentation
ln -s /usr/share/munin/plugins/php_apc_ php_apc_hit_miss
ln -s /usr/share/munin/plugins/php_apc_ php_apc_purge
ln -s /usr/share/munin/plugins/php_apc_ php_apc_rates
ln -s /usr/share/munin/plugins/php_apc_ php_apc_usage

The DocumentRoot for ​http://wiki.transitionnetwork.org/ was pointing to /web/transitionnetwork.org.webarch.net/www where there was all sorts lying around (an old Drupal install by the looks of it) so to make things tidy I created /web/kiwi.webarch.net/www/ and moved the files served at ​http://kiwi.transitionnetwork.org/ | ​http://kiwi.webarch.net/ there and edited the apache config at /etc/apache2/sites-available/kiwi.webarch.net.

The following was added to etc/munin/plugin-conf.d/munin-node:

[php_apc_*]
user root
env.url http://localhost/apc_info.php?auto

The stats this is generating are here: ​https://kiwi.transitionnetwork.org/munin/webarch.net/kiwi.webarch.net/index.html#php-apc

This seems to be working well, I'll now set this up on the live server.

[chris]

php-apc has been installed on the live server, stats here: ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/index.html#php-apc

The only difference with the dev server install was to avoid varnish serving up stale info from the php script it was bypassed by using the back-end apache port, in /etc/munin/plugin-conf.d/munin-node:

[php_apc_*]
user root
env.url http://127.0.0.1:8080/apc_info.php?auto

The docs have been updated here wiki:NewLiveServer#munin and here wiki:DevelopmentServer#Munin

Regarding memory usage on the live server, Varnish has used all the extra RAM it was given already, clearly it could use more, memcache and apc haven't yet used the additional RAM they were given. Apache hasn't had a need to use it's additional RAM yet.

If we are to get more RAM then the first application that should get some more is Varnish.

I'll leave this ticket open so we can add comments about memory use and use it to inform a decisions about getting additional RAM.

[chris]

Replying to chris:

Regarding memory usage on the live server, Varnish has used all the extra RAM it was given already, clearly it could use more, memcache and apc haven't yet used the additional RAM they were given. Apache hasn't had a need to use it's additional RAM yet.

If we are to get more RAM then the first application that should get some more is Varnish.

The above still stands, memcached has now started evicting things from it's cache ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/memcached_evictions/index.html -- I think it could use more memory, it's currently set to have 256M.

APC is using 167M of it's allocated 256M, this settings seems OK, ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/php_apc_usage.html

Varnish could use more RAM but I don't think any more should be allocated unless more is added physically.

[chris]

The Drupal sites are being migrated to wiki:PuffinServer and all the other sites have been migrated to wiki:PenguinServer, so this ticket is no longer needed.