Upgrade LIVE server to Debian Squeeze

[jim]

Many of the libraries and programs available to us are limited to older versions because LIVE is on Debian Etch. Upgrading to Squeeze will allow many components to be upgraded, and hopefully provide other nice side effects.

Elements upgraded:

• Memcached (important)
• PHP/MySQL/Apache (probably?)
• Linux kernel an all associated stuff

[chris]

Debian Squeeze has PHP PHP 5.3.3-7 and drupal.org says:

Drupal 6 notes
Although PHP 5.3 is supported by Drupal 6.14 core and higher (see the release notes for 6.14), some contributed modules may not be compatible with PHP 5.3, and some PHP 5.3 configurations still show warnings with Drupal 6.14; work is ongoing to resolve the latter in 360605: PHP 5.3 Compatibility ​https://drupal.org/node/360605.
​https://drupal.org/requirements

Jim -- would this be an issue for us?

[jim]

Well I run the site (minus SSL, Varnish and Memcache) on my Ubuntu 11.04 system which has PHP v5.3.3-1ubuntu9.3...

I think the main thing was a few pass-by-reference issues and notice-level warnings popping up... I just changed the PHP warnings level (on LIVE as well) and these went away.

The site runs fine on my box as far as I've seen. I think the vast majority of PHP5.3 bugs have been fixed in the modules we use, and if a handful have not then we can patch accordingly.

[chris]

Upgrade done on kiwi: wiki:LennyToSqueeze

[chris]

The upgrade yesterday on kiwi provides some lessons for the live upgrade:

• Put the site into maintenance mode
• Dump the mysql databases
• Upgrade to squeeze
• Check the Mysql upgrade
• Import the databases

Also there are still some issues to resolve with php:

PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/apache2/php.ini on line 977 in Unknown on line 0                            
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/suhosin.ini on line 89 in Unknown on line 0                      
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20090626/uploadprogress.so' - /usr/lib/php5/20090626/uploadprogress.so:    
+cannot open shared object file: No such file or directory in Unknown on line 0    

And the subversion backups:

== errors from /etc/backup.d/30.svn ==
                                                  
Error: command failed -- /usr/bin/svnadmin hotcopy /web/tech.transitionnetwork.org/svn/. /var/backups/svn.tmp/.                                     
Error: because of earlier errors, we are leaving svn backups in /var/backups/svn.tmp instead of /var/backups/svn    

Adding time spent last night when the timer wasn't working.

[chris]

Hmm the time got reset, it should have gone to 5 hours.

[chris]

The PHP issues have been resolved and everything looks OK here: ​https://dev.transitionnetwork.org/admin/reports/status

I'm a bit concerned though that there might be some database problem - Jim any idea what all the errors here are about? ​https://dev.transitionnetwork.org/admin/reports/dblog

For example:

	User warning: Table 'dev.block_class' doesn't exist query: SELECT css_class, module, delta FROM block_class in _db_query() (line 148 of /web/dev.transitionnetwork.org.webarch.net/www/includes/database.mysql.inc).

[chris]

The issue with the backups of the subversion repo were solved by following the steps here: ​http://linuxsysadminblog.com/2010/06/svnadmin-cant-open-file-svndbfsfs-conf-no-such-file-or-directory/

[jim]

That table "block_class" missing error is a problem with that module - it's not installing itself properly. I've gone down a version and it's fine... Will open a bug report on the module issue list.

[chris]

Quite a few config file changes to upgrade to Varnish 3.0, see ​https://www.varnish-cache.org/docs/trunk/installation/upgrade.html and ​https://drupal.org/node/1054886#comment-4871274

Have still to test varnish on dev.

[jim]

Note the latest version of the Drupal Varnish module properly supports Varnish 3.x is in SVN and going up tonight hopefully... It's already on DEV.

[chris]

Ah ha, cheers for that, I should have checked with you before doing the upgrade, dev is running varnish 3.0 now and it appears to be working OK, but I'm going to do some more testing, there were quite a few changes needed in the config file.

[chris]

I testing varnish 3.0 and have put the dev site into online mode for testing.

[chris]

Varnish seems to be OK but will need checking when the 3.0 configuration from the dev server is copied onto the live server.

One thing that I have realised is that as the site works from ​http://transitionnetwork.org/ and ​http://www.transitionnetwork.org/ varnish thinks these are different pages and will cache a copy for each url, we have two options here, munge the request with the varnish configuration (both url's would still work) or redirect at an apache level, say from ​http://transitionnetwork.org/ to ​http://www.transitionnetwork.org/. I think I'd rather do an apache redirect as it's simpler if that is OK with people.

I have listed the steps to take for the upgrade here wiki:LennyToSqueeze#LiveUpgradeSteps and it's going to take a couple of hours or so to do it and looking at times at which the site is least used it might make sense to do it late at night one evening -- would that be OK with people?

[jim]

Apache redirect sounds the simplest to me... Provided it doesn't block other subdomains of course!

[chris]

Apache redirect sounds the simplest to me

OK I have done that, it shouldn't cause any problems and should dramatically reduce the amount of duplicate pages varnish is caching :-)

I have also noticed that since the upgrade the ticket URL's int he trac emails don't have full URL's any more in the sig, eg:

--                                                                                                                                                  
Ticket URL: </trac/ticket/301#comment:16>                                                                                                           
Transition Technology </trac>                                                                                                                       
Support and issues tracking for the Transition Network Web Project.         

I'll look at this next week.

[jim]

Was going to raise the trac footer link issue, hadn't realised how useful it was until it's gone.

[chris]

Trac footer is fixed if the footer is now correct below this (on the email version of this comment)!

The debian upgrade also upgraded trac and it's behaviour changed, see ​http://trac.edgewall.org/ticket/6150 and ​http://trac.edgewall.org/ticket/5064

Would late tonight be a good time for the upgrade of the live server to squeeze?

[chris]

An uncompressed Mysql dump of the live server is 2.1G!

Just dumping and database and doing the backup jobs took a long time.

One Mysql error when inserting the backups, not sure which database this was from:

ERROR 1062 (23000) at line 12403: Duplicate entry 'läßt-18817-node' for key 'word_sid_type'

The server has been upgraded and rebooted :-)

I'm going to leave the varnish upgrade till tomorrow.

Need to key an eye on things to make sure everything is OK...

[chris]

There are a lot of errors like this:

	User warning: Table 'live.block_class' doesn't exist query: SELECT css_class, module, delta FROM block_class in _db_query() (line 148 of /web/transitionnetwork.org/www/includes/database.mysql.inc).

​https://www.transitionnetwork.org/admin/reports/event/6463570

Is this a problems caused at a Mysql level?

[jim]

No worries - as per ​https://tech.transitionnetwork.org/trac/ticket/301#comment:8 faulty module that'll be fixed shortly. I've disabled it for now.

The site and the sharing engine seem fine, so all's well!... I'll do the SVN updates on Wednesday night once this has all settled..

[chris]

Munin is now working again ​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net.html

vsftp has been switchd off due to the large number of brute force attacks -- it can be switched on as needs be.

I have tried added this to settings.php so that the dblog reports have the remote IP rather than the IP of varnish for http requests, but it's not working, but it probably doesn't matter:

$conf['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';

I'll leave the varnish upgrade till the drupal module which supports Varnish 3.0 has been installed.

[jim]

OK, I'm busy tonight but will do the updates tomorrow.

[chris]

I need to fix this graph so it is of more use -- the max number of servers is 16 not 256.

​https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/apache_processes.html

I have been also looking at the memory footprint of apache and this article explains why we have such a large one:

​http://2bits.com/articles/server-indigestion-the-drupal-contributed-modules-open-buffet-binge-syndrome.html

We have 176 modules enabled, each apache process is around 90 MB.

We should probably look at nginx or at more RAM at some point...

[jim]

That article is primarily about D5 and quite old... Not saying it' s not an issue, it is, but D6 selectively loads stuff much better than D5, hence why we've got nearly twice the modules as in the article, and we use the same amount of memory.

I've just gone through the list of modules and, of 176, only 16 could be turned off without removing critical functionality. That might save us 1-10Mb per process, so might be worth a look, but disabling some will need work. Also, there are a few more coming in, and some leaving in the next SVN update.

FYI the modules I reckon could go are:

• statistics -- logs user accesses/page views, I just disabled it.
• panels_mini -- I've just disabled this as it wasn't being used.
• translation -- (but needed for other languages -- not used properly presently)
• help -- (maybe)
• locale -- (see translation)
• calendar_ical -- are we importing/exporting iCal feeds?
• og_vocab -- Do CMS need their own vocabs now REconomy have left, Ed?
• insert -- is this used? Allows people to insert images from image fields into the wysiwyg editor
• private -- about to be replaced
• terms_of_use -- could be replaced with a mandatory CCK checkbox field
• rules_admin -- can be turned off after the SVN updates happen
• googleanalytics -- Piwik is there, let's kill this
• xmlsitemap -- provides a (huge) XML sitemap to google et al... I reckon we can do without this and its children xmlsitemap_node, xmlsitemap_taxonomy at the cost of slower indexing by search engines.

Everything else is needed, and in theory in D6 only what's needed for a page is loaded... A further line of investigation is to patch core so we get stats as to how much memory the individual modules are using... ​http://2bits.com/articles/measuring-memory-consumption-by-drupal-bootstrap-and-modules.html will investigate on my machine and report back.

[ed]

CMS don't need their own vocabs.

[jim]

OG vocab now removed and uninstalled.

If we feel that the above should be turned off, please move these requests to a new ticket.

[chris]

Update Varnish to a squeeze version, edit /etc/apt/sources.list.d/varnish.list:

#deb http://repo.varnish-cache.org/debian/ lenny varnish-2.1
deb http://repo.varnish-cache.org/debian/ squeeze varnish-3.0

Upgrade:

cd /etc/varnish
/etc/init.d/varnish stop
cp default.vcl.3.0.bak default.vcl
dpkg -r varnish
aptitude update; aptitude install varnish

This has been done, there was a few mins of downtime, I have also changed the settings at ​https://www.transitionnetwork.org/admin/settings/varnish/general to match the varnish version.