Transition Technology: Ticket #301: Upgrade LIVE server to Debian Squeeze

chris — Thu, 08 Sep 2011 14:21:43 GMT

Debian Squeeze has PHP PHP 5.3.3-7 and drupal.org says:

Drupal 6 notes Although PHP 5.3 is supported by Drupal 6.14 core and higher (see the release notes for 6.14), some contributed modules may not be compatible with PHP 5.3, and some PHP 5.3 configurations still show warnings with Drupal 6.14; work is ongoing to resolve the latter in 360605: PHP 5.3 Compatibility https://drupal.org/node/360605. https://drupal.org/requirements

Jim -- would this be an issue for us?

cc set

chris — Thu, 08 Sep 2011 14:22:15 GMT

cc jim, ed added

type changed

jim — Thu, 08 Sep 2011 14:55:11 GMT

type changed from defect to task

Well I run the site (minus SSL, Varnish and Memcache) on my Ubuntu 11.04 system which has PHP v5.3.3-1ubuntu9.3...

I think the main thing was a few pass-by-reference issues and notice-level warnings popping up... I just changed the PHP warnings level (on LIVE as well) and these went away.

The site runs fine on my box as far as I've seen. I think the vast majority of PHP5.3 bugs have been fixed in the modules we use, and if a handful have not then we can patch accordingly.

status, type changed

chris — Wed, 14 Sep 2011 08:32:27 GMT

status changed from new to accepted
type changed from task to maintenance

hours changed

chris — Wed, 21 Sep 2011 14:15:14 GMT

hours changed from 0 to 2.5

Upgrade done on kiwi: wiki:LennyToSqueeze

hours, totalhours changed

chris — Thu, 22 Sep 2011 11:11:36 GMT

hours changed from 0.0 to 2.5
totalhours changed from 0.0 to 2.5

The upgrade yesterday on kiwi provides some lessons for the live upgrade:

Put the site into maintenance mode
Dump the mysql databases
Upgrade to squeeze
Check the Mysql upgrade
Import the databases

Also there are still some issues to resolve with php:

PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/apache2/php.ini on line 977 in Unknown on line 0
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/suhosin.ini on line 89 in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20090626/uploadprogress.so' - /usr/lib/php5/20090626/uploadprogress.so:
+cannot open shared object file: No such file or directory in Unknown on line 0

And the subversion backups:

== errors from /etc/backup.d/30.svn ==
Error: command failed -- /usr/bin/svnadmin hotcopy /web/tech.transitionnetwork.org/svn/. /var/backups/svn.tmp/.
Error: because of earlier errors, we are leaving svn backups in /var/backups/svn.tmp instead of /var/backups/svn

Adding time spent last night when the timer wasn't working.

hours, totalhours changed

chris — Thu, 22 Sep 2011 11:12:52 GMT

hours changed from 0.0 to 2.5
totalhours changed from 2.5 to 5.0

Hmm the time got reset, it should have gone to 5 hours.

hours, totalhours changed

chris — Thu, 22 Sep 2011 13:18:08 GMT

hours changed from 0.0 to 0.7
totalhours changed from 5.0 to 5.7

The PHP issues have been resolved and everything looks OK here: https://dev.transitionnetwork.org/admin/reports/status

I'm a bit concerned though that there might be some database problem - Jim any idea what all the errors here are about? https://dev.transitionnetwork.org/admin/reports/dblog

For example:

	User warning: Table &#039;dev.block_class&#039; doesn&#039;t exist query: SELECT css_class, module, delta FROM block_class in _db_query() (line 148 of /web/dev.transitionnetwork.org.webarch.net/www/includes/database.mysql.inc).

hours, totalhours changed

chris — Thu, 22 Sep 2011 14:50:05 GMT

hours changed from 0.0 to 0.51
totalhours changed from 5.7 to 6.21

The issue with the backups of the subversion repo were solved by following the steps here: http://linuxsysadminblog.com/2010/06/svnadmin-cant-open-file-svndbfsfs-conf-no-such-file-or-directory/

hours, totalhours changed

jim — Thu, 22 Sep 2011 15:14:17 GMT

hours changed from 0.0 to 0.5
totalhours changed from 6.21 to 6.71

That table "block_class" missing error is a problem with that module - it's not installing itself properly. I've gone down a version and it's fine... Will open a bug report on the module issue list.

hours, totalhours changed

chris — Thu, 22 Sep 2011 15:50:29 GMT

hours changed from 0.0 to 1.0
totalhours changed from 6.71 to 7.71

Quite a few config file changes to upgrade to Varnish 3.0, see https://www.varnish-cache.org/docs/trunk/installation/upgrade.html and https://drupal.org/node/1054886#comment-4871274

Have still to test varnish on dev.

hours, totalhours changed

jim — Thu, 22 Sep 2011 16:08:57 GMT

hours changed from 0.0 to 0.05
totalhours changed from 7.71 to 7.76

Note the latest version of the Drupal Varnish module properly supports Varnish 3.x is in SVN and going up tonight hopefully... It's already on DEV.

hours, totalhours changed

chris — Thu, 22 Sep 2011 16:43:10 GMT

hours changed from 0.0 to 0.7
totalhours changed from 7.76 to 8.46

Ah ha, cheers for that, I should have checked with you before doing the upgrade, dev is running varnish 3.0 now and it appears to be working OK, but I'm going to do some more testing, there were quite a few changes needed in the config file.

hours, totalhours changed

chris — Fri, 23 Sep 2011 09:44:05 GMT

hours changed from 0.0 to 0.42
totalhours changed from 8.46 to 8.88

I testing varnish 3.0 and have put the dev site into online mode for testing.

hours, totalhours changed

chris — Fri, 23 Sep 2011 13:36:51 GMT

hours changed from 0.0 to 2.0
totalhours changed from 8.88 to 10.88

Varnish seems to be OK but will need checking when the 3.0 configuration from the dev server is copied onto the live server.

One thing that I have realised is that as the site works from http://transitionnetwork.org/ and http://www.transitionnetwork.org/ varnish thinks these are different pages and will cache a copy for each url, we have two options here, munge the request with the varnish configuration (both url's would still work) or redirect at an apache level, say from http://transitionnetwork.org/ to http://www.transitionnetwork.org/. I think I'd rather do an apache redirect as it's simpler if that is OK with people.

I have listed the steps to take for the upgrade here wiki:LennyToSqueeze#LiveUpgradeSteps and it's going to take a couple of hours or so to do it and looking at times at which the site is least used it might make sense to do it late at night one evening -- would that be OK with people?

jim — Fri, 23 Sep 2011 18:45:57 GMT

Apache redirect sounds the simplest to me... Provided it doesn't block other subdomains of course!

hours, totalhours changed

chris — Sat, 24 Sep 2011 12:08:55 GMT

hours changed from 0.0 to 0.2
totalhours changed from 10.88 to 11.08

Apache redirect sounds the simplest to me

OK I have done that, it shouldn't cause any problems and should dramatically reduce the amount of duplicate pages varnish is caching :-)

I have also noticed that since the upgrade the ticket URL's int he trac emails don't have full URL's any more in the sig, eg:

--
Ticket URL: </trac/ticket/301#comment:16>
Transition Technology </trac>
Support and issues tracking for the Transition Network Web Project.

I'll look at this next week.

jim — Sat, 24 Sep 2011 12:27:59 GMT

Was going to raise the trac footer link issue, hadn't realised how useful it was until it's gone.

hours, totalhours changed

chris — Mon, 26 Sep 2011 13:25:53 GMT

hours changed from 0.0 to 0.2
totalhours changed from 11.08 to 11.28

Trac footer is fixed if the footer is now correct below this (on the email version of this comment)!

The debian upgrade also upgraded trac and it's behaviour changed, see http://trac.edgewall.org/ticket/6150 and http://trac.edgewall.org/ticket/5064

Would late tonight be a good time for the upgrade of the live server to squeeze?

hours, totalhours changed

chris — Mon, 26 Sep 2011 22:19:43 GMT

hours changed from 0.0 to 1.8
totalhours changed from 11.28 to 13.08

An uncompressed Mysql dump of the live server is 2.1G!

Just dumping and database and doing the backup jobs took a long time.

One Mysql error when inserting the backups, not sure which database this was from:

ERROR 1062 (23000) at line 12403: Duplicate entry 'läßt-18817-node' for key 'word_sid_type'

The server has been upgraded and rebooted :-)

I'm going to leave the varnish upgrade till tomorrow.

Need to key an eye on things to make sure everything is OK...

hours, totalhours changed

chris — Mon, 26 Sep 2011 22:45:39 GMT

hours changed from 0.0 to 0.2
totalhours changed from 13.08 to 13.28

There are a lot of errors like this:

	User warning: Table &#039;live.block_class&#039; doesn&#039;t exist query: SELECT css_class, module, delta FROM block_class in _db_query() (line 148 of /web/transitionnetwork.org/www/includes/database.mysql.inc).

https://www.transitionnetwork.org/admin/reports/event/6463570

Is this a problems caused at a Mysql level?

hours, totalhours changed

jim — Mon, 26 Sep 2011 23:02:25 GMT

hours changed from 0.0 to 0.1
totalhours changed from 13.28 to 13.38

No worries - as per https://tech.transitionnetwork.org/trac/ticket/301#comment:8 faulty module that'll be fixed shortly. I've disabled it for now.

The site and the sharing engine seem fine, so all's well!... I'll do the SVN updates on Wednesday night once this has all settled..

hours, totalhours changed

chris — Tue, 27 Sep 2011 16:17:42 GMT

hours changed from 0.0 to 2.0
totalhours changed from 13.38 to 15.38

Munin is now working again https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net.html

vsftp has been switchd off due to the large number of brute force attacks -- it can be switched on as needs be.

I have tried added this to settings.php so that the dblog reports have the remote IP rather than the IP of varnish for http requests, but it's not working, but it probably doesn't matter:

$conf['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';

I'll leave the varnish upgrade till the drupal module which supports Varnish 3.0 has been installed.

jim — Tue, 27 Sep 2011 16:44:49 GMT

OK, I'm busy tonight but will do the updates tomorrow.

hours, totalhours changed

chris — Wed, 28 Sep 2011 14:11:32 GMT

hours changed from 0.0 to 0.5
totalhours changed from 15.38 to 15.88

I need to fix this graph so it is of more use -- the max number of servers is 16 not 256.

https://kiwi.transitionnetwork.org/munin/webarch.net/quince.webarch.net/apache_processes.html

I have been also looking at the memory footprint of apache and this article explains why we have such a large one:

http://2bits.com/articles/server-indigestion-the-drupal-contributed-modules-open-buffet-binge-syndrome.html

We have 176 modules enabled, each apache process is around 90 MB.

We should probably look at nginx or at more RAM at some point...

hours, totalhours changed

jim — Wed, 28 Sep 2011 14:47:47 GMT

hours changed from 0.0 to 0.5
totalhours changed from 15.88 to 16.38

jim — Wed, 28 Sep 2011 14:48:13 GMT

That article is primarily about D5 and quite old... Not saying it' s not an issue, it is, but D6 selectively loads stuff much better than D5, hence why we've got nearly twice the modules as in the article, and we use the same amount of memory.

I've just gone through the list of modules and, of 176, only 16 could be turned off without removing critical functionality. That might save us 1-10Mb per process, so might be worth a look, but disabling some will need work. Also, there are a few more coming in, and some leaving in the next SVN update.

FYI the modules I reckon could go are:

statistics -- logs user accesses/page views, I just disabled it.
panels_mini -- I've just disabled this as it wasn't being used.
translation -- (but needed for other languages -- not used properly presently)
help -- (maybe)
locale -- (see translation)
calendar_ical -- are we importing/exporting iCal feeds?
og_vocab -- Do CMS need their own vocabs now REconomy have left, Ed?
insert -- is this used? Allows people to insert images from image fields into the wysiwyg editor
private -- about to be replaced
terms_of_use -- could be replaced with a mandatory CCK checkbox field
rules_admin -- can be turned off after the SVN updates happen
googleanalytics -- Piwik is there, let's kill this
xmlsitemap -- provides a (huge) XML sitemap to google et al... I reckon we can do without this and its children xmlsitemap_node, xmlsitemap_taxonomy at the cost of slower indexing by search engines.

Everything else is needed, and in theory in D6 only what's needed for a page is loaded... A further line of investigation is to patch core so we get stats as to how much memory the individual modules are using... http://2bits.com/articles/measuring-memory-consumption-by-drupal-bootstrap-and-modules.html will investigate on my machine and report back.

ed — Wed, 28 Sep 2011 15:00:01 GMT

CMS don't need their own vocabs.

hours, totalhours changed

jim — Wed, 28 Sep 2011 17:06:23 GMT

hours changed from 0.0 to 0.075
totalhours changed from 16.38 to 16.455

OG vocab now removed and uninstalled.

If we feel that the above should be turned off, please move these requests to a new ticket.

status changed; resolution set

ed — Thu, 06 Oct 2011 13:31:06 GMT

status changed from accepted to closed
resolution set to fixed

hours, totalhours changed

chris — Mon, 10 Oct 2011 12:31:55 GMT

hours changed from 0.0 to 0.32
totalhours changed from 16.455 to 16.775

Update Varnish to a squeeze version, edit /etc/apt/sources.list.d/varnish.list:

#deb http://repo.varnish-cache.org/debian/ lenny varnish-2.1
deb http://repo.varnish-cache.org/debian/ squeeze varnish-3.0

Upgrade:

cd /etc/varnish
/etc/init.d/varnish stop
cp default.vcl.3.0.bak default.vcl
dpkg -r varnish
aptitude update; aptitude install varnish

This has been done, there was a few mins of downtime, I have also changed the settings at https://www.transitionnetwork.org/admin/settings/varnish/general to match the varnish version.