Transition Technology: Ticket #554: Site slow down and MySQL load increase

hours, totalhours changed

chris — Tue, 28 May 2013 18:11:51 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.0 to 0.25

attachment set

chris — Tue, 28 May 2013 18:12:34 GMT

attachment set to puffin-mysql_queries-week-2013-05-28.png

MySQL Queries

attachment set

chris — Tue, 28 May 2013 18:12:48 GMT

attachment set to puffin-mysql_bytes-week-2013-05-28.png

MySQL Bytes

attachment set

chris — Tue, 28 May 2013 18:13:08 GMT

attachment set to puffin-multips_memory-week-2013-05-28.png

Memory usage

attachment set

chris — Tue, 28 May 2013 18:13:24 GMT

attachment set to puffin-cpu-week-2013-05-28.png

CPU usage

attachment set

chris — Tue, 28 May 2013 18:13:38 GMT

attachment set to puffin-fw_conntrack-week-2013-05-28.png

Firewall

attachment set

chris — Tue, 28 May 2013 18:13:53 GMT

attachment set to puffin-if_eth0-week-2013-05-28.png

Firewall

attachment set

chris — Tue, 28 May 2013 18:14:06 GMT

attachment set to puffin-fw_packets-week-2013-05-28.png

Firewall

hours, totalhours changed

chris — Tue, 28 May 2013 18:20:56 GMT

hours changed from 0.0 to 0.15
totalhours changed from 0.25 to 0.4

These are the munin graphs from https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/ which have been attached to this ticket:

attachment set

chris — Tue, 28 May 2013 18:31:08 GMT

attachment set to puffin-load-month-2013-05-28.png

Load

chris — Tue, 28 May 2013 18:34:33 GMT

The load average has gone from less than 1 to over 2 since midday on 24th May 2013 as well, see (note this is a monthly graph not a weekly one as the others above):

chris — Tue, 28 May 2013 18:55:07 GMT

Using Apache banch to request the front page (just the index.php file, no css, js or images) 200 times running 10 concurrent requests at a time, from another machine in the same rack is something we have stats for from the last few years -- for comparison it now looks like the site is now roughly back to the speed it was when it was running with apache without varnish, two years ago:

wiki:LoadTimes#a2013-05-28 5.539 seconds wiki:PuffinServer
wiki:LoadTimes#a2013-01-28 0.253 seconds wiki:NewLiveServer
wiki:LoadTimes#a2013-01-28 0.262 seconds wiki:PuffinServer
wiki:LoadTimes#a2011-07-13 3.417 seconds wiki:NewLiveServer without varnish
wiki:LoadTimes#a2011-07-05 5.131 seconds wiki:NewLiveServer without varnish

hours, status, totalhours changed; resolution set

jim — Tue, 28 May 2013 22:52:39 GMT

hours changed from 0.0 to 0.1
status changed from new to closed
resolution set to fixed
totalhours changed from 0.4 to 0.5

It's because Redis couldn't be used since the global.inc replacement... the fix I said to do wiped the Redis password from /data/conf/global.inc - near the bottom: $conf['redis_client_password'] = 'isfoobared';

I replaced this with the correct pw from global.inc.bak, which is the auto-generated password BOA creates.

This explains the more load/connections to MySQL and the slowness.

Apologies for missing this, should be fixed now. Graphs settling back down...

hours, totalhours changed

chris — Tue, 28 May 2013 23:15:58 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.5 to 0.75

Thanks Jim, that did the trick, sorry for not spotting that the slowness was down to redis being unavailable, I had checked it was running a few times but didn't think to check the password.

I have checked the speed of the site and it's back to loading the front page and all the js, css and images in less than a second via pingdom.com and 200 copies of index.php loads in less than 0.2 seconds, see wiki:LoadTimes#a2013-05-29

ed — Wed, 29 May 2013 06:46:21 GMT

good work lads - going faster than shit off a shovel now. even the panels pages. nice.