Transition Technology: Ticket #593: Migrating Puffin to a ZFS file server

status, description, summary changed

chris — Thu, 26 Sep 2013 12:01:17 GMT

status changed from new to accepted
description modified (diff)
summary changed from Migrating Parrot to a ZFS file server to Migrating Puffin to a ZFS file server

chris — Thu, 03 Oct 2013 09:05:19 GMT

Alan has made a start on this, the actual switch will take around 15 mins, it's not yet been done.

chris — Thu, 10 Oct 2013 18:19:59 GMT

Alan is planning on doing this migration tonight, see https://lists.webarch.co.uk/pipermail/webarch-xen1/2013-October/000005.html

xen1.webarch.net will be offline for maintenance between 23:45 10-October 2013 and 00:15 11-October 2013 ( i.e. 30 minutes )

Apologies for the inconvenience.

I expect it'll result in frequently accesses items to be loaded faster and less frequently accessed items to be slower.

cc changed

chris — Thu, 17 Oct 2013 12:56:29 GMT

cc aland added

Adding Alan to the CC list.

hours, totalhours changed

aland — Thu, 17 Oct 2013 22:51:16 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.0 to 0.5

Replying to chris:

Adding Alan to the CC list.

Moved to new storage.

17 min prep, 7 min move

aland — Thu, 17 Oct 2013 22:54:32 GMT

Disabled rdiff-backup

( this is done with zfs snapshots now )

attachment set

chris — Fri, 18 Oct 2013 08:06:37 GMT

attachment set to puffin_2013-10-18_diskstats_latency-day.png

attachment set

chris — Fri, 18 Oct 2013 08:06:45 GMT

attachment set to puffin-2013-10-18_diskstats_iops-day.png

attachment set

chris — Fri, 18 Oct 2013 08:30:40 GMT

attachment set to puffin-2013-10-18_mysql_innodb_bpool_act-day.png

attachment set

chris — Fri, 18 Oct 2013 08:30:51 GMT

attachment set to puffin-2013-10-18_mysql_innodb_checkpoint_age-day.png

attachment set

chris — Fri, 18 Oct 2013 08:31:00 GMT

attachment set to puffin-2013-10-18_mysql_innodb_history_length-day.png

attachment set

chris — Fri, 18 Oct 2013 08:31:08 GMT

attachment set to puffin-2013-10-18_mysql_innodb_io-day.png

attachment set

chris — Fri, 18 Oct 2013 08:31:16 GMT

attachment set to puffin-2013-10-18_mysql_myisam_indexes-day.png

hours, totalhours changed

chris — Fri, 18 Oct 2013 08:34:47 GMT

hours changed from 0.0 to 0.15
totalhours changed from 0.5 to 0.65

The migration to the ZFS file server can be seen in these Munin stats:

Both the disk latency and the disk IO look better than before.

There has also been a dramatic change in these MySQL graphs, I don't know exactly what these mean:

attachment set

chris — Fri, 18 Oct 2013 17:44:56 GMT

attachment set to puffin-2013-10-18-2_diskstats_iops-day.png

attachment set

chris — Fri, 18 Oct 2013 17:45:10 GMT

attachment set to puffin-2013-10-18-2_diskstats_latency-day.png

attachment set

chris — Fri, 18 Oct 2013 17:45:21 GMT

attachment set to puffin-2013-10-18-2_mysql_innodb_bpool_act-day.png

attachment set

chris — Fri, 18 Oct 2013 17:45:39 GMT

attachment set to puffin-2013-10-18-2_mysql_innodb_checkpoint_age-day.png

attachment set

chris — Fri, 18 Oct 2013 17:45:53 GMT

attachment set to puffin-2013-10-18-2_mysql_innodb_checkpoint_age-day.2.png

attachment set

chris — Fri, 18 Oct 2013 17:46:06 GMT

attachment set to puffin-2013-10-18-2_mysql_innodb_history_length-day.png

attachment set

chris — Fri, 18 Oct 2013 17:46:20 GMT

attachment set to puffin-2013-10-18-2_mysql_innodb_io-day.png

hours, totalhours changed

chris — Fri, 18 Oct 2013 17:59:16 GMT

hours changed from 0.0 to 0.2
totalhours changed from 0.65 to 0.85

I have spent some time today looking at the [ Munin graphs] today and the disk IO and latency pattern has changed again to something closer to what it was before the move to the ZFS server, but latency is still clearly lower.

The MySQL graphs have also returned to more usual patterns, I think the changes here have more to do with the server being rebooted than with the chance to the ZFS server.

hours, totalhours changed

chris — Sun, 20 Oct 2013 12:30:21 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.85 to 0.95

Some comments on the first load spike we have had since the migration to the ZFS server have been posted to the load spike ticket.

In terms of load times for the front page it now takes about 0.4 of a second to load the site off our servers and then an additional 0.5 of a second to load the data from YouTube (half the size of the front page, which is now 1.5MB in total, is currently down to YouTube), see wiki:LoadTimesOctThirteen.

chris — Fri, 15 Nov 2013 09:28:38 GMT

Wiki:PuffinServer has been running better since it was moved to the ZFS fileserver, I suggest we do the same for wiki:PenguinServer and wiki:ParrotServer and do it prior to upgrading them to Wheezy, see ticket:535.

status changed; resolution set

chris — Fri, 29 Nov 2013 13:11:09 GMT

status changed from accepted to closed
resolution set to fixed

chris — Wed, 02 Apr 2014 09:32:45 GMT

Note that the NFS/ZFS fileserver performance was dramatically improved in March 2014, see ticket:618#comment:5