Ticket #593 (closed maintenance: fixed)

Opened 3 years ago

Last modified 3 years ago

Migrating Puffin to a ZFS file server

Reported by: chris Owned by: chris
Priority: major Milestone: Maintenance
Component: Live server Keywords:
Cc: ed, jim, aland Estimated Number of Hours: 0.0
Add Hours to Ticket: 0 Billable?: yes
Total Hours: 0.95

Description (last modified by chris) (diff)

This ticket is for the migration of PuffinServer to a ZFS file server, this will involve some downtime but should result in better IO and easier backups.

Change History

comment:1 Changed 3 years ago by chris

  • Status changed from new to accepted
  • Description modified (diff)
  • Summary changed from Migrating Parrot to a ZFS file server to Migrating Puffin to a ZFS file server

comment:2 Changed 3 years ago by chris

Alan has made a start on this, the actual switch will take around 15 mins, it's not yet been done.

comment:3 Changed 3 years ago by chris

Alan is planning on doing this migration tonight, see https://lists.webarch.co.uk/pipermail/webarch-xen1/2013-October/000005.html

xen1.webarch.net will be offline for maintenance between 23:45 10-October 2013 and 00:15 11-October 2013 ( i.e. 30 minutes )

Apologies for the inconvenience.

I expect it'll result in frequently accesses items to be loaded faster and less frequently accessed items to be slower.

comment:4 follow-up: ↓ 5 Changed 3 years ago by chris

  • Cc aland added

Adding Alan to the CC list.

comment:5 in reply to: ↑ 4 Changed 3 years ago by aland

  • Add Hours to Ticket changed from 0.0 to 0.5
  • Total Hours changed from 0.0 to 0.5

Replying to chris:

Adding Alan to the CC list.

Moved to new storage.

17 min prep, 7 min move

comment:6 Changed 3 years ago by aland

Disabled rdiff-backup

( this is done with zfs snapshots now )

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

comment:7 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.15
  • Total Hours changed from 0.5 to 0.65

The migration to the ZFS file server can be seen in these Munin stats:



Both the disk latency and the disk IO look better than before.

There has also been a dramatic change in these MySQL graphs, I don't know exactly what these mean:






Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

Changed 3 years ago by chris

comment:8 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.2
  • Total Hours changed from 0.65 to 0.85

I have spent some time today looking at the [ Munin graphs] today and the disk IO and latency pattern has changed again to something closer to what it was before the move to the ZFS server, but latency is still clearly lower.



The MySQL graphs have also returned to more usual patterns, I think the changes here have more to do with the server being rebooted than with the chance to the ZFS server.





comment:9 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.1
  • Total Hours changed from 0.85 to 0.95

Some comments on the first load spike we have had since the migration to the ZFS server have been posted to the load spike ticket.

In terms of load times for the front page it now takes about 0.4 of a second to load the site off our servers and then an additional 0.5 of a second to load the data from YouTube (half the size of the front page, which is now 1.5MB in total, is currently down to YouTube), see wiki:LoadTimesOctThirteen.

comment:10 Changed 3 years ago by chris

Wiki:PuffinServer has been running better since it was moved to the ZFS fileserver, I suggest we do the same for wiki:PenguinServer and wiki:ParrotServer and do it prior to upgrading them to Wheezy, see ticket:535.

comment:11 Changed 3 years ago by chris

  • Status changed from accepted to closed
  • Resolution set to fixed

comment:12 Changed 3 years ago by chris

Note that the NFS/ZFS fileserver performance was dramatically improved in March 2014, see ticket:618#comment:5

Note: See TracTickets for help on using tickets.