Ticket #593 (closed maintenance: fixed)
Migrating Puffin to a ZFS file server
| Reported by: | chris | Owned by: | chris |
|---|---|---|---|
| Priority: | major | Milestone: | Maintenance |
| Component: | Live server | Keywords: | |
| Cc: | ed, jim, aland | Estimated Number of Hours: | 0.0 |
| Add Hours to Ticket: | 0 | Billable?: | yes |
| Total Hours: | 0.95 |
Description (last modified by chris) (diff)
This ticket is for the migration of PuffinServer to a ZFS file server, this will involve some downtime but should result in better IO and easier backups.
Attachments
Change History
comment:1 Changed 3 years ago by chris
- Status changed from new to accepted
- Description modified (diff)
- Summary changed from Migrating Parrot to a ZFS file server to Migrating Puffin to a ZFS file server
comment:2 Changed 3 years ago by chris
Alan has made a start on this, the actual switch will take around 15 mins, it's not yet been done.
comment:3 Changed 3 years ago by chris
Alan is planning on doing this migration tonight, see https://lists.webarch.co.uk/pipermail/webarch-xen1/2013-October/000005.html
xen1.webarch.net will be offline for maintenance between 23:45 10-October 2013 and 00:15 11-October 2013 ( i.e. 30 minutes )
Apologies for the inconvenience.
I expect it'll result in frequently accesses items to be loaded faster and less frequently accessed items to be slower.
comment:5 in reply to: ↑ 4 Changed 3 years ago by aland
- Add Hours to Ticket changed from 0.0 to 0.5
- Total Hours changed from 0.0 to 0.5
comment:6 Changed 3 years ago by aland
Disabled rdiff-backup
( this is done with zfs snapshots now )
Changed 3 years ago by chris
- Attachment puffin-2013-10-18_mysql_innodb_checkpoint_age-day.png added
Changed 3 years ago by chris
- Attachment puffin-2013-10-18_mysql_innodb_history_length-day.png added
comment:7 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 0.15
- Total Hours changed from 0.5 to 0.65
The migration to the ZFS file server can be seen in these Munin stats:
Both the disk latency and the disk IO look better than before.
There has also been a dramatic change in these MySQL graphs, I don't know exactly what these mean:
Changed 3 years ago by chris
- Attachment puffin-2013-10-18-2_mysql_innodb_checkpoint_age-day.png added
Changed 3 years ago by chris
- Attachment puffin-2013-10-18-2_mysql_innodb_checkpoint_age-day.2.png added
Changed 3 years ago by chris
- Attachment puffin-2013-10-18-2_mysql_innodb_history_length-day.png added
comment:8 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 0.2
- Total Hours changed from 0.65 to 0.85
I have spent some time today looking at the [ Munin graphs] today and the disk IO and latency pattern has changed again to something closer to what it was before the move to the ZFS server, but latency is still clearly lower.
The MySQL graphs have also returned to more usual patterns, I think the changes here have more to do with the server being rebooted than with the chance to the ZFS server.
comment:9 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 0.1
- Total Hours changed from 0.85 to 0.95
Some comments on the first load spike we have had since the migration to the ZFS server have been posted to the load spike ticket.
In terms of load times for the front page it now takes about 0.4 of a second to load the site off our servers and then an additional 0.5 of a second to load the data from YouTube (half the size of the front page, which is now 1.5MB in total, is currently down to YouTube), see wiki:LoadTimesOctThirteen.
comment:10 Changed 3 years ago by chris
Wiki:PuffinServer has been running better since it was moved to the ZFS fileserver, I suggest we do the same for wiki:PenguinServer and wiki:ParrotServer and do it prior to upgrading them to Wheezy, see ticket:535.
comment:11 Changed 3 years ago by chris
- Status changed from accepted to closed
- Resolution set to fixed
comment:12 Changed 3 years ago by chris
Note that the NFS/ZFS fileserver performance was dramatically improved in March 2014, see ticket:618#comment:5
