Transition Technology: Ticket #618: Migrate Penguin and Parrot to the ZFS fileserver

attachment set

chris — Mon, 18 Nov 2013 13:18:33 GMT

attachment set to parrot-2013-11-18_diskstats_latency-week.png

attachment set

chris — Mon, 18 Nov 2013 13:18:41 GMT

attachment set to penguin-2013-11-18_diskstats_latency-week.png

hours, totalhours changed

chris — Mon, 18 Nov 2013 13:25:29 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.0 to 0.25

Alan migrated wiki:PenguinServer and wiki:ParrotServer to the ZFS server last night on ticket:535#comment:22.

I have spent some time today looking at the Munin stats and it looks to me that for twese two servers the ZFS network file server is somewhat slower that the directly attached disks that the servers were using.

wiki:ParrotServer disk latency:

wiki:PenguinServer disk latency:

It will be worth keeping an eye on these stats over the next week or so.

ed — Mon, 18 Nov 2013 14:23:40 GMT

so - the move has slowed things down? If the move has slowed things down, we measure, then move back?

sam — Wed, 22 Jan 2014 12:53:50 GMT

Hi I'm quite keen to close any tickets that we can just to get a better idea of what is a live issue. Can I close this one? Any concerns?

Thanks

Sam

attachment set

chris — Wed, 22 Jan 2014 14:32:14 GMT

attachment set to parrot_2013-01-22_xvda1-year.png

Parrot Swap

attachment set

chris — Wed, 22 Jan 2014 14:32:37 GMT

attachment set to parrot_2013-01-22_xvda2-year.png

Parrot Root

attachment set

chris — Wed, 22 Jan 2014 14:33:02 GMT

attachment set to parrot_2013-01-22_xvda3-year.png

Parrot Home

attachment set

chris — Wed, 22 Jan 2014 14:33:36 GMT

attachment set to penguin_2013-01-22_xvda1-year.png

Penguin Swap

attachment set

chris — Wed, 22 Jan 2014 14:33:54 GMT

attachment set to penguin_2013-01-22_xvda2-year.png

Penguin Root

hours, totalhours changed

chris — Wed, 22 Jan 2014 14:46:30 GMT

hours changed from 0.0 to 0.3
totalhours changed from 0.25 to 0.55

We (Webarchitects) have been somewhat concerned about the latency of the ZFS file system, last night we moved the swap partitions for all the virtual servers to local disks rather than the networked ZFS file system in the hope that this would improve performance.

On wiki:ParrotServer we have:

/dev/xvda1 swap
/dev/xvda2 /
/dev/xvda3 /home

On wiki:PenguinServer we have:

/dev/xvda1 swap
/dev/xvda2 /

Tomorrow we (Webarchitects) intend to replace a drive in the ZFS server which has been generating some SMART errors in the hope that this disk is the cause of the latency we have been seeing.

I'm afraid I think we probably need to keep this ticket open for now.

attachment set

chris — Wed, 26 Mar 2014 18:34:30 GMT

attachment set to parrot_2014-03-26_diskstats_latency-day.png

attachment set

chris — Wed, 26 Mar 2014 18:34:43 GMT

attachment set to parrot_2014-03-26_diskstats_utilization-day.png

attachment set

chris — Wed, 26 Mar 2014 18:34:56 GMT

attachment set to parrot_2014-03-26_iostat_ios-day.png

attachment set

chris — Wed, 26 Mar 2014 18:35:08 GMT

attachment set to parrot_2014-03-26_load-day.png

attachment set

chris — Wed, 26 Mar 2014 18:35:22 GMT

attachment set to penguin_2014-03-26_diskstats_latency-day.png

attachment set

chris — Wed, 26 Mar 2014 18:35:45 GMT

attachment set to penguin_2014-03-26_diskstats_utilization-day.png

attachment set

chris — Wed, 26 Mar 2014 18:36:04 GMT

attachment set to penguin_2014-03-26_load-day.png

attachment set

chris — Wed, 26 Mar 2014 18:36:18 GMT

attachment set to penguin_2014-03-26_load-day.2.png

attachment set

chris — Wed, 26 Mar 2014 18:36:37 GMT

attachment set to puffin_2014-03-26_diskstats_latency-day.png

attachment set

chris — Wed, 26 Mar 2014 18:36:52 GMT

attachment set to puffin_2014-03-26_diskstats_utilization-day.png

attachment set

chris — Wed, 26 Mar 2014 18:37:10 GMT

attachment set to puffin_2014-03-26_iostat_ios-day.png

attachment set

chris — Wed, 26 Mar 2014 18:37:24 GMT

attachment set to puffin_2014-03-26_load-day.png

hours, priority, totalhours changed

chris — Wed, 26 Mar 2014 18:50:38 GMT

hours changed from 0.0 to 0.48
priority changed from major to trivial
totalhours changed from 0.55 to 1.03

We have made a massive breakthrough on this issue, look at the following graphs from today for wiki:ParrotServer:

wiki:PenguinServer:

and wiki:PuffinServer:

This very dramatic change is a result of disabling the ZIL by running this on the NFS/ZFS zvol:

zfs set sync=disable zroot

See SOLVED: Performance Issues With FreeBSD ZFS Backed ESXi Storage Over NFS for more details.

So there is finally an answer to Ed's question from some months back:

Replying to ed:

so - the move has slowed things down? If the move has slowed things down, we measure, then move back?

The NFS/ZFS server is now probably faster than the directly attached disks, it might be worth revisiting this ticket in a few weeks to look at the annual Munin graphs to see how things are looking before closing it.

The time I have recorded with this comment is simply the time taken to upload the images and post the comment, it doesn't include any of the time spend on this issue, which included getting (and reading and deleting) an email every 5 mins for each server when the disk IO was over 1 second (countless emails over the last six months or so).

status changed; resolution set

chris — Wed, 02 Apr 2014 09:33:35 GMT

status changed from new to closed
resolution set to fixed

This is now sorted, I have also added a note on ticket:593#comment:12 for future reference.