Server time drift

[chris]

The servers are not keeping good time at the moment.

[chris]

Rdate doesn't run on wiki:PuffinServer due to the firewall, so /etc/csf/csf.conf was edited and port 37 was added to TCP_IN and TCP_OUT and the firewall restarted.

csf -r

Checking and then setting the date and checking it again:

date ; rdate -s ntp.demon.co.uk ; date
Mon Sep 30 02:22:43 BST 2013
Sun Sep 29 21:23:33 BST 2013

The following was added to the root crontab:

* * * * * rdate -s ntp.demon.co.uk

On wiki:PenguinServer

date ; rdate -s ntp.demon.co.uk ; date
Sun Sep 29 23:06:12 BST 2013
Sun Sep 29 21:25:50 BST 2013

A crontab was also added.

On wiki:ParrotServer:

date ; rdate -s ntp.demon.co.uk ; date
Sun Sep 29 21:39:25 BST 2013
Sun Sep 29 21:27:26 BST 2013

A crontab was also added.

This is just a temp workaround -- we need to solve the clock drifting, not sure what the answer is yet though.

[jim]

Puffin: This doesn't seem to be working... This has consequences for Drupal so need fixing.

tn@puffin:~/static/transition-network-d6-p005$ date
Tue Oct  8 00:38:45 BST 2013

It's 18.58 Oct 7!

rdate -s hangs, so this must have got hit but the BOA update, so I've edit /etc/csf/csf.conf to allow port 37 and set _CUSTOM_CSF to 'YES' in /root/.barracuda.cnf to avoid future clobberage.

Added crontab entry again too. Ideally we'd use a non-root user with their own cron, or another place that isn't controlled by BOA.

puffin:/data/conf# date ; rdate -s ntp.demon.co.uk ; date
Tue Oct  8 00:44:26 BST 2013
Mon Oct  7 19:02:36 BST 2013

[chris]

Replying to jim:

Ideally we'd use a non-root user with their own cron

Thanks for spotting this, and good point, I have commented the root cron job and set it to run as me.

[jim]

Per ​my comment over on #555 I have a suspicion the regular date alteration by the cron task is the cause of the load spikes...

I'd like to reduce the cron frequency to every 4 hours to see what effect that has.

I await Chris' thoughts to my brain-fart over on #555.

[chris]

I don't think the time drift has anything to do with the load spikes. The time drift only started when the motherboard was swapped and the load spikes predate this by some months.

Alan is going to try to solve the time drift issue during tonight's downtime, see ​https://lists.webarch.co.uk/pipermail/webarch-xen1/2013-October/000005.html

[jim]

There are two types of load spikes:

• Those from 'before' which were high and irregular ultimately caused by hardware issues -- fixed by the motherboard changed.
• The recent ones which are much lower in intensity and regularly spaced around the hour (or there abouts) -- these coincide with the crontab date sync being enabled as far as I can tell (see #555).

I note that the server has been rebooted (which I think was the clock fix) and the load spikes have now largely stopped.

Two questions for Chris:

1. Has the clock issue been resolved?
2. Has the crontab entry for the date sync been removed?

[chris]

Replying to jim:

There are two types of load spikes:

• Those from 'before' which were high and irregular ultimately caused by hardware issues -- fixed by the motherboard changed.

Changing the motherboard stopped the crashes and created the time drift problem but I'm not sure that anything else can be put down to it.

• The recent ones which are much lower in intensity and regularly spaced around the hour (or there abouts) -- these coincide with the crontab date sync being enabled as far as I can tell (see #555).

I note that the server has been rebooted (which I think was the clock fix) and the load spikes have now largely stopped.

The reboot was to try to fix the clock but it didn't fix it.

Two questions for Chris:

1. Has the clock issue been resolved?

No.

2. Has the crontab entry for the date sync been removed?

No, it's running as me now.

[chris]

Adding Alan to the CC list.

[aland]

Installed chrony ( replacement for ntp )
commented out crontab entry for using rdate

aptitude install chrony

watched clock with date for some few minutes

clocks now in sync

[aland]

Logged into parrot

installed chrony

clocks in sync now

[aland]

logged into penguin

installed chrony

removed cronjob

clock is running accurately now

[chris]

These two ​munin graphs changed quite dramatically when chrony was installed:

I don't know exactly what it means though.

[chris]

The server don't have a issue with time keeping since chrony has been installed as far as I'm aware, closing this ticket.

[chris]

There is a crontab entry to reset the clock after a reboot, see wiki:PuffinServer#Cron