Ticket #828 (closed maintenance: fixed)
Site down due to massive load spike 2015-01-29
| Reported by: | chris | Owned by: | chris |
|---|---|---|---|
| Priority: | major | Milestone: | Maintenance |
| Component: | Live server | Keywords: | |
| Cc: | ade, sam, annesley, paul | Estimated Number of Hours: | 0.0 |
| Add Hours to Ticket: | 0 | Billable?: | yes |
| Total Hours: | 0.25 |
Description
Seems to be recovering now, this ticket is to try to find the cause.
Change History
comment:1 Changed 22 months ago by chris
- Add Hours to Ticket changed from 0.0 to 0.25
- Total Hours changed from 0.0 to 0.25
comment:2 Changed 22 months ago by chris
There was another load spike just now, not quite as big, the ldf email:
Date: Fri, 30 Jan 2015 11:16:23 +0000 (GMT)
From: root@puffin.webarch.net
To: chris@webarchitects.co.uk
Subject: lfd on puffin.webarch.net: High 5 minute load average alert - 46.65
Time: Fri Jan 30 11:16:23 2015 +0000
1 Min Load Avg: 67.43
5 Min Load Avg: 46.65
15 Min Load Avg: 21.96
Running/Total Processes: 77/544
Output from ps:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S 2014 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 2014 24:14 \_ [migration/0]
root 4 0.0 0.0 0 0 ? S 2014 59:33 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/0]
root 6 0.0 0.0 0 0 ? S 2014 22:36 \_ [migration/1]
root 7 0.0 0.0 0 0 ? S 2014 64:31 \_ [ksoftirqd/1]
root 8 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/1]
root 9 0.0 0.0 0 0 ? S 2014 20:42 \_ [migration/2]
root 10 0.0 0.0 0 0 ? S 2014 67:15 \_ [ksoftirqd/2]
root 11 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/2]
root 12 0.0 0.0 0 0 ? S 2014 20:50 \_ [migration/3]
root 13 0.0 0.0 0 0 ? S 2014 70:08 \_ [ksoftirqd/3]
root 14 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/3]
root 15 0.0 0.0 0 0 ? S 2014 20:32 \_ [migration/4]
root 16 0.0 0.0 0 0 ? S 2014 72:41 \_ [ksoftirqd/4]
root 17 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/4]
root 18 0.0 0.0 0 0 ? S 2014 19:25 \_ [migration/5]
root 19 0.0 0.0 0 0 ? S 2014 73:49 \_ [ksoftirqd/5]
root 20 0.0 0.0 0 0 ? S 2014 0:01 \_ [watchdog/5]
root 21 0.0 0.0 0 0 ? S 2014 19:02 \_ [migration/6]
root 22 0.0 0.0 0 0 ? S 2014 74:17 \_ [ksoftirqd/6]
root 23 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/6]
root 24 0.0 0.0 0 0 ? S 2014 19:00 \_ [migration/7]
root 25 0.0 0.0 0 0 ? S 2014 74:51 \_ [ksoftirqd/7]
root 26 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/7]
root 27 0.0 0.0 0 0 ? S 2014 18:59 \_ [migration/8]
root 28 0.0 0.0 0 0 ? S 2014 74:28 \_ [ksoftirqd/8]
root 29 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/8]
root 30 0.0 0.0 0 0 ? S 2014 18:56 \_ [migration/9]
root 31 0.0 0.0 0 0 ? S 2014 76:17 \_ [ksoftirqd/9]
root 32 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/9]
root 33 0.0 0.0 0 0 ? S 2014 18:57 \_ [migration/10]
root 34 0.0 0.0 0 0 ? S 2014 74:45 \_ [ksoftirqd/10]
root 35 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/10]
root 36 0.0 0.0 0 0 ? S 2014 19:12 \_ [migration/11]
root 37 0.0 0.0 0 0 ? S 2014 75:41 \_ [ksoftirqd/11]
root 38 0.0 0.0 0 0 ? S 2014 0:01 \_ [watchdog/11]
root 39 0.0 0.0 0 0 ? S 2014 19:39 \_ [migration/12]
root 40 0.0 0.0 0 0 ? S 2014 73:13 \_ [ksoftirqd/12]
root 41 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/12]
root 42 0.0 0.0 0 0 ? S 2014 20:21 \_ [migration/13]
root 43 0.0 0.0 0 0 ? S 2014 16:52 \_ [ksoftirqd/13]
root 44 0.0 0.0 0 0 ? S 2014 0:00 \_ [watchdog/13]
root 45 0.0 0.0 0 0 ? S 2014 5:07 \_ [events/0]
root 46 0.0 0.0 0 0 ? S 2014 5:08 \_ [events/1]
root 47 0.0 0.0 0 0 ? S 2014 4:55 \_ [events/2]
root 48 0.0 0.0 0 0 ? S 2014 4:46 \_ [events/3]
root 49 0.0 0.0 0 0 ? S 2014 4:21 \_ [events/4]
root 50 0.0 0.0 0 0 ? S 2014 4:25 \_ [events/5]
root 51 0.0 0.0 0 0 ? S 2014 4:29 \_ [events/6]
root 52 0.0 0.0 0 0 ? S 2014 4:07 \_ [events/7]
root 53 0.0 0.0 0 0 ? S 2014 4:10 \_ [events/8]
root 54 0.0 0.0 0 0 ? S 2014 4:08 \_ [events/9]
root 55 0.0 0.0 0 0 ? S 2014 4:02 \_ [events/10]
root 56 0.0 0.0 0 0 ? S 2014 4:05 \_ [events/11]
root 57 0.0 0.0 0 0 ? S 2014 4:11 \_ [events/12]
root 58 0.0 0.0 0 0 ? S 2014 9:57 \_ [events/13]
root 59 0.0 0.0 0 0 ? S 2014 0:00 \_ [cpuset]
root 60 0.0 0.0 0 0 ? S 2014 0:00 \_ [khelper]
root 61 0.0 0.0 0 0 ? S 2014 0:00 \_ [netns]
root 62 0.0 0.0 0 0 ? S 2014 0:00 \_ [async/mgr]
root 63 0.0 0.0 0 0 ? S 2014 0:00 \_ [pm]
root 64 0.0 0.0 0 0 ? S 2014 0:00 \_ [xenwatch]
root 65 0.0 0.0 0 0 ? S 2014 0:00 \_ [xenbus]
root 66 0.0 0.0 0 0 ? S 2014 0:32 \_ [sync_supers]
root 67 0.0 0.0 0 0 ? S 2014 9:34 \_ [bdi-default]
root 68 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/0]
root 69 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/1]
root 70 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/2]
root 71 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/3]
root 72 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/4]
root 73 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/5]
root 74 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/6]
root 75 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/7]
root 76 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/8]
root 77 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/9]
root 78 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/10]
root 79 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/11]
root 80 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/12]
root 81 0.0 0.0 0 0 ? S 2014 0:00 \_ [kintegrityd/13]
root 82 0.0 0.0 0 0 ? S 2014 2:56 \_ [kblockd/0]
root 83 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/1]
root 84 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/2]
root 85 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/3]
root 86 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/4]
root 87 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/5]
root 88 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/6]
root 89 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/7]
root 90 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/8]
root 91 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/9]
root 92 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/10]
root 93 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/11]
root 94 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/12]
root 95 0.0 0.0 0 0 ? S 2014 0:00 \_ [kblockd/13]
root 96 0.0 0.0 0 0 ? S 2014 0:00 \_ [kseriod]
root 111 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/0]
root 112 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/1]
root 113 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/2]
root 114 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/3]
root 115 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/4]
root 116 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/5]
root 117 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/6]
root 118 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/7]
root 119 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/8]
root 120 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/9]
root 121 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/10]
root 122 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/11]
root 123 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/12]
root 124 0.0 0.0 0 0 ? S 2014 0:00 \_ [kondemand/13]
root 125 0.0 0.0 0 0 ? S 2014 0:13 \_ [khungtaskd]
root 126 0.0 0.0 0 0 ? S 2014 5:20 \_ [kswapd0]
root 127 0.0 0.0 0 0 ? SN 2014 0:00 \_ [ksmd]
root 128 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/0]
root 129 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/1]
root 130 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/2]
root 131 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/3]
root 132 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/4]
root 133 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/5]
root 134 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/6]
root 135 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/7]
root 136 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/8]
root 137 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/9]
root 138 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/10]
root 139 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/11]
root 140 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/12]
root 141 0.0 0.0 0 0 ? S 2014 0:00 \_ [aio/13]
root 142 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/0]
root 143 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/1]
root 144 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/2]
root 145 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/3]
root 146 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/4]
root 147 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/5]
root 148 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/6]
root 149 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/7]
root 150 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/8]
root 151 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/9]
root 152 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/10]
root 153 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/11]
root 154 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/12]
root 155 0.0 0.0 0 0 ? S 2014 0:00 \_ [crypto/13]
root 158 0.0 0.0 0 0 ? S 2014 0:00 \_ [khvcd]
root 203 0.0 0.0 0 0 ? S 2014 0:00 \_ [kstriped]
root 213 0.0 0.0 0 0 ? S 2014 60:23 \_ [jbd2/xvda2-8]
root 214 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 215 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 216 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 217 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 218 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 219 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 220 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 221 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 222 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 223 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 224 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 225 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 226 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 227 0.0 0.0 0 0 ? S 2014 0:00 \_ [ext4-dio-unwrit]
root 239 0.0 0.0 0 0 ? S 2014 45:24 \_ [flush-202:2]
root 2036 0.0 0.0 0 0 ? S 2014 3:25 \_ [kauditd]
root 1 0.0 0.0 10604 704 ? Ss 2014 29:21 init [2]
root 376 0.0 0.0 21484 1084 ? Ss 2014 0:00 udevd --daemon
root 55776 0.0 0.0 21480 984 ? S 2014 0:00 \_ udevd --daemon
root 2012 0.1 0.0 6640 524 ? Ss 2014 238:01 /usr/sbin/vnstatd -d
103 6926 0.0 0.0 29876 1108 ? Ss 2014 18:12 /usr/bin/dbus-daemon --system
root 7043 0.0 0.0 13084 760 ? S 2014 3:32 /usr/sbin/chronyd
root 9197 0.0 0.0 14528 468 hvc0 Ss+ 2014 0:00 /sbin/getty 38400 hvc0
root 11605 0.0 0.0 193048 3348 ? Sl 2014 108:37 /usr/sbin/console-kit-daemon --no-daemon
root 11962 0.0 0.0 122880 2188 ? Sl 2014 25:40 /usr/lib/policykit-1/polkitd --no-debug
root 3452 0.0 0.0 23360 832 ? Ss 2014 0:00 SCREEN
root 3453 0.0 0.0 19476 1328 pts/1 Ss+ 2014 0:00 \_ /bin/bash
root 23669 0.0 0.1 123312 9780 ? Sl 2014 61:30 /usr/sbin/rsyslogd -c5
daemon 30919 0.0 0.0 16628 340 ? Ss 2014 0:00 /usr/sbin/atd
root 38937 0.0 0.0 111012 1084 ? S<sl 2014 5:08 /sbin/auditd
root 38939 0.0 0.0 145536 1000 ? S<sl 2014 2:45 \_ /sbin/audispd
root 50089 0.0 0.1 53324 9996 ? Ss 2014 51:49 munin-node
root 53942 0.0 0.0 20368 948 ? Ss 2014 30:39 /usr/sbin/cron
root 19356 0.0 0.0 45780 1340 ? S 11:01 0:00 \_ /USR/SBIN/CRON
root 19362 0.0 0.0 4136 604 ? Ss 11:01 0:00 | \_ /bin/dash -c bash /var/xdrago/guest-water.sh >/dev/null 2>&1
root 19407 0.6 0.0 10768 1616 ? S 11:01 0:05 | \_ bash /var/xdrago/guest-water.sh
root 39914 0.0 0.0 28684 7860 ? R 11:16 0:00 | \_ /usr/bin/perl /usr/sbin/csf -dr 81.95.52.103
root 32301 0.0 0.0 45780 1340 ? S 11:10 0:00 \_ /USR/SBIN/CRON
root 32311 0.8 0.0 4136 600 ? Ss 11:10 0:03 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 32381 0.8 0.0 10752 1664 ? S 11:10 0:03 | \_ bash /var/xdrago/minute.sh
root 32302 0.0 0.0 45780 1340 ? S 11:10 0:00 \_ /USR/SBIN/CRON
root 32312 0.3 0.0 4136 604 ? Ss 11:10 0:01 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 32377 0.9 0.0 10848 1708 ? S 11:10 0:03 | \_ bash /var/xdrago/second.sh
root 39931 0.0 0.0 17532 2612 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 32548 0.0 0.0 45780 1340 ? S 11:11 0:00 \_ /USR/SBIN/CRON
root 32552 0.1 0.0 4136 600 ? Ss 11:11 0:00 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 32690 0.3 0.0 10748 1660 ? S 11:11 0:01 | \_ bash /var/xdrago/minute.sh
root 32549 0.0 0.0 45780 1340 ? S 11:11 0:00 \_ /USR/SBIN/CRON
root 32553 0.0 0.0 4136 600 ? Ss 11:11 0:00 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 32594 0.2 0.0 10848 1712 ? S 11:11 0:00 | \_ bash /var/xdrago/second.sh
root 39929 0.0 0.0 17532 2608 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 34353 0.0 0.0 45780 1340 ? S 11:12 0:00 \_ /USR/SBIN/CRON
root 34374 0.0 0.0 4136 600 ? Ss 11:12 0:00 | \_ /bin/dash -c bash /var/xdrago/runner.sh >/dev/null 2>&1
root 34415 0.0 0.0 10740 1600 ? S 11:12 0:00 | \_ bash /var/xdrago/runner.sh
root 34517 0.0 0.0 10720 1580 ? S 11:12 0:00 | \_ bash /var/xdrago/run-tn
root 34536 0.0 0.0 48672 1756 ? S 11:12 0:00 | \_ su -s /bin/bash - tn -c bash /data/disk/tn/aegir.sh
tn 34585 0.5 0.0 10672 1424 ? Ss 11:12 0:01 | \_ bash /data/disk/tn/aegir.sh
tn 34739 18.1 0.3 224012 30636 ? S 11:12 0:45 | \_ /opt/php53/bin/php /data/disk/tn/tools/drush/drush.php @hostmaster hosting-dispatch
root 34354 0.0 0.0 45780 1340 ? S 11:12 0:00 \_ /USR/SBIN/CRON
root 34362 0.0 0.0 4136 604 ? Ss 11:12 0:00 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 34387 0.6 0.0 10752 1660 ? S 11:12 0:01 | \_ bash /var/xdrago/minute.sh
root 39860 0.0 0.0 10752 660 ? S 11:16 0:00 | \_ bash /var/xdrago/minute.sh
root 39869 31.6 0.0 11464 804 ? R 11:16 0:01 | \_ awk {print $2}
root 34355 0.0 0.0 45780 1340 ? S 11:12 0:00 \_ /USR/SBIN/CRON
root 34381 0.0 0.0 4136 600 ? Ss 11:12 0:00 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 34396 0.3 0.0 10848 1712 ? S 11:12 0:00 | \_ bash /var/xdrago/second.sh
root 39962 0.0 0.0 17532 2616 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 35794 0.0 0.0 45780 1340 ? S 11:13 0:00 \_ /USR/SBIN/CRON
root 35803 0.0 0.0 4136 604 ? Ss 11:13 0:00 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 35892 0.3 0.0 10752 1668 ? S 11:13 0:00 | \_ bash /var/xdrago/minute.sh
root 39965 0.0 0.0 4368 708 ? S 11:16 0:00 | \_ sleep 5
root 35795 0.0 0.0 45780 1340 ? S 11:13 0:00 \_ /USR/SBIN/CRON
root 35797 0.0 0.0 4136 604 ? Ss 11:13 0:00 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 35843 0.5 0.0 10848 1708 ? S 11:13 0:01 | \_ bash /var/xdrago/second.sh
root 39926 0.0 0.0 17532 2612 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 37505 0.0 0.0 45780 1340 ? S 11:14 0:00 \_ /USR/SBIN/CRON
root 37515 0.0 0.0 4136 604 ? Ss 11:14 0:00 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 37541 0.5 0.0 10748 1664 ? S 11:14 0:00 | \_ bash /var/xdrago/minute.sh
root 39919 0.0 0.0 4368 704 ? S 11:16 0:00 | \_ sleep 5
root 37506 0.0 0.0 45780 1340 ? S 11:14 0:00 \_ /USR/SBIN/CRON
root 37514 0.0 0.0 4136 604 ? Ss 11:14 0:00 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 37565 0.9 0.0 10848 1708 ? S 11:14 0:01 | \_ bash /var/xdrago/second.sh
root 40004 0.0 0.0 17532 2460 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 40057 0.0 0.0 15980 1732 ? R 11:16 0:00 | \_ ps auxf
root 39369 0.0 0.0 45780 1340 ? S 11:15 0:00 \_ /USR/SBIN/CRON
root 39382 0.8 0.0 4136 600 ? Ss 11:15 0:00 | \_ /bin/dash -c bash /var/xdrago/guest-fire.sh >/dev/null 2>&1
root 39552 1.8 0.0 10736 1520 ? S 11:16 0:00 | \_ bash /var/xdrago/guest-fire.sh
root 39891 0.0 0.0 4368 704 ? S 11:16 0:00 | \_ sleep 5
root 39373 8.5 0.0 45780 1340 ? S 11:15 0:06 \_ /USR/SBIN/CRON
root 39430 18.7 0.0 4136 604 ? Ss 11:15 0:07 | \_ /bin/dash -c test -x /usr/sbin/metche && /usr/sbin/metche cron
root 39555 0.5 0.0 10928 1780 ? S 11:16 0:00 | \_ /bin/bash /usr/sbin/metche cron
root 39374 0.0 0.0 45780 1340 ? S 11:15 0:00 \_ /USR/SBIN/CRON
root 39380 10.9 0.0 4136 600 ? Ss 11:15 0:08 | \_ /bin/dash -c bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1
root 39559 0.8 0.0 11324 2256 ? S 11:16 0:00 | \_ bash /var/xdrago/manage_ltd_users.sh
root 39376 12.1 0.0 45780 1340 ? S 11:15 0:09 \_ /USR/SBIN/CRON
root 39446 21.0 0.0 4136 604 ? Ss 11:15 0:06 | \_ /bin/dash -c bash /var/xdrago/runner.sh >/dev/null 2>&1
root 39624 0.3 0.0 10740 1596 ? S 11:16 0:00 | \_ bash /var/xdrago/runner.sh
root 40071 0.0 0.0 4368 708 ? S 11:16 0:00 | \_ sleep 5
root 39377 11.1 0.0 45780 1340 ? S 11:15 0:08 \_ /USR/SBIN/CRON
root 39422 12.3 0.0 4136 600 ? Ss 11:15 0:05 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 39575 0.3 0.0 10748 1656 ? S 11:16 0:00 | \_ bash /var/xdrago/minute.sh
root 39378 9.3 0.0 45780 1340 ? S 11:15 0:07 \_ /USR/SBIN/CRON
root 39443 20.5 0.0 4136 600 ? Ss 11:15 0:07 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 39606 0.3 0.0 10848 1712 ? S 11:16 0:00 | \_ bash /var/xdrago/second.sh
root 39723 0.3 0.0 17532 2612 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 39379 8.8 0.0 45780 1340 ? S 11:15 0:07 \_ /USR/SBIN/CRON
aegir 39433 17.4 0.0 4136 604 ? Ss 11:15 0:06 | \_ /bin/dash -c /usr/bin/env php /var/aegir/drush/drush.php '@hostmaster' hosting-dispatch >/dev/null 2>&1
aegir 39724 15.8 0.2 212752 19112 ? S 11:16 0:00 | \_ php /var/aegir/drush/drush.php @hostmaster hosting-dispatch
root 39468 3.9 0.0 45780 1340 ? S 11:16 0:00 \_ /USR/SBIN/CRON
root 39504 0.3 0.0 4136 608 ? Ss 11:16 0:00 | \_ /bin/dash -c bash /var/xdrago/guest-fire.sh >/dev/null 2>&1
root 39682 1.8 0.0 10736 1520 ? S 11:16 0:00 | \_ bash /var/xdrago/guest-fire.sh
root 39935 0.0 0.0 4368 704 ? S 11:16 0:00 | \_ sleep 5
root 39470 5.0 0.0 45780 1340 ? S 11:16 0:00 \_ /USR/SBIN/CRON
root 39506 0.1 0.0 4136 604 ? Ss 11:16 0:00 | \_ /bin/dash -c bash /var/xdrago/minute.sh >/dev/null 2>&1
root 39620 0.1 0.0 10748 1652 ? S 11:16 0:00 | \_ bash /var/xdrago/minute.sh
root 39471 6.2 0.0 45780 1340 ? S 11:16 0:00 \_ /USR/SBIN/CRON
root 39522 0.3 0.0 4136 604 ? Ss 11:16 0:00 | \_ /bin/dash -c bash /var/xdrago/second.sh >/dev/null 2>&1
root 39726 0.3 0.0 10848 1712 ? S 11:16 0:00 | \_ bash /var/xdrago/second.sh
root 39813 0.6 0.0 17532 2612 ? S 11:16 0:00 | \_ perl /var/xdrago/proc_num_ctrl.cgi
root 39472 4.0 0.0 45780 1340 ? S 11:16 0:00 \_ /USR/SBIN/CRON
aegir 39505 0.1 0.0 4136 604 ? Ss 11:16 0:00 \_ /bin/dash -c /usr/bin/env php /var/aegir/drush/drush.php '@hostmaster' hosting-dispatch >/dev/null 2>&1
aegir 39637 3.3 0.2 212752 19112 ? S 11:16 0:00 \_ php /var/aegir/drush/drush.php @hostmaster hosting-dispatch
root 27313 0.3 0.2 58656 19412 ? Ss 2014 266:17 lfd - sleeping
root 39407 2.4 0.2 58656 18584 ? S 11:15 0:01 \_ lfd - checking system integrity
root 39414 27.0 0.0 4384 772 ? R 11:15 0:12 | \_ /usr/bin/md5sum --check /var/lib/csf/csf.tempint
root 40056 0.0 0.2 58656 18684 ? S 11:16 0:00 \_ lfd - (child) checking load...
root 40081 0.0 0.0 15976 1692 ? R 11:16 0:00 \_ /bin/ps axuf
root 61042 0.0 0.0 10864 1480 ? S 2014 0:00 /bin/bash /usr/bin/mysqld_safe
mysql 61901 3.1 24.9 4584016 2090196 ? Sl 2014 1760:48 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --open-files-limit=196608 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysq ld/mysqld.sock --port=3306
root 61902 0.0 0.0 5548 608 ? S 2014 0:00 \_ logger -t mysqld -p daemon.error
root 45229 0.0 0.1 78436 15124 ? Ss Jan29 0:32 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
www-data 35678 0.6 0.1 78436 14384 ? S 11:12 0:01 \_ nginx: worker process
www-data 35679 2.6 0.1 78436 15108 ? S 11:12 0:05 \_ nginx: worker process
www-data 35680 6.5 0.1 78436 15160 ? S 11:12 0:13 \_ nginx: worker process
www-data 35681 0.8 0.1 78436 15176 ? S 11:12 0:01 \_ nginx: worker process
www-data 35687 1.6 0.1 78436 15216 ? S 11:12 0:03 \_ nginx: worker process
www-data 35688 0.3 0.1 78436 15112 ? S 11:12 0:00 \_ nginx: worker process
www-data 35692 0.4 0.1 78436 14348 ? S 11:12 0:00 \_ nginx: worker process
www-data 35693 2.1 0.1 78436 15360 ? S 11:12 0:04 \_ nginx: worker process
www-data 35694 1.3 0.1 78436 15316 ? S 11:12 0:02 \_ nginx: worker process
www-data 35695 0.5 0.1 78436 14116 ? S 11:12 0:01 \_ nginx: worker process
www-data 35696 1.7 0.1 78436 15284 ? S 11:12 0:03 \_ nginx: worker process
www-data 35702 0.5 0.1 78436 14368 ? S 11:12 0:01 \_ nginx: worker process
www-data 35703 1.1 0.1 78436 15228 ? S 11:12 0:02 \_ nginx: worker process
www-data 35704 0.4 0.1 78436 15176 ? S 11:12 0:00 \_ nginx: worker process
www-data 35705 1.6 0.1 78436 15284 ? S 11:12 0:03 \_ nginx: worker process
www-data 35706 0.0 0.1 78436 13760 ? S 11:12 0:00 \_ nginx: worker process
www-data 35707 0.7 0.1 78436 15172 ? S 11:12 0:01 \_ nginx: worker process
www-data 35708 1.8 0.1 78436 15284 ? S 11:12 0:03 \_ nginx: worker process
www-data 35709 0.3 0.1 78436 14304 ? S 11:12 0:00 \_ nginx: worker process
www-data 35710 0.3 0.1 78436 14300 ? S 11:12 0:00 \_ nginx: worker process
www-data 35716 0.2 0.1 78436 14328 ? S 11:12 0:00 \_ nginx: worker process
www-data 35717 0.6 0.1 78436 14288 ? S 11:12 0:01 \_ nginx: worker process
www-data 35718 0.4 0.1 78436 15264 ? S 11:12 0:00 \_ nginx: worker process
www-data 35719 1.5 0.1 78436 14336 ? S 11:12 0:03 \_ nginx: worker process
www-data 35732 7.3 0.1 78436 15336 ? S 11:12 0:15 \_ nginx: worker process
www-data 35738 0.8 0.1 78436 15268 ? S 11:12 0:01 \_ nginx: worker process
www-data 35750 0.6 0.1 78436 15176 ? S 11:12 0:01 \_ nginx: worker process
www-data 35755 0.6 0.1 78436 14324 ? S 11:12 0:01 \_ nginx: worker process
www-data 35756 2.3 0.1 78436 13816 ? S 11:12 0:04 \_ nginx: cache manager process
root 57223 0.0 0.0 37832 2452 ? Ss Jan29 0:13 /usr/lib/postfix/master
postfix 57610 0.0 0.0 39900 2428 ? S 10:03 0:00 \_ pickup -l -t fifo -u -c
postfix 57612 0.0 0.0 40064 2568 ? S 10:03 0:00 \_ qmgr -l -t fifo -u
postfix 62516 0.0 0.0 42460 3464 ? S 10:07 0:00 \_ tlsmgr -l -t unix -u -c
root 933 0.0 0.0 1258964 7556 ? Ss 00:02 0:37 php-fpm: master process (/opt/php53/etc/php53-fpm.conf)
tn.ftp 36494 15.0 0.4 1268564 39200 ? R 11:13 0:27 \_ php-fpm: pool tn
tn.ftp 36495 15.1 0.2 1262948 17332 ? R 11:13 0:27 \_ php-fpm: pool tn
tn.ftp 36496 15.6 0.2 1262944 17288 ? R 11:13 0:28 \_ php-fpm: pool tn
tn.ftp 36503 15.6 0.2 1262944 17344 ? R 11:13 0:28 \_ php-fpm: pool tn
tn.ftp 36504 15.8 0.2 1262948 17276 ? R 11:13 0:28 \_ php-fpm: pool tn
tn.ftp 36516 15.0 0.2 1262948 17348 ? R 11:13 0:27 \_ php-fpm: pool tn
tn.ftp 36517 17.8 0.2 1262948 17396 ? R 11:13 0:32 \_ php-fpm: pool tn
tn.ftp 36519 14.6 0.2 1262948 17332 ? R 11:13 0:26 \_ php-fpm: pool tn
tn.ftp 37031 14.0 0.2 1262948 17236 ? R 11:13 0:22 \_ php-fpm: pool tn
tn.ftp 37035 14.6 0.4 1265172 34768 ? R 11:13 0:23 \_ php-fpm: pool tn
tn.ftp 37036 16.8 0.2 1262948 17412 ? R 11:13 0:27 \_ php-fpm: pool tn
tn.ftp 37053 15.2 0.2 1262944 17296 ? R 11:13 0:24 \_ php-fpm: pool tn
tn.ftp 37054 15.2 0.2 1262948 17280 ? R 11:13 0:24 \_ php-fpm: pool tn
tn.ftp 37056 13.4 0.2 1262932 17328 ? R 11:13 0:21 \_ php-fpm: pool tn
tn.ftp 37450 12.1 0.2 1262944 17344 ? R 11:13 0:17 \_ php-fpm: pool tn
tn.ftp 37459 13.2 0.2 1262944 17344 ? R 11:13 0:18 \_ php-fpm: pool tn
tn.ftp 37461 14.2 0.2 1262944 17308 ? R 11:13 0:20 \_ php-fpm: pool tn
tn.ftp 37468 15.4 0.2 1262944 17380 ? R 11:13 0:21 \_ php-fpm: pool tn
tn.ftp 37469 13.0 0.2 1263212 21980 ? R 11:13 0:18 \_ php-fpm: pool tn
tn.ftp 37471 17.7 0.2 1262948 17428 ? R 11:13 0:25 \_ php-fpm: pool tn
tn.ftp 37472 15.9 0.2 1262904 17384 ? R 11:13 0:22 \_ php-fpm: pool tn
tn.ftp 37473 13.1 0.2 1262944 17288 ? R 11:13 0:18 \_ php-fpm: pool tn
tn.ftp 37474 15.4 0.2 1262944 17384 ? R 11:13 0:21 \_ php-fpm: pool tn
tn.ftp 37475 12.2 0.2 1262904 17428 ? R 11:13 0:17 \_ php-fpm: pool tn
tn.ftp 38339 10.5 0.2 1262948 17292 ? R 11:14 0:12 \_ php-fpm: pool tn
tn.ftp 38351 17.4 0.2 1262904 17276 ? R 11:14 0:21 \_ php-fpm: pool tn
tn.ftp 38354 12.0 0.2 1262948 17376 ? R 11:14 0:14 \_ php-fpm: pool tn
tn.ftp 38792 9.3 0.2 1262948 17352 ? R 11:14 0:09 \_ php-fpm: pool tn
tn.ftp 38808 9.4 0.2 1262944 17272 ? R 11:14 0:09 \_ php-fpm: pool tn
tn.ftp 38820 8.6 0.2 1262908 17304 ? R 11:14 0:08 \_ php-fpm: pool tn
tn.ftp 38821 17.0 0.2 1262900 17368 ? R 11:14 0:17 \_ php-fpm: pool tn
tn.ftp 38822 14.5 0.2 1262948 17284 ? R 11:14 0:14 \_ php-fpm: pool tn
tn.ftp 38825 8.9 0.2 1262904 17280 ? R 11:14 0:09 \_ php-fpm: pool tn
tn.ftp 39335 5.2 0.2 1262948 17340 ? R 11:14 0:04 \_ php-fpm: pool tn
tn.ftp 39345 5.4 0.2 1262040 16848 ? R 11:14 0:04 \_ php-fpm: pool tn
tn.ftp 39346 16.0 0.2 1262944 17348 ? R 11:14 0:13 \_ php-fpm: pool tn
tn.ftp 39350 17.1 0.2 1263020 17948 ? R 11:14 0:14 \_ php-fpm: pool tn
tn.ftp 39351 17.4 0.2 1262904 17364 ? R 11:14 0:14 \_ php-fpm: pool tn
tn.ftp 39353 18.4 0.2 1262948 17304 ? R 11:14 0:15 \_ php-fpm: pool tn
tn.ftp 39354 24.2 0.2 1262944 17256 ? R 11:15 0:19 \_ php-fpm: pool tn
tn.ftp 39384 5.2 0.2 1262944 17308 ? R 11:15 0:04 \_ php-fpm: pool tn
tn.ftp 39415 24.0 0.2 1262948 17268 ? R 11:15 0:11 \_ php-fpm: pool tn
tn.ftp 39445 22.5 0.2 1262948 17296 ? R 11:15 0:08 \_ php-fpm: pool tn
tn.ftp 39450 23.4 0.2 1262948 17392 ? R 11:15 0:05 \_ php-fpm: pool tn
tn.ftp 39467 20.0 0.2 1262948 17352 ? R 11:16 0:02 \_ php-fpm: pool tn
tn.ftp 39564 22.0 0.2 1262948 17372 ? R 11:16 0:01 \_ php-fpm: pool tn
tn.ftp 39571 23.5 0.1 1259728 11944 ? R 11:16 0:01 \_ php-fpm: pool tn
tn.ftp 39579 21.6 0.2 1263068 22964 ? R 11:16 0:01 \_ php-fpm: pool tn
redis 20242 1.2 1.5 224816 131328 ? Ssl 03:32 5:41 /usr/bin/redis-server 127.0.0.1:6379
pdnsd 57471 0.0 0.0 145884 1000 ? Sl 10:03 0:00 /usr/sbin/pdnsd --daemon -p /var/run/pdnsd.pid
root 22338 0.0 0.0 49888 1208 ? Ss 11:03 0:00 /usr/sbin/sshd
tn 33613 22.7 0.5 240676 48052 ? R 11:11 1:05 php /data/disk/tn/tools/drush/drush.php --strict= --uri=tn.puffin.webarch.net --root=/data/disk/tn/aegir/distro/016 hosting-tasks --items=1 --quiet
tn 33623 23.4 0.5 240672 48024 ? R 11:11 1:07 php /data/disk/tn/tools/drush/drush.php --strict= --uri=tn.puffin.webarch.net --root=/data/disk/tn/aegir/distro/016 hosting-tasks --items=1 --quiet
tn 33742 24.3 0.3 224004 30596 ? S 11:11 1:09 php /data/disk/tn/tools/drush/drush.php --strict= --uri=tn.puffin.webarch.net --root=/data/disk/tn/aegir/distro/016 hosting-advanced_cron --items=1 --quiet
tn 33792 23.6 0.4 232572 39492 ? R 11:11 1:07 php /data/disk/tn/tools/drush/drush.php --strict= --uri=tn.puffin.webarch.net --root=/data/disk/tn/aegir/distro/016 hosting-advanced_cron --items=1 --quiet
Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
65 0 0 664548 470128 3648768 0 0 1 26 0 0 2 3 95 0
[-- Autoview using lynx -dump '/var/tmp/apachestatus.html' --]
Active connections: 55 server accepts handled requests 108104 108104 145980
Reading: 0 Writing: 63 Waiting: 10
Note: See
TracTickets for help on using
tickets.
These are the ldf alerts I received, starting at Thu, 29 Jan 2015 16:03:37 +0000 (GMT) and ending at Thu, 29 Jan 2015 17:12:36 +0000 (GMT):
Can't see anything is the syslog, messages, daemon.log and there is simply a gap in the Munin stats, https://penguin.transitionnetwork.org/munin/transitionnetwork.org/puffin.transitionnetwork.org/index.html
I also had a look at the Nginx access.log following the notes here, wiki:WebServerLogs#goaccess but couldn't see anything of note.
There are these in the error.log:
I don't know what caused this load spike I'm afraid.