Transition Technology: Ticket #674: Puffin locked up

chris — Tue, 14 Jan 2014 10:01:36 GMT

It's recovered.

chris — Tue, 14 Jan 2014 10:01:56 GMT

But the reboot is happening anyway...

chris — Tue, 14 Jan 2014 10:06:13 GMT

Booting puffin takes ages due to the massive number of firewall rules which need to be loaded.

It's back up now.

hours, totalhours changed

chris — Tue, 14 Jan 2014 10:15:58 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.0 to 0.25

This looks like the cause, from /var/log/messages:

Jan 14 09:44:25 puffin kernel: [4478428.647813] vnstatd       D ffff8800de4ebf90     0  1920      1 0x00000000
Jan 14 09:44:25 puffin kernel: [4478428.647825]  ffff8800de4ebf90 0000000000000282 ffff8801ffc15000 ffff880000009680
Jan 14 09:44:25 puffin kernel: [4478428.647838]  ffff8801fa651d58 000000000000000a 000000000000f9e0 ffff8801fa651fd8
Jan 14 09:44:25 puffin kernel: [4478428.647849]  00000000000157c0 00000000000157c0 ffff8801fe335bd0 ffff8801fe335ec8
Jan 14 09:44:25 puffin kernel: [4478428.647861] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.647877]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:25 puffin kernel: [4478428.647888]  [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
Jan 14 09:44:25 puffin kernel: [4478428.647897]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:25 puffin kernel: [4478428.647907]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:25 puffin kernel: [4478428.647915]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.647924]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:25 puffin kernel: [4478428.647932]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:25 puffin kernel: [4478428.647940]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:25 puffin kernel: [4478428.647947]  [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
Jan 14 09:44:25 puffin kernel: [4478428.647957]  [<ffffffff8102de30>] ? pvclock_clocksource_read+0x3a/0x8b
Jan 14 09:44:25 puffin kernel: [4478428.647967]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:25 puffin kernel: [4478428.647976]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:25 puffin kernel: [4478428.647985]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:25 puffin kernel: [4478428.648069] awk           D ffff8801eb1f8e20     0 45972  45967 0x00000080
Jan 14 09:44:25 puffin kernel: [4478428.648079]  ffff8801eb1f8e20 0000000000000286 0000000000000000 0000000000000000
Jan 14 09:44:25 puffin kernel: [4478428.648090]  0000000000000000 0000000000000000 000000000000f9e0 ffff8800bd109fd8
Jan 14 09:44:25 puffin kernel: [4478428.648102]  00000000000157c0 00000000000157c0 ffff8801fa845bd0 ffff8801fa845ec8
Jan 14 09:44:25 puffin kernel: [4478428.648113] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.648121]  [<ffffffff811976a6>] ? vsnprintf+0x40a/0x449
Jan 14 09:44:25 puffin kernel: [4478428.648130]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:25 puffin kernel: [4478428.648138]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:25 puffin kernel: [4478428.648146]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:25 puffin kernel: [4478428.648154]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648162]  [<ffffffff810f826a>] ? do_follow_link+0x1fa/0x317
Jan 14 09:44:25 puffin kernel: [4478428.648171]  [<ffffffff810f86e0>] ? __link_path_walk+0x359/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648179]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:25 puffin kernel: [4478428.648187]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:25 puffin kernel: [4478428.648195]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:25 puffin kernel: [4478428.648202]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
Jan 14 09:44:25 puffin kernel: [4478428.648211]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
Jan 14 09:44:25 puffin kernel: [4478428.648219]  [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
Jan 14 09:44:25 puffin kernel: [4478428.648229]  [<ffffffff8105cd8e>] ? do_sigaction+0x159/0x171
Jan 14 09:44:25 puffin kernel: [4478428.648236]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:25 puffin kernel: [4478428.648244]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:25 puffin kernel: [4478428.648252]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:25 puffin kernel: [4478428.648275] vmstat        D ffff8801c73c7810     0 45976      1 0x00000080
Jan 14 09:44:25 puffin kernel: [4478428.648284]  ffff8801c73c7810 0000000000000286 ffff8800d95e5e10 ffff880000009680
Jan 14 09:44:25 puffin kernel: [4478428.648303]  00000000000006fe 000000000000000a 000000000000f9e0 ffff8800d95e5fd8
Jan 14 09:44:25 puffin kernel: [4478428.648314]  00000000000157c0 00000000000157c0 ffff8801fc68b880 ffff8801fc68bb78
Jan 14 09:44:25 puffin kernel: [4478428.648326] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.648333]  [<ffffffff810414a4>] ? set_next_entity+0x34/0x56
Jan 14 09:44:25 puffin kernel: [4478428.648342]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:25 puffin kernel: [4478428.648350]  [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
Jan 14 09:44:25 puffin kernel: [4478428.648359]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:25 puffin kernel: [4478428.648366]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:25 puffin kernel: [4478428.648373]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648382]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:25 puffin kernel: [4478428.648390]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:25 puffin kernel: [4478428.648404]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:25 puffin kernel: [4478428.648404]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:25 puffin kernel: [4478428.648404]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:25 puffin kernel: [4478428.648404]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:25 puffin kernel: [4478428.648417] id            D ffff8801eb1f8e20     0 45979  45975 0x00000080
Jan 14 09:44:25 puffin kernel: [4478428.648427]  ffff8801eb1f8e20 0000000000000286 ffff8801ff0c7d80 0000000000000001
Jan 14 09:44:25 puffin kernel: [4478428.648439]  0000000000000000 0000000000000098 000000000000f9e0 ffff880029119fd8
Jan 14 09:44:25 puffin kernel: [4478428.648449]  00000000000157c0 00000000000157c0 ffff8800de4eb880 ffff8800de4ebb78
Jan 14 09:44:25 puffin kernel: [4478428.648461] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.648469]  [<ffffffff810b9b8b>] ? zone_watermark_ok+0x20/0xb1
Jan 14 09:44:25 puffin kernel: [4478428.648477]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:25 puffin kernel: [4478428.648486]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:25 puffin kernel: [4478428.648493]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:25 puffin kernel: [4478428.648501]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648509]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:25 puffin kernel: [4478428.648516]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:25 puffin kernel: [4478428.648524]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:25 puffin kernel: [4478428.648534]  [<ffffffff810d217e>] ? vma_link+0x74/0x9a
Jan 14 09:44:25 puffin kernel: [4478428.648541]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:25 puffin kernel: [4478428.648549]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:25 puffin kernel: [4478428.648556]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:25 puffin kernel: [4478428.648580] sshd          D ffff8801fa8446a0     0 45990   6977 0x00000080
Jan 14 09:44:25 puffin kernel: [4478428.648590]  ffff8801fa8446a0 0000000000000282 000000000100007f 0000000000000000
Jan 14 09:44:25 puffin kernel: [4478428.648601]  0100007f0002bcee 0000000000000000 000000000000f9e0 ffff8800dd239fd8
Jan 14 09:44:25 puffin kernel: [4478428.648613]  00000000000157c0 00000000000157c0 ffff8801fc68d4c0 ffff8801fc68d7b8
Jan 14 09:44:25 puffin kernel: [4478428.648623] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.648629]  [<ffffffff811976a6>] ? vsnprintf+0x40a/0x449
Jan 14 09:44:25 puffin kernel: [4478428.648650]  [<ffffffffa01393c6>] ? death_by_timeout+0x0/0x4a [nf_conntrack]
Jan 14 09:44:25 puffin kernel: [4478428.648660]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:25 puffin kernel: [4478428.648669]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:25 puffin kernel: [4478428.648677]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:25 puffin kernel: [4478428.648684]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648692]  [<ffffffff810f826a>] ? do_follow_link+0x1fa/0x317
Jan 14 09:44:25 puffin kernel: [4478428.648701]  [<ffffffff810f86e0>] ? __link_path_walk+0x359/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648708]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
Jan 14 09:44:25 puffin kernel: [4478428.648716]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:25 puffin kernel: [4478428.648724]  [<ffffffff810fa318>] ? do_filp_open+0x15d/0x94b
Jan 14 09:44:25 puffin kernel: [4478428.648732]  [<ffffffff8130f4c5>] ? page_fault+0x25/0x30
Jan 14 09:44:25 puffin kernel: [4478428.648739]  [<ffffffff8130f6fa>] ? error_exit+0x2a/0x60
Jan 14 09:44:25 puffin kernel: [4478428.648747]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
Jan 14 09:44:25 puffin kernel: [4478428.648756]  [<ffffffff8119845c>] ? strncpy_from_user+0x51/0x6d
Jan 14 09:44:25 puffin kernel: [4478428.648765]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:25 puffin kernel: [4478428.648772]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:25 puffin kernel: [4478428.648779]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:25 puffin kernel: [4478428.648801] ps            D ffff8801fa844db0     0 45994  45948 0x00000080
Jan 14 09:44:25 puffin kernel: [4478428.648811]  ffff8801fa844db0 0000000000000282 ffff8801ff0c7480 ffffffff810bb263
Jan 14 09:44:25 puffin kernel: [4478428.648822]  ffff88000000a6d0 ffff880014ffa000 000000000000f9e0 ffff880014ffbfd8
Jan 14 09:44:25 puffin kernel: [4478428.648833]  00000000000157c0 00000000000157c0 ffff8801faab8710 ffff8801faab8a08
Jan 14 09:44:25 puffin kernel: [4478428.648844] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.648850]  [<ffffffff810bb263>] ? get_page_from_freelist+0x56b/0x760
Jan 14 09:44:25 puffin kernel: [4478428.648858]  [<ffffffff810b9b8b>] ? zone_watermark_ok+0x20/0xb1
Jan 14 09:44:25 puffin kernel: [4478428.648867]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:25 puffin kernel: [4478428.648875]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:25 puffin kernel: [4478428.648882]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:25 puffin kernel: [4478428.648890]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:25 puffin kernel: [4478428.648898]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:25 puffin kernel: [4478428.648905]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:25 puffin kernel: [4478428.648913]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:25 puffin kernel: [4478428.648920]  [<ffffffff8130f4c5>] ? page_fault+0x25/0x30
Jan 14 09:44:25 puffin kernel: [4478428.648927]  [<ffffffff8130f6fa>] ? error_exit+0x2a/0x60
Jan 14 09:44:25 puffin kernel: [4478428.648934]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
Jan 14 09:44:25 puffin kernel: [4478428.648941]  [<ffffffff8119845c>] ? strncpy_from_user+0x51/0x6d
Jan 14 09:44:25 puffin kernel: [4478428.648949]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:25 puffin kernel: [4478428.648956]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:25 puffin kernel: [4478428.648964]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:25 puffin kernel: [4478428.648987] awk           D ffff8801eb1fe2e0     0 46000  45997 0x00000080
Jan 14 09:44:25 puffin kernel: [4478428.648996]  ffff8801eb1fe2e0 0000000000000286 00000002000280da ffff88000000da08
Jan 14 09:44:25 puffin kernel: [4478428.649008]  0000000000000000 ffffffff81012cdb 000000000000f9e0 ffff8800164d9fd8
Jan 14 09:44:25 puffin kernel: [4478428.649019]  00000000000157c0 00000000000157c0 ffff8801c73c3170 ffff8801c73c3468
Jan 14 09:44:25 puffin kernel: [4478428.649031] Call Trace:
Jan 14 09:44:25 puffin kernel: [4478428.649037]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
Jan 14 09:44:26 puffin kernel: [4478428.894222]  [<ffffffff81155196>] ? cap_inode_permission+0x0/0x3
Jan 14 09:44:26 puffin kernel: [4478428.894237]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:26 puffin kernel: [4478428.894248]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:26 puffin kernel: [4478428.894258]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:26 puffin kernel: [4478428.894265]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:26 puffin kernel: [4478428.894275]  [<ffffffff810f826a>] ? do_follow_link+0x1fa/0x317
Jan 14 09:44:26 puffin kernel: [4478428.894284]  [<ffffffff810f86e0>] ? __link_path_walk+0x359/0x6f5
Jan 14 09:44:26 puffin kernel: [4478428.894292]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:26 puffin kernel: [4478428.894300]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:26 puffin kernel: [4478428.894309]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:26 puffin kernel: [4478428.894319]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
Jan 14 09:44:26 puffin kernel: [4478428.894327]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
Jan 14 09:44:26 puffin kernel: [4478428.894337]  [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
Jan 14 09:44:26 puffin kernel: [4478428.894347]  [<ffffffff8105cd8e>] ? do_sigaction+0x159/0x171
Jan 14 09:44:26 puffin kernel: [4478428.894356]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:26 puffin kernel: [4478428.894365]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:26 puffin kernel: [4478428.894374]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:26 puffin kernel: [4478428.894399] perl          D ffff8800de4ee2e0     0 46003  45996 0x00000080
Jan 14 09:44:26 puffin kernel: [4478428.894410]  ffff8800de4ee2e0 0000000000000286 0000000000000041 0000000000000000
Jan 14 09:44:26 puffin kernel: [4478428.894422]  00000000000006fe ffffffff81012cdb 000000000000f9e0 ffff880016527fd8
Jan 14 09:44:26 puffin kernel: [4478428.894434]  00000000000157c0 00000000000157c0 ffff8800161d2350 ffff8800161d2648
Jan 14 09:44:26 puffin kernel: [4478428.894445] Call Trace:
Jan 14 09:44:26 puffin kernel: [4478428.894454]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
Jan 14 09:44:26 puffin kernel: [4478428.894464]  [<ffffffff81155196>] ? cap_inode_permission+0x0/0x3
Jan 14 09:44:26 puffin kernel: [4478428.894472]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:26 puffin kernel: [4478428.894481]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:26 puffin kernel: [4478428.894488]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:26 puffin kernel: [4478428.894496]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:26 puffin kernel: [4478428.894504]  [<ffffffff810f826a>] ? do_follow_link+0x1fa/0x317
Jan 14 09:44:26 puffin kernel: [4478428.894512]  [<ffffffff810f86e0>] ? __link_path_walk+0x359/0x6f5
Jan 14 09:44:26 puffin kernel: [4478428.894521]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:26 puffin kernel: [4478428.894528]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:26 puffin kernel: [4478428.894536]  [<ffffffff810fb5f6>] ? user_path_at+0x48/0x79
Jan 14 09:44:26 puffin kernel: [4478428.894546]  [<ffffffff81099076>] ? check_for_new_grace_period+0x98/0xa5
Jan 14 09:44:26 puffin kernel: [4478428.894556]  [<ffffffff81311626>] ? do_page_fault+0x2e0/0x2fc
Jan 14 09:44:26 puffin kernel: [4478428.894563]  [<ffffffff810f3634>] ? sys_readlinkat+0x25/0x8d
Jan 14 09:44:26 puffin kernel: [4478428.894571]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:44:26 puffin kernel: [4478428.894595] cron          D ffff8801fb270e20     0 46009   8829 0x00000080
Jan 14 09:44:26 puffin kernel: [4478428.894606]  ffff8801fb270e20 0000000000000286 0000000000000001 00000001810423b0
Jan 14 09:44:26 puffin kernel: [4478428.894617]  ffff88000bcf87c0 000000000000000b 000000000000f9e0 ffff8800cbb9dfd8
Jan 14 09:44:26 puffin kernel: [4478428.894628]  00000000000157c0 00000000000157c0 ffff8801fa458000 ffff8801fa4582f8
Jan 14 09:44:26 puffin kernel: [4478428.894640] Call Trace:
Jan 14 09:44:26 puffin kernel: [4478428.894648]  [<ffffffff811976a6>] ? vsnprintf+0x40a/0x449
Jan 14 09:44:26 puffin kernel: [4478428.894656]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:44:26 puffin kernel: [4478428.894666]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:44:26 puffin kernel: [4478428.894673]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:44:26 puffin kernel: [4478428.894681]  [<ffffffff810f892c>] ? __link_path_walk+0x5a5/0x6f5
Jan 14 09:44:26 puffin kernel: [4478428.894689]  [<ffffffff810f826a>] ? do_follow_link+0x1fa/0x317
Jan 14 09:44:26 puffin kernel: [4478428.894697]  [<ffffffff810f86e0>] ? __link_path_walk+0x359/0x6f5
Jan 14 09:44:26 puffin kernel: [4478428.894705]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:44:26 puffin kernel: [4478428.894713]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:44:26 puffin kernel: [4478428.894721]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:44:26 puffin kernel: [4478428.894729]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
Jan 14 09:44:26 puffin kernel: [4478428.894738]  [<ffffffff8130dda8>] ? thread_return+0x79/0xe0
Jan 14 09:44:26 puffin kernel: [4478428.894746]  [<ffffffff811901fb>] ? _atomic_dec_and_lock+0x33/0x50
Jan 14 09:44:26 puffin kernel: [4478428.894755]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:44:26 puffin kernel: [4478428.894763]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:44:26 puffin kernel: [4478428.894770]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
Jan 14 09:48:22 puffin pure-ftpd: (?@192.95.55.77) [INFO] New connection from 192.95.55.77
Jan 14 09:48:26 puffin kernel: [4478669.197458] vnstatd       D 0000000000000000     0  1920      1 0x00000000
Jan 14 09:48:26 puffin kernel: [4478669.197469]  ffff8801ff18bf90 0000000000000282 ffff8801fa651e10 ffff880000009680
Jan 14 09:48:26 puffin kernel: [4478669.197482]  0000000000000000 000000000000000a 000000000000f9e0 ffff8801fa651fd8
Jan 14 09:48:26 puffin kernel: [4478669.197494]  00000000000157c0 00000000000157c0 ffff8801fe335bd0 ffff8801fe335ec8
Jan 14 09:48:26 puffin kernel: [4478669.197505] Call Trace:
Jan 14 09:48:26 puffin kernel: [4478669.197520]  [<ffffffff8130f6fa>] ? error_exit+0x2a/0x60
Jan 14 09:48:26 puffin kernel: [4478669.197531]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
Jan 14 09:48:26 puffin kernel: [4478669.197542]  [<ffffffff8130e5c3>] ? __mutex_lock_common+0x122/0x192
Jan 14 09:48:26 puffin kernel: [4478669.197551]  [<ffffffff8130e6eb>] ? mutex_lock+0x1a/0x31
Jan 14 09:48:26 puffin kernel: [4478669.197561]  [<ffffffff810f7eac>] ? do_lookup+0x80/0x15d
Jan 14 09:48:26 puffin kernel: [4478669.197569]  [<ffffffff810f86aa>] ? __link_path_walk+0x323/0x6f5
Jan 14 09:48:26 puffin kernel: [4478669.197577]  [<ffffffff810f8caa>] ? path_walk+0x66/0xc9
Jan 14 09:48:26 puffin kernel: [4478669.197585]  [<ffffffff810fa114>] ? do_path_lookup+0x20/0x77
Jan 14 09:48:26 puffin kernel: [4478669.197593]  [<ffffffff810fa2a0>] ? do_filp_open+0xe5/0x94b
Jan 14 09:48:26 puffin kernel: [4478669.197602]  [<ffffffff8130f054>] ? _spin_lock_irqsave+0x15/0x34
Jan 14 09:48:26 puffin kernel: [4478669.197612]  [<ffffffff81068fa6>] ? hrtimer_try_to_cancel+0x3a/0x43
Jan 14 09:48:26 puffin kernel: [4478669.197622]  [<ffffffff8102de30>] ? pvclock_clocksource_read+0x3a/0x8b
Jan 14 09:48:26 puffin kernel: [4478669.197631]  [<ffffffff81103705>] ? alloc_fd+0x67/0x10c
Jan 14 09:48:26 puffin kernel: [4478669.197641]  [<ffffffff810eeacf>] ? do_sys_open+0x55/0xfc
Jan 14 09:48:26 puffin kernel: [4478669.197649]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b

Things I noticed during the boot:

We have lots of attempts to connect to the FTP server, pure-ftp, but none of our developers use it, can we uninstall it or does BOA require it?
We have collectd installed but not configured, can we uninstall it or does BOA require it?

Counting failed FTP login attempts from /var/log/messages:

cat messages | grep pure-ftp | grep "New connection from" | wc -l
583
cat messages.1 | grep pure-ftp | grep "New connection from" | wc -l
2367
zcat messages.*.gz | grep pure-ftp | grep "New connection from" | wc -l
7203

That is over 10k FTP login attempts in the last month.

jim — Tue, 14 Jan 2014 10:42:11 GMT

MySQL is offline -- are you still running updates?

chris — Tue, 14 Jan 2014 10:49:56 GMT

Replying to jim:

MySQL is offline -- are you still running updates?

Crap, I assumed it would start on a reboot, I have tried starting it:

/etc/init.d/mysql restart
[ ok ] Stopping MariaDB database server: mysqld.
[FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!

/etc/init.d/mysql start
[FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!

Gonna look at the logs now...

chris — Tue, 14 Jan 2014 10:53:03 GMT

Fixed:

chown mysql:mysql /run/shm/mysql
[ ok ] Starting MariaDB database server: mysqld . . . ..
[info] Checking for corrupt, not cleanly closed and upgrade needing tables..

I don't know why that was needed, this is in my crontab:

crontab -e -u chris
@reboot sudo mkdir /run/shm/mysql
@reboot sudo chown mysql:mysql /run/shm/mysql

It's in my crontab as the root one is clobbered by BOA updates.

hours, totalhours changed

chris — Tue, 14 Jan 2014 10:54:41 GMT

hours changed from 0.0 to 0.5
totalhours changed from 0.25 to 0.75

I have removed collectd-core:

dpkg -r collectd-core
(Reading database ... 58041 files and directories currently installed.)
Removing collectd-core ...
[ ok ] Stopping statistics collection and monitoring daemon: collectd.
Processing triggers for man-db ...

But there is an issue here:

aptitude search collectd
...
pB  collectd                                                  - statistics collection and monitoring daemon
c   collectd-core                                             - statistics collection and monitoring daemon (core system)
...

The flags mean:

State:
c - removed, but config-files still present (ie, not purged)
Action:
B - Broken
p - purge

So:

aptitude purge collectd
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
aptitude purge collectd-core
The following packages will be REMOVED:
  collectd-core{p}
0 packages upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Do you want to continue? [Y/n/?] y
(Reading database ... 57834 files and directories currently installed.)
Removing collectd-core ...
Purging configuration files for collectd-core ...
/var/lib/dpkg/info/collectd-core.postrm: line 23: db_input: command not found
/var/lib/dpkg/info/collectd-core.postrm: line 24: db_go: command not found
/var/lib/dpkg/info/collectd-core.postrm: line 25: db_get: command not found
/var/lib/dpkg/info/collectd-core.postrm: line 23: db_input: command not found
/var/lib/dpkg/info/collectd-core.postrm: line 24: db_go: command not found
/var/lib/dpkg/info/collectd-core.postrm: line 25: db_get: command not found
dpkg: warning: while removing collectd-core, directory '/var/lib/collectd' not empty so not removed

And now it's gone:

aptitude search collectd
p   collectd                                                                   - statistics collection and monitoring daemon
p   collectd-core                                                              - statistics collection and monitoring daemon (core system)
p   collectd-dbg                                                               - statistics collection and monitoring daemon (debugging symbols)
p   collectd-dev                                                               - statistics collection and monitoring daemon (development files)
p   collectd-utils                                                             - statistics collection and monitoring daemon (utilities)
p   kcollectd                                                                  - simple collectd graphing frontend for KDE
p   libcollectdclient-dev                                                      - client library for collectd's control interface (development files)
p   libcollectdclient0                                                         - client library for collectd's control interface

Regarding pure-ftp, it looks like it is installed from source by BOA, it's not listed as installed by aptitude:

aptitude search pure-ftp
p   mysqmail-pure-ftpd-logger                                                  - real-time logging system in MySQL - Pure-FTPd traffic-logger
p   pure-ftpd                                                                  - Secure and efficient FTP server
p   pure-ftpd-common                                                           - Pure-FTPd FTP server (Common Files)
p   pure-ftpd-ldap                                                             - Secure and efficient FTP server with LDAP user authentication
p   pure-ftpd-mysql                                                            - Secure and efficient FTP server with MySQL user authentication
p   pure-ftpd-postgresql                                                       - Secure and efficient FTP server with PostgreSQL user authentication

In terms of the kernel lockup/crash that caused the server to be unresponsive, I don't know what to suggest other than keeping an eye out for things like this in the future.

chris — Tue, 14 Jan 2014 10:56:22 GMT

For reference this is what I found in /var/log/syslog which alerted me to the reason why MySQL wouldn't start:

Jan 14 10:50:34 puffin mysqld: 140114 10:50:34 [ERROR] mysqld: Can't create/write to file '/run/shm/mysql/ibw6lZ2z' (Errcode: 13)
Jan 14 10:50:34 puffin mysqld: 140114 10:50:34  InnoDB: Error: unable to create temporary file; errno: 13

chris — Tue, 14 Jan 2014 11:02:12 GMT

This is the first email alert I got around the time of the kernel crash:

From: root@puffin.webarch.net
Date: Tue, 14 Jan 2014 09:44:08 +0000 (GMT)
To: chris@webarchitects.co.uk
Subject: lfd on puffin.webarch.net: High 5 minute load average alert - 8.79
[-- Attachment #1 --]
[-- Type: text/plain, Encoding: 7bit, Size: 0.2K --]
Time:                    Tue Jan 14 09:42:47 2014 +0000
1 Min Load Avg:          29.58
5 Min Load Avg:          8.79
15 Min Load Avg:         3.33
Running/Total Processes: 44/414
[-- Attachment #2: ps.txt --]
[-- Type: text/plain, Encoding: 7bit, Size: 0.1K --]
Output from ps:
Unable to obtain process output within 15 seconds - Timed out
[-- Attachment #3: vmstat.txt --]
[-- Type: text/plain, Encoding: 7bit, Size: 0.1K --]
Output from vmstat:
Unable to obtain vmstat output within 10 seconds - Timed out
[-- Attachment #4: apachestatus.html --]
[-- Type: text/html, Encoding: 7bit, Size: 0.1K --]
   Unable to retrieve Apache Server Status [http://127.0.0.1/server-status] -
   Unable to download: 404 - Not Found

hours, totalhours changed

chris — Tue, 14 Jan 2014 12:31:51 GMT

hours changed from 0.0 to 0.16
totalhours changed from 0.75 to 0.91

Perhaps the items in my crontab were run in the wrong order -- chowning a directory before it was created, so I have updated it to this:

@reboot sudo mkdir /run/shm/mysql ; sudo chown mysql:mysql /run/shm/mysql

And I have documented the crontab here: wiki:PuffinServer#Cron

sam — Wed, 15 Jan 2014 13:48:23 GMT

Hi Chris

transitionnetwork.org is currently unavailable. I created a new ticket, but maybe it should just be a comment here?

Not responding to a ping at the moment.

Could you have a look?

Thanks

Sam

status changed; resolution set

chris — Fri, 31 Jan 2014 14:04:34 GMT

status changed from new to closed
resolution set to fixed

Closing as this is resolved.