hours, totalhours changed

chris — Wed, 08 Jan 2014 11:51:38 GMT

hours changed from 0.0 to 0.32
totalhours changed from 0.0 to 0.32

The server is back up.

The console errors were not recent, this is from /var/log/kern.log.2.gz:

Dec 25 01:28:54 parrot kernel: [1412520.196163] INFO: task mysqld:7928 blocked for more than 120 seconds.
Dec 25 01:28:54 parrot kernel: [1412520.196183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 25 01:28:54 parrot kernel: [1412520.196191] mysqld        D 0000000000000000     0  7928  18454 0x00000000
Dec 25 01:28:54 parrot kernel: [1412520.196203]  ffff8800bfa69530 0000000000000286 0000000000000000 0000000000000000
Dec 25 01:28:54 parrot kernel: [1412520.196215]  0007ffffffffffff 0000000000000001 000000000000f9e0 ffff880074cb5fd8
Dec 25 01:28:54 parrot kernel: [1412520.196226]  00000000000157c0 00000000000157c0 ffff880002ce1530 ffff880002ce1828
Dec 25 01:28:54 parrot kernel: [1412520.196236] Call Trace:
Dec 25 01:28:54 parrot kernel: [1412520.196259]  [<ffffffffa00232bf>] ? jbd2_log_wait_commit+0xbf/0x112 [jbd2]
Dec 25 01:28:54 parrot kernel: [1412520.196273]  [<ffffffff81066332>] ? autoremove_wake_function+0x0/0x2e
Dec 25 01:28:54 parrot kernel: [1412520.196293]  [<ffffffffa003fb41>] ? ext4_sync_file+0x199/0x25c [ext4]
Dec 25 01:28:54 parrot kernel: [1412520.196304]  [<ffffffff8110d6e0>] ? vfs_fsync_range+0x73/0x9e
Dec 25 01:28:54 parrot kernel: [1412520.196319]  [<ffffffff8110d78a>] ? do_fsync+0x28/0x39
Dec 25 01:28:54 parrot kernel: [1412520.196325]  [<ffffffff8110d7b9>] ? sys_fsync+0xb/0x10
Dec 25 01:28:54 parrot kernel: [1412520.196333]  [<ffffffff81011b63>] ? sysret_check+0x17/0x5a
Dec 25 01:28:54 parrot kernel: [1412520.196341]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b

I can't see anything in the logs to indicate why it was not responding today.

There is also nothing I can see in the munin logs, https://penguin.transitionnetwork.org/munin/transitionnetwork.org/parrot.transitionnetwork.org/

I was alerted to the lask of response from the server by this email:

From: munin@penguin.webarch.net
Date: Wed, 08 Jan 2014 11:25:23 +0000
Subject: parrot.transitionnetwork.org Munin Alert
transitionnetwork.org :: parrot.transitionnetwork.org :: eth0 errors
        UNKNOWNs: errors is unknown, errors is unknown.

And I couldn't connect via SSH.

It's possible that it would have recovered without intervention.

Closing this ticket as I can't think of anything else to do on it and the server is up and running now.

status changed; resolution set

chris — Wed, 08 Jan 2014 11:51:56 GMT

status changed from new to closed
resolution set to fixed

cc, status, summary changed; resolution deleted

chris — Wed, 15 Jan 2014 15:03:20 GMT

cc aland added
status changed from closed to reopened
resolution fixed deleted
summary changed from Parrot isn't responding to Parrot lockups

wiki:ParrotServer locked up again today, again nothing in the logs, I stopped and restarted it at a xen level.

I have reopened this ticket to keep an eye on this issue and also added Alan as a CC.

chris — Wed, 15 Jan 2014 15:04:46 GMT

The alert I got about the problem from munin was as before:

From: munin@penguin.webarch.net
Date: Wed, 15 Jan 2014 13:55:22 +0000
Subject: parrot.transitionnetwork.org Munin Alert
transitionnetwork.org :: parrot.transitionnetwork.org :: eth0 errors
        UNKNOWNs: errors is unknown, errors is unknown.

status changed; resolution set

chris — Wed, 02 Apr 2014 10:28:07 GMT

status changed from reopened to closed
resolution set to fixed

Closing this, hoping the fix for the NFZ/ZFS server has resolved this, see ticket:618#comment:5

Transition Technology: Ticket #666: Parrot lockups

hours, totalhours changed

status changed; resolution set

cc, status, summary changed; resolution deleted

status changed; resolution set