Transition Technology: Ticket #724: Subscription emails from Rob's blog not arriving.

sam — Tue, 29 Apr 2014 09:31:08 GMT

Using the test function seems to generate the content correctly: https://www.transitionnetwork.org/admin/messaging/notifications/events/test

chris — Tue, 29 Apr 2014 09:34:12 GMT

If you can send me the email addresses that haven't had the notifications off-list then I can grep the maillogs for these addresses to see if I can find out where the problem is -- it could be at a Drupal level, or SMTP level.

owner, status changed

sam — Tue, 29 Apr 2014 10:51:25 GMT

owner changed from ed to chris
status changed from new to assigned

Thanks Chris. Email sent..

cc, hours, component, totalhours changed

chris — Tue, 29 Apr 2014 11:24:34 GMT

cc jim, ed added
hours changed from 0.0 to 0.45
component changed from Unassigned to Drupal modules & settings
totalhours changed from 0.0 to 0.45

Looking at the two email addresses, the first one has a couple of recent log entries that I expect relate to the problem being reported using a form on the web site and a reply being sent.

Prior to that emails were sent to that address on:

Apr 11 00:56:12
Apr 11 00:56:11
Apr 9 22:57:09
Apr 8 20:58:53
Apr 8 20:58:50
Apr 4 20:56:18
Apr 3 18:56:32
Apr 2 16:57:21
Apr 1 16:56:31
Apr 1 16:56:30
Apr 1 16:56:30

There have been none since April 11th. There are no errors in the logs.

For the other address emails were sent on:

Apr 14 10:22:24
Apr 11 12:59:48
Apr 10 18:59:58
Apr 10 10:59:51
Apr 8 18:58:40
Apr 8 10:57:20
Apr 4 08:57:15
Apr 4 08:57:14
Apr 3 10:56:37
Apr 2 08:57:18

Again no errors -- it very much looks like a Drupal issue and not a SMTP issue.

If we assume that email, on the 14th was for something else then it looks like the last notification was sent on 11th April.

I can't see what might have changed on 12th April from the https://tech.transitionnetwork.org/trac/timeline but this doesn't include comments, it would be handy if it did... perhaps that is something we can change.

The last BOA upgrade was started at Fri Apr 11 21:55:51, see ticket:707#comment:19 so I guess that rules it out as a cause since the last email appears to have been sent after that.

I have assigned the component for this ticket to "Drupal modules & settings" and have added Jim as a CC as he might instantly realise what the problem is.

hours, totalhours changed

chris — Tue, 29 Apr 2014 11:41:12 GMT

hours changed from 0.0 to 0.21
totalhours changed from 0.45 to 0.66

Following the suggestion here, http://www.gossamer-threads.com/lists/trac/users/40082 I have enabled an extra tick box on the timeline so we can see comments if we need to.

https://tech.transitionnetwork.org/trac/timeline

The blog email notifications stopping appear to only coincide with the BOA update and the Heartbleed updates, the first blog entry that the reporter of the bug didn't get appears to have been this one:

https://www.transitionnetwork.org/blogs/rob-hopkins/2014-04/impact-were-having-zsanett-and-zolt-n-transition-hossz-het-ny from looking at the RSS feed https://www.transitionnetwork.org/blogs/feed/rob-hopkins, it was published at 06:37:52 +0000

However I'm afraid that I'm at a loss to know how those updates could have broken Drupal email notifications.

hours, totalhours changed

jim — Tue, 29 Apr 2014 12:18:56 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.66 to 0.76

Cron hasn't run for 2 weeks 3 days according to https://www.transitionnetwork.org/admin/reports/status

So the upgrade didn't complete and restart the site crons... Was Octopus updated straight after Barracuda?

The hosting queues aren't running I'd guess (can't check from work) which would explain the lack of Cron.

jim — Tue, 29 Apr 2014 12:19:10 GMT

FYI I just ran cron.

hours, totalhours changed

chris — Tue, 29 Apr 2014 12:33:24 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.76 to 1.01

Replying to jim:

Was Octopus updated straight after Barracuda?

I'm really sorry but I'm afraid it appears that I forgot to do that. Is it safe to do it during the day or is it best done late at night / at the weekend when there are less people using the site?

There is an outstanding BOA update that hasn't been applied, BOA-2.2.3, this hasn't been applied due to the maintenance budget running out this month, see ticket:721

Would running it re-enable the Drupal cron task?

Following is the current root crontab:

* * * * * bash /var/xdrago/second.sh >/dev/null 2>&1
* * * * * bash /var/xdrago/minute.sh >/dev/null 2>&1
* * * * * bash /var/xdrago/runner.sh >/dev/null 2>&1
*/5 * * * * bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1
03 * * * * bash /var/xdrago/clear.sh >/dev/null 2>&1
01 0 * * * bash /var/xdrago/graceful.sh >/dev/null 2>&1
08 0 * * * bash /var/xdrago/mysql_backup.sh >/dev/null 2>&1
58 2 * * * bash /var/xdrago/daily.sh >/dev/null 2>&1
58 5 * * * bash /var/xdrago/fetch-remote.sh >/dev/null 2>&1
###EOF2014###

This is the aegir crontab:

SHELL=/bin/bash
PATH=/opt/php53/bin:/sbin:/bin:/usr/sbin:/usr/bin
*/1 * * * * /var/aegir/drush/drush.php '@hostmaster' hosting-dispatch >/dev/null 2>&1

I can't find any others.

chris — Wed, 30 Apr 2014 09:56:04 GMT

Unless I hear otherwise I'll do the BOA-2.2.3 upgrade tonight, ticket:721 and hopefully this will resolve this issue.

ed — Wed, 30 Apr 2014 15:20:29 GMT

OK!

hours, totalhours changed

chris — Thu, 01 May 2014 13:15:24 GMT

hours changed from 0.0 to 0.25
totalhours changed from 1.01 to 1.26

According to https://www.transitionnetwork.org/admin/reports/status cron is running again, there has been more mailserver activity:

So I think this ticket is ready to close if someone can confirm they are getting the emails again.

status changed; resolution set

sam — Tue, 06 May 2014 08:59:27 GMT

status changed from assigned to closed
resolution set to fixed

Hi Chris

Thanks for this. I can confirm the emails are now coming so will close the ticket.

The backlog all arrived in peoples inboxes in a mad flurry, so if this happens again I reckon we should delete the queue before restarting Cron..

Thanks

Sam

ed — Tue, 06 May 2014 09:05:43 GMT

Absolutely. Sam I'm sure you've got all the complaints under control (ie I've seen some but am not responding them so we dont' cross wires)

chris — Tue, 06 May 2014 09:20:18 GMT

Replying to sam:

The backlog all arrived in peoples inboxes in a mad flurry

Oh no, once again very sorry about this :-(

so if this happens again I reckon we should delete the queue before restarting Cron..

Good idea, but I don't know how to do that -- the queue was at a Drupal not a mailserver level.

hours, totalhours changed

paul — Tue, 06 May 2014 15:40:03 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.26 to 1.385

Delivery of subscription emails can be managed at admin/messaging/subscriptions