Transition Technology: Ticket #737: SPF / Emails rejected from the website contact form

owner, status changed

ed — Thu, 05 Jun 2014 15:51:09 GMT

owner changed from ed to sam
status changed from new to assigned

hours, totalhours changed

chris — Fri, 06 Jun 2014 09:36:41 GMT

hours changed from 0.0 to 0.56
totalhours changed from 0.0 to 0.56

Replying to sam:

I'm not sure I quite understand what's going on here.

gmx.de has a SPF record, which you can see using this web SPF Record Checker or via the command line:

dig gmx.de txt +short
"v=spf1 ip4:213.165.64.0/23 ip4:74.208.5.64/26 ip4:212.227.126.128/25 ip4:212.227.15.0/25 ip4:212.227.17.0/27 ip4:74.208.4.192/26 ip4:82.165.159.0/24 -all"
"google-site-verification=8jV5RCya962H1ur14xR3xWCFUBsExt_YibC3HkuJ9X8"

This record basically says email from gmx.de should only originate from the listed networks.

The Transition Network mail server checks for SFP records so when the TN web server sends a email from a @gmx.de address it is rejected.

There are some things that could be done:

Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see http://www.openspf.org/Best_Practices/Webgenerated

Ask the ISP providing the mailservers for TN to disable SPF checking.

Ask gmx.de to change their SPF policy.

Ask people who use the forms to use a email address that doesn't have a restrictive SPF record

Can we make an educated guess as to what proportion of email providers set this kind of SPF?

I couldn't, there are some general stats here.

I could write a Bash script to take all the account email addresses and check to see how many have SFP records -- would you like me to do this?

How many messages do we never get to see? Is it a problem? Or a small enough number of users that we just don't worry about it?

Not sure, I could do some grepping of mailserver logs if needs be.

hours, totalhours changed

chris — Fri, 06 Jun 2014 09:48:22 GMT

hours changed from 0.0 to 0.1
totalhours changed from 0.56 to 0.66

Replying to sam:

<info@…>: host mx1.spamfiltering.com[72.249.150.158] said:

550 81.95.XX.XX is not allowed to send mail from gmx.de. Please see http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.XX.XX (in reply to end of DATA command)

(User details edited as this is publicly archived)

BTW you edited out the IP address of the TN site, this is the link in the rejection email:

http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.52.103

And that page explains:

An SPF-enabled mail server rejected a message that claimed an envelope sender address of userXX@….

An SPF-enabled mail server received a message from puffin.webarch.net (81.95.52.103) that claimed an envelope sender address of userXX@….

However, the domain gmx.de has declared using SPF that it does not send mail through puffin.webarch.net (81.95.52.103). That is why the message was rejected.

cc, hours, totalhours changed

chris — Fri, 06 Jun 2014 09:59:10 GMT

cc ed added
hours changed from 0.0 to 0.1
totalhours changed from 0.66 to 0.76

Replying to chris:

There are some things that could be done:

Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see http://www.openspf.org/Best_Practices/Webgenerated

Here is a Drupal module to set the sender envelope and Reply-To header:

https://drupal.org/project/contact_reply_to

This is the best solution -- we should use this module for the current site and the new D7 site. Drupal 8 does the right thing by default.

chris — Tue, 24 Jun 2014 11:05:45 GMT

Here is an example of the email headers (including envelope address on the first line) generated by the contact form at https://www.transitionnetwork.org/contact

Return-Path: <chris@webarchitects.co.uk>
To: "Chris Croome" <chris@webarchitects.co.uk>
Subject: [Website feedback] Test - please ignore!
Content-Transfer-Encoding: 8Bit
X-Mailer: Pressflow
Errors-To: site@transitionnetwork.org
Sender: chris@webarchitects.co.uk
From: chris@webarchitects.co.uk
Date: Tue, 24 Jun 2014 11:55:33 +0100 (BST)

hours, totalhours changed

paul — Tue, 29 Jul 2014 14:48:55 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.76 to 1.01

Replying to chris:

Replying to chris:

There are some things that could be done:

Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see http://www.openspf.org/Best_Practices/Webgenerated

Here is a Drupal module to set the sender envelope and Reply-To header:

https://drupal.org/project/contact_reply_to
This is the best solution -- we should use this module for the current site and the new D7 site. Drupal 8 does the right thing by default.

Interesting conversation. It's the kind of drupal functionality that you would guess is already in core, but isn't. I agree with Chris, the way forward would be to install the Contact Reply-To module from contrib.

chris — Mon, 04 Aug 2014 17:01:45 GMT

Paul would you be happy to install the Contact Reply-To module?

Ed, are you happy with Paul installing it if he is on for it?

It's the only good solution for people using forms on the site which send email using their email address, due to SPF, so I think it should be installed.

hours deleted

paul — Mon, 04 Aug 2014 18:19:54 GMT

hours 0 deleted

I can do that. If Annesley wants to install it, that's fine. Just ask for me, if you need me.

ed — Tue, 05 Aug 2014 09:02:45 GMT

I'm happy for it to be installed. As there is a general handover from Paul to Annesley going on, it could be either for now - depending on where you are all at with the publishing process...

hours, totalhours changed

paul — Tue, 05 Aug 2014 12:44:44 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.01 to 1.135

@Annesley

Would you like to give this a go? Let me know if you run into any difficulties.

chris — Tue, 05 Aug 2014 21:22:51 GMT

Is Annesley going to pick this up or is Paul going to do it?

ed — Wed, 06 Aug 2014 16:02:15 GMT

Following an (agreement) on thursday 7/8/14, let's get Annesley and Paul to work out who goes

hours, totalhours changed

paul — Wed, 06 Aug 2014 16:20:45 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.135 to 1.26

@Annesley

Would you like to give this a go? We can then discuss any problems during our call tomorrow?

annesley — Mon, 11 Aug 2014 08:06:25 GMT

hi! sorry, just spotted this. the Drupal Module sounds like a good solution.

to check i have understood properly: in essence we are sending emails from someone else to someone else. and, because we are not actually the right domain / IP to send those emails from that other person, it will be classed as spoof / spam.

so, instead the module will send the email from us / our server / our domain but the reply-to header will ensure that when the recipient replies it goes to the right place, from the right place again.

so: i think our recent agreement, correct me if i am wrong, places this as maintenance work for Paul. i am happy to do it if necessary. i think it is a case of:

file copying the module or using tn.ftp to drush dl contact_reply_to enabling the module with tn.ftp drush en contact_reply_to and then adding the lines in to the D6 makefile and pushing to github.

correct? i ask so that i can learn the process also :) do we spin up a new site then afterwards as well? i don't understand that part still...

ed — Mon, 11 Aug 2014 08:44:23 GMT

my understanding is also that paul is doing the maintenance work following our agreement last week. It is also important that Annesley you understand what is happening on the Drupal side of things, so that although one person is doing the publishing, *at least* one other understands what is going on and how.

hours, totalhours changed

paul — Mon, 11 Aug 2014 10:16:40 GMT

hours changed from 0.0 to 0.125
totalhours changed from 1.26 to 1.385

Replying to annesley:

hi! sorry, just spotted this. the Drupal Module sounds like a good solution.

to check i have understood properly: in essence we are sending emails from someone else to someone else. and, because we are not actually the right domain / IP to send those emails from that other person, it will be classed as spoof / spam.

so, instead the module will send the email from us / our server / our domain but the reply-to header will ensure that when the recipient replies it goes to the right place, from the right place again.

That's exactly what I 'm thinking. Let me know if you want me confirm.

so: i think our recent agreement, correct me if i am wrong, places this as maintenance work for Paul. i am happy to do it if necessary. i think it is a case of:

file copying the module or using tn.ftp to drush dl contact_reply_to enabling the module with tn.ftp drush en contact_reply_to and then adding the lines in to the D6 makefile and pushing to github.

That's right. Just check it works on stage first (I know you'll do that, just saying for the record).

correct? i ask so that i can learn the process also :)

Cool. It's good that more than 1 person that can do a task. Please give it a go. Let me know if you need any help.

do we spin up a new site then afterwards as well? i don't understand that part still...

We only need to spin up new *platforms* and migrate sites to a new platform, when core moves to a new version.

Would you push your changes to the makefile to your fork on github, and send me a pull request.

chris — Wed, 03 Sep 2014 10:35:08 GMT

Any progress on this?

ed — Mon, 08 Sep 2014 12:47:52 GMT

Annesley - any progress on this?

chris — Mon, 17 Nov 2014 20:06:36 GMT

WordPress has the same problem as Drupal, see ticket:808.