SPF / Emails rejected from the website contact form

[sam]

We had a user report that they could not send a message via our contact form:

"Yesterday I sent a message to you via the contact form on the website. But obviously something went wrong: for I got a failure notice saying my message could not be delivered. Therefore I'm sending it directly via email (see below) hoping that you're receiving my message this way."

<info@…>: host mx1.spamfiltering.com[72.249.150.158] said:

550 81.95.XX.XX is not allowed to send mail from gmx.de. Please see
​http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.XX.XX
(in reply to end of DATA command)

(User details edited as this is publicly archived)

I'm not sure I quite understand what's going on here. Chris indicated in email that this would affect other users whose email provider has set this kind of SPF record.

Can we make an educated guess as to what proportion of email providers set this kind of SPF?

How many messages do we never get to see? Is it a problem? Or a small enough number of users that we just don't worry about it?

Thanks

Sam

[chris]

Replying to sam:

I'm not sure I quite understand what's going on here.

gmx.de has a ​SPF record, which you can see using this web ​SPF Record Checker or via the command line:

dig gmx.de txt +short
"v=spf1 ip4:213.165.64.0/23 ip4:74.208.5.64/26 ip4:212.227.126.128/25 ip4:212.227.15.0/25 ip4:212.227.17.0/27 ip4:74.208.4.192/26 ip4:82.165.159.0/24 -all"
"google-site-verification=8jV5RCya962H1ur14xR3xWCFUBsExt_YibC3HkuJ9X8"

This record basically says email from gmx.de should only originate from the listed networks.

The Transition Network mail server checks for SFP records so when the TN web server sends a email from a @gmx.de address it is rejected.

There are some things that could be done:

1. Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see ​http://www.openspf.org/Best_Practices/Webgenerated

2. Ask the ISP providing the mailservers for TN to disable SPF checking.

3. Ask gmx.de to change their SPF policy.

4. Ask people who use the forms to use a email address that doesn't have a restrictive SPF record

Can we make an educated guess as to what proportion of email providers set this kind of SPF?

I couldn't, there are ​some general stats here.

I could write a Bash script to take all the account email addresses and check to see how many have SFP records -- would you like me to do this?

How many messages do we never get to see? Is it a problem? Or a small enough number of users that we just don't worry about it?

Not sure, I could do some grepping of mailserver logs if needs be.

[chris]

Replying to sam:

<info@…>: host mx1.spamfiltering.com[72.249.150.158] said:

550 81.95.XX.XX is not allowed to send mail from gmx.de. Please see
​http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.XX.XX
(in reply to end of DATA command)

(User details edited as this is publicly archived)

BTW you edited out the IP address of the TN site, this is the link in the rejection email:

• ​http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.52.103

And that page explains:

An SPF-enabled mail server rejected a message that claimed an envelope sender address of userXX@….

An SPF-enabled mail server received a message from puffin.webarch.net (81.95.52.103) that claimed an envelope sender address of userXX@….

However, the domain gmx.de has declared using SPF that it does not send mail through puffin.webarch.net (81.95.52.103). That is why the message was rejected.

[chris]

Replying to chris:

There are some things that could be done:

1. Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see ​http://www.openspf.org/Best_Practices/Webgenerated

Here is a Drupal module to set the sender envelope and Reply-To header:

• ​https://drupal.org/project/contact_reply_to

This is the best solution -- we should use this module for the current site and the new D7 site. Drupal 8 does the right thing by default.

[chris]

Here is an example of the email headers (including envelope address on the first line) generated by the contact form at ​https://www.transitionnetwork.org/contact

Return-Path: <chris@webarchitects.co.uk>
To: "Chris Croome" <chris@webarchitects.co.uk>
Subject: [Website feedback] Test - please ignore!
Content-Transfer-Encoding: 8Bit
X-Mailer: Pressflow
Errors-To: site@transitionnetwork.org
Sender: chris@webarchitects.co.uk
From: chris@webarchitects.co.uk
Date: Tue, 24 Jun 2014 11:55:33 +0100 (BST)

See also ticket:746

[paul]

Replying to chris:

Replying to chris:

There are some things that could be done:

1. Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see ​http://www.openspf.org/Best_Practices/Webgenerated

Here is a Drupal module to set the sender envelope and Reply-To header:

• ​https://drupal.org/project/contact_reply_to

This is the best solution -- we should use this module for the current site and the new D7 site. Drupal 8 does the right thing by default.

Interesting conversation. It's the kind of drupal functionality that you would guess is already in core, but isn't. I agree with Chris, the way forward would be to install the Contact Reply-To module from contrib.

[chris]

Paul would you be happy to install the Contact Reply-To module?

Ed, are you happy with Paul installing it if he is on for it?

It's the only good solution for people using forms on the site which send email using their email address, due to SPF, so I think it should be installed.

[paul]

I can do that. If Annesley wants to install it, that's fine. Just ask for me, if you need me.

[ed]

I'm happy for it to be installed. As there is a general handover from Paul to Annesley going on, it could be either for now - depending on where you are all at with the publishing process...

[paul]

@Annesley

Would you like to give this a go? Let me know if you run into any difficulties.

[chris]

Is Annesley going to pick this up or is Paul going to do it?

[ed]

Following an (agreement) on thursday 7/8/14, let's get Annesley and Paul to work out who goes

[paul]

@Annesley

Would you like to give this a go? We can then discuss any problems during our call tomorrow?

[annesley]

hi! sorry, just spotted this. the Drupal Module sounds like a good solution.

to check i have understood properly: in essence we are sending emails from someone else to someone else. and, because we are not actually the right domain / IP to send those emails from that other person, it will be classed as spoof / spam.

so, instead the module will send the email from us / our server / our domain but the reply-to header will ensure that when the recipient replies it goes to the right place, from the right place again.

so: i think our recent agreement, correct me if i am wrong, places this as maintenance work for Paul. i am happy to do it if necessary. i think it is a case of:

file copying the module or using tn.ftp to drush dl contact_reply_to
enabling the module with tn.ftp drush en contact_reply_to
and then adding the lines in to the D6 makefile and pushing to github.

correct? i ask so that i can learn the process also :)
do we spin up a new site then afterwards as well? i don't understand that part still...

[ed]

my understanding is also that paul is doing the maintenance work following our agreement last week. It is also important that Annesley you understand what is happening on the Drupal side of things, so that although one person is doing the publishing, *at least* one other understands what is going on and how.

[paul]

Replying to annesley:

hi! sorry, just spotted this. the Drupal Module sounds like a good solution.

to check i have understood properly: in essence we are sending emails from someone else to someone else. and, because we are not actually the right domain / IP to send those emails from that other person, it will be classed as spoof / spam.

so, instead the module will send the email from us / our server / our domain but the reply-to header will ensure that when the recipient replies it goes to the right place, from the right place again.

That's exactly what I 'm thinking. Let me know if you want me confirm.

so: i think our recent agreement, correct me if i am wrong, places this as maintenance work for Paul. i am happy to do it if necessary. i think it is a case of:

file copying the module or using tn.ftp to drush dl contact_reply_to
enabling the module with tn.ftp drush en contact_reply_to
and then adding the lines in to the D6 makefile and pushing to github.

That's right. Just check it works on stage first (I know you'll do that, just saying for the record).

correct? i ask so that i can learn the process also :)

Cool. It's good that more than 1 person that can do a task. Please give it a go. Let me know if you need any help.

do we spin up a new site then afterwards as well? i don't understand that part still...

We only need to spin up new *platforms* and migrate sites to a new platform, when core moves to a new version.

Would you push your changes to the makefile to your fork on github, and send me a pull request.

[chris]

Any progress on this?

[ed]

Annesley - any progress on this?

[chris]

WordPress has the same problem as Drupal, see ticket:808.