Ticket #737 (assigned maintenance)

Opened 2 years ago

Last modified 2 years ago

SPF / Emails rejected from the website contact form

Reported by: sam Owned by: sam
Priority: minor Milestone: Maintenance
Component: Email Keywords:
Cc: chris, paul, annesley, ed Estimated Number of Hours: 0.0
Add Hours to Ticket: 0 Billable?: yes
Total Hours: 1.385

Description

We had a user report that they could not send a message via our contact form:

"Yesterday I sent a message to you via the contact form on the website. But obviously something went wrong: for I got a failure notice saying my message could not be delivered. Therefore I'm sending it directly via email (see below) hoping that you're receiving my message this way."

<info@…>: host mx1.spamfiltering.com[72.249.150.158] said:

550 81.95.XX.XX is not allowed to send mail from gmx.de. Please see
http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.XX.XX
(in reply to end of DATA command)

(User details edited as this is publicly archived)

I'm not sure I quite understand what's going on here. Chris indicated in email that this would affect other users whose email provider has set this kind of SPF record.

Can we make an educated guess as to what proportion of email providers set this kind of SPF?

How many messages do we never get to see? Is it a problem? Or a small enough number of users that we just don't worry about it?

Thanks

Sam

Change History

comment:1 Changed 2 years ago by ed

  • Owner changed from ed to sam
  • Status changed from new to assigned

comment:2 in reply to: ↑ description ; follow-up: ↓ 4 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.56
  • Total Hours changed from 0.0 to 0.56

Replying to sam:

I'm not sure I quite understand what's going on here.

gmx.de has a SPF record, which you can see using this web SPF Record Checker or via the command line:

dig gmx.de txt +short
"v=spf1 ip4:213.165.64.0/23 ip4:74.208.5.64/26 ip4:212.227.126.128/25 ip4:212.227.15.0/25 ip4:212.227.17.0/27 ip4:74.208.4.192/26 ip4:82.165.159.0/24 -all"
"google-site-verification=8jV5RCya962H1ur14xR3xWCFUBsExt_YibC3HkuJ9X8"

This record basically says email from gmx.de should only originate from the listed networks.

The Transition Network mail server checks for SFP records so when the TN web server sends a email from a @gmx.de address it is rejected.

There are some things that could be done:

  1. Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see http://www.openspf.org/Best_Practices/Webgenerated
  1. Ask the ISP providing the mailservers for TN to disable SPF checking.
  1. Ask gmx.de to change their SPF policy.
  1. Ask people who use the forms to use a email address that doesn't have a restrictive SPF record

Can we make an educated guess as to what proportion of email providers set this kind of SPF?

I couldn't, there are some general stats here.

I could write a Bash script to take all the account email addresses and check to see how many have SFP records -- would you like me to do this?

How many messages do we never get to see? Is it a problem? Or a small enough number of users that we just don't worry about it?

Not sure, I could do some grepping of mailserver logs if needs be.

comment:3 in reply to: ↑ description Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0.0 to 0.1
  • Total Hours changed from 0.56 to 0.66

Replying to sam:

<info@…>: host mx1.spamfiltering.com[72.249.150.158] said:

550 81.95.XX.XX is not allowed to send mail from gmx.de. Please see
http://www.openspf.net/Why?scope=mfrom;identity=userXX@gmx.de;ip=81.95.XX.XX
(in reply to end of DATA command)

(User details edited as this is publicly archived)

BTW you edited out the IP address of the TN site, this is the link in the rejection email:

And that page explains:

An SPF-enabled mail server rejected a message that claimed an envelope sender address of userXX@….

An SPF-enabled mail server received a message from puffin.webarch.net (81.95.52.103) that claimed an envelope sender address of userXX@….

However, the domain gmx.de has declared using SPF that it does not send mail through puffin.webarch.net (81.95.52.103). That is why the message was rejected.

comment:4 in reply to: ↑ 2 ; follow-up: ↓ 6 Changed 2 years ago by chris

  • Cc ed added
  • Add Hours to Ticket changed from 0.0 to 0.1
  • Total Hours changed from 0.66 to 0.76

Replying to chris:

There are some things that could be done:

  1. Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see http://www.openspf.org/Best_Practices/Webgenerated

Here is a Drupal module to set the sender envelope and Reply-To header:

This is the best solution -- we should use this module for the current site and the new D7 site. Drupal 8 does the right thing by default.

comment:5 Changed 2 years ago by chris

Here is an example of the email headers (including envelope address on the first line) generated by the contact form at https://www.transitionnetwork.org/contact

Return-Path: <chris@webarchitects.co.uk>
To: "Chris Croome" <chris@webarchitects.co.uk>
Subject: [Website feedback] Test - please ignore!
Content-Transfer-Encoding: 8Bit
X-Mailer: Pressflow
Errors-To: site@transitionnetwork.org
Sender: chris@webarchitects.co.uk
From: chris@webarchitects.co.uk
Date: Tue, 24 Jun 2014 11:55:33 +0100 (BST)

See also ticket:746

comment:6 in reply to: ↑ 4 Changed 2 years ago by paul

  • Add Hours to Ticket changed from 0.0 to 0.25
  • Total Hours changed from 0.76 to 1.01

Replying to chris:

Replying to chris:

There are some things that could be done:

  1. Change the form settings on TN site so that emails generated by forms use a Sender envelope address such as form@… or set the Reply-To the senders address. This is most (only?) practical solution, see http://www.openspf.org/Best_Practices/Webgenerated

Here is a Drupal module to set the sender envelope and Reply-To header:

This is the best solution -- we should use this module for the current site and the new D7 site. Drupal 8 does the right thing by default.

Interesting conversation. It's the kind of drupal functionality that you would guess is already in core, but isn't. I agree with Chris, the way forward would be to install the Contact Reply-To module from contrib.

comment:7 Changed 2 years ago by chris

Paul would you be happy to install the Contact Reply-To module?

Ed, are you happy with Paul installing it if he is on for it?

It's the only good solution for people using forms on the site which send email using their email address, due to SPF, so I think it should be installed.

comment:8 Changed 2 years ago by paul

  • Add Hours to Ticket 0 deleted

I can do that. If Annesley wants to install it, that's fine. Just ask for me, if you need me.

comment:9 Changed 2 years ago by ed

I'm happy for it to be installed. As there is a general handover from Paul to Annesley going on, it could be either for now - depending on where you are all at with the publishing process...

comment:10 Changed 2 years ago by paul

  • Add Hours to Ticket changed from 0.0 to 0.125
  • Total Hours changed from 1.01 to 1.135

@Annesley

Would you like to give this a go? Let me know if you run into any difficulties.

comment:11 Changed 2 years ago by chris

Is Annesley going to pick this up or is Paul going to do it?

comment:12 Changed 2 years ago by ed

Following an (agreement) on thursday 7/8/14, let's get Annesley and Paul to work out who goes

comment:13 Changed 2 years ago by paul

  • Add Hours to Ticket changed from 0.0 to 0.125
  • Total Hours changed from 1.135 to 1.26

@Annesley

Would you like to give this a go? We can then discuss any problems during our call tomorrow?

comment:14 follow-up: ↓ 16 Changed 2 years ago by annesley

hi! sorry, just spotted this. the Drupal Module sounds like a good solution.

to check i have understood properly: in essence we are sending emails from someone else to someone else. and, because we are not actually the right domain / IP to send those emails from that other person, it will be classed as spoof / spam.

so, instead the module will send the email from us / our server / our domain but the reply-to header will ensure that when the recipient replies it goes to the right place, from the right place again.

so: i think our recent agreement, correct me if i am wrong, places this as maintenance work for Paul. i am happy to do it if necessary. i think it is a case of:

file copying the module or using tn.ftp to drush dl contact_reply_to
enabling the module with tn.ftp drush en contact_reply_to
and then adding the lines in to the D6 makefile and pushing to github.

correct? i ask so that i can learn the process also :)
do we spin up a new site then afterwards as well? i don't understand that part still...

comment:15 Changed 2 years ago by ed

my understanding is also that paul is doing the maintenance work following our agreement last week. It is also important that Annesley you understand what is happening on the Drupal side of things, so that although one person is doing the publishing, *at least* one other understands what is going on and how.

comment:16 in reply to: ↑ 14 Changed 2 years ago by paul

  • Add Hours to Ticket changed from 0.0 to 0.125
  • Total Hours changed from 1.26 to 1.385

Replying to annesley:

hi! sorry, just spotted this. the Drupal Module sounds like a good solution.

to check i have understood properly: in essence we are sending emails from someone else to someone else. and, because we are not actually the right domain / IP to send those emails from that other person, it will be classed as spoof / spam.

so, instead the module will send the email from us / our server / our domain but the reply-to header will ensure that when the recipient replies it goes to the right place, from the right place again.

That's exactly what I 'm thinking. Let me know if you want me confirm.

so: i think our recent agreement, correct me if i am wrong, places this as maintenance work for Paul. i am happy to do it if necessary. i think it is a case of:

file copying the module or using tn.ftp to drush dl contact_reply_to
enabling the module with tn.ftp drush en contact_reply_to
and then adding the lines in to the D6 makefile and pushing to github.

That's right. Just check it works on stage first (I know you'll do that, just saying for the record).

correct? i ask so that i can learn the process also :)

Cool. It's good that more than 1 person that can do a task. Please give it a go. Let me know if you need any help.

do we spin up a new site then afterwards as well? i don't understand that part still...

We only need to spin up new *platforms* and migrate sites to a new platform, when core moves to a new version.

Would you push your changes to the makefile to your fork on github, and send me a pull request.

comment:17 Changed 2 years ago by chris

Any progress on this?

comment:18 Changed 2 years ago by ed

Annesley - any progress on this?

comment:19 Changed 2 years ago by chris

WordPress has the same problem as Drupal, see ticket:808.

Note: See TracTickets for help on using tickets.