General developer forum

DMARC Causing tons of bounces

 
 
Picture of Travis Johnston
DMARC Causing tons of bounces
 

My Version: Moodle 2.4.9

It seems that Yahoo!, Comcast, and AOL have been busy over the weekend and have issued a new more restricted DMARC policy on their email domains, causing students and instructors with these email addresses to no longer receive notifications from Moodle. This is because Moodle inserts the student/instructor email address into the FROM header, and sends the email on behalf of that person. 

See example header from bounced message below. In this example, it seems to be effecting anything using a Yahoo address as sending or receiving. 

------------------------------------------------------------------

    550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to
    domain's 550-5.7.1 DMARC policy. Please contact administrator of yahoo.com
    domain if 550-5.7.1 this was a legitimate mail. Please visit 550-5.7.1
    http://support.google.com/mail/answer/2451690 to learn about DMARC 550
    5.7.1 initiative. m6si942994qay.218 - gsmtp (in reply to end of DATA
    command)

Final-Recipient: rfc822; email@gmail.com
Original-Recipient: rfc822;email@gmail.com
Action: failed
Status: 5.7.1
Remote-MTA: dns; gmail-smtp-in.l.google.com
Diagnostic-Code: smtp; 550-5.7.1 Unauthenticated email from yahoo.com is not
    accepted due to domain's 550-5.7.1 DMARC policy. Please contact
    administrator of yahoo.com domain if 550-5.7.1 this was a legitimate mail.
    Please visit 550-5.7.1  http://support.google.com/mail/answer/2451690 to
    learn about DMARC 550 5.7.1 initiative. m6si942994qay.218 - gsmtp


---------- Forwarded message ----------
From: "Joe Smith" <joesmith@yahoo.com>
To: "Jane Doe" <Janedoe@gmail.com>
Cc: 
Date: Tue, 8 Apr 2014 11:00:24 -0400
Subject: Course Subject

 

-----------------------------------

There has been a bunch of uproar this week from it, mentioned here http://www.theregister.co.uk/2014/04/08/yahoo_breaks_every_mailing_list_in_the_world_says_email_guru/ John Levine talks about the impact this has and what he has been doing to address it - such as telling people to change their email address.

In Moodle, there hasn't been any discussion on this. There is no place in the administration interface to edit email headers so I have been trying to figure out how to do it in the lib/moodlelib.php file but haven't had any luck. 

What needs to be done, seeing in this section at the DMARC website http://www.dmarc.org/faq.html#s_22, is to change the FROM address to be the same as the No-Reply address or the email for the domain, and then set the REPLY TO as the student/instructor email address.

Though I am not sure how to do this and I want to bring this up to the community for review and to hopefully get some help.

 
Average of ratings: -
Picture of Travis Johnston
Re: DMARC Causing tons of bounces
 

Fixed! Below is a copy of my code. The change needs to be made in the lib/moodlelib.php file. I am in terminal right now so I can't tell what line I am on but search for the following and you will find it.

 

if (is_string($from)) { // So we can pass whatever we want if there is need

        $mail->From     = $CFG->noreplyaddress;

        $mail->FromName = $from;

    } else if ($usetrueaddress and $from->maildisplay) {

        // Changing this due to new DMARC policy requirements
        //$mail->From     = $from->email;
        $mail->From     = $CFG->noreplyaddress;

        $mail->FromName = fullname($from);

    } else {

        $mail->From     = $CFG->noreplyaddress;

        $mail->FromName = fullname($from);

        if (empty($replyto)) {

            $tempreplyto[] = array($CFG->noreplyaddress, get_string('noreplyname'));

        }

    }

 

Of course you could also remove the part of the else statement and it would still work as well. Also of note I did not change anything with the Reply To, as it is really not needed since the message body tends to include links to interact with the notification by pointing the recipient back to Moodle. This is a temporary solution which hopefully will be addressed in future Moodle patches. 

 
Average of ratings:Useful (2)
Picture of Rex Lorenzo
Re: DMARC Causing tons of bounces
Group DevelopersGroup Particularly helpful MoodlersGroup Plugins guardiansGroup Testers

A related tracker issue might be:

Assignment Submission Notifications not received at AOL accounts
https://tracker.moodle.org/browse/MDL-42580

But for your change, are you editing the email_to_user function? I believe there are other areas in Moodle that also send out email that might need to be changed.

Also, I am asking our support folks if this is affecting us.

 
Average of ratings: -
Picture of Rex Lorenzo
Re: DMARC Causing tons of bounces
Group DevelopersGroup Particularly helpful MoodlersGroup Plugins guardiansGroup Testers

Okay, I see that we have had 164 emails so far with rejection from Yahoo.

But the error messages I see in the bounced emails are:


<XYZ@yahoo.com>: host mta5.am0.yahoodns.net[98.136.216.26] said: 554 5.7.9
    Message not accepted for policy reasons.  See
    http://postmaster.yahoo.com/errors/postmaster-28.html (in reply to end of
    DATA command)

And gmail.com:

<XYZ@gmail.com>: host gmail-smtp-in.l.google.com[74.125.129.26] said:
    550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to
    domain's 550-5.7.1 DMARC policy. Please contact administrator of yahoo.com
    domain if 550-5.7.1 this was a legitimate mail. Please visit 550-5.7.1
    http://support.google.com/mail/answer/2451690 to learn about DMARC 550
    5.7.1 initiative. ha5si1628664pbc.430 - gsmtp (in reply to end of DATA
    command)

And hotmail:

<XYZ@hotmail.com>: host mx1.hotmail.com[65.55.92.152] said: 550 5.7.0
    (SNT0-MC2-F44) Unfortunately, messages from (169.232.194.164) on behalf of
    (yahoo.com) could not be delivered due to domain owner policy restrictions.
    (in reply to end of DATA command)

What is the proper fix for this? How are others handling this?

 
Average of ratings: -
Picture of ahmad shaarawy
Re: DMARC Causing tons of bounces
 

have the same error

 
Average of ratings: -
Picture of Fernando Malick
Re: DMARC Causing tons of bounces
 

This is becoming a real pain in the... sad

The problem is that forums are sending emails in which the "From:" record reflects the originals senders email address, which obviously might not correspond to the domain of the moodle platform.

This triggers the error, and the result is that emails from moodle forums are not being delivered.

¿How can this be resolved?

 

 
Average of ratings: -
Picture of André Yamin
Re: DMARC Causing tons of bounces
Group Translators

Hi Fernando,

A workaround is to change the notification setting to compiled email for everyone how have a yahoo email until the fix is ready.

You can do it doing an sql update to mdl_user. Something like "UPDATE mdl_user SET maildigest="1" WHERE email like "%yahoo%";

Be carefull with it!

HTH,

 

André Yamin.

 
Average of ratings: -
Picture of Travis Johnston
Re: DMARC Causing tons of bounces
 

Hey Rex and Fernando,

I have had no trouble since my fix, which I mentioned above, which is to change how Moodle handles the FROM variable in it's emails that are sent out. This should work fine, just make note of it because if Moodle doesn't release a patch to address this, you will need to make the change again manually next time you update. 

 

Rex,

Yes this code is located in lib/moodle.php and is inside the email_to_user() function

 
Average of ratings: -
Picture of Rex Lorenzo
Re: DMARC Causing tons of bounces
Group DevelopersGroup Particularly helpful MoodlersGroup Plugins guardiansGroup Testers

Travis, have you taken a look at the patch in https://tracker.moodle.org/browse/MDL-43669?

I am going to put this on our TEST server on Monday and see if this fixes the problem. The solution is a little better than yours, because it saves the from email address and puts it into the reply-to field. So people can still see who the poster was and contact them.

 
Average of ratings: -
Picture of Bob Martens
Re: DMARC Causing tons of bounces
 

Thanks for this, we have just pushed this out to production to try and alleviate some issues.

What I would love to see in the future is the ability to set the reply-to address as well to the users email address and have the from address continue to be the no-reply. I'll probably head over to the bug tracker and put in my thoughts.

 
Average of ratings: -
Picture of Carlos Rojas
Re: DMARC Causing tons of bounces
 

Thanks Travis Johnston your fix worked perfectly.

 

 

 
Average of ratings: -
Picture of Donna Smith
Re: DMARC Causing tons of bounces
 

Thanks! I hope this works, the bounces have been annoying.

 
Average of ratings: -
Picture of Travis Johnston
Re: DMARC Causing tons of bounces
 

Hey Everyone,

I looked at that patch but it appears that is good from 2.6 onwards. Here is the new code I wrote that keeps the Reply To email address as the student/instructor, so the email communication can continue where as before you could not reply to the email since it was the no-reply address.


if (is_string($from)) { // So we can pass whatever we want if there is need

        $mail->From     = $CFG->noreplyaddress;

        $mail->FromName = $from;

    } else if ($usetrueaddress and $from->maildisplay) {

// Changing this due to new DMARC policy requirements

//$mail->From     = $from->email;

        $mail->From     = $CFG->noreplyaddress;

        $mail->FromName = fullname($from);

        $replyto  = $from->email;

    } else {

        $mail->From     = $CFG->noreplyaddress;

        $mail->FromName = fullname($from);

        $replyto        = $from->email;

        if (empty($replyto)) {

            $tempreplyto[] = array($CFG->noreplyaddress, get_string('noreplyname'));

        }

    }

 
Average of ratings: -
Picture of Donna Smith
Re: DMARC Causing tons of bounces
 

We are at 2.3, if we upgrade to 2.6 or 2.7, does this mean this particular problem is fixed?

 
Average of ratings: -
Ben talking on the phone beside a monitor
Re: DMARC Causing tons of bounces
Group Particularly helpful MoodlersGroup Testers

No. This a patch to be applied locally, not a patch that has been incorporated into core software.

Do not think about upgrading from 2.3 without reading all the upgrade information. You are, I think, past the 2.2 Assignment upgrade, but you want to be aware of what else is changing, and this patch is not yet part of it.

Unless Mary Cooch tells me I'm wrong.smile

 
Average of ratings: -
Picture of Donna Smith
Re: DMARC Causing tons of bounces
 

Sorry, I was mistaken already at 2.6.1.

 
Average of ratings: -
Picture of Michael Aherne
Re: DMARC Causing tons of bounces
Group DevelopersGroup Particularly helpful Moodlers

Hi Donna, are you able to upgrade to 2.6.4? If so, you could use the new setting we added to stop this happening (MDL-43669), and then you wouldn't need to maintain the patch.

 
Average of ratings:Useful (1)
Picture of Donna Smith
Re: DMARC Causing tons of bounces
 

Thanks, Travis! I was missing this line in your original code: $replyto        = $from->email;

Once I added it, everything is now working as it should.

You ROCK!!

 
Average of ratings: -
Picture of Paul Baumeister
Re: DMARC Causing tons of bounces
 

Hi Travis,

I used your code and it did fix the problem with the sender using the no reply. However, if I ask for a copy from Quickmail of the email that I sent, no email copy comes to my inbox. I am assuming because it sends a copy to the email address sender and in this case it is the no-reply@ email address. Is this correct? Is there a fix for this?

 
Average of ratings: -
Picture of Rob Clark
Re: DMARC Causing tons of bounces
 

I'm running Moodle 2.7.1+.  I tried this patch, but it seems to have stopped all emails from the system.  When I reverted back to the original moodlelib.php file, there are still no emails being sent from Moodle.  Any ideas?

 
Average of ratings: -
Picture of Rob Clark
Re: DMARC Causing tons of bounces
 

Sorry.  Fixed it.  All works very well.  Thanks a lot for the workaround. smile

 
Average of ratings: -
foto_perfil
Re: DMARC Causing tons of bounces
 

Hello Guys,

My Version: Moodle 2.6.7

I followed the guidelines as MDL-43669 and "Administration >> Site Administration >> Plugins >> Outputs messages >> Emails":

I enabled the "Always send email from the no-reply address?"

enable no-replay adress


Now the emails are delivered normally.
But after you enable the option mentioned, I get the following cron messages:

cron


Code /var/www/html/moodle/lib/moodlelib.php
...

 if (!empty($CFG->emailonlyfromnoreplyaddress)) {
        $usetrueaddress = false;
        if (empty($replyto) && $from->maildisplay) {  // line 5793
            $replyto = $from->email;
            $replytoname = fullname($from);
        }
    }
...


And now?


 
Average of ratings: -
Picture of Caleb Lee
Re: DMARC Causing tons of bounces
 

Thank you so much. so far it works perfectly.

 
Average of ratings: -
Picture of Victor Martinez
Re: DMARC Causing tons of bounces
 

I'm having the same issue, had checked the tracker and not found a specific fix, I think probably yahoo and other eventually had to lax their policy or at least adjust, in the meantime we got a lot of yahoo users that are not receiving their forum messages, so I created a ticket MDL-45030 to maybe provide or organize the patches and probably fixes.

 
Average of ratings: -
Dan at desk in Moodle HQ, Perth
Re: DMARC Causing tons of bounces
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers
A somewhat related issue which would resolve this was being discussed this week: MDL-43669

 
Average of ratings:Useful (2)
Picture of jeff w
Re: DMARC Causing tons of bounces
 

I have been suffering through this all week. What I ended up doing was going into Modules / Activities / Forum and unchecking "Use email address in reply" (default is set to yes). It fixed the immediate problem of messages bouncing from forums so the reply email matches the domain. But I noticed today that the messages from the messaging system have also started to bounce. 

 
Average of ratings:Useful (2)
Picture of Robin Sitten
Re: DMARC Causing tons of bounces
 

Thanks Jeff - this was a good suggestion that worked perhaps TOO well. Changing the Forum checkbox did stop the bouncing, but we discovered it had also stopped the SENDING of forum update notices, even to the Message box in-platform.

We still get assignment submissions notifs in the Messages box, but not forums.

so I rechecked that box in  Site Administration > Plug Ins > Activity Modules - Forums

and also changed our noreply email address to our helpdesk email. We'll now get nuisance replies, but I was eager to see if this made any difference. It didn't seem to.

Messages are still being rejected, and discussion forums have not restarted. I am going to subscribe to this forum, to see if I can get notifications.

Has anyone had any luck?


AOL Postmaster was kind enough to provide this acknowledgement

http://postmaster-blog.aol.com/2014/04/22/aol-mail-updates-dmarc-policy-to-reject/

but I wasn't sure how to use the information in Moodle admin settings.

 
Average of ratings: -