Rethinking email delivery from Moodle 3.2

Rethinking email delivery from Moodle 3.2

by Dan Poltawski -
Number of replies: 17

Starting in Moodle 3.2, we are considering doing relatively significant changes to the way Moodle delivers email, following on from various issues reported over the years.  Adrian and Simey have been working on a patch in MDL-44467 and I have been discussing the issue quite a bit with them.

I was particularly reminded of these problems by Jamie Kramer's presentation at #mootus16 - Moodle Email Problems and Solutions and it brought me back to my own experience doing Moodle hosting. But I would be interested to hear from other system administrators views on the current plan.

From my point of view, the main issue that we need to fix is that Moodle in it's out of the box configuration very often tries to send emails as the users of the system either as the 'From' or the 'Reply-to' address and this behaviour is incompatible with current spam-prevention measures in use today. 

The proposed solution (and you can see the patch in progress in MDL-44467) is:

  1. We will remove the current 'Only send from no reply address' flag.
  2. We will replace this with a list of allowed email domains  (which defaults to none)
  3. We will start delivering all emails from the no reply address at all times, unless the email is  in this list of allowed domains 
  4. We'll update the From name to include a 'via Moodle' (much like other websites do), to prevent address book confusion when sent via no-reply

This means that out of the box, the only task for anyone configuring Moodle to send email will be getting the no-reply address set to something which is not flagged as spam (e.g. has SPF records which trust the Moodle host as the sender).

For more advanced configurations, e.g. a university - where you are under control of a mail domain, and you want to allow emails to be sent out from Moodle directly as the user - you can add this domain to the allowed domains list. But, unlike the previous noreply setting - Moodle still will not try and send the mail directly if sending 'from' gmail.com and instead fallback to from the no reply address.

I'd love to hear your feedback on this proposal.

(this is not exactly on topic for this forum, but in my experience has the audience most interested)

Average of ratings: Useful (5)
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Sounds very much on the right track.

However, one question, if the user installs Moodle and does nothing at all (i.e. relies on local Sendmail processing - assuming that is working) what will the 'from' address be by default. In fact will mail work at all?

Mail processing is a thing of great confusion to users who have never had to administer mail servers and I wouldn't want to see things get even more confusing. For many people, "you'll need to set a noreply address that your MTA will deliver" will be a big ask (yes, I know they may have to do it regardless).

Average of ratings: -
In reply to Howard Miller

Re: Rethinking email delivery from Moodle 3.2

by Dan Poltawski -

However, one question, if the user installs Moodle and does nothing at all (i.e. relies on local Sendmail processing - assuming that is working) what will the 'from' address be by default. In fact will mail work at all?

It will be the default no-reply address (which is no-reply@your.moodle.host.name IIRC). I think that is the best that we can do without configuring anything.

I think that is the same/better to the situation now? Or is there another suggestion?

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

That'll do... it won't be any worse than now.

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Using the noreply@yourdomain.com was going to be my suggestion and I think that maybe it does that now but could not guarantee it.  

As long as there is good documentation to go along with the changes, I think this will work.  Initially, the excluded domains could be confusing...

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Ken Task -
Picture of Particularly helpful Moodlers

Will also say ... on towards 'the right track', like Howard.

2 cents more ... some of which is already mentioned ... and I'll try to refrain from 'examples'/stories! :|

Much of the successfull delivery of mail is dependent upon DNS and SPAM checkers/services on the other end (even what providers might do**)

1. the moodle server has a FQDN - forward as well as reverse.

2. "has SPF records which trust the Moodle host as the sender"

3. the no-reply to address/sending address/from address, etc. should be of the FQDN of the Moodle server itself.  Headers of the message as such that nothing is obscured/changed where SPAM checking could immediately block having failed the 'first checks'.

"a list of allowed email domains" .... and that would be determined how?
User entry?  It was amazing to see how in-experienced users configuring Windows XP
for DNS actually used the 'example' provided by Microsoft.

** one provider actually submits their own entire range of IP addresses in SBL (probably are other providers as well).

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Rethinking email delivery from Moodle 3.2

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Of course, if you are effectively sending email "on behalf of" a domain which is not your own (we do a lot of this as we provide hosting) then all bets are off determining the ideal way to make sure those are delivered. 

In reality, it's probably not a brilliant idea. In an ideal world mail should be sent by the 'home' email server. Any variation on this requires some expertise and experience. As I like to say, "it's not a Moodle problem". 

The new proposals seem to give us some flexibility in this area though.

Average of ratings: -
In reply to Ken Task

Re: Rethinking email delivery from Moodle 3.2

by Dan Poltawski -
Much of the successfull delivery of mail is dependent upon [..]

I agree with your points - unfortunately that is outside of the control of what we can do in Moodle - so we're trying to do the best we can to make that part as straight forward as possible . We can improve the diagnostics to help identify these issues (and I have discussed it, but I doubt it'll make it in this release - maybe someone could extend https://moodle.org/plugins/local_mailtest to do this in the meantime?). 

"a list of allowed email domains" .... and that would be determined how?

It would be manual user entry. I take your point, but it would not be something which I expect typical sites to manage. In fact, if I was designing from scratch I would only allow the no-reply option - but talking to admins, the ability for Moodle to send email as the user in question is something which people like to keep, I think this option helps us achieve that.

Average of ratings: Useful (1)
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm not promising (hours in the day and all that) but email is my secret "thing", so I'll try to take a look at that plugin.

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Matteo Scaramuccia -

Hi Dan,

It would be manual user entry.

Could it be a regexp too?

TIA,
Matteo

Average of ratings: -
In reply to Matteo Scaramuccia

Re: Rethinking email delivery from Moodle 3.2

by Dan Poltawski -

Yes, well sort of - there is some more limited wildcard matching support.

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Eric Hagley -

I would love to see an option where email is not required at all.

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Dan and all

I haven't yet looked at MDL-44467 in detail. For a start, if the old default and new default are identical, no harm could be done.

Talking of the old default, the e-mail address of the admin used to play a role in e-mail sending. Is it still the case or has it been isolated and removed?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Rethinking email delivery from Moodle 3.2

by Dan Poltawski -

For a start, if the old default and new default are identical, no harm could be done.

That isn't the case - the current default has $CFG->emailonlyfromnoreplyaddress disabled - so Moodle tries to email out as real user email addresses (and is a broken default IMO, which is why we are trying to fix it).

Talking of the old default, the e-mail address of the admin used to play a role in e-mail sending. Is it still the case or has it been isolated and removed?

There is also the support user which has been used. These will all be subject to the rules about allowed email domains, otherwise it sends using noreply address.

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Dan

Thanks for the clarifications. You wrote:

> the current default has $CFG->emailonlyfromnoreplyaddress disabled - so Moodle tries to email out as real user email addresses (and is a broken default IMO, which is why we are trying to fix it).

Yes, it was an unfortunate default. Sending mail on behalf of all its users was optimistic. It could have worked a decade ago. But today the success rate is low. Then there was an organizational weakness in that choice too. People just press "reply" to those mails and, knowingly or unknowingly, switch to private mail!

>> For a start, if the old default and new default are identical, no harm could be done.
>
> That isn't the case -

OK. Then this is break in the tradition. Needs to think it out from the beginning. But I still have a fundamental problem with the plan, sorry. The description sounds like a) defining the address(es) Moodle will use to send mail b) defining the address(es) Moodle is allowed to send.

That doesn't sound right. Moodle will tipp over its own feet. I am reluctant to be old fashioned, but don't we need to make the distinction between Mali User Agent (MUA) and the Mail Transfer Agent (MTA)? Moodle is a MUA, not a MTA, and wants to remain that way. Am I right?

>> Talking of the old default, the e-mail address of the admin used to play a role in e-mail sending. Is it still the case or has it been isolated and removed?
>
> There is also the support user which has been used.

So it was the site's support address I was talking about?

To make matters worse, there is also an Envelope Sender. We are talking about the body From: header, right?
Average of ratings: -
In reply to Dan Poltawski

degrama

by Grant Mucha -
My Solution and two cents. This may not be entirely related to your initial issue but ties into email delivery.


Please do not remove or eliminate the Site Administration -> Plugin -> Message Outputs -> Email option to configure SMTP hosts. I also struggled with email deliver-ability for many years. Since finding this option I was able to configure Moodle to work with MailGun a transactional email provider that removes the need for me to even run mail based services on my server. On top of that, all of my delivery issues are gone and my plan at MailGun is free under 10,000 emails per month. No email services on my servers, no security issues, no IP trust issues, no headaches.


I am really not a fan of sending out any email from no-reply based accounts. This simply closes the loop if clients or students are having problems and they have to find an alternative means to communicate. All of my email transactions are from a real email address which is checked and responded to daily to assist anyone having problems.

I believe From: and To: should include the users name or business name. This generally improves delivery.

To: 'degrama@gmail.com' => 'Degrama'

From: 'info@school.com' => 'School'

Typically if the to name matches the users it will be less likely to be designated spam.

Obviously as Dan indicated all domains should have SPF and DKIM setup these days. And both are extremely easy to setup in your DNS settings.

Hopefully some of this is helpful to someone.

Cheers,

Grant

Average of ratings: -
In reply to Grant Mucha

Re: degrama

by Dan Poltawski -

Nothing will be changing about the existing smtp hosts options nor the 'to' address.

In the scenario you describe above - you would set your no-reply address to info@school.com.

Average of ratings: -
In reply to Dan Poltawski

Re: Rethinking email delivery from Moodle 3.2

by Michael Aherne -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Dan, this seems sound to me, and much better than the current situation. I take it the no reply address would still be able to be used for certain types of email where no reply is expected, regardless of the whitelisting of the person originating it?

Average of ratings: -