Email sent via email_to_user() needs to be queued for background sending

Email sent via email_to_user() needs to be queued for background sending

by Chris Yust -
Number of replies: 7

Hi Everyone,

Previously on assorted forums, the question has been asked "does Moodle queue outgoing emails for background processing", and the answer generally is "no".  Couple of threads for reference:

https://moodle.org/mod/forum/discuss.php?d=367633
https://moodle.org/mod/forum/discuss.php?d=322747#p1484226

We have observed that various Moodle features - such as the messaging system between users, or notifications when a student submits an assignment, end up attempting to send one or more emails synchronously within the user's browser thread via email_to_user() in lib/moodlelib.php.  If the email can be sent out quickly, it's not really a problem.  However, if the target mail server happens to take some time to respond (5, 10, 30, 120 seconds or more!) the user's browser is just completely hung until the process completes.  We have observed this a number of time and fielded complaints from teachers and students.

Best architectural practice in this situation is, of course, to always queue potentially long-running tasks for background execution and not try to complete them in the user's foreground thread.  I would like to propose the following solution - we have implemented this locally in a test environment and it seems to work, but I appreciate any opinions on hidden "gotchas" that are not obvious.

1.  We have created a new Moodle database table (e.g. email_queue) that replicates the essential parameters fed to email_to_user(), plus a couple of other things. 

2. We have modified lib/moodlelib.php to REPLACE the existing email_to_user() function with a version that simply creates a new record in the email_queue table and returns immediately.  The original email_to_user() function was renamed to email_to_user_sync().  Now, by definition, any Moodle process that tries to call the original email_to_user() function will end up queuing the message and returning immediately.

3. We have created a new local plugin that does nothing more than launch a scheduled task every minute.  That scheduled task will read the new email_queue table for un-sent messages.  Upon finding them, it will call email_to_user_sync() to actually send the messages.  There is also some minor tweaking to try and maintain the mailer object persistently, in case that lets us preserve SMTP sessions and reduce connection churn.   We'll also purge the email_queue table of sent messages every couple of days to keep it tidy.

Can anyone spot any major architectural flaws with this approach, such as some feature that demands an email actually be sent by the time email_to_user() returns?

Thanks for your time!


Average of ratings: Useful (1)
In reply to Chris Yust

Re: Email sent via email_to_user() needs to be queued for background sending

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Immediate reaction without reading everything...

As the vast majority of email interaction is via the Forum activity - which *does* queue emails - precisely what real life problem are you fixing here?

Average of ratings: -
In reply to Howard Miller

Re: Email sent via email_to_user() needs to be queued for background sending

by Chris Yust -

Thanks for responding...there are at least two specific scenarios:

1. Teacher sends a message to student or vice-versa, via the "Participants" list when inside a course (/user/index.php).  "With selected users...", then "Send a Message". We have received user complaints and verified user browser hangs in this scenario due to time spent in email_to_user().

2.  Student submits an assignment, which can trigger email notification to student and/or teacher depending on the notification settings of the assignment and the individual users.  We have received user complaints about hanging browsers in this scenario and, while we haven't done the same level of analysis as #1, are reasonably confident it's the same root cause, as the problem goes away when email notification is disabled.

A quick search on the code base shows other core areas where email_to_user() is called, e.g. "/user/edit.php", where the same problem might occur.  I suspect if you added a hard-coded 2-minute delay into email_to_user() on a test system and then exercised those features where email_to_user() was present, browsers would similarly hang in every case.

I can see a number of plug-ins that use the function as well (e.g. Intelliboard).  While not strictly a "core Moodle" problem at that point, if you expose a service that is likely to be abused (and indeed, almost cannot be used correctly by any user-facing task), why not bullet-proof it against known architectural issues?


Average of ratings: -
In reply to Chris Yust

Re: Email sent via email_to_user() needs to be queued for background sending

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Have you considered that whatever mail relay you are using is either (a) slow or (b) has some problem?  

I'm disputing what you are saying but I have run some very large Moodle sites and never had any issues about email sending. However, in all cases we've run our own, local mail relay.

Average of ratings: Useful (1)
In reply to Howard Miller

Re: Email sent via email_to_user() needs to be queued for background sending

by Chris Yust -

We know the mail relay *is* sporadically slow...we've seen it take 2+ minutes to respond to a SMTP connect.  We happen to be using Google (smtp.google.com), but ANY mail relay (or, architecturally, external system that you try to connect to in order to perform a task) is a risk for being slow or unresponsive.  Systems that talk to other systems should understand this and design accordingly.

So, the question is, should Moodle put the user's browser at risk of hanging because, under the covers, it attempts to send a synchronous email based on the user's actions and the target mail relay happens to be slow or temporarily down?

Running your own local mail relay may reduce the risk factor, perhaps significantly enough that you rarely or never see a problem.  But not everyone has a local mail relay, and Moodle (at least superficially) does not require it.





Average of ratings: -
In reply to Chris Yust

Re: Email sent via email_to_user() needs to be queued for background sending

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Chris,

It really does sound like you want Moodle to act as an MTA and that really shouldn't be part of its responsibility. I've never encountered these kinds of issues when using a local MTA, even with much larger sites.

My general recommendation is to always run a local MTA on your server. Personally I'd be more than happy to see the SMTP settings removed from Moodle and require people using a properly configured local MTA instead. A fully and properly configured server should be a pre-requisite. E-mail is part of running a properly configured server.

Adding a queue and task system adds another layer of complexity and I really don't think that it belongs in Moodle.

Generally I'd recommend that we use the right tool for the right job.

Andrew

Average of ratings: Useful (3)
In reply to Andrew Lyons

Re: Email sent via email_to_user() needs to be queued for background sending

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I agree. The MTA doesn't have to be anything fancy. In fact I would caution against trying to deliver mail to its final destination. That's a whole bunch of hassle. But just passing it on to the next hop (a "smart host") is a good idea. 

Of course, for non technical users this would probably be big ask. Especially on Windows. However, I don't get the impression this would be beyond the scope of the OP. 

Average of ratings: -
In reply to Howard Miller

Re: Email sent via email_to_user() needs to be queued for background sending

by Chris Yust -

Hi Everyone,

Thanks for your thoughtful responses!  Since the general consensus is that a locally installed mail relay with quick response times is required for smooth functioning of all Moodle servers, we'll look at adding one into our environment.

Average of ratings: Useful (1)