Slow email after enabling airnotifier service.

Slow email after enabling airnotifier service.

by Trevor Jones -
Number of replies: 4
Picture of Core developers Picture of Plugin developers

Hi All,

I wanted to share our experience we recently had at our University. For context our University runs ~3000 courses per term with ~47,000 daily active users.

The past few weeks  (since the start of term), we noticed our forum email cron was significantly slow and had been taking at times up to 12 hours to process a single forum cron run.

After a lot of digging we found the primary factor was related to our enabling of the push notification service using messages.moodle.net

See the following transaction trace:

Transaction trace

The disabling the push notification service increased the processing speed from approximately from ~2000 notifications/h to ~16000/h 

Has anyone else run into this issue? 

Has there been a degradation in the messages.moodle.net service performance?

What have others done for push notifications?

Average of ratings: -
In reply to Trevor Jones

Re: Slow email after enabling airnotifier service.

by Juan Leyva -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Trevor,

yes, the public Push Notification server has been recently experiencing some issues.

Please see https://tracker.moodle.org/browse/MDL-65970

You can safely apply that patch if you can't upgrade your site. This should mitigate the problem.

Regards, Juan

Average of ratings: -
In reply to Juan Leyva

Re: Slow email after enabling airnotifier service.

by Trevor Jones -
Picture of Core developers Picture of Plugin developers
Hi Juan,

Thanks for this! Good to know about this issue, though I don't think this patch quite resolves the issue for us.
We have peak message sending rate of 4.4 messages/s which works out to about 0.2s per message. In our worse case sending rate where our cron ballooned and could not keep up, we were sending ~0.55 messages/s which is ~1.8s per message. So hard coding the timeout to 2s wouldn't alleviate the issue in the worst case for us. It seems that a proper solution would be to either decouple the push notification from the email sending process or provide a service with some level of guarantees that meets our needed threshold.

To this end we've begun exploring the use of amazon's SQS with Lambda to capture the push messages and provide async processing.
Perhaps a local node server could be used in a similar manner as it doesn't suffer from iowait issues like php does.

In addition, we've found a few more things. For one, we have found many device tokens in the system which are expired or no longer valid. Testing has shown we get 400 and 500 error codes depending on the token, we're not clear yet which scenarios are causing each. But more importantly there doesn't appear to be a cleanup step for tokens/devices that doesn't depend on the individual user (as far as we're aware).
Average of ratings: Useful (1)
In reply to Trevor Jones

Re: Slow email after enabling airnotifier service.

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi,

thank you for these posts.
We experiment the same kind of issue but we're still at Moodle 3.7.1+.
I see that the fix is for Moodle 3.7.3 and I suppose it's already in 3.8.

We've an average of 18.000 emails (up to 26000) send only for forum (annonce/auto subscribe) notifications by day.
And we've a delay of 12h at most before the emails weres send. The delay go back to normal at the morning when the queue is empty.
(delay are incremental of course linked to the queue of waiting notifications)
We plan to put the cron task send notification on a separate server that will only do that.
But I'm still afraid that the queue will be too big to respect the 30 minutes.
If anyone have experience with that can of issue, any advice will be welcome.

This post help us to examine if the airnotifier slow down when you've so much notification by days.

I hope that my explanation is clear...

But I've a question which tool do you use to have the "transaction trace" display here ?

It can really help us in your daily  work.

Thanks in advance.

Dominique.
Average of ratings: Useful (1)
In reply to Dominique Palumbo

Re: Slow email after enabling airnotifier service.

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi,

We're still in Moodle 3.7.1+ but we've desactivate the airnotifier and it solve the problem.

When we upgrade your Moodle we'll reactivate it. And we examine the possibility to install the server for airnotification in-house if performance is still a problem.

Thank you very much all.

Dominique.
Average of ratings: -