Multiple email messages - Moodle 3.7 - mdl_notifications

Multiple email messages - Moodle 3.7 - mdl_notifications

by Paulo Cerqueira -
Number of replies: 28

Hello,

After upgrade to Moodle 3.7, we noticed that forum messages appear multiple times (email). 

Server: Debian 9.9, Mysql: 5.7.26, PHP: 7.3.

Crontab is running under user user www-data: 

*/1 * * * *        /usr/bin/php /var/www/html/moodle/admin/cli/cron.php > /var/log/cronmoodle2.log 2>&1


Any ideas ?

TIA



Attachment md_notifications.png
Average of ratings: -
In reply to Paulo Cerqueira

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Jon Bolton -
Picture of Particularly helpful Moodlers Picture of Testers
I might be missing something obvious, but they’re NOT the same. The first one you’ve highlighted is 478800 and the second one is 478807 🤔
Average of ratings: Useful (1)
In reply to Jon Bolton

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Paulo Cerqueira -

They haver the same postid and content. This situation is really strange, because a lot of users are complaining about multiple (3x, 4x) messages.

I tried in one of my foruns. Posted once, three messages received in a 12 hours interval.

Thank anyway, this thing is getting me crazy.


Regards,


Paulo Cerqueira 


Average of ratings: -
In reply to Paulo Cerqueira

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Jon Bolton -
Picture of Particularly helpful Moodlers Picture of Testers
But in the two you've highlighted, then useridfrom and useridto is different for each.
First one is from userid 4388 to userid 3763 and the second is from userid 4388 to userid 3120.

I'm not saying there isn't a problem, just that the screenshot isn't helping to diagnose it.
Average of ratings: Useful (1)
In reply to Jon Bolton

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Paulo Cerqueira -
You are correct! The screenshoot is somehow vague. Please, take a look at these other two. You can see that only one message has been posted. The other screenshot, is my inbox. You can see that the same message has been sent multiple times.

Last semester, Moodle 3.5 was a charm with messages, no multiple messages. We´ve upgraded to 3.7, no errors, everything looked ok, but, after a LOCATE change in the server (just that, we changed the server language from en-us to PT-BR), we noticed this strange behavior. I can´t see a relation between the server language (Debian 9.9) and multiple email messages. I think it was just a coincidence. Just posting more information about the problem...

Obs: English is not my first language, please disregard any grammatical errors.

TIA

Paulo Cerqueira

original post - forum

my inbox
Average of ratings: -
In reply to Paulo Cerqueira

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Jon Bolton -
Picture of Particularly helpful Moodlers Picture of Testers
What (specific) version of 3.7 are you running? You can find the “build number” in Site administration > Notifications

There was a similar problem on when 3.7 was released but it was fixed in v3.7.1 - see MDL-65655
Average of ratings: Useful (1)
In reply to Jon Bolton

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Paulo Cerqueira -
Moodle 3.7.1 (Build: 20190708). I think it´s the newest build.
Average of ratings: -
In reply to Paulo Cerqueira

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Jon Bolton -
Picture of Particularly helpful Moodlers Picture of Testers
Have you checked that your cron settings are correct on the server?
Did you change the timezone as well as the locale?
Average of ratings: Useful (1)
In reply to Jon Bolton

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Paulo Cerqueira -
crontab -l (www-data user):
*/1 * * * * /usr/bin/php /var/www/html/moodle/admin/cli/cron.php > /var/log/cronmoodle.log 2>&1

timezone is ok (GMT-3)

date output: qua ago 21 17:04:25 -03 2019



but, looking the output of the cron.php in /var/log/cronmoodle2.log, I´ve found this error:

 tail /var/log/cronmoodle2.log

Marked 15 posts as read
... used 185 dbqueries
... used 56,665079116821 seconds
Adhoc task complete: mod_forum\task\send_user_notifications
... used 186 dbqueries
... used 56,665494918823 seconds
Adhoc task failed: mod_forum\task\send_user_notifications,Erro ao gravar na base de dados (Erro ao gravar na base de dados = "Error writing database")
Execute adhoc task: mod_forum\task\send_user_notifications
... started 17:07:11. Current memory use 39,7Mb.
Sending messages to 11910500 (5278)


BTW: no errors in /var/log/mysql/error.log, just a bunch of login failures from the Internet

Average of ratings: -
In reply to Paulo Cerqueira

Re: Multiple email messages - Moodle 3.7 - mdl_notifications

by Paulo Cerqueira -
Another output from /var/log/cronmoodle2.log:

tail /var/log/cronmoodle2.log
... used 173 dbqueries
... used 60,872830867767 seconds
Adhoc task complete: mod_forum\task\send_user_notifications
... used 174 dbqueries
... used 60,873253822327 seconds
Adhoc task failed: mod_forum\task\send_user_notifications,Erro ao gravar na base de dados
Stopping processing of adhoc tasks as time limit has been reached.
Cron script completed correctly
Cron completed at 17:27:02. Memory used 40,7Mb.
Execution took 1860,764983 seconds
root@eadgrad:/var/www/html/moodle/mod/forum/classes/task# tail /var/log/cronmoodle2.log
Post 21012 sent
Post 21013 sent
Post 21018 sent
Post 21025 sent
Post 21034 sent
Post 21035 sent
Post 21069 sent
Post 21086 sent
Post 21088 sent
Post 21089 sent
Average of ratings: -
In reply to Paulo Cerqueira

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators
I have exactly the same problem.
Ubuntu 16.04, php7.2, mysql5.7, Moodle 3.7.1+ (Build: 20190824)

Forum posts are getting send out until I delete them.
mdl_notifications gets filled up with a line for every repost of the forum message.
In mdl_forum_post is marked as mailed (mailed = 1)
In the logs I only see it been send once, without an error message (see log at the bottom of my post
Due to the holidays, the server has not been used for two months. I have upgraded from 3.6 latest to the current latest version and it started happening with the first post made on the server (the second and the tirth show the same problem). Changes are the upgrade of Moodle, upgrade of the php version (7.0 to 7.2) and adding of the tool https://moodle.org/plugins/tool_cohortdatabase

I noticed https://tracker.moodle.org/browse/MDL-65888, but I can't find any error messages in Moodle logs.


(Execute scheduled task: Forum-mails en onderhoudstaken (mod_forum\task\cron_task)
... started 21:30:04. Current memory use 31.4MB.
Removing old digest records from 7 days ago.
Removed all old digest records.
Fetching unmailed posts.
Done
Processing post information
Processed 1 posts
Filling caches
Filling course cache
Done
Filling forum cache
Done
Filling discussion cache
Done
Filling user subscription cache
Done
Filling digest cache
Done
All caches filled
Queueing user tasks.
Processing 1 users
Queued 0 digests and 1 messages for 2
Queued 0 digests, and 1 individual tasks for 1 post mails. Unique users: 1 (0 ignored)
All tasks queued.
... used 15 dbqueries
... used 0.037278890609741 seconds
Scheduled task complete: Forum-mails en onderhoudstaken (mod_forum\task\cron_task)
Average of ratings: Useful (1)
In reply to koen roggemans

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators
So my test message got sent 7 times over a time period of 70 minutes.
The intermediate time increased: 3 minutes, 5, 9, 17, 33 - so more or less double.
The two previous messages got sent out to 150 people, so I deleted them when I noticed the repeated sending and that stopped the sending as well.
Average of ratings: Useful (1)
In reply to koen roggemans

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

Sorry for your troubles sad

Can you try and run cron manually from the command line, or otherwise have its output captured. It should show there was an error while processing the queue of notifications. IIRC, there is a bug/regression in the 3.7 forum code that causes the notifications to be scheduled for emailing again in case of unexpected error.

Average of ratings: Useful (1)
In reply to David Mudrák

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators
hm. May be I should create a post first and wait until it starts sending ...
Server Time: Fri, 30 Aug 2019 10:18:39 +0200


Cron script completed correctly
Cron completed at 10:18:39. Memory used 15.1MB.
Execution took 0.089769 seconds
Average of ratings: Useful (1)
In reply to koen roggemans

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators
Aha - error writing to database, but what did it try to write and where? And why did it fail?

Adhoc task complete: mod_forum\task\send_user_notifications
... used 25 dbqueries
... used 0,087355136871338 seconds
Adhoc task failed: mod_forum\task\send_user_notifications,Fout bij het schrijven van de databank
Cron script completed correctly
Cron completed at 10:55:04. Memory used 40,2MB.
Execution took 2,277948 seconds
Average of ratings: Useful (1)
In reply to koen roggemans

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators
At the moment I'm trimming the logs. I noticed the server is set to "keep all logs" (which is the default setting) and that they contain 12 million rows. May be that is the write error?
Average of ratings: Useful (1)
In reply to koen roggemans

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Paulo Cerqueira -
An important update about this bug - I think it is (I don't know why) related with the server LOCALE. My problems started AFTER I changed the LOCALE to pt_BR. Before, everything was working fine, then the duplicated messages began. Well, I decide to rollback the LOCALE. Guess what ? No more duplicates and database error !

I just can´t understand what the LOCALE has to do with this issue, but it's related. No problem with en_US.UTF-8 UTF-8.
Average of ratings: Useful (3)
In reply to Paulo Cerqueira

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Camilo Ravelo Durán -
hi, where specifically do i make the change? I have the same problem. thank you
Average of ratings: -
In reply to Paulo Cerqueira

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Erick de Azevedo Lima -

Hello everyone!

Yesterday I've updated the bug MDL-66561 with some additional info I found when debugging. And yes, the locale causes the trouble. I've tracked the problem and found that it is located in the Moodle MySQL database component that performs an ugly hack when building the database query (it is documented in the code as an ugly hack, these are not my words). It converts a floating point numbers (in this particular case, the timestart and timeend fields are the culprits) to string. When doing so, the PHP internal engine uses the locale to perform the conversion and... it exchanges the decimal separator dot (.) for comma (because here in Brazil we use comma as the decimal separator), making the MySQL think that the decimal part of the number is another value. In this case, the database query end up with 12 values x 10 database columns and does not know what to do, throwing an error. This bug will only occur when the decimal separator is other than dot, because MySQL only accepts dot as decimal separator in raw database queries. I've suggested where the Moodle devs can make the correction, because I don't know the worflow for contributing with a source code change. Anyway, now we do know a root cause for this problem and the suggested workaround is all we can do while waiting for a patch. Sorry for all the tech talk, but I could not find better words to explain. If there are any additional doubts, just let me know.

Average of ratings: Useful (3)
In reply to Erick de Azevedo Lima

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Brian Caballero -

Hi @Erick,

I had the same issue with multiple notifications.  I checked this thread and followed the fix on LOCALE settings.   I changed it from `C.UTF-8` to `en_US.UTF-8"`.  But I'm still having same issues.  What other server configs do I need to change? Thank you.

Average of ratings: -
In reply to Brian Caballero

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Erick de Azevedo Lima -
Hello, Brian. Sorry for the delay in my response. I got a lot of e-mails and the one containing the notification from your post just "got buried" in my mailbox. I only noticed your mention reading the post here in the moodle.org site. In fact, I could not change server settings for my Moodle sites. What I've done in each one of them was disabling the logging for tasks. The error occurs during the log operation, so I disabled this task logging it and will keep it disable until the Moodle team apply a patch correcting the real bug, that is on the database layer and not on logging/task. I documented a patch on the bug tracker on January 14th, because the bug is known and already had a "bug number", but, until this moment, I had no reply at all . And the saddest part is: Even it is a little piece of code, I think the patch will take a long time until it gets applied to stable Moodle, because this bug only affects sites from that uses locales that does not use the dot(.) as a decimal separator. Sad, but there's nothing more I can do.
Average of ratings: Useful (1)
In reply to Erick de Azevedo Lima

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Brian Caballero -
Thank you for the response Erick. Another thing i noticed related to the issue might be our Load Balancing. Locally I manage to replicate the multiple notification issue by executing two or more cron at the same time. The result depends on the number of subscribers within the forum.

On our case, all of our front-end executes cron. There are chances that two or more process run at the same time thus causing this error. This now leads me to question this part of moodle doc recommendation:
https://docs.moodle.org/310/en/Cron#Running_cron_for_several_Moodle_servers

I'm now trying to look into this ticket for more info on how to set up a dedicated cron server for Moodle:
https://moodle.org/mod/forum/discuss.php?d=127108
Average of ratings: -
In reply to Brian Caballero

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Camilo Ravelo Durán -
Hello Brian,
I don't think it is the load balancer. I have the same problem in a moodle with a load balancer and another moodle without a load balancer.
Average of ratings: -
In reply to Erick de Azevedo Lima

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Camilo Ravelo Durán -
Thanks Erick for the answer, it's good to have an idea where the problem is coming from. In my case our moodle works with MICROSOFT SQL SERVER. I had not seen the comments because my solution was to block the emails in the forum activity, but I would like to activate them since they are useful. Unfortunately we do not have a solution for this problem.
Average of ratings: -
In reply to Camilo Ravelo Durán

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Anastasia Leyva -
Hi Camilo,

Have you found the solution to this issue? I am having the same problem with Moodle 3.9 and changing the LOCALE settings hasn’t fixed the issue. This is only happening for forum notifications no other emailing services (quickmail, open forum, messages…)
Any luck with this?
Average of ratings: -
In reply to Anastasia Leyva

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators
If you notice this but in Microsoft SQL server too, then it should be mentioned in the bug rapport MDL-66561 because there is nothing about MSSQL there yet, only Mysql.

As mentioned in the bug rapport: if you can't get it fixed, you can always switch off logging as a workaround on moodle/admin/settings.php?section=tasklogging
Average of ratings: -
In reply to Anastasia Leyva

Re: Duplicate forum notification emails being sent - Moodle 3.7.1

by Camilo Ravelo Durán -
Hi Anastasia,

Unfortunately we still have not found a solution for the problem. We also have moodle 3.9 and we use sql server.
Average of ratings: -