Running cron from multiple webservers

Running cron from multiple webservers

by Christos Savva -
Number of replies: 9

Hello Moodlers smile,

I know this was asked before, but since there was no answer I though I will give it another try

Other Posts about this:

https://moodle.org/mod/forum/discuss.php?d=358434

https://moodle.org/mod/forum/discuss.php?d=344181


Based on the latest Moodle Docs, we can now run the cron on multiple webservers (link).

Today I tried to do just that, while the cron was running on node1 I tried to run the cron on node 2.

Node 1 finished the cron ok, node 2 gave an error (below) and stopped processing the cron and exited


The error

Scheduled task complete: Forum mailings and maintenance jobs (mod_forum\task\cron_task)

Execute adhoc task: core\task\refresh_mod_calendar_events_task

... started 14:11:23. Current memory use 27.3MB.

Refreshing events for assign

Default exception handler: Coding error detected, it must be fixed by a programmer: A lock was created but not released at:

[dirroot]/lib/classes/task/manager.php on line 448

 Code should look like:

 $factory = \core\lock\lock_config::get_lock_factory('type');

 $lock = $factory->get_lock(Resource id #854);

 $lock->release();  // Locks must ALWAYS be released like this.


 Debug:

Error code: codingerror

* line 117 of /lib/classes/lock/lock.php: coding_exception thrown

* line 91 of /lib/cronlib.php: call to core\lock\lock->__destruct()

* line 61 of /admin/cli/cron.php: call to cron_run()


!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:

[dirroot]/lib/classes/task/manager.php on line 448

 Code should look like:


 $factory = \core\lock\lock_config::get_lock_factory('type');

 $lock = $factory->get_lock(Resource id #854);

 $lock->release();  // Locks must ALWAYS be released like this.

!!!



So is it possible to run the cron on different webservers for the same Moodle instance or not?

Since cron found that there was a lock already in place for that task, why the cron didnt continue with the next task and it exited?


Kind regards

Christos

Average of ratings: -
In reply to Christos Savva

Re: Running cron from multiple webservers

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Christos,
that's a good question: AFAIK everyone running a cluster is used to have a node not serving web requests but running Moodle cron.

That being said, the lock is there to preserve multiple nodes to perform the same operation with unpredictable results when overlapping on the same operation: I never thought that it was been designed to allow skipping a locked task and passing over the next.

What is your Moodle version?

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron from multiple webservers

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Matteo,

It is now possible to run cron on multiple nodes simultaneously.

The locking system is designed to lock individual tasks and if a task is locked, it will fail to obtain the lock and move on to the next task.

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Running cron from multiple webservers

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

TNX Andrew for stepping into the discussion Yes !

Guessing we're talking about MDL-55980, available on 3.3+: https://docs.moodle.org/dev/Moodle_3.3_release_notes#For_administrators .

Edit: I was wrong, MDL-25499 including several minor issues being resolved in the latest 2.7 (https://docs.moodle.org/dev/Moodle_2.7_release_notes#Cron).
MDL-51707 looks like relevant too: fixed in 2.8.9, 2.9.3 and above.

HTH,
Matteo

In reply to Christos Savva

Re: Running cron from multiple webservers

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Christos,

It is possible to run Cron on multiple servers simultaneously.

This lock messages is unlikely related to running on multiple servers, and more likely due to a bug somewhere in one of the cron tasks which has meant that the task failed (maybe is threw an Exception, or experienced a Fatal error). As a result, the thread was ended and the lock was not released via the normal processes. There is a shutdown handler which runs during the PHP exit which is still called. One of the tasks it performs is to release locks. If it finds any locks to release, it warns because they have not been closed properly for some reason.

It looks like there is a bug of some kind in mod_assign which prevents the refreshing of the calendar events there.

We have made improvements to the way in which this works, including better diagnostics in more recent Moodle versions.

Which version of Moodle are you using? Have you ensured that you have the same version of PHP installed on all nodes?

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Running cron from multiple webservers

by Christos Savva -

Hi Andrew and thanks for your reply

My Moodle version is 3.3.2+ (Build: 20170915)

And yes the 2 nodes (web servers) are exactly the same (created the second instance from Image of the first).

Could you please let me know how I will identify the error and maybe fix it?

Kind regards

Christos

In reply to Christos Savva

Re: Running cron from multiple webservers

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

I'd recommend enabling the debugging and debugdisplay. You can also run a specific scheduled task (admin/tool/task/cli/schedule_task.php --execute="\core\task\refresh_mod_calendar_events_task").

Hopefully this should give a stacktrace and some more debugging information.

In reply to Andrew Lyons

Re: Running cron from multiple webservers

by Christos Savva -

I executed


sudo -uapache /usr/bin/php task/cli/schedule_task.php --execute="\core\task\refresh_mod_calendar_events_task"

And I got this

Task '\core\task\refresh_mod_calendar_events_task' not found

I then checked /admin/tool/task/scheduledtasks.php

and indeed there is no such task :/


In reply to Andrew Lyons

Re: Running cron from multiple webservers

by Christos Savva -

I just tried again to run the cron on both nodes. This time both competed.


Node 1 gave this output

Scheduled task complete: Background processing for question engine (core\task\question_cron_task)

Execute adhoc task: core\task\refresh_mod_calendar_events_task

... started 10:10:48. Current memory use 44.5MB.

Refreshing events for assign

Refreshing events for assignment

Refreshing events for book

Refreshing events for bootstrapelements

Refreshing events for chat

Refreshing events for choice

Refreshing events for choicegroup

Refreshing events for data

Refreshing events for feedback

Refreshing events for folder

Refreshing events for forum

Refreshing events for glossary

Refreshing events for imscp

Refreshing events for label

Refreshing events for lesson

Refreshing events for lti

Refreshing events for page

Refreshing events for quiz

Refreshing events for resource

Refreshing events for scorm

Refreshing events for survey

Refreshing events for thesisproposition

Refreshing events for turnitintooltwo

Refreshing events for url

Refreshing events for wiki

Refreshing events for workshop


Node 2 didn't execute: core\task\refresh_mod_calendar_events_task at all