I have a scheduled task running every 5 minutes in an environment where multiple cron jobs are running. I'd like to restrict the scheduled task to run only one cron instance, as we are getting errors because things are being done twice. I'm looking into the Lock API, however, I'm not sure that's what I need. Any insight would be appreciated!
I have tried enabling blocking, but since two cron processes are running, locking the cron doesn't work for me. Is there a setting in Moodle to tell moodle two separate cron instances are running the scheduled tasks? Should we set up our two server cron tasks to stagger their run time (i.e. have one run at mm:00 and the other run at mm:30? I'm really confused as to why a scheduled task could possibly run on two separate crons at the exact same time (down to the second).
Could you explain more fully the environment where multiple cron jobs are running? Is that to say you have a single host, where there are multiple entries in crontab that run the admin/cli/cron.php script? Or that you have two or more hosts connected to the same Moodle db (e.g. web farm), each running one cron task for cron.php at the same time?
You may have seen from your coding already, that unless you specify a lock factory via configuration, or provide one for your db family (postgres appears to be the only one), the lock factory defaults to file_lock_factory.php. If you're arrangement is with multiple hosts, then task locks on different hosts are not observed.
But if your arrangement is the former, one machine, many crontab entries, I would need to look further; but, then, would staggering the crontab entries by a minute or so resolve the issue?
We have two hosts connected to the same Moodle db, each running one cron task for cron.php at the same time. If task locks on multiple hosts aren't observed, how do we circumvent this issue?
As I think about it, I may be wrong altogether about locks being observed between multiple hosts. It depends on how you're implementing a shared moodledata directory, in which there is a lock subdirectory where the file locks are placed. NFS shares are pretty common, and that's how we share the file system data between many VMs. But to avoid the problem, we run cron tasks on only one of the VMs.
That you're having an issue seems to indicate that your shared file system is not (or does not support) locking files, because of its configuration, or limitations (NFS v3 vs. v4). The next step might be to have your sysadmins verify the file sharing clients on the hosts (assume NFS) are configured correctly (/etc/fstab), and that the NFS server is configured to allow file locking.
If it turns out you can't resolve the file locking issue, then, and I have not explored this option, you can try using the db_record_lock_factory. To implement it, $CFG->lock_factory is assigned the value of the classname to use, in this case 'db_record_lock_factory', easiest to do in config.php I suppose. Check the docs, or review the code in /lib/classes/lock, specifically lock_config.php, and db_record_lock_factory.php.
Can I suggest that instead of running cron on both hosts, just run it on one of them? It would simplify things. Yes, it might take more time to finish having one host do the cron tasks, but you'll only have one set of logs to monitor and collect, and you'll avoid the headache you're experiencing now. Only one of the hosts needs to run the cron task after all.
Thanks for your suggestions! The scheduled task that I'm running doesn't perform any file references, but it does do database calls. I don't want to lock the database tables, as we may have students accessing them while the scheduled task is running.
Unfortunately, I am not the administrator of our Moodle environment. We are using a GPFS file system between two nodes. The documentation actually does specify that you should run cron on both servers, and that the processes should lock to avoid running them at the same time. (https://docs.moodle.org/36/en/Cron#Running_cron_for_several_Moodle_servers). But the question is, how do I lock the task? In the TASK API documentation, it states that if blocking is set to 1, no other scheduled task will run at the same time. I have tried this, but it isn't stopping the second cron from running. If you look at /lib/classes/task/manager.php lines 478-483, you can see that there is logic to set up a lock for the task, however, it is using the cron lock factory. This seems to only work on one instance of cron. Is there some setup option where we can specify that we have two servers running cron tasks, so that the task lock is referenced by both crons? Or should we schedule our cron processes to run staggered from one another?
Reading the MoodleDoc you referenced, it states "Tasks can run in parallel..." not "should" or "must"... they just left the door open, so to speak, so you could. But when in a multi-host environment, config-dist.php notes that the lock sub directory must be someplace shared, and it must support filelocking.
The issue is that you have two hosts trying to run the same task at the same time. Let's go ahead and assume that either the GPFS client or the GPFS distributed lock manager is misconfigured--either way, you currently can not use file locking for synchronizing task execution.
Solution option 1: fix your GPFS environment so file locking will work.
Solution option 2: run cron tasks only on one of the hosts. All of your tasks will be taken care of by the one host. You need not worry that half of them will be skipped, or that house cleaning will be overlooked on the one server not running the cron task. Only one of the hosts needs to run the Moodle cron.php periodically.
Solution option 3: use the db_record_lock_factory instead. It'll be slower, but by how much is anybody's guess. Might turn out to be just fine.
The question you keep asking, "how do I lock the task?" is not the right question. The task is being locked. It's being locked on host A and seen on host A. It's being locked on host B and seen on host B. The question should be, "how do I lock the task so it is seen on both hosts, or however many hosts I have?"
The answer is to either fix your file locking problem, circumvent the problem altogether by running the Moodle cron,php on only one machine, or use the database for task locks. Option 2 is the simplest option, and carries the least amount of risk, near zero best I can tell.
The task "blocking" setting just keeps the manager from going to the next available task on the same host. Again, won't stop each host from running your problematic task.
Hi Fred,
I jumped into this thread as I am trying to figure out the scheduled tasks as well. Do most scheduled tasks get controlled by going to mdl_task_scheduled? Does it follow this logic flow?
- Cron checks to see if the nextruntime is less than the current time.
- If yes, update the nextruntime and lastruntime.
- Run the scheduled task
If we have greater than 1 server running cron, the cron job should still try to update the nextruntime and lastruntime. if two servers hit that task at the same second whomever got to lock those two fields first gets to run the job right?
Yes, the table you mentioned, mdl_task_scheduled, is used.
Looking in /lib/classes/task/manager.php at the get_next_scheduled_task() function, the query used to fetch the scheduled task candidates looks for lastruntime is null or past AND nextruntime is null or past AND enabled. This function, despite it fetching multiple rows, returns only a single task though. The rows are ordered so the task with the oldest lastruntime is first.
But the one task returned to the caller, a while loop in cron_run(), will be the first one for which get_next_scheduled_task() can acquire a task lock (a file lock given the file_lock_factory is used by default).
That one task is then returned to the while loop in cron_run(), and it is executed by passing it to cron_run_inner_scheduled_task(). When done, the task lock is removed.
So, when two or more hosts are processing scheduled tasks at the same time, and assuming all hosts detect file locks in the shared $CFG->dataroot/locks directory, they begin iterating over the rows of potential tasks (ones needing to be run), and use the file locks as the way of saying before any updates are made to the task in the DB, "MINE! I HAVE THIS ONE!" It's one of the basic IPC mechanisms--like a semaphore or mutex. All the other hosts that fail to get the file lock for a given task just go on to the next candidate in the list (of rows returned by the query).
Once a host (or process in the case of multiple cron tasks running on a single host) has the task lock established, then it can go about doing the work of the task and updating the database row associated with that task.
Hope all that is useful somehow. But it explains why, in a multi-host environment, it's easiest to designate one host to handle the Moodle cron task. Even then, because any task could take longer than the interval between cron runs, the likelihood of overlapping cron runs is high, and the file lock will prevent the current run from stepping on an already running tasks.