Are cron.php and adhoc_task.php capable of parallelizing?

Are cron.php and adhoc_task.php capable of parallelizing?

by Visvanath Ratnaweera -
Number of replies: 5
Picture of Particularly helpful Moodlers Picture of Translators

We have a Scheduled task concurrency limit (task_scheduled_concurrency_limit) and an Ad hoc task concurrency limit (task_adhoc_concurrency_limit), which may be set to numbers higher than 1. The default being 3.

Are cron.php and adhoc_task.php really parallelizable? Or as https://docs.moodle.org/en/Cron#Scaling_up_cron_with_multiple_processes says, those scripts need to be invoked that many times to create those parallel tasks?

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Are cron.php and adhoc_task.php capable of parallelizing?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Yes - if you set off several long running ad-hoc tasks (e.g. restore a big course) then they all "run" at the same time up to the selected concurrency limit. Find a big course, click the 'copy' function a few times and watch it in action.
Average of ratings: Useful (1)
In reply to Howard Miller

Re: Are cron.php and adhoc_task.php capable of parallelizing?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Howard

Yes, I can understand the use of task_adhoc_concurrency_limit in this situation: if users request (interactively) ad-hoc tasks concurrently, up to the selected concurrency limit they will run concurrently.

Does it also mean admin/cli/adhoc_task.php too would spawn that many ad-hoc tasks, if required?

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Are cron.php and adhoc_task.php capable of parallelizing?

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Each CLI task can only run one item in parallel at a tiem. If you run an additional CLI runner it will take another parallel thread, but will still respect the task_adhoc_concurrency_limit.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Are cron.php and adhoc_task.php capable of parallelizing?

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi Visvanath,

Each individual task, and each cron runner is not parallelised, but the cron system as a whole is. If you have multiple runners then they will not block one another.

The task_scheduled_concurrency_limit setting allows a larger site to ensure that adhoc tasks are given some priority, whilst the task_adhoc_concurrency_limit setting ensures that any heavy adhoc tasks do not prevent scheduled tasks from running.

It does not make sense to run more than one copy of a single adhoc task so these cannot be parallelised, and scheduled tasks use a lock to ensure that each task is only run once (not in parallel with itself).
Average of ratings: Useful (4)
In reply to Andrew Lyons

Re: Are cron.php and adhoc_task.php capable of parallelizing?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hello Andrew

Thanks for providing the information. I read a couple of your other posts including https://moodle.org/mod/forum/discuss.php?d=417100#p1680821 and https://moodle.org/mod/forum/discuss.php?d=417100#p1680824 and also the docs https://docs.moodle.org/311/en/Scheduled_tasks, https://docs.moodle.org/311/en/Administration_via_command_line#Scheduled_tasks, https://docs.moodle.org/311/en/Cron#Low_latency_adhoc_tasks. I can't say, I am a step closer to understanding the mechanism, it must be somebody's (yours?) doctoral thesis.
smile

Very specific to my original question, I am interested in understanding the differences amoung having two lines like this in the crontab:
* * * * * /usr/bin/php /path/to/moodle/admin/cli/adhoc_task.php --execute --keep-alive=59
* * * * * /usr/bin/php /path/to/moodle/admin/cli/adhoc_task.php --execute --keep-alive=59

compared to having only one such line,
* * * * * /usr/bin/php /path/to/moodle/admin/cli/adhoc_task.php --execute --keep-alive=59

and not calling adhoc_task.php at all from crontab at all, i.e. crontab just has the single line:
* * * * * /usr/bin/php /path/to/moodle/admin/cli/cron.php --execute --keep-alive=59
Average of ratings: -