I thought it might have had something to do with this code:
/// Run all core cron jobs, but not every time since they aren't too important.
/// These don't have a timer to reduce load, so we'll use a random number
/// to randomly choose the percentage of times we should run these jobs.
srand ((double) microtime() * 10000000);
$random100 = rand(0,100);
if ($random100 < 20) { // Approximately 20% of the time.
mtrace("Running clean-up tasks...");
But it's not happening anything like 20% of the time
Have you compared the output of the long running cron instances to the output of other instances? The differences in the output might reveal what is causing the cron instances in question to take longer to complete.
Hi Adam,
Thanks for your contribution. Appreciated.
I've separated out about 10 cron outputs that run normally Vs half a dozen that run for 2 hours. Then I ran file differences between all of them. Every time I thought one task might be causing the delay it existed in one file but not another. There wasn't any consistency of long running crons vs not long running crons.
I can confirm that course backups are not run via cron and that the clean-up tasks that run 20% are _not_ the cause.
It's very suspicious how close the duration in minutes of these tasks. In the last 12 hours there have been 5 of them. 4 ran between 120 and 121 minutes. 1 ran for 122 minutes.
It's either:
- A process in cron.php that occasionally runs and always runs on a very similar data set (hence the similar completion time) but doesn't trace its presence in the cron output.
- A server environment issue
Further information
MySQL is not locking.
No crontabs exist for any other users of the server
The commands are being run with flock to prevent concurrent crons
The commands are being run with nice to run at a lower priority
Does anyone know if Redhat (CentOS) priority scheduling like... ramps up the priority if it reaches a maximum (say... 120 mintes?)
-David
I haven't used CentOS or RHEL recently, but when I did use both, there wasn't anything that dynamically adjusted process priorities as you are describing, and I doubt anything like that would have been added.
Do you have system level monitoring in place so that you can review what is happening on the system during the long cron runs? There might be something else running on the system at the same time as cron that explains the long cron run times, especially if you have the priority on the cron processes set low. You could also try running cron for a few days without nice to see if the issue is related to process priorities and competing for CPU time.
Thanks Adam,
That's a great idea. I'll run the cron without nice for a few days and see what happens
