Moodle in English: cron job fails

cron job fails

by George Schaathun - Wednesday, June 10, 2020, 5:02 PM

We suddenly detected that the moodle cron job is failing, and has consistently been failing for three months, which is as far back as the logs go, across at least one upgrade.

The error message is pasted in below. I have been trying to look for stale locks, but I am not sure what I should be looking for. The mdl_lock_db table has 915 entries, I understand, maybe, that's supposed to be empty? If it is safe to close down and delete the contents, I could do that ... I am just not reckless enough to do so before hearing advice ...

Does anyone have any ideas or recommendations?
TIA

Server Time: Wed, 10 Jun 2020 10:52:59 +0200


Execute adhoc task: core_course\task\course_delete_modules
... started 10:52:59. Current memory use 18.8Mb.
!!! Kodefeil er oppdaget, det må fikses av en programmerer: A lock was created but not released at:
[dirroot]/lib/cronlib.php on line 154

 Code should look like:

 $factory = \core\lock\lock_config::get_lock_factory('type');
 $lock = $factory-&gt;get_lock(259206106);
 $lock-&gt;release();  // Locks must ALWAYS be released like this.

 !!!

Re: cron job fails

by Howard Miller - Wednesday, June 10, 2020, 5:50 PM

Usually, that error message is a symptom of the problem - not the actual one. Some other error stops the task completing properly and you get the lock not released error. Make sure you have Debugging on (right up to Developer) before you do anything else.

Average of ratings: -

Re: cron job fails

by Ken Task - Wednesday, June 10, 2020, 9:38 PM

+1 to what Howard says. This 2 cents more.

To avoid ... "suddenly detected that the moodle cron job is failing, and has consistently been failing for three months," - which is bad news ... do this instead ...

Where the directions say to redirect to la-la-land ... redirect to a log file which is overwritten on each cron run.

touch /var/log/moodlecron.log

Then redirect to > /var/log/moodlecron.log

You can then watch in real time: tail -f /var/log/moodlecron.log

Good to be cautious .... nearly all in moodledata (exception filedir) can be and is regenerated by moodle crons. Two methods of possible fix ... manipulate releated moodle DB tables ... never the first option ... or the second is easier.

In your moodledata directory move locks to .locks - the dot in front hides the directory from moodle - like a backup in a way. The locks folder/directory should be re-created on next cron run which you could do manually via code/admin/cli/cron.php

Locks are created to prevent usage of X while moodle acts upon X. They are to be 'erased' (released) when process using them is done. Permissions on moodledata/locks could/would prevent that from happening and one would/might see an error such as the one you are seeing. So make sure ownerships/permissions on moodledata are ok for the user under which your cron runs has access to write (which also means erase) locks.

Problem solved?

IF so, remove the .lock directory from moodledata.

'SoS', Ken

Average of ratings:Useful (1)

Re: cron job fails

by Howard Miller - Wednesday, June 10, 2020, 9:46 PM

You can move the locks into the database quite easily (see config-dist.php). The upshot is that it's much easier to manage. As said, I doubt this is a lock issue.

Average of ratings: -

Re: cron job fails

by Ken Task - Wednesday, June 10, 2020, 10:50 PM

Points back to the issue of updating or upgrading. Don't think I've seen warnings about checking config-dist.php after updating or upgrading. Should be emphasized then.

What I shared was from experience with a users system that had the issue - almost exactly the same as described. No way to do forensics.

As long as OP gets it solved.

'SoS', Ken

Average of ratings: -

Re: cron job fails

by Howard Miller - Thursday, June 11, 2020, 1:48 PM

Well... there's a lot of "stuff" in config-dist.php that you can add to config.php. I would describe much of it as "only if you know what you're doing".

I just mean that if you are having locking issues (or think you might be) - it's a lot easier to monitor a database table than the lock state of files flying around in a directory.

Average of ratings: -

Re: cron job fails

by George Schaathun - Thursday, June 11, 2020, 5:13 PM

Well there is no locks dir under moodledata. What does that mean?

Isn't there a way to release all locks? A stale lock seems to be the only plausible hypothesis, and I was hoping to find a solution simpler than wipe and reinstall ...
The server is only really used in the Autumn semester, so any lock can be assumed to be stale.

Average of ratings: -

Re: cron job fails

by Leon Stringer - Thursday, June 11, 2020, 7:49 PM

If there's no lock directory in Moodledata but there are rows in mdl_lock_db then presumably $CFG->lock_factory is set to some non-default value in config.php probably \core\lock\db_record_lock_factory (or maybe you have PostgreSQL?).

However, I think what Howard and Ken are saying is that the stale locks aren't the issue, it's what's causing these stale locks that needs solving which is presumably a failing task. (I think stale locks will get tidied as they expire). If the same message appears each time, i.e. "Execute adhoc task: core_course\task\course_delete_modules" then that's what we need to look into.

To add to what Ken suggested I think we need to capture the error device (STDERR) too, so redirect with > /var/log/moodlecron.log 2>&1 (that bit on the end capturing STDERR).

You can run the adhoc tasks with admin/tool/task/cli/adhoc_task.php so you could run:

php -f /path/to/moodle/admin/tool/task/cli/adhoc_task.php -- --execute --showdebugging > /tmp/adhoc.log 2>&1

I normally run these as the web server user, apache on my server, so I prefix this with sudo -u apache. Check the site cron's command line (presumably in crontab) to see if you should do this.

The above command should send all output and errors to /tmp/adhoc.log which will hopefully help identify any tasks failing to complete.

If more help is needed, please include any messages from the above log file, the crontab entry (remove/change anything you don't want to share), the $CFG->lock_factory setting (if any), the Moodle version, and the database type (MySQL, MariaDB, PostgreSQL, etc.) and database version.

Average of ratings:Useful (3)

Installing and upgrading help

cron job fails