The operation timed out while waiting for a lock

The operation timed out while waiting for a lock

by Steven Swanepoel -
Number of replies: 12

Hi,

I'm running Moodle 2.8.9 (Build: 20151109).

Our cron job isn't running automatically anymore. When I manually try and run it via the web interface /.../cron.php I get this error: 

!!! The operation timed out while waiting for a lock. !!! 

How would I fix it? I've been in contact with my hosting provider for the past couple of days, who is not proving useful at all. They've asked to create a new cron job script for .../admin/cli/cron.php .

However this is not running automatically. They've been able to run it via shell manually.

I've googled it and found that there are trackers for it but no solution. I just don't know what else to do...


Cheers,

Steven

Average of ratings: -
In reply to Steven Swanepoel

Re: The operation timed out while waiting for a lock

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

That's not an error - well it probably isn't. 

It just means that (when you try to run cron manually) cron is already running somewhere else. I actually submitted a tracker report some time ago complaining that message isn't very helpful. 

In reply to Steven Swanepoel

Re: The operation timed out while waiting for a lock

by Damyon Wiese -
Actually there probably is an error. I would not expect you to see this message ever, unless something is really "stuck". The lock is only used when calculating which scheduled task to run next. This should happen really quickly. What could be happening, is that your cron jobs are crashing, or being terminated - and are not automatically cleaning up their locks.

Moodle supports different lock factories - and the best one depends on your database/OS etc.

Here are the docs about lock factories from config-dist.php:
// Moodle 2.7 introduces a locking api for critical tasks (e.g. cron).
// The default locking system to use is DB locking for Postgres, and file locking for
// MySQL, Oracle and SQLServer. If $CFG->preventfilelocking is set, then the default
// will always be DB locking. It can be manually set to one of the lock
// factory classes listed below, or one of your own custom classes implementing the
// \core\lock\lock_factory interface.
//
// $CFG->lock_factory = "auto";
//
// The list of available lock factories is:
//
// "\\core\\lock\\file_lock_factory" - File locking
// Uses lock files stored by default in the dataroot. Whether this
// works on clusters depends on the file system used for the dataroot.
//
// "\\core\\lock\\db_record_lock_factory" - DB locking based on table rows.
//
// "\\core\\lock\\postgres_lock_factory" - DB locking based on postgres advisory locks.
//
// Settings used by the lock factories
//
// Location for lock files used by the File locking factory. This must exist
// on a shared file system that supports locking.
// $CFG->lock_file_root = $CFG->dataroot . '/lock';


I would recommend you change to a different supported lock_factory if you are having trouble with your current one.

Average of ratings: Useful (1)
In reply to Damyon Wiese

Re: The operation timed out while waiting for a lock

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I beg to differ (at least about the symptoms) wink

I see it all the time on real site (admittedly big ones) when attempting to run the cron manually from the command line. Probably as much as 75% of attempts. 

In fact, I just picked a site at random and ran the cron through the web interface (I have access to it on this site)...

First go...  (and there's nothing unusual about this site)

I have to admit that I don't really have a clue what this lock code is trying to do. I assumed it was simply preventing the same scheduler processes running at the same time (or at least part of it - like the zip file example on that page) but I guess not (most/many scheduler processes are far from fast). I read https://docs.moodle.org/dev/Lock_API but I'm no better off.  Off topic I know - but just while it was in my head!




In reply to Howard Miller

Re: The operation timed out while waiting for a lock

by Steven Swanepoel -

Damyon, this is my current setup:

Apache Version: 2.4.16

PHP Version: 5.4.44

MySQL Version: 5.5.42-cll

Architecture: x86_64

Operating System: linux


I noticed on my searches that you created a local lock factory plugin. Would this be the best bet to change my lock factory? If so can you give me specific details how to do this? I've had a look at the read me in the plugin and I'm none the wiser about what to do next.
I still confused as to how cron can be run from command line (/cli/cron.php) but not from the web interface (/admin/cron.php)? Surely if the was some kind of critical lock both would be affected?


Howard, cron hadn't run for over a week before it was picked up. Our dashboard depends entirely on cron running and users alerted me to the fact that stuff wasn't showing up as it should. I don't think in my instance the lock was from manual intervention.

In reply to Steven Swanepoel

Re: The operation timed out while waiting for a lock

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm not suggesting that manual intervention is the cause. I would assume that if manual intervention causes this then so do 'normal' cron runs.

In reply to Howard Miller

Re: The operation timed out while waiting for a lock

by Steven Swanepoel -

Latest reply from hosting provider:

Hello Steven,

Thank you for your patience and apologies for the delayed response!

I tried a couple of fixes suggested online for similar issues but none of them worked. I also found out that Moodle had this reported as a bug, but they did not fix it as the error you get from the cron is supposed to mean that another instance of the cron is already in progress, which is not the case. 

I see that you are running an older version of Moodle: 

$release = '2.8.9 

Is it possible that you try to upgrade to the latest and see if this will fix the issue?


Good initiative shown from them to do some Googling. However I'm not keen on their upgrade suggestion.

In reply to Steven Swanepoel

Re: The operation timed out while waiting for a lock

by Damyon Wiese -
Because you are on MySQL, the default lock factory you will be using is "file locking" - I would try switching to DB locking to see if it helps.

$CFG->lock_factory = "\\core\\lock\\db_record_lock_factory";

in config.php should do the job. (Make sure there are no cron processes running before switching this).

I wouldn't recommend you try any of the lock factories from my plugin in the plugins db - they are more exotic and only intended for adventurous types smile

Average of ratings: Useful (3)
In reply to Damyon Wiese

Re: The operation timed out while waiting for a lock

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

What sort of additional load might this cause on the database?

I'm thinking in terms of database sessions - i.e. on a busy production site they are unusable due to burning up all the connections within a relatively short period of time. 

In reply to Howard Miller

Re: The operation timed out while waiting for a lock

by Damyon Wiese -
"What sort of additional load might this cause on the database?"

Pretty much 0 extra load - you already have a DB connection, this is only (currently) used by cron, mod_choice and theme css generation. It makes a few DB queries instead of one call to "flock". In theory the flock call should be quicker - but it looks like the underlying filesystem is being un-reliable in this case so it's better to use the DB.


In reply to Damyon Wiese

Re: The operation timed out while waiting for a lock

by Steven Swanepoel -

Thanks Damyon, this change to config.php worked! Great news just before the weekend!

In reply to Steven Swanepoel

Re: The operation timed out while waiting for a lock

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I think I may have been "guilty" of entering that bug report based on my experiences (on multiple sites). However, I may have misinterpreted the underlying cause. 

As I say above, the underlying thinking behind this locking code is very opaque to me so I'm not really qualified to comment. 

In reply to Howard Miller

Re: The operation timed out while waiting for a lock

by Steven Swanepoel -

Well we run a light weight Moodle compared to our organisations so I was surprised when I saw the cron locking up. 1600 user where 50% have never logged in. 5 or so courses, the biggest one of which isn't in use yet. I've learnt a lot about cron this week that's for sure!