A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Number of replies: 19
Picture of Particularly helpful Moodlers Picture of Translators
The following error took me by surprise. It is a testing server, so flogged much. Still I'm careful not to disturb the LEMP operation, I thought.

===
!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:
[dirroot]/lib/classes/task/manager.php on line 913

 Code should look like:

 $factory = \core\lock\lock_config::get_lock_factory('type');
 $lock = $factory->get_lock([longhash]);
 $lock->release();  // Locks must ALWAYS be released like this.

 !!!
!!
Error code: codingerror !!
!! Stack trace: * line 135 of /lib/classes/lock/lock.php: coding_exception thrown
* line 73 of /lib/cronlib.php: call to core\lock\lock->__destruct()
* line 178 of /admin/cli/cron.php: call to cron_run()
 !!
===

The error is reported quite late. Checking logs, it started on 17 Nov 16:08 after a server update.

- Server: Ubuntu 22.04.3 LTS
- PHP 8.1.2-1ubuntu2.14 (cli)
- Nginx with php-fpm
- Moodle 4.1.6+ (Build: 20231110)

The Moodle (test) instance functions perfectly.

Never seen that. Started debugging. Posting here, in case this has been reported.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
I have a clue. There could be a connection to its special setup. On that Linux machine non-privileged shell users run their own Moodle instances. The file permissions of moodledata is delicate but I had this system and its predecessor running for over an year now.

The differences now are
a) now we are on Moodle 4.1 LTS, last time 3.9 LTS and 3.11
b) the admin/cli/cron.php is called by the shell users crontab, not from the user www-user's crontab.

The latter seems to upset the permissions somewhere under moodledata.

I thought, not 100% sure, that the system ran that way under Moodle 3.x. Is it possible that the server task framework behaves differently in Moodle 4.1 LTS?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
My specific question is whether Moodle will allow a shell user other than the owner of the web server to run its Cron? I can provide the Unix side of file permissions - the usual things with groups and the more obscure guid, suid, sticky bit or whatever. What I don't know is the Moodle  side, whether Moodle, specifically starting from 4.0, won't allow that.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:

If I remember correctly this is the symptom, i.e. that a task did not complete correctly, the A lock was created… message is not the actual error to be resolved. So one needs to identify the task that was running and see if there are any errors reported.

My specific question is whether Moodle will allow a shell user other than the owner of the web server to run its Cron?

I don't think Moodle does any checking of file owners. The scripts are run by a process – typically php-fpm for the web server, and systemd impersonating the crontab's owner for cron.php – and this process' user account must be able to read/write all files in Moodledata.  Presumably these could be two different users as long as each has the ability to read, modify and delete files created by the other (I don't have an environment set up like this to confirm). This might necessitate granting full permission (chmod -R 777 moodledata) which is rarely ideal from a security perspective.

Is it possible that the server task framework behaves differently in Moodle 4.1 LTS?

I'm not aware of any changes between versions that might cause this.

Average of ratings: Useful (1)
In reply to Leon Stringer

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Leon

You wrote:
> I'm not aware of any changes between versions that might cause this.

Thanks for that pointer. Then I will have a second look. I know that my 3.9 LTS setup worked, because it still does. I recovered that VM.

Here are the ownerships and permissions of its moodledata:
$ ls -l
total 24
drwxrwx---   2 user008  www-data    6 Sep 22 00:45 antivirus_quarantine
drwxrwx---   3 www-data www-data   55 Sep 21 03:07 cache
drwxrwx---   2 www-data www-data   24 Aug 15 16:04 diag
drwxrwx--- 258 user008  www-data 8192 Nov  1 14:26 filedir
drwxrwx---   2 user008  www-data    6 Sep 21 04:45 lang
drwxrwx---   7 www-data www-data  101 Sep 21 03:10 localcache
drwxrwx---   6 user008  www-data   42 Sep 21 19:00 models
drwxrwx---   3 user008  www-data   40 Aug 11 11:11 moodle008data
drwxrwx---   2 www-data www-data   24 Sep 21 03:06 muc
drwxr-x---   3 www-data users      27 Feb  8  2023 repository
drwxrwx---   2 www-data www-data 4096 Nov 26 02:35 sessions
drwxrwx---   5 www-data www-data   73 Nov 30 06:55 temp
drwxrwx---  94 user008  www-data 4096 Dec  1 14:25 trashdir
-rw-rw----   1 user008  www-data  168 Aug  5  2022 warning.txt
They look messed up but through the write permissions for the group www-data both the shell user and the web-server can write and delete files.

$ id user008
uid=1008(user008) gid=33(www-data) groups=33(www-data)

$CFG->directorypermissions = 0770;
Note that there are no "777".

So the only possibility is a difference between the two LEMP installations - both are Nginx, php-fpm with fastcgi. The old (working) one is Debian 11, the newer (not working) one is Ubuntu 22.04.

> If I remember correctly this is the symptom, i.e. that a task did not complete correctly, the A lock was created… message is not the actual error to be resolved. So one needs to identify the task that was running and see if there are any errors reported.

But every shell user who owns a Moodle instance gets this. In this setup any shell user can install a single Moodle instance under a subdirectory mysite/siteXYZ. So all sites can't get one fine day, when I rebooted the machine after a minor apt upgrade!

So on to Nginx and/or PHP pecularities.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

To recap: the issue is that cron.php is failing with A lock was created…, but there's no evidence it's caused by file permissions, this is just a suspicion at this stage?

And this happened after a system update and reboot? If this is a CentOS server I'd be suspecting SELinux enforcing by default after being turned off temporarily, but presumably with Ubuntu that's not a factor?

You could try running cron.php (sudo -u user008 php admin/cli/cron.php) and seeing which task it gets up to when the error appears. Then try running that task individually (sudo -u user008 admin/cli/scheduled_task.php --execute='\class\path\to\task'). Of course, lots of tasks run as ad hoc tasks in recent Moodle versions which complicates this kind of troubleshooting.

Average of ratings: Useful (1)
In reply to Leon Stringer

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
At the highest debug level:

user033@server$ php admin/cli/scheduled_task.php --execute=\\core\\task\\session_cleanup_task
!!! Invalid permissions detected when trying to create a directory. 
Turn debugging on for further details. !!!
!! /tmp/requestdir/7hkq can not be created, check permissions.
Error code: invaliddatarootpermissions !!
!! Stack trace: * line 1577 of /lib/setuplib.php: invalid_dataroot_permissions thrown
* line 1668 of /lib/setuplib.php: call to make_writable_directory()
* line 106 of /lib/classes/task/logmanager.php: call to get_request_storage_directory()
* line 242 of /lib/cronlib.php: call to core\task\logmanager::start_logging()
* line 167 of /admin/cli/scheduled_task.php: call to cron_run_inner_scheduled_task()
 !!
Fatal error: Uncaught coding_exception: Coding error detected, it must be fixed by a programmer: 
A lock was created but not released at:
/var/www/html/site033/admin/cli/scheduled_task.php on line 154
 Code should look like:
 $factory = \core\lock\lock_config::get_lock_factory('type');
 $lock = $factory->get_lock(be146bb29840cb8d4896f6b40a5fabb76c419d79);
 $lock->release();  // Locks must ALWAYS be released like this.
 in /var/www/html/site033/lib/classes/lock/lock.php:135
Stack trace:
#0 [internal function]: core\lock\lock->__destruct()
#1 {main}  thrown in /var/www/html/site033/lib/classes/lock/lock.php on line 135
What it says is true:

user033@server:$ ls -ld /tmp/requestdir
drwxr-x--- 6 www-data www-data 4096 Dec  3 17:35 /tmp/requestdir
user033@server:$ mkdir /tmp/requestdir/foobar
mkdir: cannot create directory ‘/tmp/requestdir/foobar’: Permission denied

For me it looks like this /tmp/requestdir has been created without respecting $CFG->directorypermissions = 0770; in the config.php. Expected pattern would be drwxrwx--- 6 www-data www-data 4096 Dec  3 17:35 /tmp/requestdir

???
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Justin Hunt -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
It does look like the permissions are wrong on requestdir for some reason. It seems to be 750.
Which for user www-data will work. But for other members of www-data group will not.
Is it possible the 0770 directory permissions were a change you made to config.php after the moodle data dir folders were already made?
I would check what the permissions were on newly created folders in the moodle data dir. If they are correct, then maybe manually updating permissions on existing folders is going to be ok.
If its not an existential problem, reworking your cron set up so that cron.php is ultimately called by www-data might also be an idea
Average of ratings: Useful (1)
In reply to Justin Hunt

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Oh, yes. Manually changing: # chmod 770 /tmp/requestdir/ makes the user033 to run his cron!

There are dozens of users on this server doing the same. I think, now I know why some have this problem and some don't. It depends on who ran in to that ad-hoc task first, the user executing the cron.php manually or the crontab of the user www-data. If the user runs cron the directory permissions are set respecting the config.php but the background ad-hoc process doesn't. This is a bug!

Thanks Leon and Justin! I'm waiting for your comments before filing a tracker issue.

@Justin, this setup is not existential. This machine is a test bed. But the whole idea of this test bed is for dozens of non-privileged Linux shell users to install, upgrade, backup, delete and restore a Moodle instance each, at wish! I may have to keep on correcting this permission malfunction until the bug is fixed.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Ken Task -
Picture of Particularly helpful Moodlers
Pardon interpution ...
"But the whole idea of this test bed is for dozens of non-privileged Linux shell users to install, upgrade, backup, delete and restore a Moodle instance each, at wish!"

A thought ... have you looked into Virtualmin?

'SoS', Ken



Average of ratings: -
In reply to Ken Task

Re: dozens of non-privileged Linux shell users to install, upgrade, backup, delete and restore a Moodle instance each, at wish

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Is it one of the earliest GUI to manage Linux servers? I remember using such thing in an early life. No! GUI is not my thing!

I love CLI and scripts. This is the third "test bed" I have made. If I did the GUI thing, I have to start anew every time. So boring. Instead I have a set of scripts to create N (non-privileged) shell users, give them N random passwords, create N moodle and moodledate directories with correct permissions, N databases with the same password as the shell user, a config.php template they need to edit, ... 

In each iteration they get better.

Each time more fun!

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: dozens of non-privileged Linux shell users to install, upgrade, backup, delete and restore a Moodle instance each, at wish

by Ken Task -
Picture of Particularly helpful Moodlers
I understand where you are coming from!  As long as you are having 'fun' and with each iteration it gets better.

A Virtmin/Webmin combo is like re-seller setups with some providers (Web Hosting Management/cPanel) - the webmin/cPanel's are what the 'customer' uses.

Webmin's themselves can export a config of itself and that can be applied to other Webmin's.   Those could include modules used for php/apache/nginx/mysql/mariadb.

Both of them come with Terminal - so no need to setup remote ssh access with mfa - one less port to protect thus reducing attack surface.

And Webmin's can be interlinked ... if on the same wide area private network to be able to use 'Custom Commands' .. which are buttons pointed to shell scripts or mysql/mariadb queries.

Webmin's update themselves also!  It also handles OS updates.

I asked persons in the know one time whether Moodle (HQ/Partners/whomever) offered a class in admin of a moodle via CLI and answer was 'no' ... reason ... hosting platforms too diverse.   Well, yeah ... Windows vs Linux that's true ... but Debian/Ubuntu/CentOS are soooo simular - what one learns with one can be applied/used with another.

Anyhoo ... thought I'd share! smile
'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: dozens of non-privileged Linux shell users to install, upgrade, backup, delete and restore a Moodle instance each, at wish

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
I meant Webmin. Reading your description I must say that they have come a long way. If I have a "customer", who know, a colleague wants to do the same, then I will recommend Webmin/Virtmin, especially if he doesn't like playing in the Shell.
wink

Those who said, no sysadmin of Moodle since the platforms too diverse, true, if you compare the GUI (and partly non-free software), but the ven. Unix shell precedes all that "diversity", remained virtually same for 50 years and still have a big enough user base. So if they're compelled to play Shell, they know in which direction to look.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
There is more to it than initially observed!

Now I have the same permissions in moodledata:

drwxr-x--- 4 www-data www-data 4096 Dec  3 14:45 cache
drwxr-x--- 7 www-data www-data 4096 Nov 14 19:47 filedir
drwxr-x--- 7 www-data www-data 4096 Dec  3 14:45 localcache
drwxr-x--- 2 www-data www-data 4096 Dec  3 14:45 muc
drwxr-x--- 7 www-data www-data 4096 Nov 20 08:50 temp
although the config.php has:

$CFG->directorypermissions = 0770;
So the same problem: The permissions above should be drwxrwx---. Why doesn't moodle respect $CFG->directorypermissions?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

Is $CFG->umaskpermissions set in config.php? If this was 0020 that would cause the permissions shown in your listing. That would be quite an obscure setting to have alter but it's the only explanation I can come up with. I've tried changing $CFG->directorypermissions to various values and file permissions are set as expected.

Average of ratings: -
In reply to Leon Stringer

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Leon

That was a good piece of sleuth work!

Before we document what Moodle does, do you know how $CFG->directorypermissions and $CFG->umaskpermissions supposed to work in combination? Guessing from their names, they seem to do the same thing, the umask is just the inversion of permissions.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

From reading the code and change history it appears $CFG->umaskpermissions was added to ensure the expected permissions were set for third-party code that neglected to set $CFG->directorypermissions when creating directories. It was added in Moodle 2.6 (MDL-40642).

One thing to watch out for: a PHP script's umask setting applies across all threads for a process (see the note for umask() in the PHP docs). Presumably the php-fpm service could have threads for different Moodle sites in a single process, and if these had different $CFG->directorypermissions or $CFG->umaskpermissions then one site could potentially end up with the setting of another (for the lifetime of that thread). Presumably one should configure PHP-FPM with per-site pools in order guarantee processes – and thus their threads – are segregated by site. (I have limited knowledge on this).

Average of ratings: Useful (2)
In reply to Leon Stringer

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913 [ad-hoc solution found]

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Many thanks, Leon, for all the investigative work. You must have spent hours on this!

AFAIC we can put the matters to rest. Not that it is clear to me what the developers had in mind. In fact, "The reason for relaxed permissions is CLI execution of scripts, if you restrict the permissions and execute CLI as root you end up with fatal errors in web UI.some developers forget to use permissions in mkdir() call or forget to chmod()" makes me uneasy. I don't believe in one developer correcting the other developer's mistakes! The PHP documentation you linked is clear. chmod to set permissions, umask to remove permissions. But, I'm not a preacher.

My final take is, "Manually changing: # chmod 770 /tmp/requestdir/ makes the user033 to run his cron" https://moodle.org/mod/forum/discuss.php?d=453101#p1821236 eliminates the symptom. The trainee sysadmins work on this machine since then without this lock happening. So leaving the thread with this single chmod as an ad-hoc solution.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

I wasn't expecting admin/cli/cron.php to be writing to $CFG->localrequestdir. To me this suggests the setting name is misleading as surely "request" refers the Apache process responding to an HTTP request from a browser. My initial reaction is that admin/cli/cron.php ideally shouldn't use this location (but what do I know!).

You could try setting each site's $CFG->localrequestdir to '.../moodledata/temp/requestdir'. Then you could manually set permissions to ensure this directory is readable and writable by both www-data and user033 (or whichever user the site's cron runs as). There may be a performance impact if I/O to Moodledata is slower than /tmp. Moodle would still create files and directories with the $CFG->directorypermissions/$CFG->umaskpermissions permissions so there may still be a problem.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: A lock was created but not released at: [dirroot]/lib/classes/task/manager.php on line 913

by Justin Hunt -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
I also struggled with an issue like this recently. The "lock was created but not released .." error is just a symptom of an earlier error.
In my case it was because local_aws was using Guzzle, which moodle 4.2 also started using, and there was a version/namespace/something clash
That caused a process to crash without releasing the lock.
I think there will be an error in the cron processing, or maybe in the web server logs that should lead you to it.
I usually just run the cron from the command line, as Leon suggested, and the error reports show up in the output.
Average of ratings: Useful (1)