Upgraded to 3.9 on CentOS 8. I tried to use the new copy a course. It hung and never completed the copy. I have now tried to do a backup of the course and I get an error message "Backup pending for the resource". Question, How do I clear this?
The Course copies in progress just show the course in current operation as "backup" status as "process pending".
I am starting to panic as this course is to go live next week. Any idea on how to reset the copy process?
seLinux still present in CentOS 8 ... permissive or enforcing?
Cron job you show 'running' is what server has for config of crontab and the very end redirect which goes off into la-la-land > /dev/null could mean you really don't know if it's running.
If logs in /var/log
touch /var/log/moodle-cron.log
then change cron tab and rather than > la-la-land ... > /var/log/moodle-cron.log
You can watch realtime: tail -f /var/log/moodle-cron.log
Check moodledata/temp/backup/
That's the area moodle uses to build backups. Anything found in there can be manually removed safely in case you are approaching a space crunch.
IF ... IF ... you see folders with long names (like a contenthash value) change into one/all to inspect contents. The very last action a backup process uses is to 'copy' (that's 'copy') the .mbz file to destination ... which, unless you change it via backup preferences is moodledata/filedir/ and by no humanly recognizable name.
There should be a moodle_backup.xml file present and a .mbz file if it indeed finish ... the .mbz is good - can be used to restore.
If no moodle_backup.xml nor .mbz didn't finish - timed out - task in cron job rompted on cleaning up before backups/restores completed ... other ????
Do drill down into one of those directories. ls -l looking for the largest .xml file. Usually that would mean heaviest.
No science to this ... just a lot of poking/probing until a clue emerges. :|
DB is also involved and that's another level ... however, check DB real quick with:
select userid,contenthash,filearea,filename,filesize from `mdl_files` where filename like "%.mbz%" order by filesize DESC;
'SoS', Ken
In the DB I see:
| 2 | da0bc005fd24de83a6decb396576bb75f7b30796 | draft | backup-moodle2-course-5-conflict_management-20200625-0952.mbz | 10240876 |
| 2 | da0bc005fd24de83a6decb396576bb75f7b30796 | draft | backup-moodle2-course-5-conflict_management-20200625-0952.mbz | 10240876 |
Could I delete those records without upsetting the whole db?
Leave DB alone ... but you do see info and the size of the backup now:
Both of same course ... size 10240876 - 10 Megs - so whatever is done with course/backups will take a little while ... won't be instant!
On to next posting ... which discloses the real issue.
'SoS', Ken
!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:
[dirroot]/lib/cronlib.php on line 154
Code should look like:
$factory = \core\lock\lock_config::get_lock_factory('type');
$lock = $factory->get_lock(4f23a73256f500bf9c7d78cb8f15f77398be7ffb);
$lock->release(); // Locks must ALWAYS be released like this.
So, am gathering from this, the cron wasn't running correctly ... maybe for a long time. :| *Fix your cron job.*
Issue: "lock was created but not released"
This is not normally the way to 'fix' but considering.
Using command line or anything you have the can browse files (create/delete) ... go to your moodledata directory.
remove the 'lock' directory - directory and it's contents.
Safe to do cause moodle will re-create it if needed.
BTW, 'lock' present in older versions of moodle in moodledata ... but not in a 3.9+. So upgrade may not have removed it.
Manually run your cron job ... maybe multiple times ... until noting to do.
From moodlecode/admin/cli/
php cron.php [ENTER]
up arrow ... hit enter ... repeat until you see nothing actually ran:
Something like:
[root@server cli]# php cron.php
Server Time: Thu, 25 Jun 2020 16:59:05 +0100
Ran 0 adhoc tasks found at Thu, 25 Jun 2020 16:59:05 +0100
Cron script completed correctly
Cron completed at 16:59:05. Memory used 15.5MB.
Execution took 0.032055 seconds
See adhoc task above ... where the copy course stuff is done? ;)
'SoS', Ken
[root@m2 maol-onlinedata]# php /var/www/html/maol-online/moodle/admin/cli/cron.php
Server Time: Thu, 25 Jun 2020 20:14:11 +0100
Execute adhoc task: core_course\task\course_delete_modules
... started 20:14:11. Current memory use 3.5MB.
!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:
[dirroot]/lib/cronlib.php on line 154
Code should look like:
$factory = \core\lock\lock_config::get_lock_factory('type');
$lock = $factory->get_lock(4f23a73256f500bf9c7d78cb8f15f77398be7ffb);
$lock->release(); // Locks must ALWAYS be released like this.
From command line, in moodlecode/admin/tool/task/cli/
run php adhoc_task.php
That will bring up help on that script
Ad hoc cron tasks.
Options:
-h, --help Print out this help
--showsql Show sql queries before they are executed
--showdebugging Show developer level debugging information
-e, --execute Run all queued adhoc tasks
-k, --keep-alive=N Keep this script alive for N seconds and poll for new adhoc tasks
-i --ignorelimits Ignore task_adhoc_concurrency_limit and task_adhoc_max_runtime limits
Run this one first:
php adhoc_task.php --showdebugging --execute
What are ownerships/permissions on moodledata?
Is it located in /var/www/
Dunno but we might be looking at botched upgrade. - things missing?
Upgraded to 3.9 from what version?
What version is your 3.9 now?
How did you upgrade?
'SoS', Ken
moodledata apache:apache 777
Process for upgrade...
- upload moodle-latest-39.tgz to server
- backup up moodle/ and moodledata/
- backup moodle.sql
- move moodle/ to moodle.old/
- untar moodle-latest-39.tgz
- cp moodle.old/config.php to /moodle
- restart httpd
- login and run the upgrade
Here is the output of the adhoc_task.php
[root@m2 cli]# php adhoc_task.php --showdebugging --execute
Server Time: Fri, 26 Jun 2020 11:57:21 +0100
Execute adhoc task: core_course\task\course_delete_modules
... started 11:57:21. Current memory use 2.3MB.
instantiating backup controller e199b52d7200db9cfeb379a71d4cce13
setting controller status to 100
loading controller plan
Default exception handler: Coding error detected, it must be fixed by a programm er: A lock was created but not released at:
[dirroot]/lib/cronlib.php on line 154
Code should look like:
$factory = \core\lock\lock_config::get_lock_factory('type');
$lock = $factory->get_lock(4f23a73256f500bf9c7d78cb8f15f77398be7ffb);
$lock->release(); // Locks must ALWAYS be released like this.
Debug:
Error code: codingerror
* line 117 of /lib/classes/lock/lock.php: coding_exception thrown
* line 126 of /admin/tool/task/cli/adhoc_task.php: call to core\lock\lock->__des truct()
!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:
[dirroot]/lib/cronlib.php on line 154
Code should look like:
$factory = \core\lock\lock_config::get_lock_factory('type');
$lock = $factory->get_lock(4f23a73256f500bf9c7d78cb8f15f77398be7ffb);
$lock->release(); // Locks must ALWAYS be released like this.
!!!
!!
Error code: codingerror !!
!! Stack trace: * line 117 of /lib/classes/lock/lock.php: coding_exception throw n
* line 126 of /admin/tool/task/cli/adhoc_task.php: call to core\lock\lock->__ destruct()
!!
Does this mean I really need to change the code in the PHP file?
Have always hated the 'numbers' on permissions ... can never remember which sets what.
So what does the following look like on your server:
ls -ld /var/www/moodledata
Like this?
drwxrwsrwx. 11 apache apache 4096 Jun 25 16:32 moodledata
Recursively, where, if present due to config, is where one would fine a 'lock' directory. Like this?
drwxrwsrwx. 3 apache apache 4096 Jun 25 16:33 cache
drwxrwsrwx. 17 apache apache 4096 Jun 25 10:01 filedir
drwxrwsrwx. 2 apache apache 4096 Jun 22 12:32 lang
drwxrwsrwx. 7 apache apache 4096 Jun 25 17:10 localcache
drwxrwsrwx. 6 apache apache 4096 Jun 25 10:01 models
drwxrwsrwx. 2 apache apache 4096 Jun 22 12:34 muc
drwxrwsrwx. 2 apache apache 4096 Jun 26 00:01 sessions
drwxrwsrwx. 9 apache apache 4096 Jun 25 16:33 temp
drwxrwsrwx. 3 apache apache 4096 Jun 25 10:01 trashdir
Note there is no 'lock' directory.
In config-dist.php of a fresh git acquired 3.9+ (Build: 20200626)
There are several config lines related to your issue.
Clips:
// Some filesystems such as NFS may not support file locking operations.
// $CFG->preventfilelocking = false;
// Moodle 2.7 introduces a locking api for critical tasks (e.g. cron).
// The default locking system to use is DB locking for Postgres, MySQL, MariaDB and
// file locking for Oracle and SQLServer. If $CFG->preventfilelocking is set, then the
// default will always be DB locking. It can be manually set to one of the lock
// \core\lock\lock_factory interface.
// $CFG->lock_factory = "auto";
// The list of available lock factories is:
// "\\core\\lock\\file_lock_factory" - File locking
// Uses lock files stored by default in the dataroot. Whether this
// "\\core\\lock\\db_record_lock_factory" - DB locking based on table rows.
// "\\core\\lock\\mysql_lock_factory" - DB locking based on MySQL / MariaDB locks.
// "\\core\\lock\\postgres_lock_factory" - DB locking based on postgres advisory locks.
// Settings used by the lock factories
// Location for lock files used by the File locking factory. This must exist
// on a shared file system that supports locking.
// $CFG->file_lock_root = $CFG->dataroot . '/lock';
// Session lock warning threshold. Long running pages should release the session using \core\session\manager::write_close().
// Set this threshold to any value greater than 0 to add developer warnings when a page locks the session for too long.
// The session should rarely be locked for more than 1 second. The input should be in seconds and may be a float.
// $CFG->debugsessionlock = 5;
Your errors show it started Execute adhoc task: core_course\task\course_delete_modules and couldn't.
'destruct' ... which means to me, erase/remove.
Since you have no lock directory in moodledata, has to be in DB ... and there is a table for locks. I've never had to use it.
mysql> show tables like 'mdl_lock%';
My 3.9 DB has the table but no data in it!
mysql> select * from mdl_lock_db;
Empty set (0.00 sec)
For some reason, your server is confused.
Suggest using config.php and setting
// $CFG->file_lock_root = $CFG->dataroot . '/lock';
which, if I understand it, should create a lock directory in moodledata.
If it doesn't, manually create a 'lock' directory in your moodledata and give it liberal permissions:
mkdir lock
chmod ugo+rwx lock -R
chown apache:apache lock -R
should look like:
drwxrwsrwx. 9 apache apache 4096 Jun 25 16:33 lock
The run the adhoc task again to see if we can't get by that!
Fingers X'd!
'SoS', Ken
Been looking over GUI config of a 3.9+
Can't find 'lock' setting anywhere in GUI config that appears to
relate to issue with cron.
However, in
Experimental settings
Context freezing contextlocking
Default: No
This setting enables read-only access to be set for selected categories, courses, activities or blocks.
Context freezing applies to administrators contextlockappliestoadmin
Default: Yes
If disabled, administrators remain with write access to any frozen contexts.
What does your site have?
Now I am grabbing at straws!!!! :|
'SoS', Ken
Have looked at all of the above. In the past when I would upgrade I would just copy the old config.php file to the new moodle directory. This time I modified the config-dist.php. I did uncomment
//"\\core\\lock\\file_lock_factory" - File locking
That took the whole site down. So I replaced the comment.
I am out of airspeed, altitude and ideas. Do you know any of the developers that could help with this issue?
Clueless... Dennis
May we see what ownerships/permissions looks like for your server's moodledata directory?
'SoS', Ken
drwxrwxrwx 11 apache apache 4096 Jun 27 11:55 maol-onlinedata
[root@m2 maol-onlinedata]# ls -la
total 48
drwxrwxrwx 11 apache apache 4096 Jun 27 11:55 .
drwxr-xr-x 7 root root 4096 Jun 22 01:17 ..
drwxrwxrwx 3 apache apache 4096 Jun 26 02:37 cache
drwxrwxrwx 258 apache apache 4096 Jun 1 23:09 filedir
-rwxrwxrwx 1 apache apache 128 May 17 12:23 .htaccess
drwxrwxrwx 2 apache apache 4096 May 21 14:33 lang
drwxrwxrwx 7 apache apache 4096 Jun 27 15:08 localcache
drwxrwxrwx 6 apache apache 4096 May 21 14:33 models
drwxrwxrwx 2 apache apache 4096 May 17 12:24 muc
drwxrwxrwx 2 apache apache 4096 Jun 27 15:02 sessions
drwxrwxrwx 13 apache apache 4096 Jun 27 12:00 temp
drwxrwxrwx 2 apache apache 4096 Jun 27 11:55 trashdir
Earlier post ... response to how upgraded ... you said copied back into new code ... now we are saying you edited.
So there's a change you don' t normally do.
The edited config-dist.php was copied to config.php to make it active.
A config-dist.php file has a lot of stuff! Do you re-call having to comment out something else? Or did you decide to set something you hadn't noticed before?
And since you were working with the file owneerships/permissions on the copied config.php are in play. The config.php file must be readable by all:
chmod ugo+r config.php
But not writeable ..
chmod ugo-wx config.php
This makes config.php only readable.
Asked about selinux ... CentOS 8 still uses selinux.
Is selinux in enforcing or permissive? IF in permissive if selinux doesn't like something it will log it to apache logs.
sestatus will show if set to permissive or not.
So in config.php right now ... is there a line that tells moodle to use moodledata/locks/ rather than DB?
In the db for your moodle, are there any records in mdl_lock_db?
'SoS', Ken
config.php is
-r-xr-xr-x 1 root root 55522 Jun 26 16:28 config.php
selinux is disabled
No records in mdl_lock_db
What's x-fw-server: Flywheel/4.1.0?
'SoS', Ken
??? Where did this come from?
Try to keep this short ... your server is having issues that others are not reporting in great numbers. There's server config and we may have reached an end there, but there is also network ... and other.
Tried to slueth your site url and did some remote test from my location ... things like a curl -I ... nothing invasive. Now if I did find the right site, a test showed a funny response ... what I asked about.
'SoS', Ken
Today I created a new VPS using CentOS 8. I was very careful with the installation of apache, PHP, and MariaDB. Created a new Moodle site, not moving data from another site etc. Tried the copy and the same issue. It gets hung on the backup and never completes.
Well, that's 'special'! At first you say it didn't work ... just hung ... BTW, since it is dependent upon a task/cron to do it's thing .. the task/cron must have kicked in and volia ... success!
Didn't think there were many options when it comes to installing AMP stack on CentOS 8 - did you start with minimal ISO (only sshd - no GUI desktop - no AMP stack) and build via package manager (dnf)? Did you install any repos ... like epel or MySQL or remi or ?
This time ... was there a 'lock' directory in moodledata when you started?
Curious mind ... wanna know!
Most of yesterday and today with a VMWare CentOS 6 and a VB CentOS 6 and trying to get 5.7 MySQL and PHP 7.2 for migrating a 3.5.highest moodle to new hosting on CentOS 7 later this year (actually summer).
Nothing replaces reading up as much as one can ... and still ...hickups! (GRRRRR!)
Had issues with MySQL ... on the VMWare CentOS 6 finally had to un-install MySQL 5.5 from another repo and re-install new 5.7 from MySQL Repo. Fresh install didn't so the temp root password at first ... and all things I tried didn't work. Un-installed, re-installed, and what do ya know ... it worked. more GRRRRRR! Settings in my.cnf no longer applied .. less to tweak with Tuner ... good to go ... and that was on a production server.
mysql_upgrade -u root -p with force option on one ... and
mysql_secure_installation for fresh 5.7 ...
side note: starting with MariaDB 10.4.6 mariadb-secure-installation is same thing but it's a symlink to mysql_secure_installation.
All I can say it ... ain't technology grand! :| (when it works ... but when it doesn't ... it sucks! Never started out to be a DB admin ... but learn as I must. (typical public school teacher, huh?)
12:30 AM here ... had all the 'fun' I want for a weekend now ... am ready for ZZZZZZZZ's
Anyhoo ... let us know what you find ... for you are on the 'bleeding edge'!
'SoS', Ken
Line referenced in lib/cronlib.php (154) is in this section of code - line 154 is **'d
if ($checklimits) {
$adhoclock = null;
for ($run = 0; $run < $maxruns; $run++) {
// If we can't get a lock instantly it means runner N is already running
// so fail as fast as possible and try N+1 so we don't limit the speed at
// which we bring new runners into the pool.
** if ($adhoclock = $cronlockfactory->get_lock("adhoc_task_runner_{$run}", 0)) {
break;
}
Wonder if one can actually see the process listed in server processes:
ps aux |grep php
Could get nasty and kill any such process seen with:
kill -9 PID (where PID is the process ID number of the php lock factory. I don't know what that looks like cause).
The only time I've had to do that was with Automatted backups process and disk nearing 100% full.
'SoS', Ken
Also asynchronous backups are NOT enabled.
I have just tried to do a course restore from a backup, and I have the same issue. It is stuck at "Course restoration in process" and never completes.
I wonder if there is a file permission issue??
Have you found a way to remove the "backup pending"? I can't edit, move, or delete the course because it is waiting on the backup to complete. Need to find a way to delete the course. I have now been successful using backup and restore to move the courses.
@both ...
In the copy screen are you using 'Copy and Return' or 'Copy and view'.
Just tested.
In moodledata/backup/temp/ a directory was created.
e618bcfcaa562a47cf3ffb1b6a26db5a
but that's as far as it got ... no activity in that directory.
Immediately went to Course Copies in progress screen and has
a 'Process Pending'.
Manually ran cron.php in admin/cli/
Adhoc task complete: core\task\refresh_mod_calendar_events_task
Execute adhoc task: core\task\asynchronous_copy_task
... started 16:01:40. Current memory use 55.9MB.
Course copy: Processing asynchronous course copy for course id: 3
Course copy: Backing up course, id: 3
Course copy: Restoring into course, id: 4
Course copy: Copy completed in: 0 seconds
... used 929 dbqueries
... used 0.68147683143616 seconds
Adhoc task complete: core\task\asynchronous_copy_task
After running cron job completed, the screen for Course Copies in progress shows a check mark in status ... completed.
So ... check your cron job ... adhoc task
Cron job running frequently enough? 1 time every minute
Check moodle settings for Task Processing
admin/settings.php?section=taskprocessing
Check task logs
/admin/tasklogs.php
Check Scheduled task
/admin/tool/task/scheduledtasks.php
Cleanup course recycle bin \tool_recyclebin\task\cleanup_course_bin
'SoS', Ken
In the settings.php file, no Section=taskprocess, but there is
$section = required_param('section', PARAM_SAFEDIR);
Don't know what I am looking for in the
Check task logs
/admin/tasklogs.php
Check Scheduled task
/admin/tool/task/scheduledtasks.php
Also not sure where and what cleanup_course_bin as I can find no file by that name.
'Don't know what I am looking for in the ... ' links ... anything related to issue you are having.
cleanup_course_bin should be a row in
/admin/tool/task/scheduledtasks.php
And it should show:
Cleanup course recycle bin \tool_recyclebin\task\cleanup_course_bin | Recycle bin tool_recyclebin | Thursday, 25 June 2020, 4:58 PM | ASAP | */30 | * |
---|
Recyclebin kicks in when anything is deleted ... creates backups (.mbz files) in case user wants to restore something they deleted. Am assuming that since copying course is using a backup process when/if successfull clean ups would be needed. Dunno really cause 3.9 is right out of the shrink wrap and only a 3.9+
Am not seeing your problem in the 3.9+ am using so am looking everywhere I can think of!!!
'SoS', Ken
Has anyone managed to solve this problem? I've tried everything and nothing, I have the same problem! Damn updates.
I have same issue in Debian 9
A lock was created but not released at: [dirroot]/lib/cronlib.php on line 154
Code should look like:
$factory = \core\lock\lock_config::get_lock_factory('type');
$lock = $factory->get_lock(709937f245ddec70ed665f995e47c9858dd6d40e);
$lock->release(); // Locks must ALWAYS be released like this.
Is safe execute?
cd /var/moodledata
rm -f ./cache/ ./lock/ ./temp/ ./trashdir/ ./sessions/ ./localcache/ ./muc/