Has Ghostscript something to do with a half a million lock files?

Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Number of replies: 13
Picture of Particularly helpful Moodlers Picture of Translators

I am looking at a powerful machine which became unusable recently. Not my machine. It had the bad luck of happy system administrators, who were happily doing other things. I am just investigating after disaster struck. So, need to start with vague information.

First thing I found was that moodledata/lock contains 500 k files! Apparently the admins had to quadruple Kern.maxvnodes to some 30 million in October. Going through various old log files I found the same ominous message **** File has unbalanced q/Q operators (too many Q's) **** in millions. Web searches on this string point to Ghostscript, for example https://bugs.ghostscript.com/show_bug.cgi?id=694310.

So the first question is, could the two things be related?

Interestingly almost all the lock files have date stamps from June and July this year. Going back in time I came to know that they've activated assignfeedback_editpdf early this year and installed Ghostscript somewhere in April. And they conduct assignments mass-scale where students upload PDFs. That is where my suspicion comes from.

Since the lock files are months old, I deleted the lock files and recovered a million vnodes! (Yes, stopped the cron job before doing that.)

So the second question is, how many lock files are "normal" in a busy site where hundreds of students upload PDFs?

The cron job runs every 2 min. The OS is FreeBSD 13. Web server Apache 2.4, DB server MySQL 5.7, PHP 7.3, configured mpm. All the web server instances are in a single Jail, to which I have super user access.

Related discussions in these forums:

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Has Ghostscript something to do with a half a million lock files?

by Ken Task -
Picture of Particularly helpful Moodlers

Shame on you ... the one thing you didn't mention ... version of Moodle! smile

Ghostscript is needed if moodle setup to use annotation in grading online/uploaded docs assignments.  If they set up an assignment and were uploading .doc or .docx or other document type that can be converted, moodle could convert to a PDF for 'grading'.

I think even if they uploaded a PDF it would still be processed same way.

Thus all those locks.

Using which file converter?  Unoconv or Google?

Is there any ad-hoc task pending completion?

in admin/tool/task/cli/

php schedule_task.php --execute="\mod_assign\task\cron_task"

also/maybe:

php schedule_task.php --execute="\core_files\task\conversion_cleanup_task"

'SoS', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Ken

Nice to see you! (again)
;)

Ooh, Moodle is, are rel. 3.10 and 3.11. A detail I didn't mention earlier, in production there are two Moodle instances involved, an instance for exams 3.10 (Build: 20201109)  and another for teaching 3.11+ (Build: 20210625) . The exam system authenticates from the teaching system through MNet. That was the subject of this discussion https://moodle.org/mod/forum/discuss.php?d=429280#p1727578.

Both instances run in the same BSD Jail, both get unusable during heavy loads. I only have admin access to a third one for testing, which is a copy of the exam system 3.11.4+ (Build: 20211112), again in the same Jail. I avoided admin access to the two production instances, looks like I need to change my mind.

Students upload 90% PDFs. The others PNG. To my information no DOCX/ODT are involved in the current exams. As such no document converter, Site administration > Plugins > Document converters > Manage document converters, is active;  neither unoconv nor Google.  Only assignfeedback_editpdf | default is set, the path to Ghostscript is /usr/local/bin/gs, the path test reports success.  '/usr/local/bin/gs --version' gives 9.06. Yes, my understanding too is that all PDF submissions go through it.

teaching instance # php admin/tool/task/cli/schedule_task.php --execute="\mod_assign\task\cron_task"
Execute scheduled task: Background processing for assignment module (mod_assign\task\cron_task)
Processing 1 assignment submissions ...
Processing assignment submission 180207 ...
Done
Done processing 1 assignment submissions
... used 27 dbqueries
... used 0.57677006721497 seconds
Scheduled task complete: Background processing for assignment module (mod_assign\task\cron_task)

exam instance # php admin/tool/task/cli/schedule_task.php --execute="\mod_assign\task\cron_task"
Execute scheduled task: Background processing for assignment module (mod_assign\task\cron_task)
... used 6 dbqueries
... used 0.10258007049561 seconds
Scheduled task complete: Background processing for assignment module (mod_assign\task\cron_task)

both instances # php admin/tool/task/cli/schedule_task.php --execute="\core_files\task\conversion_cleanup_task"
Execute scheduled task: Cleanup of temporary records for file conversions (core_files\task\conversion_cleanup_task)
... used 5 dbqueries
... used 0.022668838500977 seconds
Scheduled task complete: Cleanup of temporary records for file conversions (core_files\task\conversion_cleanup_task)

I didn't expect bg jobs since right now there is no exam load and we killed those old lock files yesterday. We expect light exam loads today. Will monitor those two commands when the candidates enter the exams.

The bad news is that vnodes still grow, even now. There must be something more probably at the system level, not directly related to Moodle, for example background mails generated from crontab. Unfortunately making little progress in this unfamilier BSD Jails system. Planning to go that path with the sysadmin.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Has Ghostscript something to do with a half a million lock files?

by Ken Task -
Picture of Particularly helpful Moodlers

Been looking for reasons ...

Locking was intro'd in 2.7 and designed to prevent cron job from acting on some process that was already running ... conflicts ... such as clean up of moodledata/temp/backup area when autobackups still working on a large course backup - something like that.

So has the troubled instance of moodle had any issues with the running of cron?

You have 2 options for locking ... files or DB.  The presence of a 'lock' directory in moodledata indicates that's 'files' (default? ... I think).

Look in config-dist.php file for some settings to change behavior of the lock_factory.

fgrep lock_factory config-dist.php

Set to DB I would think that would fix your problem with file inode limits of system, but it would, at the same time, put greater 'stress' on DB.  That's a guess ... not scientific at all.

Fingers crossed for ya! smile

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Ken

Not seeing any light still. The explosion in moodle/data must have happened either due to a breakdown at the system level or indeed due to a buggy Ghostscript. After cleaning the mess 24 h ago Moodle has not even created a lock directory. There were exams during that time but smaller ones, totaling 200 candidates over three time slots, two overlapping.

Thanks for the pointer "locking api" in config-dist.php. I must admin, I understood nothing. This thread has more readable information https://moodle.org/mod/forum/discuss.php?d=417100 especially this https://moodle.org/mod/forum/discuss.php?d=417100#p1680821.

Yes, the lock files go to the disk, which is the default. I don't want them to congest the DB, prefer to have the machine running without creating very many locks.

A buggy Ghostscript is still a possibility. But the present Ghostscript passes Moodle's internal test. So will leave it for the time being.

Same thing with the vnode limits. According to the local FreeBSD specialist the numbers we see are normal for the size of the operation.

Will keep on observing the server.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Has Ghostscript something to do with a half a million lock files?

by Alain Raap -
Picture of Particularly helpful Moodlers
Visvanath, lockfiles make me think of Redis or php session lock files on the file system, but I assume this isn't what you've configured? Did you check the session settings in your config.php file? Lock files in the moodledata directory I've never heard of before, but this directory can become very large in size and number of files.
Average of ratings: -
In reply to Alain Raap

Re: Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hello Alain

If you mean moodledata/sessions/, no, this is about moodledata/lock/. There is nothing on lock in config.php.

From config-dist.php:
// Moodle 2.7 introduces a locking api for critical tasks (e.g. cron).
...
// Settings used by the lock factories
//
// Location for lock files used by the File locking factory. This must exist
// on a shared file system that supports locking.
// $CFG->file_lock_root = $CFG->dataroot . '/lock';

This lock/ is a "little known" directory in moodledata/. Agree, moodledata/filedir/ can get very large depending on what users upload. But I don't see a reason why moodledata/lock/ should explode to half a million files!

No memcache(d), no Redis, BTW.

@Ken and all
Interestingly, config-dist.cfg says:
// The default locking system to use is DB locking for Postgres, MySQL, MariaDB  and
// file locking for Oracle and SQLServer. If $CFG->preventfilelocking is set, then the
// default will always be DB locking. It can be manually set to one of the lock
// factory classes listed below, or one of your own custom classes implementing the
// \core\lock\lock_factory interface.

Although there is nothing related in the config.php, this lock factory ravaged the disk. How come?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Has Ghostscript something to do with a half a million lock files?

by Ken Task -
Picture of Particularly helpful Moodlers

Sandbox server.  Multiple versions of Moodle 3.8.highest through and upto 3.11.highest

No settings in any of the config.php's for lock.

in the 3.8.highest DB:
mdl_lock_db
+-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| id          | bigint(10)   | NO   | PRI | NULL    | auto_increment |
| resourcekey | varchar(255) | NO   | UNI |         |                |
| expires     | bigint(10)   | YES  | MUL | NULL    |                |
| owner       | varchar(36)  | YES  | MUL | NULL    |                |

Empty set - no data

in moodle38data there is a lock directory.

Contains:
00  06  0f  15  1d  22  2b  31  37  3c  40  4c  5c  68  6d  72  79  85  91  96  9e  a5  b8  c5  cf  dc  e7  f9
02  07  10  16  1f  25  2d  32  39  3d  45  4e  5d  69  6f  74  7b  87  93  98  9f  aa  ba  c6  d1  e2  f1  fa
03  08  12  17  20  27  2e  34  3a  3e  47  52  60  6b  70  76  7d  8b  94  9b  a1  af  c3  ca  d4  e5  f4  fe
04  0b  14  1c  21  29  30  35  3b  3f  49  53  66  6c  71  78  7f  90  95  9d  a2  b1  c4  cc  db  e6  f6  ff

00 above:
./00:
total 0
-rw-rw-rw-. 1 apache apache 0 Aug  6 13:21 00a741993be6359dd581fdac7dfd8c31
-rw-rw-rw-. 1 apache apache 0 Nov 25 15:36 00f466af106e0caa5b26aff1ed671a85

Note those files are 0 bytes.

In the config-diet.php section related to lock:
//      $CFG->lock_factory = "auto";
But there is no explanation of 'auto'.

php cfg.php |grep lock_factory
shows no settings related.

If I manually remove all files contained in moodle38data/lock/
and then run the main cron as root user that directory is re-populated.
now looks like:
00  12  1c  20  29  2e  32  3f  4c  69  76  8b  a1  ba  c5  ca  e6  fe
06  14  1d  21  2d  31  3d  49  4e  6c  7d  9d  aa  c3  c6  cc  f9

00 looks now looks like:
-rw-rw-rw-. 1 root apache 0 Nov 25 15:51 00f466af106e0caa5b26aff1ed671a85

Note 0 byte size file above.

Checking DB again:

mysql> select * from mdl_lock_db;
Empty set (0.00 sec)

Conclusion ... some setting for 'auto' exist somewhere and 'auto' does files.

Can find no cli task/adhoc task for cleaning up - moodledata/lock/

Yes, know your version is higher than the 3.8 examples given, but it appears 3.9, 3.10, 3.11 behaves the same way?

Soooo ... wonder if setting up a cron job in operating system that would manually remove all contents of your moodledata/lock/ ... frequency to be determined by you ... would be a working 'bandaide' and avoid inod limits.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Ken, some detective work you've done!

> Sandbox server. Multiple versions of Moodle 3.8.highest through and upto 3.11.highest

Teaching machine, 3.11+ (Build: 20210625)

> No settings in any of the config.php's for lock.

Same here.

> in the 3.8.highest DB:
> mdl_lock_db
> +-------------+--------------+------+-----+---------+----------------+
> | Field | Type | Null | Key | Default | Extra |
> +-------------+--------------+------+-----+---------+----------------+
> | id | bigint(10) | NO | PRI | NULL | auto_increment |
> | resourcekey | varchar(255) | NO | UNI | | |
> | expires | bigint(10) | YES | MUL | NULL | |
>| owner | varchar(36) | YES | MUL | NULL | |
> > Empty set - no data

mdl_lock_db;
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| id | bigint(10) | NO | PRI | NULL | auto_increment |
| resourcekey | varchar(255) | NO | UNI | | |
| expires | bigint(10) | YES | MUL | NULL | |
| owner | varchar(36) | YES | MUL | NULL | |
+-------------+--------------+------+-----+---------+----------------+

Empty set (0.00 sec)

Identical.

> in moodle38data there is a lock directory.

There is no moodledata/lock/ since we cleared it three days ago.

> Note those files are 0 bytes.

> In the config-diet.php section related to lock:
> // $CFG->lock_factory = "auto";
> But there is no explanation of 'auto'.

This https://docs.moodle.org/dev/Lock_API is what I found.

> php cfg.php |grep lock_factory
> shows no settings related.

Nor in my (teaching) Moodle.

> If I manually remove all files contained in moodle38data/lock/
> and then run the main cron as root user that directory is re-populated.
> now looks like:
> 00 12 1c 20 29 2e 32 3f 4c 69 76 8b a1 ba c5 ca e6 fe
> 06 14 1d 21 2d 31 3d 49 4e 6c 7d 9d aa c3 c6 cc f9

So the question is which core components uses lock API and/or which of our plug-ins talks to lock API?

> 00 looks now looks like:
> -rw-rw-rw-. 1 root apache 0 Nov 25 15:51 00f466af106e0caa5b26aff1ed671a85
> > Note 0 byte size file above.

Sorry, didn't keep a record. From memory, 'du' gave many hundreds of MB.

> Checking DB again:
>
> mysql> select * from mdl_lock_db;
> Empty set (0.00 sec)
>
> Conclusion ... some setting for 'auto' exist somewhere and 'auto' does files.

Yep. That (locks on the file system) suits me.

> Can find no cli task/adhoc task for cleaning up - moodledata/lock/

Equally important to find out the code which creates the locks.

> Yes, know your version is higher than the 3.8 examples given, but it appears 3.9, 3.10, 3.11 behaves the same way?

Considering https://docs.moodle.org/dev/index.php?title=Lock_API&action=history I'd expect too.

> Soooo ... wonder if setting up a cron job in operating system that would manually remove all contents of your moodledata/lock/ ... frequency to be determined by you ... would be a working 'bandaide' and avoid inod limits.

I am in the lucky situation, touch wood, not even the lock/ directory reappear. So, no "bandages" are necessary.

I am almost where I started. We search in a very broad area: starting from corrupt file systems (ZFS) through mal-configured OS (FreeBSD), system components (Ghostscript) to misbehaving applications, Moodle / its plug-ins (Annotate PDF). We have a system specialist. I am looking at the Moodle side.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Has Ghostscript something to do with a half a million lock files?

by Ken Task -
Picture of Particularly helpful Moodlers

Ok, some more 'exploring' ...

Site is a 3.8.highest sandbox so not as much usage as a 'normal site'.

added to config.php

$CFG->lock_factory = "\\core\\lock\\db_record_lock_factory";

manually removed all files from moodledata/lock/

As root user ran cron.php from admin/cli/ ... no errors.

Then checked mdl_lock_db table:

mysql> select * from mdl_lock_db;
+----+---------------------------------------------------------+---------+-------+
| id | resourcekey                                             | expires | owner |
+----+---------------------------------------------------------+---------+-------+
|  1 | cron_scheduled_task_runner_0                            |    NULL | NULL  |
|  2 | cron_\auth_mnet\task\cron_task                          |    NULL | NULL  |
|  3 | cron_\enrol_category\task\enrol_category_sync           |    NULL | NULL  |
|  4 | cron_\enrol_flatfile\task\flatfile_sync_task            |    NULL | NULL  |
|  5 | cron_\enrol_imsenterprise\task\cron_task                |    NULL | NULL  |
|  6 | cron_\enrol_lti\task\sync_grades                        |    NULL | NULL  |
|  7 | cron_\enrol_lti\task\sync_members                       |    NULL | NULL  |
|  8 | cron_\enrol_meta\task\enrol_meta_sync                   |    NULL | NULL  |
|  9 | cron_\enrol_paypal\task\process_expirations             |    NULL | NULL  |
| 10 | cron_\repository_dropbox\task\cron_task                 |    NULL | NULL  |
| 11 | cron_\repository_onedrive\remove_temp_access_task       |    NULL | NULL  |
| 12 | cron_\logstore_legacy\task\cleanup_task                 |    NULL | NULL  |
| 13 | cron_\core\task\session_cleanup_task                    |    NULL | NULL  |
| 14 | cron_core_cron                                          |    NULL | NULL  |
| 15 | cron_\core\task\send_new_user_passwords_task            |    NULL | NULL  |
| 16 | cron_\core\task\send_failed_login_notifications_task    |    NULL | NULL  |
| 17 | cron_\core\task\legacy_plugin_cron_task                 |    NULL | NULL  |
| 18 | cron_\core\task\grade_cron_task                         |    NULL | NULL  |
| 19 | cron_\core\task\completion_regular_task                 |    NULL | NULL  |
| 20 | cron_\core\task\portfolio_cron_task                     |    NULL | NULL  |
| 21 | cron_\core\task\plagiarism_cron_task                    |    NULL | NULL  |
| 22 | cron_\core\task\calendar_cron_task                      |    NULL | NULL  |
| 23 | cron_\core\task\blog_cron_task                          |    NULL | NULL  |
| 24 | cron_\core\task\question_preview_cleanup_task           |    NULL | NULL  |
| 25 | cron_\core\task\question_stats_cleanup_task             |    NULL | NULL  |
| 26 | cron_\mod_assign\task\cron_task                         |    NULL | NULL  |
| 27 | cron_\mod_forum\task\cron_task                          |    NULL | NULL  |
| 28 | cron_\mod_quiz\task\update_overdue_attempts             |    NULL | NULL  |
| 29 | cron_\mod_quiz\task\legacy_quiz_reports_cron            |    NULL | NULL  |
| 30 | cron_\mod_quiz\task\legacy_quiz_accessrules_cron        |    NULL | NULL  |
| 31 | cron_\mod_workshop\task\cron_task                       |    NULL | NULL  |
| 32 | cron_\mod_workshop\task\legacy_workshop_allocation_cron |    NULL | NULL  |
| 33 | cron_\repository_filesystem\task\cron_task              |    NULL | NULL  |
| 34 | cron_\tool_messageinbound\task\pickup_task              |    NULL | NULL  |
| 35 | cron_\tool_monitor\task\clean_events                    |    NULL | NULL  |
| 36 | cron_\workshopallocation_scheduled\task\cron_task       |    NULL | NULL  |
| 37 | cron_adhoc_task_runner_0                                |    NULL | NULL  |
| 38 | cron_\quiz_statistics\task\quiz_statistics_cleanup      |    NULL | NULL  |
| 39 | cron_\tool_cohortroles\task\cohort_role_sync            |    NULL | NULL  |
+----+---------------------------------------------------------+---------+-------+
39 rows in set (0.00 sec)

logged onto site ... clicked around some ... changed no config, etc.

Did same query after and now the output is as:

mysql> select * from mdl_lock_db;
+----+---------------------------------------------------------+---------+-------+
| id | resourcekey                                             | expires | owner |
+----+---------------------------------------------------------+---------+-------+
|  1 | cron_scheduled_task_runner_0                            |    NULL | NULL  |
|  2 | cron_\auth_mnet\task\cron_task                          |    NULL | NULL  |
|  3 | cron_\enrol_category\task\enrol_category_sync           |    NULL | NULL  |
|  4 | cron_\enrol_flatfile\task\flatfile_sync_task            |    NULL | NULL  |
|  5 | cron_\enrol_imsenterprise\task\cron_task                |    NULL | NULL  |
|  6 | cron_\enrol_lti\task\sync_grades                        |    NULL | NULL  |
|  7 | cron_\enrol_lti\task\sync_members                       |    NULL | NULL  |
|  8 | cron_\enrol_meta\task\enrol_meta_sync                   |    NULL | NULL  |
|  9 | cron_\enrol_paypal\task\process_expirations             |    NULL | NULL  |
| 10 | cron_\repository_dropbox\task\cron_task                 |    NULL | NULL  |
| 11 | cron_\repository_onedrive\remove_temp_access_task       |    NULL | NULL  |
| 12 | cron_\logstore_legacy\task\cleanup_task                 |    NULL | NULL  |
| 13 | cron_\core\task\session_cleanup_task                    |    NULL | NULL  |
| 14 | cron_core_cron                                          |    NULL | NULL  |
| 15 | cron_\core\task\send_new_user_passwords_task            |    NULL | NULL  |
| 16 | cron_\core\task\send_failed_login_notifications_task    |    NULL | NULL  |
| 17 | cron_\core\task\legacy_plugin_cron_task                 |    NULL | NULL  |
| 18 | cron_\core\task\grade_cron_task                         |    NULL | NULL  |
| 19 | cron_\core\task\completion_regular_task                 |    NULL | NULL  |
| 20 | cron_\core\task\portfolio_cron_task                     |    NULL | NULL  |
| 21 | cron_\core\task\plagiarism_cron_task                    |    NULL | NULL  |
| 22 | cron_\core\task\calendar_cron_task                      |    NULL | NULL  |
| 23 | cron_\core\task\blog_cron_task                          |    NULL | NULL  |
| 24 | cron_\core\task\question_preview_cleanup_task           |    NULL | NULL  |
| 25 | cron_\core\task\question_stats_cleanup_task             |    NULL | NULL  |
| 26 | cron_\mod_assign\task\cron_task                         |    NULL | NULL  |
| 27 | cron_\mod_forum\task\cron_task                          |    NULL | NULL  |
| 28 | cron_\mod_quiz\task\update_overdue_attempts             |    NULL | NULL  |
| 29 | cron_\mod_quiz\task\legacy_quiz_reports_cron            |    NULL | NULL  |
| 30 | cron_\mod_quiz\task\legacy_quiz_accessrules_cron        |    NULL | NULL  |
| 31 | cron_\mod_workshop\task\cron_task                       |    NULL | NULL  |
| 32 | cron_\mod_workshop\task\legacy_workshop_allocation_cron |    NULL | NULL  |
| 33 | cron_\repository_filesystem\task\cron_task              |    NULL | NULL  |
| 34 | cron_\tool_messageinbound\task\pickup_task              |    NULL | NULL  |
| 35 | cron_\tool_monitor\task\clean_events                    |    NULL | NULL  |
| 36 | cron_\workshopallocation_scheduled\task\cron_task       |    NULL | NULL  |
| 37 | cron_adhoc_task_runner_0                                |    NULL | NULL  |
| 38 | cron_\quiz_statistics\task\quiz_statistics_cleanup      |    NULL | NULL  |
| 39 | cron_\tool_cohortroles\task\cohort_role_sync            |    NULL | NULL  |
| 40 | cron_\core\task\delete_incomplete_users_task            |    NULL | NULL  |
| 41 | cron_\core\task\badges_cron_task                        |    NULL | NULL  |
| 42 | cron_\core\task\badges_message_task                     |    NULL | NULL  |
| 43 | cron_\mod_chat\task\cron_task                           |    NULL | NULL  |
| 44 | cron_\mod_scorm\task\cron_task                          |    NULL | NULL  |
| 45 | cron_\block_rss_client\task\refreshfeeds                |    NULL | NULL  |
+----+---------------------------------------------------------+---------+-------+
45 rows in set (0.00 sec)

note from 39 rows to 45 rows.

Ran cron.php from command line as root user.

No errors and rows have been added to table

| 46 | cron_\core\task\backup_cleanup_task                     |    NULL | NULL  |
| 47 | cron_\enrol_cohort\task\enrol_cohort_sync               |    NULL | NULL  |
| 48 | cron_\enrol_manual\task\sync_enrolments                 |    NULL | NULL  |
| 49 | cron_\enrol_manual\task\send_expiry_notifications       |    NULL | NULL  |
| 50 | cron_\enrol_self\task\sync_enrolments                   |    NULL | NULL  |
| 51 | cron_\enrol_self\task\send_expiry_notifications         |    NULL | NULL  |
| 52 | cron_\assignfeedback_editpdf\task\convert_submissions   |    NULL | NULL  |
+----+---------------------------------------------------------+---------+-------+
52 rows in set (0.00 sec)

cron job on this site is running as per recommendations ... 1 time per minute.

So lock is directly related to cron jobs and task list as well as modules used.  *and there are no files in moodledata/lock/ *

Guess I might have to tinker in the site with stuff ... wait a day to see what changes.

Am guessing that once all the task that are turned on will eventually have an entry/row in the DB table and it won't grow ... but update a row.

If you are *brave*, you could do same on your site with issues ... if DB is in good shape.   That would end the inode issues in moodledata/lock/.  Could easily turn it back to old behavior by removing the added line in config.php.

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Ken

Thanks for the deep-dive in to the "lock factory". Went in to details and couldn't find anything suspicious. The task processing from the GUI, now I am a Moodle admin too, all seem to be clean. And no new lock files appear, Moodle has not even created the moodledata/lock/ directory. I would say no need to go the "lock" path any further.

Instead the focus has shifted to two things: The one thing related to the OP is, I need to check how the sea of PDF files, some 118 k, are related to the 420 k PNG files. I believe they are related, PNGs are the individual pages of PDFs. Whether they are clean in the sense that the PDF annotator still works. The other thing is missing files, files which are recorded in Moodle but missing in filedir/. Will have a look. Will open a new thread if necessary.

Many thanks and pl. accept a boat load of good Karma from the "customers" (a state institution I was helping pro bono).
Average of ratings: -
In reply to Ken Task

Re: Has Ghostscript something to do with a half a million lock files?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Possibly a side-track, still: The moodledata/filedir of the teaching instance is 3.1 TB big! The plugin Course size reports 3,202,136 MB. They tally at least.

I'm still keen to know how much of this are swollen PDFs due to a bug (in Ghostscript?). There have been many requests in the past. See https://moodle.org/mod/forum/discuss.php?d=412381, for example.

Does anybody know?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Has Ghostscript something to do with a half a million lock files? [RESOLVED]

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

[RESOLVED]

I don't know whether I said this somewhere else, the cause was indeed a buggy Ghostscript. This is what our FreeBSD specialist found out:

Replacing ghostscript9-base 9.06_14 with ghostscript9-agpl-base 9.52 solved the problem.

So a bug from A.D. 2011 resolved in A.D. 2014 was sleeping in our FreeBSD in A.D. 2020 and caused some havoc. It created 900 GB (no typo) big log files, the 500 k files in filedir/lock.

Possibly this is at least partly responsible for exhausted memory - although 256 GB physical memory was available. Found log files with records like

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted \
(tried to allocate 2323399 bytes) in /usr/local/www/.../

running for months.

Hope to lay the matter at rest.

Average of ratings: Useful (1)