Moodle consuming disk space and CRON task log

Moodle consuming disk space and CRON task log

by Okhtay Gh -
Number of replies: 11

Hello,

I am using :

moodle 3.8.1

Apache 2.4.41

PHP 7.3.13

 

HW spec : VPS CPU 4 core. 4GB Ram. 100GB Space

SSL : Let's Encrypt installed with certbot

My moodle keeps filling up the server's disk space regardless of if I delete backups or purge cache/logs. I checked my CRON tasks to see if they are running and I found a couple of errors one which indicates a cleanup task failure and is expected to run normally.

Log file path /opt/bitnami/apps/moodle/moodledata/moodle-cron.log

 Default exception handler: خطا در نوشتن بر روی پایگاه داده Debug: Lock wait timeout exceeded; try restarting transaction

UPDATE mdl_config_plugins SET value = ? WHERE id = ?

[array (

  0 => 1599034381,

  1 => '1934',

)]

Error code: dmlwritDefault exception handler: خطا در خواندن از پایگاه داده Debug: MySQL server has gone away

SELECT * FROM mdl_task_scheduled WHERE classname = ?

[array (

 

  0 => '\\core\\task\\session_cleanup_task',

 

Here is a wider part of the log file showing more errors :

 

sh: 1: /usr/bin/gs: not found
Potential coding error - existing temptables found when disposing database. Must be dropped!
Potential coding error - existing temptables found when disposing database. Must be dropped!
Default exception handler: خطا در نوشتن بر روی پایگاه داده Debug: Lock wait timeout exceeded; try restarting transaction
UPDATE mdl_config_plugins SET value = ? WHERE id = ?
[array (
0 => 1599034142,
1 => '1934',
)]
Error code: dmlwriteexception
* line 489 of /lib/dml/moodle_database.php: dml_write_exception thrown
* line 1634 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1809 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->set_field_select()
* line 1427 of /lib/moodlelib.php: call to moodle_database->set_field()
* line 66 of /lib/cronlib.php: call to set_config()
* line 61 of /admin/cli/cron.php: call to cron_run()

Default exception handler: خطا در نوشتن بر روی پایگاه داده Debug: Lock wait timeout exceeded; try restarting transaction
UPDATE mdl_config_plugins SET value = ? WHERE id = ?
[array (
0 => 1599034201,
1 => '1934',
)]
Error code: dmlwriteexception
* line 489 of /lib/dml/moodle_database.php: dml_write_exception thrown
* line 1634 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1809 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->set_field_select()
* line 1427 of /lib/moodlelib.php: call to moodle_database->set_field()
* line 66 of /lib/cronlib.php: call to set_config()
* line 61 of /admin/cli/cron.php: call to cron_run()

Default exception handler: خطا در نوشتن بر روی پایگاه داده Debug: Lock wait timeout exceeded; try restarting transaction
UPDATE mdl_config_plugins SET value = ? WHERE id = ?
[array (
0 => 1599034261,
1 => '1934',
)]
Error code: dmlwriteexception
* line 489 of /lib/dml/moodle_database.php: dml_write_exception thrown
* line 1634 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1809 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->set_field_select()
* line 1427 of /lib/moodlelib.php: call to moodle_database->set_field()
* line 66 of /lib/cronlib.php: call to set_config()
* line 61 of /admin/cli/cron.php: call to cron_run()

Default exception handler: خطا در نوشتن بر روی پایگاه داده Debug: Lock wait timeout exceeded; try restarting transaction
UPDATE mdl_config_plugins SET value = ? WHERE id = ?
[array (
0 => 1599034321,
1 => '1934',
)]
Error code: dmlwriteexception
* line 489 of /lib/dml/moodle_database.php: dml_write_exception thrown
* line 1634 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1809 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->set_field_select()
* line 1427 of /lib/moodlelib.php: call to moodle_database->set_field()
* line 66 of /lib/cronlib.php: call to set_config()
* line 61 of /admin/cli/cron.php: call to cron_run()

Default exception handler: خطا در نوشتن بر روی پایگاه داده Debug: Lock wait timeout exceeded; try restarting transaction
UPDATE mdl_config_plugins SET value = ? WHERE id = ?
[array (
0 => 1599034381,
1 => '1934',
)]
Error code: dmlwritDefault exception handler: خطا در خواندن از پایگاه داده Debug: MySQL server has gone away
SELECT * FROM mdl_task_scheduled WHERE classname = ?
[array (
0 => '\\core\\task\\session_cleanup_task',
)]
Error code: dmlreadexception
* line 486 of /lib/dml/moodle_database.php: dml_read_exception thrown
* line 1247 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1587 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->get_records_sql()
* line 1559 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
* line 1538 of /lib/dml/moodle_database.php: call to moodle_database->get_record_select()
* line 680 of /lib/classes/task/manager.php: call to moodle_database->get_record()
* line 229 of /lib/cronlib.php: call to core\task\manager::scheduled_task_failed()
* line 119 of /lib/cronlib.php: call to cron_run_inner_scheduled_task()
* line 73 of /lib/cronlib.php: call to cron_run_scheduled_tasks()
* line 61 of /admin/cli/cron.php: call to cron_run()

Exception ignored in shutdown function Closure::__invoke: جدول «task_log» وجود ندارد
PHP Fatal error: Uncaught coding_exception: یک خطای برنامه‌نویسی شناسایی شد. این خطا باید توسط یک برنامه‌نویس برطرف شود: A lock was created but not released at:
/opt/bitnami/apps/moodle/htdocs/lib/classes/task/manager.php on line 557

Code should look like:

$factory = \core\lock\lock_config::get_lock_factory('type');
$lock = $factory->get_lock(Resource id #742);
$lock->release(); // Locks must ALWAYS be released like this.

in /opt/bitnami/apps/moodle/htdocs/lib/classes/lock/lock.php:117
Stack trace:
#0 [internal function]: core\lock\lock->__destruct()
#1 {main}
thrown in /opt/bitnami/apps/moodle/htdocs/lib/classes/lock/lock.php on line 117
Default exception handler: یک خطای برنامه‌نویسی شناسایی شد. این خطا باید توسط یک برنامه‌نویس برطرف شود: moodle_database::update_record_raw() id field must be specified. Debug:
Error code: codingerror
* line 1536 of /lib/dml/mysqli_native_moodle_database.php: coding_exception thrown
* line 1593 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->update_record_raw()
* line 624 of /lib/classes/task/manager.php: call to mysqli_native_moodle_database->update_record()
* line 312 of /lib/cronlib.php: call to core\task\manager::adhoc_task_failed()
* line 166 of /lib/cronlib.php: call to cron_run_inner_adhoc_task()
* line 76 of /lib/cronlib.php: call to cron_run_adhoc_tasks()
* line 61 of /admin/cli/cron.php: call to cron_run()

Exception ignored in shutdown function Closure::__invoke: جدول «task_log» وجود ندارد
Exception ignored in shutdown function tool_log\log\manager::dispose: خطا در خواندن از پایگاه داده
Potential coding error - existing temptables found when disposing database. Must be dropped!
PHP Fatal error: Uncaught ddl_change_structure_exception: خطا در اجرای DDL sql in /opt/bitnami/apps/moodle/htdocs/lib/dml/moodle_database.php:492
Stack trace:
#0 /opt/bitnami/apps/moodle/htdocs/lib/dml/mysqli_native_moodle_database.php(1072): moodle_database->query_end(false)
#1 /opt/bitnami/apps/moodle/htdocs/lib/ddl/database_manager.php(77): mysqli_native_moodle_database->change_database_structure('DROP TEMPORARY ...', Array)
#2 /opt/bitnami/apps/moodle/htdocs/lib/ddl/database_manager.php(332): database_manager->execute_sql_arr(Array, Array)
#3 /opt/bitnami/apps/moodle/htdocs/lib/dml/moodle_temptables.php(140): database_manager->drop_table(Object(xmldb_table))
#4 /opt/bitnami/apps/moodle/htdocs/lib/dml/moodle_database.php(405): moodle_temptables->dispose()
#5 /opt/bitnami/apps/moodle/htdocs/lib/dml/mysqli_native_moodle_database.php(610): moodle_database->dispose()
#6 /opt/bitnami/apps/moodle/htdocs/lib/dml/moodle_database.php(160): mysqli_native_moodle_database->dispose()
#7 [internal function]: moodle_database->_ in /opt/bitnami/apps/moodle/htdocs/lib/dml/moodle_database.php on line 492
sh: 1: /usr/bin/sendmail: not found

 

 

 

Average of ratings: -
In reply to Okhtay Gh

Re: Moodle consuming disk space and CRON task log

by Ken Task -
Picture of Particularly helpful Moodlers

If you have automated backups turned on turn it off.

Look in moodledata/temp/backup/ - manually remove any and all files folders in there.

Look in moodledata/trashdir/ - manually remove any and all files folders in there.

Running out of disk space if DB is hosted on same server (check config.php to find out of DB server is 'localhost') is serious business.  Tables can get corrupted.   Nothing in moodle code, cept the command line only scripts in code/admin/cli/ will give a hint or clue.   Do you have command line access to server - would hope so as your moodle is really a bitnami distribution of the entire AMP stack + moodle.

Some of the errors in your logs might be results of errors prior to other events - like the gs not found ... or sendmail not found.

By chance have you attempted to install any plugin/addon that isn't compatible with your core version of Moodle?

Can you still login as an admin level user?

'SoS', Ken



Average of ratings: Useful (2)
In reply to Ken Task

Re: Moodle consuming disk space and CRON task log

by Okhtay Gh -

Thank for the reply Ken,

The folder moodledata/temp/backup/ was empty.

Also moodledata/trashdir/ had 3.5gb files in the morning ( peak time of users ) and now has 7.5mb which I assume it is working as it should be ( Is CRON cleanup task running? ).

I have a whole stack on a single server and I have root access to the server and admin access to the site.

I checked all plugins versions and found 3 of them out of compatibility with my moodle version, therefore I removed them.

 

Average of ratings: -
In reply to Okhtay Gh

Re: Moodle consuming disk space and CRON task log

by Ken Task -
Picture of Particularly helpful Moodlers

In paths, add the path to php-cli.   That will enable 'run now' buttons under each scheduled task that is enabled.   Can run them at will via GUI then so you can see if that task has errors/issues.   Cleanups one should be able to run even during  prime time.

One can manually remove in a space crunch all files/folders in trashdir to gain space.

Ditto moodledata/temp/backup/ *if* autobackups not still running.  But one can remove failed autobackups with some CLI investigation and knowing what course is actively being backed up.

Check settings of for recyclebin ... defaults are to hide it in course menu unless there is something that teacher has deleted.  I'd set to display all the time so you can check.  When teacher removes a module in a course moodle makes a .mbz (a backup) and tags that file where it could be restored by teacher.

IF one has deleted a category or courses the .mbz's are there but there is no restore button anywhere.  You can set the length of time to retain recylebin files.

Once files have reached that time, moved to trashdir where it remains 4 more days before removed. (that time is hard coded)

select id,filename,filearea,timecreated,contenthash from mdl_files where filename like '%.mbz';

if filearea shows automated backup created by auto backups.
Otherwise ... that backup made by teacher or other admin.

Admins cannot see teachers private file backups.

If files show up in trashdir, something is working! smile

No magic bullet here ... each system set up is different, so it takes a little digging to figure out so admin can adjust settings, etc.

Also, see your site is a Bitnami .... not standard install etc. ... so suggest checking with others about issues that run the same stack via:

https://community.bitnami.com/c/moodle

'SoS', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Moodle consuming disk space and CRON task log

by Okhtay Gh -
Hello Ken,
Just putting some updates here.

I did few changes on my instance And I am not sure which one solved the issue big grin
 
  • I modified httpd.conf as the following

StartServers 10
MinSpareServers 10
MaxSpareServers 30
ServerLimit 1032
MaxRequestWorkers 1024
MaxConnectionsPerChild 5000

ServerLimit 48
StartServers 32
MinSpareThreads 1024
MaxSpareThreads 1536
ThreadsPerChild 64
MaxRequestWorkers 3072
MaxConnectionsPerChild 5000

  • I enabled a cloud-firewall before any traffic passes to my moodle and blocked a few untrusted IP ranges
  • Optimized high hit images mostly on the user's dashboard ( coarse image )

About the CRON task, with the help of Ken, I discovered where to observe them and they seem to be working fine. smile

Average of ratings: -
In reply to Okhtay Gh

Re: Moodle consuming disk space and CRON task log

by Ken Task -
Picture of Particularly helpful Moodlers

"Is CRON cleanup task running?" ... looks like it is.

How frequently do you have cron running?   Recommendation these days is once every minute.  Sound like too frequent? .... layman's explanation ...

Cron consist of task ... which are seen in scheduled task.   Those have, as the name suggest, schedules of their own if enabled ... some ASAP ... some delayed.

Sites that have cron running as it should are in better shape that sites that don't - well, yeah, duh!   Thought I'd say it anyway ...

So is your issue fixed?

'SoS', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Moodle consuming disk space and CRON task log

by Okhtay Gh -

Sorry Ken for the delayed reply.

Since the server is in the production phase, I have to wait for the prime time to pass ( mostly up to midnight ) in order to execute the commands and extract logs and analysis.

As you have already said, CRON tasks are running successfully and can be seen in the "Scheduled task". Thank you again for the reminder.

Now I am suspicious of a plugin that using up the disk space. Tonight, I will disable them one by one and will have a look at their directories to see if there are unusual logs or "~" files to delete. I the past 2 days, I have around 10gb of additional space in my disk which definitely is not the actual content.

Average of ratings: -
In reply to Ken Task

Re: Moodle consuming disk space and CRON task log

by Gilbert Loyogoy -
Hi Ken , sorry for interruption is it safe to delete files and folders inside the backup, temp, and trashdir ? Cause we are having a full storage problem
Average of ratings: -
In reply to Gilbert Loyogoy

Re: Moodle consuming disk space and CRON task log

by Ken Task -
Picture of Particularly helpful Moodlers
@Gilbert ... in a space crunch ... yes, it is 'safe' to manually remove any/all files in moodledata/temp/backup/ Cron jobs/task list has a cleanup for that. Any files/folders in moodledata/trashdir/ can also be safely manually.
Best, of course, if the task that clean up those areas work on their own.
Set the path to php-cli in admin paths settings and on the scheduled task for cleanups there should be a 'run now' link which will execute just that scheduled task. Any errors should display .. .if it does show errors, please provide them for further cussing/discussing.
'SoS', Ken
'SoS', Ken
Average of ratings: -
In reply to Ken Task

Re: Moodle consuming disk space and CRON task log

by Clay Asbury -

@Ken Task - please pardon the interruption, but I've got nearly an identical situation as @Okhtay Gh.   Ken, would you be willing to review offline with me?  Happy to pay for your time.  

I setup Moodle for a school in NC in August and it's worked well for them, but disk space is becoming an issue.   I do a fair amount of front end web development but I'm a little unclear on how to manage storage/backups.   

Average of ratings: -
In reply to Clay Asbury

Re: Moodle consuming disk space and CRON task log

by Ken Task -
Picture of Particularly helpful Moodlers
Sent you a PM - contains how to contact via voice.
As a prep for conversation when you call, please PM to me url to your moodle.
'SoS', Ken
Average of ratings: -