[3.9] Automated backups consumes all disk space and trashdir is full

[3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -
Number of replies: 23
Hello,

We have updated our Moodle from 3.7 to 3.9.1
We have automated backups run by crontab command every Saturday morning and it finishes in Sunday afternoon/evening.
We keep 4 weeks backups in Moodledata and in specific directory (.mbz files), we skip courses not modified for 30 days.
So basically every week we make ~2200 backups and it's ~240GB.
But as we keep 4 weeks backups so basically oldest 2200 backups (~240GB) is deleted. As I understand moved to moodledata/trashdir and scheduled task should delete it from trashdir?
We have this setup for years. Currently ~600GB free disk space and this happened second time after Moodle update.

Problem:
This Saturday morning automatic backups started and Sunday afternoon we couldn't login to Moodle (Database reading error), so I checked server via SSH it was 0% space left on disk, then I checked size of moodledata/trashdir it was 600GB there..
In Filezilla I saw that moodledata/trashdir folder has 256 directories. Which is strange to me, because it seems that files are not being deleted fast enough?

We run our cron every 60 minutes.
In scheduled tasks "Cleanup files in trash  \core\task\file_trash_cleanup_task" runs every 6 hours by default, and from the logs I see that it was running without problems on Saturday and Sunday.
As I understand this task should delete files from moodledata/trashdir?

So why files are not deleted from trashdir and why it takes so much disk space?



Because as I said we also make .mbz backup files to specific folder (separate disk), so there is no change in that disk space. There is no huge swings in .mbz file size, maybe few GB.
Last week  was made 2254 .mbz backup files and 245GB, and two weeks ago it was 2206 files and 236GB

So for some reason when automatic backups starts Moodle put a lot of files to trashdir but file_trash_cleanup_task is not capable deleting so many files at a time?

Thanks,
Marius.
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Ken Task -
Picture of Particularly helpful Moodlers

An area to check config for:
Administration -> Server -> Tasks -> Task processing

What does
Site administration -> Server -> Tasks -> Task logs
when filter set to 'auto' and drop down set to Success show?


Compare \core\task\automated_backup_task with \core\task\backup_cleanup_task
is clean up romping on backups?

backups will use moodledata/temp/backup/

What, if anything, is in that directory or than 9 byte .log files?
Are there hash named directories and inside those a backup.mbz with
all the xml's and related directories?

There has always been a hard coded number of days to wait before removing anything from moodledata/trashdir/

"the draft files are deleted from the files table after 4 days, the trashed files (not present in files table) are deleted after 24 hours. This is intentional."
From https://tracker.moodle.org/browse/MDL-25592

Also remember that moodledata total space usage is affected by other modules/actions/activities ... like assignments and annotation/grading of assignments - this to say with a space crunch other areas might be involved ... like a teacher that decides to make their own backups of courses ... goes to user private files which admin level users cannot see.

In a space crunch ... like running out of space ... one can manual remove any/all files folders from moodledata/trashdir/ without issues.

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -
Hello Ken,

Administration -> Server -> Tasks -> Task processing all values are set to defaults. Scheduled task concurrency limit: 3, Scheduled task runner lifetime: 30min,  Ad hoc task concurrency limit: 3,  Ad hoc task runner lifetime: 30min.

If I go to Site administration -> Server -> Tasks -> Task logs and search for automated_backup_task Success and backup_cleanup_task Success
For both I get 24 lines and the Start time is almost the same, in some lines time is different in 1 minutes, but mostly it's the same Start time.

In moodledata/temp/backup currently there is 3500+ .log files from dates Aug 17th to Aug 24th. But all files are 0 byte size.
Also there is 12 directories with dates Aug 17 & Aug 23. Inside these directories there is xml files, and other folders like sections,course,files,activities.

Yes, I understand that moodledata space usage if affected by assignments, teachers uploading new files/deleting old ones etc. But this is still strange because this never happened before and we had like 200-300GB free space and we didn't face this problem. Now we have 600GB free space and we ran out of space two times and both times was during weekly course backups in the weekend.
Semester is not even started so there is not much action in the Moodle system right now, so I highly doubt about teachers taking backups, uploading tons of files, etc.

Today in the morning weekly backups finished (as I deleted files manually from moodledata/trashdir so automated backups continued and finished this morning). So I checked on our Specific location where additional .mbz backups are create, so for this week it was 2370 courses and total .mbz backups size 248GB.
So I suppose in Moodledata backups takes similar size and oldest backups are deleted when new are created, so there should be no problems.. Maybe in Moodle 3.9 something changed in backups/deletion process?

Thanks,
Marius.
Average of ratings: Useful (1)
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Ken Task -
Picture of Particularly helpful Moodlers

"For both I get 24 lines and the Start time is almost the same, in some lines time is different in 1 minutes, but mostly it's the same Start time."

Think about that ... same start time?

"Also there is 12 directories with dates Aug 17 & Aug 23. Inside these directories there is xml files, and other folders like sections,course,files,activities."

Where there any backup.mbz files in those directories?

If so how large were they? 

If there wasn't a backup.mbz nor a moodle_backup.xml then that course backup failed for some reason.   View moodle_backup..xml file to see course name then go visit that course to see what's in it.  Those can be manually erased without consequence.

"Semester is not even started so there is not much action in the Moodle system right now, so I highly doubt about teachers taking backups, uploading tons of files, etc."

Seems to me that would be a very good time for teachers to be active in their courses ... including making a backup.   But ... only you would know that about your own site ... if you'd been monitoring and have some historical info.

That would also mean that courses haven't changed .. right?   Why would you need another backup of that course if it hadn't changed?

Is the finish time consistent?

Believe you said your cron set to run every 60 minutes?   Think the recommendation is to run it once every minute now.  Everything that cron does won't be executed at each run ... task list.

You might also consider increasing those time outs ... slightly higher.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -

Compare \core\task\automated_backup_task with \core\task\backup_cleanup_task:
Tasks

These course backups in /moodedata/temp/backup/ maybe failed because we ran out of space. At the moment I see only 4 directories, so other 8 courses was completed or was deleted automatically. For these 4 remaining courses only two had moodle_backup.xml file and checked these course, they had automated backups done last week. The other ones maybe had to be course backups but didn't started because run out of space? Just my guess...
Currently /moodledata/temp/backup/ is only ~300MB.

I would like to get back to /moodledata/trashdir/ then.. So now in Filezilla I see that it has 256 directories.
And this is the maximum directories size which can be as I understand..
Trashdir

And I am sure that these directories at 4-5AM for sure are not teachers who deleted something, so these are the oldest backups files from moodledata which was deleted because new was created. So even if this is old backup it will stay in trashdir for like 24hours or so?

Or if we would run our cron.php every minute it would get deleted much faster?

Also about which timeouts are you referring to?

Thanks,
Marius.

Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Richard van Iwaarden -
Picture of Particularly helpful Moodlers
Marius, I have had the exact same problem a couple of times with different Moodle versions. The problem has always been the recycle bin, not the automated backup. What I do now is 1) disable the recycle bin and 2) delete everything from the temp backup folder.

After a few days I turn the recycle bin back on and the problem seems solved for some reason.

I know this is a bad solution and many other people will find this problem (it has been reported several of times) but it solves it for now.
Average of ratings: -
In reply to Richard van Iwaarden

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -
Thanks Richard, but if you disable recycle bin and later enable it, so all content which teachers deleted will be lost?
Because recycle bin is enabled in our Moodle and we set to keep files for 30 days, so if teacher deletes something we don't need to restore all course/item from backup file, he can find his deletes files in recycle bin for 30 days.

Marius.
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Richard van Iwaarden -
Picture of Particularly helpful Moodlers
I don't know if it's all lost. Could be. I just do it like this, because else my whole Moodle crashes. And ours is huge. We are talking many terabytes and if the recycle-bin crashes like you describe in you opening post, our site generates several terabytes per day. Could also be related to the 20.000+ courses we have.
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Ken Task -
Picture of Particularly helpful Moodlers

PHP settings for time outs.  Site Admin Menu - Server - PHP info.

All of this controlled by settings in moodle + what code has in tables.

If it is recyclebin, run this query via mysql client:

select userid,contenthash,filename,filearea,filesize,timemodified from mdl_files where filename like '%.mbz' order by filesize DESC

filearea will show recyclebin if a teacher deleted something ... userid tells you who.

One could turn the recyclebin link display to all the time displayed ... default is to hide it unless there is something in recyclebin.

That way ... one could go into a course and in a course admin menu you could inspect what's in recyclebin ... and force deletion of each item.

The recommendation of running cron job every minute has been the recommendation for a long time.   Not every part of cron/task list will run each time.   The only way I've found to get info/handle on this is to have the cron job redirect to a moodle_cron.log file in /var/logs/ so I could watch in realtime/at any time with tail -f /var/log/moodle_cron.log.

I would hope teachers are cautious about deleting anything in their courses ... keeping for 30 days might be too much 'hand holding'.

What all must understand ... moodle is a shared resource ... by that I mean, a teacher could be using up more than their 'fair share' in a moodle.   Sometimes that is due to nature of course ... obviously a course in digital media (audio/video/etc) is gonna use up more space ... backing up a course is good practice especially after a major assignment due, but is there a reason to keep 10 backups in teachers private area files?

'SoS', Ken

Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
Have you all seen this checkbox in the automated backup settings? This is a new setting that I haven't seen before in Moodle 3.5.x, maybe in a later version it's already included?

backup | backup_auto_files

When it's checked, all the course files (in moodledata) are also included in the backup. I haven't taken a further look at this point yet, but I'm going to test our new upgrade with and without this setting checked to see the difference and if more data is stored after the automated backup.
Average of ratings: -
In reply to Alain Raap

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Richard van Iwaarden -
Picture of Particularly helpful Moodlers

That is interesting.

But what happens in this scenario:

  • auto-backup enabled
  • Teachers deletes course X which holds files X, Y, Z (it is the only course that holds these files)
  • Backup is restored (but only contains hyperlinks - not the actual files)

Will everything still be there? And if so... would it mean that step 2 (teacher deletes files) doesn't really delete files?


Average of ratings: -
In reply to Alain Raap

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Richard van Iwaarden -
Picture of Particularly helpful Moodlers

Answered my own question I think:



So that would mean that if I increase 'filescleanupperiod' to say 30 days, I can run autobackup with the setting not to include files... theoretically.

Average of ratings: -
In reply to Richard van Iwaarden

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
Richard, I think it’ll cost you a lot
of diskspace when you increase that number. Maybe try to test it as I said, what’s the difference in diskspace used when the checkbox I mentioned is enabled/disabled?
Average of ratings: -
In reply to Alain Raap

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Richard van Iwaarden -
Picture of Particularly helpful Moodlers
I don't know, let's see what Ken thinks. 'In the spirit of sharing' smile

We should think of two things:
  1. What if file x,y,z are in every daily backup stored
  2. OR... what if I keep deleted files for 30 days

I have no clue as to what would take more disk space...

Average of ratings: -
In reply to Richard van Iwaarden

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Ken Task -
Picture of Particularly helpful Moodlers
At this point, I don't know what to think ... do know there is no recycle bin for deleted courses ... only resources deleted from a course.

Think I'd set recyclebin button/link to display all the time ... normal behavior is to display the link in course admin menu only when there is something in recyclebin.   Go to one of these troubled courses and see if there is anything in recyclebin.

The only way to know for sure if files that were linked in a course are used else where in the site is via direct mysql query of mdl_files table.

And the only way to know for sure if a file is in a backup is to copy the mbz file out to some test area, uncompress it. Then check files.xml (humanly recognizable name + contenthash value seen) against what one finds in the uncompressed files directory.

The files.xml will show a path to a folder structure like that of moodledata I think 2 levels deep ... for finding using contenthash value seen in files.xml ... 1st two letters/numbers is a subdir in files folder ... 2nd two letters a subdir in the first subdir ... then the file which should be named the same as the contenthash value. Do file -b contenthashfilename to see if it's mimetype.

I do have a 3.9.highest sandbox using auto backups ... no issues .. of which I am aware ... but then again those courses would be small, etc. so ... not fair to compare.

moodledata size increases ... failed backups where those failures are not removed/cleaned up will take up space ... recyclebin files which are stored in the sea of files ... moodledata/filedir/

Query mdl_file table looking for any file using component recyclebin.

Fraid there is no admin tool to sort this out ... just a lot of inspecting various things ... only tool that does come to mind is moosh ... it does have optional commands for finding 'orphaned' files and checking that meta data in mdl_files table matches what can be found in moodledata/filedir/

'SoS', Ken
Average of ratings: -
In reply to Ken Task

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Colin Fraser -
Picture of Documentation writers Picture of Testers
I have no idea how the IT guys set up backups or how they handle this issue, I've not been a part of that. We actually don't delete courses quickly, the end Semester 1, all semester long courses are reset, then archived. At the end of each year, all courses are reset and archived to the same Category, e.g. Archive2019. (Our school year is the same as the calendar year in Oz.) Those courses to be repeated the next year are brought forward, those courses not repeating, and there are always a few, remain for future reference, or as resources for others to use. All teachers have access to all courses in the archive, they are set as non-editing Teachers at the Category level, I was lazy when setting it up. At the end of each year, the earliest Archive we have is deleted entirely so a course remains within the Moodle for 5 years. Works for our need.
Average of ratings: -
In reply to Alain Raap

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -
Hello Alain, we have this option enabled in automated backup settings, and 'filescleanupperiod' is set to 3 weeks.
However as I mentioned as we are doing automated backup to moodledata + specific directory (mbz files), so checking specific directory I can see that every week is similar number of backups made and it's about 240-280GB. So I assume that backups in Moodledata uses more or less same space. So each week ~240GB of new backups are stored in moodledata and ~240GB should be deleted.
We have 600GB free space if moodledata/trashdir and moodledata/temp/backup is empty.
And this weekend I checked moodledata/trashir after automated backups finished so we had ~540GB in trashdir and only 60GB free space.

Also now we run our cli cron every 5 minutes. So I don't understand why moodledata/trashdir are getting so big when we start automated backups on weekends.

Even if oldest backups are moved to trashdir and deleted only after 24 hours it should be plenty of free space left as new backups size are the same every week ~240GB..

Marius.
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
Hi Marius,

I think your filescleanupperiod setting could be the problem that your trashcan stays that big as you describe.
I see this is also a new setting that wasn't present in Moodle 3.5. I find it strange that during your automated backup process the trashcan is growing, only files that are deleted are backups inside your .mbz directory and not files in your moodledata folder, or am I wrong? I thought the backup process only deletes the older backups and the temporary course in the backup folder. But these files aren't part of the moodledata folder.

I run the automated process outside Moodle cron and with the Linux cron that runs a separate script I wrote for the automated backup. This script runs every day and excludes the course that runs 10 hours to make a backup.

Here you can find my post about this subject, because of NFS we used I ran the automated backup process on the DB server, it was much faster then. Now we don't use NFS anymore, the automated backup runs on the Moodle server again:

https://moodle.org/mod/forum/discuss.php?d=383815

I included the current shellscript that runs on our Moodle server, I changed the automated backup_auto_active in 'manual', to run the process outside Moodle cron.
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
I found this thread yesterday, maybe it helps finding the cause of your problem:

https://moodle.org/mod/forum/discuss.php?d=409314
Average of ratings: -
In reply to Alain Raap

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -

It seems that old files is not being deleted from moodledata/trashdir even after 24 hours (as Ken mentioned it should be).

Last week everyday I checked moodledata/trashdir size, so on tuesday, wednesday, friday it was 37-39GB.
Then automated course backups started on saturday 4AM and finished on ~11AM. I checked moodledata/trashdir on saturday and it was 290GB.

So as I mentioned weekly backups is around 240-280GB, so this is the number we get. On friday it was ~40GB + ~250GB after automated backups on sutarday and we have 290GB in moodledata/trashdir.

I checked on Sunday, on Monday and there was same 290GB used in moodledata/trashdir. So somehow oldest backups which are moved to trashdir is not being deleted even after 48 hours.
Server has access to moodledata folder, it's set to 777, so it's not permission issue.


Also I believe something has changed in Moodle backups process because now backups are made "randomly".
Can someone check in their Moodle and confirm that you have the same process?

Before update to 3.9.1 automated backups was made from the first course (id=1) to the last course (id=xxxx).
So using filezilla I could short mbz files by time created and all course were ordered by increasing id.


And now after update to 3.9.1 automated backups are being done "randomly". For example first automated backup done on Saturday 4:00AM was ID=1614, next was id=857, etc. It was done 2930 automated backups.

Screenshot below:
Courses created in random order?


Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
Hi Marius,

I'm also testing the automated backup process at the moment and you're right about the random creation of backups. I see these messages
in the automated backup log:

Putting backup of Startopleiding in adhoc task queue ...
complete - next execution: Thu, 10 Sep 2020 22:00:00 +0100
Putting backup of Communicatieve vaardigheden in adhoc task queue ...
complete - next execution: Thu, 10 Sep 2020 22:00:00 +0100

Adhoc task queue wasn't present in a previous version. Just as you mention, backups were made from increasing courseid 1 to last courseid
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
Here I found a tracker about the adhoc task and the automated backup task:

https://tracker.moodle.org/browse/MDL-65729 in https://docs.moodle.org/dev/Moodle_3.9_release_notes
Average of ratings: -
In reply to Alain Raap

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Marius S -
But as I see this bug was in 3.7, 3.8 and was fixed in 3.9. So as we are using 3.9.1 I think this should be already fixed.
Did you notice much faster automated backup process in Moodle 3.9?
Because now we get automated backup done in like 6-7 hours. On Moodle 3.7 it was about 28-32 hours I believe.
But we also moved to new server, so not sure if it's because of server or because of new automated backup process.

Update about moodledata/trashdir, I checked yesterday and it was empty (well ~200mb). So basically it took ~4 days to delete oldest automated backups from moodledata/trashdir.
I will try to check next week if it will be the same situation.
Average of ratings: -
In reply to Marius S

Re: [3.9] Automated backups consumes all disk space and trashdir is full

by Alain Raap -
Picture of Particularly helpful Moodlers
I haven't tested with a large number of courses yet to check the performance of the automated backup task.
Our automated backup task takes about 10 hours to complete in the present Moodle version (3.5).
What I noticed is that when I added new courses to my new Moodle 3.9 environment, they weren't backup'ed
the same evening but they were scheduled for the backup run the next day.
Average of ratings: -