Reduce Moodle's disk usage

Reduce Moodle's disk usage

by Eric Phetteplace -
Number of replies: 10

We're about to see a big increase in Moodle usage as all our classes move online. I want to free up disk space on our storage drive, which is nearly full. Unfortunately, Moodle doesn't really seem to delete things when you delete them, or at least the disk usage on the mounted file store never decreases. I'm trying to backup old courses before deleting them, but this actually increases disk usage. I'm mostly using moosh commands to automate things and we're on Moodle 3.2. Here's what I've been doing:

- identifying courses to backup, then running `moosh course-backup $ID`, zipping together all the course backups, moving the archive to permanent storage, removing the individual course backup files

- this appears to create a number of "mbz" Moodle backups in the "files" database table even though moosh's backups were in a different location, so now I have to track those down and delete them with `moosh file-delete $ID` then follow up with `moosh file-delete --flush` to empty the trashdir

- in the admin interface, delete the parent course category for that semester and when it asks what to do with child objects select "delete all - cannot be undone"

Am I missing something? Why does the disk usage stay the same or even increase after these activities? Are related files like assignment submissions not deleted when a course is deleted? And if so, how would I go about finding and actually deleting these files (not just moving them to the trashdir but actually removing them from the disk)? Please help!

Average of ratings: -
In reply to Eric Phetteplace

Re: Reduce Moodle's disk usage

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Due to some strange design decision that I could never understand, Moodle waits several days before deleting files using a background task. It can take up to 4 days. Or you need to use Moosh.
Average of ratings: -
In reply to Howard Miller

Re: Reduce Moodle's disk usage

by Eric Phetteplace -
What moosh command? I am already using `moosh file-delete --flush` but that doesn't seem to do anything. I'm beginning to suspect that the recycle bin is somehow involved but I don't know how to empty it using the command line .
Average of ratings: -
In reply to Howard Miller

Re: Reduce Moodle's disk usage

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Years ago, there was a steady stream of people asking how to restore something they had accidentally deleted, so the deleted file/course/resource, etc., four day residence in the recycle bin, was implemented. Hate to admit that it has saved me on more than one occasion, as recently as last month. As many users discover, the downside is a temporary increase in disk usage, until the files are actually deleted, four days later.

As to running out of disk space when self hosting, with today's "cheap" disk drive prices, it is almost inexcusable to run out of space. It should be possible to replace a current drive with a new, LARGE, drive, with just a relatively short downtime for the copy/move data process that would be required.
Average of ratings: Useful (1)
In reply to AL Rachels

Re: Reduce Moodle's disk usage

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think this predated the recycle bin. Files were marked for deletion but left for a few days. However, due to the way Moodle hashes files your chance of actually recovering it was minimal (and required knowledge of the file system)

I always argued that it was a mistake not to have an "empty the trash" option - as you could be deleting files because you had run out of space.
Average of ratings: Useful (2)
In reply to Eric Phetteplace

Re: Reduce Moodle's disk usage

by Ken Task -
Picture of Particularly helpful Moodlers

Check out your settings for recyclebin

https://docs.moodle.org/38/en/Recycle_bin

Set that to display all the time and one should be able to 'delete' more quickly.

When backing courses via command line there is a backup.php in admin/cli/ and it has an option of saving to an alternative directory outside of moodedata.   If that directory was a mount point for something like attached storage the .mbz ends up there.

Moodle will still use moodledata/temp/backup/ to build the .mbz's (so there is additional space used) and if any backup process fails for some reason, only the clean up task will remove stuck contenthashed named directories there.    You could however manually remove any/all in moodedata/temp/backup/ if in a real pinch.

Then, there is also course backups made by teachers/others that you cannot see as admin.  For those, db query of md_files to find contenthash and then a find from commandline in moodledata/filedir/ for the location ... like ./xx/j7/xj7blah and finally manually removing, then removing the row in mdl_files table that referenced that .mbz file.

How do I know all this?   Have had the displeasure of working on systems that ran out of space before ... automated backups had been set to keep all and were not sent to a designated directory! :\  ... not fun ... took hours.

'SoS', Ken


Average of ratings: Useful (3)
In reply to Ken Task

Re: Reduce Moodle's disk usage

by Eric Phetteplace -
Thank you, that was helpful, mostly just to confirm that I'm not missing something. It appears that savings disk space immediately, and not in two weeks when a series of recycle bin and temp directories are cleaned up, means:

- delete a course or category of courses
- look in the files database table and delete all the backups created under the "tool_recyclebin" component
- delete the temp/backup folder in the data directory
- delete the trashdir (or run `moosh file-delete --flush`) in the data directory

I've disabled recyclebin entirely for now and that (sort of?) seems to help, it lets me skip the second item above which is the most tedious to do anyways.
Average of ratings: Useful (1)
In reply to Eric Phetteplace

Re: Reduce Moodle's disk usage

by Mawan A. Nugroho -

Within 6 months, Moodle took up hundreds of GB of disk space. Upon investigation, this was caused by assignfeedback_editpdf plugin. I have reset some courses, but the storage space is still full.
From this case, I learned several lessons:

  1. Never activate the assignfeedback_editpdf plugin.
  2. If the assignfeedback_editpdf plugin has been activated and used by students and teachers, then uninstall this plugin. Moodledata immediately shrank in size. Don't worry, this plugin will be reinstalled by Moodle. Once installed, immediately deactivate this plugin.

Sorry, my English is bad.

Greetings from Tangerang, Indonesia.

Average of ratings: Useful (2)
In reply to Mawan A. Nugroho

Re: Reduce Moodle's disk usage

by Swen Dahlhaus -

Hi there, thanks for your input so far smile

We're facing issue 2 you wrote about and would like to know:

1. Do you know whether already uploaded pdf's handled by the plugin will be lost upon uninstalling assignfeedback_editpdf plugin?
Our issue is the vast amount of copys that are generated, but we need to keep the originals.. 

2. Would you by chance know whether deinstalling the assignfeedback_editpdf would interfere with the actually used mod_pdfannotator?

looking forward for your reply smile

Average of ratings: -
In reply to Swen Dahlhaus

Re: Reduce Moodle's disk usage

by Zahid Castaneda -
Did you get any feedback or tried uninstalling the plugin?

I am having some problems with storage as well.
Average of ratings: -
In reply to Zahid Castaneda

Re: Reduce Moodle's disk usage

by Swen Dahlhaus -
No, no feedback, but I have unstalling the plugin. No Problems anymore. After some tries, you have to uninstall it 3 or 4 times, maybe in cause of many data, the plugin was gone. Moodle reinstalled it automaticly, but if you dont allow the plugin to make Copys in the Admininterface no new storage problems anymore.

Greeting from germany
Average of ratings: -