Ubuntu/Moodle drive inexplicably and rapidly filling

Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
Number of replies: 24

When I last checked (via Webmin) the local disk space on my VPS (which was some time on Tuesday afternoon) I was at 62% (of 143/158 GB) full, which is a good amount for the start of an academic year.

When I checked today, it was at 95% and filling rapidly, despite no backups having been scheduled, no (planned) additions to the Websites or disks. As I was trying trying to puzzle out what had happened, I could see the percentage full creeping up until, at one point it hit 100% and Moodle crashed.

I was able to delete about 200 MB of media files from the repository, at which point I was able to put Moodle into maintenance mode via command line.

I could find nothing amiss, however: I used Moosh file-dbcheck to locate some orphaned files and deleted those using the FileTrash plugin, then deleted a number of backups of course backups and successfully got disk usage down to 95%, but I can find no processes running that should be not,

At present, I've taken Moodle out of maintenance mode and disk space is not running out on its own, but I am at a loss regarding how to regain the lost space, what's occupying it, and how to prevent its being filled again, presuming I am able to recover the space.

General specs:

  • Ubuntu 20.04.4 on virtual private server (Kernel 5.15.0-22)
  • NginX 1.21.4
  • Moodle 3.11.5+, build 20220308
  • mariadb 10.5.15
  • php 7.4.3
  • all recommended extensions installed and without problem reports
  • 47 Additional plugins, but none that have proved problematic until now

Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
What do these commands bring?

$ cd /path/to/your/moodledata
$ du -sh *
$ df -h
What is the promised disk quota by your VPS hosting?
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
du -sh *
  • 4.0K antivirus_quarantine
  • 12M cache
  • 8.0K CourseArchives
  • 140K environment
  • 77G filedir
  • 64M geoip
  • 27M lang
  • 8.7M localcache
  • 1.1M lock
  • 16K models
  • 32K muc
  • 3.3G repository
  • 28K sessions
  • 2.9M temp
  • 2.2G trashdir
[relevant part of] df -h
  • /dev/sda        157G  143G  6.4G  96% /

Promised disk size is 160 GB.

It's the /path/to/data/filedir that has exploded. Overnight, at least, there's been no more upward creep in the /filedir size or disk usage, but it's virtually unusable now, and I still have to add four more courses and maybe 500 new accounts even before the academic year begins here.

At the end of the previous academic year (before I archived two years' of courses to different computers) and deleted about 2000  students total disk usage was about 82%. After cleanup it was at about 62%, then went to 100% with no action on my part.

du, Webmin, and my account page at the VPS provider's site all show the same information.


Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
A couple of observations.

$ du -sh /path/to/moodledata/* total to about 83 GB. Where are the remaining 143 - 83 = 60 GB? May be you have to navigate heavy directories starting at '$ du -sh /your/root'. /your/root is what '$ mount' will show against /dev/sda.

"It's the /path/to/data/filedir that has exploded" Are you sure. Do you have records of its development in the past?

If it is really data/filedir, du you have Site administration > Courses Backups > Automated backup setup: Active Enabled? If yes, what is in Automated backup storage? If disabled now, did you have it enabled in the past?

Do you know what are the 3.3 GB in data/repository ($ ls -lR | less)?

2.2 GB in data/trashdir is not good.  Also '$ls -lR | less' in it. Any chance of your hosting is under powered?

But their total of 6 GB is 1/10 of the unaccounted 60 GB!

Warning: As a shared hosting may be your database is outside of the disk which overflows. But remember, disk full will damage the database irrevocably. Even it wasn't, the database and the content of data/filedir may go out of sync, making Moodle unusable. That said, did you have such a crash in the past?

Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
  • Automated backups are set up and have been for the entire 11 months and two weeks this Virtual Machine has been running. Everything that should be in the Automated backup is supposed to be there.
  • The repository directory is fine: only what is supposed to be in there is in there, and nothing extra or unused is in there.
  • No, I have never had such a problem in the past, neither when I was using a hosted system nor when I was working with a VPS.
Previous sizes of /moodledataarea/filedir/:
  • 1 December 2020: a little more than 25 GB (average of several backups from a VPS that had been running for two academic years)
  • 1 April 2021: <1 GB (before academic year began and after fresh Moodle installation/VPS spin up)
  • 22 February 2022: < 15 GB (after the effective end of the academic year)
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

Where are automated backups saved?  And other settings for auto ... how many for each course? etc, etc.   Study those options.

In your info on contents of moodledata directories, I see an 8.0K CourseArchives.  Are auto's going there?

That shows only 8.0k.

IF auto's are now going (for some reason) to default /moodledata/filedir/ that would explain increase of filedir, would it not?

Auto can be set to save in more than one location.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
  • "CourseArchives" is a now mostly empty directory to which I used to save some partial backups. It's so long ago I don't remember exactly what had been in there.
  • Auto backups go to a different directory in the parent directory the moodledata directory is in. Everything seems to be in order there. In those courses I've checked, there is nothing in the course backup areas. Backups are set to be recorded only in the autobackup directory and have been for a few years.

What I'm contemplating now is just spinning up a new VPS next week and restoring Moodle there. However, will Moodle reconstitute the filedir there automatically? I'll restore the courses—backups include files and user information—from my latest backups, but I'd really rather not do that (a) because if I don't know the cause of the problem I fear it will happen again and (b) I'll have to pay for the extra VPS for the time it takes to be sure everything's working again and (c) I won't be able to do my real work in the meantime.

Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
These are the steps I'm contemplating now:
  1. Start new VPS
  2. Install Moodle with Git
  3. Create new database with same name, users, passwords, etc. as the old one
  4. Replicate the /data directory in which current course backup directory and three unproblematic Moodle installations currently live. Make an empty directory for the problematic Moodle installation's dataroot
  5. Finish Moodle installation in browser
  6. Install plugins
  7. Restore contents of repository folder
  8. Replicate category and course structure from old, broken installation
  9. Restore all courses (including users and user data from latest backups (assigning the same course numbers as from the problematic installation)
  10. Copy old config.php to new system (but temporarily use "SCFG->wwwroot = http://[bareIPaddress]" here?)
  11. Cross fingers

If everything works after I log in (using the IP address), I'll go back to the old server and move the currently unproblematic Moodle installations to the new server and point the domain names to the correct places.

I guess instead it might be easier to move everything that works to the new server, then set up another Moodle on the old one (if there's enough space) and see if I can't get the /filedir to behave normally.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

Just a couple of thoughts on your last plan here ... use a FQDN ... use a https site on new server ... using ip and http is like taking one step forward to only have to go back 2 to go forward again!

So from your description of latest plan ... a full site migration ... and changing DNS for the new IP ... same FQDN - which you could cheat upon using your workstations host file - new IP mapped to the same FQDN as old server and you wouldn't have to do much more ... cept your cert for https.

New server will have more space right????

Sounds like a plan! smile

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
>> New server will have more space right????
Nope. I can't afford it. I'm paying for everything from my own pocket. The next step up with servers is double the cores and double the disk size at double the price. I can add storage only in 10 GB units, but it'll still be money I'd rather we put toward our own lives.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

"... will Moodle reconstitute the filedir there automatically?" ... answer nope!

You would be doing a moodle migration to a new server ... backup of code directory, sql dump of DB and import of that dump into new DB on new server, and, minimally, a tar ball of moodledata/filedir/ transferred to new server and setup appropriately - web service + tweaks, DB server + tweaks, PHP version + extensions and settings ... a rebuild!

"Auto backups go to a different directory in the parent directory the moodledata directory is in"

So the following is what you showed in moodledata ... CourseArchives an old place for course backups that you say you no longer use for course backups.   Guess we'll have to trust ya when you say that cause no info on what might be in there!   So now you have a directory (unkown/named here) that's at the same level as moodledata.

  • 4.0K antivirus_quarantine
  • 12M cache
  • 8.0K CourseArchives
  • 140K environment
  • 77G filedir
  • 64M geoip
  • 27M lang
  • 8.7M localcache
  • 1.1M lock
  • 16K models
  • 32K muc
  • 3.3G repository
  • 28K sessions
  • 2.9M temp
  • 2.2G trashdir
Are you ever going to share a listing of the files in that other directory that shows size of the backups?   Can tell you from experience that backups of over 100Gig will struggle from command line and would probably fail if using Admin GUI and not tweaking some settings upward to accomplish.

"but I'd really rather not do that" then your justifications gave me a 'chuckle' ... uhhhh ... who are you trying to convince?   No offense, I hope not me!!!   Not my server.   Not my problem ... got problems of my own!!!! soooooooooooo. smile

But, if you can't solve the space issue with what you have now ... what are your options?   That question is just food for thought!

Yeah, moodle sites can get to a state where one has to bite the bullet and do X ... like it or not ... am gonna be facing one of those when my CentOS 7 servers reach EOL.    Won't be fun, but ....

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
The largest course backup I have to deal with now is 577 MB (full of a semester's worth of workshops, forum posts, and the like) and the smallest 67 MB (for a course that has not begun yet), for a total of 13 GB spread and about 65 files.
I've got only 10 GB of free space now, though. I feel like I'm trying to turn a car around inside a shipping container.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
> I feel like I'm trying to turn a car around inside a shipping container.

No, you have the foot on gas pedal and overlooking sign posts!

>> $ du -sh /path/to/moodledata/* total to about 83 GB. Where are the remaining 143 - 83 = 60 GB?
>> Any chance of your hosting is under powered?
https://moodle.org/mod/forum/discuss.php?d=432591#p1740767

I know, you have answered some of the questions. If you quote the question you are answering, it will be easier for me. See my preferred https://en.wikipedia.org/wiki/Posting_style.

Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
>>$ du -sh /path/to/moodledata/* total to about 83 GB. Where are the remaining 143 - 83 = 60 GB?
Accounted for:
  • (Automated backup files) 13 GB
  • Test Moodle installation A /moodledata: 67 MB
  • Test Moodle installation B /moodledata: 79 MB
  • Another Moodle installation /moodledata: 29 GB
  • Problematic Moodle installation /moodledata: 80 GB (has fluctuated between 1 GB when the VPS was first started and about 25 GB on 7 January 2022, two days before the data explosion)
  • The remainder are system files, the only ones larger than 1 GB being /var (16 GB), /usr (3 GB), and /snap (2 GB).
>> Any chance of your hosting is under powered?
I have never had problems with it before: it's a shared CPU with 4 cores for my installation and 8 GB memory. There have never been more than 40 users logged in at any given time, The network can handle 5 Gps in and 40 Gps out, and the VPS is on a physical machine in the same city all of the users are. The VPS company always shows ping speeds of less than 20 ms (and usually less than 15 ms), and ping from my home ip (going through a VPN) average less than 15 ms.


Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

Hmmm ... first we've heard about having 4 moodle installs!
The one you describe as 'Another Moodle installation: 29 Gig.
Obviously, you have command line ... and there was a suggestion
that moosh (a command line only tool) might help.  Guess you didn't
try moosh cause of in-ability to install software like that.

So your next trip to your terminal, issue the following:
env |grep 'PATH=' [ENTER]

is there a /customeridlogin/bin/ at the tail end of that path?

If there is, you can manually create the bin directory, and inside there, install moosh - as well as use ln -fs command to make it easy to use 'moosh' as a command without path.   Moosh has a couple of commands that could disclose orphaned files in moodledata/filedir/.  orphaned files have no info in DB mdl_files table and cannot be seen in moodle.

At any rate, the issue is ballooned moodledata ... need more space.  Gonna suggest that test moodles in a crunch for space might need to go - every little bit one can gain space wise takes precedence over tinker/testing - since you've also ruled out spinning up another server - that also would cost.  So we're working on getting things fixed on the server you have.

Trying to help here, but the target seems to be moving! :|

'SoS', Ken

Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
The interrogation was aimed at distinguishing between:
Case a) something "technical" is wrong. Means hardware, system software, Moodle, supporting software or in their settings are not doing what they are expected to do.

Case b) the swelling is genuine, in the sense, it is the amount of data you either uploaded or instructed the software to create.

We can only help if the case is a), showing you how to follow the evidence and dig the cause up. In the case of b) no technology is needed, only some basic arithmetic. From your previous post I conclude it is case b). So all you need is "mathematics" like if you delete 2.2 GB you can't create a 26 GB Moodle instance. For that you need to delete 26 GB data or buy 23.4 GB of disk quota. Mathematics and logic at that level.

Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

2.2G trashdir

You can manually remove/erase what is in trashdir and gain 2.2 Gig of space right now.

In temp of moodledata 2.9M isn't much and could also manually remove/erase what's in there but suggest taking a look at what is there.  In temp, is there a backup directory?   What does it contain?

Don't think you've mentioned if you have autobackups turned on or not.   If on disable for now.

When you run cron from command line, the last thing seen is ad-hoc task.   Is there any ad-hoc task pending.   Those are jobs that didn't complete for some reason the last cron run.  Among those are several 'clean ups'.

How will you be creating those 4 new courses?

You could install moosh.  It's command line only and has a few commands relating to files - one finds orphaned files ... present in moodledata/filedir/ but not in DB mdl_files table.

https://moosh-online.com/

https://moodle.org/plugins/pluginversions.php?id=522

https://moosh-online.com/commands/

'SoS', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
Removed the files in trashdir and gained 2% free space.
No adhoc tasks pending, and database shows none, either.
Installed moosh already and ran through the orphaned files, though there were not many; also ran the File trash plugin.

As for the 4 new courses, three will be created by copying extant courses and one from scratch.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
Update: I've got the disk down to 34% capacity, I presume by deleting about 80 old courses (after backing them up) and perhaps by deleting a few thousand users and, so far, there has been no repetition of the rapid upswing in amount of disk used.

Moosh file-datacheck gave about 35 files orphaned in the filedir, but none of them was large (max about 8kb) and I knew what they were and why they were orphaned. Moosh file-dbcheck shows no problems now.

I'm still mystified, however, regarding why I was able to watch the local disk space graph on Webmin go up a percentage point every minute or so until reaching 100% and crashing both Moodle and Webmin.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

Answer: Recyclebin - it's a hidden 'place' when there are massive deletions of courses.

Course deleted - goes to recyclebin backup - since there is no link to display courses in that state, can't hurry them along to trashdir where the cron job cleanup task finally removes the files.

Default for recyclebin is to keep deleted course mbz's for 1 week.   If one sets that to a lower time span ... I did one today for 1 second ... ran the clean up task more than once to finally get those .mbz's to trashdir where I could manually remove anything in trashdir.

But ... congrats ... advise checking settings for recyclebin if/when in the future you do any massive clean up in the future! smile

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
I'm afraid that wouldn't be the answer because I did not delete the courses until after the problem happened. I did it while attempting to fix the problem.
When I finished working on 8 March (my time) everything was OK inasmuch as the disk was at an acceptable level. Some time the next morning, the disk started filling, moving up a percentage point or so each time Webmin refreshed until everything crashed.
I did search for hidden files on the system using ls -l .?* and different variations of that, along with some finds with various flags. I didn't find anything that seemed to not belong. I'd also tried running cron from the command line several times to move things along, turning off automated backups, and so on. I never got any adhoc tasks running.
Everything is backed up in multiple places on and off site and on and off shore, for that matter. I'll keep an eye open for any further inexplicable disk-space eating and probably start a new machine next year.
Thank you for the fast and thorough help.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Mathew Gancarz -
Picture of Core developers
Hi Doug, you could try this report to see if there's a specific user or course that might be consuming the extra space: https://moodle.org/plugins/report_coursesize
Average of ratings: Useful (1)
In reply to Mathew Gancarz

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
That was something I used and checked. No course seemed to be using substantially more space than expected.
Average of ratings: -
In reply to Doug E. Wray

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Ken Task -
Picture of Particularly helpful Moodlers

Please see:

https://askubuntu.com/questions/57224/a-sudden-drop-in-free-disk-space

where there is a suggestion to install or use a 'Baobab'.   Truth in sharing ... I've not used it.

https://help.ubuntu.com/community/Baobab

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Ubuntu/Moodle drive inexplicably and rapidly filling

by Doug E. Wray -
Thank you for the reply. Unless I'm missing something, Baobab is a Gnome program that does pretty much what ncdu does. I'll try it next time I run a desktop Linux, but I'm just on command line with my Moodle server.
Average of ratings: -