How do I check for orphaned files?

Re: How do I check for orphaned files?

ved Ken Task -
Antal besvarelser: 8
Billede af Particularly helpful Moodlers

There has been reported issues with annotated PDF files remaining ... not removed ... and space usage.   From what I can tell, no resolution to that issue.

Deletion of anything in moodle is of concern ... deletion of users from backend does create issues in other areas ... like files that belonged to the users deleted.    Always best to find a way to get moodle code/cron/adhoc task jobs to do it.

As far as space usage ... yep, if moodle is used, it only grows in the needs for more space/processing power etc.   moodledata doesn't need to be on same server (am sure you are aware)... so if the web service isn't really stuggling to keep up and it's a space issue, moodledata could be on a server dedicated for storage of filedir - just a mount point for moodledata/filedir/ where the other directories in moodledata that require speed remain on same box as web service.

Have found that using git for installation/updates/upgrades by far is less stressful for those who wear that hat - and does make it less stressful for those involved in other areas of moodle server support as well.

Much also depends upon plugins ... and themes are plugins.

Any entity today certainly has options for their online learning options and the two you mentioned have made inroads in the K12 world (at least where I live).  But, like anything, there is + and - to any choice.

Did you get moosh to work on your server?

'SoS', Ken

Gennemsnitsbedømmelse: -
I svar til Ken Task

Re: How do I check for orphaned files?

ved Paul Ritchings -
Thanks Ken,

The thing is there were approx 19 thousand users. There are now 15 thousand. How can this have no effect on disk space?

Thanks for the info on the annotated PDFs.

I was not at work yesterday and in any case there's a fairly urgent bespoke project I'm working on at the moment.

I'll see what I can do with Moosh (development server first).
Gennemsnitsbedømmelse: -
I svar til Paul Ritchings

Re: How do I check for orphaned files?

ved Leon Stringer -
Billede af Core developers Billede af Particularly helpful Moodlers

I don't know if this adds to what Ken's already explained but:

Users by themselves have very little effect on disk space each being a row in database table mdl_users. When database rows are deleted the database's file storage does not decrease, it makes the space in the file available for future rows. Reclaiming unused space can be done manually, for example with OPTIMIZE TABLE in MySQL/MariaDB.

What generally takes up space in Moodle is files, typically either 1) course content created by teachers, 2) assignment submissions uploaded by users, and 3) Moodle system files like automatic course backups. One might expect users' assignments to be deleted when you delete the user but the assignments are owned by the course. So if a user has uploaded a 200 MB assignment, deleting the user won't remove that, you would have to delete the course (or – I think – the instance of the assignment activity).

The exception to the above is user deletion in response to a subject access request. If the site has the GDPR policy configured then if a user sends a "Delete all of my personal data" request to the privacy officer, and this is approved by the privacy officer, then their course files are deleted too.

Courses and activities may get placed in the recycle bin if this is configured so space not be recovered immediately when these are deleted. By default this is emptied after one week by the cron.

I wrote a script to list files in moodledata/filedir that weren't in the database (documentation). I think it does the same thing moosh file-dbcheck does. To download it, e.g. to the home folder:

$ wget -P ~ https://raw.githubusercontent.com/leonstr/moodle-utils/master/moodledata_orphans.php

To run it, assuming the Moodle source code is in /var/www/moodle:

$ cd /var/www/moodle
$ php ~/moodledata_orphans.php
Gennemsnitsbedømmelse:Useful (3)
I svar til Leon Stringer

Re: How do I check for orphaned files?

ved Paul Ritchings -
Thank you Leon

This could be very useful. Just tried the script on our staging server. It seems to be working.

I will probably (during a quieter period) put the site into maintenance and ask for snapshot to be taken of the live server then use the optimise table command.
Gennemsnitsbedømmelse: -
I svar til Leon Stringer

Re: How do I check for orphaned files?

ved Ken Task -
Billede af Particularly helpful Moodlers

I also, just tried.   Works ... on 3.10.x as well (where moosh does not) !   Thanks for that!   Feature request?  Justification: if the file on disk (orphaned ... ie, not in DB) could have a mimetype ID (image,gzip, etc.) one would have a clue as to where in moodle to look ... especially (?) true for .mbz's and .pdf's (annotated?).

Thanks, Leon!

'SoS', Ken


Gennemsnitsbedømmelse: -
I svar til Ken Task

Re: How do I check for orphaned files?

ved Leon Stringer -
Billede af Core developers Billede af Particularly helpful Moodlers

Ken: Thanks for the feedback. What I think I'll do is have an option to suppress the "Not in database:" prefix so the output can be piped e.g. to file. You can do this now using cut, e.g. with Moodledata in /var/www/moodledata:

$ php ~/moodledata_orphans.php | cut -d ' ' -f 4 | xargs -I '{}' file /var/www/moodledata/filedir/'{}'

Results in output like:

/var/www/moodledata/filedir/d3/d7/d3d7b7ffd2bb57fde1b9e7e28d0c8087b59eeb97: JPEG image data, ...
/var/www/moodledata/filedir/db/18/db1897ce8c5b24138b7101cc9f2e13c86d15b12d: ASCII text
⋮
Gennemsnitsbedømmelse: -
I svar til Leon Stringer

Re: How do I check for orphaned files?

ved Leon Stringer -
Billede af Core developers Billede af Particularly helpful Moodlers

Ken (and anyone interested): You can now add -b (bare output) and -p (show full path) options. With both options the output is suitable for piping to another command.

Examples:

See the file type of the orphaned file:

$ php ~/moodledata_orphans.php -b -p | xargs file

See the date and size:

$ php ~/moodledata_orphans.php -b -p | xargs ls -l

Or – if you've got a backup of the files – delete them:

$ php ~/moodledata_orphans.php -b -p | xargs rm
Gennemsnitsbedømmelse:Useful (2)
I svar til Leon Stringer

Re: How do I check for orphaned files?

ved Ken Task -
Billede af Particularly helpful Moodlers

That's great, Leon.   Thanks.   Should be made part of core code in admin/cli/ scripts IMHO! smiler

'SoS', Ken

Gennemsnitsbedømmelse: -
I svar til Leon Stringer

Re: How do I check for orphaned files?

ved Paul Ritchings -
Sorry to come back to this after weeks.

I've used the information both you and Ken have provided amd found that there are mbz file which while not being orphaned from mdl_files (ie they are on disk and in the table) they do not show up in the course indicated by for instance:

SELECT cxt.*, mf.*
FROM mdl_files mf
join mdl_context cxt
on mf.contextid = cxt.id
where mf.contenthash in ('b335685ddedd72b69e2ba88eecf976b4f9cf4220')

This gives a contextlevel of 50 and when I use the instanceid to view the course and go to "restore" there are no backup files shown.

Are there other tables which might cause the backup files not to be shown in the frontend? Has something become corrupt?
Gennemsnitsbedømmelse: -