General help

Missing files on Moodledata filesystem

 
Picture of Alain Raap
Missing files on Moodledata filesystem
 

We recently migrated a database and Moodledata filesystem from another system and I found out that with the automated backup a lot of Moodledata files were missing on the filesystem. So I wrote a bash script to check all the files in the database table mdl_files that were missing on the filesystem. I checked the unique contenthash entries in the table mdl_files against the files on the system (Moodledata filedir subsystem).

Who has also experienced this problem and how did you search for the missing files? And also, how did you solve this problem (update the database or restore the missing files)?

The bash script delivers an output file with all the missing files on the Moodle filesystem.
Who is interested can send me a message for more info about the script.

 
Average of ratings: -
Picture of Howard Miller
Re: Missing files on Moodledata filesystem
Group Core developersGroup Documentation writersGroup Particularly helpful MoodlersGroup Plugin developers

I missing something here...

Do you mean it was ok before the migration but not afterwards?

 
Average of ratings: -
Picture of Alain Raap
Re: Missing files on Moodledata filesystem
 

Howard, we're still investigating why the files are missing after the migration. That's why I made the bash script.

 
Average of ratings: -
Picture of Ken Task
Re: Missing files on Moodledata filesystem
Group Particularly helpful Moodlers

To assist with the 'sea' of files in filedir and DB, consider installing moosh - a cli only moodle tool.

https://moodle.org/plugins/view.php?id=522

There are a couple of options to moosh commands that can help with files/meta data in DB:

https://moosh-online.com/commands/

'spirit of sharing', Ken


 
Average of ratings: -
Picture of Alain Raap
Re: Missing files on Moodledata filesystem
 
Ken, thanks for the tip. I knew Moosh already, it wasn't enough for me so I wrote a script that looks from the metadata in the database at the files on the filesystem. Moosh looks for files without metadata (orphans).
 
Average of ratings: -
Picture of Ken Task
Re: Missing files on Moodledata filesystem
Group Particularly helpful Moodlers

Thought moosh had more then one command dealing with files .... and one did find orphans ... at one time.   File-dbcheck below? then diff a raw listing?

data-stats

Provides information on size of dataroot directory, dataroot/filedir subdirectory and total size of non-external files in moodle. Outpus data in json format when run using --json option.

moosh data-stats

file-datacheck

Go through all files in Moodle data and check them for corruption. The check is to compare file's SHA to their file names.

moosh file-datacheck

file-dbcheck

Check that all files in DB do exist in Moodle data.

moosh file-dbcheck

file-list

Search and list files from mdl_files table. The argument should be a valid SQL WHERE statement. Interesting columns of possible search criterias are: contextid, component, filearea, itemid, filepath, filename, userid, filesize, mimetype, status, timecreated, timemodified.

You can also use some special values:

  • course=NNN to list all files that relate to a course

The output will contain some defaults or nearly all possible file information if "-a|--all" flag is provided. The meaning of the flags column is (in order):

  • mdl_files.status
  • lowercase letter "d" if entry is a dicrectory
  • "e" if external file
  • "i" if a valid image
  • "m" if time created and time modified differ

With "-i" option only IDs are returned. This can be used when pipe-ing into other file-related commands.

Example 1: Show all legacy files for a course, which context id is 15

moosh file-list "contextid=15 AND component='course' AND filearea='legacy'"

Example 2: Display full information on file with ID 17

moosh file-list -a id=162

Example 3: Show all files from course 6

moosh file-list course=6

Example 4: Super-combo. Get all course files and tar/bzip2 them up.

moosh file-list -i course=2 | moosh file-path -s -r | tar -C $(moosh config-get core dataroot) -T - -cjf files.tar.bz2



 
Average of ratings: -