Moodle crashed, and after it was rebooted, Moodledata is no longer connected, and we do not have files that students submitted or any files that teachers uploaded. Please help!
I should also mention that the ldap server authentication was hidden for some reason.
We can actually find some of the files - up until the Spring 2018. We can upload new data and that works. Just can't find the files between Spring 2018 and up until yesterday.
There is very little information here...
1. Moodle crashed - what does that mean? What stopped working? What made it work again? Is is possible that there was a server backup restored?
2. What server is this running on? What version of Moodle are you using?
3. Where exactly did you find the files? In Moodle? On the server? What does this mean?
4. How do you know that files are missing from 2018 on? Is this a file search on the server by date or looking within Moodle?
5. I presume you are using ldap authentication? Does it work correctly after reenabling?
1. Moodle crashed - what does that mean? What stopped working? What made it work again? Is is possible that there was a server backup restored?
2. What server is this running on? What version of Moodle are you using?
3. Where exactly did you find the files? In Moodle? On the server? What does this mean?
4. How do you know that files are missing from 2018 on? Is this a file search on the server by date or looking within Moodle?
5. I presume you are using ldap authentication? Does it work correctly after reenabling?
Moodle 3.6.5, running on Ubuntu 16.04 lts. Database is on a separate server running MYSQL 8.0.18.
Students were taking quizzes and Moodle website stopped responding. We stopped a large database load that was happening at the same time. That made Moodle responsive, but nobody could log in except admins. We rebooted the Moodle website server and that didn't solve the login problem. We found that the LDAP authentication module was disabled. We re-enabled that, which solved the login problem, but now we have found that we are missing uploaded files from the last three years. We can upload new files, we can find files from the prior years (Fall 2014- Fall 2018).
It does look like the longer we wait, the more files show up and can be selected. Like it is slowly populating the files and connecting them to Moodle. We are up to getting the files from Fall 2018, so we'll just wait and see what magically happens!
By "finding files", I mean that I can go and look in Moodle for files submitted by students, and I can't find them. Or I can go and try to click on a syllabus that a teacher uploaded, and it's not there.
Students were taking quizzes and Moodle website stopped responding. We stopped a large database load that was happening at the same time. That made Moodle responsive, but nobody could log in except admins. We rebooted the Moodle website server and that didn't solve the login problem. We found that the LDAP authentication module was disabled. We re-enabled that, which solved the login problem, but now we have found that we are missing uploaded files from the last three years. We can upload new files, we can find files from the prior years (Fall 2014- Fall 2018).
It does look like the longer we wait, the more files show up and can be selected. Like it is slowly populating the files and connecting them to Moodle. We are up to getting the files from Fall 2018, so we'll just wait and see what magically happens!
By "finding files", I mean that I can go and look in Moodle for files submitted by students, and I can't find them. Or I can go and try to click on a syllabus that a teacher uploaded, and it's not there.
We are not getting all of the files restored from Moodledata. I thought it maybe was repopulating them somehow, but it's not. Not sure how we will do a restore of a backup from last night and also include all the changes made today with Moodle quizzes for the finals.
'Observed behavior' isn't necessarily an indicator of where the issues might lie. Moodle doesn't 're-populate' links to files unless you are making reference to a cache set up for moodle files. So does your site use a content delivery network or caching for files of some sort? If so, there would be/should be a reference in config.php, I would think.
As far as restoring yesterdays backup ... full site? Links to files really involve DB meta data in mdl_files table contenthash column that matches physical files located in moodledata/filedir/XX/YY/XXYYbunchofletters
Moodle admin user doesn't have any admin interface to check files. Only way it to use something like adminer (which you have installed in your /local/ of the code/web server - or - a no web way using mysql client on the web server making a query of mdl_files table.
So if you restored DB from 24 hours ago, yes, all quiz data collected during that 24 hour period would not be there ... like students never took their finals. No one would be happy with that.
Where is moodledata/filedir/? config.php has that path.
Is moodledata also another server (like NSF)?
As far as issues with logging on ... using DB for sessions or files?
Burst in DB traffic you mentioned ... was autobackups running ... did someone schedule the cli only LDAP sync for the wrong time?
How about 'health' of the DB server and databases themselves? Have to confess sorta strange seeing an old LTS Ubuntu (16.04 - which is still supported I guess) with another dedicated DB server running MySQL 8.x. What's the DB operating system?
Dedicated DB servers are great if they are configed to use as much memory as they can for the DB for Moodle. That way there is less I/O and most of the queries to the DB are answered from memory ... not looked up in a table, etc.
So what does the DB server look like? OS, free memory, memory used for DB, any automated process that would eat up CPU/DB?
Do the test students are taking use many files in questions ... like audio files or videos hosted locally?
How many more students need to take test? Is there any time that, due to number of classes taking test, there would be a burst of usage?
Sorry ... no answers ... just too many places one needs to look for the reason of unfortunate events of a day ago.
'SoS', Ken
Follow up ... what's important concerning files for you right now is if they exist for questions.
So if that is correct, here's a way to check on files used in quizzes:
Command line Query
mysql -u root -p'' -e "use moodle;select contenthash,filename,filearea,filesize,component from mdl_files where component='question';" > qfiles.txt; cat qfiles.txt;wc -l qfiles.txt
will produce a file called qfiles.txt
qfiles.txt looks like
contenthash, filename,filearea,filesize,component
061a936175eeada6969166901477c2f4e9efb005 ppg__examview__U2_MC__00001_res__nar001-2.jpg questiontext 29516 question
bf8fc7c80e30c7b84968acd44c0fa917cac7bbaf ppg__examview__U2_MC__00001_res__nar001-3.jpg questiontext 23383 question
5d675b5726c83e54b4be51f2ffc176462d653437 ppg__examview__U2_MC__00001_res__nar001-4.jpg questiontext 27807 question
10e16b2d96a428a1d29c220424165504c5daac4e ppg__examview__U2_MC__00001_res__nar001-5.jpg questiontext 29198 question
e0ff6b7ad01f32b134463144c261196577e2a30c ppg__examview__U2_MC__00001_res__nar001-1.jpg questiontext 27984 question
Use the contenthash value to find if the file exist in moodledata/filedir/.
From moodledata/filedir/ and using the last line above as example:
/moodledata/filedir
find ./ -name e0ff6b7ad01f32b134463144c261196577e2a30c
command might take a long time looking through what might be 100's of Gigs of files.
should show:
./e0/ff/e0ff6b7ad01f32b134463144c261196577e2a30c
note the ./e0/ff/ subdirectories
and to get mimetype of file ... database meta data showed it to be a .jpg
[root@moodle filedir]# file -b ./e0/ff/e0ff6b7ad01f32b134463144c261196577e2a30c
JPEG image data, JFIF standard 1.00, comment: "LEAD Technologies Inc. V1.01"
So whatever question uses ppg__examview__U2_MC__00001_res__nar001-1.jpg image in it the image should show in the question.
Could do another query for only the contenthash value to imagefiles.txt
Then where moodledata/filedir/ is located on your web server create a looping bash shell script that would read each line of imagefiles.txt and do a find in moodledata/filedir/ while redirecting output to foundfiles.txt
Search foundfiles.txt for lines that would indicate file not present to see if there are any.
Best of luck!
'SoS', Ken
I don't have all the details, but apparently, because we were doing a backup of the database during the time when all of the exams were happening, Moodle slowed way down, and when we rebooted Moodle, the database reverted back to the backup that happened in the early hours of the day (3 a.m.). As I said, files were not showing, but slowly they would start appearing. By the end of the day, we had reverted back to all of the files that were backed up at 3 a.m. Which meant that we had lost all of the work that students or teachers had done in the morning. Fortunately, some of the teachers had saved their grades before everything disappeared.
Sounds like backups process/procedures need reviewing/tweaking.
For moodledata ... large as it is ... have found progressive rsync to be the fastest. Basically a mirror .. rather than a tar ball. And to a directory that could be mounted in a pinch. Actual tar ball archive happens during non-prime time especially when students wouldn't be taking finals (finals week always an exception).
Some communications between DB server and Moodle so that Moodle is put into maintenance mode with an informative screen for users. DB backups (sql dumps) should probably lock DB and prevent any admin/teacher/student access to Moodle.
My 2 cents.
Glad you have it sorted though and thanks for getting back!
'SoS', Ken