Automated backups: successful after 1.5 hours but continues to update db for 5 hrs slowing system to a halt

Automated backups: successful after 1.5 hours but continues to update db for 5 hrs slowing system to a halt

by Susan Mangan -
Number of replies: 3

Running version 2.4.5.  Automated backups scheduled to run at 3 AM on Tues and Fri. to a different server.  Backup reports indicate backups successful after approx 1.5 hours.  Yet ... seeing strange activity in db where mdl_backup_courses is being updated continuously for more than 5 hours after backup has completed. (see screenshot).  This is slowing our system down to a halt.  The queries indicate that moodle is updating a prior backup that was last backed up approx 2 months ago, and had an error status of 3.  The update is setting the next backup to occur for the next schedule backup date.  This seems logical given we didn't have the automated backup process running for a while during the summer and changed the parameters from storing the backup to the course area to a different server... however, why is this happening hours after a backup completes?  And why does this particular process seem to slow our system down to a halt? Does it have to do with the fact that the backups are on a different server?  That would explain it I suppose, however, the fact that the backup report indicates it was completed successfully after 1.5 hours while churning out queries hours and hours after the fact still concerns me.  Just wondering if this is normal....

The queries are being executed in a timely manner but during normal activity the mysql screen never looks this full. (see screenshot)  Our dba confirms adequate resources on the DB as well, our infrastructure staff confirm system resources (cpu, memory, etc.) are adequate. (VM) Normally, the performance is quite good.

Has anyone seen this behavior and/or have any suggestions as to how to fix? (this thread was started in the hardware/performance forum but when I confirmed it was the automated backup process causing the problems I thought it might be good to post here as well?)

Attachment moodle-db2.jpg
Average of ratings: -
In reply to Susan Mangan

Re: Automated backups: successful after 1.5 hours but continues to update db for 5 hrs slowing system to a halt

by Susan Mangan -

Hmm, it seems I have missed a day here.  The backup ran successfully on Tues at 3 am and the behavior I am referring to above actually occurred the following day.  The last time this happened the behavior occurred the day of the backup as well as the following day.  Can't make heads or tales of it.  Still mystified.

In reply to Susan Mangan

Re: Automated backups: successful after 1.5 hours but continues to update db for 5 hrs slowing system to a halt

by Ken Task -
Picture of Particularly helpful Moodlers

Well, no one seems to be venturing guess here, so I'll begin ...

From what I've learned about autobackups ... Moodle uses first moodledata/temp/backup/ as a build area to make the backups.  Check that area for residual files/folders remaining - should only be 0 byte .log files with loooooong file names - which indicate success.  If you see any greater then 0 .log files open and inspect.  Might be a clue in there.  It is safe to manually remove all files folders inside the backup directory.

Once the backup file has been built, Moodle then copies (not move) the built .mbz file to the designated folder.   You've said this is on another server so am guessing thats a mount point.  It's been my experience that courses of 4.x+ Gig size have issues with being copied (not moved) on a CentOS system (if running RHEL) probably the same.  Has to do with kernel and there is a fix, but one would have to re-compile kernel to get that support.

In the DB (version 2.5), there are three tables related to backups:

mdl_backup_controllers, mdl_backup_courses, and mdl_backup_files_template.  Check the first two tables for info ... the last usually contains nothing.

Check out the first table for info.  It would be rare probably but maybe, just maybe, some instructors are running backups of their own courses (snow balls chance, but ...).

Think autobackups would be run by user 2 - that's the admin account.

Courses that have not changed within the config limits are not backed up but skipped ... one can see that while running cron.  Also, there is a command line utility just for autobackups in /moodlecode/admin/cli/ which might be worth running to see what's going on.

'spirit of sharing', Ken

Average of ratings: Useful (2)
In reply to Ken Task

Re: Automated backups: successful after 1.5 hours but continues to update db for 5 hrs slowing system to a halt

by Susan Mangan -

As always, THANK-YOU for your continued brilliance.  I'll have a look to see what I can uncover.