automated backups failing - sharing resolution

automated backups failing - sharing resolution

by Ken Task -
Number of replies: 6
Picture of Particularly helpful Moodlers

For months have been getting daily email notifications that one or two courses didn't complete.  Finally decided it's time to investigate/fix.

CentOS: el6.x86_64 16 Gig memory
PHP vr PHP 5.5.30, MySQL 5.5.32

Moodle version: 2.8.9+ (Build: 20151119)

Small site: only 17 courses (this to say, what I've done with this site may/may NOT work for larger sites)

Inspecting moodledata/temp/backup/
would find hashnamed directories that were used to build the backups
with no moodle_backup.xml file in them.  In other words, in-complete
backup attempts.

The scheduled task cron job to clean up not working.
Believe this is due to the backups never completing so Moodle
never really knew to clean up.

Resolution:

1. truncated all the data contained in the following tables:
mdl_backup_controllers
mdl_backup_courses
mdl_backup_logs
Tables remain ... just removed all the rows/records that contained any status/tracking/dates/ etc. info about courses and backups.

2. set automated backups 'active' to 'manual'.  Unchecked Skip hidden
Skip courses not modified set to 'NEVER'.
(desired to get at least ONE backup of every course then would go back and
reset those items)

3. in scheduled task - disabled:
Automated backups \core\task\automated_backup_task
Clean backup tables and logs \core\task\backup_cleanup_task
The latter set to:
10     *     *     *     *     0

4. manually removed all directories/ .log files in moodledata/temp/backup/

5. purged caches

6. Ran, from command line, the moodlecode/admin/cli/ automated_backukp.php script

In one course - the largest and the one containing the most number of
quizzes/test watched the contenthash/activites directory creation via linux command:
watch "ls -l" and discovered many, many, many quizzes.
That course finally backed up ... that one course took about 1 hour alone.
When watch could see the .xml files being created ... good sign ... close to
completing.

*** and for the first time in months ... ***

Sending email to admin
Automated backups complete.
Automated cron backups completed correctly
Execution took 2600.372029 seconds

And the Email notification:

Summary
==================================================
  Courses; 17
  OK; 17
  Skipped; 0
  Error; 0
  Unfinished; 0
  Warning; 0
  Automated backup pending; 0

  Backup completed successfully

Changed php max_exection_time to 2700 seconds from default of 30 - yeah, I know ...

Re-set automated backups to enabled, start at 0:05, skip hidden courses
skip courses not modified since 30 days.
But left, disabled, the scheduled task.

Will have to wait a day to see if it runs without error.

But, it's progress (I think!)

'spirit of sharing', Ken

Average of ratings: Useful (3)
In reply to Ken Task

Re: automated backups failing - sharing resolution

by Ken Task -
Picture of Particularly helpful Moodlers

Just to follow up and report on progress ... success!   Autobackups ran without error.

This is the second server/instance in which tables were truncated and, in both, 'control' of autobackups was re-gained.  In both instances, autobackups functioned as expected.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: automated backups failing - sharing resolution

by Dale Siggerfjell -

Interesting, I ran this and eventually got backups but unfortunately after truncating the three tables that you mentioned my "automatic backups" remained in the course restore space. The backups write to disk properly, just aren't updated in the table that houses this information and those links.

It's recently occurred to me that despite the backups taking place successfully, on the site it's only kept the 5 last courses before I changed my backup location settings. I can't manually remove them as the links are all dead, so it's a bit of a hassle to go through manually creating 0 bite files. 

Not sure really how to address this yet. 

Attachment Screen Shot 2016-04-19 at 07.00.57 .png
Average of ratings: -
In reply to Dale Siggerfjell

Re: automated backups failing - sharing resolution

by Ken Task -
Picture of Particularly helpful Moodlers

So if you clicked the button that says "manage backup files", what happens?   There should be a page refresh and in the box it should show those files listed in the table as icons file their .mbz filenames.   To remove them, right click or control click on single file and there should be a button atop the next popup window to delete the file.   After deleting all of them, remember to save ... that last action updates the tables.   Remember, you are looking at meta-data read from database tables ... not the real files themselves.   The files themselves should be moved to trashdir on the next cron job run and there (in trashdir) they remain until 4 days passes, at which time, via another cron job, the files are finally removed from the drive.

"course restore space" IF the backups are going to the default area for backups, which in the config for automated backups is  Automated backup storage backup | backup_auto_storage and is set to Default: Course backup filearea.   Translated that means in moodledata/filedir/   That's why they show in the area of your screen shot.

There is also another setting for

Save tobackup | backup_auto_destination

which is a directory you manually create and make readable/writable by the apache user.

There are other settings which, if the tables were truncated, would from this time forward come into play.

Keep number.  Skip courses not modified. Skip hidden courses.

So it's really a combination of factors that determines what one sees.   Truncating all tables removes any data Moodle was keeping on courses, but doesn't necessarily change the behavior of the options chosen for automated backups.

If I re-call correctly, with the two sites I was having trouble with, I had to truncate the new data created those tables. then fiddled with the settings for automated backups again to get desired results.

I ran the automated backups from the command line.    After running, needed to get into any course and click restore to see the backup files IF they were saved in an area I can see via Moodle UI.   Moodle doesn't provide an admin tool/view of all course backups.  Would also check, via command line, that alternate directory I manually created for the presence of the backups.   I saved those to a directory like: /home/m29backups which is writable by apache.

'spirit of sharing', Ken


'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: automated backups failing - sharing resolution

by Dale Siggerfjell -

Right, I understand what you mean and that process but unfortunately none of the backups listed exist anymore. They had long been deleted by automatic backups and thus the links are dead and I get the dreaded error (which i've become far too familiar with below):

Debug info: [dataroot]/filedir/d0/4f/d04f5e92e7a37c4ab807ed89d2b581507abe26c4

Error code: storedfilecannotread


Going in one by one to find every single instance of these backups and putting a 0 byte file in it's place would take forever. So I'm not really sure how to approach clearing the references to the files. 

Average of ratings: -
In reply to Dale Siggerfjell

Re: automated backups failing - sharing resolution

by Ken Task -
Picture of Particularly helpful Moodlers

You've reported other problems with DB matching files.    The things you are reporting are very strange and I don't believe I've ever seen similar.     Sounds like an older backup of the moodledata directory was restored to the site for some reason and thus the data in tables like mdl_files doesn't match what's in moodledata/filedir/.   

Things like this don't happen by themselves.   Ok, that might sound like a 'finger wag' (and it really is, but OK, lesson learned? ), sooooo ...

If there are only 5 or so backups referenced in the DB for which there is no physical file in moodledata/filedir, then do the same thing done with the user pics problem.   Carefully remove records in mdl_files.

What do you get with this query on the DB for Moodle:

select contenthash,filename from `mdl_files` where (`component` like "%backup%" and `filename` like "%.mbz%")

In checking over that output, see any that match up with the one error you've shown:

d04f5e92e7a37c4ab807ed89d2b581507abe26c4

If so, remove the record.

Just to be safe ... would backup the DB first before removing any records AND

I'd take the time to explore /moodledata/filedir/d0/4f/ to see if there is a file with that content hash name.   There shouldn't be.

If you had linux and could use CLI, one could check easily with:

cd /pathto/moodledata/filedir/

find ./d0/4f/ -name d04f5e92e7a37c4ab807ed89d2b581507abe26c4

That should kick back nothing.

Do the same for all those errors seen ... there are only 4-5 right?

Nothing found ... safe to delete the records.

When you finish cleaning up, remove any DB backups you have archived so you won't use errant ones.   Also remove any data directory backups you have so you won't use errant ones.   And do a fresh full site backup ... DB dump, an archive of the code directory, and an archive of the data directory.   Restores are only as good as the backups.

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: automated backups failing - sharing resolution

by Dale Siggerfjell -
Sorry for not responding earlier - I just saw the post right now.


What do you get with this query on the DB for Moodle:

select contenthash,filename from `mdl_files` where (`component` like "%backup%" and `filename` like "%.mbz%")

I ended up getting all of the automatic backup references, that I imagine most (if not all) are non-existent files. In the off hours of today I'll do a backup and try removing some references to a hidden/unused course reference. 

Hopefully I won't break everything. 

Average of ratings: -