Backup error on large courses

Backup error on large courses

by John Jackson -
Number of replies: 3

Greetings Moodlers,

I've been wrestling with a backup problem for a while, and have collected quite a bit of experimental data, and think I'm going to put in a ticket into tracker. The automated course backup simply does not seem to work consistently with course with more than 4 GB of files.

Here's the environment: Moodle 2.8., VPS dedicated server, running PHP 5.4, suPHP handler.

I have several courses that include MP3 files so that the total course size exceeds 4 GB. When automated backups run, one of the courses throws an error of "unfinished" in the BU log. The automated BU script leaves a lot of files in /moodledata/temp/backup, which if left to its own devices, will completely fill up the drive, for they system tries again and again to backup, and each time leaving gibabytes of temp files.

I have turned on the "experimental" setting for the new zip format, which provides for courses > 4GB.

I can run a manual course backup from the web interface within the course, and it executes without failure.

The automated script worked with the course last when the resulting zipped course file was 3.5 GB. It now fails when the file is 4.5 GB (that is the size generated by the manual course BU). 

When I run the CLI automated backup script, it fails without any error, and leaves the same mess of temp files.

I have set the maximum amount of RAM for the BU and CRON process via the system settings. I have also dramatically increased the timeout settings in PHP.ini, so the bu script has time to execute a large BU.

Sine the manual course BU works, but the automated BU script fails, this seems to indicate a bug in the automated script, hence my inclination to file a ticket in tracker. But if any of you have a great idea of how to resolve the problem I'm all ears. In the interim I have to turn off automated backups and BU each course manually, which is not a viable long term solution. Perhaps one of you knows how to eliminate a single course from automated backups?

Thanks in advance,

John


Average of ratings: -
In reply to John Jackson

Re: Backup error on large courses

by John Jackson -

I have created a Tracker ticket: MDL-50255

Further testing with CLI: I got CLI of the automated backup script to run. It reports that it backs up the course successfully, but actually does nothing. It creates no temp files and no zipped file. It also executes in 2 seconds, indicating it just skips the course, though it says it completes.

For more details watch the tracker ticket.

Average of ratings: -
In reply to John Jackson

Re: Backup error on large courses

by Ken Task -
Picture of Particularly helpful Moodlers

As I understand it, the new 'experimental' backups setting uses gunzip and/or tar to create the archives rather than zip.  I think most Linux systems come with 'tar' but maybe not 'gunzip'.   Does your system have tar/gzip?   Reason I ask is your statement about runs and executes in 2 seconds.

And the command line version uses the settings one has in

/admin/settings.php?section=backupgeneralsettings

Plus ... have you checked /admin/settings.php?section=automated

There are settings in there that would affect autobackups as well as the command line versions.

Keep 1, skip hidden, skip courses not modified in X days, automated backup storage I would set to 'specified directory for automated backups' and the destination to be outside of moodledata.

Also check into suPHP rules.  It might be the culprit.   Apache error logs show nothing?

'spirit of sharing', Ken

PS - now that there is a command line backup script, wonder if there will be a 'restore' script to compliment it.   Hope so cause if Moodle is struggling with creating the backups it's likely (?) to struggle with restores via the Apache/Moodle UI.


Average of ratings: -
In reply to John Jackson

Re: Backup error on large courses

by Ken Task -
Picture of Particularly helpful Moodlers

Your post is 'perfect timing' as I've been working trough such issues on multiple sites.   As the end of the school year approaches was checking into end of year process for teachers which includes backups and resetting courses.   Some courses have now grown to the point where autoBU of them fails, etc.  ... same things you've described.   The moodledata/temp/backup/ data might be worth checking into as a completed backup might be contained in one of those directories ... Moodle simply couldn't *copy* them from the build directory to it's destination.   Other wise, manually removing is necessary recoup space.

Anyway, check into /moodlecode/admin/cli/ in 2.8.x one will find a 'backup.php' script there now.

Help on the script shows:

Perform backup of the given course.

Options:
--courseid=INTEGER          Course ID for backup.
--courseshortname=STRING    Course shortname for backup.
--destination=STRING        Path where to store backup file. If not set the backup
                            will be stored within the course backup file area.

Would recommend using a directory outside of Moodle for archiving backups ...  /home/backups or some such.

The command then: php path_to_moodle_code/admin/cli/backup.php --courseid=# --destination=/home/backup/

It's been capable of backing up such large courses ... takes Apache out of the loop and only involves PHP and MySQL server.

The only thing one needs is a brief listing of courses.

So in moodlecode/admin/cli/ I've created a little bash command to create a 'courses.txt' file to help:

mysql -u root -p'[yourDBpass] -e "use [yourmoodleDB];select id,fullname,shortname from mdl_course;" > courses.txt;cat courses.txt

catting courses.txt looks like this:

id    fullname    shortname
1    Some ISD    SISD
4    Government - Cofey    Government
5    AP Computer Science - All Teachers    APCS

So to backup the Government - Cofey course:

php ./backup.php --courseid=4 --destination=/home/backup/ [ENTER]

Your largest courses (those that have been failing with other methods) could be put into a 'backupall' which contains single lines like the one above.

One could then execute it like so on linux:

source backupall [ENTER]

'spirit of sharing', Ken

Average of ratings: Useful (3)