Would help to know version of Moodle and OS under which running ... so consider this an 'educated' guess ... (having had some issues in this area before on boxen I help admin)
Might I suggest inspecting moodledata/temp/backups/ and it's contents? Successful course backups leave only a 0K .log file. Course backups which have issues might leave a contenthash named directory Moodle was using to build the .mbz's and/or a large file by contenthash name (which is a zip/mbz) that couldn't be 'copied' to the designated directory for some reason.
A quick fix, until issues can be discovered and resolved, might be to run a script that removes all files/folders from moodledata/temp/backups/ before one runs another script to run the autobackups 'manually' ... ie not automated. Script can be found in /moodlecode/admin/cli/ and is called 'automated_backups.php'. Comments in that file say:
'This script executes automated backups completely and is designed to be
called via cron.'
While designed to run from a cron job, one can run it from command line.
One can see what's going on with moodledata/temp/backups then with a watch command in moodledata/temp/backups/ and a watch command on the mnt point directory as well.
Don't think turning on Debugging will show much in this case, BTW, as Moodle code might be successful in creating the .mbz file but cannot *copy* it to the mnt point.
'spirit of sharing', Ken
Hi Ken thank you for your swift response.
I am running 2.6.3 on Linux (CentOS). The problems seem to have started since I upgraded to 2.6.
My log files are arounf 1kb each which suggests there might be a problem. I have disabled automatic backups for the time being until I find the route of the problem.
I have attached a screenshot of the backup dir
Your help is greatly appreciated.
Thanks
Ben
Happen to run CentOS myself ... so the specifics below I have done/experienced first hand:
The .log files are ascii/text. One can open using nano or just cat 'em:
cd /pathto/moodledata/temp/backup/
nano some.log file
or
cat some.log file
Can't tell you specifically what you'll find ... but they will be clues. From the looks of the number/size/dates, could be a specific course that is having issues.
Those two files at the top of your screen shot ... how large are they?
ls -l output please (shows size)
Can do a quick peak at the file type:
file 71....
should show if it's a zip ... in Moodle-ese a .mbz file ... that might fall into that category of too large for CentOS to use the 'copy' command to move them to your /mnt/win/dir.
CentOS 5.x and 6.x as well has an issue with copying files over 4.+ Gig's. An OS bug that's not been fixed yet, BTW. (so it's not Moodle code)
Could do the following in that directory.
mkdir work
mv 71[tab key] ./work
cd work
If file 71... above said it was a Zip, unzip it:
unzip 71...
Open the extracted moodle_backup.xml with nano OR use head for the top 50 lines:
head -n 50 moodle_backup.xml
That will show the humanly recognizable name Moodle would have used and give you a clue as to which course to explore in the Moodle UI.
Once one knows the humanly recognizable name Moodle would have used for the filename to use in the autobackups designated directory, you could *move* it yourself.
mv 71[tab key] [humanly recognizable file name].mbz
then mv [humanly recognizable file name].mbz again to your /mnt/win/directory:
mv [humanly recognizable file name].mbz /mnt/win/[directory]
Now if that 71[tabkey] file doesn't unzip, or the unzip command complains about an in-complete archive, then we've some more sluthing to do.
At any rate, once you've manually moved .mbz's to your /mnt/win/dir all files and folders could be removed from moodledata/temp/backup/ Don't delete the 'backup' directory, just it's contents. cd moodledata/temp/backup/ [enter] then rm -fR *
Then run your command line autobackup.php script to see what happens.
In one terminal session cd /pathto/moodledata/temp/backup use 'watch ls -lR".
in another terminal session logged on as root user cd /pathto/moodlecode/admin/cli/ run
php automated_backups.php
Also, check those log files to see if it's an issue with a mod/block not compatible with your current version of Moodle being used in 'troubled courses'. If those mods/blocks not all that important one could remove them from the system and especially in the courses that used them.
'spirit of sharing', Ken
Hi Ken, sorry for the delayed response.
I will look to create a delete script as a temporary solution.
I think I have narrowed the problem down to incomplete/error back ups from the logs. I have attached a recent log to see if anything raises alarm bells with you.
Referencing the hash against the fildir - shows that there are no files that exist with the same code which might be throwing up the bugs.
I await your wisdom
Best,
Ben
Oh, are we in trouble ... 'await ... wisdom'!!! I did disclose that much of the 'wisdom' comes from trying things that don't work ... at least, then, I know what doesn't work. :|
Ok, looked at the log file and see references to a couple like:
checksum 7fbc5565f00d2a24e02e5c73d3b76036
And you're telling me that 7fbc5565f00d2a24e02e5c73d3b76036 doesn't exist in filedir?
checksum suggest the system is checking the accuracy of the creation of that file - ie, nothing missing. So that .log file doesn't look like it's reporting any issues.
Are you using the operating system 'du'?
And there is nothing left in /moodledata/temp/backup/ ?
Moodle must be able to create temp files ... and it must be able to remove them once it's done as well.
One of the methods I use to troubleshoot things like this is the 'watch' command - while the backup process is running. Can see Moodle create folders and xml files and then build a contenthash named .mbz file. Sometimes, Moodle cleans up all of it with the exception of the .mbz file (which I've mentioned before, in certain situations, cannot be 'copied' to designated directory for archiving). It can, however, be 'moved'.
So can you use command line and 'watch'?
'spirit of sharing', Ken
I'm running into the same problem after the 2.6 upgrade (from 2.4)
I've enabled the new 'experimental' backup format, so perhaps that will help, as I suspect it might be courses over 4GB that are crashing the backup. In 2.4 it simply failed that course backup, but in 2.6 is it possible it doesn't clean up at all once it hits one of those?
That brings me to the design decisions that may be influencing this: I'd argue it should be creating, copying to target for course backups, and clearing the temp for each course as it completes, if it's not. I've got over 100GB in my Moodle install, and can't float multiple copies of the temp files for the whole mess when backups fail. I understand when dumping files into a tar that a local scratch disk has performance advantages, but either handle the temp files during backup or maybe better let me flip a switch to work on the target as scratch as well?
Don't know if anyone involved in writing watches these threads, or if I should submit that elsewhere for consideration, but thought I'd mention it.
At any rate - I'm curious if you found resolution on this, and what that looked like?
You've not mentioned your operating system ... do know that there is an issue with backups of courses over 4+Gigs and it's not Moodle code ... it's operating system support for the copy command. The backup .mbz that is built in moodle/temp/backup/somecontenthashedidfolder/ is copied ... not moved ... to the course backup area ... ie, back into filedir somewhere and/or to the alternate directory for automated backups. It's the copy command on CentOS that lacks support. They say it can be fixed ... if you want to re-compile the kernel. Uhhhh ... no thanks!
Got a big backup box of some sort that you could mount a folder upon from the Moodle? Could use that as your alternative backup area. Mount Point could then also be symlinked in Moodle's file system repo in some courses so one could still access those backups ... even though they are no longer in moodledata.
Might have to increase PHP limits on time for a script to run ... as well as memory used by a script.
If you are running automated backups on cron job, switch it to manual. One could then write a small script using the moodlecode/admin/cli/auto_backup.php script (read the comments in the script) and then call that script via cron job (not Moodles) to execute at non-peak days/hours.
Anyway ... there are issues. Solutions are not 'one size fits all' am afraid.
'in the spirit of thinking', Ken
Thanks for the reply, Ken.
Sorry about missing the context info. I'm relatively new to Moodle-specific admin nuances...
Moodle 2.6.4+
Exp.Backup - On; didn't change behavior
PHP ini resource settings:
Mem: 128M
Parse & Exec times: 60sec
Ubuntu server 12.4
Linux ##### 3.2.0-57-generic #87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Course Backup path is a mount:
//########/Backups/#### on /media/offsite type cifs (rw) ---- (this is a Win Server 2008R2 share)
Auto Course backup enabled for 6pm daily, to /media/offsite/moodle_course_backup, keeping 2, etc.
Also run a manual nightly full system backup in a cron job for DR site, and VM backups, but the course backups are nice for when a prof blows up a course - so easy to recover versus full system backups.
So, let's see if I follow you (and in case anyone else is seeing this issue they'd know what I did):
One suggestion is that I set a symlink between the temp/backup folder and an external big disk? I'm not sure I follow how that fixes the problem. I've got courses over 4GB (some almost 30GB - need to have my instr.tech.guy sort that out with them!) which wasn't a big problem in 2.4; those course backups would fail but get flagged in the report and I'd know we need to work with the faculty on those, but least the other courses would still complete, and clean-up would wipe the temp, including the files from the failed backups. So even a huge external disk now (2.6) slowly fills (60GB of failed backup at a time). Likely I'm missing something in understanding this one?
Another suggestion was to ditch auto-backups for manual variant - principally to avoid peak times, etc. Worth looking at the script, perhaps, to see if I could check course size first, and skip the big ones, but still doesn't answer why the failed course files aren't cleaned up.
Am I getting that? What am I missing about why those failed course files aren't cleaned up? What I did today, finding that the exp.backups setting didn't help, was add a line in my nightly routine to wipe the temp/backup folder. Doesn't get my large course backups running, but at least the disks don't fill up.
I'll post any other solution I find here as well. Thanks for working at this with me.
Thanks for the additional information. Can now be a little more specific as to what to 'watch'.
Might need to increase the php settings for exec to something larger than 60 seconds. Those Gig+ courses take time to build. That could be the reason large courses fail to complete. BTW, how would Moodle know to cleanup a directory IF the script creating the backup fails? Answer ... don't think it would.
So when I run into one of those, I'll first clean the /moodledata/backup/temp/ folder of anything that resides there ... .log files, temp hashnamed directories and hashednamed files.
Then, from command line, while in /moodledata/backup/temp/, issue the following command:
watch "ls -lR" [ENTER]
You can watch that window and see Moodle build the backup ... can also see where it stops.
In another terminal session, from the command line, I'll use the following:
tail -f /var/log/httpd/error_log
On Ubuntu think that's tail -f /var/log/apache2/error.log (or something like that)
That allows me to watch that log in realtime.
And yet, in another, run 'top' ... that will show php kicking in from time to time ... and how much memory it's using, etc..
In the browser, I'll go to that course and begin a backup process on one large course that failed. If you can run the browser backup on from another workstation it's easier to see what is happening/or not on the workstation you are using for terminal access. Or, in yet another terminal session, run the automated backup script from the command line. Yeah, I know three terminal screens. :\
Have discovered different issues (doing the above) so can't say for sure what you'll see/find.
Basically, moodle creates a 'roadmap' of what to backup in moodle_backup.xml so if the script dies before it can write info about files to that .xml file, the files.xml file and then copy the contenthashed named file from moodledata/filedir/whereever ... one might be able to determine which file.
Will also have to do a query of the DB to see if teachers are making their own backups and how many they have. Moodle creates those backups with date/time stamps in the names so they are not the same files each time a teacher backs up the course. You could have multiple backups IN the backup you are attempting to create.
So this query should get some info for ya:
select contenthash,filename,filesize from `mdl_files` where `filename` like "%.mbz%"
If the mount point for moodledata/backup/temp is on another box, that will make the build process slower (which means an increase in the time to exec) but it's off the web server.
'spirit of thinkering', Ken
Hmmmm time to edit expired so ...
http://www.sosoftexas.org/videos/mdl2-watchingbackup.swf
shows (kinda) the process described above.
'spirit of sharing', Ken
Great suggestions - I'll certainly make progress now.
I hadn't really considered that manual course backups could be that big a chunk, but wow - we've got plenty of cleanup to do. That's what was hitting the biggest courses hardest - large media files + multiple backups = bloat.
Thanks for helping me get started in more proactive management of our rig - this gives me specifics to go to our Ed.Tech. and faculty with, in addition to sorting out my own maintenance tasks.
-Russ
We have a similar problem but we found that the backups from deleted courses was not purged.
I just created a plugin to remove this kind of backups:
https://github.com/leblangi/moodle-tool_purgeautobackups
Also, you may vote on this task:
https://tracker.moodle.org/browse/MDL-48808
I hope this will help.
