Backup and restore

 
 
Picture of koen roggemans
the story of the misteriously filling up hard disk
Group DevelopersGroup Documentation writersGroup Particularly helpful MoodlersGroup Translators

I just want to share my learning experience for future reference - it might help some others too wink
We have a Moodle 2.1.2 with automatic backups enabled, once a week on friday, keep 1. After the upgrade in august from 1.9 to 2.1 we used about 400 GB from the 1000 GB disk for the moodledata folder. Last week, Nagios warned me we went over 80% disk usage, this week we were over 90%.
After a search, it appeared that the 'keep 1' setting didn't do the trick and I had all the backups since 20 august for all courses in the automated backup area. Why?
I found the cause in the 'Automated backup storage' setting. That was set to 'Specified directory for automated backups'. The 'Save to' setting was left black, which is ok, since the help message says 'Full path to the directory where you want to save the backup files
(leave blank to save in its course default dir)'. Therefore the backups were nicely saved in the 'Course backup file area', but it also caused the deleting of old backups not working anymore. Apparently if you set a 'Specified directory for automated backups', the automatic removing doesn't work.
I was happy to find that by running the cli script automated_backups.php the number of files was shrinking now to the expexted 1 in my course backup area. But the disk size kept on increasing. So not solved.
Deleted files apparently move to moodledata/trashdir. That contained now over 400GB. The trash dir is emptied accourding to the code, every 4 days. I didn't want to wait for that. In the database table mdl_config is a record 'fileslastcleanup', that says when the cleanup has last run (in time since epoch). I changed that time stamp, using http://www.epochconverter.com/ to a time stamp 4 days ago, started admin/cron.php. The trashdir was emptied and I'm back on 402GB disk usage.
Conclusion: be carefull with the 'Automated backup storage setting' wink

 
Average of ratings:Useful (22)
Picture of Cory Troup
Re: the story of the misteriously filling up hard disk
 

Thanks! Well done.

 
Average of ratings: -
Picture of Adam Watkins
Re: the story of the misteriously filling up hard disk
 

Thank you smile

 
Average of ratings: -
Ying Yang
Re: the story of the mysteriously filling up hard disk
 

Thanks for the great write up. I was unaware of the automated_backups.php script and that has simplified a problem I was having with runaway backups.

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: the story of the misteriously filling up hard disk
 
Average of ratings:Useful (1)
Rosario playing soccer
Re: the story of the misteriously filling up hard disk
Group Particularly helpful Moodlers

I think this should be reported as a bug in the Tracker. Autmatic deletion of old backup-zip-files should work in any circumstance.

Rosario

 
Average of ratings: -
Picture of Chris Kenniburg
Re: the story of the misteriously filling up hard disk
Group Particularly helpful Moodlers

Could there have been a permissions issue with the "Specified Directory" that was preventing the deletion of old backups?  When we used a specified directory it had to be changed for the www-data user to have proper permissions to write and modify documents.

Just a thought on the issue.

 
Average of ratings: -
Picture of Melinda Kraft
Re: the story of the misteriously filling up hard disk
 

It's not a permissions problem, or even a code one. It appears to be a hard coded setting that retains deleted fiels in the the Moodledata trash for 4 days. In a perfect world, I think Moodle admins should be allowed to set the trash clearing time to suit their particular needs, otherwise files should delete immediately.  In the case of backups, the problem can expand expontially if you retain a number of backups.

 
Average of ratings: -
Ray Lawrence
Re: the story of the misteriously filling up hard disk
Group Particularly helpful Moodlers

Watch out for this too: MDL-29262 Massive database tables.

 
Average of ratings: -
Ray Lawrence
Re: the story of the misteriously filling up hard disk
Group Particularly helpful Moodlers

Created an issue here: MDL-32100

 
Average of ratings:Useful (1)
Picture of koen roggemans
Re: the story of the misteriously filling up hard disk
Group DevelopersGroup Documentation writersGroup Particularly helpful MoodlersGroup Translators

Thank you for the rapport and adding me as a watcher Ray. Very thoughtfull of you.

 
Average of ratings: -
Picture of Tony Holt
Re: the story of the misteriously filling up hard disk
 

Just had this exact issue.

 

I'm okay to wait for the 4 days to be up, but our actual usage after moving the files to trash: 50GB
usage as a result of backups : 968GB.

 

Lesson Learned smile

 
Average of ratings: -
Picture of Nadav Kavalerchik
Re: the story of the misteriously filling up hard disk
Group DevelopersGroup Particularly helpful MoodlersGroup TestersGroup Translators

Please vote on this: http://tracker.moodle.org/browse/MDL-30725 solution to this issue.

 
Average of ratings: -
Rosario playing soccer
Re: the story of the misteriously filling up hard disk
Group Particularly helpful Moodlers

Thanks, my data disk was filled more than once, when Moodle 1.9 did not correctly remove the existing zip-files. I just voted for it.

Rosario

 
Average of ratings: -
Picture of David Price
Re: the story of the misteriously filling up hard disk
 

this was incredibly helpful, thanks.

 
Average of ratings: -
Picture of Janine Hempy
Re: the story of the misteriously filling up hard disk
 

You are my new best friend! Thanks for posting a detailed description of the problem and a solution. You rock!

 
Average of ratings: -
Picture of Miki Alliel
Re: the story of the misteriously filling up hard disk
Group Translators

Hey, we  use Moodle 2.3.3

and we set the "specified directory for automated backups" and set a valid Path. the backups are stored in the path (outside from moodledata) successfully but I can still see lot's of files in the list of the Automated bacuckup, I looked at the "moodledata/trashdir" and it is empty.  but still having this list of files in the  Automated bacuckup in the restore page of each course.

how can I delete them? our "moodledata/filedir" is huge now!

please Help

thanks

 
Average of ratings: -
Picture of David Andrew
Re: the story of the misteriously filling up hard disk
 

This is happening to me too.

My Filedir is growing by 6Gig every night, I've already doubled the disk allocation once.

any help please

 
Average of ratings: -
Picture of Erdem Sahin
Ynt: Re: the story of the misteriously filling up hard disk
 

Hi all,

I have the same problem with filedir. I need to find a way to decrease the size of filedir.

Is there a way to eliminate unnecessary files in filedir directory?

 
Average of ratings:Useful (1)
Picture of Chris Swaffield
Re: Ynt: Re: the story of the misteriously filling up hard disk
 

I, too, would like suggestions on how to solve this huge fieldir problem. 

 
Average of ratings: -
Picture of Bill Antonia
Re: Ynt: Re: the story of the misteriously filling up hard disk
 

Have a look at my comments towards the end of this thread:

https://moodle.org/mod/forum/discuss.php?d=219826

I had the same issue, loads of backups despite limiting the number to keep. It's a little drastic as you have to alter the PHP code temporarily. However I gained around 180GB if I remember correctly.

Just make sure you have a good backup of your server first.

Regards

Bill

 
Average of ratings: -