Backup eats storage space (duplicate temp folders)

Re: Backup eats storage space (duplicate temp folders)

by Eloy Lafuente (stronk7) -
Number of replies: 10
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi A,

An MySQL dump containing your mdl_course, mdl_backup_courses and mdl_backup_log should be enough for me. With them, I can see where (course and step) is crashing your scheduled backup. You can send it privately, of course.

Anyway, making a few calculations:

240Mb * 12 executions per hour (once every 5 minutes) * 4 hours = 11Gb so really your disk is getting full before the 4 hours time limit!!

As a quick fix, you can try this:

- If you are running Unix, use external zip and unzip executables, specifying them in the Administration/Variables page. In the past, a memory bug in the PCLZIP library (used when such executables are not defined) has been the cause of similar issues. This could solve your problem with the course if the problem is its compression.

- In admin/cron.php change line 147 from "@ini_set("memory_limit","128M");" to "@ini_set("memory_limit","256M");". The backup process itself doesn't need that amount of memory, but the PCLZIP library need as much memory as the total size being compressed. This could solve your problem with the course if the problem is its compression.

- In backup/lib.php, change line 34 from "$hours = 4;" to "$hours = 1", this make that old temp files were deleted in 2 hours, so they will be deleted before your disk become full (you will need 2-3Gb "only"). But this doesn't solve the problem with your 240Mb course!!

Ciao smile
Average of ratings: -
In reply to Eloy Lafuente (stronk7)

Re: Backup eats storage space (duplicate temp folders)

by Timothy Takemoto -

Hi Eloy

I see that this is is related to the issue being reported here.
http://moodle.org/mod/forum/discuss.php?d=6548

My Moodle has been saving a temp folder every 30 minutes for the past month. I have just got a message from my sysadmin that I am using the entire hard disk space of my university's server = 430GB.

There are a very large number of temporary folders each with a large number of files. (about 35 days x 2 per hour x 24 hours = 1700 folders, each with a large number of files?). I cannot seem to delete them using ftp because it takes a long time to get the file list.

From reading the above it seems that the problem occurs when the backup fails, often due to the use of too much memory, and then tries again 30 minutes later, each time creating a new temporary file which is not deleted. 

However, in Eloys quick fix line 34 of backup/lib

        $hours = 4;
is just that in my moodle but it seems that backup files have been stockpiling for the past month. Am I understanding this correctly? Should my my Moodle have been deleting temporary files that are more than 4 hours old?

For the time being I have turned off automatic backup in my moodle (I have never had cause to use it). I hope that will stop the temporary files from being created but perhaps the once every thirty minutes temporary file is related to a backup command that has yet to be completed from one month ago?

I am using a 1.4 that I downloaded on the 4th August. Does 1.4.1 solve this problem?

Tim

Average of ratings: -
In reply to Timothy Takemoto

Re: Backup eats storage space (duplicate temp folders)

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi Tim,

I've been done some changes to the scheduled process just now. I think that, with them, all the "disk-full" errors and the "repeating-backups-every-half-an-hour" problems will be out definitively. I've simulated all types of (imaginable by me) errors in my test servers and everything has worked fine).

You can see more info in Bug 1954.

This changes have been included in MOODLE_14_STABLE so I think that 1.4.2 will contain them.

Ciao smile
Average of ratings: -
In reply to Eloy Lafuente (stronk7)

Re: Backup eats storage space (duplicate temp folders)

by Timothy Takemoto -

Thanks Eloy,

Thanks for the bugfix. Quick as always. And anyway your backup system is saving my life now as someone trying to create 26 course pages, by backing up prototypes and adding them to existant pages. You are a saint.

I need to fix the memory limitations that are preventing my course backups from working. In the meantime I think that I will be able to delete the large course that was stopping my backups from working.

I have spent a while hacking my 1.4.1 so I would like to be able to patch rather than upgrade to 1.4.2.

I have joined the inner circle (cvs mailing list) as a read only member and read of your two mails reporting changes to  

backup_scheduled.php log.php
and
backuplib.php lib.php

Can I just download these two (or four?) files from cvs and upload them or will this do something horrible to my 1.4.1?

Tim
Takemoto

P.S. *In case* I have some spare money in the future, do you have a paypal account? Please mail me off list.

Average of ratings: -
In reply to Timothy Takemoto

Re: Backup eats storage space (duplicate temp folders)

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi Tim,

I think that you can safely upgrade that 4 files (from MOODLE_14_STABLE branch). Details of required versions can be found in Bug 1954. Anyway, manual patching isn't the correct way, you know. wink

Ciao smile

P.S. In *that case* http://moodle.org/donations/ is the best place (as you know cool already).
Average of ratings: -
In reply to Eloy Lafuente (stronk7)

Re: Backup eats storage space (duplicate temp folders)

by Timothy Takemoto -
Dear Eloy,
Thank you. I will try and make a manual patch, even though I know it is not not correct. Or I may try and download 1.4.2 and hack all over again. It is quite quick when you get used to it.
Tim

Average of ratings: -
In reply to Timothy Takemoto

Re: Backup eats storage space (duplicate temp folders)

by Andy Hawkins -

Thanks for your suggestions. For various reasons we delayed upgrading to try your fixes, until forced to do so by the recent security alert. Now we are running 1.4.2 (2004083125) and tested the scheduled backup again.

All except 4 courses backed up successfully (as reported by email). The 'failure' courses listed in the log all had end date/times earlier than start date/times.

The overall backup started this time on 7 Nov at 2300 and finished at about 0121 the next morning. The first 3 'failures' showed an ending date/time which corresponded with the previous backup attempt (11 Sept) which started this thread. (ie the detailed log showed an attempt on 7 Nov, and then below this previous attempts on 11 Sept). [the last of these interestingly showed the end date of 12 Sept, possibly because it actually began at 0015 on 8 Nov].

The last failure only showed starting on 8 Nov (0050) and then ending on 1 Jan at 0000.

For each 'failure' there was a delay of approx 30 mins in the log before the next, successful, course backup.

The disc did not fill up (I had deleted previous backup files so it would not have found any) which I am pleased about and the log and email report is slick.

Where should I look further to try and find why these particular courses failed and why the dates are set strangely? (I set Sunday 2300 as the backup time when least user activity was likely)

TIA

Andy Hawkins

Average of ratings: -
In reply to Andy Hawkins

Re: Backup eats storage space (duplicate temp folders)

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi Andy,

what the exact status sent to you by email?

There are three possible situations:
  • OK. Nothing to do. Course Backup finished correctly.
  • ERROR. The backup didn't finish due to any "controlled" error. Common errors can be bad file names, some inconsistency in your DB...
  • UNFINISHED. The backup didn't finish due to any "uncontrolled" exception. Typically, they are related to some "resource limits, like "memory allowed", "execution time", "disk quota", "zip problems"...

Anyway, you've to take a look inside your wrong backup details to see when the process is finishing! Can you post here the last 10 lines of every wrong course in the backup details log?

Ciao smile
Average of ratings: -
In reply to Eloy Lafuente (stronk7)

Re: Backup eats storage space (duplicate temp folders)

by Andy Hawkins -

Thanks for response Eloy.

Email: 

Summary
==================================================
  Courses: 146
  OK: 142
  Error: 0
  Unfinished: 4

  Some of your courses weren't saved!!

  Please take a look to your backup logs in:
======================
First 'error' log:
begins ...
Sunday, 7 November 2004, 11:04 PM
11:04 PM:59Start backup course GCSE ICT(B) Module 3    2003-5
11:04 PM:59    Phase 1: Checking and counting:
11:04 PM:59        checking parameters
11:04 PM:59        calculating backup name
ends (on 7 Nov) and then shows previous log ...
11:05 PM:18                glossary
11:05 PM:18                label
11:05 PM:18                lesson
11:05 PM:18                wiki
11:05 PM:18                scorm
11:05 PM:18                questionnaire
11:05 PM:18        copying user files
11:05 PM:24        copying course files
Saturday, 11 September 2004, 10:35 PM
10:35 PM:51Start backup course GCSE ICT(B) Module 3    2003-5
10:35 PM:51    Phase 1: Checking and counting:
10:35 PM:52        checking parameters
10:35 PM:52        calculating backup name
Second course does this :
11:40 PM:39                questionnaire
11:40 PM:39        copying user files
11:40 PM:40        copying course files
11:40 PM:40        zipping files
Saturday, 11 September 2004, 11:14 PM
11:14 PM:24Start backup course Physical Education Specification A
11:14 PM:24    Phase 1: Checking and counting:
11:14 PM:24        checking parameters
11:14 PM:24        calculating backup name
3rd:
12:15 AM:08                label
12:15 AM:08                lesson
12:15 AM:08                wiki
12:15 AM:08                scorm
12:15 AM:08                questionnaire
12:15 AM:08        copying user files
12:15 AM:09        copying course files
Sunday, 12 September 2004, 10:00 PM
10:00 PM:35Start backup course AS ICT Video Challenge
10:00 PM:35    Phase 1: Checking and counting:
10:00 PM:35        checking parameters
and the last ..
12:50 AM:48            lesson
12:50 AM:48            wiki
12:50 AM:48            scorm
12:50 AM:48        calculating users
12:50 AM:48        calculating user files
(this one does not have a log from the previous attempt - presumably the course was created since.)
Hope this is what you wanted. It does look like some were still calculating/copying, but from looking at the finished backup file size, not necessarily the biggest courses. However this would depend on the compression actually applied - is there a way of finding out how big a course is before backing up?
It does seem it could be a resource/execution time etc issue. PHP memory is set to 64MB and script execution to 90 seconds.

                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Andy Hawkins
                                        
Re: Backup eats storage space (duplicate temp folders)

                                        

                                            by Eloy Lafuente (stronk7) - 
                        

                                

                
                                        

                                                        Picture of Core developers
                                                        Picture of Documentation writers
                                                        Picture of Moodle HQ
                                                        Picture of Peer reviewers
                                                        Picture of Plugin developers
                                                        Picture of Testers
                                        

                            

                
                            

                
                                

                                    

                                        Hi Andy,



thanks for your feedback. Summarising it your errors seem to be:



GCSE ICT(B) Module 3 - copying course files



half an hour later (this is normal):



Physical Education Specification A - zipping files



again half an hour later:



Start backup course AS ICT Video Challenge - copying course files



and, of course, half an hour later:



The 4th Course - calculating user files



Half an hour is the time that the backup waits until it assumes that something really bad has happened. Then it continues with the rest of courses (skipping the problematic one. That's the reason to say that "this is normal".



About your errors, some thoughts:

 
  • copying course files:: Something has gone wrong when copying all the data under the course directory. Reasons can be wrong names and/or some type of restriction in effect. I suggest you to enable debug and then try to execute the backup of that course manually to see if it finish and/or there is some specific error showed!
    

    
 
  • zipping files: Problems when zipping files use to be caused by the size of the course when using internal zip libraries. If you are under Unix (linux, macos x...) try to use external zip executables instead. They can be defined under "Administration/Configuration".
    

    
 
  • calculating user files: Never seen this before! At this stage the backup process simply counts the number of users with images and store them in backup tables for future use. Have you manually created/modified any user image? Enabling debug and executing the backup manually could show us more details about what's happening...
    
 
 

And this is all. If you can test above thoughts...feedback will be welcome to continue...ciao smile


                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Eloy Lafuente (stronk7)
                                        
Re: Backup eats storage space (duplicate temp folders)

                                        

                                            by Andy Hawkins - 
                        

                                

                
                                        

                                        

                            

                
                            

                
                                

                                    

                                        
OK, thanks again Eloy. I will need to find a suitable slot in my timetable to sit and play with the manual backups so it may have to wait a few days.
Please to hear the problems are 'normal'.smile
We are using Windows - so external zip does not seem to be an option.
I also have a feeling that I switched off error reporting in php.ini - so I'll have to see whether debug on will show any more than now.
We have not knowingly changed any user image manually (not even sure I would know how) - but we do have a number of users who have changed profiles to include scripts (now filtered out) remove names etc. Their underlying user id is still there but, for example, name fields are blank. However there are enough of these pests to show in more than the one course only.
Thanks for your patience and help - I'll get back to you as soon as I can.
Andy

                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  -