How to prevent Backup Timeouts

How to prevent Backup Timeouts

by Daniel Brouse -
Number of replies: 9
Hello, I have been combing through information on how to make sure my backups occur correctly for quite a while (on and off for months) and still have some questions.

First off, my Moodle 1.6.1 installation is setup on a shared hosting account and the sever settings are:
Operating system - Linux
Kernel version - 2.6.17-11_4.BHsmp
Apache version - 1.3.37 (Unix)
PHP version - 4.4.4
MySQL version - 4.1.21-standard-log

Now that we are in the winter break, I am wanting to upgrade to Moodle 1.7 but I have had some problems with backups breaking. I would like to get good backups before I try to upgrade. OK, I must admit, I tried to use cPanel's fantastico to upgrade to 1.7 but it didn't work and I suspect that the same problems I am having with backups are preventing a fantastico script based upgrade from working. One of these days I need to do an install/upgrade without relying on fantastico but that is another issue. So, now, I need to get the backups working correctly.

I have a rather small Moodle installation with just 7 active classes and 9 inactive ones. There are 150 student accounts with no more than 60 active during any one academic term. I HAVE been recording lectures via camtasia and posting them in the moodle classrooms in flash format so the total zip for some of my classes is > 200megs.

Anyway - ALL of my active classes are having problems with scheudled backups. Most often they make it to phase 2 of the backup process and then stop at the "logs" section. I have had some of the classes make it all the way to the "zipping files" section of the backup and then not finish.

Some questions:
  • Is it possible that having html tags in the reource name could cause problems on the backup. I often place <font color="red"> - Video Clip</font> in the name of my resources (because it looks nice and stands out) but I have wondered if that could cause problems with backups. One of the reasons I suspect this is that when I do a manual backup the resources that have the html formatting are often not selected for backup by default and they have some "funky" extra text associated with them. EXAMPLE:
    Video Clip" checked="checked" XonClick="this.form.elements['backup_resource'].checked=1;" /> Ch7 Skeleton 3 - Video Clip
  • Is there a process/location for storing the flash video files/folders so that they will not be included in the scheduled backup but they will be available to students who are enrolled in the class? I would be happy to NOT back the video clips up if that provides better reliablity for the backups of the rest of the course.
  • How do I setup an external zip program to zip the files. This seems to be a recommended way of zipping files for backup (rather than just using internal routines) but I don't know how to set this up.
  • How do you determine if your timeout settings on the server need to be increased and what are reasonable settings?
  • Does anyone have any other recommendations?
Thank you, Daniel
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by Daniel Brouse -
An update:

  • I deleted the html font formatting (see the original post if this doesn't make sense) on all of the resources and, although the backup dialog looked a lot nicer, I ran into the same problems. The backup stalled on the "logs"
  • I then turned off the backup of the logs and now instead of halting on the "logs" it is halting on the "zipping files."

Thanks in advance to all/any who respond.
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by John Isner -
See this discussion for how to increase the time limit on php scripts.

Average of ratings: -
In reply to John Isner

Re: How to prevent Backup Timeouts

by Daniel Brouse -
John,

Thanks for the link to the discussion. I learned something that I was wondering about -- where the updated php.ini file needs to reside. In the past I didn't know that it really needed to be in the moodle/admin folder and so I just copied it everywhere! It appears that some other people have had the same issue.

Thanks again.

Daniel
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by Simon Fraser -
Hi Daniel

You can go into the admin ->backup section and select what files you want to backup. I suspect that if you set course files to no the size of your backup zips will drop dramatically.

I just did this and will see what happens at 1:00pm tomorrow morning. I hope my "big" class will go through easily.

Simon
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by Simon Fraser -
Hi Daniel,

I am running on a similar config at www.bluehost.com. Seems that shared hosts limit the script execution time so when your backup script exceeds their limit they just kill it.

I switched to the external zip and unzip routines and I can now manually backup my courses. The automated nightly backup fails on one course with 25 mb of files but I don't mind. However, I can back it up manually.

What I did was go to my config page in the moodle admin section and put in the path for my external zip routine (/usr/bin/zip) and unzip routine (/usr/bin/uzip) in the operating system subsection.

You may have to get the path from your own host. This works for me but I don't know if it will work for you.

I suspect that problem lies in the size of your camtasia files. Why don't you host these outside your moodle class so that the backup does not have to deal with them. You could link to them so all the backup would have to store is the link and not the entire file.

Hope this helps
Simon



Simon
Average of ratings: -
In reply to Simon Fraser

Re: How to prevent Backup Timeouts

by Daniel Brouse -
Simon,

Thank you for the response. I smiled when I saw that you specifically mentioned bluehost.com as that is the host provider that I use. I almost mentioned them in my original post but decided to not mention them because I didn't want to seem to be advertising for them. Thank YOU for mentioning that you are having a similar problem with bluehost as it makes your other comments particularly useful for my purposes. I think that in the future I will provide such information.

I am pretty sure that I can get the automated backup to work with files larger than 25MB but I know that significantly higher than that, and the automated backup seems to timeout. I have updated the php.ini file with the following:
max_execution_time = 60 ; Maximum execution time of each script, in seconds. DCB updated from 30
memory_limit = 12M ; Maximum amount of memory a script may consume (8MB) DCB updated from 8
upload_max_filesize = 16M ; Maximum allowed size for uploaded files - DCB updated from 8M
Does anyone have any pointers on determining the optimal settings of the above values? Should I just find out the maximum value permitted and use that or is there a problem with increasing any of these values above a certain level?

I haven't designated external processes for the zip and unzip but I have noticed that manual backups (at the course level using the backup tool) will succeed when automated ones fail. At this point I am unable to get the manual backups to successfully complete when everything is included in the course backup (logs, files, etc.) and so I am getting a little worried. As you are using bluehost and have used the external backup processes, I guess I will give it a try also. I have been hesitant because I don't understand how specifying a path (directory structure) changes the zip and unzip processes. If anyone can explain that to me I would be obliged.

I have thought of putting all of my big files in the "files" subdirectory rather than include them as part of the course files. I will do this if necessary but I would prefer that the files not be accessible outside of the course, and I will really miss being able to easily look up the hyperlink information inside Moodle. If I place them in the "files" directory I will need to use something other than Moodle to determine the link to use for those resources.

Thank you again for your help.
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by Daniel Brouse -
Follow-up

  • I changed the zip & unzip to the external processes by pasting the suggested paths [zip routine (/usr/bin/zip) and unzip routine (/usr/bin/uzip)] into the appropriate fields in the configuration variables page.
  • I also increased the php.ini max_execution_time to 120, the memory_limit to 16M, and the upload_max_filesize to 24M. I pasted this to the admin directory and then confirmed that the configuration: variables page allowed 24M upload instead of the prior 16M.
I probably should have done these separately, to make it more scientific, but I did all of the above and then tried manual backups on three of my biggest courses. I still got timeouts on my "logs" for the two of the three classes (the other timed out on "assignments") but once I turned off the "logs" portion of the backup I ended up with at 220M, a 337M, and a 186M backup file for each. So it appears that I can backup these large (filesize) classes. I bet that if I increased the max execution time I could backup the logs also but I haven't tried it yet.


I found AN answer to my questions about how high you can safely go with php.ini settings in the "help" section of my host provider. It is not terribly informative (doesn't give very many numbers as guidelines) but it does provide some information about what you can change in the php.ini file and which ones may cause problems if you increase them too much (memory_limit).
http://helpdesk.bluehost.com/kb/index.php?x=&mod_id=2&id=124

Daniel
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by Simon Fraser -
Hi Daniel,

This is good information on the php.ini settings. I will experiment with these as well before the start of the semester.
Average of ratings: -
In reply to Daniel Brouse

Re: How to prevent Backup Timeouts

by Simon Fraser -
Hi Daniel,

Do you have an update on your changes? Are your backups working now? I made some of the changes you suggested and will see what happens tonight.

Simon
Average of ratings: -