CRON Scheduled Backups Randomly Failing (>85% of the time) plus a "Work Around"

CRON Scheduled Backups Randomly Failing (>85% of the time) plus a "Work Around"

Caleb Benefiel -
回帖数:5
I posted this earlier as a reply to another thread - but then I realized the thread was over 6 months old!  I think this deserves a new thread since I've compiled a lot of suggestions into a single post.

First some background information:
The problems started in 1.3.x and continued in 1.4.3 for awhile.  I upgraded to 1.5.1 and then 1.5.2 and the problem continued. 
In an attempt to resolve this issue, I deleted the old installation of Moodle and removed all related databases.  I performed a clean installation of 1.5.2 and then restored the courses.
My Moodle has 5 courses.  Manual backups work great.  Automatic/scheduled backups fail ~85% of the time.

CRONTAB
Every ~8 minutes, I run:
lynx -connect_timeout=300 -dump http://www.constructiveteacher.com/admin/cron.php > /dev/null via the CRON daemon.

(I just recently added the -connect_timeout=300 string to see if that would resolve the problem - it did not)

When an automatic backup fails, I receive this email from the CRON Daemon at my webhost (Total Choice).

----------------------------------------------------------
Looking up www.constructiveteacher.com
www.constructiveteacher.com
Making HTTP connection to www.constructiveteacher.com
Sending HTTP request.
HTTP request sent; waiting for response.
Alert!: Unexpected network read error; connection aborted.
Can't Access `http://www.constructiveteacher.com/admin/cron.php'
Alert!: Unable to access document.
lynx: Can't access startfile
----------------------------------------------------------

Once I receive this email, the cron.php script reports that a "Backup is in Progress" for the next ~45 minutes or so.  At some point the program realizes the backup has stalled and then it sends me a backup report.

----------------------------------------------------------
Summary
=====================
 Courses: 4
 OK: 3
 Error: 0
 Unfinished: 1
----------------------------------------------------------

I've tried some different user suggestions that I've read.
  • Optimize the database via phpMyAdmin (credit: Peter Cameron)
  • I changed the backup directory to /moodledata/globalbackups/ instead of the individual course directories.  (credit: James Depow)
  • I manually set the path to zip / gzip in the Moodle config. (credit: Tim Allen)
Unfortunately, none of these suggestions seemed to have helped me or many other people who have run into similar problems.

Here is some info about the actual backups I have:

----------------------------------------------------------
Click on a folder icon to navigate.
Click on a name to view its properties.
       / mdata / globalbackups / (Current Folder)
       backup-biology_1-20050729-0826.zip      42986 k 2777
       backup-cteacher-20050729-0826.zip       92 k    2777
       backup-gs-20050729-0828.zip             1708 k  2777
       backup-life_science_1-20050729-0826.zip 7152 k  2777
       backup-ls_-_1-20050729-0828.zip 16 k    2777
----------------------------------------------------------

The problem usually occurs on the biology backup - it's the 43MB backup.  Perhaps that has something to do with it?  It occasionally fails on life_science as well - the 7MB backup.

Work Around
If I change the moodle config to not include the course files, the backups are significantly smaller - all are less than 500k.  When I do this, the backups very rarely fail - only ~5% of the time or less.
Then I just have to remember to do a manual backup every now and then to make sure that the course content has been backed up.

Also, I haven't thoroughly tested this, but if I use my web browser to call /admin/cron.php instead of letting the CRON Daemon call it, a full backup fails much less frequently.  I'd say a full backup is successful ~25%+ of the time!

Has anyone determined the cause of this?  There seems to be some random element involved.  The CRON daemon itself seems to be somewhat involved as does the size of the backup.

Caleb Benefiel
回复Caleb Benefiel

Re: CRON Scheduled Backups Randomly Failing (>85% of the time) plus a "Work Around"

Caleb Benefiel -
Today was an interesting day!

I think I've finally found a solution to the infamous...
-----------------------------------------------
HTTP request sent; waiting for response.
Alert!: Unexpected network read error; connection aborted.
Can't Access `http://www.constructiveteache
r.com/admin/cron.php'
Alert!: Unable to access document.
lynx: Can't access startfile
-----------------------------------------------
...error!

I had a feeling this was a Lynx related problem, but it was hard to tell.  From doing some google searches, I found that the average Lynx timeout was ~110 seconds.  For some reason that I do not understand, this timeout is a bit variable.  That could be why the problem starts out as an occasional problem and then gradually happens more and more often as your Moodle grows.  (on a bad Lynx day, the timeout can be as low as 25 seconds).

So... why use Lynx anyway?

I guess some hosts do not allow you to directly call the PHP binary.  But Total Choice Hosting does let you call PHP.  On my server it's kept in /usr/local/bin/php

So, I altered my CRON process to call:

/usr/local/bin/php /home3/constea/public_html/admin/cron.php > /dev/null

And the problem seemed to go away.

I let 3 complete backups run over the course of a couple hours to make sure it wasn't just a fluke that this fixed the problem.

And I also have implemented all the suggestions I mentioned in the previous thread, so some of those changes could be contributing to this success.

Good luck if you're having this problem!
Caleb Benefiel


回复Caleb Benefiel

Re: CRON Scheduled Backups Randomly Failing (>85% of the time) plus a "Work Around"

Martin Dougiamas -
Core developers的头像 Documentation writers的头像 Moodle HQ的头像 Particularly helpful Moodlers的头像 Plugin developers的头像 Testers的头像
Yes, using the php binary is by far the preferred solution.
回复Caleb Benefiel

Re: CRON Scheduled Backups Randomly Failing (>85% of the time) plus a "Work Around"

Rodney Holloman -

I am having the same problem using wget.

What would the string in crontab look like using this method. I have played around with it and can't seem to get it right.

Thanks,

Rodney

www.sfwbc.edu/online

回复Caleb Benefiel

Re: CRON Scheduled Backups Randomly Failing (>85% of the time) plus a "Work Around"

Mohamed Hossam -

Hi Caleb.

I registered to only thank you! I know this topic is old but you might be saving me from being cindrella for the last four months! blush

I have a website that HAS TO perform some PHP processing work at 12:00 every night. But I faced the same problem you had. Now, I am trying your way (php command line) and will tell you if things go fine tomorrow big grin