Automated backups stopping after an hour of running

Automated backups stopping after an hour of running

by Paul Lindgreen -
Number of replies: 10
Picture of Particularly helpful Moodlers

Automated backups run for 1 hr and backup 30 courses, then the backups appear to stop even though there are a few hundred more visible courses to backup.

The last course worked on indicated in reports->Backups indicates 'unfinished', there are no cron log errors or php errors, the last cron log entry looks like
'Backing up XXXXXXXXXXXXX..
Server Time: Wed, 27 Apr 2016 09:52:27 -0300'

The next cron log entry for backups indicates the backups are already running even though no backups are occurring:
'Execute scheduled task: Automated backups
... started 09:54:12. Current memory use 32.6MB.
Checking automated backup status...RUNNING
automated backup are already running. Execution delayed
... used 1 dbqueries
... used 0.2500011920929 seconds
Scheduled task complete: Automated backups'

Do the automated course backups stop if the scheduled server task for automated backups try to run again before the first job is completed?

The 550 courses should take roughly 6hrs to complete, my scheduled server task for auto backups runs once every hour, as the default value, should it run once a day instead (or every 8hr?) so as not to interfere with the running task?

Thanks

========environment=======
moodle 2.7.8, windows 2008r2/iis7.5, mysql

Average of ratings: -
In reply to Paul Lindgreen

Re: Automated backups stopping after an hour of running

by Paul Lindgreen -
Picture of Particularly helpful Moodlers

Ran the backups again last night, still stopping after 1 hr of running, starts up 2hr later and repeats pattern afterwards.

Settings for last nights autobackups:
Server scheduled task for autobackups default value of every 50th minute
Course backups 8pm

Results:
850pm-950pm courses backed up
1150pm-1250 courses backed up
250am-350am courses backed up
550am-650am courses backed up
190 couorses total, which grows after each run by about 45, still about 450 left to do.

Clear pattern of 1hr of backing up, followed by 2 hrs of no backing up, then starting up again.

The backups stop and start on the 50th minute so it looks like the server scheduled task for autobackups is the culprit stopping the backups. I will consider running the server scheduled task for autobackups every 8 hr next.

Average of ratings: -
In reply to Paul Lindgreen

Re: Automated backups stopping after an hour of running

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

I suggest running once a day, probably in the middle of the night.  

Average of ratings: -
In reply to Emma Richardson

Re: Automated backups stopping after an hour of running

by Paul Lindgreen -
Picture of Particularly helpful Moodlers

Tried the following schedule in our development environment

Course automated backup schedule: every day at 135pm
Server Scheduled Task, automated course backups: 430, 1630

Results:

courses back up from 430pm-630pm every day

cronlogs reveal at 6:32 and 18:32 'Execute scheduled task: Automated backups' surprisingly. Its a mystery as to why its even running at these times as its not scheduled to do so? Here's the output from the mysterious task runs:

'Checking automated backup status...RUNNING
automated backup are already running. Execution delayed
... used 1 dbqueries
... used 0.70313000679016 seconds
Scheduled task complete: Automated backups'

There appears to be an unscheduled task running 2 hours after the server scheduled task run, which shuts down the first one for auto course backups.

Average of ratings: -
In reply to Paul Lindgreen

Re: Automated backups stopping after an hour of running

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Ok, so cron is running almost exactly two hours following the scheduled tasks.  I suspect a time zone issue...I would still get down to one a day till you get through an initial run.  I have been noticing that mine are taking hours to run and initial run took several hours (I think I have something going on though that I have not had time to trouble shoot because they should not take as long as they are on my server!)  

Occasionally, a problem with one course will stall out the run too.  I  suggest turning them off for a day just to clear anything out (hopefully) and then turning it back with just one time selected.  See if you can then get through it once.

Average of ratings: -
In reply to Emma Richardson

Re: Automated backups stopping after an hour of running

by Paul Lindgreen -
Picture of Particularly helpful Moodlers

re: I would still get down to one a day till you get through an initial run.

I'll give that a go next week

re: Ok, so cron is running almost exactly two hours following the scheduled tasks.

Server scheduled auto course backup running every hour at the 50th minute. Backups start on the 50th minute and end at the 50th minute than restart 2 hours later at the 50th minute.

re: Occasionally, a problem with one course will stall out the run too.

Wouldn't that result in irregular stop times rather than this 50th minute pattern of run terminations? Reports->Backups also indicates that the course on the 50th minute results in an 'Unfinished' status too.


Average of ratings: -
In reply to Paul Lindgreen

Re: Automated backups stopping after an hour of running

by Ken Task -
Picture of Particularly helpful Moodlers

What's your settings on Clean backup tables and logs in scheduled task?

and or other scheduled task that might have affect on Auto backups?

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Automated backups stopping after an hour of running

by Paul Lindgreen -
Picture of Particularly helpful Moodlers

re: What's your settings on Clean backup tables and logs in scheduled task?

Left at default of every 10 minutes.

Not sure how this setting or others affect the schedule, sure wish I could backup all the courses in 1 run though.  I'ld like to restrict it to one long run in the evening during our slow time. My backup of 750 courses took over 2 days while running at a pattern of 1 hr on followed by 2 hr off.

Average of ratings: -
In reply to Paul Lindgreen

Re: Automated backups stopping after an hour of running

by Ken Task -
Picture of Particularly helpful Moodlers

Haven't really tested/research this, but there was another poster that had a 'timing' issue with autobackups and the new task.   At that time, thought it might have been a bug ... clean up started while autobackup was running ... that clean up involved the very directory that autobackups (or manual) uses to build the backups ... moodledata/temp/backup/   Poster reported finding in-complete backup builds ... directories were there but didn't have moodle_backup.xml or other parts ... and no backup.mbz temp files. (which is how ti wroks, from viewing/watching it realtime ... ie, a backup.mbz file is created ... and the very last step in the process is to "copy" that backup.mbz to the destination chosen in config (file system or alternative directory) while changing the name of the file according to options chosen in making backups ... like use course short name, ID number, etc..

That last process of "copy" hits limitations in operating systems - 32bit vs 64bit and other limits to copy. (move command, however, works with any size)

I too have had issues with autobackups ... and I finally ended up: turning off the clean up task, setting autobackups to go to a file system repo (so they could be seen by courses set to use that file system repo), and creating my own script running the commands found in moodlecode/admin/cli/  tied to a cron job ... that I could schedule for off peak times.

To regain control of it all had to truncate tables related the the scheduling of autobackups first.   Set autobackups to manual, but did choose options in backups such that they were full course backups ... users, logs, etc.

The script is bash shell and runs through a text file listing of courses - courses1.txt would contain nothing but the course ID's:

2

6

33

50

I had to use two scripts ... one pointed to small courses and one pointed to the large ones - largecourses.txt.

This way ... I could either trust my manually created cron jobs to run or I could, if I desired to do so, run the scripts myself at any time.

Know this is not what you wanted to hear, but ....

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Automated backups stopping after an hour of running

by Paul Lindgreen -
Picture of Particularly helpful Moodlers

re: That last process of "copy" hits limitations in operating systems - 32bit vs 64bit and other limits to copy. (move command, however, works with any size)

I didnt think it was a server limitation issue as the process stops on the automated course backups schedule. I'll retest this in our development environment by changing the minute of the hour this task runs. We are on 32bit php and will probably go to 64 bit in the near future though.

To further complicate things my development environment runs 2hr on, 2hr off, with the same automated course backup schedule. This dev machine is M2.8 while production is M2.7. I notice one difference between the two versions in Scheduled Task defaults for Legacy log table cleanup, its once an hour in M2.8 and every minute of the fourth hour in M2.7, the M2.7 doesn't like correct but also doesnt look like its influencing my auto course backups either.

Thanks for the details on your manual work around. I'll look into the admin cli side of things, its looking a little more complicated than I hoped, do you think it would be easier to upgrade to Moodle 3 in the hopes its refined auto course backup schedules?

Average of ratings: -
In reply to Paul Lindgreen

Re: Automated backups stopping after an hour of running

by Ken Task -
Picture of Particularly helpful Moodlers

Am not aware that 3.x specifically addressed the issue ... for that matter not sure there ever was an official tracker bug reported with directions on how to reproduce the error.   One would have to have a sandbox clone of a production site that had the issues and purposely wreck/disable/putz to get consistent behavior, me thinks.    This to say, don't think I've ever been able to say .... xyz is cause in every case.

To many variables involved ... recently investigated a shared hosting environment where op had option to increase package restrictions that appeared to be the culprit ... cap on how much extra memory could be called by a Moodle script.   A standalone or even a virtual where op had full control that wouldn't have been an issue .. or the 'fix' could have been applied.

Fingers crossed that and upgrade wouldn't make it worse! ;)

'spirit of sharing', Ken


Average of ratings: -