Course backup hangs - no error logs

Course backup hangs - no error logs

by Josiah Carberry -
Number of replies: 38

My course backup function has started to fail, where it had worked fine. This is true for the daily scheduled backup, as well as the interactive backup. All I see in the interactive backup is that it gets to about 97% and then advances no further. 

There are no logs for the backup itself that help diagnose the issue.

Furthermore, there is nothing in either the apache or the mysql logs relevant to this question.

It is definitely not an issue of disk space, nor have there been any changes in directory access rights.

So, I am stymied in trying to figure out what is going wrong. The problem may be related to an upgrade from version 2.8.1 to 2.8.6. Since, I have upgraded to 2.9, but the problem is still there. Any suggestions?

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

With no debugging errors (turned on and using manual backups), no errors in logs (Apache or MySQL nor in 'messages'/PHP logs) then the only thing one could suggest is inspection of the directory used to build backups: moodledata/temp/backup/

There might be 0 byte files remaining in that directory with names such as:

87a616e0f6a01dbcb56be8f666f10f57.log

Those indicate successfully completed backups.

If one sees directories with such names: efa616e0f6a01dbcb56be8f666f10f57 with associated .log files those are failed backups (or failed to complete the very last step ... indicated by the % of progress you've shared.   That last step is to *copy* the built backup .mbz file to it's destination depending upon options chosen in the backup preferences.

Please take a look at that directory and post back what you see.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

OK. So I have plenty of such log files of 0 length and associated directories. There are no .mbz files, however. There are, instead a while load of directories and files corresponding to the various types of objects that I want to back up. Most of the files are xml files, although some have no extensions. A picture of the top level listing in one of those directories is attached.

Do I understand correctly that the last step is to make a compressed tar archive of all these files? If so, this is apparently where the backup is failing.

By the way, I note that all the file permissions in the backup area are 666. Does anyone need access to these files other than www-data, the owner? Shouldn't the permissions be 660, at best, or even 600? 


Attachment Capture.PNG
Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Yes, the second to last step is for Moodle to use moodle_backup.xml to build a .mbz file - which is the complete backup of all those items.

So NONE of the folders with those long hash names have temporary backup.mbz.AYgzi9 files in them?

Normally, there is a reason for failure and they could be Apache related, PHP related, or even MySQL related.  Normally, there are errors reported in logs, but you say there is not.   So without more specific info, a suggested list:

Permissions on the data directory ... that directory should be outside of document root and the only user/group that should be able to write to it is the apache user.   That being said, it looks like apache is able to write to it.  You don't show ownership/permissions in your screen shot.  So set the ownership/permissions liberally to begin with ...

@ the location of moodledata: chmod ugo+rw moodledata -R

Since NONE of the hashnamed folders have .mbz or temp .mbz files in them, they may not contain all the building blocks (ie, moodle_backup.xml file hadn't been completed yet).   So basically all those are useless.   Thus remove them all.

@ moodledata/temp/backup/: rm -fR *

Now some things for PHP ... increase the variables for time a script can run.  Default is normally 30 seconds.   Make it something like 120.   Amount of memory a script may consume:  Default is normally 128M ... increase it to 256M.  Changes to php.ini required restart of apache service.

MySQL: large course backups will involved larger data packets.  So one setting in MySQL is max_packets:

max_allowed_packet=500M

In building temp tables, MySQL might need to be able to open more files:

open_files_limit=6000

After you've made changed to my.cnf (MySQL config) and php.ini (site side PHP), *turn off* automated backups in the Moodle Admin interface - set it to manual.

Then run the cron job several times from the command line:

cd moodlecode/admin/cli/

php cron.php; php cron.php; php cron.php

Since you are down on the command line and in cli directory, let's backup a course via command line:

First make a directory outside of Moodle for the destination directory for the backups.  Since you are running these scripts as root there should be no issues with writing to whatever directory you create.

Example: mkdir /home/moodlebackups/

php backup.php --courseid=2 --destination=/home/moodlebackups/

The course ID above is the course ID of the desired course to backup.   You can see those by accessing a course with web browser and looking at URL line.   Think I'd pick the course with the least number of resources (links to files you may have uploaded to the course) and the least number of quizzes in it (quiz/test is a heavy process and memory/resource hungry).

Running backups this way take apache out of the loop ... it's just PHP.  The script won't display everything it's doing and will look something like:

== Performing backup... ==
Writing /home/m26backups/backup-moodle2-course-2-sa-20150525-1021.mbz
Backup completed.

If you want to watch what's going on in another terminal window:

cd /path_to_location_of/moodledata/temp/backup/

@ /path_to_location_of/moodledata/temp/backup/ type:

watch "ls -lR"

When you begin the cli backup in the other terminal window, click back on the second terminal session and 'watch'.

Am assuming you are not on a shared system.

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

Thanks for this detailed advice.

To make matters short, after making the new settings for apache2 and mysql, and restarting both services, nothing changed. The backup process still creates all the backup xml files etc. , but fails to create the mbz file.

BTW, all the backup files have 666 permissions, so this is not the issue.

I note that when the command line backup is performed, you say that I should expect something like:

== Performing backup... ==
Writing /home/m26backups/backup-moodle2-course-2-sa-20150525-1021.mbz
Backup completed.


In fact, I never see the "Writing.....mbz" line (which is in line with the general issue).

Furthermore, I note that the same issue occurs when I try to backup a course that basically has nothing in it.


Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Ok ... how about some specific information concerning your system?  

Operating System is?   Reason asked: *If* CentOS and still running SELinux in enforcing mode, that would cause issues in some cases.

Hosting is?   Reason asked: some providers have built in restrictions and set things such that NOTHING is logged.

Have been attempting to assist by looking at clues provided.

Since you appear to have command line, let's see if you can manually complete the last step.

Run the automated php script again from the command line.

Then cd into moodledata/temp/backup/[thatlongdirectoryname]

Make your own .mbz file

tar -zxvf testcourse.mbz ./*

That creates the backup file located in the [longdirectoryname].   Success?  Go on to next step.

Then let's do it just like Moodle is programmed to do it ... by using copy.   Still in that [logdirectoryname]:

cp *.mbz /home/m29backup/copiedtestcourse.mbz

Just make sure the directory to which the .mbz file is to be copied exist prior to executing the command.

After this, am at ropes end and really don't know what to tell you.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

I appreciate your having taken all this time to work through a diagnosis of the problem.

I can create a tar archive (albeit with different options from the ones you suggest) and I can copy the archive to a home subdirectory.

I am running on a private Ubuntu 14.04.2 LTS server platform.

I think I might reinstall moodle, as I can think of no other explanation of why the backups should suddenly start to fail.



Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Well, am at a loss.

Another poster is also having issues with backups and has found that certain tables have just grown too large ... in backup for example, you have options to include histories and logs or not.   The settings you choose in the form, affect automated backups either cron'd or run from command line.

If that were the case however, one might see MySQL log errors ... of course, many do not run constant MySQL logs as they tend to get very,very large.

It might be related to updating/upgrading method.   Downloading zip and un-zipping zip (ie, overwiting) isn't recommended.

With command line access, think that git installation and git for updates as well as upgrades IS the way to go. 

Just one more thought ... should have asked this before ... what are your php settings for memory a script is allowed to consume, length of time a php script is allowed to run?

Larger courses do require upwards tweaks from 128M to something higher in the first, from 30 seconds to something higher) 120 seconds? for the second.

'spirit of sharing', Ken


Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by david millians -

I had an issue where M. said they were running when they weren't. Solved the problem by duping the 3 backup tables (on 2.6) then emptying them of records. It enabled the automatic backup to run again.

Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Dan Kirk -

Good afternoon All,

Thank you Ken for spending such a large amount of time to help resolve this issue. I too have been having head ache with Moodle backups ever since I upgraded my VPS server package.

First of all I started to notice the Moodle file dir started to grow rapidly from about 7gb to 30gb in the matter of days. I was receiving auto backup error emails from Moodle but assumed it was coming from the ghost copy of the site I was about to take offline. Upon trying to reduce the file size of this dir, I notices the temp backup file was huge. I am not happy deleting anything out of the Moodledata dir in case it screws everything up. I do know that the php version of the new server has been upgraded slightly and also a newer version of CentOS is on the new VPS however I have followed all your advice in tweaking the php.ini files from within WHM.

I would like some expert advice on this issue. If I disable the autobackup will the cron sort all the unfinished files? I too have nothing but blank log files with no backup logs inside of Moodle, so really no real information as to why it is failing. I have temporarily shut down the auto backup to conserve server storage space but I am anxious to get this issue fixed.

Many thanks in advance.

 

Dan

 

Average of ratings: -
In reply to Dan Kirk

Re: Course backup hangs - no error logs

by Josiah Carberry -

Dan,

Your issue sends a lot like mine. All I can tell you is that I was never able to solve it (well, I solved the issue by ignoring it, depending on a full site backup instead). A workaround in the hand is worth ten in the bush. I have, however, recently migrated by site to a new platform with an entirely different structure of file ownership. I will let you know here if that makes any difference.

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Dan Kirk -

Hi Josiah

I feel the pain.

I would like the option to use the Moodle backup solution otherwise i would have to completely restore a whole hosting account. I always want to have a plan b when it comes to backups.

Dan

Average of ratings: -
In reply to Dan Kirk

Re: Course backup hangs - no error logs

by Sam Stevens -
Bit of a thread resurrection but have you checked your php settings max_runtime setting? If you don't have a long enough time there then a backup will fail due to hitting that.
Average of ratings: -
In reply to Sam Stevens

Re: Course backup hangs - no error logs

by Dan Kirk -

Yes i have followed all the advice as described in previous comments but still no luck.

Thanks

Dan

Average of ratings: -
In reply to Dan Kirk

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Define 'expert advice'.  Somewhat difficult with Moodle as everyone's Moodle setup isn't necessarily the same.   Can only 'share' what has worked for me - thus, am 'expert' on the systems I admin ... your mileage might vary.

1. I don't use filedir for automated backups - which is the default location for saving the autobackups.  Instead, I use a file system repo (moodledata/repostiory/autobu) and tell auto backups to save files there.   This way, filedir doesn't increase ... that's the main storage area for all files in Moodle courses.

2. to be able to use that repo for restoring courses, should that be needed, simple to add a file system repo to a system admin level only course.   Can restore courses there.

3. Preferences for autobackups makes a diff ... are you backing up courses with users or without users.   Those prefs determine what it is that get included in autobackups.  Obviously, during 'peak time', one would want full backups ... users + their work.   That, however, is a factor in backing up.  Do you really need logs?

As reported/shared by another poster in this thread, he had to re-gain control of the whole process so he truncated the 3 tables related to autobackups.    That cleared all dates and flags for the functioning of autobackups. (I've had to do that also) ... Problem resolved.

4. the contents of moodledata/temp/backup/ can be removed manually ... and safely ... it's in temp.   But, before one does that, might be a good idea to get a look see at them ... it might help discover your problem.

Example: IF in one of those contenthash named directories one sees a 'backup.mbz' file (as well as all the directories and related xml files [moodle_backup.xml being very important as it's the roadmap for restores]) it could mean that your system (think you said CentOS but didn't mention version nor if 64 bit or not), there is an issue with Moodle using 'copy' to move the backup file to it's desiginated area (filedir/other).  On CentOS there is an operating system limit on how large a file it can copy ... one can move just fine, but not copy.   Moodle code cannot overcome that.   IF one is using the repo suggestion above, one can readily see all the backups by their humanly re-cognizable names.   One can also use a command to "mv" what really is a valid backup file that Moodle couldn't 'copy' to that repo directory.

Another factor ... how many courses?   And what is nature of those courses?   Any course in a K12 environment that is about 'digital' anything, will be the largest (normally).   Those courses will always fail with the autobackups.    So, knowing their course ID numbers, one can write a bash script to use the moodlecode/admin/cli/backup.php script that saves to the suggested repo directory *and* set a cron job for that script to run.   Not at same time as autobackups.   You are then assured you have those large courses backed up.  Using such tactics have been successful in getting large courses (all students and their work) in a backup that's over 20+ gig.

Taking the time to do the above means one doesn't have to rely on full site backups alone ... besides, how would one extract 1 course from a site backup to restore to a production Moodle?

Yeah, it's not perfect ... what open sourced project/app, constantly improving/changing, is?

Hopefully, in the future, with future releases/fixes, etc. these sorts of things will be finally resolved.   In the meantime, strongly suggest doing whatever it is you have to do to get course backups.   One day, you might be glad you took the time.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Dan Kirk -

Hi Ken.

Only a  true 'expert' is usually that humble. It is obvious you have vast experience in your field and a superior knowledge to myself of the inner workings of Moodle.  All the data inside the temp folder is hashed littered with XML files with no completed archives. It does seem to be the largest course that the backup is failing on, but it isn't that large.  The old server i was using was CENTOS 6.7 x86_64 xenpv using MySQL 5.5.48-cll where as i am now using CENTOS 7.2 x86_64 hyper-v with the same MYSQL version. SHould i upgrade it to MariaDB 10.0? It would be a nightmare if Moodle stopped working as a result however.

I will endeavor do continue with my research to try and resolve this issue. Once again, Thank you for all your time within this community.


Regards

Dan

Average of ratings: -
In reply to Dan Kirk

Re: Course backup hangs - no error logs

by Josiah Carberry -

On my new platform I have command line capabilities so I have run a course backup manually using cli/backup.php

By now I am on Moodle 3.0.3

Needless to say, it failed spectacularly. But at least, for the very first time, I have errors displayed! The output from the script is below. Unfortunately, it is beyond my skills to identify what is going wrong:

I have looked around at other threads to see if I can find any info. Not much luck.

However, I can say that mysql max_packet_size is 16M (which is a lot higher than the recommended size of "at least 4M")

Socket timeout is set at 60 seconds.

Anyway, here is the output:


== Performing backup... ==
instantiating backup controller 7d0988a029a83084cd3a8549ed47d79d
setting controller status to 100
loading controller plan
setting controller status to 300
applying plan defaults
setting controller status to 400
setting file inclusion to 1
checking plan security
setting controller status to 500
setting controller status to 700
saving controller to db
calculating controller checksum 5b2b6833bd202233ae404b391bb73d7a
loading controller from db
setting controller status to 800
PHP Warning:  Error while sending QUERY packet. PID=4444 in /blahblah/web/moodle/lib/dml/mysqli_native_moodle_database.php on line 1079

Warning: Error while sending QUERY packet. PID=4444 in /blahblah/web/moodle/lib/dml/mysqli_native_moodle_database.php on line 1079
Default exception handler: Error reading from database Debug: MySQL server has gone away
SELECT COUNT(*)
                  FROM mdl_backup_ids_temp b
                  JOIN mdl_user u ON u.id = b.itemid
                 WHERE b.backupid = ?
                   AND b.itemname = 'userfinal'
                   AND u.mnethostid != ?
[array (
  0 => '7d0988a029a83084cd3a8549ed47d79d',
  1 => '1',
)]
Error code: dmlreadexception
* line 443 of /lib/dml/moodle_database.php: dml_read_exception thrown
* line 1080 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1480 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->get_records_sql()
* line 1553 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
* line 1763 of /lib/dml/moodle_database.php: call to moodle_database->get_field_sql()
* line 471 of /backup/util/dbops/backup_controller_dbops.class.php: call to moodle_database->count_records_sql()
* line 1638 of /backup/moodle2/backup_stepslib.php: call to backup_controller_dbops::backup_includes_mnet_remote_users()
* line 88 of /backup/util/plan/backup_structure_step.class.php: call to backup_main_structure_step->define_structure()
* line 181 of /backup/util/plan/base_task.class.php: call to backup_structure_step->execute()
* line 177 of /backup/util/plan/base_plan.class.php: call to base_task->execute()
* line 120 of /backup/util/plan/backup_plan.class.php: call to base_plan->execute()
* line 320 of /backup/controller/backup_controller.class.php: call to backup_plan->execute()
* line 100 of /admin/cli/backup.php: call to backup_controller->execute_plan()

PHP Fatal error:  Can't use method return value in write context in /blahblah/web/moodle/theme/contemporary/lib.php on line 38

Fatal error: Can't use method return value in write context in /blahblah/web/moodle/theme/contemporary/lib.php on line 38

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

This:

MySQL server has gone away

tells you the issue is still the setting for max_packet_size is still (even at 16M) is not enough.   Set it to 32M.  Restart MySQL service/server and TIA (try it again from command line).

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

Easier said than done, Ken. In fact, that's simply not allowed on my platform.

Any other suggestions?

PS: The issue is very hard to understand. Why, if the packet is not big enough, is it not sufficient to simply send more packets?

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

What's not allowed on your platform?    On shared remote hosting there's lots of things one cannot do.  Try submitting helpdesk ticket or whatever and inquire.  If they won't, guess you'll need to shop around for a better hosting solution.

Any app backended by a DB, needs to be programmed as efficiently has it can.    Sending too many query request at the DB gets one into a bind also.   Some data in Moodle can't be broken into smaller packets ... not without a lot more coding ... which, of course, just adds to complexity.   That kinda reminds me of dialup internet where one went after a 1 Meg file with a 2400 baud modem - that never connected higher than 1200 baud.   Well, in a two words ... 'it sucked!' (that's when internet got the rep of the 'world wide wait').

As to technical explanation: https://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html

Let's see ... the way technology goes ... back when one could use only 24K  ... programs had to be written in such a fashion as to fit into those memory limitations ... fast forward ... we all now use laptop tops/desktops that have the memory of a very, very, very powerful server 20 years ago ... and the race goes on and on.

Gates got it wrong when he said 'no one would need over 640K'!!! ;)

'spirit of sharing', Ken



Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

I understand the point about performance, but I think that is putting the cart before the horse. An application should not FAIL simply because of a limit to packet size. The application should read that limit and ensure that it's packets never exceed it. If, as a result, the performance is poor, then the limit can be adjusted (if allowed). I have seen other applications that have programmed workarounds for the cases where the server limit was lower than desired. But outright failure is not a workaround.

In any case, when the Db server is not the localhost, those huge packets need to be broken down into much smaller packets, meaning a lot of added overhead. It strikes me that a lot of inelegant assumptions have been made by the programmers.

Enough of that rant.

I note in the documentation: "Both the client and the server have their own max_allowed_packet variable, so if you want to handle big packets, you must increase this variable both in the client and in the server." I understand that the server is configured in a file such as my.cnf. Where is the client configured? In php.ini? In the application code? As an env. variable?

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

You mention other programs ... please share what other programs?

A google search shows Drupals, Joomlas appear to have or could have the same issue.
And they all provide the same solution ... set max_packets_allowed higher
on server.  From what I understand, an attempt to use higher settings
in a client still does not override the server settings.  Attempting changes
in client side php.ini begets error 500's from what I've read.

max_packet_allowed - client
http://stackoverflow.com/questions/13417371/can-you-increase-max-allowed-packet-from-the-client

Shows command of mysql client:
https://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html

But, as I understand it, it's a session setting.  So even if one had command line
access and could use a client on the server, that setting, issued from
command line, would apply only to the session of the command line client.
Thus, a process from Moodle code would be another session and the command issued
would not apply.

Think the only hope one has if remotely hosted and on shared host is to contact hosting provider and inquire.

Otherwise, one might have to run backups manually and select only half of the items in a course on one run and then select the other half on run two.

Or, if one could figure out what resource in the course is causing the issue, de-select the one item.  In your previous posting with debug output ... 'userfinal'.   Which, without seeing it, indicates a quiz/test?

I know neither one of those is really acceptable (so no rants please - I know it sucks!).   But to get a backup ... any backup ... it's the only way I can think of to do it ... given situation.

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

I have finally been able to come back to this issue. My hosting provider has been convinced to raise the value of max_packet_size from 16M to 32M. Also, Moodle has been upgraded various times to 3.1.1+ (Build: 20160714) 

These changes have made no difference at all to the course backup issue when run via the client interface. I still get the same error messages as above when debugging is turned on.

However, when run via the CLI, the backup gets as far as creating the mbz file in the moodledata/backup/temp/blahblah directory. However, it fails with the messages:

!!! Error reading from database !!!
Potential coding error - existing temptables found when disposing database. Must be dropped!

The mbz file is not moved to the targeted directory, even though I can do that manually. The is no permissions issue.

So, the question now is what is causing that database error and what can be done about it.

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

This topic is about as wide as it is long ... much depends ...

'hosting provider' indicates remotely hosted and from the fact you can run the backups via CLI means you have shell access.   IF this is related to DB server config (not saying the error reported is the culprit), do you have access to the (assuming Linux here) my.cnf file to make any tweaks?

Was helping someone else with similar issue ... a reseller account on GoDaddy that was used to allocate a server to themselves for Moodle.   While I could login as the user provided, it clearly was in a 'jail' (ie, user space like /home/customerid/).   Could see public_html where the Moodle code resides.

After installing MySQLTuner on the Moodle box and having to use the WHM on the reseller box ... not the moodle box ... found that the max for InnoDB data could be set to as high as 128Megs and no higher.   Tuner, on the true Moodle box, was recommending 9Gig.  On the moodle box, could see an /etc/my.cnf file but didn't have access rights to be able to edit/change logged on ... and NO appearent way from the WHM box to add/change more parameters in /etc/my.cnf on the moodle box.

All that leading up to asking ... what's your setup with provider?

The fact you can complete the backup via the command line and can see a .mbz file (you've not mentioned the size of that file but it could be a clue) but that .mbz file hasn't been moved to the destination ... begets a couple of questions (sorry) ... 1. the CLI script does provide an option to save the backup to an alternate location.   Did you try/do that?  2. Even though the error says something about temp tables exist and must be removed, am not at all certain that's true ... temp tables may not exist ... but in the config of MySQL is it using the operating system /tmp directory for 'temp' tables?    ls -lR /tmp/ on a CentOS 5 box as root user would show any/all things in there.   See any files belonging to mysql user?   See anything that looks like a .frm file?  Actual temp tables, if in the DB, look like #sql7a79_1_1.  MySQL defaults set the server to remove temp tables when the session that created them ends.  If session that was used to create backup stopped without actually clearing the session ... extremely rare, I would say .. then they could be there.   Ask provider to see if there are any temp tables remaining in DB.   But then again, maybe the error reported isn't really accurate as the backup process does make/records steps etc. in tables related and it didn't get to record the epoch time stamp column that indicates the 'finish time' so Moodle code throws that error.

What's operating system ... specific?   Should be 64 bit whatever.

Do know that on CentOS 5 32 bit this sort of thing happened when course backups reached the 32bit OS limitation of working with files over 4+ gig (same thing happened with Windows servers when using 32 bit versions of PHP on a 64bit box).

There could still be a bug in code ... even in the latest and greatest ... when it comes to autobackups and timing of things ... M31 now handles cron jobs via task list.   There's a task for cleaning up tables.   If that task kicks in while yor autobackups is running it might actually remove tables that were being used to track auto backups.   Shouldn't happen, but ...

The routine for backing up ... when it reaches the very end ... I think it's still using php copy as opposed to php unlink (unlink is like command line 'move') but I couldn't tell you where specifically.

This backup.mbz file you can see in the temp backup directory ... can you cp it to another location OR do you have to use the 'mv' command?

Ok, none of that helps, right?   Sorry bout that!  *You are NOT alone!*

I've had to resort to turning off autobackups on some servers and using a couple of bash shell scripts set to run via OS cron jobs using the 'backup.php' script per course ID and using a destination directory outside of filedir to get them (large) to complete - talking 20+Gig'ers.

Could share back how I do those CLI scripts if you like.

Maybe a real Moodle programmer involved in backups would give us a hint?

Some reading related:

https://www.redhat.com/archives/ext3-users/2010-June/msg00002.html

http://superuser.com/questions/101676/is-there-some-difference-between-mv-and-cp-rm-the-old-file-on-unix

Here's one suggestion (haven't tried it):
https://blndxp.wordpress.com/2016/03/03/fixed-moodle-error-reading-database/

'spirit of sharing'. Ken

Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

Thanks, Ken, for helping me to think through this issue. The error message about temp tables is the closest I have come to some useful hint, but I simply do not have superuser access to mysql, so it is impossible for me to try any changes. At best, I can ask the hosting provider to make a change, but that is just shooting in the dark and it takes forever...if they agree to do it.

I have provided answers to your questions, but I don't see how they really help.

>>All that leading up to asking ... what's your setup with provider?

Although I have telnet/ssh access, I can see and manage only the files under my home directory. I cannot see, for example, the /etc directory. I don't even know if my.cnf is to be found there. In any case, I cannot read the mysql config values directly.I

>>1. the CLI script does provide an option to save the backup to an alternate location.   Did you try/do >>that? 

I give the backup location explicitly - no error there

>>2. Even though the error says something about temp tables exist and must be removed, am not at all >>certain that's true ... temp tables may not exist ... but in the config of MySQL is it using the operating >>system /tmp directory for 'temp' tables?  ls -lR /tmp/ on a CentOS 5 box as root user would show >>any/all things in there.   See any files belonging to mysql user?

mysql.sock

>>  See anything that looks like a .frm file? 

No

>>What's operating system ... specific?   Should be 64 bit whatever.

me:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.9 (wheezy)
Release:        7.9
Codename:       wheezy
me:~$ uname -a
Linux myserver 3.14.40-imu #1 SMP Mon May 4 15:44:25 TZ 2015 x86_64 GNU/Linux

>>There could still be a bug in code ... even in the latest and greatest ... when it comes to autobackups >>and timing of things ... M31 now handles cron jobs via task list.   There's a task for cleaning up tables.   >>If that task kicks in while yor autobackups is running it might actually remove tables that were being >>used to track auto backups.   Shouldn't happen, but ...

The problem occurs too regularly for this to be a random occurrence.u where specifically.


>>This backup.mbz file you can see in the temp backup directory ... can you cp it to another location OR >>do you have to use the 'mv' command?
I can do both

>>Maybe a real Moodle programmer involved in backups would give us a hint?

That would be great. We shouldn't be shooting in the dark about this.


Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

The questions may not appear to help but the answers do help determine your environment - and a guess as to what you can do and what you can't.

First, I do hope you never use telnet to access via terminal/shell.   Am surprised that's even available.   Always use ssh.

Basically, the account you have with service provider is in a 'jail' - which gives you, the user, very little you can do to investigate or arrive at some alternative solution (work-around ).  Unless when you use the ssh account you have you can su to root (substitute user - ie, become root user for the execution of some commands that require root level).   In order to see/know the difference, one would need to acquire a setup from providers that offer that sort of package ... like RackSpace.   You'd have to spend some time exploring the differences but think, just focusing on the issue at hand, those differences are glaring.

I agree that the GUI appears to be broken ... but only in certain circumstances.   The process (long established) for attracting attention to an issue is to submit a tracker item.   In a tracker submission one has to describe the issue *AND* provide a way for those investigating a way to replicate the issue - there's the rub ... not everyone has access to a Wheezy (your environment - same exact ... version PHP, version MySQL, version apache, etc., etc.) setup with your account restrictions AND you'd have to provide them with a FULL user backup of the course ... course contents makes a difference ... so how do you do that IF backups are failing?   Kinda a catch 22 isn't it?

Only other way ... grant an investigator access to your server ... not only as Moodle Admin but also command line.  But, they, then, are in the same jail you are in now.    Been there ... done that.

Example ... the error you saw earlier about 'server has gone away' ... you couldn't fix - had to contact help desk and get them to do it ... and they probably were reluctant to do that.  *IF* you are paying for a VPS - virtual private system - you are the only user on that system, you should be able to su to root AND do it all knowing the package purchased might still have max limits.

But, folks don't want to do things via command line ... especially those who are really teachers/instructors and have little experience at administering a web server.   And because Moodle is pretty heavy on DB, then one also has to deal with being a DB Admin - from command line.

Until Moodle Association members vote for 'fixing' the issue and the issue attains the number of votes to make the issue a priority, then it stays status quo.  

In the meantime, to have backups of courses, individual users ... non-Moodle Association folks ... one will have to resort to 'work-arounds' ... and the only way I've found is command line using my own bash shell scripts.

@Al Beal has said the same thing and provided exactly what I have also discovered ... the CLI does work ... even with large courses ... but use the alternative location ... don't put the backup into the 'soup' called 'filedir' ... the 'new file system' as that adds more queries/etc with the DB itself.

The fact you see a mysql.sock should mean the DB server is on the same machine as the moodle code.  That connection to the DB server supposed to be used only for localhost.

But, the DB user that's setup may not have full priv's to do all things to the DB for Moodle.

Use the info in the config.php file for DB host, DB user, DB password and see if you can connect to the server via command line:

If DB host is localhost:

mysql -u [DBUSER] -p[DBPASSWORD] [ENTER]

IF DB host is an IP:

mysql -h [DBHOST] -u [DBUSER] -p[DBPASSWORD]

There is no  space after the -p AND if the DBPASSWORD contains characters that might be interpreted as escapes ... you might have to enclose the [DBPASSWORD] in 'ticks' ... '!#s\9*!!201lsekessl'

In the above you sub [DBUSER] with the DB user seen in config.php

This is a test to see if you can get there.

To quit the mysql> prompt, type: \q [ENTER]

There are some things one might be able to do from the command line and at the mysql prompt on a temporary basis that could possibly help the backup situation.

'spirit of sharing', Ken



Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

Ken, what are you thinking of to try from the mysql session that one could not do via phpmyadmin, for example?

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Simple question ... not so simple answer! :\  But here goes (this from my perspective):

See, that's an item you've not mentioned ... you have or could install phpmysladmin ...
So you have access to a phpmysqladmin on your VPS server.   That's good.
But ... is it the highest/most secure version?

https://www.cvedetails.com/vulnerability-list/vendor_id-784/cvssscoremin-7/cvssscoremax-7.99/Phpmyadmin.html

Whenever there is a security issue with PHPMyAdmin I see scans on servers to which I have full access where the probe is seeking a phpmyadmin directory by version number.
Some folks installing PHPMyAdmin would follow ... almost verbatum ... directions to install from some blog or even PHPMyAdmin info provided by them.   I've seen experienced server admins try to obscure the location by installing in /p/m/a/phpa/  But guess what ... the pokers/probers have read the same blog!!!   So I see attempts to access /p/m/a/ in hopes of hitting it.   No harm done ... this just an example.

Just for reference:
http://stackoverflow.com/questions/17536425/how-to-set-max-allowed-packet-in-phpmyadmin

Another app ... outside of Moodle that one has to make sure is fixed/patched and
protected IF you manage the server yourself.  And, PHPMyAdmin doesn't have
a button to update itself ... like Webmin does, like WP does, like Joomla does, like Drupal does (Moodle still lacks).

Yes, there is an add-on called Moodle Adminer that is supposed to be like
phpmyslamin.   Dunno that either one will allow the setting of global variables.

There are some MySQL global settings that can be set on a temporary basis. Global variables ... like max_allowed_packet.

When you saw in logs server has gone away in apache error log, one could have used a mysql to see the current settings for that variable AND set it globally to a higher value on a temporary basis.

The catch 22: The SUPER privilege is required to set global variables.

If your setup of DB user is a user that doesn't have SUPER priv's, then can't
set them but could inform the helpdesk with info/exact values, etc..

I have, in my series of scripts to work with large course backups, a getcourses script to show the course ID's and the course title/short name and dump to a courses.txt file.    That allows me to find those large courses ID numbers ...
I use an add-on in Moodle called course size to help with that.
Once I have the course ID number, I know what to provide the backup.php script to backup a specific course from the command line.

What I am thinking ... if one could use a bash shell script to set variables that needed to be increased run the backup then exit.  From the command line I could use the SUPER priv user.

The global variables set via this method would be in affect as long as
the mysqld daemon was running.   If the mysqld daemon restarts (maybe when it's logs are rotated) those are lost.

Now ... am NOT a certified MySQL DB Admin ... just BSEd and MA in something that used to be called 'Instructional Technology' attempting to use/support Moodle.   Won't ever be a MA member (am retired public school educator).

'spirit of sharing', Ken

Average of ratings: -
In reply to Dan Kirk

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Ok, let's clear the air here ... my 'expertise' is being 'retired'.   Hold no 'certificates/degrees/etc' in DB administration nor Moodle Admin ... it's all been OJT over years and years and as needed.   I only know that I don't know enough! ;)

There is still debate about MySQL vs MariaDB ... have servers that run either and no real difference to me ... but didn't have bench marks from when Moodle was first installed/used anyway .... sooooo ... this to say, don't think your issue is related to DB flavor.   If it were, more than likely either would fail.   So check server logs ... not the logs in Moodle ... the MySQL logs and your apache logs.  Might find a clue there. (MySQL server has gone away is a clue ... and fixable)

 Install a plugin called course sizes.   Run it.   That will show approximate size of the course that's having issues.

Also inspect the course ... using a plugin that's not being used in other courses?   If so, what/which?

And again, please inspect the contents of those failed backup directories .... if there is no moodle_backup.xml there it didn't reach a critical stage in building a backup.  But looking at what .xml files are there might give a clue as to where it failed.

Since you've CentOS, you should have commandline access ... run the moodlecode/admin/cli/backup.php script from the command line for the troubled course.  @Josiah has done so and has seen the error that's key to getting the GUI backups to work - that of max_packets.   Maybe that's where your backup is failing as well.

And I know this sounds far fetched ... but not .. running a Linux guest OS inside a Windows Virtual server might also be a factor.   M$ has recently announced they have a Linux of their own now to help in their Virtualized environments (something over 60% of the guest OS's are Linux based they have discovered).   Hmmmmm ... no surprise there!    IF that is a factor, you'll never see anything that makes sense in your CentOS 7 logs.    Not saying that's the issue, but wonder if you've asked the Virtual Server admin if he/she has seen anything out of the ordinary.   Once had a Moodle on VMWare.   Every hour at 37 after the hour the Moodle was in-accessible for 5 minutes.  "Customer" kept telling me it was something in the virt CentOS and/or Moodle causing that outage.   Finally they looked at VMWare ... yep, sure enough ... some other process or guest OS on the same physical VMWare server as the Moodle and Guest OS CentOS 6 was given 100% resources to do something at 37 after the hour.

It's fixable ... keep digging! ;)

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Course backup hangs - no error logs

by Al Beal -
I had problems where my course backup would fail too, it kept hitting a percentage and that would be it. 


I found CLI backups worked for me. In fact I just did another this morning, it does dump it with full options, so one of my courses was 1.8gb! I restored it locally, then backed it up again but without user data for a more manageable size.


They are done by course ID. So go to your database, look for mdl_course . Here you will have a list of the courses and their IDs. In the CLI, it will be something like:

php moodle/admin/cli/backup.php --courseid=x --destination="where ever you want"


Al

Average of ratings: Useful (2)
In reply to Al Beal

Re: Course backup hangs - no error logs

by Josiah Carberry -

Thanks for this feedback, Al. It seems we are seeing similar phenomena, although who knows if the causes are the seem. Be that as it may, if the CLI course backup works, but the client GUI backup fails, surely there is an issue with either a) a bug in the software that only gets hit in certain circumstances; and/or insufficient information about how to set certain parameters.  I do wish the developers of this functionality could shed more light on the subject.

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

I have no expertise on this, but I wonder if the GUI backup relies on some handshaking with the browser, and maybe the browser is timing out.  How long did the CLI backup take?  And how many MB/GB was the course backup file?

Average of ratings: -
In reply to Rick Jerz

Re: Course backup hangs - no error logs

by Josiah Carberry -

The session timeout is set to the default 2 hours. The whole backup procedure takes only a few minutes (until it fails), be it with the GUI or the CLI. The resulting mbz file is about 700M, but moving (or copying) it to the final destination should only take an instant (at least, that's how long it takes when I give the command manually). So, I don't suppose the client is timing out.

Average of ratings: -
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Have been digging into code related to backup.   Moodle uses moodledata/temp/backup/ as the build directory for a backup.   It assigns a directory like a content hash (long name of letters and numbers).  It also creates a text based .log file by the same hash name with .log extension.   Prior to Moodle 3.1 when the backup concluded successfully the .log file was cleared of content and thus remained a 0 byte size .log file.   And the hash  directory was also removed.

Now, in 3.1, that log file remains and IF the built backup has failed at the very last action in the process, so does the built .mbz ... in the hash named directory.   Think the 'spyglass' icon in the autobackups report one sees next to each backup actually reads that .log file.   I had been trying to find what the numbers in that log meant ... finally found it:

In backup.class.php of /backup/ of code

   // Execution mode
    const EXECUTION_INMEDIATE = 1;
    const EXECUTION_DELAYED   = 2;

    // Status of the backup_controller
    const STATUS_CREATED     = 100;
    const STATUS_REQUIRE_CONV= 200;
    const STATUS_PLANNED     = 300;
    const STATUS_CONFIGURED  = 400;
    const STATUS_SETTING_UI  = 500;
    const STATUS_NEED_PRECHECK=600;
    const STATUS_AWAITING    = 700;
    const STATUS_EXECUTING   = 800;
    const STATUS_FINISHED_ERR= 900;
    const STATUS_FINISHED_OK =1000;

First, don't ask me what they mean ... am not a programmer ... so am trying to apply what logic I can to what I see.

I don't have a site right now that has backups failing ... got them working via bash shell scripts, etc. and don't want to break them again.    But, since you do have a system having issues, please check:

What's the last entry in that hash.log file that remains?

From what you've described, the hashed directory name that was being used to build the backup *does have* a completed backup ... a .mbz file ... that you can *move* out of the temp build directory to another location.

Trying to determine exactly where that logging stops - 700, 800, or 900?   If it reaches 1000 then I assume it's been able to move/copy the .mbz file to destination AND remove the hash temp directory it was using to build leaving only the hash.log file.

Something else found in the code of the backup process:

core_php_time_limit::raise(1 * 60 * 60); // 1 hour for 1 course initially granted
raise_memory_limit(MEMORY_EXTRA)

On systems that are restricted the process might not be able to raise the extra memory needed ... not increase the time needed to complete - that's a guess.

There are, however, two variables that Moodle Admin can set in the UI that relate.   Found in:

Admin -> Server -> Performance

Extra Memory limit ... has a drop down pick list.   Default for it is 512M.    Comment there in settings says:

Some scripts like search, backup/restore or cron require more memory. Set higher values for large sites.

What does your drop down for that variable have?    What would happen if that were set higher?

Also a Maximum time limit (default is 0) with a comment ... comment says:

"To restrict the maximum PHP execution time that Moodle will allow without any output being displayed, enter a value in seconds here. 0 means that Moodle default restrictions are used. If you have a front-end server with its own time limit, set this value lower to receive PHP errors in logs. Does not apply to CLI scripts."

If you change those parameters, make note of them ... and remember where you set them in case you need to tweak them again (using less).

Am gathering that each 'module' has it's own backup routine ... quizzes backup routine is unique to quiz ... forums backup routine is unique to forums ... etc.   This to say (and have seen this in realtime via terminal) the  backups hit one of those that is heavy ... it takes a long time to complete.   Maybe longer than is allowed/restricted by something.

'spirit of sharing', Ken



Average of ratings: Useful (1)
In reply to Ken Task

Re: Course backup hangs - no error logs

by Al Beal -

Yeah Josiah, my adventure ended here though. (pretext: I took over a erroneous Moodle site, after battling with it for 2/3 years this summer I did  fresh install).

I did however, largely with Kens help (i was a former lurker trying to be more active now I have some experience) limit my problems. After give a second glance at previous posts though i see everything has been said already!

My crons kept failing too, my log files were empty but there were hash folders with partial course backups. I would manually reset the complete status in the database so they could try again the next day, i tried increasing all associated parameters too. Eventually my site would eat itself due to the cron not getting to the delete older copies bit.

I can't help technically but my logical step would be to duplicate the course with no user data, if it backs up clean then you can focus on the user data. I had a feeling my quizzes were the problem, but couldn't prove it.



Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

So I have experimented with changing the extra memory limit setting (which can also be set in config.php).

512M: mbz not created

1024M: mbz created; size is 774383413; move to target directory fails

2048M: mbz created; size is 774383325; move to target directory fails

3072M: mbz created; size is 774382973; move to target directory fails

3072M: mbz created; size is 774383225; move to target directory fails

4096M: mbz created; size is 591577088; move to target directory fails

Next, I tried to understand the different sizes of the mbz file. I suppose that the smaller size for the 4096M setting might be due to the simple failure to allocate all that memory, or perhaps allocating that memory reduced the available memory for something else. But I was more concerned with the two different sizes for the two runs at 3072M. So I tried to determine if the problem were in the compressed archive or in the xml files being archived. I stopped this analysis when I found that the block.xml files created for a certain block (random glossary entry) were not the same! In one case, the <configdata> value was different in the two cases was different, They started off the same, but diverged after a while, and one xml file had 452 more characters than the other. In fact, all the block.xml files for this block are different. Maybe the file contains the value for the next random entry, so the difference is normal.

Otherwise, the xml files seem to be the same, although I have not rigorously compared them all.

It goes without saying that 1) there were no other open sessions during these tests; and 2) there were absolutely no changes to the course at all during these tests.

Finally, when I have debugging enabled, I get a log similar to the following:

[Sun 31 Jul 2016 08:56:16 CEST] [info] instantiating backup controller e1a8ec8142bcd6bb5a8874cf45ee893b
[Sun 31 Jul 2016 08:56:16 CEST] [debug] setting controller status to 100
[Sun 31 Jul 2016 08:56:16 CEST] [debug] loading controller plan
[Sun 31 Jul 2016 08:56:16 CEST] [debug] setting controller status to 300
[Sun 31 Jul 2016 08:56:16 CEST] [debug] applying plan defaults
[Sun 31 Jul 2016 08:56:16 CEST] [debug] setting controller status to 400
[Sun 31 Jul 2016 08:56:16 CEST] [debug] setting file inclusion to 1
[Sun 31 Jul 2016 08:56:16 CEST] [info] checking plan security
[Sun 31 Jul 2016 08:56:16 CEST] [debug] setting controller status to 500
[Sun 31 Jul 2016 08:56:16 CEST] [info] checking plan security
[Sun 31 Jul 2016 08:56:16 CEST] [info] checking plan security
[Sun 31 Jul 2016 08:56:16 CEST] [info] checking plan security
[Sun 31 Jul 2016 08:56:16 CEST] [debug] saving controller to db
[Sun 31 Jul 2016 08:56:16 CEST] [debug] calculating controller checksum 6af85d2608395aac78197dee7a4edd34
[Sun 31 Jul 2016 08:56:22 CEST] [debug] loading controller from db
[Sun 31 Jul 2016 08:56:22 CEST] [info] checking plan security
[Sun 31 Jul 2016 08:56:22 CEST] [info] checking plan security
[Sun 31 Jul 2016 08:56:22 CEST] [debug] setting controller status to 700
[Sun 31 Jul 2016 08:56:22 CEST] [debug] saving controller to db
[Sun 31 Jul 2016 08:56:22 CEST] [debug] calculating controller checksum 54588700c04bdaf3d8aac09f933c0eb9
[Sun 31 Jul 2016 08:56:22 CEST] [debug] loading controller from db
[Sun 31 Jul 2016 08:56:22 CEST] [debug] setting controller status to 800

When the backup finally hangs, it fails to add a final entry showing either a 900 or a 1000 status.

So, my conclusion is as follows:

1. In my case, the default 512M for extra memory is not sufficient to create the mbz file

2. At least 1024M appears to be sufficient

3. However, it is possible to put the setting too high, depending on the available memory and how the server is configured and what other memory needs there might be.

4. I am assuming that the differences in mbz file size are due to the differences in the block.xml described above, and hope that these differences are normal. I will try to confirm with the plugin author.

5. The failure to copy the mbz file to the final destination and the deletion of the temporary xml files remains unexplained. I note, however, that this failure also occurs when doing the backup using the CLI.

Average of ratings: Useful (1)
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Ken Task -
Picture of Particularly helpful Moodlers

Similar to you ...

The -d option to a PHP CLI will allow you to set a custom value for any of the configuration directives allowed in php.ini.

Example: to extend the memory limit of a script (default might be 512M) to a
Gig php -d memory_limit=1024M ... script.php

for the backup script in moodlecode/admin/cli/ it works.

php -d memory_limit=2048M backup.php --courseid=# --destination=/var/www/unifsrepo/m31/

In trying this out on a 16Gig machine (2.9.7 (Build: 20160711)) that has 2741080k free while watching top, could see PHP kick in AND  if I increased the memory limit from the command line the backup actually took less time on smaller courses (appeared to).

Then, went after the largest ...

[root@moodle cli]# php -d memory_limit=2048M backup.php --courseid=214 --destination=/home/clibackups/
== Performing backup... ==
packet_write_wait: Connection to 65.xxx.xxx.xxx: Broken pipe

After about 3 hours or so ... had to zzzz out on the couch (am old and need naps!)

The ssh session times out before the CLI backup script completes.

*** Yet the script is still running on server.   Using watch "ls -l;df" on the hash directory being used to build the .mbz shows the .mbz growing.

It completed.  It is a valid archive - I see a backup.mbz file among the xml files and directories used to build - key xml file ... moodle_backup.xml.

58435706443 Aug  8 20:42 backup-moodle2-course-214-dim_-_cole-20160808-1836.mbz

That's a 58.4 Gig backup.

The course itself has no quizzes - largest xml file in the archive is files.xml
and the largest directory is, of course, files.

In this version of Moodle. the log file in moodledata/temp/backup/ is 0'd out
so there were no references to to anything.   In version 3.1 those log files
remain.

In this version of Moodle, there are the scheduled task.
In the task list there is an item:

Clean backup tables and logs \core\task\backup_cleanup_task
and
\core\task\file_temp_cleanup_task

IF enabled and run via cron job think there is potential for conflict with
whatever backup one is attempting unless there is something that checks
to see if an auto backup or backup process is still in progress and reschedules
for later.

From command line one can get a quick overview of all:
php admin/tool/task/schedule_task.php --list

View of top:

Mem:  16333944k total, 16054560k used,   279384k free,   579508k buffers
Swap:  8208380k total,    22872k used,  8185508k free, 11478420k cached

It's an all in one box ... apache + DB.  A MySQL tuner clip:
[OK] Maximum reached memory usage: 6.2G (40.03% of installed RAM)
[OK] Maximum possible memory usage: 7.4G (47.70% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available

In the DB: mdl_backup_controllers
shows one row for the course with a status of 700 and
time modified has 0 (other entries have an epoch time stamp reference)
There is a long controller field. So never recorded 800 ... which is executing
but it was executing cause I watched it in realtime with 'watch'
and never reached 900 or 1000.
 
Nothing in mdl_backup_logs

[root@moodle log]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited

so I am at a loss ... the .mbz appears to be a valid archive/backup, although
I have NOT attempted to restore the course.   Something/somewhere is causing
termination of the last steps in process ... that of copying the .mbz to
destination and thus also cleaning up the temp directory used to build.

One thing for sure ... with a course that size, automated backups will never complete successfully on that site.  And, will have to be very creative with any bash shell script.

If anyone has any ideas would love to hear 'em! ;)

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Course backup hangs - no error logs

by Josiah Carberry -

This message is a sort of postscript on this issue. I recently did a major change in my site, involving the replacement of all videos that were previously stored on the server and played using the videofile plugin. They are now stored on one of the principal video sites and played using the URL resource. I did this because of playback reliability on various client platforms, rather than due to the course backup issue. At the same time, I upgraded the site to Moodle version 3.1.2.

The upshot is that the course backup function now works extremely quickly and no longer hangs. Of course, the resulting backup file is a tiny fraction of the previous size, given that there are no longer any video files involved.

This leads me to believe that all the previous issues were largely due to insufficient resources on my server, which may have led to timeouts (or perhaps not). Now, perhaps it is my fault for trying to run Moodle on a platform that is not adapted to the application's requirements. On the other hand, I did allocate more than an order of magnitude greater resources (number of CPUs and memory) than either the Moodle documentation or any anecdotal remarks in this forum suggested as necessary. So I wonder whether it might be appropriate to rethink the algorithms in use that require so many resources. I wonder what Donald Knuth would say?

Average of ratings: Useful (1)
In reply to Josiah Carberry

Re: Course backup hangs - no error logs

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
On the topic of performance check the resources in the "Hardware and performance" forum: https://moodle.org/mod/forum/view.php?id=596. (Yes, the resources are visible only when you compose a new discussion.)

P.S. Moodle is not the only software construction which is not compatible with Don Knuth nor Niklaus Wirth Plea for lean software.
;-P
Average of ratings: -