Backup fails

Backup fails

by Jonathan Rider -
Number of replies: 21

I have a site on a shared server and am using Moodle 2.9.5+ (Build: 20160324). The course content has been restored many times from previous courses over about 2 years. Unable to get auto backups working, I have been doing manual backups for some time. However now I am getting persistent failures. I put the site in maintenance mode and turned on debugging - the following error messages resulted; I am unclear about next steps - can anyone help? Please advise if additional information may be critical (I'm not too tech savvy):

Warning: Error while sending QUERY packet. PID=180775 in /home/ride1527/public_html/eco16/lib/dml/mysqli_native_moodle_database.php on line 1079
Error reading from database

Fatal error: Uncaught exception 'dml_read_exception' with message 'Error reading from database'
in /home/ride1527/public_html/eco16/lib/dml/moodle_database.php:
443 Stack trace: #0 /home/ride1527/public_html/eco16/lib/dml/mysqli_native_moodle_database.php(526):
moodle_database->query_end(false) #1 /home/ride1527/public_html/eco16/lib/ddl/sql_generator.php(230):
mysqli_native_moodle_database->get_tables() #2 /home/ride1527/public_html/eco16/lib/ddl/database_manager.php(99):
sql_generator->table_exists(Object(xmldb_table)) #3 /home/ride1527/public_html/eco16/lib/ddl/database_manager.php(318):
database_manager->table_exists(Object(xmldb_table)) #4 /home/ride1527/public_html/eco16/lib/dml/moodle_temptables.php(140):
database_manager->drop_table(Object(xmldb_table)) #5 /home/ride1527/public_html/eco16/lib/dml/moodle_database.php(370):
moodle_temptables->dispose() #6 /home/ride1527/public_html/eco16/lib/dml/mysqli_native_moodle_database.php(481):
moodle_database->dispose() #7 /home/ride1527/public_html/eco16/lib/dml/moodle_datab in /home/ride1527/public_html/eco16/lib/dml/moodle_database.php on line 443

Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Well, its a database issue.  Have you converted your database to barracuda? - https://docs.moodle.org/28/en/Administration_via_command_line

You might try running a check and repair on the database too - just make sure you backup first.


Average of ratings: -
In reply to Emma Richardson

Re: Backup fails

by Jonathan Rider -

I am aware of the Barracuda thing - however my service provider does not permit that format.

As the persistence of the problem is recent, I believe the barracuda format is not the issue. Any other ideas? I thought the stack trace points to some temp tables that need to be dropped?

Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

Think that temp table thing is really mis-leading and may NOT be the cause.    In reading over MySQL docs on temp tables, they are created only for the session and are removed after a session ends.   That session is initiated by the DB user as in your config.php file.    However, having said that, and you've probably done this correctly but I think worth a check ... the priv's of the DB user as in config.php need to pretty much have all priv's to the DB for Moodle.    Think you can look at users and the priv's via PHPMyAdmin (I don't use that) or in your hosting providers tool (whatever that is) to see DB users and their priv's.

Something else ... guess without really seeing ... this course has been used many times.   As courses go, it seems to me they only grow .. more activities/etc.  ... they generally don't get smaller.    Two of the heavier processes for Moodle are quiz and the backups - either by the manual method or automatted.

Since you are on a shared hosting plan, what are the limits related to DB usage?

Don't think I"ve ever seen "sending QUERY packet. PID=180775"  but that's a definite clue!

PID is the process ID and doesn't tell us much unless in DB error logs there is a reference to that PID.   That packet stuff however, might be related to max_packets_allowed ... which by default is 16M ... with an expanding course  am almost willing to bet some data in some activity/quiz etc. has reached that max.

Normally, one sees a error that says 'Server has gone away' but due to shared hosting, etc. could be you are seeing something specific to the package/hosting.

Ask provider what is max_packets_allowed for DB's on the DB server.  And ask if your's can be increased to double 16M ... 32M (I can't give you a value ... that's a guess).

If you google for 'sending Query packet' types of errors, you'll see other apps ... not just Moodle ... have similar issue.   And in nearly all I've found, responses say it's max_packets_allowed.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Jonathan Rider -

My host has confirmed my max_packets_allowed = 33MB.

I am surprised if the quizzes have passed the limit as the course is not that big (e.g. .mbz backup sizes range around 300MB).

To workaround - should I try manual backup of the course in parts and then to restore, merge the parts into one course?

Any other ideas?

Thanks

Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Jonathan Rider -

I have attempted (unsuccessfully) to backup parts and merge in restore.

I have confirmed the DB user has all privileges.

I have been trying different privileges for the config.php file but no success yet; I have been unable to find what these should be in the moodle documentation; is it 0644? 0744?

Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

If everything else is working, I doubt that the config file is your issue.  644 should be sufficient, I believe.  I am sure Ken can confirm.

Have you tried by chance enabling the new backup method?  Found in server settings?  Probably won't help but worth a try.  Is this all on one course?  Or does this occur with every course?


Average of ratings: -
In reply to Emma Richardson

Re: Backup fails

by Jonathan Rider -

Thanks for your kind attention - I will try to get the auto backup working.

Everything on the site works fine as far as I can tell.


Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

This has nothing to do with config.php file permissons.   The only permissions needed on config.php are:

readable by owner/group (that would be the user under which apache service runs), writable by owner, (optional) writable by group all others need no permissions.  

On CentOS Linux server looks like:

-rw-r----- 1 apache apache 732 Apr 26 00:07 config.php

It is a database error .. not in writing to the DB on the DB server, but reading from.

In the config.php file, there is a line for dbhost.   On standalone/not shared servers, that is normally 'localhost' .. which means the DB server *is* on the same server as apache and your Moodle code.

If that variable has an IP address or some hostname, its an external DB server ... some where inside the providers network.

Am pretty sure providers won't give cusotmers tools to test connectivity to shared DB servers  ... like mysqlslap (I think it's called).   About the only thing one could do IF they had ssh access to their moodle server is ping the IP address of the remote DB server.   That should return no errors but that's at the network layer and what needs to be tested is really at the application layer .. the kinds of things mysqlslap can do.

There is no tool in Moodle, of which I am aware, that can test in this fashion.

Am really at a loss as to what to tell you ... other than communicating with your provider ... the specific error and ask if there are caps/maxes in reading from DB's.

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Jonathan Rider -

Thanks for your suggestions - my problem may be some corruption in the DB from recycling the course so many times and upgrading to successive versions. I may need to simply rebuild a clean site.

My moodle site is in a directory of my main site - I doubt this is a factor as I usually can obtain a backup but recently I get persistent failures. That said - I have never been able to get the auto backup functioning properly.

I am using the Essential theme - could that be a factor?


Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

The location of the site should have no affect, as long as you have permissions right on all your files - you might double check.  When you say you have never got auto backups running, what exactly do you mean.  What happens?  

If you cannot pull a backup manually, then you will not be able to pull one automatically.  You could try with clean theme but I don't think that is the issue either.

Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

We keep reaching for straws here when bringing up themes (although there have been isssues with themes in the past, but don't think any of those issues affected backups).   Doesn't have any reference in the error ... concentrate on the error.

Clue: moodle site in a directory of main site - means you are running something else ... like a WordPress?

And ... never able to get autobackups running ... at all ...

So let's quite trying to hide the provider - if they are doing such a terrible job supporting Moodles then community users deserve to know,   And, you might have to seek another provider that is more Moodle friendly.   Since your site is already on the internet and accessible (am assuming it's not a private anything) then why hide provider?   You're not training for a government entity on espionage or weapons or such, are you?

So who is the provider?

Sometimes, providers have FAQ's/Docs/their own forums available to the public ... sometimes they have private forums for customers only.   Would do some 'leg work' for ya IF I knew the provider.

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Jonathan Rider -

The site host is Hostpapa - they have provided good support in my opinion. No intention to hide the host - I didn't think that was pertinent.

The main site is a joomla site but I have had no issues running moodle on a directory site. I did have a moodle-specific site but have cancelled it as I was not experiencing problems with the directory site.

I was able just now to successfully backup one course - but the problem remains with a course that prompted this query originally.

Could the directory site be an issue with the auto backup? I have wondered about this before - and at one time set up a separate site for that reason, but then was able to get the original site working in a directory. If this is a problem, I can take steps to have the moodle site on the main address.


Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks for sharing the hoster.   Will seek and see what can find for ya.

There isn't a problem, or shouldn't be a problem, running Moodle in a subdirectory, but op might be able to create issues ... unknowingly.    Example: to harden Joomla sites, one might resort to rewrite rules or using a recommended (for Joomla) .htaccess file.   It's possible those could create issues for Moodle.   In your case, don't think you are using either of those to harden a Joomla ... but  only you (the op) would know.

Something else to check ... Moodle now has task ... which is like a cron job ... one of those is to clean up temp space/area in Moodle.   Guess where those backups are built?   Yep, same directory ...

moodledata/temp/backup/ is used to build backups and it's also used to restore courses.

With what ever you have to browse files, check out moodledata/temp/backup/ to see if there is anything in there.    There should not be.   It is safe to manually remove any/all files/directories found in moodledata/temp/backup/ IF Moodle hasn't done it.

Now ... just so I get this straight ... you just today was successful in making another course backup.   But, this troubled course still can't be backed up.   OR it can, but the backup can't be restored .... which is it?

If that course backup or restore fails, does it fail with the SAME exact error every time?

If I provided you a place to upload this backup for me to inspect, would you be interested?   Once one has a .mbz it can be inspected ... and if need be, edited, then rebuilt as a .mbz backup.   Only if you know what you are doing .... not to say I will be successful with this one, but I have been succesful recovering troubled courses before.

If interested ... PM me and I'll send a URL where you can go to acquire an account.   You'll have to PM me when done so I can setup a tinker area for ya so you can upload.

'spirit of sharing', Ken




Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

Ok a quick look at your hosting provider ... what plan did you purchase?

MySQL 5.1 Databases is what I saw .... at:

https://www.hostpapa.com/web-hosting-plan/detailed-chart/

Moodle 3:

https://docs.moodle.org/dev/Moodle_3.0_release_notes#Database_requirements

says: 5.5.31 is minimum required for 3.

The issue you are having is a DB problem .... sooooo??????

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Jonathan Rider -

My host provides PHP version 5.5.30; my moodle version is Moodle 2.9.6 (Build: 20160509) - I think this is OK. The environment check in moodle confirms min requirements are met - but it does mention the barracuda format (which is unavailable to me).

Regarding the backups - I just upgraded to the above build from 2.9.5 and have a backup of one course. The other course (only running two currently) is the one I cannot seem to backup.

The moodledata/temp/backup file has many 4KB and 0KB files (failed backups?) I'm not sure if one of those is the file you are offering to examine.

The backup failure usually occurs at around the 96.4% point.

If you think there is some benefit to examine one of these I can make it available - let me know; I have not used PM but will watch for messages.

Really appreciate your interest - wish my tech skills were worthy!


Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

Well, then provider needs to update their web pages ... that said they offer MySQL 5.1

For 2.9 Moodle recommends MySQL 5.5.31:

https://download.moodle.org/releases/security/

The environmental checks code can be hacked to hide skip thus not error certain requirements ... like that of the version of MySQL recommended by Moodle HQ.   Am not saying your provider did that, but ....

Those 4KB files indicate failed backups.   They have 'contenthash' file names .... ie, a bunch of letters and numbers and have extension of .log.   They are text/asciii.   If you could, open one of them to see what it says inside.   Think they might show the stages (plan) by an ID number ... 100, 200, etc.   If that log gets to something like 800 (sure would be nice if program would use a single word to indicate what those numbers mean)  see below:

The fact backups get to 96.4% is significant.   The very last thing the backup process does is 'copy' the built backup.mbz file to it's destination according to choices/options OP (ie, you) has chosen.   THe default it to save the backup in the 'file area' (moodledata/filedir/) in an obscured fashion (contenthash named - not humanly recognizable).   Many systems that are under-powered fail at 'copy'.   If one had command line access, however, one can 'move' a backup.mbz file to a location outside of moodledata/filedir/.    It sounds like the package you have purchased on a shared system (might be very affordable - saw they offer $2.95 a month but generally means there are caps/restrictions really unknown to customers, until they try to run something, like a Moodle, that pushes over those caps/restrictions.

In your version of Moodle (2.9.x), Site Admin, Courses, Backups, General Backup defaults, one could select items to backup or deselect.    Since this course has been used a lot, suggest unchecking some items in that list and exclude things to get a minimum backup ... exclude users, exclude logs, etc.   Goal is to reduce the backup and thus the processing required to back the course up.   Am assuming we want a course backup that could be restored like a fresh course.  If the logs for this course have never been reduced, that could possibly lead to issues with the amount of data contained therein.

There is also something new called 'task' which is like a laundry list for cron jobs broken out    There's one for Clean backup tables and logs   That should be off for now.  You might need to turn it back on after this issue is resolved.

You can check on versions of things ... Site Admin -> Server -> PHP Info will show not only php version, but things like Operating system - the first line in the info displayed.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

Follow up ... been in PM and other communications with OP.   One completed backup uploaded to a sandbox server (not restircted)  restored without issue.   Course really not a 'large one'.   One failed backup that was uploaded (op found in moodledata/temp/backup/) did not contain a moodle_backup.xml file ... the road map for restoring.    The log files ... reach 800 ... but didn't reach 1000 - the latter is the last step of copying the built backup to preferred area ... filedir.

There's only one more suggestion I have ... which I should have suggested before ....

The error is strange but could be due to either a PHP time out or a PHP memory limit.

Cannot find in Hostpapa's FAQ's KB any reference to how a client could increase the time out from the default 30 seconds NOR how a client could increase the memory used by a script.    Those values can be seen by going to Site Admin -> Server -> PHPInfo

IF possible with hosting provider .. OP will have to communicate with them ... increase the time out to 60 seconds and the memory limit to at least 128.

Did find postings for Drupal and the onversation said there Hostpapa had a 30 Meg limit for memory and wern't gonna increase/change - but that was in 2009 (full truth).

Providers KB is, however,  seriously out of date:

http://hostpapasupport.com/kb/versions-of-php-mysql-cpanel-linux-etc/

and in-correct .. the page above list MySQL has being version 14.14

Clip from the link above (appears to be CentOS 5 (the el5 seen below) or RedHat Enterprise 5 ... which, will soon fall into 'non-supported for updates'

All new accounts receive the following versions:

PHP: 5.3.26
MySQL: 14.14
cPanel: 11.36.1 Build 8
Theme: cPanel X v3
Operating system: Linux
Apache: 2.2.20 (Unix)
PERL: 5.10.1
Path to PERL /usr/bin/perl
Path to sendmail /usr/sbin/sendmail

IMHO ... the bottom line is this ... they might be affordable but not a good hosting solution for 2.9 IF the site grows over the requirements (php memory, php time outs, etc.) - which, I imagine is true of nearly all hosting providers.

Can't drive a bus that's governored for 55 MPH over 55 MPH ... no matter how hard one mashes foot on acellerator!

Did discuss with OP how courses might be re-designed to be able to backup sections in one backup and then other sections for resources in a 2nd backup ... then restore both to a single course.   Not ideal, but might be the only way with situation such that it is.

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Jonathan Rider -

Thanks again Ken for your insights. As discussed, I will revise backup defaults to shed unnecessary baggage. I understand my problem is not a moodle issue but is caused by my hosting plan's limitations. I will most likely muddle along with my current plan.

This query can be closed now with much thanks!

Average of ratings: -
In reply to Jonathan Rider

Re: Backup fails

by Doug Moody -

Jonathan,


Though you have closed this post, I see Ken Task has been helping you. Ken is a wonderfully knowledgeable poster, and has helped me numerous times (thank yo uvery much Ken)

This particular issue has plagued me for a long time in my own shared hosting of Moodle. I suspect now that the issue is mostly due to the tim eit takes to perform the final step in backing up, that is, copying it to the designated directory.

If I could only know for certain what causes a timeout in copying, I could contact my provider and "beg" for more resources. But I don't want to do that unless I could state with certainty it would fix my problem.

For me, my automated backups backup courses that are generally less than 300MB. Anything more than that always fails. Curiously, if I try to manually back them up, it works! That's my conundrum...why it works manually but not automated?

Additionally, I have experimented with what will work in a backup and what won't. Apparently, when I don't ask it to backup assignments as part of the backup package, it always works. Of course, this is not acceptable, since I always need a full backup, not a partial one.

So, I am asking if, since you closed this post, if you have found anything else that worked for you. If so, would you mind sharing?

Thanks.

Average of ratings: -
In reply to Doug Moody

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

@Doug ... Thanks for that ... guess it was a way to say ... stay out and let Jonthan answer ... a polite way. smile

Did some more digging ... much to be discussed with hosting provider ... can't offer a solution.

On this page:
http://support.hostgator.com/articles/hosting-guide/hosting-plan-comparison/hosting-comparison-chart
There is an interesting section of information entitled: Server Resources

Under a column they call 'Feature'

Server Resources Limit and for 'shared' I see a % - 25%
(does this mean there are only 4 customers on a shared system?)
http://support.hostgator.com/articles/cpu-resource-usage
It is unlikely that you hit the 25% CPU usage for 90 seconds at a time, unless your site is script intensive and requires more processing.

One thing they don't say there ... a process that hits those limits is killed and reports NO errors - it's like a kill -9 on a process via command line.

Simultaneous Processes and for 'shared' it's 25.
http://support.hostgator.com/articles/processes-limit

Now that talks about just the Web server.   What of the DB server where the DB resides?  There are lots of queries of the DB in a backup process.

On backups of a large course in a dedicated server running it all ... web and DB ...
it's not unusual to see 'top' show the php process hit 100% of the CPU on
a 16Gig Mem - 4 core server.  It doesn't stay 100% very long ... a second or two ... then
drops down somewhere in the 80-90% range and then vastllates 70->80_>90-> etc.  ... but it's consistently high compared to other processes - top command polls every 2 seconds I think.

In an automated run of all courses, one can tell when one course backup has been completed
cause just before the next course in the list begins it's process, CPU usage/MySQL usage
drops to 'normal levels' (whatever that means) ... but doesn't stay there long.

A backup in Moodle of a single course done from the course menu is one thing.
That's using the scripts on the web server that talk to that backhanded DB server and your DB.

Automated backups for the site ... all courses ... is another.  It's really the same process as backing up a single course from the course menu, it's just one right after another as quickly as Moodle can run it.   Dunno how often a HostGator setup would start a timer on a process and stop watching but could imagine the 'watcher' is still watching/calculating/timing and very well could reach that 90 second limit on a process that did hit 25% for an instance ... enough to trigger.

If you stay with your hosting plan, the only thing that I could offer/suggest that might help is to look at those assignments and see if you couldn't find a way to make them 'smaller' ... or not as many.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Backup fails

by Ken Task -
Picture of Particularly helpful Moodlers

@ both of you ...

more digging found this:

Possible work-around ... don't host with them so dunno but from what I read ..
might be possible:

hostgator reference

http://support.hostgator.com/articles/cpanel/what-do-i-put-for-the-cron-job-command

in /moodlecode/admin/cli/

create a text file called courses.txt  make sure it's readable.
In courses.txt put the course ID's of the courses you want to backup one course ID per line.
1 is your front page.

So courses.txt **might** look like:
1
24
3
407
133

Next create a file called 'runbu.sh' (stands for 'run backups')
In runbu.sh these lines ** adjust the paths for your account.   Just following what the HG reference
showed.

#!/bin/bash
#
for i in `cat /home/username/public_html/[moodle?]/admin/cli/courses.txt`
do
    echo "Site in que: $i";
    echo "---------------------------------";
    php /home/username/public_html/[moodle?]/admin/cli/backup.php --courseid=$i --destination=/full/path/to/densination/directory/;
    echo "***** done with $i *****";
done

** make the file runbu.sh executable.

The php line above wrapped here, I think ...  that one line ends with ';'
Run from cron job

The destination directory could be in /fullpath/to/moodledata/repository/backups/

You *must* create the 'backups' directory manually and set ownership/permissions such that the user thus Moodle can read/write to it.   This script will NOT create that directory for you.


You could then setup a file system repo from any course to point to that repo to restore courses if you needed to
AND you'd see the backup files sizes as well as their full humanly recognizable names.

Otherwise, if you leave off the designated directory option, it will go into moodledata/filedir/ ... but who knows where!

The cli/backup.php script takes preferences for what to backup from the Moodle Admin UI settings for backups - not automated backup preferences.   Defaults for that are usually a full course backup including users and their work.

Why would this work and the automated backups normally run fail?

Each course is backed up as a separate process ... one at a time.

Automated backups is ONE process that might/probably IS going over limitations/restrictions on shared accounts.

If this works, great ... you have a work around to backup.  ** BUT ** wouldn't trust that I could restore such a course without testing.   No testing could mean you fool yourself.

Shouldn't hurt to try it.

'spirit of sharing', Ken

Average of ratings: -