Interesting backup and restore conundrum

Interesting backup and restore conundrum

by Tammy Moore -
Number of replies: 12

Primary Question:

I remember in Moodle 1.9, when the restore to new course was proceeding there was a line that actually gave text-based information as to what step it was on. Though Moodle 2.x doesn't seem to have that, I figure there is a way to still dig down somewhere to see it. I am pretty shaky at SSH, bit confident in Control Panel. If there is a Moodle or Control Panel way to do it I would jump for joy. But this is a conundrum important enough to solve to do SSH if needed. I need to get over being intimidated by making a mistake there and crashing something anyway. I am one of those teachers that learned as we grew types (we are going into our 12th year of the project)  instead of being a server side guru. It makes me feel like I have many holes in my skill set, but my time is loaded down with admining, building, and teaching, so there isn't a daily need for the server side stuff and it is lost in competing with other time needs. 


Want to help more? ...

If anyone wants to jump in with further suggestions, please do. Here is the needed background if you want to help more ...


We are having issues with our math course backups and restores specifically. All other courses, even some bigger in terms of mb are backing up and restoring fine. So there is some commonality to these math courses I am trying to track down.


Here is the behavior in backups  ...

Automated backups work on all of them. Manual backups stalls at the 99% point in the backup for about 80% of them. 


Here is the behavior in the restore to new course of the few math courses that do backup fine ...

Of the manual backups that did work and of the automated restores, the restore gets to 25% and hangs. The spinning progress wheel in the browser tab stops as well, so it appears Moodle/server is no longer working on anything. This occurs in about ten minutes or less and is well below our timeout settings on the server. No errors in error log. It just stops. Looking into the course shell that is created (title of corse restore in progress) shows it successfully makes the empty course shell but transfers no data at all into it. Nothing on the course page and nothing in the question bank. This makes me wonder if it is failing at some sort of check done prior to adding content. For the non-math courses that successfully restore, they move past this screen at the 25% mark and the screen changes in the way it looks with the progress bar actually starting over at 0%. It would seem an entirely different stage is occurring here as it transitions into adding content. Math courses just are not kicking over into it. 


Details about these math courses:

All of them have between 2000 and 4,000 questions in the question bank with each having a few subcategories throughout in how it is structured. Our next closest course in question bank size that restores fine has about 2500 questions also with nested subcategories. II can see question bank size as being a possible cause of the issue. The same size question banks all backed up and restored fine in Moodle 1.9 when we used them beofre. Today we have some experimentation on if we can split the full year courses into semester courses and drop the number, but without being able to backup and restore to a new course, you might see how that isn't an easy task to tackle.

The math courses are the only courses that have had Latex copy-pastes into the questions and answers. I wouldn't think that would make a difference since the questions do behave fine in quizzes, but in the off chance it might be important, I will mention it. 


Some experimenting that we have already done has led us to find that one source of backups failing was in the way the cloze questions can be coded throws off the html because it interprets the greater than symbol as html instead of a character.  We replaced those particular question types with the character code for greater than and less than and that cleared several of the courses to then start backing up correctly. We are using the same process of narrowing down the questions into small groups of question pools to see if any other issues of like nature could be causing problems. 

I am not in these math builds, so I am bringing in the lead builder into the trouble-shooting since she will be very familiar with any out of the norm methods that might be used across all of the math courses. I know last year she was building using Word as the question template and we discovered it was adding in hundreds of megabytes worth of formatting gobbledy-gook. That was just in one course and we did clear that. I know she is not using Word now, but it is possible she is using some other question templating outside of Moodle for new questions and importing questions in from that. We will be meeting after the holidays to follow up on this possibility.


Details about the Moodle and server environment:

Moodle 2.9 which was a fresh install just a bit over a year ago on a dedicated 4-core, 16 GB RAM server. Centos, LAMP


We have the max execution time up to a whopping 10,000 seconds to give some of the other courses enough time to backup and restore. I did one, not a math course, last night that took over an hour but it restored fine. It wasn't large in terms of mb (11.8), but the question bank is pretty big and takes up most of that. We use links to content housed outside the course to reduce the total mb since we get many copies of the same course over time. 


We had a 1.9 Moodle before, but we opted to rebuild courses over upgrading to it since we knew we would need to touch about every question and course page anyway to set images to responsive (we are using a Bootstrap-based theme) and to add the hint-levels to take advantage of the redo feature for quizzes. Questions from 1.9 went through a custom-coded process to convert them to what was needed for 2.9. Essentially, it added the hint levels with some templated feedback and made images responsive.  Courses with those have backed up and restored fine for all but the math courses. 


From reading this you might get a feel for what an intense build year this has been and will continue to be for at least another year. Yes, we builders are exhausted. lol. I am open to server setup mistake or performance issue suggestions if there is any chance that could be an issue. Maybe something with the Latex handling since that would be math specific, database (MySQL) settings, etc. The new school year is right around the bend and I am sure hoping we can find the source of trouble in time to get new course copies. We can use the fall-back of continuing to use last year's copy for maybe one more year's worth of student load, but not more than that.

Average of ratings: -
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

Wow! That was painful to read ... can't imagine how painful it must
have been to write, but very good description.

I admin more than one CentOS server with similar specs but maybe not
the number of users/courses.   Still though, (hate to tell ya this, but you asked ...) your server usage sounds like it's reached a stage where
some things might be better accomplished from the command line.

Before I dive into that, believe it or not, am gonna ask you to do something
in the GUI admin side of your site.

There is a hidden tool that might help some ...
http://yoursite/admin/tool/health/

Imagine it will take a long time to run on your system so be patient.  Might consider putting the site into maintentance mode OR run health check at a non-prime time - think I'd try the latter first.

If the tool discovers any health issues you will get dialog boxes with
statements about the problem and a button to "view solution".

Just so happens I have a sandbox Moodle 2.9.highest (as of today) running
on CentOS server and just ran the health tool.

One box says:
Only multianswer and random questions should be the parent of another question

You have questions that violate this in your databse. You will need to investigate to determine how this happened.

Now remember, your server may NOT/probably WON'T say exactly that but WILL have other.

Clicking the button for "View Solution" will present to you either
a box that says something to the affect ... dunno, try the quiz forum or
it might (hopefully) present to you an SQL query you can run (from the mysql> prompt - which means command line).

The sql queries may/may not fix all, but you are closer to better health
than you were before! ;)

The above may/may not fix issues with backups, however, I can tell you
it's been my experience on one heavily used Moodle 2.9 that command line
tools found in moodlecode/admin/cli/ have become necessary in backing up
courses with large number of users and large number of quizzes.

Hate put you into overload mode right away, so how about running the health
tool first, before diving into CLI tools that will help.

Just one more thought ... have decided it's not worth comparing 1.9 to 2.x and now 3.x.   Those days a gone.  Concentrate on what issues you see before you and resolving those problems.  Old behaviors of 1.9 will probably never return (may they rest in peace) ;)

Do have suggestions for using CLI which will follow this posting and after hearing back from you about health.

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

Part II ... command line tools.   One major advantage ... takes apache out of the loop ... just php and mysql involved in whatever they do.

First, some of the scripts in moodlecode/admin/cli/ take defaults from how the admin user has setup things like backup.   If in the UI admin interface, one has selected options for course backups such that those backups are essentially no user, no grades, no work by students, etc. etc. ... ie, a reset course.

Create a directory outside of Moodle code/data to hold the backups - something like /home/backups/ would do nicely.   Why outside of Moodle data?  So you can see IF it does complete ... the tail end of the backup routine takes what it has built in /moodledata/temp/backups/[sometiempdirectory]/backup.mbz and actually 'copies' it to destination ... which, by defaults is normally 'file area' ... which is in "the mess" of  filedir (new filing system) where there are no humanly recognizable names of files and where one cannot 'inspect' a backup file.

One can backup a single course ... will try that first.

So, know the course ID ([#] in example below) of one of those troubled courses and set the defaults to backups such that the backups would be essentially NO user backups (like reset courses), and then from the command line as root user:

cd /var/www/html/[moodle] or /poathtomoodlecode/

php backup.php --courseid=# --destination=/home/backups/ [ENTER KEY PRESSED HERE]

What you should see is similar to this:

== Performing backup... ==
instantiating backup controller 9d45061642731167bd5c34ea43e246f0
setting controller status to 100
loading controller plan
setting controller status to 300
applying plan defaults
setting controller status to 400
setting file inclusion to 1
checking plan security
setting controller status to 500
setting controller status to 700
saving controller to db
calculating controller checksum d02d16ae506741789a9acbfe49f455b8
loading controller from db
setting controller status to 800
setting controller status to 1000
saving controller to db
Writing /home/backups/backup-moodle2-course-17-moogoojoo-20160704-1333.mbz
Backup completed.


If that fails ... then the above did NOT reach 1000 but stopped at some other number/point, then  one will have to inspect where/how it failed and here again the only way to do that is via command line.

**IF it fails**

cd /var/www/moodledata/temp/backup/

Looks for a directory that is long and has a bunch of letters numbers in it.

Does that directory have a backup file in it? .... like: backup-something.mbz

If the .mbz file is there, then the backup failed at the very last step (that 99% point you see in the GUI).   That last step is *copying* the backup .mbz file to it's destination.  Known issue with any courses that's in the Gig range and OS using 'copy'.   Would be willing to bet it's a good backup.   One can manually move the .mbz file to /home/backups/ thusly.

In /var/www/moodledata/temp/backup/d02d16ae506741789a9acbfe49f455b8 (long directory name will be different this one is an example)

mv backup-something.mbz /home/backups/

Notice that's the 'move' command.

ls -l /home/backups/ and one should see the .mbz file in /home/backups/ and the .mbz file is no longer in the (remember example directory) d02d16ae506741789a9acbfe49f455b8

Ok, once backup is 'out there', how does one restore it?

A file system repo can be setup in any course that points to /home/backups/.   Then the .mbz file can be restored without having to download it just to turn around and upload it.

Now if the .mbz file doesn't exist, need to check for the existence of a moodle_backup.xml ... not there?   Then what is there?  Are there are .xml files with related directories .... files.xml is related to the contents of the files directory, as an example.

At this point couldn't describe exactly what, but think you might find, if one were to compare the number of quizzes in the course via the Moodle UI of the course and what one sees in the tempdirectory/quiz and quiz.xml file one will see items missing (quiz ID numbers).   Those would be suspect quizzes to check in the Moodle UI then.

That's about it ... given info ... more specifics would require access.

One last suggestion ... something you didn't mention you tried ... breakup the backup of those troubled courses.  The first run of the backup do NOT select ALL the quizzes/sections ... stop at half way point .... 52 sections ... stop at Section 26 and run the backup - name it with 'sec1-26' in the backup name.  If that completes, then run backup again only this time selecting Section 27 to the end - using 'sec27-end' in the backup file name.

Restoring of that course, would then have to be done twice and to the same course (this usually leads to having to re-order things and might not be desired way to go).

Am certain others have different suggestions/ideas. ;)

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Interesting backup and restore conundrum

by Tammy Moore -

Thanks so much! I mentioned the 1.9 data really more as extra info in this thread, but I certainly was whinny about a year ago when I was in the thick of the transition, so touché' I earned the comment. It was way too little time, too little sleep, new level of server tools with the dedicated server to figure out, and a big jump in terms of learning how the new version worked compared to ancient 1.9. I have felt ashamed of the few posts I made during that time. they definitely shoed the stress I felt. You, however, have always been the first to jump in to help. You are the greatest help here at Moodle.org. thanks.  I have had a year to get most of the workload done, I get more than the four hours of sleep I was getting a year ago, and I have grown to love Moodle 2.9. 

Now, apologies aside, lol, to the task ...

It is no doubt a bad sign, but the Health Check never finished.

I set up a Moodle 29 on my own computer over a year ago just to play around with theming a little while waiting for our real server to be provisioned and set up that we planned to use for Moodle 2.9. I realized that using it instead of the production server for doing some personal training runs at command line level work would be wise. Plus, I could practice in the day without worrying about server load with active students and be more alert (not a night owl).

I used one of the successful math course backups and did find out through that process that our max_allowed_packet was not adequate. Once I bumped that up to an insane number I was able to successfully restore the Algebra 2 course onto my personal Moodle 29.

I then ran a health check on my own 29 to see if the successfully restored course cause any health check errors to be thrown. None other than the fact I never did set up the cron for my own little MAMP server. 

Today, more practice at cli. Tonight, when the site settles down, I plan to try again for the real restore on the production server now that the max_allowed_packet is as high as was needed to do the restore successfully on my test Moodle. Then I will try again at a backup and see if the increase helps there too. I will report in again and update how it all went.

Average of ratings: -
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

Oh, no need to apologize for wishing to see some older behaviors of 1.9 ... I too!  Unfortunately, 'progress' generally means more resources.   The newer issues need to be addressed to continue to have a functioning Moodle server.   That's kinda what I was trying to get across. ;)

Sounds like some tool to check the health of and settings for the DB is in order.   Unfortunately, one of the better tools for this is command line.

Yes, max_allowed_packets setting is/has been known to be an item to tweak.   Back when restoring 1.9 backups to early versions of 2.x found another ... number of open files - open_files_limit=6000 (number was a guess).   Maybe needed when converting the 1.9 file system to the newer 2.x file system, but I've now made that a 'standard tweak' on any new installation of Moodle DB.

With 2 and 3 and beyond ... one also needs to check into innoDB tweaks.

A tool that will help: mysqltuner.pl - a perl script.   When run it does offer suggestions as to  what settings need tweaking (always upwards) to optimize DB performance.

If running a Moodle as an 'all-in-one' (DB and file storage on same server as moodle code), I pay close attention to the balancing act between apache and the DB server.

Example of output items from mysqltuner.pl

[OK] InnoDB data size / buffer pool: 184.6M/13.0G

[!!] Maximum possible memory usage: 13.5G (86% of installed RAM)

If you note the second example, seems I have tweaked the DB settings too much on the server from which these example items come.

Dropped connections ... MySQL has a default of 151 for Max number of connections and on busy sites I"ve had to increase that to handle the load.   One 'client' (an ISD) didn't let me know they were going to allow usage of student devices to make up for ISD's inability to provide a true 1-1 for all students.   That sort of thing, however, might be different with your Moodle ... truely asynchronous ... not a like a school whose students from one class (say 25 of them) would hit a moodle course at the same time and hammer away at it for the remainder of the class period.   Could always tell when the 'change classes bell' rang ... big drops ... 5-10 minutes later ... right back up there.

Ok, enough about that ... are you backing up courses to restore them to use as new courses (same prof) for next semester ... or something like that?

Are there courses where one could simply use reset after archiving a full course backup to get some classes ready for the next semester?

'spirit of sharing', Ken



Average of ratings: Useful (1)
In reply to Ken Task

Re: Interesting backup and restore conundrum

by Tammy Moore -

As to the question about resets vs restores - 

More than half of the students are in at your own pace option courses since we max out in capacity for our live online courses fast.  We can get some really big enrollments in these, so they take a while to 'cool off'. Biology this year was over 1000 students. Other science courses were not far from hitting 1000. I have made new copies annually and moved the current courses off to an archive folder to give students already working time to finish. Typically in a year to two years all students have retired out past the 365 days since last activity point and then I delete that copy from the server. Not bellyaching, just noting that courses with near 1000 enrollees in Moodle 2 did give us a learning curve this year. Teachers had page load time outs as enrollment numbers grew that we did finally track down to the Grade Me block add on. We are now using Marking Manager with great success in those big courses. Changes to the grade book, forget it. It just was not doable. There were other things as well. This coming year, we are considering shifting to multiple course copies released throughout the year and keep the enrollment size down in each one.  That opens up a whole new kettle of fish, but we want to give it a go. The roll out of multiple copies through the year may actually mean these cool off faster that the single large enrollments. Resets instead of new copies, maybe. 

Course copies that are for the live, online classes usually run anywhere from 50 to up to 400. Those have more solid end points because they are synchronous. Conceivably, these could eventually reach a point where a reset is all that is needed. Though, it may be a year or two to finish shaking out errors yet from rebuilding the courses, so we are now entering the phase of having a master course copy so we can make fixes as we find problems. So many things found in an active course cannot be fixed while students are in it whether is is that the course enrollment is too large to make a change to the grade book or it is something in a quiz and once a student has started it we are locked out of changes unless we dump all the student work in that activity up to that point. 


Acknowledged Training Need - 

What I need is a really good course, even better if it is Moodle specific. I can see a message "If the number is too high, ..." or increase the size of ..." in recommendations, but if I don't know what "too high is" or have some sense of scale for how much to increase something by I can way overshoot or not nearly set something high enough to see a performance gain. Plus, I don't know things well enough to realize repercussions to watch for since so much is inter-related. Plus, I want to learn the whys as much as the what. I want to get over fearing the server side stuff and get as comfortable with is as I am with Moodle's innards. 

There isn't anything locally available since I live very rural. I am going to have to figure out how to find the online training I need; though, I am not even quite sure what my search term I need. LAMP Stack training? Maybe full Stack, or maybe Database Training?


Changes I Have Made - 

Thanks for the heads up on max number of connections. I now have that set to 200 up from the default of 151. Here is a good example of following a recommendation to increase something, but not knowing what scale to increase by. I figure 200 is a safe number to try and yet see a performance improvement. That with the dramatic increase I made in max_allowed_packets I hope will give us enough tweak to find success tonight when I make the next attempt at a restore and give another try with one of the course backups that has been stalling.


Plans in the works:

I have researched the MysqlTuner and plan to install it and give it a try. I also remembered seeing a recommendation page in PHPMyAdmin and thought I would browse it. I saw a lot of recommendations related to tables not being properly indexed and things related to Innodb buffer pool. I will dig in and try to learn and understand those and see if I can get myself ready to tackle improving things there. I also want to dig in and get a handle on MyiSAM engine vs InnoDB. Another area is to look to moving to Amazon's elastic web if that can give us more resources at peak than staying with standard servers. It is that or go to the next step of renting another server to split off the database from the web server. We need more hamster wheels for peak use. We often pegged out our CPU/4 cores at peak hours of the day last school year. 

Average of ratings: -
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

Those details you've given are important and definitely related ... thanks.  Much discussion/questions but you seem to have a handle on how it's used and are already aware of some issues ... just not clear resolutions (yet).   That will come.

I wouldn't be surprised if you were contacted by someone at Moodle Partner! ;)

However, some more sharing related to training needs in administering a Moodle server and a prelim comment ... rare is the case where the Moodle Admin user is also the Operating System Server Admin and a DB person as well.   Most of the time there might be at least 3 persons involved ... Moodle Admin (inside Moodle UI), Server Admin (OS) and a DB Admin.   Wearing all those hats could get 'heavy' on ones head.

How did you find out you needed to tweak max packets?
Surely you got that from either Debugging in Moodle OR from server logs or a combination thereof.

Example: in debugging restoring a course one saw 'Server has gone away'.
Google search: 'server gone away MySQL'   Notice I left out Moodle.
If you did exactly that search the first hit should be something from
mysql.com.   Maybe Percona Blog (good resource for MySQL DB info).   That would be a technical page but that's what one needs.

One will notice that piwiki, drupal, maybe faqs from Amazon related to the
topic.   There are hosting providers who have rather extensive docs/faqs etc.
for their customers that are available for anyone to read.  Sometimes, one can find some nuggets in the others not specifically related to Moodle.   One just has to be aware of translations (not language, but OS specific) ... their system vs your system.

So not sure there is a Moodle related specific training site for those things that
make Moodle go in a typical LAMP stack - Linux, Apache, MySQL, PHP (but there's also now some Python related items in the latest Moodle 3.1 and search as well as Java Runtime and an office suite - if one wanted all the bells and whistles). Would be great, but like documentation, might be valid for a while, but then rather quickly falls into platform/version specifics and, sigh, out of date.

Maybe the best option is to learn how to fish.  Do think I see that phrased differently in those descriptions of ohhh, say ISTE's Standards for students and teachers ... 'problem solving' etc.

Here's a short list of my 'tools'.   Most command line because I want to use them on a live server and in realtime if at all possible (examples below might be specific to CentOS).   They are not in order of importance ... those will eventually filter themselves to the anyone's top 10.

Learning how to create bash shell scripts for repetitive investigations and task.
Example: an 'update' script to update the moodle code using git as well as running the php scripts found in /moodlecode/admin/cli/

Learning system information tools that come with OS - top, ps, df, du, tail, head etc. as well as tools that would be involved in backups: tar, rsync, mysqldump.

Learning where the logs are kept ... apache access/error, MySQL/MariaDB logs
(multitail, tail -f, fgrep (searching logs).

That's a very short list ... am certain that others could contribute.

Do know this ... the more one does something the greater the chance it will
eventually be internalized.   If at all possible be proactive ... not reactive.

'spirit of sharing', Ken


Average of ratings: Useful (3)
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

Since your Learning Plan involves DB ...

More on DB and tuner - just sharing ... from perspective of a Moodle Admin who doesn't hold any DB credentials.

Before beginning any tweaking .... *do a mysqldump of the DB*.

Recommendations from MySQLtuner should not be applied blindly and without research.
There are some changes/additions/tweaks to config that should be used in combination with
another tweak and should be increased gradually.  best reference is/are the docs from
the source of the software. Using MySQL 5.5? .Then use the MySQL site for that version.

https://dev.mysql.com/doc/refman/5.5/en/

Try to focus when you do read the technical ... try not to get distracted or off into a tangent which
you may not need at present.

Something can be tweaked to high which leads to diminishing returns.
After making tweaks, MySQL daemon (server/service) must be restarted.
So one command needed for this is related to restarting MySQL services safely/properly which is dependent upon OS.   IF, your tweaks have actually lead to MySQL failing to restart, no problem ... just edit the my.cnf file again and comment out your added parameters.   Start service.
Research some more.

Right after restarting strongly advise checking the MySQL error log to see if everything appears
to be OK.

Consider putting Moodle site in maintenance mode while you tweak and restart.

If you made tweaks that was designed to increase the memory usage of MySQL, use tools
that would show that: top with ctrlF + P on centos as an example.

Database server now running under your tweaks needs some time (actually being used) before you
to run tuner again to evaluate your tweaks.

There's a lot one could write about this.

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Interesting backup and restore conundrum

by Tammy Moore -

Many thanks to Ken Task! Even though I am still hitting the wall with the course restores I have learned so much and the site is better for it. I have experienced lot of growth in SSH skills for this troubleshooting effort going from total newbie to at least now being able to get to and run the cli scripts, update my.cnf using nano, what and how to run SQLTuner, and more. Best of all, my command line terror has abated somewhat. lol


For the sake of others that may hit the same issue, I wanted to post an update of what I tried and what I plan to try next.


Failed restore behavior summary: 

Most courses restore fine. Large question bank courses fail at the 25% progress point with the Chrome tab spinning wheel disappearing being a very reliable indicator of failure. This failure happens within two minutes of reaching the 25% restore progress bar point. All timeout settings I can find are well above 2 min. Every once in a while I will get the behavior where it pauses at 25% then zooms to about 65% and fails. When it does get that far, there are actually questions in the quiz bank, though they do not have their categories. Nothing but the course shell everywhere else.


One new oddball course has popped up that may or may not be related:

Chem course restored fine. That new copy was updated with some hint levels added. Then it was going to be kept as a master. I then made a copy of that one for a seating copy and it failed like the other courses described above. I tried again both with the master course and even with the course that worked fine to restore to the master in the first place. Now even it fails at 25%. I saved the copy of the master to my hard drive and tried it on a local MAMP Moodle install and it installs fine there. . I hope there is a clue in these details which will lead to the root cause of the restores failing.


The following have been tried ...

☑ Settings that have been updated

SQL: Each confirmed after a SQL restart with a show variables like '' query in SQL.

innodb_buffer_pool_size increased to sqltuner recommendation of over 9GB

key_buffer_size is increased to greater than 17 mb per SQLTuner recommendations

max_heap_table increased to 16 mb

tmp_table_size increased to 16 mb

max_allowed_packet is at SQL max of 1073741824

max_connections=300 well above default of 151

php: restarted

mssql.connect_timout=5000 (default was 5)

mssql.timeout=6000 (default was 60)

mysql.connect_timout=6000 (default was 60)

PHP max execution time=2160 (this was actually increased when the server was set up)


☑ Barracuda and Innodb is in place

There was a recommendation on the command line Moodle page to use Barracuda if the site is large , so I switch us over from a default of Antelope to Barracuda. I ran the cli script mysql_compressed_rows and it confirms we now are all Barracuda. 


I read in a post here at Moodle.org by Joshua Bragg that he hit the same wall and thought he had settings all high enough as well and still his restores failed. He used MOOSH for the first time and it did the job for him. That is my next thing to try. 

Average of ratings: Useful (1)
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

Welcome.   What is it 'they' say ... You Go Girl! ;) (not trying to be sexist so please, not the intent).

On your way to be becoming a CLIN (command line ninja)!   It's a 'trip' ... but well worth the journey if administering a Moodle.

Would be nice if when moodle records in the .log file the stages it's reached in backing up/restoring *IF* those numbers (100,200,300,400, etc.) had 'text' hints ... even if just ONE word - like quiz or assign etc.   That's enough of a humanly understandable clue to help the Admin level user inspect troubled courses.

'spirit of sharing', Ken

Average of ratings: -
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Tammy Moore -

Here is an update - 

One thing we suspected was involved in the issue was the large number of quizzes (150 to 180) and the large quiz bank (smallest math course had over 5000 questions with some math courses having over 7000).

So one of the avenues of testing has been to keep trimming down the size to see if we could then get a successful copy. Today we literally got our very first successful restore to a new course copy for any of our math courses. It was a quarter the number of weeks of the original and only 2100 questions including labels. 

So now we are getting a sense of the top end of the question bank size. We are disappointed that Moodle 2 (or maybe still some server setting) has capped our courses down so tight to having to work in course quarters, but a step forward in this issue we have battled for 6 weeks is a cause to celebrate. 

We also went ahead and moved the server to the next tier up with full host server support and I moved us from 16 GB RAM to 32. They did a server optimization and that immediately moved one of the courses that we couldn't back up into successfully backing up. Though that was a full course copy and it did not restore. All the backups did speed up. Whereas we did have a few that would take close to an hour to backup before, now the longest we wait is ten minutes (of the ones that do work). 

So in this chapter of what feels like a novel length adventure, ends with some progress toward the goal of getting past the math course gremlin. We still have a ways to go yet, but there is some faint glint of light from the end of the tunnel. 

Average of ratings: -
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Ken Task -
Picture of Particularly helpful Moodlers

First of all, congrats!  Know it's been painful and has required much of your valuable time away from that for which you use Moodle - teach!

I suspect that the quiz banks themselves might be in need of repair.

Now that server has a lot more resources, don't forget about trying to run the 'health check' tool by URL.    Suggestions provided for quiz fixes can be applied via ssh session at the mysql> prompt.

And, IF running a 2.9> version of Moodle, there a script in sitecode/admin/cli/ called fix_orphaned_question_categories.php

whose help says:

Fix orphaned question categories.

        This scripts detects question categories that have had their
        context deleted, thus severing them from their original purpose.

        This script will find the orphaned categories and delete the unused
        questions in each category found.  Used questions will not be
        deleted, instead they will be moved to a rescue question category.

Might be a good thing to run and might result in shorter backup times, more accurate backups which means on the flip more accurate restores as well.

'spirit of sharing', Ken


Average of ratings: -
In reply to Tammy Moore

Re: Interesting backup and restore conundrum

by Peter West -

I know last year she was building using Word as the question template and we discovered it was adding in hundreds of megabytes worth of formatting gobbledy-gook. 

Hi Tammy,

I'm not a fan of Word for generating HTML, but recent versions (since 2010?) support saving as Filtered HTML.

I just tried it on a short letter, and the filtered version was 12Kb, as against 78Kb for the "normal" HTML version.

Average of ratings: Useful (1)