Moodle Crashing Every Night

Re: Moodle Crashing Every Night

by Ken Task -
Number of replies: 6
Picture of Particularly helpful Moodlers

Sorry, cannot do a Vulcan Mind Meld (StarTrek/Spock) with a remote system.

Obviously, Moodle is trying to keep track of what failed and attempts again ... but, let's deal with preventing it from crashing right now knowing that we're trading off individual course backups (which I know is desired) to one of site backups.

First, turn automated backups to manual.

Install an add on called courses sizes.   It's not accurate, but good enough for our purposes.   We just need to see the largest courses and work with backing up those with the CLI script.

In the automated backups one has no control over which courses to exclude and when ... so let's say it's course ID 5 that's failing ... and causes issues with the remaining 50 - 6-50.

That's the course then (ID 5) to use with the CLI backup script.

Automated backups as well as the CLI backup script do take parameters you've set in the backups settings for the site.

http://yoursite/admin/settings.php?section=backupgeneralsettings

Consider reducing the number of days for logs ... or no logs at all in backups.

Consider other items such as: including users or not ... excluding them would also mean there are no assignments submitted by them in the backup.   That has to reduce the backup size ... thus time to create the backup, space needed to create the backup and archive etc.

Get a base course backup for troublesome course, then change those settings back to what is desire.  At least, now, you have some backup of the course ... even though it has no users no logs, no grades, etc.. - kinda like a re-set course.   That backup could be your fall back position (last resort).

'spirit of sharing', Ken

Average of ratings: Useful (2)
In reply to Ken Task

Re: Moodle Crashing Every Night

by Ken Task -
Picture of Particularly helpful Moodlers

Just thought of something else to tack onto the last reponse.

The admin level user can go to the course and manually backup the course.   This time selecting no students (no users) and de-selecting those activities which might be large.  If the course has a lot of quizzes/test, that's the other item in a course that could be large - not file size, but in processing.  Bad thing about Moodle and a quiz/test link, one doesn't know how many questions there are in it (that would make a nice 'infotip' popup for users if they hoovered over it with their mouse) until one clicks into it.

More questions means more processing ... period.   And, depending upon how the question bank has been setup in that course, you might be downloading the entire quiz bank!

'spirit of sharing', Ken



Average of ratings: Useful (1)
In reply to Ken Task

Re: Moodle Crashing Every Night

by Scott Tolan -

Thanks for the advice Ken, I've manually backed up the courses by reducing the size via deselecting users, quizzes etc. just so the base of the course is there. I've also spoke with the tutors of that course to see if there is anyway that they can reduce the content of the course, so maybe remove old content etc.

See how it goes.

Average of ratings: -
In reply to Scott Tolan

Re: Moodle Crashing Every Night

by Jeff White -

Depending on your error messages (have yet to see a post on your cron job logs or backuplogs showing what the error was), your course just could be so large that it is taking over an hour for the course to finish backing up which Moodle has a set time limit for backing up each course.


Try modifying moodle/backup/controller/backup_controller.php on line 310

        core_php_time_limit::raise(1 * 60 * 60); // 1 hour for 1 course initially granted

Change the 1 in bold to a 2 or 3 and see if that fixes things when it runs its next backup. 

I had a course that had +3,000 students with videos and +15 quizzes and moodle was doing over a million queries to the database to get the grades, quiz attemps, etc together for the backup of this 1 course. Adjusting the backup time limit resolved things for my moodle instance until we moved to a faster enviroment.

How do you have your cleanup settings configured for your site? Go to /admin/settings.php?section=cleanup. Adjusting the settings here can really help improve your site's performance and make the backups finish faster. Just keep in mind that if you site has been running for years and your server was set never to do a cleanup job, it will take a long time for the cleanup job (cronjob) to finish. 

Average of ratings: Useful (2)
In reply to Jeff White

Re: Moodle Crashing Every Night

by Scott Tolan -

I'll be honest, as an example of how novice I am, I neither knew what and how to retrieve the backuplogs or the cron job logs. However I have figured this out. The images below are the backuplogs (I think!) and I have also managed to pull off the cronjob logs, but what is it I would be looking for? There were a lot of errors :/ ctrl + F showed a 100 errors, some around quizzes and others around external RSS feeds.


Backuplogs:






Quiz error:

Error while processing attempt 3758 at 182 quiz:Coding error detected, it must be fixed by a programmer: There is no question_attempt number 885 in this attempt.#0 /var/www/moodle/question/engine/questionusage.php(239): question_usage_by_activity->check_slot('885')#1 /var/www/moodle/question/engine/questionusage.php(186): question_usage_by_activity->get_question_attempt('885')#2 /var/www/moodle/mod/quiz/attemptlib.php(556): question_usage_by_activity->get_question('885')#3 /var/www/moodle/mod/quiz/attemptlib.php(480): quiz_attempt->number_questions()#4 /var/www/moodle/mod/quiz/cronlib.php(78): quiz_attempt->__construct(Object(stdClass), Object(stdClass), Object(stdClass), Object(stdClass))#5 /var/www/moodle/mod/quiz/lib.php(480): mod_quiz_overdue_attempt_updater->update_overdue_attempts(1432824503, 1432824443)#6 /var/www/moodle/lib/cronlib.php(259): quiz_cron()#7 /var/www/moodle/admin/cron.php(88): cron_run()#8 {main}


I will change the snippet of code you have suggested though, because those four courses that are failing are huge so maybe they are timing out. And the clean-up settings seem pretty tidy in all honesty, we delete everything after 7 days, except grade history which remains for 365 days.


Cheers

Average of ratings: -
In reply to Jeff White

Re: Moodle Crashing Every Night

by Ken Task -
Picture of Particularly helpful Moodlers

@Jeff White  Useful and rated as such ...

Thanks for the info.   Was hoping your suggested fix would fix a 2.6 version of Moodle, however ....

In 2.6 the file backup_controller.php does not exist.
Only these:
backup_controller.class.php
base_controller.class.php
restore_controller.class.php

Lines do not exist:
fgrep "core_php_time_limit" *.php

Renders no finds

In a 2.7 you'll find that core_php_time_limit referenced in more than one file:

ktask$ fgrep "core_php_time_limit" *.php
backup_controller.class.php:        core_php_time_limit::raise(1 * 60 * 60); // 1 hour for 1 course initially granted
restore_controller.class.php:        core_php_time_limit::raise(1 * 60 * 60); // 1 hour for 1 course initially granted
restore_controller.class.php:        core_php_time_limit::raise(1 * 60 * 60); // 1 hour for 1 course initially granted
restore_controller.class.php:        core_php_time_limit::raise(1 * 60 * 60); // 1 hour for 1 course initially granted

So it looks like upgrading from 2.6 to at least a 2.7 is needed ... 2.8 does have the same lines - as does a 2.9.   And, think it wise to change the restore controller files to match those of backup IF restorting will take as long as the backup!

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Moodle Crashing Every Night

by Jeff White -

Hi Ken,

Looks like they changed the call variables. Look for something like this in backup/controller/backup_controller.class.php

        // Basic/initial prevention against time/memory limits

        set_time_limit(1 * 60 * 60); // 1 hour for 1 course initially granted

This is from Moodle 2.5 build so hopefully they have not changed it in 2.6 smile. Maybe do a search for "1 hour for 1 course" or "60" which is what I normally look for when modifying it between relelases. 

Average of ratings: -