Hello again,
It has come to my attention that after running backups and restores of quizzes and courses that the HTTP 500 error message appears while trying to do these actions. I've examined the server and database error logs, turned on debugging, and searched the Moodle support forums yet I'm not a step closer to identifying the cause or this problem or solving this.
This is for running Moodle v.2.5.6 and an upgrade to a present version of Moodle is forthcoming.
Any help is appreciated.
Thank you.
Some more information: I've examined the .mbz files that are used for this attempted backup/restore and found that the .xml files within are non-problematic. So the problem, I would assume, rests more with the Moodle database but I'm still unsure.
Any help is appreciated.
Any help is appreciated.
Hello again,
We've recently upgraded to v.3.0.9 but our backup/restore problem has carried over across to this newer version. Furthermore, the HTTP 500 error message is still the only error message even after turning debugging on. I've checked the server logs and found no clues that could aid in resolving this.
Any help is appreciated.
Probably would help to disclose some more info ... php version, mysql version as well a Operating system. Am Linux person myself and use command line a lot, so wonder if you'd like to try this to see if there is something that displays ...
In moodlecode/admin/cli/ there is a php script that backs up individual courses with the defaults as set by Moodle UI in backups. Try runnning it for a front page backup which is course ID 1 and save the backup outside of Moodle file system.
cd /pathto/moodle/code/admin/cli/
php backup.php --courseid=1 --destination=/root/
Even without debugging turned on one might see something of the routine itself.
If successful, then it suggest configuration of web service might be the issue.
'spirit of sharing', Ken
Hi Ken,
Thank you for your response.
I've ran the backup.php script and it did provide a *.mbz file output that looks appropriate but we've received another HTTP 500 error when trying to restore this.
It's looking more like our web service indeed may be the cause of concern here.
We are using:
- PHP v.5.6.30
- IIS 7.0
- MS SQL Management Studio 2008
How should we proceed from here?
Thanks again.
Not an IIS person myself ... and think you are correct ... 500's are general catch all and there could be all sorts of reasons. ownerships/permissions on moodledata/temp/backup/?
That assumes that the backup.php script did create a valid backup file ... mbz.
One way to see about valid .mbz ... un-archive the backup .mbz into a test directory. One should see a 'moodle_backup.xml' file at the root of the 'test' directory. That file is like the crontoller for restores.
Turn on debugging to developer and try the restore again - might get something ... although 500's will even stop that.
Check IIS server logs ... not only web service but anything you can think of that involves creating files/directories in moodledata/temp/backup/ .... that includes the levels of access to also remove or change ownerships in that directory.
'spirit of sharing', Ken
Hi again,
I've de-archived the .mbz file and it does look like that this backup from the command line php backup.php worked properly. I do see the moodle_backup.xml file and it is populated with data.
I've then checked the folder permissions and IIS does indeed have permissions for all folders in question.
Next, I have the server log files open and I do see that there are some entries within that note HTTP 500 for the 'sc-status' header.
In the moodledata/temp/backup folder there are log files here, all from 8/15 (when we did the command line backup) and these are all 0 KB in filesize with a few folders also listed for the same date/time.
Un-archived backup sounds good.
Those .log files with 0 byte size normal - means a successfull backup. Any .log file with more than 0 bytes, you can open and inspect. What you'll find is references to a 'level' like 100,200,600,700,900 .... think 1000 is success.
If there are directories that have long filenames ... those were being used when building the backup file. They should have been un-linked and then removed upon successful backups. If there are some remaining, that could be the error 500 thing.
It is safe to manually erase those directories/folders. Do so before your next attempt at backups.
The 'sc-status' header is the clue:
Those should provide a number ... which you might find in:
Like I said, don't run IIS so .... time for you to dig a little more.
Is your PHP a 64 bit version? Need 64 bit .... even if the backup files don't appear to be large there could be 'large chunks of data' which 32bit PHP couldn't handle.
'spirit of sharing', Ken
Hi again,
In the error logs following the '500' that has been popping is 0, which refers to the sc-substatus column header. This suggests to me that the error would be an HTTP 500.0 error and the link provided states that this is a '500.0 - Module or ISAPI error occurred'.
The directories within the moodledata/temp/backup have been removed. I'll check this folder again after the next backup attempt.
As I recall this is PHP v.5.6.30 64-bit. Furthermore, I've been told that these courses backed up (or attempted to be backed up) are small in file-size.
Thanks again.
Unless a true Windows person pops in here to 'save the day', I could be confusing the issue because, as I've stated before, don' t run IIS. Had your initial posting in this thread stated the issue was on Windows, I probably wouldn't have responded - not having first hand experience with Windows servers and Moodle .... not since NT 4.0.
Please learn to use whatever search tool you like to dig more.
In a quickie search, did hit on tha fact that logs show more than just Module or ISAPI error if one digs into what that means and even suggest testing 'locally' ... ie, from the desktop of the server itself.
Here's one such hit:
That may/may not apply to your situation ... more food for thought?
Guess we'll have to wait for a Windows admin to jump in here.
How about it Windows folks?
'spirit of sharing', Ken
Hello again,
I've enabled the Handler Mappings in IIS to enable php-cgi and isapi.dll and then further gave Windows permissions to php-cgi.exe as specified in the link but; unfortunately, our results were identical.
Small update here. We're now seeing partial success now as our recent testing shows that one course was able to be backed up but another cannot. The unsuccessful attempt still results in the HTTP 500 error.
I'll post more details as they become available.
One further item to note is that the error log has been giving the sc-win32-status error code of 258. Looking around this code seems to indicate it is for a WAIT_TIMEOUT error along with the HTTP 500.0. Further searches have not indicated how to resolve this, however.
I've looked up for further resolutions to resolving this HTTP 500 and (likely) Wait Timeout error but nothing is effective. I've further adjusted IIS settings such as script time-out to a higher value, and set error messages to display within the browser, and I'm still not closer to a solution.
If there is anyone familiar with this problem and IIS/Windows your help is appreciated.
Thanks.
Another update, still not closer to a solution however.
We've managed to get a proper error message displayed and what appears is:
| Module | FastCgiModule |
|---|---|
| Notification | ExecuteRequestHandler |
| Handler | PHP |
| Error Code | 0x80070102 |
We've then updated the Activity Timeout and Request Timeout to 300 seconds but the page still gave this HTTP 500 error.
And in the IIS logs this HTTP 500 error is reflected in the error message:
2017-09-06 17:45:20 10.0.10.206 POST /backup/backup.php - 80 - 10.33.10.172 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/60.0.3112.113+Safari/537.36 http://mysite.com/backup/backup.php training.mysite.com 500 0 258 303359
Partial success now.
I've updated the FastCGI and applicationHost.config settings and managed to get proper outputs of course backups. These settings were for the Activity Timeout, Idle Timeout, and Request Timeout (I'm still a little unsure on which exactly is the most relevant here) and the Instance Max Requests.
This has enabled me to complete about a half-dozen backups but others would go on for a prolonged period of time and in moodledata\temp\backup I would see the working files/folders in production here but work to a certain point, stall, and then wait out the remaining time limits.
I've also been looking in the DB at the mdl_files table and see that is has about 1/3 or its remaining KB capacity. Should I clear this table or assign more memory space her to get past these stalled backups?
Think I'd Leave the mdl_files table alone. '1/3 remaining KB capacity'?????? ' Pardon my ignorance, but I don't re-call ever having to be concerned about such a thing. Now config of mysql for data size and number of instances, yes but that's the entire DB ... not just certain tables.
Are you absolutely sure that web service user can: create new files, copy files, delete files in moodledata/temp/backup/ as well as copy what could be *large* .mbz files from the build area to destination ... which, if left with defaults, would be moodledata/filedir/
IF backups/restores have been failing might be due to php settings ... time for a script to run, memory a script can consume. Are there any long directory names in moodledata/temp/backup/? Are there any larger than 0 byte .log files in same directory.
'spiirt of sharing', Ken
Hi Ken,
I appreciate your response.
1) In mdl_files I was examining the possibility whether the table had exceeded its memory limits or not and whether that is relevant to what I've been experiencing.
2) Yes, IIS webservice does have full control and permissions to read, write, modify, execute, etc. the moodledata folder, subfolders, and its contents.
3) Earlier today I've managed to backup a large course of roughly 140 MB whereas others occupy low MB or fewer in file size. Even smaller courses would timeout after the set period.
4) The moodledata/temp/backup folder does contain log files with 0 KB in size. The folders are larger than 0 KB and both the files and folders contain the long string of letters and numbers as their names.
5) I've been examining a series of PHP and FastCGI settings that could be relevant here yet successful backups are random.
Thank you once again.
Successful backups/restores leave only a .log file. Anything else and it's failed. If you went into one of those failed course backups one will see .xml file. Looking at those and knowing what a course has in it might help track this issue down.
To avoid confusion, remove anything in /moodledata/temp/backup/ Then attempt to backup again one of the courses that failed. That way you have only one attempt to sort through.
148M mbz isn't really a 'large course' .... have some servers that multiple G's .... like 20+.
What counts is heavy processing .... is there anything common to the courses that fail? Like a bunch of quizzes/test?
Let's remember that while you are backing up courses others are using the site. So it's possible your backup added with the usage by users at the same time could cause server slow down and a backup to 'time out'. Plus there are Moodle task/cron jobs. Depending upon site, task/cron jobs could spike from time to time.
How about web service error logs again ... same errors today 500's? Did you track those down?
BTW, if a web based backup fails, one can do the same course ID via commands line using the backup.php script in moodlecode/admin/cli/
Try that ... should take web service out of the loop. And, the CLI script might show/reveal a little more about errors, too.
'spirit of sharing', Ken
Hi again,
1) For the incomplete/failed backups I've looked into their folders within moodledata/temp/backup and the .xml files do have data populated within.
2) I have removed this folder of any existing folders or .log files and it does not contribute to the success of a backup.
3) I have inquired about any common quizzes or other course element that may be common across all these failed backup attempts. I'm presently awaiting a response from the users that regularly perform the backups.
4) Yes, I do see the HTTP 500s and HTTP 200s (for successes) in the IIS logs.
5) The command line backups do work but for the users that regularly do these backups it is impractical, and they would much rather prefer to handle it via Moodle. I have informed them that this is an option and can be done if needed while work continues to remedy this existing problem.
Thank you once again for your continued assistance.
In response to:
1 - size of the xml files gives you clues as to how heavy proceesing course backup might be ... large quiz.xml file ... lot's of quizzes .... large files.xml ... lots of files.
2 - did say manually removing would fix the problem ... just gives you a target to look at.
5 - indicates that it is truly an IIS issue then.
How long did the command line backup take? That takes php time outs out of the loop.
Do the same course you backed up via the command line via web interface.
IF it fails, inspect the moodledata/backup/temp/longnamedirectory's .log file.
That is text and can open to inspect with notepad. Backups build a plan then execute the plan recording the stage in the .log file. To what stage did the backup get? 100, 200, 300, ?
Have asked this before .... don't re-call if you reponded or not:
what is max time for a php script to run?
what is max memory any script can consume?
Both of those found in Site Admin -> Server -> PHP Info.
You might need to increase those values.
Also, the backup.php script in admin/cli/ has an option to give the path to where the backup is to be eventually copied from moodledata/temp/backup/ When you ran the command line backup on that course did you provide a path? If NOT then the cli script uses defaults as set in the backup of courses via the Moodle Admin UI.
Like I've said before ... I don't run Windows.
Guess there's are no Windows experts in these forums - hmmmmm .... what does that tell ya?
'spirit of sharing', Ken
Hi again,
- The temporary folders do include quizzes and they do take up the bulk of the file size in these temporary folders. To answer an earlier question, these quizzes are not commonly used across several courses and each quiz is unique.
- I've tried backing up using both methods for a course that repeatedly did not backup. The Moodle approach ran the full duration (ten minutes, 600 seconds) whereas the command line approach will state '== Performing backup. . . ==' and then wait indefinitely.
- Examining the server logs, the Moodle approach did run POST /backup/backup.php and did provide two HTTP 200 messages and one HTTP 500 message after timeout.
- When I run the command line script I do specify a destination - D:\Moodle\ but no completed backups are relocated here.
- Opening the .log files in notepad (or any other text editor) comes up blank.
In the phpinfo.php display, the following is listed:
- max_execution_time = 600
- memory_limit=128M
Thank you once again for your continued assistance.
On a failed backup, for whatever reason, there should still be a larger than 0 byte .log file. As the backup process progresses that .log file is written to .... at each stage of backup. Take it you did change the extension from .log to .txt before attempting to open with NotePad.
What are ownerships/permissions on moodledata ... on temp ... on backups directories?
When you run the command line version and give a destination does D:\Moodle\ ownerships/permisisons allow PHP to *copy* the built 'backup-whatever.mbz' file from moodledata/temp/backup/longfilename build directory?
Where the CLI might fail in Linux via command line is when running as apache user and apache user doesn't have access rights to write to destination (that's why I run those as the root user) OR the backup is too large for the copy command. Evidence of that is a .mbz file that remains in the build directory.
Take a screen shot of what the temp build directory looks like for one failed backup and attach in a response.
Really don't think this is a moodle code issue but a IIS config issue from what you've described. Sorry ... 500 errors are hard to track down.
If this issue were rampant with Windows installs running IIS there would be many many postings with 'me too', etc..
'spirit of sharing' .... still looking for some IIS expert to jump in here!
Ken
Hi again,
- I've not changed the files from .log to .txt. Regardless, it still provides a blank if I do change these file extensions.
- moodledata, temp, and backup each have IIS_IUSRS listed with full control of the directories.
- Specifying a destination in the command line approach does place the backup file in the specified location for courses that can be backed up. So I can specify D:\Moodle, or moodledata\temp\backup and it will appear wherever specified.
Would quizzes or some other course content be responsible for this stalling problem? Could there be dependencies on this data which does not enable it to undergo these backups?
Screenshot below: 
Thank you again.
Sigh ... again ... 0 byte sized log files indicate a *successful backup*. .log files larger than 0 bytes indicate a failed backup and show the stages of the backup process it was able to complete before failing. Viewing that larger then 0 byte file might give you a clue for what to look for in the course - * see below.
In an eariler posting you said when you ran the command line script and gave a destination the script did not complete ... no .mbz file in the destination directory. Now you say there are .mbz files in desitination?
Too bad Windows can't show realtime processes and activity in directories as they occur ... or fail. :\
Since there is no log file created when the process begins, me thinks that's the point at which if fails ... then times out.
Now to the last question ... yes, there could be a part of the course that's causing the failure, but there should be a contenthash named directory and a .log file begun IF the process can even begin in any attempt.
* IF you suspect it's something in specific courses causes, then the only way I know is to inspect each of the troubled courses and determine what's NOT core. Step through a manual backup where you select the items to be backed up ... excluding the addons ... include only core activities/mods, etc.
IF that completes then what would that show us?
I take it that if you do have addons they are all up to date and current for your version of Moodle.
'spirit of sharing', Ken
To clarify, I've tested the command line approach for courses that have proven to be successfully backed up already and those that presently cannot be backed up, by either method. If the backup performed successfully, the .mbz file would be placed wherever specified. If the backup was not performed successfully there would be no .mbz file placed wherever specified.
I will test another backup attempt and exclude any add-ons or miscellaneous materials and see if this has any difference.
Quick update:
It did work if I uncheck all other backup settings and leave activities and resources. So it looks like the error(s) was within one of these settings.
Another quick update:
If I uncheck the 'Include enrolled users' it will then disable five other fields - Include user role assignment, Include comments, Include badges, Include user completion details, and Include grade history. I've now made successful backups by setting this one field off.
Next, I'll try turning these settings off individually to see where specifically the problem is occurring.
Also, the plugins and add-ons for the site were updated back in August but this problem has existed prior to this.
Thanks.
You are getting closer!!! Progress is good ... to find additional plugins for the site:
Site Admin Menu -> Plugins -> Plugins overview -> on that form there should be a link for 'Additional plugins only'.
Might see if this applies:
https://tracker.moodle.org/browse/MDL-34388
Look for any related to quiz - and how many instances of that addon are in the site.
The version you are currently running 2.5.x is soooooo old now the closest thing I have to compare is a 3.0.x
Have re-read your original posting ... going to be upgrading soon ... need to as 2.5.x is very old.
Think I'd map out strategy for upgrading beginning now - right after or even before starting the upgrade 'march'. It will be a 'march' meaning you'll have to upgrade from 2.5.x to at least 2.7.highest and then I'd do some research on what the next version should be.
Especially take stock/notice whatever plugin is causing the issue and check if there is a compatible version for your destination version by searching Moodle plugins site.
Bsst of luck!
'spirit of sharing', Ken
Hi again.
Yes, we are getting closer to resolving this problem.
As mentioned earlier, we've upgraded to Moodle v.3.0.9 and updated the plugins afterwards, but since this problem was preexisting we were unclear whether upgrading Moodle would impact this backup/restore and HTTP 500 problem.
Also, I've tried unchecking the other five backup settings would still result in a backup failure. Only unchecking the 'Include enrolled users' will result in a successful backup now. This leads to my next question of what is included when I select this setting, and how can I distinguish what user/course information goes into a successful or unsuccessful backup.
Thanks again.
Well, guess I missed 'already upgraded to 3.0.9 ...' ... which is also still behind and no longer getting fixes and updates for anything ... but you know that.
I do re-call there being issues with certain role conflcts (superuser/course creator?) on restores, but not on backups - and not error 500's.
Windows systems use 'Administrator' as super user ... there is a config.php line that dictates 'admin' in Moodle being admin in moodle ... supposedly not to confuse WinDoze whose system admin might have changed 'Administrator' to 'admin'.
$CFG->admin = 'admin';
Dunno if that would be a factor or not ... again ... don't do Windoze.
Approach to backing up .... check to see who on the system has superuser/admin rights. Write them down. Go into troubled courses. See if those users are also assigned as course creators or Teachers in the course. Remove those users .... leaving only the students.
Try a backup.
What are you using for authentication? On Windoze would assume LDAP. Are there any users in troubled courses that are authenticating with something other than LDAP?
Moodle either does all users or none ... nothing in between - no user select list.
IF one can get a .mbz backup and restore fails ... might be due to roll conflict. I've had to un-gizip an old backup, edit users.xml removing the user ID'd as having a conflicting role (Moodle doesn't tell you which user ... just that there is a role conflict), then re-gzipped the backup making sure moodle_backup.xml is at the root of the gzip compressed file. Then, and only then, would the course restore.
Still however, that situation didn't throw a 500 error. ?????
what happens if you make a no user ... no students no work no assignments no files ... no teacher ... backup of a troubled course?
What addons do you have installed?
Am nearing ropes end here and some of what I might suggest just might send you down a rabbit hole ... illusive 'wabbit' (Elmer Fudd here).
Shame shame on Windows experts in these forums.
'spirit of sharing', Ken
'spirit of sharing', Ken
Hello again.
Sorry for the delay. I've gotten sidetracked.
1) v.3.0.9 is the present version in use and plans to modernize again will commence in the near future.
2) The $CFG->admin variable has never changed from 'admin' and I don't suspect this would be the cause of the problem as other errors would likely have appeared prior to this backup problem.
3) Yes, we are using LDAP on this site for login authentication. I will investigate whether this has any effect on enrolled users in courses during the backup process.
4) A troubled course will successfully backup if I remove the 'Include enrolled users' setting during the backup initial settings page. All other settings are inconsequential.
I'll post more information as it becomes available.
Thank you again.
Hello again,
I've gotten side-tracked again but I've been able to give Moodle some more attention today.
Since the 'Include Enrolled Users' checkbox is a factor here for the success or failure of a backup, which tables in particular are being included in this backup? This way I can run some queries to better identify where specifically this user data is faulting the backup process.
I have discussed LDAP as a possible factor in this situation and we're in the middle of devising some action to investigate this further. This could possibly lead to disabling LDAP-based user accounts and see if this has any effect in getting proper course backups.
Thanks.
Will say this ... you are relentless! (compliment) .... not sure that you'll find fault with what the tables for logs contain for users or there would be other issues present.
Having said that, the issue really might be too much data per user might be causing this issue.
Anyhoo,
SHOW COLUMNS FROM mytable FROM mydb;So if using MySQL plugin your DB replacing mydb above and take a gander at:
mysql> show tables like '%log%';
which will output more than one table but the two to view are: mdl_log and mdl_logstore_standard_log
You might want to get a heavy user ID so you can query the last table above to see just how many entries there are for just one heavy user.
OR you could add the code suggested to skip course ID 1. ;)
'spirit of sharing', Ken
Hi again,
I've looked at the log and storage tables and they are indeed large but not so large that I would expect them to interfere with the backups here. I've adjusted the php.ini file to provide additional variable handling but this doesn't seem to do the trick. Any other ways we can increase the volume of user data into these backups?
Thanks again.
Hi again,
There are many logs to view here (hundreds of thousands) and I've looked at increasing my PHP variable limit but to no avail.
Been thinking some more on this (dangereous) ... what you might have to do is upgrade the 2.5.x to highest 2.5.x and then to 2.6.highest as that's where the new compression (.gzips rather than zips) for backups were first introduced. Prior to 2.6 and that change/use of new compression many many sites were having problems with backups ... Windows servers primarily due to 32 bit version of PHP but even Linux servers (CentOS as an example) had issues with the tail end of backup process where the code used *copy* rather than *move* of the completed .mbz file to destination *IF* the backup file was 4 Gig or larger (operating system limitation on copy command).
To be on the safe side, clone your site and do the upgrades to 2.6.x highest on the clone. First thing to check, of course, is one of those troubled courses and backups.
May not have been your doing, but a take away here might be don't allow Moodle code to get that far behind in the future. Not saying one has to get the latest/greatest ... it's good to stay *near* the 'leading edge' but not so far that one is on the 'bleeding edge'. It's open source software thus, even if marked/tagged as 'stable', anyone using .0 or latest and greatest of any version of Moodle is really an 'omicron tester' (that's my humble opinion, of course).
2 cent advice ... spirit of sharing', Ken