504 Gateway timeout after editing SCORM settings

504 Gateway timeout after editing SCORM settings

by Ronald Vyhmeister -
Number of replies: 11

I've got a non-tribial SCORM package (60+MB). When I change the settings on the package (turn autocommit on or off) it takes forever and eventually I get the 504: Gateway Timeout error... and not just for me, but for everybody! After 5+ minutes it comes back like if nothing happened.

Any advice is welcome... Running 3.9 also tested in 3.10 with the same results...

Thanks!

Average of ratings: -
In reply to Ronald Vyhmeister

Re: 504 Gateway timeout after editing SCORM settings

by Ronald Vyhmeister -
Further testing shows the following:

  1. The timeout only occurs with editing a large class (20,000 + students). The same class without students has an instant response.
  2. It is only the editor's session that freezes. A different session (different user, different browser, or incognito window) is not affected by the timeout.

Any ideas welcome!


Average of ratings: -
In reply to Ronald Vyhmeister

Ri: Re: 504 Gateway timeout after editing SCORM settings

by Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
This behaviour happens when the session lock is not released. You should check your web server error log to see why. Usually, a lock is requested before doing a critical operation on your data (e.g. editing a resource/activity) and released when the job is done (correctly).
If the lock is not released, something has gone bad in the "middle", maybe your PHP script timeout is too short for editing such a big resource?
Average of ratings: Useful (1)
In reply to Sergio Rabellino

Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by Ronald Vyhmeister -
Hmmm... done some more digging... and it apparently only happens with SCORM courses with more than 6000 users... There is one error in the httpd server logs, no error at all in php-fpm.

The error happens if I just add/remove a period from the course description, or change any setting of the course, whether it's SCORM or AICC, and whether the package is hosted locally or in the cloud. It puzzles me that a quick update of the description should cause this problem and only for courses with large numbers of users....

See the log below (sanitized)... Any feedback welcome!

[Thu Feb 24 11:17:06.401900 2022] [proxy_fcgi:error] [pid 2321:tid 139646289831680] (70007)The timeout specified has expired: [client a.b.c.d.206:52584] AH01075: Error dispatching request to : (polling), referer: https://moodleserver/moodle/course/modedit.php?update=21802&return=0&sr=0
Average of ratings: -
In reply to Ronald Vyhmeister

Ri: Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
I'm short of ideas, but I would like to exclude the number of users enlisted in the course as the root cause itself. The log you posted says that php-fpm did not respond in time: could you try to set it to a larger value (more time to wait for a response) inserting a directive as follow (for an apache 2.4+ installation)
ProxySet timeout=600
inside the section.
Obviously it make no sense set an apache timeout larger than the PHP set_time_limit of your fpm pool, so check also this consistency between the two values. If it's only a matter of wait some time . this could solve, otherwise you have to search were the php script goes in an infinite loop.

HTH
Average of ratings: -
In reply to Sergio Rabellino

Re: Ri: Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by Ken Task -
Picture of Particularly helpful Moodlers

Another HTH ...

check logs for a reference to a 'checknet.txt' file request ...   SCORM's are an application shipped to client and run from client machine.  Those could be large and involved taking quite a lot of time to do whatever is in them.    Maybe over the session limit setting.  Like a ping back to Moodle saying, 'hey, am running SCORM and I'm still here!'.   If the checknet ping doesn't get to moodle, one might see the 504 gateway error.

Once got the advice from the person who, at the time, maintained SCORM for Moodle (or so I was told), to make the SCORM's smaller or break them apart into smaller 2 part things.   Probably would have worked, but the authors of the content rejected the suggestion.   Nearly all of their courses were 100% SCORM - site was a large site with load balancer etc. (just to give you an idea of size/usage).   Eventually, because this issue didn't appear to be resolveable, entity moved to Canvas.   Moodle lost a chunk of 'market share' as a result because the entity was followed by other smaller entities ... 'whatever the Joneses' did', kinda thing.

Or, like Videos, host the larger/longer SCORM's else where and use moodle as a portal to them.

https://rusticisoftware.com/products/rustici-dispatch/

which won't be free hosting, of course.   Never tried that myself, soooooo ?????

Anyhoo ...

HTH also.

'SoS', Ken



Average of ratings: -
In reply to Ken Task

Re: Ri: Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by Ronald Vyhmeister -
Thanks Ken for your feedback... yes, checknet.txt is gotten repeatedly, with no errors... the weird thing is that it's not an issue with running the big SCORM package (that's another issue that you've identified well), but in merely editing the course text description... and why should it be different to edit the text with 1 student or 10000? Or is it trying to regrade everything? In that case, yes, we'll have to put a bigger timeout... but users won't like 300 seconds smile
Average of ratings: -
In reply to Sergio Rabellino

Re: Ri: Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by Ronald Vyhmeister -
Hi Sergio,

I could set the timeout... the WEIRD thing is that when I take the course, export it (stripping users) and then import it (no users), it takes like 1 second after editing to come back to me... NEVER a problem... and yes, there is a longer time the more users are in the system... At this point I'm at 60 seconds, which is more than enough to modify the text of the course description...I guess I wonder why the effort of saving a different text description varies based on the number of enrolled students... seems like that is really off! Or is it trying to rescore everybody?
Average of ratings: -
In reply to Ronald Vyhmeister

Re: 504 Gateway timeout after editing SCORM settings

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
That's a known issue - been around for a while, the proper fix is MDL-46279 but there's a chunk of work in that and no one has cared enough about it to pay for my time to fix it or submit a pull request to fix it.

this helps (clearing out some of the data in the tracks table so it doesn't have to update it all.)
https://moodle.org/mod/forum/discuss.php?d=398448#p1607266
Average of ratings: -
In reply to Dan Marsden

Ri: Re: 504 Gateway timeout after editing SCORM settings

by Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Reading your hints, the root cause is the SCORM that, when you go to edit mode, starts to analyze/update all the records in the mdl_scorm_scoes_track table. If your table is a big one (and the number of users impact on this, obviously) the time exceed the set_time_limit.
So the only strategy available is to set-up a custom task that periodically trash away the older records...

thanks dan
Average of ratings: Useful (1)
In reply to Sergio Rabellino

Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by Ronald Vyhmeister -
Thank you Dan and Sergio... I see what the issue is, and will work accordingly... I wonder why SCORM requires all the tracking to be reviewed when anything is modified... I can understand that review might happen if the SCORM package is changed... When I update a quiz with 20,000 students, it does not even try to regrade unless I instruct it to...

For now I know what do to in order to avoid the error message... I do think there may be a simpler way to help the performance (waiting 3 minutes is no fun) than fixing the database organization (which I understand--I've taught databases for years). If the review of the tracking is necessary when certain fields are updated, then let's do it then, but not all the time.
Average of ratings: -
In reply to Ronald Vyhmeister

Re: Ri: Re: 504 Gateway timeout after editing SCORM settings

by David Pesce -
Picture of Plugin developers
I'm organizing a group of individuals interested (feeling the pain) in funding this development. Reach out to me if you are interested in contributing! The total cost is reasonable, and we are not far from having all the funds.
Average of ratings: -