Moodle web servers locking up

Moodle web servers locking up

by Hittesh Ahuja -
Number of replies: 12
Hi All,


This is a shout out to any Moodle server admins out that could help us with their experience of configuring and working with the VLE. 

We face this serious issue at our institution with Moodle every now and then where both the servers ( load-balanced) gets used to their maximum capacity and then fall over. The only way to bring it back  to life is to restart it and things go normal after that. 

To be a bit technical with the issue we are facing, below is the response from our Corporate Apps and Database Manager : 


Same old problem of both web servers locking up in the W state, i.e. sending reply, with all workers being used. The servers get really wound up to a huge load, and just stay there

Doesn't matter whether we have one or two servers, all that happens is that you end up with 400 or 800 stuck workers


Anyone out there facing a similar issue where the Apache workers get stuck after a certain point which clogs up the server. 


Feel free to ask me any specifications about the server and I'll try my best to be accurate. 


Many thanks in advance,


Hittesh 

Average of ratings: -
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Your starting point is the information on https://moodle.org/mod/forum/post.php?forum=94.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moodle web servers locking up

by Hittesh Ahuja -

Hi visvanath,

Not sure your link is taking me anywhere except to edit the same post ! 


Thanks,

Hittesh

Average of ratings: -
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Hittesh

I know, it is confusing. The information section you now see on https://moodle.org/mod/forum/post.php?forum=94 were supposed to be the header of this, Hardware and performance, forum https://moodle.org/mod/forum/view.php?id=596. (I am sure because I wrote it following the German "Moodle-Technikforum" https://moodle.org/mod/forum/view.php?id=949 ;egg

Then our community manageress found it too much for the poor help seekers here, and pruned it (and other forums) to the present ten-word formula. Now I have the privilege of explaining the story (repeatedly) and still maintaining the original page by proxy. (See https://moodle.org/mod/forum/discuss.php?d=322223.)

Coming back to the relevance of that info, if you've seen "Please include as much background information as possible about your hardware, the operating system, the web server, the database server, the PHP scripting framework, etc." the helpers wouldn't need to pull that information one-by-one from you.
Average of ratings: Useful (1)
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Michael Aherne -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Hittesh, I remember our infrastructure guys encountered the same sort of problem and wrote a script to monitor the worker processes and restart Apache if they got beyond a certain number. It's maybe not the most elegant solution but as far as I know it has worked pretty well!

Average of ratings: Useful (1)
In reply to Michael Aherne

Re: Moodle web servers locking up

by Hittesh Ahuja -
Was that for a specific Moodle version or it happened irrespective ? We are currently on Moodle 2.8.7 (Build: 20150706) 
Average of ratings: -
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Michael Aherne -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

I think it was put in place for 1.9, but it still kicks in from time to time and we're running 2.8.3 at the moment.

Average of ratings: Useful (1)
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Galin Vassilev -

Hello,

A lifetime ago, I have dealt with a similar issue where the apache processes would just pile up an lock up for no apparent reason. Eventually I figured out back than that I was using the default mpm-worker with mod_php on an SSL site which is not supported. Switching to mpm-prefork did the trick and fixed the issue. I obviously could not tell if that is the problem you are having but checking into that would be a good starting point.

Another thing to consider is, assuming you are using mod_php, would be to switch to php-fpm. mod_fastcgi is the glue between apache and php-fpm I would recommend, although using just ProxyPass would work just fine as well.

Hope this helps.

Average of ratings: Useful (1)
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by James McLean -

I assume you're running Linux? It sounds like the Apache processes are waiting on IO, or perhaps a file or even a database lock - it's hard to tell based on the information provided. We have seen similar when we had issues with shared storage.

Can your sysadmins do some further investigation with tools such as 'lsof' et al when the problem next occurs?

The load average seen is based on the number of threads, but they're all waiting on something and not actually very busy so it's something of a bum-steer.

Increase your Apache logging up to a higher level and see if any helpful errors are logged also.

Average of ratings: -
In reply to James McLean

Re: Moodle web servers locking up

by Hittesh Ahuja -

We're running it on a Solaris/SunOs . I'll shall make your points known to our sysadmins. I'll surely post any findings or fixes here if we manage to crack it  . 

Average of ratings: -
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Paul Verrall -

"Solaris/SunOs" - Good luck with that smile

If I was to take a punt at a quick fix I would look at KeepAliveTimeout in your Apache config and try dropping it to around 5 seconds (if it is not already in that region).

Average of ratings: -
In reply to Paul Verrall

Re: Moodle web servers locking up

by Hittesh Ahuja -

apache2handler section under phpinfo suggests we have KeepAliveTimeOut set to 5 sec already. 


TimeoutsConnection: 45 - Keep-Alive: 5

Average of ratings: -
In reply to Hittesh Ahuja

Re: Moodle web servers locking up

by Ken Task -
Picture of Particularly helpful Moodlers

Some resources for resolving the issue *might* be found in the following links:

Apache Tuning
https://httpd.apache.org/docs/2.4/misc/perf-tuning.html
NOTE: there are some specific references to Solaris/SPARC

Script to individually terminate 'stuck' web server threads without restarting apache
http://giantdorks.org/alain/script-to-individually-terminate-stuck-web-server-threads-without-restarting-apache/

If anyone were to help, think sharing what you can from httpd.conf especially the parts related to 'workers' as there are 3(?) configs for workers.

There has already been a suggestion to check mpm-prefork and/or change config from mod to php-fpm. mod_fastcgi.

'spirit of sharing', Ken

Average of ratings: Useful (2)