Moodle web servers locking up

Moodle web servers locking up

de Hittesh Ahuja -
Número de respuestas: 12
Hi All,


This is a shout out to any Moodle server admins out that could help us with their experience of configuring and working with the VLE. 

We face this serious issue at our institution with Moodle every now and then where both the servers ( load-balanced) gets used to their maximum capacity and then fall over. The only way to bring it back  to life is to restart it and things go normal after that. 

To be a bit technical with the issue we are facing, below is the response from our Corporate Apps and Database Manager : 


Same old problem of both web servers locking up in the W state, i.e. sending reply, with all workers being used. The servers get really wound up to a huge load, and just stay there

Doesn't matter whether we have one or two servers, all that happens is that you end up with 400 or 800 stuck workers


Anyone out there facing a similar issue where the Apache workers get stuck after a certain point which clogs up the server. 


Feel free to ask me any specifications about the server and I'll try my best to be accurate. 


Many thanks in advance,


Hittesh 

Promedio de valoraciones: -
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de Visvanath Ratnaweera -
Imagen de Particularly helpful Moodlers Imagen de Translators
Your starting point is the information on https://moodle.org/mod/forum/post.php?forum=94.
Promedio de valoraciones: -
En respuesta a Visvanath Ratnaweera

Re: Moodle web servers locking up

de Hittesh Ahuja -

Hi visvanath,

Not sure your link is taking me anywhere except to edit the same post ! 


Thanks,

Hittesh

Promedio de valoraciones: -
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de Visvanath Ratnaweera -
Imagen de Particularly helpful Moodlers Imagen de Translators
Hi Hittesh

I know, it is confusing. The information section you now see on https://moodle.org/mod/forum/post.php?forum=94 were supposed to be the header of this, Hardware and performance, forum https://moodle.org/mod/forum/view.php?id=596. (I am sure because I wrote it following the German "Moodle-Technikforum" https://moodle.org/mod/forum/view.php?id=949 ;huevo

Then our community manageress found it too much for the poor help seekers here, and pruned it (and other forums) to the present ten-word formula. Now I have the privilege of explaining the story (repeatedly) and still maintaining the original page by proxy. (See https://moodle.org/mod/forum/discuss.php?d=322223.)

Coming back to the relevance of that info, if you've seen "Please include as much background information as possible about your hardware, the operating system, the web server, the database server, the PHP scripting framework, etc." the helpers wouldn't need to pull that information one-by-one from you.
Promedio de valoraciones:Useful (1)
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de Michael Aherne -
Imagen de Core developers Imagen de Particularly helpful Moodlers Imagen de Peer reviewers Imagen de Plugin developers

Hi Hittesh, I remember our infrastructure guys encountered the same sort of problem and wrote a script to monitor the worker processes and restart Apache if they got beyond a certain number. It's maybe not the most elegant solution but as far as I know it has worked pretty well!

Promedio de valoraciones:Useful (1)
En respuesta a Michael Aherne

Re: Moodle web servers locking up

de Hittesh Ahuja -
Was that for a specific Moodle version or it happened irrespective ? We are currently on Moodle 2.8.7 (Build: 20150706) 
Promedio de valoraciones: -
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de Galin Vassilev -

Hello,

A lifetime ago, I have dealt with a similar issue where the apache processes would just pile up an lock up for no apparent reason. Eventually I figured out back than that I was using the default mpm-worker with mod_php on an SSL site which is not supported. Switching to mpm-prefork did the trick and fixed the issue. I obviously could not tell if that is the problem you are having but checking into that would be a good starting point.

Another thing to consider is, assuming you are using mod_php, would be to switch to php-fpm. mod_fastcgi is the glue between apache and php-fpm I would recommend, although using just ProxyPass would work just fine as well.

Hope this helps.

Promedio de valoraciones:Useful (1)
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de James McLean -

I assume you're running Linux? It sounds like the Apache processes are waiting on IO, or perhaps a file or even a database lock - it's hard to tell based on the information provided. We have seen similar when we had issues with shared storage.

Can your sysadmins do some further investigation with tools such as 'lsof' et al when the problem next occurs?

The load average seen is based on the number of threads, but they're all waiting on something and not actually very busy so it's something of a bum-steer.

Increase your Apache logging up to a higher level and see if any helpful errors are logged also.

Promedio de valoraciones: -
En respuesta a James McLean

Re: Moodle web servers locking up

de Hittesh Ahuja -

We're running it on a Solaris/SunOs . I'll shall make your points known to our sysadmins. I'll surely post any findings or fixes here if we manage to crack it  . 

Promedio de valoraciones: -
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de Paul Verrall -

"Solaris/SunOs" - Good luck with that sonrisa

If I was to take a punt at a quick fix I would look at KeepAliveTimeout in your Apache config and try dropping it to around 5 seconds (if it is not already in that region).

Promedio de valoraciones: -
En respuesta a Hittesh Ahuja

Re: Moodle web servers locking up

de Ken Task -
Imagen de Particularly helpful Moodlers

Some resources for resolving the issue *might* be found in the following links:

Apache Tuning
https://httpd.apache.org/docs/2.4/misc/perf-tuning.html
NOTE: there are some specific references to Solaris/SPARC

Script to individually terminate 'stuck' web server threads without restarting apache
http://giantdorks.org/alain/script-to-individually-terminate-stuck-web-server-threads-without-restarting-apache/

If anyone were to help, think sharing what you can from httpd.conf especially the parts related to 'workers' as there are 3(?) configs for workers.

There has already been a suggestion to check mpm-prefork and/or change config from mod to php-fpm. mod_fastcgi.

'spirit of sharing', Ken

Promedio de valoraciones:Useful (2)