go unresponsive after 100 online users

go unresponsive after 100 online users

by Jason Boscorelli -
Number of replies: 15

Hello,

Our Moodle 3.9 is installed on ubuntu 20 with apache and mysql. (installed with Step by step ubuntu installation document). Our server has 32core xeon, 256gb ram and SSD HDDs but after 100-150 online users moodle site stop responding. I think there a software limit i missed because when i check the resources of the server while it was not responding, i can see that there are a lot of free resources (ram and CPU). Can you please help me in this case?


Thanks in advance.


Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Ken Task -
Picture of Particularly helpful Moodlers
Check apache and mysql logs ... error logs especially. When the event happens is there anything a browser displays error wise? What does it say?

How were you able to 'check resources' of server when server not responding?   Sounds like you were able to ssh into server but web service had issues.

ps aux |grep apache2

and apache error logs.

Some config areas to check ... DB config ... in error logs of web service ... see any 'MySQL has gone away' lines? Suggest installing MySQLTuner and run it with MySQL super user creds.

'SoS', Ken
Average of ratings: -
In reply to Ken Task

Re: go unresponsive after 100 online users

by Jason Boscorelli -

Thank you for your quick response.

I got no error on browser just no response but i can connect via ssh. Also its installed on vmware esxi so i can see the free resources. 

I will also check the error log messages next time i have the same issue. 

Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Ken Task -
Picture of Particularly helpful Moodlers
Check the error logs now .... not later when you have the same issue. Might be able to detect what the issue is and do something about it now.
'SoS', Ken
Average of ratings: -
In reply to Ken Task

Re: go unresponsive after 100 online users

by Jason Boscorelli -
I checked the error logs and i found a lot of errors like
[Wed Aug 26 06:48:51.891849 2020] [php7:notice] [pid 509521] [client XXX] [client XXX] http://.com/moodle Invalid Login Token: 22495566762 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36, referer: http://.com/moodle/login/index.php

and this error:
[Wed Aug 26 08:51:00.423398 2020] [mpm_prefork:error] [pid 1356] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Ken Task -
Picture of Particularly helpful Moodlers

Concentrate your efforts on this error
[mpm_prefork:error] [pid 1356] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

There is a script like MySQLTuner that helps check config of apache and makes recommendations.

https://support.rackspace.com/how-to/using-the-apache-2-buddy-script-for-performance-and-stability/

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: go unresponsive after 100 online users

by Jason Boscorelli -

Thanks a lot for your help. I will use these scripts and let you know about the result.


Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Jerry Lau -
check your mysql log.. it maybe a rouge plugin?
Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Alex Rowe -
Have you completed any configurations of MySQL, PHP, Apache etc?

The base/default configurations probably wouldn't be enough to support more than that.

With those specs, it should be "easy" to have around (or over) 1000 concurrent.
Average of ratings: -
In reply to Alex Rowe

Re: go unresponsive after 100 online users

by Jason Boscorelli -
I only enabled opcache.
Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Alex Rowe -
This is most likely your issue then.

The errors above around prefork workers means that you are getting more requests than Apache can handle with the current configurations.

Prefork is not the best mpm module to use, you should be using the event mpm module and then configure PHP to use php-fpm via proxy from Apache.

The apache event mpm might only need a small amount of configuration of its config file but best to test first then change only if you need to.

PHP fpm config would also need to be edited so that you are running the right amount of processes to handle incoming requests. Default is 5 I think, but the standard way is to work out how much RAM each process takes then divide RAM available to get the top value. You can use static or dynamic process handling, up to you. RAM available also depends on system RAM and RAM given to the DB and everything else.

Opcache might also need to be configured if it was only enabled, most likely giving more RAM but default might be ok for now. It can be tweaked later easily.

DB would need to be configed too, mysqltuner is a good start but make sure to research each recommendation as they may not work for your specific circumstances. That applies to everything though.

Also look into your MUC and sessions config. With a single server using SSDs there might not be much improvement, but you can use the tests available in the admin settings. Redis is the better option these days and I prefer two instances, one for MUC with an LRU policy and sessions without any purging. Redis is also only single threaded.

I would probably need to see the different config files to know more.
Average of ratings: Useful (1)
In reply to Alex Rowe

Re: go unresponsive after 100 online users

by Jason Boscorelli -
I really appreciate your help. I will make these changes and try. Thanks!
Average of ratings: -
In reply to Jason Boscorelli

Re: go unresponsive after 100 online users

by Ariden Lei -
Our Moodle is version 3.8, Apache 2.4.8, PHP 7.2 and MySQL 5.7, We use rotating disc HDD only.

I encountered the same problem with a large number of error message "Invalid Login Token" as below:

[Mon Jul 12 23:36:16 2021] [notice] [client M] moodle.yyyy.xxx [client M]  https://moodle.yyyy.xxx  Invalid Login Token:  180020  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36

I have already rise the MaxRequestWorkers in MPM directive to 800 but it is unfortunate that the error message of "Invalid Login Token" persists for less than 100 online users.

<IfModule mpm_worker_module>
ServerLimit                     25
StartServers                    3
MinSpareThreads           25
MaxSpareThreads          75
ThreadLimit                     64
ThreadsPerChild             32
MaxRequestWorkers      800
MaxConnectionsPerChild  10000
</IfModule>

The apache2buddy result is listed as below:
[ OK ] Current Apache Process Count is 3, including the parent PID.
Your server's physical RAM:                                             105588 MB
Remaining Memory after other services considered:  92033 MB
MySQL Detected => Using 13490.16 MB of memory
Largest Apache process (by memory):                            525 MB

[root@moodle ~]# ps -ylC httpd --sort:rss

S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S     0 14455     1  0  80   0 20432 137436 poll_s ?       00:00:24 httpd
S     2  1239 14455  0  80   0 277820 826450 pipe_w ?      00:00:53 httpd
S     2   339 14455  0  80   0 305512 832082 pipe_w ?      00:00:53 httpd

It is strange that the Apache process is as large as 525MB and the apache process count is so small. This makes the calculated MaxRequestWorkers value becomes very small. 

Please advise what causes the error message "Invalid Login Token". How can I minimize the apache process size and increase the Apache Process Count? Thank you very much.


Average of ratings: -
In reply to Ariden Lei

Re: go unresponsive after 100 online users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Does your Moodle go unresponsive after 100 online users and throws these errors like in the post https://moodle.org/mod/forum/discuss.php?d=409658#p1652482 (see the subject line)? If not, your symptoms do not match those of the original poster. Please start a new discussion, or even better, please request the moderator to split this discussion with a proper subject line - riding piggy-back is not appreciated.
Average of ratings: -