Issue: consider raising the MaxRequestWorkers setting

Issue: consider raising the MaxRequestWorkers setting

by Dave Weninger -
Number of replies: 10

We had held an online Moodle quiz with 350+ users. Photos of answer scripts are required to uploaded to Moodle, Moodle was overloaded and most users were unable to login or encountered webpage frozen. We found the below error messages in apache error log besides numerous "invalid login token".

[mpm_worker:error] [pid 2207:tid 140268152399744] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

[mpm_worker:error] [pid 2207:tid 140268152399744] AH00286: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

I wonder whether it is correct to include the below setting in httpd.conf file so as to raise the value of MaxRequestWorkers for more concurrent users. As we hadn't change these setting before, it is believed that Moodle was using the default setting - MaxRequestWorkers = 400.

<IfModule mpm_worker_module>
ServerLimit		32
StartServers		10
MinSpareThreads		75
MaxSpareThreads		250
ThreadLimit		64
ThreadsPerChild		32
MaxRequestWorkers 	1024
MaxConnectionsPerChild 10000
</IfModule>

I found that our Linux version (CentOS 7.4, source compiled Apache 2.4.9) does not include the file 00-mpm.conf nor mod_mpm_worker.so .Also, I can't find the statement LoadModule mpm_worker_module modules/mod_mpm_worker.so in the httpd.conf config file. I wonder how can I change the above setting.

According to apache2buddy result, the average apache process is 287MB and the available free memory is 51GB.

I wonder if the actual MaxRequestWorkers value is only 51GB/287MB=178 instead of the value in setting = 400.

Please feel free to advise. Thank you very much.

Average of ratings: -
In reply to Dave Weninger

Re: Issue: consider raising the MaxRequestWorkers setting

by Ken Task -
Picture of Particularly helpful Moodlers

Qualifications for this response ... none ... just run CentOS 7 and doing some sharing/leg work for ya ...

CentOS 7 - yours may not look/have same

/etc/httpd/

./conf:
httpd.conf - main config file
last lines:
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
IncludeOptional sites-enabled/*.conf

./conf.modules.d:
00-base.conf
00-dav.conf
00-lua.conf
** 00-mpm.conf - if running
00-proxy.conf
00-ssl.conf
00-systemd.conf
01-cgi.conf
10-mod_security.conf
15-php.conf

modules are loading from:
./conf.modules.d/
What's in 00-base.conf
fgrep mpm *.conf

httpd -V | grep MPM

https://httpd.apache.org/docs/2.4/mod/prefork.html
https://httpd.apache.org/docs/2.4/mod/worker.html

apachebuddy does have a

### GENERAL FINDINGS & RECOMMENDATIONS ###

What does it say?

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Issue: consider raising the MaxRequestWorkers setting

by Dave Weninger -
Sorry, I really cannot find the ./conf.modules.d nor /etc/httpd/conf.d

Under /usr/local/apache2/conf/extra, I can only find the below files:
httpd-autoindex.conf
httpd-dav.conf
httpd-default.conf
httpd-info.conf
httpd-language.conf
httpd-manual.conf
httpd-mpm.conf
httpd-multilang-errordoc.conf
httpd-ssl.conf
httpd-userdir.conf
proxy-html.conf
httpd-vhosts.conf

I can only find httpd-mpm.conf under conf/extra which includes the MaxRequestWorkers directive.

I can find the below line in httpd.conf but it cannot call the content of httpd-mpm.conf even I uncomment it. The apache2buddy resulted that MaxRequestWorkers was still 400.
# Server-pool management (MPM specific)
#Include conf/extra/httpd-mpm.conf

--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: ???? (x.x.x.x):

Settings considered for this report:
Your server's physical RAM: 75350 MB
Remaining Memory after other services considered: 51014 MB
Apache's MaxRequestWorkers directive: 400 <--------- Current Setting
Apache MPM Model: worker
Largest Apache process (by memory): 298 MB
Apache appears to be running in worker mode.
Please check manually for backend processes such as PHP-FPM and pm.max_children.
Apache2buddy ONLY calculates maxclients for prefork model.
--------------------------------------------------------------------------------
A log file entry has been made in: /var/log/apache2buddy.log for future reference.

If I directly add the below in httpd.conf and run the apache2buddy, it shows that value MaxRequestWorkers is 1024. Does it mean that I have already added them in the correct place?
ServerLimit 32
StartServers 10
MinSpareThreads 75
MaxSpareThreads 250
ThreadLimit 64
ThreadsPerChild 32
MaxRequestWorkers 1024
MaxConnectionsPerChild 10000
It seems that apache2buddy can only calculate the MaxRequestWorkers for mpm_prefork but not mpm_worker.


Average of ratings: -
In reply to Dave Weninger

Re: Issue: consider raising the MaxRequestWorkers setting

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi, you wrote:

> the average apache process is 287MB

Unusual. Last time I measured in a busy site, the Apache processes took anywhere between 50 to 75 MB https://moodle.org/mod/forum/discuss.php?d=414977#p1674081. What differences your site has from the plain core Moodle?

> and the available free memory is 51GB
> > I wonder if the actual MaxRequestWorkers value is only 51GB/287MB=178 instead of the value in setting = 400.

Possible. You need a record of memory utilization at that happened to see whether Apache consumed all the memory.

What is the Moodle version, BTW? Theme? Additional plug-ins? And the PHP?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Issue: consider raising the MaxRequestWorkers setting

by Dave Weninger -
I just run the apache2buddy again and found that the DB has grown from 24GB to 37GB after 3 months from last online exam.

[ -- ] Your PHP Memory Limit (Per-Process) is 512M.
[ -- ] MySQL Detected => Using 37150.14 MB of memory.
[ -- ] Redis Detected => Using 29.87 MB of memory.
[ -- ] Memcache Detected => Using 39.40 MB of memory.

### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: moodlea.?????? (x.x.x.x):

Settings considered for this report:
Your server's physical RAM: 75350 MB
Remaining Memory after other services considered: 38130 MB
Apache's MaxRequestWorkers directive: 400 <--------- Current Setting
Apache MPM Model: worker
Largest Apache process (by memory): 321 MB
Apache appears to be running in worker mode.
Please check manually for backend processes such as PHP-FPM and pm.max_children.
Apache2buddy ONLY calculates maxclients for prefork model.

[ -- ] httpd is currently using 380.40 MB of memory.
[ -- ] The largest apache process is using 321.88 MB of memory
[ -- ] The smallest apache process is using 314.83 MB of memory
[ -- ] The average apache process is using 317.32 MB of memory

The actual MaxRequestWorkers = 38.1GB/317.32MB=120.  Does it mean that our Moodle can only support 120 concurrent users?

We have already split the webserver and database to two server.
Our Moodle version is 3.6.1, PHP 7.2.16, Theme: Essential 3.6.01 .It is strange that the apache process size is abnormally large. The size of DB is also too large. 

Our Moodle is integrated with Zoom and Panopto. Teachers make use of zoom for teaching video class frequently in the recent months. Some of them used to upload large MP4 files (size: 1~2GB) directly to Moodle. I wonder if this is the cause that makes the DB grows up quickly. Or, even makes the apache process size becomes large. If teacher add video to Moodle, which is stored in Panopto cloud, space of Moodle would not be occupied. The video displayed is just a redirected URL link. Unfortunately, some teachers don't know that. They like to download MP4 files from zoom cloud and then upload them to Moodle directly.


Average of ratings: -
In reply to Dave Weninger

Re: Issue: consider raising the MaxRequestWorkers setting

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
The next question is, which path to take, this Apache tuning approach or tackling the session lock https://moodle.org/mod/forum/discuss.php?d=419645 ? And/or whether there is a connection to this https://moodle.org/mod/forum/discuss.php?d=419704 ?

And you resolve this https://moodle.org/mod/forum/discuss.php?d=415355 ?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Issue: consider raising the MaxRequestWorkers setting

by Dave Weninger -
There is no connection to error message "Undefined" on https://moodle.org/mod/forum/discuss.php?d=419704 ?. As we had split the web server and database, the error message "504 Gateway Timeout" on https://moodle.org/mod/forum/discuss.php?d=415355 should be partially resolved. And, it is also related to number of concurrent users can be supported.

For the session lock issue, it think the cause is that students successively upload image files before uploading of the first one had completed. This leads to concurrent http request. Data writing to database seems to be not fast enough. It is not easy for us to switch to SSD. Therefore, the solution is that we do not require students to upload photos of answer scripts in the future.

However, It is the biggest problem that the number of supported concurrent users is only 120. I think it is good to raise the value of MaxRequestWorker to 1024 but it only raises the ceiling. The actual number of concurrent users is limited by the apache process size. The question comes. How can we reduce the apache process size? Or, can the problem be solved by adding more memory?
Average of ratings: -
In reply to Dave Weninger

Re: Issue: consider raising the MaxRequestWorkers setting

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Still, it is a good idea to "Once your problem is solved, reply to the thread, adding [Solved] to the subject line of your post and providing links to the documentation, discussion or tracker issue that helped you." https://moodle.org/mod/page/view.php?id=7080. In addition the doctors here need to know whether a case is new or a complication, specially from those patients who own thick dossiers.
wink

Yes, you need to find out why your Apache processes are huge. I have no idea, https://moodle.org/mod/forum/discuss.php?d=419825#p1691964.
Average of ratings: -
In reply to Dave Weninger

[Solved]: Issue: consider raising the MaxRequestWorkers setting

by Dave Weninger -

I found that I missed the below result of apache2buddy. 

"[ OK ] Current Apache Process Count is 5, including the parent PID."

There are actually 5 apache processes. The apache process size should be 317.32MB/5= 63.46MB. The previous value apache process size 317MB was wrong.

This matches with the result of the below command. https://www.erol.name/calculate-total-and-average-apache-process-size/

ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Process Size (MB): "x/((y-1)*1024)}'

The actual MaxRequestWorkers should be 38.13GB/63.46Mx0.8 = 479

Hence, the solution is to add more physical memory. Change the MPM directive to httpd.conf file so as to raise the ceiling MaxRequestWorkers settings. . In this way, the maximum number of supported concurrent users can be increased from the default value 400 to the targeted numbers.


Average of ratings: -
In reply to Dave Weninger

Re: [Solved]: Issue: consider raising the MaxRequestWorkers setting

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
63 MB for an Apache process. That fits to my 50-75 MB reported earlier https://moodle.org/mod/forum/discuss.php?d=414977#p1674081.

I don't understand your question though. Do you ask, since your MaxRequestWorkers 1024 was a high enough ceiling, according to the calculation 38.13GB/63.46Mx0.8 = 479 your server should be able to support the targeted number of 400. But why was it overloaded already with 350?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: [Solved]: Issue: consider raising the MaxRequestWorkers setting

by Dave Weninger -
We haven't changed the ceiling MaxRequestWorkers to 1024 yet. Our targeted number is about 800~900 concurrent users attempt the online quiz. We have to add the MPM directive carefully. Maybe need to wait for opportunity of testing at the next online exam.

Yes, our server should be able to support 400 concurrent users - the default MaxRequestWorkers value, but it was overloaded with 350. I think the reason is that each student submitted about 5 jpg files to the Moodle. Data was not fast enough to be written to hard disk, student might upload the second jpg file without waiting for the completion of the previous upload. This led to parallel http post request. That is why the error message "cannot obtain session lock for sid" appeared at the same time. This act behaved like multiplication effect of the number of concurrent users. 350x5 = 1750.

Perhaps, the http request was queuing up initially. When certain amount of "invalid login token" accumulated, it reached the limit of MaxRequestWorkers. Therefore, we saw the error message "consider raising the MaxRequestWorkers setting" accompanying by the error message of "cannot obtain session lock..".
Average of ratings: -