Moodle performance - examples of configurations

Moodle performance - examples of configurations

од Jukram Aspalanta -
Број на одговори: 21

Sorry, again new performance topic, but I am feeling I need some good advice and configuration examples.
 
Main goal: support 1000+ "concurrent users" to take quiz at the same time (ideal to support ~2500 users).
We have a VMWare and we have enough resource to give our VMs. Disks are SSD. So this shouldn't be a problem. I think the main problem is how to configurate it properly. In addition, it's not very easy to emulate the users activity and test how the server can handle it in real situation. I have run some tests with Apache JMeter, but I haven't done well yet.
 
Idea:
Two Virtual Machines, one for web/moodle and another one for DB. (Moodle cluster maybe to overkill.)
OS: Centos 7
Moodle: 3.7
PHP: 7.3.16
Web: Apache 2.4.41
DB: Postgres 11.5
 
For high-traffic web it is recommended to use Apache (MPM event) + PHP-FPM and PostgreSQL (https://docs.moodle.org/37/en/Performance_recommendations).
Can someone give some examples (and hardware) how they have configured their settings for:
Apache MPM event:
StartServers =
ServerLimit =
MaxRequestWorkers =
ThreadsPerChild = 
ThreadLimit = 
MaxConnectionsPerChild =
MinSpareThreads = 
MaxSpareThreads =
 
PHP-FPM:
pm =
pm.max_children =
pm.start.servers =
pm.min_spare_servers = 
pm.max_spare_servers =
pm.process_idle_timeout =
pm.max_requests =
 
For PostgreSQL I have followed this configurator: https://pgconfigurator.cybertec-postgresql.com/
 
Last time when we had a quiz test (questions and some are with images), performance issues occurred - very slow, especially if there were ~600 users.
Output, when users were taking a quiz:
[root@moodle ~]# ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Process Size (MB): "x/((y-1)*1024)}'
Apache Memory Usage (MB): 4897.9
Average Process Size (MB): 349.85

Any good advice/ideas or examples how you have configured your Moodle server settings/configuration files are welcome, please share смешко

Просек на рејтинзи:Useful (2)
Во одговорот до Jukram Aspalanta

Re: Moodle performance - examples of configurations

од Howard Miller -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
My suspicion is that you are in the realms of some of the Moodle Partners who specialise in this sort of thing. There will be *very* few Moodle users who have sites configured to run 1000+ *concurrent* quiz sessions. I've been doing this a long time and it would frighten me.

I'll say what I always say... it's your quiz and your users. Start with some manageable pilots and do LOTS of measuring.

BTW... it may not be what you meant but VMs buy you convenience, not capacity. For the most part it's just another layer of software pulling processor cycles. There are still times when it makes sense to run on 'bare metal' to extract the last ounce of performance.
Во одговорот до Jukram Aspalanta

Re: Moodle performance - examples of configurations

од Thorsten Bartel -
Слика од Core developers
Be sure to utilize the following performance enhancing settings:
- Enable the PHP zend opcache
- Set up Redis or Memcached as Session handler and for the MUC (Moodle Unified Cache) *
- Look into disabling swapping

* If you're using Memcached, go for session handler instead of MUC, in my experience the performance gain is much greater and having the MUC reside on a SSD should be fine. I'm adding this because it's not trivial to implement more than one memcached on the same machine and you do [b]not[/b] want MUC and sessions residing in the same instance (MUC being purged results in all sessions being lost).

These are some things I can think of, but we have not yet tried all of them myself. So please report back if these were helpful (and if, how so).

As for the configuration of the MPM Prefork module, these are our current settings:

StartServers 5
MinSpareServers 0
MaxSpareServers 0
MaxRequestWorkers 900
ServerLimit 900
MaxConnectionsPerChild 0

StartServers is pretty much irrelevant, since you won't restart your system that often. We've found that performance runs best with Min- & MaxSpareServers set to 0 so load distribution will be completely dynamic. We didn't have much time to really test these settings against others, though. MaxRequestWorkers and ServerLimit basically depend on how much your hardware can handle. Take a good look at how much your average Apache process takes up in RAM (minus shared memory) and check how many of these your system can accomodate. Don't forget to consider resources for the OS and other services (especially the opcache and Redis/memcached).
Во одговорот до Thorsten Bartel

Re: Moodle performance - examples of configurations

од Jordan Picard -

It's always worthwhile to keep in mind that MaxRequestWorkers is the actual amount of workers that can use up to the memory_limit you have set in your php.ini file. The default is 128M (minimum 96M, 128M recommended, increase if you have issues or large moodle instances/backups). If you have a server with 16GB of RAM, you should always set aside some for overhead of the system (usually 2GB is common) and then do your calculation.

(16GB-2GB)/128M = 112

This number, 112, is the amount of workers that your system can handle using the max amount of memory on your system before it starts to swap (page) to disk and your performance will tank. That doesn't mean you can't set it higher but depending on your scenario it is usually better to keep in memory for your applications so the performance is consistent. Some people use the average apache process memory usage to calculate the amount of request workers they should have but if you go this route you will need to be careful about with performance issues when you run at capacity. Only you truly know your workload so adjust to what you see is best for your environment.

Во одговорот до Jordan Picard

Re: Moodle performance - examples of configurations

од Tasos Koutoumanos -

This depends a lot on the configuration of Apache and PHP. It doesn't apply in the usual case of php-fpm. The Apache process may be handling all kind of files, most of them static, and offload the processing to php processes. It's typical in such a setup that the apache processes are around 5MB, with PHP process being around 50MG.

Во одговорот до Thorsten Bartel

Re: Moodle performance - examples of configurations

од Uğur Ünsal Yavuz -

Hi Thorsten,

Are you still working with these settings? Any performance or leak experience with the MinSpareServers 0, MaxSpareServers 0 values? 

Во одговорот до Uğur Ünsal Yavuz

Re: Moodle performance - examples of configurations

од Thorsten Bartel -
Слика од Core developers
Hey Uğur,

please excuse the late reply. I haven't had access to my mail-account for the past week, otherwise I would have noted your question earlier.
We have since moved away from MPM-Prefork in favor of MPM-Event and have seen a significant performance boost.
Some additional configuration is needed, but the essentials are these:

In file /etc/apache2/mods-enabled/mpm_event.conf :
StartServers 4
ServerLimit 32
MinSpareThreads 64
MaxSpareThreads 512
ThreadLimit 128
ThreadsPerChild 72
MaxRequestWorkers 2304
MaxConnectionsPerChild 16384

In file /etc/php/7.2/fpm/pool.d/www.conf :
pm = static
pm.max_children = 1024
pm.max_requests = 1500

It is also possible to run the PHP-FPM in dynamic mode, but we experienced faster response times without the overhead of dynamic process creation / termination.

As for memory leaking (as you specifically asked): The settings MaxConnectionsPerChild and pm.max_requests define the amount of connections an apache-process or requests a php-process may handle during their lifetime before being restarted gracefully, freeing up allocated memory.
Во одговорот до Jukram Aspalanta

Re: Moodle performance - examples of configurations

од Visvanath Ratnaweera -
Слика од Particularly helpful Moodlers Слика од Translators
Hi

You have identified the difficulty of the problem:
> it's not very easy to emulate the users activity and test how the server can handle it in real situation.

But you expect the solution in a wrong place, I'm afraid:
> I think the main problem is how to configurate it properly.

Look at it this way. You said the server failed at 600 concurrent quiz users. Let's take 500 as a safe number. 500 concurrent quiz users are a huge load. You server is definitely not malconfigured. Then I would be hugely surprised if you double that number through fine tuning. Best case 10-20%. If it is more, the server was malconfigured.

> We have a VMWare and we have enough resource to give our VMs.

How much were they (during the 600 user session)? That was a single VM, right?

> Idea:
> Two Virtual Machines, one for web/moodle and another one for DB.

OK, that is a big change (not a mere configuration change). You can expect a big improvement.

> OS: Centos 7
> ..
> Web: Apache 2.4.41
> ..
> For high-traffic web it is recommended to use Apache (MPM event) + PHP-FPM

There is a raging debate whether Nginx with FastCGI perform better. That would be a big change which you need to experiment separately. Always the same question: How do you measure the improvement.

> I have run some tests with Apache JMeter, but I haven't done well yet.

I too think only JMeter tests on Moodle itself give comparable results. Recent Moodle releases come bundled with a whole package of tools for JMeter. They are not well documented though. I made a start here: Is moodle-jmeter-script-generator still maintained? then stalled. You are welcome to extend the document with your findings.

> Can someone give some examples (and hardware) how they have configured their settings for:
> Apache MPM event:
> ..

Since you already have decent server, please submit its values as an example. There are many visitors here looking for 500 concurrent quizz situations.

> Output, when users were taking a quiz:
> [root@moodle ~]# ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Process Size (MB): "x/((y-1)*1024)}'
> Apache Memory Usage (MB): 4897.9
> Average Process Size (MB): 349.85

I think the process size is on the high side. Anyway, there is no single number to measure how a server is performing. Have a look as Munin output for example to see how many parameters are involved.

All that said, apart from RAM and fast database response, efficient cache systems improve Moodle performance best.
Во одговорот до Visvanath Ratnaweera

Re: Moodle performance - examples of configurations

од Howard Miller -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
Load testing is very, very tricky on a complex system like Moodle. It tends to work best to fix a problem you know you have rather than identifying a problem you're not sure you have.

It requires a very narrow view of the system to test and a very clear idea of how to reproduce the "problem". It is then useful to see if any changes are producing a positive result.
Во одговорот до Jukram Aspalanta

Re: Moodle performance - examples of configurations

од Leonardo Martinez -
Hi!, we are also after a good configuration to run quizzes for a big number of users (3000+) Up to now we are going well with apprx 1200 concurrent users taking a quiz at the same time, but more than that, moodle "hangs"...

In our infrastructure, we have 3 dedicated VMs for moodle over vmware, with 3 HP hosts and an external Storage SAS with 10K 3.2gbps SFF discs.
We are not overcommitting memory or cores in vmware

1.- A reverse proxy over nginx and Varnish as cache server. Also doing the SSL offload. Lot of work happens here, but the server seams very very quiet. 16cores 24 gb ram.
2.- The Moodle Server, running Apache 2.4.6 MPM Event based and PHP-FPM 32 cores 32 gb ram
The moodle is not overloaded with plugins.. near a plain install.
3.- And finally, the DB Server, a Percona MySql. 32 cores 32 gb ram

Any advice where to invest next to acomplish the task and take big quizzes will be appreciated.
If it was possible to reach 1200, It must to be possible to reach 3000 too! (well maybe... ) смешко

Here my main configs. Any other info, let me know.

In a normal day:
Apache Memory Usage (MB): 105.691
Average Process Size (MB): 26.4229

php-fpm Memory Usage (MB): 6157.7
Average Process Size (MB): 47.7341

curl -L 127.0.0.1/status
pool: www
process manager: dynamic
start time: 23/May/2020:01:11:40 -0300
start since: 501442
accepted conn: 5016900
listen queue: 0
max listen queue: 0
listen queue len: 0
idle processes: 123
active processes: 5
total processes: 128
max active processes: 193
max children reached: 0
slow requests: 0


apachectl -V
Server version: Apache/2.4.6 (CentOS)
Server built: Aug 8 2019 11:41:18
Server loaded: APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)


mpm.conf

ServerLimit 1200
StartServers 32
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 1200
MaxConnectionsPerChild 1000


php-fpm pool
[www]
listen = /var/run/php-fpm.sock
pm = dynamic
pm.max_children = 260
pm.start_servers = 128
pm.min_spare_servers = 64
pm.max_spare_servers = 128
pm.max_requests = 1000

my.cnf

innodb_log_file_size=2G
innodb_flush_log_at_trx_commit=1
innodb_flush_method=O_DIRECT
max_connections = 950
innodb_buffer_pool_size=15589934592
innodb_buffer_pool_chunk_size=2073741824
innodb_buffer_pool_instances=20
thread_handling=pool-of-threads
Во одговорот до Leonardo Martinez

Re: Moodle performance - examples of configurations

од Uğur Ünsal Yavuz -

Hi Leonardo,
Let me share out experience alongside with yours

We get a huge improvement when we disable the timeline and calendar modules on the dashboard
With Loadium tests, we reach 3000 concurrent users on the infrastructure below. Working very well on production.

HA Proxy LB - 3xCluster>>16 core, 24g ram>3Par Storage (10,000 rpm disks)

Apache MPM Prefork mode 

StartServers 50 
MinSpareServers 50
MaxSpareServers 100
ServerLimit 2800
MaxRequestWorkers 2800   (recommended by apache-buddy / each server)
MaxConnectionsPerChild 0

+some MariaDB tuning settings (i think the most important tricks are hidden here)


Во одговорот до Uğur Ünsal Yavuz

Re: Moodle performance - examples of configurations

од Leonardo Martinez -
Thanks Uğur,
Nice storage you have, we cannot afford that up to now......so we are with a middle-end HP one, it's not bad, but not as good as yours.

I'm also after a good Percona/MariaDB DBA who can optimize the database and parameters... but everybody is asking me to switch to PostgreSQL, and now, it's a bit late...

I'll be fine touching parameters trying your values and comparing.. thanks...
Disabling the timeline and calendar it's a good idea too, I'll be testing it.

Thanks a lot !

Regards,
Во одговорот до Leonardo Martinez

Re: Moodle performance - examples of configurations

од Uğur Ünsal Yavuz -

I recommend you to use Firefox Developer Edition to inspect the loading performance of the timeline and calendar modules when a  page loads. Especially the /my dashboard is strangely called on almost every different subpage click and refresh. This cause most of the performance issues. We saw 4 seconds of SQL calls only for the timeline per page refresh. When you do your tests, you will see this clearly.  JMeter based Loadium on cloud is a great free tool for your tests. I do not think it is very necessary to switch PostgreSQL at this stage. 



Во одговорот до Uğur Ünsal Yavuz

Re: Moodle performance - examples of configurations

од manoj joseph -
I am also facing the issue above 1000+ concurrent quizzes. Anybody knows how to tune my server to handle 3000 concurrent users.As I am using ec2 instances i can manage configurations.Only thing the setup l.I configured apace PFM . But user goes above 1000 site is very slow.
Cron running on my server. I kept default in task processing.
I am using 16core CPU with 64GB ram EC2 instance.
Now I separated the DB with other instance
I thought the issue was related to db and queries caches .
Any body suggest to set such things in moodle
Во одговорот до manoj joseph

Re: Moodle performance - examples of configurations

од Visvanath Ratnaweera -
Слика од Particularly helpful Moodlers Слика од Translators
@Manoj, Isn't this the separate thread: Conducting quiz for 2000 users?
Во одговорот до Visvanath Ratnaweera

Re: Moodle performance - examples of configurations

од manoj joseph -

yes.I just followed the forum. My actual aim is to conduct the quiz.No body is giving the exact explanations.I did quiz .It was getting slow after reaching  1000 users.

Во одговорот до manoj joseph

Re: Moodle performance - examples of configurations

од Visvanath Ratnaweera -
Слика од Particularly helpful Moodlers Слика од Translators
Hi Manoj

I understand you wanting to change the doctor if you didn't see results. It is not that simple in a public forum like the moodle.org. All the "medical records" are still under the old consultation so the new doctors have to ask for them again. In fact, it is worse than that: The new doctors are for the old doctors for the most part, they've simply forgotten the old records! In the case of two clinics, that doesn't matter as long as you can pay. But here, it is free. So revisiting and roll the case from the beginning you are wasting their energy and robbing others. On top of all that, what you've just done is knows as thread hijacking and not appreciated in forums.

Long story, the short answer is, if you are still waiting for an answer, just reply to your own post (in the original thread) with your additional findings in the mean time.
Во одговорот до Leonardo Martinez

Re: Moodle performance - examples of configurations

од Leonardo Martinez -
We found a query affecting seriously the logon times, involving the notification table. Seems a bug with our moodle version.

https://tracker.moodle.org/browse/MDL-64400

Our mdl_notifications was with more than 25.000.000 records, 60gb and the auto purge was not working.
So, after manually cleaning, we hope to have better time in login when 2000 users try at the same moment...
That query was resposable for 6/7 secs at logon...