Web server performance for 5000 simultaneous users

Web server performance for 5000 simultaneous users

by Eimantas Jakas -
Number of replies: 28

Hello,

we have Moodle 3.7 in our university server and we have problem if we have > 1000 online users Moodle is getting very slow. In server performance overview we do not see any slow processes. What best web server performance settings do we need to set? Then do we need to think about load balancer and scale?

Average of ratings: -
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
It's impossible to comment without knowing a great deal more about your current configuration.

If you haven't done already, read Performance
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
Already read, but I didn't find anything specific to help. At the moment we are using:
- Ubuntu 18.04
- PHP 7.3 (FPM/FastCGI)
- Apache 2.4
- MySQL 5.0.12
- Memcached 3.1.4

Are there any spec performance settings to which attention should be paid? Maybe MySQL performance any upgrade?
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
MySQL 5.0?? Really?

Server specification(s)? Architecture?

Given that the above configuration isn't at all unusual, we need to figure out what is the problem.
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -

Sorry, we are using MySQL 5.7.30

What king of server specifications and architecture (its a lot information), can You make an example?

Are You asking this:

Server: Dell PowerEdge
CPU: 2x Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Hardisk: 4x300GB 10k + 4x1TB 7.2k SAS
Raid: Raid10 in both cases
Ram: 48GB
Firmware versijos: the latest
Network: 10 GbE
OS: Ubuntu 18.04.2 LTS

The server operating system is written to Raid10 4x300GB 10k RPM diskus. Moodledata deployed to Raid10 4x1TB 7.2k RPM SAS disks. 

We think its is not a server spec problem, but PHP configuration or/and MySQL (or all WEB server), because Moodle benchmark report always says very slow read/write to database, to file system.

In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

'We think its is not a server spec problem,'

You may well be right, but at the moment it is hard to work from 

'Moodle is getting very slow. '

Where and what is slow and for who/how i.e. >1,000 people doing what. Simply having them logged in doing nothing is unlikely to be an issue.   Is it slow all the time or is there a pattern. I once spent 6 months tracing down a slowness that was linked to the cron kicking off every 15 minutes (these days it should run every 1 minute so that is unlikely to be your issue).

Also for testing I suggest installing a parallel copy of Moodle with no plugins/integrations and see if that shows the same symptoms.  See if you can get some page load times.

In reply to Marcus Green

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
At the moment our University and students are working online and have quiz to take and every day from 10h till 13h and about 15h till 17h we have about 1,000 users online and in that moment Moodle loads very slow quiz are very slow page loading. Page load time ~10s - 15s. In other time Moodle is fast.
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
'Moodle loads very slow quiz are very slow page loading'
That is useful information and narrows things down significantly. Can I confirm that you can have around 1,000 users taking a quiz at the same time? If not can you confirm what the typical number is taking a quiz at any one time.
In reply to Marcus Green

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
Not one Quiz but at one same time different quizzes is taking about 1,000 users in different courses, we have now settlement period.
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Beta G -
First of all you have to use PHP Opcache for example Zend opcache and a newer MySQL version and Nginx because Apache is slow and consumes lot of resources. After this your bottleneck will not be web server but MySQL server.

I run up to 2000-2500 concurrent connections in a single virtualized server with my customizations in technology is Nginx web server, SSD NVME storage, sessions on webserver not in database etc and database is on remote server ( database is bottleneck most of times )

So first I think you need to upgrade mysql and change apache to nginx, tweak mysql and use innodb and lots of ram memory and so on.
In reply to Beta G

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

"First of all you have to use PHP Opcache for example Zend opcache and a newer MySQL version and Nginx because Apache is slow and consumes lot of resources"

If he's using PHP 7, then he's using Opcache (unless he turned it off which is a bit unlikely)

A newer version of MySQL is MySQL 8 which comes with its own challenges. However, I doubt it would be noticeably faster than MySQL 5.7

I know it's something of a religious thing but I think saying Nginx is faster than Apache (certain so you'd notice) is open to debate. I am well aware that Nginx performs well in some circumstances but I doubt you'll see much difference with Moodle. Assuming Apache is configured correctly (but assuming Nginx is configured correctly too). 

So I don't think any of these things will make any difference to the OP. However, your other points about storage, sessions and RAM may well be closer to the truth. 

In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Beta G -
I agree with you but I tried just to show some things which I believe are better and important. I use Apache Web server more in environment where I have more small sites together under a server and for that I like Apache more but when it comes to handle more concurrent users online etc its not a myth why all big sites use Nginx instead of Apache anyway I am not to make more debate for it. I have run my Moodle site before good also with Apache so perhaps its more in case of him storage and miss configuration.
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Is the Moodledata RAID attached by NFS or natively?
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
natively
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Ok - I'm interested to see the answer to Marcus's question, above. That many users taking a quiz would account for it being slow. Your system is probably just not powerful enough.
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
The exact time when people start a quiz can make a difference. Staggering over a few minutes can help.
In reply to Marcus Green

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
...and having one (or very few) questions per page.
Average of ratings: Useful (1)
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -

We have rules for all teacher that quiz one question per page must be. But hear we have problem that then student clicks second question page loads very slow about 10 - 15 sec., sometimes its brings "The page cannot be displayed".

'Staggering over a few minutes can help'
what do you mean? how could i do that? maybe you can explain more?

In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
He means split the quiz into groups and don't give them all exactly the same start time.
Average of ratings: Useful (1)
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
Ou... thats not possible, because we have about 200 teachers and to control all teachers to split quiz and groups to start in diff time.
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
How about WEB servers settings any PHP or MySQL config acceleration or request processing optimization
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think you're probably going to need more (virtual) hardware in that case. That's a huge amount of people to be doing a quiz at the same time.

However, some general thoughts (all of which are documented, however)

- If you don't already, organise all the monitoring you can. You need to know what your server is doing. Munin is a good start
- Get a redis server. Use Redis for caching and for sessions
- Consider using the database for file locking.
- Tune the database (if you've done nothing, it will be using far too little resource)
- Tune the webserver (usually the opposite of the DB. They come configured to serve static web pages and can use too much resource)
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -

Can You suggest how can I tune the database and webserver?? and how can I do database for file locking and how can it help?

In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
- https://github.com/major/MySQLTuner-perl
- Performance_recommendations
- Database file locking is set up in config.php. You'll find the lines you need in config-dist.php
- https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04
- Caching
- If you have Redis you can put the sessions there as well. like file locking, find the lines in config-dist.php
Average of ratings: Useful (2)
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

Performance good practice is one thing. Finding the bottleneck in a particular case is something different. The OP has yet to disclose the Moodle Benchmark results https://moodle.org/mod/forum/discuss.php?d=402542#p1625784.

P.S. We are definitely looking at the Hardware and performance forum. ;)

In reply to Visvanath Ratnaweera

Re: Web server performance for 5000 simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
He actually posted them when I asked him to and then deleted them again. Which is odd.

Sometimes, if you can't get to the root cause, you just have to do *something* and see what happens IMO.

Yep - good point - moving.
In reply to Howard Miller

Re: Web server performance for 5000 simultaneous users

by Eimantas Jakas -
at the moment we just noticed that if we have over 500 users online page loads about 20sec if its <500 then load drops.
In reply to Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

Hi

You wrote:

We think its is not a server spec problem, but PHP configuration or/and MySQL (or all WEB server), because Moodle benchmark report always says very slow read/write to database, to file system.

I can't follow the logic. Could you post a screen-shot of the Moodle benchmark results?