Web server performance for 5000 simultaneous users

Web server performance for 5000 simultaneous users

από Eimantas Jakas -
Αριθμός απαντήσεων: 28

Hello,

we have Moodle 3.7 in our university server and we have problem if we have > 1000 online users Moodle is getting very slow. In server performance overview we do not see any slow processes. What best web server performance settings do we need to set? Then do we need to think about load balancer and scale?

Μέσος όρος βαθμολογίας: -
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
It's impossible to comment without knowing a great deal more about your current configuration.

If you haven't done already, read Performance
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -
Already read, but I didn't find anything specific to help. At the moment we are using:
- Ubuntu 18.04
- PHP 7.3 (FPM/FastCGI)
- Apache 2.4
- MySQL 5.0.12
- Memcached 3.1.4

Are there any spec performance settings to which attention should be paid? Maybe MySQL performance any upgrade?
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
MySQL 5.0?? Really?

Server specification(s)? Architecture?

Given that the above configuration isn't at all unusual, we need to figure out what is the problem.
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -

Sorry, we are using MySQL 5.7.30

What king of server specifications and architecture (its a lot information), can You make an example?

Are You asking this:

Server: Dell PowerEdge
CPU: 2x Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Hardisk: 4x300GB 10k + 4x1TB 7.2k SAS
Raid: Raid10 in both cases
Ram: 48GB
Firmware versijos: the latest
Network: 10 GbE
OS: Ubuntu 18.04.2 LTS

The server operating system is written to Raid10 4x300GB 10k RPM diskus. Moodledata deployed to Raid10 4x1TB 7.2k RPM SAS disks. 

We think its is not a server spec problem, but PHP configuration or/and MySQL (or all WEB server), because Moodle benchmark report always says very slow read/write to database, to file system.

Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Marcus Green -
Φωτογραφία Core developers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Plugin developers Φωτογραφία Testers

'We think its is not a server spec problem,'

You may well be right, but at the moment it is hard to work from 

'Moodle is getting very slow. '

Where and what is slow and for who/how i.e. >1,000 people doing what. Simply having them logged in doing nothing is unlikely to be an issue.   Is it slow all the time or is there a pattern. I once spent 6 months tracing down a slowness that was linked to the cron kicking off every 15 minutes (these days it should run every 1 minute so that is unlikely to be your issue).

Also for testing I suggest installing a parallel copy of Moodle with no plugins/integrations and see if that shows the same symptoms.  See if you can get some page load times.

Σε απάντηση σε Marcus Green

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -
At the moment our University and students are working online and have quiz to take and every day from 10h till 13h and about 15h till 17h we have about 1,000 users online and in that moment Moodle loads very slow quiz are very slow page loading. Page load time ~10s - 15s. In other time Moodle is fast.
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Marcus Green -
Φωτογραφία Core developers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Plugin developers Φωτογραφία Testers
'Moodle loads very slow quiz are very slow page loading'
That is useful information and narrows things down significantly. Can I confirm that you can have around 1,000 users taking a quiz at the same time? If not can you confirm what the typical number is taking a quiz at any one time.
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Beta G -
First of all you have to use PHP Opcache for example Zend opcache and a newer MySQL version and Nginx because Apache is slow and consumes lot of resources. After this your bottleneck will not be web server but MySQL server.

I run up to 2000-2500 concurrent connections in a single virtualized server with my customizations in technology is Nginx web server, SSD NVME storage, sessions on webserver not in database etc and database is on remote server ( database is bottleneck most of times )

So first I think you need to upgrade mysql and change apache to nginx, tweak mysql and use innodb and lots of ram memory and so on.
Σε απάντηση σε Beta G

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers

"First of all you have to use PHP Opcache for example Zend opcache and a newer MySQL version and Nginx because Apache is slow and consumes lot of resources"

If he's using PHP 7, then he's using Opcache (unless he turned it off which is a bit unlikely)

A newer version of MySQL is MySQL 8 which comes with its own challenges. However, I doubt it would be noticeably faster than MySQL 5.7

I know it's something of a religious thing but I think saying Nginx is faster than Apache (certain so you'd notice) is open to debate. I am well aware that Nginx performs well in some circumstances but I doubt you'll see much difference with Moodle. Assuming Apache is configured correctly (but assuming Nginx is configured correctly too). 

So I don't think any of these things will make any difference to the OP. However, your other points about storage, sessions and RAM may well be closer to the truth. 

Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Beta G -
I agree with you but I tried just to show some things which I believe are better and important. I use Apache Web server more in environment where I have more small sites together under a server and for that I like Apache more but when it comes to handle more concurrent users online etc its not a myth why all big sites use Nginx instead of Apache anyway I am not to make more debate for it. I have run my Moodle site before good also with Apache so perhaps its more in case of him storage and miss configuration.
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
Is the Moodledata RAID attached by NFS or natively?
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
Ok - I'm interested to see the answer to Marcus's question, above. That many users taking a quiz would account for it being slow. Your system is probably just not powerful enough.
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Marcus Green -
Φωτογραφία Core developers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Plugin developers Φωτογραφία Testers
The exact time when people start a quiz can make a difference. Staggering over a few minutes can help.
Σε απάντηση σε Marcus Green

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
...and having one (or very few) questions per page.
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -

We have rules for all teacher that quiz one question per page must be. But hear we have problem that then student clicks second question page loads very slow about 10 - 15 sec., sometimes its brings "The page cannot be displayed".

'Staggering over a few minutes can help'
what do you mean? how could i do that? maybe you can explain more?

Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
He means split the quiz into groups and don't give them all exactly the same start time.
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -
Ou... thats not possible, because we have about 200 teachers and to control all teachers to split quiz and groups to start in diff time.
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
I think you're probably going to need more (virtual) hardware in that case. That's a huge amount of people to be doing a quiz at the same time.

However, some general thoughts (all of which are documented, however)

- If you don't already, organise all the monitoring you can. You need to know what your server is doing. Munin is a good start
- Get a redis server. Use Redis for caching and for sessions
- Consider using the database for file locking.
- Tune the database (if you've done nothing, it will be using far too little resource)
- Tune the webserver (usually the opposite of the DB. They come configured to serve static web pages and can use too much resource)
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Eimantas Jakas -

Can You suggest how can I tune the database and webserver?? and how can I do database for file locking and how can it help?

Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
- https://github.com/major/MySQLTuner-perl
- Performance_recommendations
- Database file locking is set up in config.php. You'll find the lines you need in config-dist.php
- https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04
- Caching
- If you have Redis you can put the sessions there as well. like file locking, find the lines in config-dist.php
Σε απάντηση σε Howard Miller

Re: Web server performance for 5000 simultaneous users

από Visvanath Ratnaweera -
Φωτογραφία Particularly helpful Moodlers Φωτογραφία Translators

Performance good practice is one thing. Finding the bottleneck in a particular case is something different. The OP has yet to disclose the Moodle Benchmark results https://moodle.org/mod/forum/discuss.php?d=402542#p1625784.

P.S. We are definitely looking at the Hardware and performance forum. ;)

Σε απάντηση σε Visvanath Ratnaweera

Re: Web server performance for 5000 simultaneous users

από Howard Miller -
Φωτογραφία Core developers Φωτογραφία Documentation writers Φωτογραφία Particularly helpful Moodlers Φωτογραφία Peer reviewers Φωτογραφία Plugin developers
He actually posted them when I asked him to and then deleted them again. Which is odd.

Sometimes, if you can't get to the root cause, you just have to do *something* and see what happens IMO.

Yep - good point - moving.
Σε απάντηση σε Eimantas Jakas

Re: Web server performance for 5000 simultaneous users

από Visvanath Ratnaweera -
Φωτογραφία Particularly helpful Moodlers Φωτογραφία Translators

Hi

You wrote:

We think its is not a server spec problem, but PHP configuration or/and MySQL (or all WEB server), because Moodle benchmark report always says very slow read/write to database, to file system.

I can't follow the logic. Could you post a screen-shot of the Moodle benchmark results?