Hardware setup for 5000 concurrent users

Hardware setup for 5000 concurrent users

by afshan shahzad -
Number of replies: 7

We are expecting 5000 concurrent users attempting a quiz. What is the ideal hardware/software setup required to cater to the users.

Average of ratings: -
In reply to afshan shahzad

Re: Hardware setup for 5000 concurrent users

by Ken Task -
Picture of Particularly helpful Moodlers

Before anyone could suggest, might be good idea to share info on your current setup ... this a single/standalone moodle with DB and moodledata on same server?   Memory of server, etc.   Running MySQL/MariaDB?   Suggest installing MySQLTuner and run with superuser credentials.   There will be suggestions at the bottom of the run.

Basically, it's a balancing act between web service and DB if on same server.   The more memory one can allocate to the DB the better ... very little Disk I/O then for the DB.

Let the discussion/cussion begin! smile

'SoS', Ken


In reply to afshan shahzad

Re: Hardware setup for 5000 concurrent users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I can only add... if at all possible don't do that. If you genuinely have 5,000 users all starting a quiz at exactly the same time you are going to have to pay a LOT of money to build a stable system.
In reply to Howard Miller

Re: Hardware setup for 5000 concurrent users

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
"If you genuinely have 5,000 users all starting a quiz at exactly the same time you are going to have to pay a LOT of money "
And you might have to pay a lot less if the users start the quiz spread over a 2 to 4 minute time frame. Setting up systems to do this is a specialist area and I recommend contacting a Moodle partner, as that is the sort of thing they do for a living.
Average of ratings: Useful (2)
In reply to afshan shahzad

Re: Hardware setup for 5000 concurrent users

by Ken Task -
Picture of Particularly helpful Moodlers

According to your other posts in these forums:
Multiple sites that use LDAP for authentication.
Not sure of which platform(s) you have moodle(s) running, nor DB server.

So is the question about 5000 users taking a quiz in context of above?  Combining users from the various moodles you are running?   ... kinda like a 'testing' server/instance of Moodle?


Before your students could take a quiz they have to login.
If still using AD/LDAP how is that working (speed/memmory consumption/etc)
for the largest number of students you have?

Of all the moodle modules, quiz is the heaviest processing.
Also, how the quiz is presented to takers makes a difference.
100 question test where all questions are presented to the takers at one time,
very heavy processing and would drag an server to a creepy crawl.

Other things - are all students taking the test from designated locations?
Meaning, all students are required to report to lab's A, B, C to take it.
Or are they allowed to be on campus, using their own device, connected to your
network and test is restricted access in a network sense.

In the docs for performance ... a couple of things kinda stand out:
DB server and it's config (web service to a lesser extent although equally important).
And memory of server if single box (shudder).  Not to mention SSD/moodledata and of course what comes before ll that networking.  One that stands out ... hits per second!

Anyhoo ... if you have been running moodle(s) and are now attempting to centralize
testing, you have some data you can collect now to get an idea of what you seek!

Not the best tool, but enough to get an idea.

Webalizer

Has a section on Top 10 or X number Total Entry Pages table - by month (bellow is Apr).
A production k12 site that is used primarily these days as a test/quiz machine:

Top 10 of 140 Total Entry Pages
#     Hits     Visits     URL
1     39954     5.51%     8605     41.85%     /
2     16228     2.24%     3150     15.32%     /course/view.php
3     124597     17.18%     2817     13.70%     /lib/ajax/service.php
4     13358     1.84%     1838     8.94%     /login/
5     11137     1.54%     879     4.27%     /course/
6     50153     6.91%     330     1.60%     /mod/quiz/attempt.php

The typical path for students on test days for any course.
1. login (they use LDAP)
2. navigate to Course
3. locate the test/quiz and begin.

Other such tables for Webalizer in example site for Apr:
Top 10 of 5854 Total URLs By KBytes
#     Hits     KBytes     URL
1     50153     6.91%     11025662     23.89%     /mod/quiz/attempt.php

Top 30 of 5854 Total URLs
#     Hits     KBytes     URL
1     124597     17.18%     1665940     3.61%     /lib/ajax/service.php
2     50153     6.91%     11025662     23.89%     /mod/quiz/attempt.php

** and above info is for 'legit traffic' ... server is scanned quite frequently.

Hits by response codes:
Code 400 - Bad Request     0.07%     481
Code 403 - Forbidden     0.00%     19
Code 404 - Not Found     0.66%     4758
Code 407 - Proxy Authentication Required     0.00%     3
Code 408 - Request Timeout     4.63%     33563

Do you have any such tool running right now?

Lots to investigate and consider!

'SoS', Ken

In reply to Ken Task

Re: Hardware setup for 5000 concurrent users

by Ken Task -
Picture of Particularly helpful Moodlers

And 2 cents more ... am guessing that you are a tech given charge by 'powers that be' to setup such a beast and that you, yourself, have never conducted a moodle class/course with testing.

JMeter Testing ... not on a production server, but one hosted/configured however you like.

https://docs.moodle.org/dev/Load_testing_Moodle_with_JMeter

https://moodle.org/mod/forum/discuss.php?d=425057

https://docs.moodle.org/39/en/JMeter_test_plan_generator

Comment ... had Amazon, Google, Azure existed 25 years ago, I wouldn't have
purchased 3 servers ... all of which have been decommissioned and scripted for parts.

Wonder ... the cost of setting up this system for 5000 (buying the hardware) vs a leased setup via a Moodle Partner or robust/burstable setup with Amazon, Google, or even Azure [guest OS Ubuntu] just for the purpose of such testing.

'SoS', Ken

In reply to afshan shahzad

Re: Hardware setup for 5000 concurrent users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I was wondering... we get this sort of request a lot. "I'm going to do a quiz for [Insert huge number here]  simultaneous users". If you can tell us, what actually is this quiz? I'm just interested.