Cluster design for 15,000 concurrent users in Moodle

Cluster design for 15,000 concurrent users in Moodle

by Talhah Khan -
Number of replies: 14

Dear all,

I am sure this question has been asked by other people. I saw many existing threads but I didn't get a clear answer.

We need to use Moodle for a big school network where they have a requirement of 15,000 concurrent users during the school time (for taking videos and attempting quiz) between 8 AM to 12 PM. 

  1. What configuration would be required to support 15,000 users? I saw that 1 GB RAM is required to support 20 users. So in theory, probably 720 GB ram is need across multiple application servers. Would it be able to handle 15,000 concurrent Quiz simultaneously?
  2. We have assumed 5 web application servers each having 96 GB Ram, 48 cores, 1920 GB SSD, 8 TB networking etc. And 2 database servers each having 512 GB ram, 64 cores, 7200 GB SSD, 12 TB network. So a total of 450 GB ram for application servers and 1024 GB for DB Server. Is it a good estimate?
  3. Would we need any caching as we are going to use multiple app servers? How much cache (Redis) etc we should assume if needed?
  4. How to handle files across application servers? Do we need to use network storage or Amazon storage etc?
  5. Would one DB server be enough in addition to multiple application? I am afraid one DB server may not suffice. Can we use PostgreSQL for multiple database servers? How to configure multiple DB Servers (not as ready only replica for backup but as a cluster)
  6. We are thinking to use dedicated virtual machines from Linode.com for both DB and application server. Does anyone has any good/bad experience with Linode?
  7. Do we need to consider anything else (in addition to cache, load balancer, shared network storage) for this cluster deployment?

Thank you,

Talhah

Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Alex Rowe -

How sure are you of the 15k concurrent users? As an example, you might have 2000 users using the site over the day, but only 700 are active (5 minute window) at the one time. Of those 700 you might only have 20 users doing anything at the EXACT same time.

If you have 15,000 users doing a quiz at the same time, that could be 15,000 users clicking start quiz simultaneously which results in 15k database connections (unless pooling), 15k MUC connection, session connections, etc, etc.

If you don't have expertise running a Moodle system of this size, it might be wise to reach out to a Moodle Partner to spec a system specifically for you.

Otherwise a basic LAMP stack with almost a TB of RAM is probably not going to work without a lot of tweaking of DB, OS, networking level settings.

For the other questions:

3. Redis at that size, it runs quite small, so maybe 8GB RAM would be enough to start. Use it for MUC and Sessions, but use two different hosts or run two instances on the same hosts. It's also only single threaded.
Also, you should (definitely) set up SSL offload and caching on the load balancer for all static assets.

4. Definitely need shared storage for the dataroot accessible from all application servers. Fast SSD is best

5. Your choice, MDL-19711 may help here.

7. Backup storage, monitoring, alerting, upgrades, probably lots of others too.

Average of ratings: Useful (3)
In reply to Alex Rowe

Re: Cluster design for 15,000 concurrent users in Moodle

by Talhah Khan -
Dear Alex, many thanks for your replies. We need 15,000 users because 1) system will be used in a network with 100,000 users between 8 to 12 pm 2) on some instance during the year, tests will be done during a specific hour using the system 3) we are asked to provide an SLA of 15,000 concurrent users specifically by client. We gave lower numbers based on tradition ratios of registered to daily login users to concurrent users. Based on some history, they know concurrent usage will be higher.

Thank you for you suggestion to reach out a moodle partner. I am concerned that a basic LAMP installation even in a cluster may not be that simple. I just wanted to know all the risks and considerations - right now we need to do sizing and get back to our client with a quote.

We may or may not decide to go with a partner. It all depends on the cost/pricing. But I agree, it would be a better choice.
Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
15,000 concurrent users is huge! How did you arrive at that figure?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Cluster design for 15,000 concurrent users in Moodle

by Talhah Khan -
Dear Visvanath,

We need 15,000 users because 1) system will be used in a network with 100,000 users between 8 to 12 pm 2) on some instance during the year, tests will be done during a specific hour using the system 3) we are asked to provide an SLA of 15,000 concurrent users specifically by client. We gave lower numbers based on tradition ratios of registered to daily login users to concurrent users. Based on some history, they know concurrent usage will be higher.
Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Usman Asar -
Picture of Plugin developers Picture of Testers
realistically it would be 1000 concurrent, no more than that. try to understand the meaning of concurrent (not registered, not logged in) meaning all concurrent users will click to fetch database at the exact same instance.
btw, which institute is it to hold that many students?

PostgreSQL is being used by OU-UK and the moodle admin particularly referenced it being used for that many users (I dont recall the exact phrase he used).


Average of ratings: -
In reply to Usman Asar

Re: Cluster design for 15,000 concurrent users in Moodle

by Talhah Khan -
Dear Usman,

I understand the difference between registered users (user base), daily logged in users (DAU) and concurrent users (users that are sending requests to web/db server concurrently at any point in time).

The client has specifically asked for this number and we also need to do load testing as part of providing a solution. Are there any reference cases available for such big concurrency?
Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Usman Asar -
Picture of Plugin developers Picture of Testers

See if Tim Hunt can assist, they are running OU's servers with 10 web facing web servers and database on PostgreSQL with Master-Slave replication (not load balanced), if your client's user base is local, then ideally get servers locally as Linode will cost nearly $15k monthly.

Average of ratings: Useful (2)
In reply to Usman Asar

Re: Cluster design for 15,000 concurrent users in Moodle

by Talhah Khan -
Thanks Usman - how many concurrent users they have at OU UK? I doubt they have 15,000 concurrent users. I think we will need a lot more fire power for 15,000 concurrent users (we have to ensure 15,000 assessments/quiz as well).
Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Usman Asar -
Picture of Plugin developers Picture of Testers

That I am not sure how many concurrent users they have, even if we assume 15k, all wont be taking quizzes and quizzes are most resource demanding in moodle, you can always setup 2 DB servers with 3 web servers, and add on top if need be, as OU's hardware must be decade old, and architecture does effect performance. Rather than going to Linode, get AMD Epyc on NVMe drives. I am more into Windows server setups therefore wont be able  to assist much on Linux I am afraid.

CatalystIT have as well deployed a larger solution with 3+ million users for Saudi Government, they must have plenty concurrent to seek their assistance.


Average of ratings: -
In reply to Usman Asar

Re: Cluster design for 15,000 concurrent users in Moodle

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
Thanks for the shout-out Usman! smile

Yes - 15K concurrent users is something we can definitely support on our systems - Talhah - as others mention, you really need to get a Moodle Partner involved in this - trying to do design and support a system like this without prior experience is really tough and could end up with the project being a total failure. There is a lot to think about in terms of both infrastructure design and configuration as you will have seen from your research and the posts here in the forums.

It sounds like the 15K concurrent users is also something that you don't need to support during the normal day-to-day periods either so you'd want to have a system that allows you to easily scale up or down when the large online quizzes are running so you're not paying crazy amounts of $$ for hardware that is under-utilised 95% of the time.
Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

The reason why I asked is, we get lot of enquiries from people who overestimate the concurrency drastically. You know that servers do not scale linearly. So an overestimation can sky rocket the costs.

Anyway, you have been given this figure by your customer. So no point in discussing whether it is a real estimation or an overestimation. Your product has to pass this test. Sorry, I don't claim to know an answer. FYI, this https://docs.moodle.org/en/Performance_FAQ#How_do_you_define_.22concurrent_users.22.3F is what we mean, when we say concurrent users.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Cluster design for 15,000 concurrent users in Moodle

by Talhah Khan -
Many thanks Visvanath! Yes we have to comply to the number and also do load test as part of solution provisioning. I saw the thread you shared. So there is no guidance available on how to design a solution for 15,000 users. Moodle has thousands of installations, we should have some standard configurations for say 1,000 or 5,000 or 10,000 or 15,000 concurrent users...
Average of ratings: -
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Mathew Gancarz -
Picture of Core developers
Hi Talhah, the challenge is because Moodle is capable of so much and can be used in so many different ways, there are no 'standard configurations' that would always work. Even putting aside all the challenges in defining a 'concurrent user', 5,000 concurrent users watching a video is very different than 5,000 concurrent users reading some text and again very different than 5,000 concurrent users taking a quiz.

We ran a project recently that was specced to handle tens of thousands of concurrent users and used this Maximum template: https://github.com/Azure/Moodle to get something up quickly and cheaply without engaging a Moodle partner. We ended up with about 100-200 concurrent users and that project ended up switching to a partner so that the in-house team wouldn't have to worry about the details of the platform. The complex clustered setup could have just been a single beefy LAMP server.

I would strongly suggest engaging a Moodle partner to dig into this a bit more, there is too much details that are unique to each institution's use at that scale that can break your deployment.

The cheaper alternative is to just go with something like what you suggested or a templated build, start with smaller deployments and adjust as needed, but only you know your budgets and the risks involved if things don't work.

For load testing, see https://docs.moodle.org/dev/JMeter but you will run into bandwidth issues trying to simulate that many users unless you build a cluster of machines to test with. I've been successful testing up to about 200 'concurrent users' using a single machine. So in theory to test 15,000, you would need about 75 machines running the jmeter testing scripts, but you'll probably run into bottle necks from the internet connection those are hooked up to, unless you geographically disperse them.
Average of ratings: Useful (3)
In reply to Talhah Khan

Re: Cluster design for 15,000 concurrent users in Moodle

by Amir Fouladvand -
Hey Talhah,
Did you finally run the mentioned project? and if yes, would you please share your experience? because Im also interested to know the result of 15K concurrent quiz makers with above HW capacity.
Thanks in advance
Average of ratings: -