Moodle in English: Server Configurations required

Hi,

I am using Moodle 4.4.1

I have round 7 live moodle instances and 40+ moodle backup instances.

Live instances carries current session (i.e. June 2024) examinations for 7 different universities.

Backup instances carries old sessions examinations like June 2023 or January 2024.

All 7 live instances having different traffics:

One have 60000+ users per session (around 5000+ users login at a same time (Per second)).

One have 40000+ users per session (around 3000+ users login at a same time (Per second)).

Other live instances have 5000+ users per session (around 1000+ users login at a same time (Per second)).

So you can say around 110000+ users login at the same time per session in thousands of quizzes.

Now as I said above we have 40+ instances old old sessions. These 40+ instances have only admin/manager access which means do not have much traffic.

Instances having 60000+ and 40000+ students crashes often.

Can anyone suggest right server configurations for above scenario?

Please help.

平均分： -

Re: Server Configurations required

Visvanath Ratnaweera - 2024年06月21日星期五 17:34

That is a major operation! I think it is beyond a level where you can put random tips and tricks of a community together and build the solution. Yours needs professional support like from Moodle HQ or a Moodle Partner. You can also try A new "Large scale Moodle deployment" support group. (I don't know any details other than that discussion.)

I have some counter questions to understand your current set up and the demand:

- When you talk of "sessions", they are something like exam seasons, right? Twice a year, in January and in June?

- If we take the 60000+ users session as an example, do you mean 60k users take a synchronous exam, say today 08:00-12:00, and those 60k users open the exam at the rate of 5 k users per second, that is all 60 k have loaded the exam 12 seconds?

- Is there anything called a typical exam? How does it look like?

- How do you do that today? You talk of Moodle "instances". Does it mean, there is a Moodle for that 60 k users and another for the 40 k users? Are they separate machines (hardware) or somehow in a one big server architecture? Could you give more details, like as requested in Before you post.. Read this..?

- Have you collected performance matrices of the servers when they crash and also their load (stress and strain)?

平均分：Useful (2)

Re: Server Configurations required

Aditya Dubay - 2024年06月27日星期四 16:17

Hi Vishwanath Ji,

Thanks for reply.
In one instance of Moodle we have 60000+ users. For Example 23000 in DCA and 37000 in PGDCA and today is Fundamental of computers exam (MCQ Quiz) for both courses. Quiz is opened from 00:00 to 23:59 today.
We cant restrict users like if 5000 users logged in then others can wait for their turn. So we do not know out of 60000 users how many users logged in together.
You can count Jan and June sessions as Semesters.
All Instances (60k and 40k and others) are in same server.

平均分： -

Re: Server Configurations required

Visvanath Ratnaweera - 2024年06月27日星期四 16:35

So 23000 students taking one MCQ quiz and 47000 students taking another MCQ test within a time frame of 24 hours, all on one server. There are other Moodle instances, one with 40000 users, on the same server.

No wonder the server "crashes" irrespective of how big it is!

I have to agree with the "insane" comment by Howard. Unfortunately I never had the chance to run a system a fraction this size. So can't be of help. (See my first reply.)

You read the documentation linked to the header of this forum, right?

Just curious: How long is the MCQ quiz today (in hours)? And how big is it: Number of questions? the question types? how many per page? whether students have to watch videos or listen to audio in those questions? And the other students, apart from those 60000, what do they do today (on that server)?

平均分： -

Re: Server Configurations required

Aditya Dubay - 2024年06月28日星期五 14:02

What to say now if old Moodlers are not confident about "How moodle perform with large amount of users.".
I feel instead of calling "insane/insanity", you people can say that you people dont have any idea about it.
Also I never see any limitation on user count from Moodle.

I feel you people need to see this link:

https://docs.moodle.org/404/en/Large_installations#:~:text=Currently%20the%20largest%20known%20Moodle,with%20over%203.5%20million%20users.

平均分： -

Re: Server Configurations required

Visvanath Ratnaweera - 2024年06月28日星期五 14:50

I understand that you don't want to discuss with me. As I repeatedly said, I have no experience on installations of this scale. I was trying to express the problem in Moodle terms for others to rhyme in.

I don't know how you conclude that "old Moodlers" must be confident about "How moodle perform with large amount of users." In my case, the first part maybe true, not necessarily the second part when it comes to multiple-ten-thousand synchronous quiz users!

BTW, You read the documentation linked to the header of this forum, right?

About the link: Thanks, I think I've seen it. But it doesn't say how to make them. Don't mix up the menu with the menu, Reminder: Don't eat the menu!

P.S. I have an Alice Quote for that. But came to know some readers don't appreciate them. So decided against.

平均分： -

Re: Server Configurations required

Ken Task - 2024年06月28日星期五 23:24

Hmmm ... I detect a hint of 'age discrimation/bias' ... reference "old Moodlers".

Uhhhh ... "old Moodlers" are the persons or company that have the most experience with the app but not necessarily the size of the implementation you seek (scale).

So maybe you need to explore 'load balancing' on Amazon or Google?

Or hosting with a certified Moodle Partner!

'Spirit of Sharing' ... an 'old Moodler',

Ken

平均分：Useful (1)

Re: old Moodlers are not confident [side topic]

Visvanath Ratnaweera - 2024年06月29日星期六 02:47

Either way "old Moodlers" have nothing to fear. See https://moodle.org/mod/forum/discuss.php?d=459061#p1843667.

平均分： -

Re: Server Configurations required

Aditya Dubay - 2024年06月29日星期六 14:40

Old Moodler in case of experience.

You know Ken and Visvanath sometimes we came here on forum with hope that someone can help us but when we heard "insanity" kind of comment we get demoralized.
Moodle can handle millions of users.
I am sorry if "Old moodlers" hurts you.

平均分： -

Re: Server Configurations required

Howard Miller - 2024年06月29日星期六 15:36

Would you rather we said "sure that's easy" or - using our combined experience - would you rather we said, "that's not going to work, modify your plans"? I'll just leave that with you....

平均分：Useful (1)

Re: Server Configurations required

Ken Task - 2024年06月29日星期六 18:42

Am not 'hurt' at all!! Am a 'crusty' 75+ 'something'!

These forums are more often than not a source of information for 'do it yourself' moodlers (small to medium sized projects).

Do contact a Moodle Partner.

https://moodle.com/services/certified-service-providers

If not an MP, then use

https://stats.moodle.org/

Registered sites by size @ above link to find sites the size similar to one that you seek.

Good luck!

'SoS', Ken

平均分： -

Re: Server Configurations required

Mary Cooch - 2024年06月29日星期六 19:44

Also, again to repeat what Visvanath linked to earlier - there is a Large scale Moodle installation Telegram group (I am not in it, but understand it is popular) where you can ask.

平均分： -

Re: Server Configurations required

Howard Miller - 2024年06月22日星期六 23:08

What are all these users doing? What sort of activities and materials do you have?

110,000 users logging in at the same time and starting a quiz is - sorry - insanity. We have a VERY large Moodle setup and we get worried about starting quizzes in the high hundreds.

平均分： -

Re: Server Configurations required

Aditya Dubay - 2024年06月27日星期四 16:18

Please elaborate "Insanity"

平均分： -

Re: Server Configurations required

Howard Miller - 2024年06月29日星期六 15:34

The normal old/fashioned meaning.

We have EIGHT very big web servers, a substantial Enterprise MySQL database, loads of Redis servers and so on. We get very nervous when simultaneous quizzes of around 1,000 students start. We look at all sorts of things that make a difference (e.g. randomisation makes a big hit, too). So, on that basis, if you want to start a quiz with 110,000 students all at the same time. You'll need (to start with) around 1,000 clustered VMs running Moodle. All will be BIG and all will be expensive. I've no idea how you would configure a database to handle that without smoking out.

Even 23,000 (reading the above) is a going to require a massive Moodle installation for which you're going to need consultancy from someone who really understands this area.

So, if you SERIOUSLY want to start a quiz with over 100,000 students I think you will need the funds equivalent to the GDP of a small, European country and you will need support from someone with more experience of huge installations than most of us have.

So, anyway, insanity

平均分：Useful (1)

Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Visvanath Ratnaweera - 2024年06月30日星期日 14:08

> to start a quiz with over 100,000 students I think you will need the funds equivalent to the GDP of a small, European country

Yes, an European country. The US still seem to maintain their "American size". See https://federalnewsnetwork.com/contractsawards/2022/12/dod-ends-cloud-contracting-saga-with-four-awards/.

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Marcus Green - 2024年06月30日星期日 16:27

The answer to this question is to use a Moodle partner with experience of very large sites. I was attracted to the Moodle partner I work for many years ago when there was a reference to them running a Moodle site with 2 million "active users". Two years ago I was delighted to be employed by them (not on the performance optimisation side I should add). They now run significantly larger sites. They run huge quizzes, typically on sites dedicated for the purpose that are "scaled up" for those events.

You need access to people who do this sort of thing both for a living and for a vocation and it requires a significant financial cost.

To add context to the other replies in this thread, many of us have been reading questions for over a decade about Moodle sites with huge number of users that turn out to be somewhat "theoretical".

平均分：Useful (1)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Visvanath Ratnaweera - 2024年06月30日星期日 16:42

Oh, yes! That is why I always ask people to read past discussions in this forum, in this case "How can I have N simultaneous users?" type of questions, claiming that they build a knowledge base. In fact they are also entertaining.

When N > 10 k, they tend to be more entertainment that binding recipes - in this forum I must stress. Sure there are people "who do this sort of thing both for a living and for a vocation". I don't expect them to distribute such recipes. Even if they do, the recipients must be quite exceptional system administrators with hoards of hardware, money and labour at their disposal. If one needs an analogy, it can't be very difficult to collect the blueprints on how to build a mega city. But building one needs more than mere blueprints.

And in our altruism let's not forget budgets of that scale are not always not technocratic. To come to the off-topic of this branch read Pentagon cancels $10bn cloud contract awarded to Microsoft. (That sage continued.)

平均分：Useful (1)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Marcus Green - 2024年06月30日星期日 18:34

"I don't expect them to distribute such recipes"
To extend the recipe analogy, if a chef gives you a recipe, it will not make you a chef.

平均分：Useful (1)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

aykut ozkaya - 2024年07月11日星期四 07:27

Thanks for asking a brave question like this.
Yes scale and concurrency is something we all face.

I like to see more discussion on what it takes to run Moodle on a larger scale? Yes it may be costly to setup.

Aditya- I am sure you already have considered the webserver, db server and nginx config to serve at this scale. Could you share more about your experience with the community please? How is it you are scaling for 100K users? In my recent stress tests I was able reach 10K but 100K is out of my reach at the moment

Thanks
In addition:
it is a matter of adding loadbalancers into the mix and adding additional
servers as required.

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Ben Girard - 2024年07月13日星期六 03:17

We have the same problem with hyperscale and concurrency with Moodle. We're debating staying with Moodle running in the cloud or switching platform since we can't find anyone or a firm to hire provide a firm auto-scale architecture that would handle >50 user per second login rate and 1,000,000 users in the system.

Apparently, Moodle US can handle Air Force and Army but there is no "recommended architecture plan" telling exactly how is this supposed to be setup?

Currently, JMETER load testing chokes at the SSO login stage.

Any recommendations to find real professionals with hyper-scale knowledge of Moodle configurations on AWS or Azure and JMETER for load testing? If yes, let me know, we want to retain your services ASAP.

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

aykut ozkaya - 2024年07月13日星期六 04:12

Hi Ben

I think moodle can easily handle this provided you have the right networking infrastructure. May that be AWS elastic, or load balancer configuration on data centre with required number of servers to spin as an when needed or indeaded your own nginx proxy to distribute the traffic to multiple servers. ( the last case - I found getting moodle communicate in to reverse proxy with nginx a little tricky). Nevertheless, really achievable, as I said I was able to hit 10K all it needs a scale up to reach 100K.

What kind of of time scale you guys have?

平均分： -

Re: Recommendations to find real professionals with hyper-scale knowledge of Moodle configurations on AWS or Azure and JMETER for load testing?

Visvanath Ratnaweera - 2024年07月17日星期三 03:12

The "real professionals" are by definition the Moodle Partners as already pointed out in the original thread. You are free to get second opinion in the Large scale Moodle installation Telegram group Mary linked above.

There is also the Moodle Jobs database. It uses the same login as in the forums.

P.S. I am not convinced that you have the same problem as the OP. There are so many factors involved in a high performance server situations that it is impossible for two people to have the same factors.

平均分：Useful (2)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Howard Miller - 2024年07月17日星期三 18:01

"...but there is no "recommended architecture plan" telling exactly how is this supposed to be setup?"

and there is never going to be as there are too many variables.

If you are coming at this from a standing start then you have a BIG problem. There are one or two Moodle Partners who might be able to help you but that's about it. Failing that, I would always recommend starting out with a smaller pilot and gaining some real-world experience.

平均分：Useful (1)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

aykut ozkaya - 2024年07月17日星期三 23:35

Variables are not necessarily Moodle specific but rather to do with server configuration.
Tim Hunts OU architecture caters for one of the largest university in UK https://www.slideshare.net/slideshow/ou-moodle-hosting/49708981
Since the above publication the server technology improved a huge amount. I found its more CPU core rather than RAM which determines performance at this scale.
But having powerful machines is half the battle. Configuration is another.
In the posts above the server load sometimes referred as number of concurrent users and more recently number of users per second. I think this needs to be clarifed in order to realisticly Jmeter test the upper limits of a high performance server architecture.
For example when jmeter test to simulate 10K users in a 10 question quiz with thinking time and 380 s delay:
Server logs did show traffic around 70K hits at its peak. Obviously these are user page request and activities. However , Moodle site log data shows around 600 -700 log records per second at its maximum. Moodle main dashboard confirms 10K test user in the last 5 minutes window.
I will be scaling my jmeter test as soon as I can add new dev servers with a loadbalancer and a stand alone db server. I expect to see an increase in concurrent users ( at the moment I get nginx 504 s around 10k jmeter quiz test).

However, jmeter alone is not enough. Should we be looking more at Moodle log data? Any suggestions ?

平均分：Useful (1)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

aykut ozkaya - 2024年07月18日星期四 02:26

I was curious about the scalability but also questioned the actual need for concurrent 100K quiz users. I found Tim's blog very insightful. Especially the peak load of an actual university with around 150 K Moodle enrolments.

https://tjhunt.blogspot.com/2014/04/load-testing-moodle-262-at-ou.html?m=1

Jmeter case study above states 100K page visits an hour is average peak for OU - what kind of setup requires 100K users a second?

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Ben Girard - 2024年07月18日星期四 03:12

I made a typo... In our current custom platform, we reach 50 to 100 users per second, not 100K. Our company manages assessments for 200+ School districts, which is about 600,000 students. When a few districts decide to take an assessment during the same week, we have days where we have 50,000 students logging in for 1 day. If they start an assessment (quiz, assignment) at around 9am (8:55 to 9:05) we can easily reach 50 user per second, thus why I need to work with a highly experienced company that can help us configure a cloud hosted Moodle that will auto-scale up and down to manage such volume for such day.

In other words, we badly need help...

平均分： -

Re: Recommendations to find real professionals with hyper-scale knowledge of Moodle configurations on AWS or Azure and JMETER for load testing?

Visvanath Ratnaweera - 2024年07月18日星期四 03:48

This is something entirely different from the original post. and has no to do with "Pentagon's $10 billion cloud contract". I propose to split the thread at this post and change the title, for example to "Recommendations to find real professionals with hyper-scale knowledge of Moodle configurations on AWS or Azure and JMETER for load testing?"

About who can help, have you read this post?

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

aykut ozkaya - 2024年07月18日星期四 04:02

Thanks for clarifying this Ben
Now it makes sense. I like to help. If the moderators wanting this conversation to move elsewhere thats OK too.
Can you let me know more about your current set up please? You can email me directly if the forum rules are not allowing anymore on this topic

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Tim Hunt - 2024年07月18日星期四 04:20

Note, that presentation is from 9 years ago. Our Moodle is now hosted on AWS. In the AWS docs include a 'Moodle reference architecture' and what we do has a lot of similarity with that. However, as everyone here alwasy says - because it is true - the only way to know how much server resources you need is to set something up and then test with simulated load which is similar to the type or real load you expect.

(With AWS, and other cloud providers, there is also the approach of using the flexibility to scale things up and down. Once your system is running, you can use the live monitoring to see where you are paying for larger capacity that you really need, and scale down to save money. Or, find a bottle-neck and scale it up to save money.)

平均分：Useful (3)

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

aykut ozkaya - 2024年07月18日星期四 04:47

Thanks Tim
I think your post sum this matter up nicely.

平均分： -

Re: Resource comparisons: Pentagon's $10 billion cloud contract [OT]

Visvanath Ratnaweera - 2024年07月19日星期五 15:46

Even the billion clouds need a rest. Read Microsoft Outage Hits Users Worldwide, Leading To Canceled Flights. Their outages cause airlines to cancel flights, our outages leave children without education. Make your choice.

平均分： -