Moodle workload capabilities

Moodle workload capabilities

by Benjamin Kraft -
Number of replies: 7

Hey everyone,

I hope this is the right place to ask these questions. If not, please feel free to move this entry to the appropriate section. I've read the performance FAQ but I'm afraid I lack the technical understanding to answer the questions myself.

I am a student at an german university that heavily relies on moodle, especially for course-registration.

Registration-days are, to put it mildly, absolutely disastrous. Here is how it works:

- There is a moodle-room called "course-registration for semester xy"

- enrollment is done by poll

- Link to the poll for specific courses is unlocked at certain day/time

- registration is staggered like this:

Modul 16     - 22.09.2020 ab 17.00 Uhr
Modul 4       - 22.09.2020 ab 18.00 Uhr
Modul 6-2    - 22.09.2020 ab 19.00 Uhr
Modul 6-1   -  22.09.2020 ab 19.30 Uhr
Modul 7       - 22.09.2020 ab 20.00 Uhr
Modul 8       - 23.09.2020 ab 17.00 Uhr
Modul 11-1   - 23.09.2020 ab 18.00 Uhr
Modul 22    - 23.09.2020 ab 19.00 Uhr
Modul 9      - 24.09.2020 ab 16.00 Uhr
Modul 10    - 24.09.2020 ab 17.00 Uhr
Modul 18    - 24.09.2020 ab 18.00 Uhr
Modul 19    - 24.09.2020 ab 19.00 Uhr
Modul 12    - 25.09.2020 ab 16.00 Uhr
Modul 13    - 25.09.2020 ab 17.00 Uhr
Modul 14    - 25.09.2020 ab 18.00 Uhr
Modul 15    - 25.09.2020 ab 19.00 Uhr
Modul 20    - 25.09.2020 ab 20.00 Uhr

Currently there are ~15k students enrolled across ~ 30 faculties. The above schedule is from the biggest faculty, by far. So worst case scenario (highly unlikely imo) there are 15k simultanious server requests when registration links unlock. [I would generously guesstimate more like 5000 simultanious requests = 15k students divided by 30 faculties, multiplied by 10]. AFAIK moodle is hosted on a server on campus.

Issues: Login to moodle takes forever. Server can't handle the load when links unlock (constant 503) -so everybody refreshes until they "get a spot". Which can take between 15 seconds and 5 minutes, by that time the course you've tried to register for is likely already full. Or you get an "enrolled" message, but are not actually enrolled when checking the poll results...(I'm doing this from a 100 Mbit cable line with a 8ms Ping and Jitter 1 to DEAC, but you run into the same issues if you're logged in to the campus network, during registration you can't even ping the university server, 100% packet loss). So course registration becomes complete luck...which is very frustrating to say the least.

Now the university is blaming the students for this, claiming that people who log in with multiple devices  "clog up the server"... And they claim that there is nothing they can do to change the contstant server overload on registration days.

Questions:

Am I right in assuming this is an rediculous excuse? Isn't it more likely that the university is too cheap to provide the appropiate ressources needed to handle this load? Or is moodle badly implemented/administrated on the server? Is the Poll-feature even the appropiate way to do course-registration via moodle?

Is this an issue with scaling in the moodle software or an issue in providing the appropriate ressources?

What would be the requirements to run moodle smoothly for this task?

Is this even possible on an on-campus server?

Would it be an easy/cheap solution to host moodle on e.g. AWS compared to running it on an on-campus server?

The reason I came here with these questions is that I would like an expert opinion I can point to - and who better to ask than the developer community itself. I find it highly offensive that the university is blaming the students for this, and is claiming they can't do anything about it (this has been an ongoing issue for years now).

I'm by no means an expert in these matters, but I think I know enough to ask at least reasonable questions. Thanks in advance for anybody who is willing to help smile


tl;dr University is blaming Students for constant 503 on course-registration days; says theres nothing they can do about it. Is this an excuse to hide unwillingness to pay for ressources / incompetence?

Average of ratings: -
In reply to Benjamin Kraft

Re: Moodle workload capabilities

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
This is a shame.
> the university is blaming the students for this
And 15 k students learning to hate Moodle?

Even if it is Moodle's fault, the university must not take a system on-line which can not handle the load! The students' behaviour is to be expected. I can't imagine this happening in Germany of all places. It is not a digital third world and indeed there is a very active an competent German speaking Moodle eco. See Deutsche Moodler.

Now to the details:
- You said, "Login to moodle takes forever." What is the authentication system of the Moodle site? Usually it is some variation of LDAP. But there are many. See https://docs.moodle.org/en/Authentication.

- Could you explain the registration procedure in more detail?
> - There is a moodle-room called "course-registration for semester xy"
This is a Moodle course, I believe.

> - enrollment is done by poll
What is the exact Moodle activity used?

> - Link to the poll for specific courses is unlocked at certain day/time
So there must be a link (the Moodle activity above) per course, which are switched on and off. Is that right.

> Or you get an "enrolled" message, but are not actually enrolled when checking the poll results...
Sounds like a race condition, due to a delay between accepting the registration command and actual registration. You need to know the intricacies of the work-flow, which only the Moodle administrator knows, I am afraid.

- The infrastructure needed
> What would be the requirements to run moodle smoothly for this task?
Not a sensible question. By "this task" you mean keeping the current system as it is, right? Firstly, before throwing resources, one needs to scrutinize whether the plan is bad. Secondly, how much resources (hardware, infrastructure, man power) needed for a give "size" of a Moodle site has no answer. See https://docs.moodle.org/en/Performance and https://docs.moodle.org/en/Performance_FAQ. In fact, there is a whole forum dedicated to the subject of Hardware and performance?.

- Some random answers to your other questions:
> Am I right in assuming this is an ridiculous excuse?
Ridiculous or not, no excuses!

> Isn't it more likely that the university is too cheap to provide the appropriate ressources needed to handle this load? Or is moodle badly implemented/administrated on the server?
It is the same question. Whether university saved money or its money was wasted, we don't know.

> Is the Poll-feature even the appropriate way to do course-registration via moodle? Is this an issue with scaling in the moodle software or an issue in providing the appropriate resources?
Need to know about the "poll-feature". See point 2 above.

> What would be the requirements to run moodle smoothly for this task?

No answer, as explained above.

> Is this even possible on an on-campus server?

What are the alternatives?

> Would it be an easy/cheap solution to host moodle on e.g. AWS compared to running it on an on-campus server?

I an no fan of the "Cloud". From what I hear, this pay as you use feature can burn the wallet, if you tap the power of a dedicated server from the cloud. In addition, if the authentication is the bottle neck, the remote authentication will make it worse.

You need to revisit the final set of questions after studying the first three points.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moodle workload capabilities

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers
"Would it be an easy/cheap solution to host moodle on e.g. AWS compared to running it on an on-campus server?"
Easy/cheap - no, not necessarily. But AWS would have the distinct advantage that a properly/well-configured infrastructure could be scaled up and tuned for the registration period and then scaled back down for normal usage, so yes potentially costly while scaled up, but short term-temporary costs. You also need to balance that cost against reputational damage of a service not functioning on your on-premises hosting and the damage that does to the students' use of Moodle from day 1.
Average of ratings: -
In reply to Benjamin Kraft

Re: Moodle workload capabilities

by Colin Fraser -
Picture of Documentation writers Picture of Testers
What is "enrollment done by poll"? This course has polled 65% of approval from students so only 65% of applications to take this course are accepted? "MoodleRooms"? MoodleRooms is a separate service provider that uses Moodle at its base, and I suspect they have been modifying it (read making a mess of it) which is why it's not working. If using MoodleRooms, of course. That aside, your use of terminology is, for me, confusing, but then I am easily confused.
Assuming your University is self hosting, Richard's comments are most enlightening. Moodle is an LMS, not an SMS, so if the University is trying to use it for that purpose, then it's like using a chisel for a screwdriver. Not going to work well.
Also, bear in mind that at 15,000 students, that's an awful lot of server demand in just three hours. The University is setting itself up here, an obsession with time that is working against you. If you have 2000 students wanting to enroll in 1 course at 1700hrs, then at 1700hrs, a lot of them are just going to try at the same time, the "get in first" or "first in, best dressed" mentality. Not a clever use of server time, I suggest. I would recommend that you don't try and regiment enrollments, obviously not working. Think about a different process, like spreading enrollments out over a longer time frame, but you will still get heavy loads at opening times. Change the way in which enrollments are made. Using an LDAP, as Visvanath suggests, has always been a winner for me.
Average of ratings: -
In reply to Benjamin Kraft

Re: Moodle workload capabilities

by Benjamin Kraft -
Thanks for your answers so far! And sorry for the delay on my side. Also sorry for confusing terminology/language - I will try to use the accurate english terms, where I simply translated the german terms before (pls keep in mind that I'm not a native english speaker).

I will try to answer your questions as far as I can:

>What is the authentication system of the Moodle site?

It is "Email-based self-registration". So pretty much anyone can create an account. But I think you need a valid University-eMail-account to gain acces to all functions.

>Could you explain the registration procedure in more detail?
>What is "enrollment done by poll"? / What is the exact Moodle activity used?

I'll try.
- Between semesters the university publishes a class-directory for the next semester
- Every Module is offered by multiple professors, on different schedules
e.g. the module "sociology 102" is offered by prof x on mondays 8-10am
                                                                                 prof y on tuesdays 10-12am
                                                                                 prof z on fridays 5-7pm
- there is an unrestricted moodle-course called "module (class) enrollment for semester 2020/2021"
- this moodle course lists all modules for the upcoming semester
- under every module is a link to an activity that starts at a certain date/time (see OP schedule for example)





- this is the "choice" activity. This is how you choose which class/prof of a given module you want to enroll in
- once the link unlocks it looks like this:

Your choice for Module 102
+ Prof X / coursenumber 102-123
+ Prof Y / coursenumber 102-435
+ Prof Z / coursenumber 102-678
+ Prof B / coursenumber 102-900





- You "vote" for the class/prof you want
- class-size is restricted, so if a class has 24 spots, once 24 ppl "voted" for it, it is "full" and you can't choose it anymore




- If your "vote" was succesfull you are "enrolled" in this class
- Prof will get a list of students in his class, only these students have the right to partake in the exams for this class
- registration for exams however, is done on a completely seperate service outside of moodle on a different date

The 503-Issues on registration-days occur when trying to log-in to your moodle account and when you try to use the "choice" activity, once the links to the activity are unlocked



Average of ratings: -
In reply to Benjamin Kraft

Re: Moodle workload capabilities

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
You are welcome! The "delay" is no problem for me, it is your problem after all. Or, isn't it your University's problem? How come you are working on it? Are you a student interested in a project?

Now to the situation:
1. Login process
I wanted to know whether an external component like an authentication server is your bottleneck. You confirmed that Moodle uses its internal authentication, so the answer negative. The logins are simply blocked because the registration process overloads the server.

Still curious: Why don't the clever students just log in an get ready before the appointed hour?

2. The registration process
Now I understand better.
- There is a Moodle course to which the students can register freely.

- In that there is a series of Choice activities.

- Once the student enters his choice, he is taken to either a Feedback activity or to a Questionnaire.

- Once he answers that, the registration for a course is done.

- The student repeats the process for other courses he wants to register.

I can imagine these activities, specially the Choice, were never meant to be scaled to this level. You need to do some reverse engineering, for example follow SQL queries being generated, and follow the code looking for optimizations. That is at the application level. The usual system level optimizations, like monitoring the server load, finding the bottleneck, releave either by system tuning or just adding more resources, etc. they are always valid. See the documentation to the forum Hardware and performance. Also read as many related discussions as possible.

Obviously you need access to the Shell and to the database. Without those, we can only speculate.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moodle workload capabilities

by Benjamin Kraft -
Dear Mr.Ratnaweera
Thanks for taking the time answering my questions, it is really appreciated lächelnd

> isn't it your University's problem? How come you are working on it?

Well, I'm not working on it, and it is indeed the university's problem. The reason I'm seeking information on it is an eMail the university administration sent out to all students a couple of weeks ago:

"[...]today we got numerous complaints that it was impossible to enroll in classes via moodle - if a lot of students are using 4 devices for enrolling it is no surprise that the system is overloaded...
We know that the enrollment-system leaves much to be desired and we are sorry for that. However, there is very little we can do to fundamentally change the situation[...]"

This deeply offended me in two ways :
1) blaming students for bad infrastructure - if the system would run properly, no one would have to try things like using multiple devices. Also, I highly doubt that "a lot" of students are doing this - and even if - it should not make any difference if the server is already unable to handle the load (And how do they know about this? If 4 students live in a shared appartement, which is very common, of course there are 4 devices connecting from the same ip...)

2) The "there is nothing we can do about it"-attitude. For me, this translates to "we know it's bad, but we are unwilling to pay for the infrastructure upgrades necessary or the man-hours needed to implement the system properly. So you'll have to deal with it. Sucks to be you."

The limited knowledge I have in this area stems from running a dedicated gaming server in the late 90s ;) and hosting an eMail-server for family+friends. But the limited knowledge I have is enough to make me suspicious about the administrations excuses.
As I mentioned before, this has been an ongoing issue for years (at least the 2.5 years I've been enrolled to the university).

The reason I'm trying to gather detailed information on what might be the actual problem(s) and possible soulutions is that I'm considering to write an open letter to the administration - in hope of creating enough pressure from the student body that forces the administration to act / supplying the IT-faculty with the appropriate ressources.

Statements made in this thread like
- "Even if it is Moodle's fault, the university must not take a system on-line which can not handle the load! The students' behaviour is to be expected."
-"I can imagine these activities, specially the Choice, were never meant to be scaled to this level."
-"AWS would have the distinct advantage that a properly/well-configured infrastructure could be scaled up and tuned for the registration period and then scaled back down for normal usage"

can help me to formulate a fact-based refutation of the administrations excuses. Also the hardware and performance forum you pointed me to has a lot of similar threats, including solutions that show that something can be done - provided the time and ressources are given to the IT-staff.
I imagine the IT-faculty knows exactly what they need and asked for it, but are afraid to publicly speak out about the universitys unwillingness to provide.
Average of ratings: -
In reply to Benjamin Kraft

Re: Moodle workload capabilities

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Benjamin (Die Community ist per du. wink )

Now I understand the situation. But I must say, it is only remotely related to Moodle. The performance part may belong to Hardware and performance forum provided that the Moodle activities were chosen rightly. I would start at the design stage. For that there is the General help forum. Whichever, I don't know what the General developer forum can contribute.

Talking of the contribution of MoodIle, I am a private individual in a (loose) community - no official connection to Moodle HQ. So all personal opinion here!

Anyway, although I understand the situation, I agree with you only partially. At an administrative level the University has apologized, "the enrollment-system leaves much to be desired and we are sorry for that". So far good. Then they continued, "However, there is very little we can do to fundamentally change the situation". A lame excuse. Is it technically impossible? (Why? Was the design bad?) Or, are they incapable? Or, no money, no resources? Or, money spent on wrong things? Or, no priority? Depending on the answer you might accept the situation or not. But blaming it on students, "if a lot of students are using 4 devices for enrolling it is no surprise that the system is overloaded". that was very bad. In PSTN the engineers knew that the subscriber will dial again if the line is busy. They had stochastic models to take that in to account a century ago during WW I. Why the IT dept. of a major German university doesn't know that today, I can't think of any reasons.

At the technical level, I won't jump so far as to say "AWS would have the distinct advantage that a properly/well-configured infrastructure could be scaled up and tuned for the registration period". a) I don't know whether the design (the registration course and the chosen Moodle activities) was bad. Even if it was, b) AWS is the solution. (Note that I am talking in first person. I don't have experience in AWS.)

My previous statement, "I can imagine these activities, specially the Choice, were never meant to be scaled to this level", is a just a possibility. We benchmark the quiz activity all the time, but not Choice, Feedback, Group choice, etc., at least not in the Hardware and performance forum. The Testing and QA people might.

In any case, I won't invest time at the technical level, unless I have full access, to the design and to the server.
Average of ratings: -