A moodle setup for 10K simultaneous users

A moodle setup for 10K simultaneous users

by Srujan K -
Number of replies: 7
Hi All,

We are planning to conduct our assessments online. We won't use it for maintaining courses or so. Our assessments are just multiple choice questions.

.We will have a peak load of 10k CONCURRENT users for a maximum of 1:30 Hrs.
.We are planning to buy 5 web servers and 2 DB servers.

  • Can Moodle be customized only to conduct assessments and draw statistics out of that.
  • We would also like to export assessment results to MS-EXCEL.
  • What are the best configurations for web servers and DB servers (both HW and SW).
  • We are also thinking of setting up a repeater in every assessment hall. Can we support this kind of setup with Moodle (is it necessary at all?).
regards,
tecgaty.
Average of ratings: -
In reply to Srujan K

Re: A moodle setup for 10K simultaneous users

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Yes Moodle can run quizzes, compute statistics, and export the results in Excel format.


You are talking about extreme loads. Be aware that you would be the first people to try to run Moodle quizzes at this sort of load, so expect problems. They are probably surmountable, but since no-one has tried it, no one can tell you for sure.

Whatever you do, you must perform extensive load testing before putting this in front of students.


Generally, you can expect three peaks in the load, which are when problems are most likely.

1. When everyone logs in.

2. When everyone starts their test.

3. When everyone clicks 'submit all and finish'.

You will find it much easier if you can spread the load out in some way. The worst thing you can do is have someone stand at the font of a big hall and say 'Start now'. That will guarantee that every student hits the start button at exactly the same time, which maximises the chance that the server will crash.


Since you only want to use Moodle to run You can massively improve performance by completely stripping Moodle down to the bare bones.

1. Uninstall all the modules except quiz and forum.
2. Uninstall as many blocks as possible.
3. Ensure all filters are turned off.
4. Do not create any courses. Instead put the quiz on the site front page.
5. Delete all the roles except Administrator and Logged in User.
6. Add the necessary capabilities to the Logged in User Role, so any logged in user can attempt the quiz.
7. Remove all the capabilities from the Admin role except moodle/site:doanything.
8. Remove as many capabilities as possible from the Logged in User role as possible, so they just have the ones needed to see the front page and do the quiz.
9. Remove as many blocks as possible from the front page, if you were not able to uninstall them at step 2. above.
10. Uninstall any question types you are not using. (That is, everything but multiple choice.)

These should dramatically speed up you Moodle site for taking quizzes.

In terms of some key database tables, you are aiming for to get them down to this many rows:

course 1
course_categories 1
course_modules 1 or 2
quiz 1
role 2
role_capabilities ~10
role_assignments 1 (administrator)
context ideally 3 or 4, but this may not be possible.
user ~10000
question (just the ones you are using)
question_categories 1


Once you have done all that, start load-testing. As I say, no-one has tried this before, so you will almost certainly find some potential quick performance wins in the quiz code, if you have a developer available. Post patches in the tracker, and keep talking to me (I am the maintainer of the quiz module) and it will probably be possible to get your patches reviewed and incorporated into the official Moodle distribution.

With a developer on hand, there are other ways of stripping down your site that start to look like nasty hacks, but might be worth it for the extra performance. For example;

11. Eliminate the code that updates the user_last_access table.
12. Change the add_to_log function so that it does not do anything.
13. Since you know that certain tables only have very few rows, change functions that normally load data from those tables to use hard-coded values rather than connecting to the database.
14. If you will not be using the Moodle gradebook, rip out the code that updates the scores stored there.


If you do get a good load-testing set-up, there are some questions that it would be really useful to have answers to:

A. Do my suggestions 1-10 above actually make a significant difference?

B. Which works better, putting all the questions in the quiz on one page, or splitting them up?

C. Does the choice of database engine (MySQL or Postgres) make a significant difference?

D. Which PHP accelerator works best in this case?


So, in summary, this project would be a big step into the unknown. Therefore, it is essential you budget for plenty of load-testing. However, it is a very interesting project to see if Moodle is capable of working at these loads, so you would probably find lots of people in the Moodle community interested in what you are doing, and able to offer advice and help.
Average of ratings: Useful (8)
In reply to Tim Hunt

Re: A moodle setup for 10K simultaneous users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
See that info, Tim, it should be in the docs somewhere smile I'm not sure what the title should be. "Moodle for Mentalists"???
In reply to Tim Hunt

Re: A moodle setup for 10K simultaneous users

by Srujan K -
Hi Tim,

That's an optimistic start. By the by I forgot to mention that we will be using 100 Mbps network. What Hardware and Software configurations will be ideal for my situation? Also is the concept of repeaters necessary?
In reply to Srujan K

Re: A moodle setup for 10K simultaneous users

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I am not an expert on networking. Hopefully someone else will answer that.
In reply to Srujan K

Re: A moodle setup for 10K simultaneous users

by Dan Poltawski -
I would've thought 100Mbps is sufficient, my experience of moodle is that its pretty bandwidth light (unless you are delivering media-rich content).

Remember to configure your webserver to have sensible image expiry headers so each reload isn't requesting theme images continually..

I don't really come across repeaters in amny modern network topologies is it not a switched network?
Average of ratings: Useful (1)
In reply to Srujan K

Re: A moodle setup for 10K simultaneous users

by James Logan -
My prioritized choices for best disk storage reliability and performance options are SAN, RAID, SAS, or SCSI. Stay away from directly attached IDE, ATA, and SATA disks in high load environments. Also, if you're using directly attached disks, keep the operating system and Moodle software on separate spindles (disks).

Network thoughts ...
The optimum performance of a 100 Mbps network is about 70 Mbps, and pushing to higher bandwidth utilization causes serious network degradation. Even if you could insure delivery of 70 Mbps, that comes out to 7K bps (or 700 Bps) per user, which is comparable to very slow dial-up modem speed. Suppose that only 1/8 of your users are competing for that bandwidth at one time, then you're still averaging only 56K bps at best - still dial-up modem speed. Unfortunately ethernet does not throttle down so that the available bandwidth is evenly distributed among workstations, instead each workstation tries to use all the bandwidth it can get - in your case this will very likely exceed the optimum performance threshold and lead to serious network degradation.

Hopefully your 10K users are not all hanging on a single 100 Mbps ethernet wire in your server room - if so, just blindfold your users and require that they type with one finger. Preferably you have multiple ethernet lines that you can funnel through gigabit switches at your server farm. Suppose that you attach a single 1 Gbps 16 port switch to each of the 5 servers, and funnel (roughly) equal numbers of workstations (about 125) to each port of the 5 switches. In this configuration, all of your workstations can simultaneously obtain 350 Kbps from their respective dedicated server, or 1/8 of the workstations can obtain 2.8 Mbps simultaneously. These bandwidth limitations are on the server side of each gigabit switch, not on the workstations side.

Adding more servers increases the average available bandwidth proportionally, and probably the best solution for performance problems. Don't connect multiple servers to the same (shared) network. Connect your Moodle servers to the DB servers (different network) on a separate Network Interface Card (NIC), which means that you need servers with at least dual NICs. Be suspicious of solutions that boast greater than 2 Gbps bandwidth because your servers might not be able to keep up at higher speeds.

Network performance is hard to predict precisely, but easy to deal with if you adopt a flexible network topography. For example, install one server with a single 1 Gbps 32 port network switch. Then connect as many workstations to the switch as you can while observing overall system performance under full load. What you learn will give you enough information to predict how many servers you need, and what size/speed switches you need.

Regards

Average of ratings: Useful (6)
In reply to Srujan K

Re: A moodle setup for 10K simultaneous users

by Elvedin Trnjanin -
There are quite a few excellent suggestions here, but I think you're using the wrong tool for the job. While for Moodle it would be easier to write authentication modules and other customizations, why not go with survey software instead as that doesn't have all the things that slow down Moodle (and make it useful)?

I'd suggest looking at something like "LimeSurvey" as I believe it should meet your requirements and it stores very few things in the database so you don't have to invest in more than one database server.
Average of ratings: Useful (2)