Hardware requirements to support 1000 simultaneous quiz takers?

Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -
Number of replies: 19

Hi all,

We are in the process of setting up a new moodle installation to support a minimum of 1000 concurrent test takers who will be taking an entrance test into our University. The quiz will be text-based only and will contain 50 multiple choice questions, one question per page. We have previously handled 100 test takers on a 2.4GHz Core2Duo HP Compaq Desktop Machine with 4GB RAM running Ubuntu Server 10.04 (Apache, MySql, PHP with eaccelerator).

We are ready to buy new server(s) but not quite sure on the right specs to adequately support the project without issues. We will be using the installation only for moodle quiz at the moment.

Is it preferable to use one single high-end server for both Web and Database? If not, what are the advantages of separating the web server and the database server? And what hardware specs would be adequate for both scenarios, i.e, single server and dual servers?

Thanks in anticipation of your expert contributions.

 

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Are you going to lock all 1000 people in a room, and make them all do it at exactly the same time, or will this quiz be available for about a week, and people can do it any time?

If the former, give up now, if the latter, you should not have any problems.

Average of ratings: Useful (2)
In reply to Tim Hunt

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -

All 1000 persons will be acessing the quiz at the same time, from different rooms though. Is this not achievable?

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by ben reynolds -

I'm fascinated by the logistics of 1,000 users and 1,000 computers and simultaneous test taking. I can't imagine how you organize such a thing.

Tim, didn't I see a comment from you last week that quiz runs very slowly with about 100 simultaneous users?

Average of ratings: -
In reply to ben reynolds

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -

We've got 5 computer labs with each equipped with 200 laptops. Logistics is not a problem for us, rather, optimal server(s) specs is/are.

Any suggestions pls? We need to make the purchase ASAP.

Average of ratings: -
In reply to ben reynolds

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
To be honest, I have not hard data. I just know that the quiz is quite complex, and so puts a lot of load on the server when you have concurrent users. And I have seen lots of posts from people who tried to get too many people sitting a quiz at the same time and had their server fall over. And, if you are using MySQL (MyISAM engine) when your server falls over, it corrupts your database. Nice. However, even staggering the start times by 10 or 15 minutes, so the users are not all submitting absolutely at the same time seems to help a lot. Anyway, this is a popular topic in these forums. I suggest you search through the old threads.
Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -

Thanks guys for your responses so far.

However, I still really want to know the (dis)advantages (if any) of having the web server and database on same (or separate machines). I have read through the perfomance docs/threads and they don't say why.

 

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

The advantage of having everything on one server is that the network is not involved when data is loaded from the database, so database access is quicker. So on a system with a small number of users, everything on one server will be fastest. The disadvantage is that you are limited by the amount of hardware you can get in one machine, and if that one machine crashes, your Moodle site is down.

The advantage of multiple servers is that you can have much more hardware (or certainly more hardware for less money). You can then have lots of load-balanced web servers. That scales much better with lots of simultaneous users. The disadvantage is that each database access then incurs network latency, so for single users, things are a bit slower. Also if one web server falls over, the others can keep your site up. With the database, you can use master-slave replication with fail-over, if you want to remove the DB server as a single point of failure.

Average of ratings: Useful (4)
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Robert Brenstein -
When you say you want to run quiz, do you actually mean using the quiz module or some other module, like feedback, which possibly loads the server less?

Also, do you need to have all data in the same database? I mean that if this is an entrance exam, you could actually have a farm of independent servers, let's say 2-3, even 4, per room, and after the test, your staff can export the data from all servers and combine as needed.
Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Ixbalanque Millan -

Check this:

http://docs.moodle.org/en/Performance#Web_server_performance

Best regards

Average of ratings: -
In reply to Ixbalanque Millan

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -

Thanks but I already read that. Any suggestions on ideal server specs?

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Ixbalanque Millan -

Having to 1000 users concurrently connected, I think it's very difficult to achieve with a sample of 1000 laptops, because you need to have 1000 processes running  in a single second; i. e., the processes running both parallel or simultaneously in your server. I think that you have to check the real concurrency that exist in your server, and you should evaluate your server resources.

You can also use Webserver Stress Tools to simulate the behavior of 1000 users or more in your server.

The hardware requirements say you need 1G of RAM to support 50 concurrent users:

http://docs.moodle.org/en/Installing_Moodle#Requirements

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

I am sure that nobody knows the answer to this type of question. My recent script (look for 'e-peen' thread title) was an attempt to help give approximate answers, but nobody much has actually run it yet so we don't have a dataset of how different systems score, nor any confirmation that the script corresponds in any way to real usage. Maybe that script is no good either...

If you have already done this test with 100 people you should be able to work out the number of Moodle php requests per second that this caused. This is the most useful concurrency figure (not number of users - requests per second, and what type of requests they are). Once you have that number, if it was obtained from tests in a similar environment, you can hopefully just multiply by 10 to get your target.

Using this information, you might be able to create a script in a load testing system such as JMeter which makes that many requests to a quiz page of a similar kind. You could run this script against your current server, and any proposed system, to see what maximum estimated requests per second the server can handle.

I don't think there is any real way to evaluate hardware ahead of time because no data exists. At the Open University we serve a lot of requests and (leaving aside things which are only in place for reliability and don't help peformance) we do it with a single fast Postgres database server (iirc it has 16 cores, 64GB RAM), four reasonably fast webservers, and an NFS server linked to our SAN for dataroot [and sessions which is a performance/reliability issue].

With 1,000 people simultaneously taking a quiz, you are probably around or ahead of OU peak load. We typically see about 1,000 different users in a 5 minute period (I looked right now, last 5 minutes it was 964 users) but many of those users are probably just dropping in to look at a course page and see if there's anything new, or they loaded a long forum page and are reading through it - whereas answering a single multiple choice question could potentially be much quicker, resulting in more frequent requests...

--sam

Average of ratings: Useful (1)
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I'll hazard a guess but just to give you an idea of the scale of the problem... From my experience/observations I would have thought that you were doing very well to run 100 concurrent (whatever that means) quiz sessions on that hardware. It's impossible to know how your users were using Moodle (e.g. is it a "quick fire" quiz or do they have to think for 5 minutes between answering each question - big difference). However, let's say that 100 users is the near the maximum achievable for that hardware.

In which case you need 10 of those servers and a load balancer, plus something very hefty to run the database AND I'm really not sure how to spec that without trying it.

I think what everybody is trying to tell you is that this is a bad plan and it would be a lot better if you could modify your plans. Is there no way of staggering your quiz into 10 sessions of 100 (which you already know will work)?
Average of ratings: Useful (2)
In reply to Howard Miller

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -

Hmmn ... kinda looks like we're trying to embark on some mission impossible with this quiz  thoughtful.

Problem is we will be having about 9000 candidates in all, and we have only 3 days to conduct the quiz. We've decided to handle 3000 per day, spread across 5 sessions (with each session lasting an hour). That means we now need to handle only 600 concurrent quiz takers.

We have acquired two servers:

Server One: HP Proliant DL380 G6

  • Quad Core Intel Xeon CPU E5506 @ 2.13 GHz
  • 2 x 146GB 10k RPM SAS drives
  • 18Gb Registered ECC RAM
  • Gigabit NICs

Server Two: HP Proliant ML350 G5

  • Dual Core Intel Xeon CPU 5120 @ 1.86 GHz
  • 2 x 146GB 10k RPM SAS drives
  • 16Gb Registered ECC RAM
  • Gigabit NICs

I need your suggestions on which to use as web server or database server. I'll also appreciate your suggestions on what we might need to upgrade on each of them and other things we need to watch out for, in order to have a successful quiz.

Thanks!

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
So you literally have 600 machines all connected to your network and you can organise 600 people doing this at the same time? Are you sure your network infrastructure can handle this, never mind Moodle.

I have no idea exactly what it would take to do this but I do know you are a million miles away with the hardware you have.

Does this all have to be done on the *same* Moodle site. Could you duplicate your Moodle over several installations each sitting on a subnet reasonably local to the quiz takers? This would lighten up the Moodle load and the network load.
Average of ratings: -
In reply to Howard Miller

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Chris Akus -

Yes we have over 600 machines all connected on the network and we can for sure organize 600 persons to take the quiz at the same time (remember we need to test 9000 in 3 days).

When you say I am a million miles away with the hardware I have, how exactly do you mean? Are the server specs too low? What do we need to increase?

We've been thing about duplicating the moodle instance and splitting the users accross both instances but with the limited hardware (only 2 servers), that would imply we'll have to do web/database on a single machine for each instance. Is this a good idea? And should we follow this path, is there a script or quicker way of doing this the duplication?

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Eric Hagley -

So let's put this together.

You have 5 labs, you want 600 students to do the quiz in each "batch". That is 120 students in each lab.

Is it absolutely imperative that they all start at "exactly" the same time?

For example, could you "organize" for a rather long password to be hand carried to one lab, then after it was written on the board / posted, it be carried to the next lab etc so that there was a 3 to 5 minute lag between the start of each lab? If you add the fact that the long password has to be typed in you might be looking at 10 to 20 test takers hitting the "start quiz now" button within the same 1 to 3 second window. (My understanding of what "concurrent" users means.) More manageable I would think.

How are the users logging in? Have they all been given separate user names and passwords? The log in will also be another time of stress on the server I would imagine...

Using the server outlined in this post I have carried out a quiz with 100 students all accessing at around the same time - probably a 5 minute window, and all finishing (I had the timer set) at around the same time too. Though 5 students received an error message when their time ran out, their scores and answers were recorded. That quiz contained audio files as well!

If you can set it up so that the actual logging in and clicking the "start quiz" button is staggered you may well be able to get away with the hardware you have - it is much better than what I was using! Not sure if that is feasible though...

Good Luck and please let us know how you got on.

Eric

Average of ratings: -
In reply to Chris Akus

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Robert Brenstein -
Since you said you have 5 computer labs, get 5 servers, one for each lab, to run the test. Isolating each lab from campus network and internet will have additional benefits for both supervising the test and overloading campus net. With 600 test takers in 5 labs, you will have manageable load of 120 users per server. Surely, with 9000 takers, you have a budget to buy required hardware. You could even consider buying more affordable hardware by putting 2 servers in each computer lab, thus reducing load to 60 users per server. Installing 5 or 10 Moodle instances is trivial and not that much work and quizes can be loaded thru export/import. Combining databases will require rekeying userids but you probably just need scores and those can be extracted programmatically from each database and then reports combined.
Average of ratings: Useful (2)
In reply to Howard Miller

Re: Hardware requirements to support 1000 simultaneous quiz takers?

by Darko Miletić -

Your case seems like ideal for implementing set of HA Load balanced web servers tied with HA load balanced mysql servers. Where you have one write server and x replicated read servers.

This would be expensive and complicated setup.

What did you do in the end?

Average of ratings: Useful (1)