Hardware and performance

Login 1000 student at a time on one quiz

 
Picture of Lalit Thakur
Login 1000 student at a time on one quiz
 

I am running a moodle 1.9 for 1500 student i have a problem that 1000 student are not give the exam at a time . all student can login and also open a quiz but when they click on next button for next question that it take more time about 2 to 3 min. how to resolve this problem.

I have :
ubuntu 14.4 server OS
apache2
mysql
php

And System Config:
100 GB RAM
80 GB HD
every thing is higher level

my.cnf and apache2.config both fiel are atteched

(Edited by Howard Miller to remove contact details - original submission Tuesday, 14 July 2015, 7:04 AM)

 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

100GB RAM? is that correct?

You should stagger the quiz. You have almost zero chance of this working if it is genuinely 1000 student simultaneously. 

If I really wanted to do that, I would be thinking about a lot more hardware. I have a system with four load-balanced front-ends all with 32GB of RAM and we worry when quizzes exceed 100 simultaneous users. I just can't imagine 1000 on a single front-end having any chance of working. If the database and the web server are the same machine then you really are in trouble wink

However, I *hope* you have been monitoring the system when performing the quiz so you can identify where the pinch-point is?

 
Average of ratings: Useful (2)
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 

Thanks For quick reply Mr. Howard Miller


Really i feel that problem so pls suggest me what should i do for 1000 user simultaneously.

what should be my RAM size ??

How to change my front and and database both are diffrent systems and which system have more RAM sizes.


 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

To be perfectly honest with you... if you are asking these sort of questions you probably need more help than can easily be provided in these forums. 

You need to analyse your systems to see what is consuming (or more likely running out of) resources at these times of high load. My guess would be that this is going to need a clustered system of some sort - several web servers. I would be *very* worried if I was given this requirement. I would also strongly recommend that you simply don't try to start a quiz with 1000 people at the same time. Can you not stagger the start at all?

Other than that, I would contact a Moodle Partner and purchase some consultancy, as this isn't simple at all. 

 
Average of ratings: -
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 

Thanks For Reply MR. Howard Miller


Ok no problem,  but can i provide the quiz for 500 student simultaneously . if yes,

Then what is my RAM Size,

ans Also tell me system configuration.

??????is it need for database on different system.

????


 
Average of ratings: -
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 

Also tell me Howmany Student attempt the quiz at a time if My RAM size is 100 GB

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers
These are not simple questions. There is some documentation about these topics in the initial information of this (Hardware and performance) forum. Start a new discussion then you'll see them.
 
Average of ratings: Useful (1)
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 

Thanks For Reply MR. Visvanath Ratnaweera


Ok no problem,  but can i provide the quiz for 500 student simultaneously . if yes,

Then what is my RAM Size,

ans Also tell me system configuration.

??????is it need for database on different system.

????

 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

I'm really sorry... but it doesn't work like that.

It takes running pilots, carefully monitoring the system and making some informed decisions based on that.

We can't tell you the answers you want because we are not in your server room and we are not analysing your system. What you are trying to do is very difficult and - I'm sorry - you sound like you are completely out of your depth. 

You need to bring somebody in who understands and has experience of very big Moodle systems. I could take a guess what you might need but it is way more hardware than you have. How much is your budget for this?

 
Average of ratings: -
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 
Sir,

acully First Understand my problem , and please again read my questions , if you can reply the ans of that question then send me appropriate reply.

Thank you
 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

...please read (again) my answers wink

I haven't got anything else to add. Anyone else want to have a go?

 
Average of ratings: Useful (1)
Picture of Frankle Lee 李智高
回复: Re: Login 1000 student at a time on one quiz
 

Hi, Howard.

I am a Chinese moodle user, I have been using moodle from 1.7 and now I am using moodle 2.6.

I work in a college with nearly 10,000 students, and I organized a lot of final exams on moodle these years.

I use a single ThinkServer RD630, 2CPUs, 32G RAM, SAS Raid5, and it can supoort  nearly 600 students to take a quiz at the sametime from our computer room. It seems to me 1000 students is not so hard if you have a more powerful server with more ram, so I think 100G ram is more than enough.

My 600 student quiz is that 600 student in 6 computer rooms and login and start the quiz at the same time in 3 to 5 minutes, random questions, 10 questions per page, mostly are multiple choice, or open questions.

Are we talking about the same thing? Or, 1000 student taking quiz simultaneously is not the same idea with my situation?

 
Average of ratings: Useful (5)
Tim at Lone Pine Koala Sanctuary
Re: 回复: Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Thank you for sharing that data. It is a really useful indication of how much you can scale up use of the Moodle quiz.

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: 回复: Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers
Hi Frankle Lee 李智高

Could you provide a bit more detail?

You say:

> I have been using moodle from 1.7 and now I am using moodle 2.6.

Did you do the same kind of exams with the same load from the beginning? Since 1.9 was less resource hungry than 2.x, In your judgement, would your current server handle 1000 users on Moodle 1.9? I ask because the OP uses Moodle 1.9.

> It seems to me 1000 students is not so hard if you have a more powerful server with more ram, ...

Have you data of RAM usage during those peaks? All load graphs would be intersting.

> ... 600 student in 6 computer rooms and login and start the quiz at the same time in 3 to 5 minutes, ...

Do you mean, within a period of 3 to 5 minutes all 600 students have opened the first page of the exam?
 
Average of ratings: -
Picture of Frankle Lee 李智高
回复: Re: 回复: Re: Login 1000 student at a time on one quiz
 

Hi, here is my answer:
I conduct all my 600-user-final exams on moodle 2.6 not 1.9

It seems to me that 50MB ram for every user is quit true to my situation, so I guess more than 60Gb ram should be OK for 1000 users. And of course, it also depends on the CPU, Cache, and harddisk speed. More Powerful CPU, faster Cache and SSD will help to reduce the ram requirement.

The last question, all my 600 students will login and click start quiz then load the first page of questions within a period of 3-5 minutes.

The loading time at the beginning will last longer, sometimes over 30 seconds, but once the quiz has began, the loading time for the next page will be much faster because of the clicking time for next page would be various for different users.

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers
Hi Frankle Lee 李智高

Thanks for providing those information.

> I conduct all my 600-user-final exams on moodle 2.6 not 1.9

I knew that. The point i wanted to bring was that your server might handle close to 1000 users taking the same exam if you were running Moodle 1.9. Note that this discussion is about Moodle 1.9.

> It seems to me that 50MB ram for every user is quit true to my situation, so I guess more than 60Gb ram should be OK for 1000 users.

Your argument is that, since your machine handle close to 600 users with 32 GB, raising it to 64 GB it might reach 1000 users. Possible, if the RAM was and will remain the bottle neck. That is why I asked about monitoring.

> And of course, it also depends on the CPU, Cache, and harddisk speed. More Powerful CPU, faster Cache and SSD will help to reduce the ram requirement.

Generally true. But you will notice the difference only if you help the bottle neck.

> The last question, all my 600 students will login and click start quiz then load the first page of questions within a period of 3-5 minutes.
>
> The loading time at the beginning will last longer, sometimes over 30 seconds,

Your machine is very powerful, I must say. Still 30 sec can make an examination candidate nervous. There were various tricks discussed in this forum to flatten the initial peak, like delaying each computer room by 5 min, and/or assigning the exam a very long password, ...

> but once the quiz has began, the loading time for the next page will be much faster because of the clicking time for next page would be various for different users.

Yes, that is to be expected. Also because the various caches continue "learning".

All that said, you are trying to put a cap on the number of users:
Is there a way to limit the number of online users for a moodle site?
https://moodle.org/mod/forum/discuss.php?d=317711
Is it related in any way?
 
Average of ratings: Useful (2)
Picture of Frankle Lee 李智高
回复: Re: Login 1000 student at a time on one quiz
 

> I conduct all my 600-user-final exams on moodle 2.6 not 1.9

I knew that. The point i wanted to bring was that your server might handle close to 1000 users taking the same exam if you were running Moodle 1.9. Note that this discussion is about Moodle 1.9.

> It seems to me that 50MB ram for every user is quit true to my situation, so I guess more than 60Gb ram should be OK for 1000 users. 

I read this 50mb per user many years ago, it looks like from the documentation of 1.9, but still true to me right now, so I always keep that in mind.

Your argument is that, since your machine handle close to 600 users with 32 GB, raising it to 64 GB it might reach 1000 users. Possible, if the RAM was and will remain the bottle neck. That is why I asked about monitoring.

I will try to monitor the bottle neck next time, but usually a 600 user exam only happen in mid term or finals, so need to wait for next semester. Now is the summer vacation, YEAH!!!

 

> And of course, it also depends on the CPU, Cache, and harddisk speed. More Powerful CPU, faster Cache and SSD will help to reduce the ram requirement.

Generally true. But you will notice the difference only if you help the bottle neck.

> The last question, all my 600 students will login and click start quiz then load the first page of questions within a period of 3-5 minutes.

> The loading time at the beginning will last longer, sometimes over 30 seconds, 

Your machine is very powerful, I must say. Still 30 sec can make an examination candidate nervous. There were various tricks discussed in this forum to flatten the initial peak, like delaying each computer room by 5 min, and/or assigning the exam a very long password, ...

Thanks for your tricks, I will try it later. the long password is a Brilliant tricks!

My server is not so powerful, it works for my for over 2 years. And it's very cheap, about USD3000.00, now even much cheaper I am sure.

> but once the quiz has began, the loading time for the next page will be much faster because of the clicking time for next page would be various for different users.

Yes, that is to be expected. Also because the various caches continue "learning".

All that said, you are trying to put a cap on the number of users:
Is there a way to limit the number of online users for a moodle site?
https://moodle.org/mod/forum/discuss.php?d=317711
Is it related in any way?

Actually I am working on a project to invite teachers from other school to use our platform to build and share courses with us. In this case, I prefer more control of the user numbers from each school.

 
Average of ratings: -
Picture of Howard Miller
Re: 回复: Re: 回复: Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

I might add that I am entirely guilty of the "50MB per user" figure. It was worked out on the basis of watching some monitoring over a period of about 10 minutes on one of my live systems. It was meant to provide an "order of magnitude" figure rather than a hard rule. I suspect it is a bit on the high side. However, some teacher/admin activities can consume a LOT more memory so I still think it's about right. 

As has been said, if performance is poor there is more to go wrong than not having enough ram. It is easy enough to see if a system is going into swap and it is easy enough to check the load figures. However, just guessing (what we get asked to do all the time in this forum) is not really going to help anyone. 

 
Average of ratings: Useful (1)
Picture of Matteo Scaramuccia
Re: Login 1000 student at a time on one quiz
Core developersParticularly helpful MoodlersPlugin developers

Hi All,
here are some refs about :

HTH,
Matteo

 
Average of ratings: Useful (1)
Picture of Marcus Green
Re: Login 1000 student at a time on one quiz
Core developersParticularly helpful MoodlersPlugin developersTesters

Quote "You are going to need a bigger boat"

If you want to have 1000 students take an exam on Moodle at the same time you are going to have to do a great deal of testing and piloting. This is way beyond what can be helpfully advised in a forum like this, nobody can give you exact figures.  If all these students are taking the exam at the same time I would assume it is important and so you cannot afford to have failures.  You will need to test with software like Apache JMeter but you will need to also pilot with real students.

You will need to identify where the bottle necks are, for example is it writing to disk, is it networking is it memory.  Where will the students be located, are they all on the same network?  One of the worlds largest providers of distance learning is the UK Open University and they don't have anything like 1,000 students taking an exam at the same time.

What you really need is advice from people who are expert at large deployments of Moodle. This company from  New Zealand are worth considering.

http://catalyst.net.nz/news/2-million-users-new-training-platform


 
Average of ratings: Useful (1)
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 

Thanks for the reply Mr. Marcus Green

Can You Tell me That Howmany student attend a quiz at a time on 100 GB RAM

 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

With moderator hat firmly on...

No he can't.   There's no point asking again and again for a simple answer to your problem when there is no simple answer. 

I'm sorry if you have got in too deep with this project but this is what happens when you either don't get the full requirements or you try to build something without sufficient experience and knowledge. 

If you had asked before starting this project we would have told you how hard it would be. Maybe you wouldn't be in this situation now. 

 
Average of ratings: -
Picture of Lalit Thakur
Re: Login 1000 student at a time on one quiz
 

All Student located on a server and database and front hand both are on a single system

 
Average of ratings: -
Picture of Marcus Green
Re: Login 1000 student at a time on one quiz
Core developersParticularly helpful MoodlersPlugin developersTesters

I do't know,  I don't have any background in high numbers of students on high stakes quizzing. The main experience I have is with cheap hosting (e.g. $US70 a year) and getting 18 students to take an exam, which worked OK, but that was low stakes testing. I would guess (and it is a guess) that you will get tens rather than hundreds of students with the configuration you mentioned but I could be entirely wrong.

I do have a decade of monitoring questions like yours and there are no simple answers. There are things you should be wary of such as using Networked Attached Storage or NFS mounting, you should look very closely at caching such as memcache/mongodb.  You should consider SSD storage (solid state drives) as they can be much faster than magnetic media.

Be aware that performance issues can come from unlikely places. For example I know of a site that got a big improvement in performance by removing the technology that had been put in place to increase performance (go figure). There have been many performance issues that turn out to be slow loading of material from other sites, e.g. rss feeds or Javascript. So for example if you use mathjax (which I think is on by default in modern Moodle), then make sure it is loaded locally. I have seen significant problems caused by cron scripts running. I believe that the bottleneck for quizzing can be the database so look into optimising that, e.g. Maria db.

One piece of good advice is to ensure your exam machines(s) are dedicated to exams and don't do anything else.



 
Average of ratings: Useful (2)
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Also, is the OP running VMs? 

Everybody + dog runs on VMs these days and, for large Moodle sites, it's nearly always a bad idea. Unless you really know what you are doing. Unless.... dead

 
Average of ratings: -
Picture of Marcus Green
Re: Login 1000 student at a time on one quiz
Core developersParticularly helpful MoodlersPlugin developersTesters

Can't what Howard? I think having an extra layer of VM lard between Moodle and the hardware is a top way of enhancing performance smile

 
Average of ratings: Useful (1)
Picture of Jeff White
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers

What would you consider a large site too large for VMs? I am keeping my VMs spread out among multiple VM host servers to decrease the impact of the main culprit of your VMs slowing to a crawl of noisy neighbors on the same VM host server.  We operate under the process of increase demand on our Moodle site, we spin up more VMs. That is something not easily done with physical servers.

 

Oddly our environment is 300% faster running on a dedicated VM LAMP cluster over our old physical shared LAMP cluster even though that environment hardly ever was maxing out in its resources. The VMs have additional overhead due to the virtualization with much less resources of RAM and CPU that the physical servers had but yet it’s average page generation time is .4 seconds.

 
Average of ratings: Useful (1)
Tim at Lone Pine Koala Sanctuary
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

At the Open University, almost all our server are now VMs. The one exception is the DB server which hosts the database for our main Moodle site. We get good performance (e.g. ~0.4s for a typical page-load time) but then it is probably quite high spec hardware (I don't really know) and we have hardware load-balancers.

 
Average of ratings: Useful (1)
Picture of Visvanath Ratnaweera
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers
I don't know how virtualization popped up here. The OP and the control example, that of Frankle, both are dedicated hardware.
 
Average of ratings: -
Picture of Ken Task
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers

Taking a stab at it here, Howard! ;)

No one can answer your question ... at least not something you will accept.   Truth is, no one can.  It's up to you.   So here are some references for you to use so you can setup a test of your server:

http://www.iteachwithmoodle.com/2012/10/10/3-free-tools-to-test-if-your-moodle-server-can-cope-with-large-amounts-of-students/

http://www.iteachwithmoodle.com/2012/10/13/how-to-load-test-your-moodle-server-using-jmeter/

Since you are testing a production server (one that is in use), strongly suggest testing during off peak hours of usage.

And a warning ... expect issues IF you set the test to a quiz and 1000 users.

When doing the test, suggest one learns a few things about live monitoring.   These might require opening multiple terminal sessions (via ssh) first, before running a jmeter test.

In one ssh screen run 'top' ... that shows processes running and which are using the most resources as well as memory of server ... how much free, how much is swap.

Install a thing called 'mytop' ... which is top for MySQL.   Run it in another terminal screen when running jmeter testing.

By watching those and seeing when jmeter test begins to fail, etc. one can then determine what is max number of users for given system.

Prior to testing, also suggest installing mysqltuner ... http://mysqltuner.com/ and running it.

Have read that MySQL is most efficient when it can use 80% of the RAM available.

So with MySQL and Apache on the same server, one will need to tweak both configs.   NOTE: I've seen in your configs the number 1000 for a couple of settings ... which appear to be guesses.   Guesses, without testing the system, could be counter productive.  One is actually making the situation worse, not better.

Top will show the number of apache services up and running as well as how much memory is being used by Apache.   Just like it will show the mysqld daemon and how much memory it's using.   If at all possible the idea is to keep the server/services from using SWAP space ... that means data is written to and read from a temporary area of the Disk to complete whatever which means server response will be slower.

Also, one will never be able to get just Apache and MySQL on a single server to use ALL the memory ... the operating system itself needs to reserve enough memory to ... well ... operate!

Some things to check with the quiz ...

How is the quiz set?   To display all questions at once?  1 question at a time?  5 questions at a time?

Makes a difference.  Quiz IS, by it's nature, one of the heavier mods of Moodle in server processing.   That is a known fact.

Since you've not established a base line when Moodle was first installed and used, then it's a game of catch up ... testing, tweaking, testing, tweaking ... etc..  until one figures out what the server can handle.

Ok, just trying to help here ... so please, don't respond to this with the same question.  You've some work to do!

'spirit of sharing', Ken


 
Average of ratings: Useful (2)
Picture of Ken Task
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers

One more thing ... just thought of it (duh!)

You've given the goal and some server info ... but server info leaves some guessing on any one's part.

Is the server a 64 bit box?   The version of Moodle you are running (1.9) could run well on 32 bit ... depending upon other factors.

And ... a description of overall setup ... example:

1 ISD with total population of 6500 students/teachers.  Single server - meaning Apache/MySQL running on same server.   Site has over 120 courses.   They even offer Professional Development for teachers ... those sessions are the ones that are the heaviest in server usage - most number of users at one time.

So in this site, on any given day, there could be 4 High School classes whose students have a device or computer taking a quiz.   Each class, however, could be from 10 students to 25 students.

So in your situation ... 1000 users are in a lecture hall or x number of students per room taking the quiz at the same time ... like an End of Course test?

Which leads to yet another possible bottle neck ... all the students are on local LAN/WAN and the Moodle server is remotely hosted - ie, outside the local LAN/WAN.   What's the data pipe max for outbound traffic?   Got a 2 Gig circuit?   Is the LAN/WAN doing any video conferencing at the same time ... video conferencing is usually given quality of service in routers meaning a VC gets whatever bandwidth it needs ... leaving less bandwidth for other traffic ... such as X number of students take a quiz on a remotely hosted Moodle server.

What is your situation/overall setup?

'spirit of sharing', Ken


 
Average of ratings: -
Picture of Marcus Green
Re: Login 1000 student at a time on one quiz
Core developersParticularly helpful MoodlersPlugin developersTesters

I'm not sure I have ever seen a Moodle performance issue that was a core Moodle performance issue. I have seen slow disks, slow networks, shared hosting that would have been slow with anything, poor choice of clustered file systems, timeouts due to non existant rss feeds, slow cron jobs due to the tasks they were performing, systems with theme designer mode turned on, third party plugins that hang, but not anything that at root cause could be blamed on core Moodle code.

I suspect that others do know of such instances, but I have no personal experience of them.

 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Yes, it was called Moodle 1.8. 

We laugh about it now....

 
Average of ratings: -
Picture of Howard Miller
Re: Login 1000 student at a time on one quiz
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Ken, stop overcomplicating things and tell him how much memory he needs!!

I'm getting my coat.... clown

 
Average of ratings: Useful (1)
Picture of Visvanath Ratnaweera
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers
Howard

When you say "how much memory he needs" you mean the gray and wobbly kind, right?
smile
 
Average of ratings: Useful (1)
You either love it or you hate it
Re: Login 1000 student at a time on one quiz
Particularly helpful Moodlers

100GB ram! thoughtful


I have 4 x web nodes and all these only add up to 48GB

Holy Moly,  that's enough to power at least a million commodore 64s

Even so 1000 simultaneous quizzes would kill anything

 
Average of ratings: -