Hi,
We are facing/anticipating a performance issue when it comes to quizzes in moodle. The problem is that we have several teachers that intend to perform online exams at the end of the semester... and those exams may comprise some few hundreds of students... and it is possible some of the exams will be released simultaneously (so it may get pretty heavy for short periods).
Adding to this scenario we have our everyday moodle activity... that we absolutely don't want to compromise with the exams activity at the end of the semester!
I wonder if this is an usual scenario on your institution and what solutions do you use to solve this problem.
Thank you,
susana
Yes, other people have done things like this, and you have to be very careful.
I strongly suggest that you search the forums for similar threads.
1. You must do some testing in advance to check that your sever can cope with this. If the first time you run a quiz with hundreds of students is on the day of the exam, you will regret it.
2. Do not allow two teachers to run an exam at the same time, that just increases the load for no good reason.
3. It is even better if you can get the proctor/invigilator to get each student to start at a slightly different time, perhaps a few students each minute. That way they do not all finish at exactly the same time. Finishing seems to be the thing that causes the highest load.
4. If you care about your data, and about concurrency, you should not by using MySQL with MyISAM tables for your database. You get terrible lock contention. Use InnoDB on MySQL, or Postgres, if at all possible. But remember point 1. do some load-testing of whatever solution you adopt before the day of the exam.
I strongly suggest that you search the forums for similar threads.
1. You must do some testing in advance to check that your sever can cope with this. If the first time you run a quiz with hundreds of students is on the day of the exam, you will regret it.
2. Do not allow two teachers to run an exam at the same time, that just increases the load for no good reason.
3. It is even better if you can get the proctor/invigilator to get each student to start at a slightly different time, perhaps a few students each minute. That way they do not all finish at exactly the same time. Finishing seems to be the thing that causes the highest load.
4. If you care about your data, and about concurrency, you should not by using MySQL with MyISAM tables for your database. You get terrible lock contention. Use InnoDB on MySQL, or Postgres, if at all possible. But remember point 1. do some load-testing of whatever solution you adopt before the day of the exam.
Hi Tim,
Thank you for your suggestions!
Our problem is also how to control quizzes that are released at the same time. Some teachers are pretty autonomous and don't give us any feedback about quizzes and other activities like assignments they release to students... and we want to give them maximum freedom to do what they want...
We are also considering the following solutions:
Thank you for your attention!
Cheers,
susana
Thank you for your suggestions!
Our problem is also how to control quizzes that are released at the same time. Some teachers are pretty autonomous and don't give us any feedback about quizzes and other activities like assignments they release to students... and we want to give them maximum freedom to do what they want...
We are also considering the following solutions:
- Set our moodle production server in order it can handle normal activity users and quiz peaks (... Expensive, and wouldn't that be a waste of performance when it comes to non exam periods?)
- Set a separate server only for quizzes. It would not interfere with everyday activity and could be set to handle peaks:
- We could use some quiz application like question mark (hummm...plus license...)
- Or use a moodle instance, only for quiz activity (ok, but is it possible to integrate with production main moodle?)
- Create a scheduler module for exams which won't allow teachers to schedule quizzes with some X hundreds of users simultaneously.
Thank you for your attention!
Cheers,
susana
(This is mostly just a note in sympathy...)
We have had very poor results with even a small number of students (~15) taking quizzes simultaneously. We have found that there is an immediate spike in CPU load which renders the server non-responsive. Immediately restarting Apache seems to knock the CPU load back to previous, minimal levels. Sometimes, the quiz then progresses with little-to-no significant CPU load. Other times, there can be one or more recurrences of the spike.
Anecdotally, the spike seems to occur more immediately when a teacher also happens to be using the gradebook...
We went through the exercise of ensuring that there were only a few questions per page but this didn't seem to make any difference.
We have installed monit to automatically restart Apache when the CPU load spikes. This seems to help, although we still get student complaints about the sudden, brief periods of non-responsiveness while they are taking the quiz.
As to your possible solutions list (and I would be delighted to find a "silver bullet" for this problem there):
- 1.0 Beefing up the server might work. We're running a pretty beefy webserver with PHP accelerator, lots of RAM, the PostgreSQL database on a second server. I guess the question is, how much horsepower do you need to be assured of good results? Don't know the answer to that one as we haven't hit that point yet.
- 2.1 - Separate quiz application & server would certainly help, one imagines, but aside from license costs, there's the not-integrated-with-Moodle-course or -gradebook problem. I'd also wonder about making sure all students could find the new URL.
-2.2 - Don't know that moving to a second Moodle instance on a separate server would solve the problem. It'd prevent the interference with non-quiz activities, but no guarantee quizzes would work, and the subsequent re-integration seems fraught with peril.
-3.0 - Scheduler idea might work, but seems like an investment in a workaround where in an ideal world, we'd invest instead in solving the problem within the quiz module itself.
IIRC, a comment was made elsewhere about the possibility of re-factoring the quiz module, but I don't know if anything has come of this...
And of course, having teachers & students stagger their quiz start/stop activities would be a big help. (All but impossible to effect, sadly, but wouldn't it be nice?) In our case, and it sounds like yours also, teachers and students expect the quiz module to just work as advertised, whenever they want to use it. When they find it doesn't, they're ready to abandon the whole application.
Perhaps we could take up a collection to sponsor a quiz redevelopment effort?
Otherwise, should you find another, better solution I'll be eager to hear it!
Cheers,
Worth
We have had very poor results with even a small number of students (~15) taking quizzes simultaneously. We have found that there is an immediate spike in CPU load which renders the server non-responsive. Immediately restarting Apache seems to knock the CPU load back to previous, minimal levels. Sometimes, the quiz then progresses with little-to-no significant CPU load. Other times, there can be one or more recurrences of the spike.
Anecdotally, the spike seems to occur more immediately when a teacher also happens to be using the gradebook...
We went through the exercise of ensuring that there were only a few questions per page but this didn't seem to make any difference.
We have installed monit to automatically restart Apache when the CPU load spikes. This seems to help, although we still get student complaints about the sudden, brief periods of non-responsiveness while they are taking the quiz.
As to your possible solutions list (and I would be delighted to find a "silver bullet" for this problem there):
- 1.0 Beefing up the server might work. We're running a pretty beefy webserver with PHP accelerator, lots of RAM, the PostgreSQL database on a second server. I guess the question is, how much horsepower do you need to be assured of good results? Don't know the answer to that one as we haven't hit that point yet.
- 2.1 - Separate quiz application & server would certainly help, one imagines, but aside from license costs, there's the not-integrated-with-Moodle-course or -gradebook problem. I'd also wonder about making sure all students could find the new URL.
-2.2 - Don't know that moving to a second Moodle instance on a separate server would solve the problem. It'd prevent the interference with non-quiz activities, but no guarantee quizzes would work, and the subsequent re-integration seems fraught with peril.
-3.0 - Scheduler idea might work, but seems like an investment in a workaround where in an ideal world, we'd invest instead in solving the problem within the quiz module itself.
IIRC, a comment was made elsewhere about the possibility of re-factoring the quiz module, but I don't know if anything has come of this...
And of course, having teachers & students stagger their quiz start/stop activities would be a big help. (All but impossible to effect, sadly, but wouldn't it be nice?) In our case, and it sounds like yours also, teachers and students expect the quiz module to just work as advertised, whenever they want to use it. When they find it doesn't, they're ready to abandon the whole application.
Perhaps we could take up a collection to sponsor a quiz redevelopment effort?
Otherwise, should you find another, better solution I'll be eager to hear it!
Cheers,
Worth
You might try using a monitoring tool like Cacti to discover exactly where your load is coming from, then tune that part of your system. Often performance issues are multi-causal, so you might fix one problem only to uncover another. From your report of the issue happening when teachers look at the gradebook, it *sounds* like a database performance issue, perhaps in how it is tuned, or in how it's connected, or...(my team once worked for weeks with a client to debug a weird performance issue only to find that they had connected gigabit fiber to a 10mbit card).
15 students in a quiz with a teacher viewing gradebook shouldn't be any problem for a site, esp. with a separate db server (we have folks with much larger loads than that on single box servers that run very fast), so I'd recommend you look into tuning the servers before upgrading them, or the other ideas on your list.
15 students in a quiz with a teacher viewing gradebook shouldn't be any problem for a site, esp. with a separate db server (we have folks with much larger loads than that on single box servers that run very fast), so I'd recommend you look into tuning the servers before upgrading them, or the other ideas on your list.
Hi
Did you discuss those issues in this forum? If so please post the link to the discussion. Otherwise post as many details about your system: OS, webserver, php, mysql/postgres - all with version numbers. Also Moodle version and how you installed.
Did you go through http://docs.moodle.org/en/Performance and http://docs.moodle.org/en/Performance_FAQ ?
As pointed out by Michael, 15 taking a quiz simultanously is nothing for a state of the art server.
Did you discuss those issues in this forum? If so please post the link to the discussion. Otherwise post as many details about your system: OS, webserver, php, mysql/postgres - all with version numbers. Also Moodle version and how you installed.
Did you go through http://docs.moodle.org/en/Performance and http://docs.moodle.org/en/Performance_FAQ ?
As pointed out by Michael, 15 taking a quiz simultanously is nothing for a state of the art server.
I'm also experiencing loading issues with only 34 students concurrently loading a 60 question randomly generated exam. Each question has a 60-90kb image. The curious thing is the exact same exams run from the hosted moodle site at the college I work at and load fine. My own site is running on a dedicated quad core kentsfield 3210 server with 4gig ram, 2x 500gb drives and 2500 gb bandwidth. I'm thinking my server hardware is adequate to have 34 concurrent 60 question exams run would it not? I'm really hoping to have the issues resolved as I expect the user base to grow significantly over the next year, but I'm more than concerned over this loading issue. Automated assessment is one of the main reasons I use an LMS. I've been reluctant to upgrade the site to 2.0 from 1.9 while I have paying customers, but I'll do that over the Christmas break and hopefully it will resolve the issue.
Is there any information on just what hardware is required, or what the Moodle software limitations are to achieve adequate usability for hundreds or even thousands of users concurrently?
I'm certainly up for contributing to development to resolve the issue.
Do you have APC installed on the server (mentioned earlier in this thread)? That's what helped my load issues. We had about 120 students concurrently taking exams. Completely solved the problem, and reduced the overall load by more than 50%.
Just wondering if your quiz is a long page (or has several long pages rather than short ones with a few questions on each). I have read that breaking up long pages can significantly lighten the load when many students start a test at the same time.
Well we had 10q per page and then dropped it to 5. Still had issues loading pictures. My site service cannot add the caching app to speed things up apparently because mod_php would be required and they run mod_suphp for the added security it provides over mod_php. From what I've read caching won't really help with random quizzes anyway. Perhaps the code gurus will consider providing a caching function by having the randomizing engine pre script a randomized test and cache it prior to running an exam by selecting the number of quizzes expected to be required simultaneously? I'm really hoping for a solution as I see automated randomized assessment as one of the cornerstones of improved education. It enables me to minimize ineffective time in the teaching leanring process for both myself and my students, particularly for experiencial learners.
Does anyone actually know of any way of deciding just how much hardware and what software is required for a given number of students to have seamless access to Moodle course content?
Your teachers may be autonomous, but when it comes to using a shared resource, they have to coordinate with each other.
Suppose two teachers both wanted to run their exam in the school hall. And suppose that they both just turned up with their students at the time they wanted to start and found the other one there, and that there was not space for them both to fit. That would be a disaster.
However, it is obvious that the school hall is a fixed size resource, and so if you want to use it for an important activity, you should book it in advance. With Moodle the situation is less transparent, but no less clear.
You need to explain this to your teachers.
Worth Bishop said, "a comment was made elsewhere about the possibility of re-factoring the quiz module, but I don't know if anything has come of this..."
Well, you don't need to take out a collection for that to happen, because I have been working on it for the last 6 months or so, paid for by my employer, the Open University. It is nearly finished, but, annoyingly, it was not finished in time to include in Moodle 2.0. Details on Development:Question_Engine_2 if you are interested.
One of the not finished bits is that I have not yet run the load tests to see whether it performs better than the old version. However I expect it to perform better, and the way the code it written, it should be easier to isolate and do something about any performance bottlenecks.
Suppose two teachers both wanted to run their exam in the school hall. And suppose that they both just turned up with their students at the time they wanted to start and found the other one there, and that there was not space for them both to fit. That would be a disaster.
However, it is obvious that the school hall is a fixed size resource, and so if you want to use it for an important activity, you should book it in advance. With Moodle the situation is less transparent, but no less clear.
You need to explain this to your teachers.
Worth Bishop said, "a comment was made elsewhere about the possibility of re-factoring the quiz module, but I don't know if anything has come of this..."
Well, you don't need to take out a collection for that to happen, because I have been working on it for the last 6 months or so, paid for by my employer, the Open University. It is nearly finished, but, annoyingly, it was not finished in time to include in Moodle 2.0. Details on Development:Question_Engine_2 if you are interested.
One of the not finished bits is that I have not yet run the load tests to see whether it performs better than the old version. However I expect it to perform better, and the way the code it written, it should be easier to isolate and do something about any performance bottlenecks.
Hi,
We are using:
moodle 1.9
app server (CentOS - 2x Xeon 3,2GHz 4GB RAM, 6x 146GB SCSI)
db server (CentOS - 2x Xeon 3,6GHz 4GB RAM, 3x 300GB SCSI)
2 webcache servers
We usually make exams by shifts (30-60 students). We had already more than 100 students taking a quiz simultaneously because we had two different quizzes at the same time... with success. But we also had some incidents, most of them because of simultaneous activities:
I understand Tim's point of view as quizzes are a shared resource. Well, the whole moodle is a shared resource... But we are realizing that some modules, (like quizzes and assessments), because of their nature (used by many users concurrently in short periods) require special treatment and may compromise a simple browsing of thousands of users. On the other hand an exam is also something very sensible... even a 10 sec waiting for a server response stresses out any student. We don't want to see a final exam compromised by a regrading or an excited forum discussion in another course.
That's why we are thinking about to isolate exam tools, like quizzes, from the rest of moodle... We also thought about doing offline quizzes, but it requires some logistics and security procedures that may become the tool less agile.
... or maybe most of the institutions invest in a beefy server/architecture, as Worth Bishop mentioned, and have no headaches with this issue(?)...
I just want to remark that I think that the moodle quiz module is a great tool... and that's the main problem: all our teachers want to use it. Hope the improvements will be available soon in moodle2.0.
Thanks for your interest in the subject and hope to see some more opinions and experiences!
Cheers,
susana
We are using:
moodle 1.9
app server (CentOS - 2x Xeon 3,2GHz 4GB RAM, 6x 146GB SCSI)
db server (CentOS - 2x Xeon 3,6GHz 4GB RAM, 3x 300GB SCSI)
2 webcache servers
We usually make exams by shifts (30-60 students). We had already more than 100 students taking a quiz simultaneously because we had two different quizzes at the same time... with success. But we also had some incidents, most of them because of simultaneous activities:
- a gradebook browsing and regrading at the same time of a quiz
- an assignment and a quiz simultaneously
- simultaneous quiz and assignment view (a teacher tries to download hundreds of submissions)
I understand Tim's point of view as quizzes are a shared resource. Well, the whole moodle is a shared resource... But we are realizing that some modules, (like quizzes and assessments), because of their nature (used by many users concurrently in short periods) require special treatment and may compromise a simple browsing of thousands of users. On the other hand an exam is also something very sensible... even a 10 sec waiting for a server response stresses out any student. We don't want to see a final exam compromised by a regrading or an excited forum discussion in another course.
That's why we are thinking about to isolate exam tools, like quizzes, from the rest of moodle... We also thought about doing offline quizzes, but it requires some logistics and security procedures that may become the tool less agile.
... or maybe most of the institutions invest in a beefy server/architecture, as Worth Bishop mentioned, and have no headaches with this issue(?)...
I just want to remark that I think that the moodle quiz module is a great tool... and that's the main problem: all our teachers want to use it
Thanks for your interest in the subject and hope to see some more opinions and experiences!
Cheers,
susana
Hi,
Nice to meet you susana, i am a newly started moodle developer, you gave a very useful information regarding the moodle quiz.
Actually i have also had the same kind of problem when i conducted a online quiz for students of National Police Academy. The server was heavily loaded with the students requests.
There are 130 students, and our server configuration is 2GB RAM, P4 Processor, ubuntu 9.04 server, moodle 1.9.4 we are using.
And I liked your suggestion that parting the moodle quiz from moodle is a very nice idea and fantastic.
Do you have any suggestions to do that? If u have please guide me.
Awaiting your response regarding this solution...
Regards
chiru
Nice to meet you susana, i am a newly started moodle developer, you gave a very useful information regarding the moodle quiz.
Actually i have also had the same kind of problem when i conducted a online quiz for students of National Police Academy. The server was heavily loaded with the students requests.
There are 130 students, and our server configuration is 2GB RAM, P4 Processor, ubuntu 9.04 server, moodle 1.9.4 we are using.
And I liked your suggestion that parting the moodle quiz from moodle is a very nice idea and fantastic.
Do you have any suggestions to do that? If u have please guide me.
Awaiting your response regarding this solution...
Regards
chiru
I never knew moodle can be used for exams. Hope to learn more.
internet marketing expert
internet marketing expert
Interesting idea - trying to schedule online quizzes to reduce load & comparing to lecture halls - large US institutions often run software like Banner or Peoplesoft to schedule rooms. These are multi-million $ pieces of software - though of course they do other things like transcripts and financials, curriculum management, etc.
However, in many cases a big selling point of an LMS is that an institution can be free of the need for complex room schedules - in fact a large Uni where a friend works got around the limits of a space constrained campus by using blended courses where students can choose to show up for lecture or view the lecture video from elsewhere. This let the uni increase the size of a popular lecture courses from 200 students that could fit in the lecture hall to 1000 students. As this is in a traffic congested place, this also helped reduce parking and commuting issues students had. But the course had a weekly quiz that closed on a Friday at 11:59. And since it turned out that a good number of the students started that quiz at about 11:30 PM, that put a bit of a strain on their server*.
Does OU have a method of scheduling quiz open/close times? This almost seems like it would be more difficult than scheduling rooms. To automate it one would need to check the times of all the quizzes on the system, I guess, and inform instructors if there were too many at a given time for the available system resources... interesting problem.
* I call this 'academic peak load' - since it is different from corporate LMS loads which tend to be fairly steady M-F. I counsel folks to configure and tune systems with this load in mind - in higher ed loads tend to peak on academic and student calendars, which means often just before a vacation, Friday afternoons or evenings, and during mid-terms and finals. Often load will be very low at other times and then spike dramatically, which can be challenging for SysAdmins and System Architects (fortunately ours are quite well versed in this issue).
However, in many cases a big selling point of an LMS is that an institution can be free of the need for complex room schedules - in fact a large Uni where a friend works got around the limits of a space constrained campus by using blended courses where students can choose to show up for lecture or view the lecture video from elsewhere. This let the uni increase the size of a popular lecture courses from 200 students that could fit in the lecture hall to 1000 students. As this is in a traffic congested place, this also helped reduce parking and commuting issues students had. But the course had a weekly quiz that closed on a Friday at 11:59. And since it turned out that a good number of the students started that quiz at about 11:30 PM, that put a bit of a strain on their server*.
Does OU have a method of scheduling quiz open/close times? This almost seems like it would be more difficult than scheduling rooms. To automate it one would need to check the times of all the quizzes on the system, I guess, and inform instructors if there were too many at a given time for the available system resources... interesting problem
* I call this 'academic peak load' - since it is different from corporate LMS loads which tend to be fairly steady M-F. I counsel folks to configure and tune systems with this load in mind - in higher ed loads tend to peak on academic and student calendars, which means often just before a vacation, Friday afternoons or evenings, and during mid-terms and finals. Often load will be very low at other times and then spike dramatically, which can be challenging for SysAdmins and System Architects (fortunately ours are quite well versed in this issue
Well, you say "selling point". In other words marketing.
In reality it is not that simple. An LMS is more flexible. It costs less to buy more memory for you web server, or buy a whole new faster server, than it does to make your school hall twice as big. Also, whatever limits there are, they are softer than the physical limits of walls. You can squeeze in more students, but the server becomes a bit slower for everyone.
At the OU, I think there is some effort to make sure that not all courses choose the same deadlines, but I've just released that I have no idea how that happens. Certainly it is not a heavy-handed system. And that applies more to our assignment submission system (still separate from Moodle because it has been around for a long time and works well) than the Moodle quiz, since our assignment submission system has much higher use levels.
OU quizzes tend to be open for several weeks, which spreads the load. We have only a moderate last-minute rush. Here is the data I got out of our database: http://tjhunt.blogspot.com/2010/03/when-do-students-submit-their-online.html
At the OU, typically a course is created, and then taught for 5, 7 or 10 years with only minor modification. So, actually we are still in the start-up phase of our use of the Moodle quiz. The usage numbers for this year (about 35000 quiz attempts per month) are about three times what they were last year. Most of that usage is formative. Looking at the numbers, only about 15% of the quiz attempts are summative at the moment.
In reality it is not that simple. An LMS is more flexible. It costs less to buy more memory for you web server, or buy a whole new faster server, than it does to make your school hall twice as big. Also, whatever limits there are, they are softer than the physical limits of walls. You can squeeze in more students, but the server becomes a bit slower for everyone.
At the OU, I think there is some effort to make sure that not all courses choose the same deadlines, but I've just released that I have no idea how that happens. Certainly it is not a heavy-handed system. And that applies more to our assignment submission system (still separate from Moodle because it has been around for a long time and works well) than the Moodle quiz, since our assignment submission system has much higher use levels.
OU quizzes tend to be open for several weeks, which spreads the load. We have only a moderate last-minute rush. Here is the data I got out of our database: http://tjhunt.blogspot.com/2010/03/when-do-students-submit-their-online.html
At the OU, typically a course is created, and then taught for 5, 7 or 10 years with only minor modification. So, actually we are still in the start-up phase of our use of the Moodle quiz. The usage numbers for this year (about 35000 quiz attempts per month) are about three times what they were last year. Most of that usage is formative. Looking at the numbers, only about 15% of the quiz attempts are summative at the moment.
To be honest I think most of the problems are caused strictly by the 'face-to-face' approach (herd hundreds of students into one/several rooms and make them all take a quiz at the exact same time). Entirely distance learning institutions aren't likely to have this problem because we don't have the option to do that. (Or to be more specific... where there are formal exams that do require students to be in specific places at one time, those are still done with pen and paper here, thankfully.)
Tim is the expert on quiz and I'm sure it has some specific performance issues, but I wonder if this is not necessarily a problem with quiz specifically but just concurrency in general. We have over 150,000 students currently on courses. By most measurements, that's a pretty big institution, and we have a pretty big Moodle system to cope with it. But if you take concurrency as 'unique users in a 5 minute period', a typical value for us - like right now say - is about 1,100. So if we forced students even on just one course (~1,000 students) to take a quiz at the exact same time, we could easily double our concurrency. Two courses? We've tripled it. And so on.
It's possible to do queries in Moodle log table to find out what your system's typical concurrency in a 5 minute period is (for each period across a day, say). If that number's much higher than you expect the number of people to be simultaneously taking a quiz, then cool, maybe it'll work. But if it's not, then running a simultaneous quiz - or simultaneous anything - with a large number of users takes you into uncharted territory.
--sam
PS 'Concurrency' numbers are a bit of a crock as they don't really relate to system load - in Moodle you can use something like 'log rows per minute' to do that - but it's easier to talk about numbers that way...
Tim is the expert on quiz and I'm sure it has some specific performance issues, but I wonder if this is not necessarily a problem with quiz specifically but just concurrency in general. We have over 150,000 students currently on courses. By most measurements, that's a pretty big institution, and we have a pretty big Moodle system to cope with it. But if you take concurrency as 'unique users in a 5 minute period', a typical value for us - like right now say - is about 1,100. So if we forced students even on just one course (~1,000 students) to take a quiz at the exact same time, we could easily double our concurrency. Two courses? We've tripled it. And so on.
It's possible to do queries in Moodle log table to find out what your system's typical concurrency in a 5 minute period is (for each period across a day, say). If that number's much higher than you expect the number of people to be simultaneously taking a quiz, then cool, maybe it'll work. But if it's not, then running a simultaneous quiz - or simultaneous anything - with a large number of users takes you into uncharted territory.
--sam
PS 'Concurrency' numbers are a bit of a crock as they don't really relate to system load - in Moodle you can use something like 'log rows per minute' to do that - but it's easier to talk about numbers that way...
Actually, one point that has been made in the past, but I don't think has been repeated in this thread yet: MySQL sucks. More specifically, MySQL with the default MyISAM storage engine sucks when you are doing concurrent writes. It can only lock entire database tables, so when it is processing one students quiz submission, it cannot process another student's submission. By contrast, a real database like Postgres or the INNODB storage engine for MySQL, can lock individual table rows, so it can concurrently update the data relating to two different students' attempts concurrently.
(While the above analysis is logical, I am still hoping that someone will actually do some load testing and come out with experimental verification that it is true. I have not seen anyone actually measure this yet.)
P.S. When sam says, "where there are formal exams that do require students to be in specific places at one time, those are still done with pen and paper here, thankfully." you can tell he is not a student. I have studied OU courses, and I have done three hours exams with a pen and paper, and it is horrible. It hurts to write for that long. But, at least the server can't crash
(While the above analysis is logical, I am still hoping that someone will actually do some load testing and come out with experimental verification that it is true. I have not seen anyone actually measure this yet.)
P.S. When sam says, "where there are formal exams that do require students to be in specific places at one time, those are still done with pen and paper here, thankfully." you can tell he is not a student. I have studied OU courses, and I have done three hours exams with a pen and paper, and it is horrible. It hurts to write for that long. But, at least the server can't crash
More specifically, MySQL with the default MyISAM storage engine sucks when you are doing concurrent writes
IMO, good to be clear that MySQL with MyISAM is not the ideal tool for large quiz concurrency, while MySQL with INNODB is much better. I'm just saying this because I've seen a few people who have read "MySQL bad/Postgres good" on a forum, switched over to Postgres, expecting a silver bullet, and finding they had even worse performance than they did before*.
My recommendation to people using MySQL, set up monitoring (so you know which of your steps are helping vs. hurting), switch to INNODB, tune your config for your kind of load, then only if all that doesn't work, think about changing your db. In most cases the first few steps solves the problem.
*And the system I mentioned above with the 1000 student quiz & dying server were using Postgres - w/as in the places I've worked we've handled very large loads via well tuned MySQL
.
I'd agree that many of the issues are due to the face-to-face model of day/time/room scheduled classes - this may well be an out of date model, however most LMS using K-20 systems I know of are still in a blended model, with online supplementing face-to-face, and probably will be in that model for at least the next decade.
I'd also hat to have to take a multi-page exam via Pen and Paper again (filled many little blue books with scrawl in my undergraduate days) in this day and age, I'd call that barbaric.
IMO, good to be clear that MySQL with MyISAM is not the ideal tool for large quiz concurrency, while MySQL with INNODB is much better. I'm just saying this because I've seen a few people who have read "MySQL bad/Postgres good" on a forum, switched over to Postgres, expecting a silver bullet, and finding they had even worse performance than they did before*.
My recommendation to people using MySQL, set up monitoring (so you know which of your steps are helping vs. hurting), switch to INNODB, tune your config for your kind of load, then only if all that doesn't work, think about changing your db. In most cases the first few steps solves the problem
*And the system I mentioned above with the 1000 student quiz & dying server were using Postgres - w/as in the places I've worked we've handled very large loads via well tuned MySQL
.I'd agree that many of the issues are due to the face-to-face model of day/time/room scheduled classes - this may well be an out of date model, however most LMS using K-20 systems I know of are still in a blended model, with online supplementing face-to-face, and probably will be in that model for at least the next decade.
I'd also hat to have to take a multi-page exam via Pen and Paper again (filled many little blue books with scrawl in my undergraduate days) in this day and age, I'd call that barbaric
I think many people would call the Moodle quiz offering (in terms of complexity of question types and quality of automated marking) far worse things than 'barbaric'
Of course you can use entirely manually marked questions but then really it's almost the same as pen and paper except that pen and paper doesn't crash when you get a thousands students to use it at once... (OK you also get records etc but.)
@tim - Yes my hand gets tired if I write too these days - but by 'Thankfully' I really meant, we do not have either (a) the server load, or (b) the mega-fallout if things go wrong...
Agree Postgres is not a silver bullet. If you haven't started using a system and you don't have experience managing either then for goodness' sake pick Postgres - much simpler. But if you already use MySQL and especially if you have experience tuning it then obviously it is a good idea to stick to it if possible.
You can certainly break postgres performance... we have done quite frequently. (Hint: Turn off auto-vacuum and don't vacuum/analyse frequently-changed tables anywhere near enough. Then never run full vacuum. Give it a few months and your database will be practically dead! This is the point where you try to extort a pay rise from your employer in the event that you fix it.)
As for face-to-face, yes I understand this still happens! Nothing against it, just I think people using it on a large scale need to be willing to accept Tim's performance suggestions (e.g. trying to make sure different courses don't have tests at the exact same time; staggered starts).
I went to a presentation in the UK moot where they (i think it was the university in Beirut?) discussed scheduling quizzes. I think they had integrated custom forms into their moodle so teachers could book a particular time and also book the computer room at the same time. And for a big test you could book an IT support person to be on hand as well in case anything breaks. All these requests went through the administrators; I don't recall much discussion of performance issues but presumably, if too many teachers wanted to book labs for tests at the same time, administrators could say 'sorry please reschedule'.
--sam
Of course you can use entirely manually marked questions but then really it's almost the same as pen and paper except that pen and paper doesn't crash when you get a thousands students to use it at once... (OK you also get records etc but.)
@tim - Yes my hand gets tired if I write too these days - but by 'Thankfully' I really meant, we do not have either (a) the server load, or (b) the mega-fallout if things go wrong...
Agree Postgres is not a silver bullet. If you haven't started using a system and you don't have experience managing either then for goodness' sake pick Postgres - much simpler. But if you already use MySQL and especially if you have experience tuning it then obviously it is a good idea to stick to it if possible.
You can certainly break postgres performance... we have done quite frequently. (Hint: Turn off auto-vacuum and don't vacuum/analyse frequently-changed tables anywhere near enough. Then never run full vacuum. Give it a few months and your database will be practically dead! This is the point where you try to extort a pay rise from your employer in the event that you fix it.)
As for face-to-face, yes I understand this still happens! Nothing against it, just I think people using it on a large scale need to be willing to accept Tim's performance suggestions (e.g. trying to make sure different courses don't have tests at the exact same time; staggered starts).
I went to a presentation in the UK moot where they (i think it was the university in Beirut?) discussed scheduling quizzes. I think they had integrated custom forms into their moodle so teachers could book a particular time and also book the computer room at the same time. And for a big test you could book an IT support person to be on hand as well in case anything breaks. All these requests went through the administrators; I don't recall much discussion of performance issues but presumably, if too many teachers wanted to book labs for tests at the same time, administrators could say 'sorry please reschedule'.
--sam
Great to hear that, Tim,
Another teacher and I are completing a proposal to our employer for a new Moodle server because the salvaged machine that we used this year (a 5-6 year old Dell PowerEdge w/ 512MB of RAM running Linux) was choking on 25 students starting a quiz concurrently for their final exam.
It's somewhat disheartening to learn that this issue has also hit people with more up-to-date systems that ours. The load was heaviest on the Apache processes which were processing the PHP. Granted, this was a chemistry final with tons of LaTeX images for Lewis dot diagrams and chemical equations and such, but I was disheartened by the sluggish response though.
Will your work on Question Engine 2 be available during an incremental 2.x release of Moodle sometime in the next year?
Take care,
-Tom
Another teacher and I are completing a proposal to our employer for a new Moodle server because the salvaged machine that we used this year (a 5-6 year old Dell PowerEdge w/ 512MB of RAM running Linux) was choking on 25 students starting a quiz concurrently for their final exam.
It's somewhat disheartening to learn that this issue has also hit people with more up-to-date systems that ours. The load was heaviest on the Apache processes which were processing the PHP. Granted, this was a chemistry final with tons of LaTeX images for Lewis dot diagrams and chemical equations and such, but I was disheartened by the sluggish response though.
Will your work on Question Engine 2 be available during an incremental 2.x release of Moodle sometime in the next year?
Take care,
-Tom
The Question Engine is a major change, so it can only go into a a major release. That is, it might go into Moodle 2.1 (that is the plan) but it could not go into 2.0.1.
I read through this entire forum and am now even more nervous than I was prior to reading it because we have scheduled a final exam to be delivered via our Moodle site. The exam will be taken by a little over 400 students in two waves. Half first and the remainder an hour later. So, we're looking at something like 200 at one time. We can schedule it so there is a lag in the start time for 42 students at a time (e.g. start 42 wait 2 minutes start another 42 etc.) but I'm still nervous and am not sure of a good load testing method.
Our server is an Apple XServe (G5 Dual 2.3Ghz with 5Gig of RAM) running Ubuntu LAMP. 2 7200 RPM drives in the bays on the XServe. One drive contains the LAMP and the Moodle instance, the other contains the Moodle DataDir.
Any advice? Besides, "reschedule your test!"
Thanks in advance.
Jason
Our server is an Apple XServe (G5 Dual 2.3Ghz with 5Gig of RAM) running Ubuntu LAMP. 2 7200 RPM drives in the bays on the XServe. One drive contains the LAMP and the Moodle instance, the other contains the Moodle DataDir.
Any advice? Besides, "reschedule your test!"
Thanks in advance.
Jason
Tweaking should be divided into two parts:
Ad.2 Use as much RAM as you can to postpone disk IO.
Moodle dataDir should not be placed on same partition (drive) as database. The best would be to have 3 separate IO pools (system, dataDir, DB). In this case dataDir should go with system and DB separate. Usually during quizzes Db is a bottleneck.
- application server (Apache, php)
- database server (MySQL)
Ad.2 Use as much RAM as you can to postpone disk IO.
Moodle dataDir should not be placed on same partition (drive) as database. The best would be to have 3 separate IO pools (system, dataDir, DB). In this case dataDir should go with system and DB separate. Usually during quizzes Db is a bottleneck.
Bartosz,
Thank you very much for the reply. This type of tweaking is pushing my skills but I'm trying. I'm going to max out the RAM on our server with an addition of 4 Gb to start with.
The hard part for me is going to be the tweaking of Apache, PHP and MySQL. I remember some great threads (or maybe Moodle Docs) that contain detailed information about this type of tweaking so I'll do a search and hopefully find something to get me started.
As for the drive issue, I'm not going to be able to get a third drive set up prior to our test administration (I don't think) so I'll be forced to use the two drives we have now. I'll have to work on moving the DB over to it's own dedicated drive and the dataDir on to the drive with the System but that won't be too hard.
Thanks again
Jason
P.S. Does anyone know how I can do a load test without actually having 200 students sit down and access the site?
Thank you very much for the reply. This type of tweaking is pushing my skills but I'm trying. I'm going to max out the RAM on our server with an addition of 4 Gb to start with.
The hard part for me is going to be the tweaking of Apache, PHP and MySQL. I remember some great threads (or maybe Moodle Docs) that contain detailed information about this type of tweaking so I'll do a search and hopefully find something to get me started.
As for the drive issue, I'm not going to be able to get a third drive set up prior to our test administration (I don't think) so I'll be forced to use the two drives we have now. I'll have to work on moving the DB over to it's own dedicated drive and the dataDir on to the drive with the System but that won't be too hard.
Thanks again
Jason
P.S. Does anyone know how I can do a load test without actually having 200 students sit down and access the site?
There is a load testing tool called Apache Jmeter which allows you to create scripts that simulate multiple users. I created a little script that simply logged in and logged out again and used it to bring moodle to a standstill with a simulation of 200 simultanous users on my fairly low spec Windows laptop to demonstrate the concept of load testing to my students.
Jason,
Apart from providing physical RAM chips, you should also let MySQL use it. Usually default my.cnf config file just works. I would start from choosing proper size of innodb_buffer_pool_size.
Talking about PHP, I would start from installation of APC.
As it was said before Jmeter from apache is a really good stress testing tool. However you have to spent some time to prepare scenarios that would give you objective results.
If you are short on time, maybe I would be able to help you? I've already done that kind of stuff remotely several times. And I don't rip my clients off (:
Apart from providing physical RAM chips, you should also let MySQL use it. Usually default my.cnf config file just works. I would start from choosing proper size of innodb_buffer_pool_size.
Talking about PHP, I would start from installation of APC.
As it was said before Jmeter from apache is a really good stress testing tool. However you have to spent some time to prepare scenarios that would give you objective results.
If you are short on time, maybe I would be able to help you? I've already done that kind of stuff remotely several times. And I don't rip my clients off (:
Hello all,
I'm developing a mild panic at this point as I've been trying to get the following system set up and am not having much luck:
Macintosh XServe G5 Dual 2.3GH with 8 GB of RAM and three drives (all 7200rpm)
My plan was to have the LAMP on drive 1, the Moodle datadir on drive 2 and the MySQL db on drive 3 but....
I upgraded the server to Ubuntu 10.04 and am having an issue with sudden shutdowns......ouch!!!
I ran Ubuntu 7.04 for several years without one issue and am contemplating reverting to that and giving it a try. Actually, not contemplating, I will be doing so tomorrow and letting it run for a while to see how it performs.
My question is this:
I've been running our Moodle site on a Mac Mini for the past year and never had an issue with performance. The specs on the mini are:
2.0 GHz Intel dual core
4 GB Ram (but only about 3 is addressed)
Internal hard drive (120 gb) - little laptop hard drive in the mini (5200 rpm..?)
I've been attempting to revert to the XServe with the following specs:
Dual 2.3Ghz processors
8 GB RAM
3 7200 rpm drives (each 400 GB+)
but am having the OS issue.
So, is it crazy to try to run a quiz with the Mini with approx 180 students taking the quiz simultaneously? I can optimize the db (innodb) and allocate more ram to it but am still thinking that I don't have enough to work with....
If I can't get this silly XServe working I'm going to be in a bind and am starting to work on formulating a 'backup' plan in case I can't get it up and running reliably soon.
Any advice will be most appreciated.
Jason
I'm developing a mild panic at this point as I've been trying to get the following system set up and am not having much luck:
Macintosh XServe G5 Dual 2.3GH with 8 GB of RAM and three drives (all 7200rpm)
My plan was to have the LAMP on drive 1, the Moodle datadir on drive 2 and the MySQL db on drive 3 but....
I upgraded the server to Ubuntu 10.04 and am having an issue with sudden shutdowns......ouch!!!
I ran Ubuntu 7.04 for several years without one issue and am contemplating reverting to that and giving it a try. Actually, not contemplating, I will be doing so tomorrow and letting it run for a while to see how it performs.
My question is this:
I've been running our Moodle site on a Mac Mini for the past year and never had an issue with performance. The specs on the mini are:
2.0 GHz Intel dual core
4 GB Ram (but only about 3 is addressed)
Internal hard drive (120 gb) - little laptop hard drive in the mini (5200 rpm..?)
I've been attempting to revert to the XServe with the following specs:
Dual 2.3Ghz processors
8 GB RAM
3 7200 rpm drives (each 400 GB+)
but am having the OS issue.
So, is it crazy to try to run a quiz with the Mini with approx 180 students taking the quiz simultaneously? I can optimize the db (innodb) and allocate more ram to it but am still thinking that I don't have enough to work with....
If I can't get this silly XServe working I'm going to be in a bind and am starting to work on formulating a 'backup' plan in case I can't get it up and running reliably soon.
Any advice will be most appreciated.
Jason
I forgot to add that my db is 543 MB
&
The quiz I'm preparing to deliver is 60 questions. 30 listening and 30 reading (all multiple choice each with one correct answer)
Jason
&
The quiz I'm preparing to deliver is 60 questions. 30 listening and 30 reading (all multiple choice each with one correct answer)
Jason
Updating my inquiry here for informational value (hopefully)
I ended up reverting to Ubuntu 7.04 because newer versions did not implement the fan control properly (windfarm controls) and I was getting sudden shutdowns, which, of course, won't do.
So, back in 7.04 now and happily stable. I have changed MySQL settings as follows to hopefully speed it up under quizzes:
default-storage-engine=InnoDB
innodb_buffer_pool_size = 750M
innodb_log_files_in_group = 2
innodb_log_file_size = 256M
innodb_file_per_table = 1
I'm wondering however, if I really need InnoDB as default and if I could just convert the tables associated with a quiz to innodb to achieve the efficiency I'm looking for....?
It seems completely unrelated but I've noticed now that a block I have on the front page of our site (MyCourses) is loading much more slowly since I've made these changes. This was an issue brought up by the developer (here) but under MyISAM on a machine that had less RAM and less processor power, I never noticed a significant lag. Maybe I've gained performance in one area and lost a little elsewhere.
Also, I did go ahead and install APC as was suggested. Wow! Surprised and how much more quickly pages load!
Here are my system specs (also for informational value)
Mac XServe G5 Dual 2.3Ghz processors
8 GB RAM
3 7200rpm drives
LAMP on drive 1
Moodle datadir on drive 2
MySQL datadir (db) on drive 3
I haven't messed with Apache at all yet aside from changing the php.ini max post setting to allow for larger uploads. Guess I need to get in there and tweak it a little...or not?
Any observations and or feedback will be appreciated.
Sincerely
Jason
I ended up reverting to Ubuntu 7.04 because newer versions did not implement the fan control properly (windfarm controls) and I was getting sudden shutdowns, which, of course, won't do.
So, back in 7.04 now and happily stable. I have changed MySQL settings as follows to hopefully speed it up under quizzes:
default-storage-engine=InnoDB
innodb_buffer_pool_size = 750M
innodb_log_files_in_group = 2
innodb_log_file_size = 256M
innodb_file_per_table = 1
I'm wondering however, if I really need InnoDB as default and if I could just convert the tables associated with a quiz to innodb to achieve the efficiency I'm looking for....?
It seems completely unrelated but I've noticed now that a block I have on the front page of our site (MyCourses) is loading much more slowly since I've made these changes. This was an issue brought up by the developer (here) but under MyISAM on a machine that had less RAM and less processor power, I never noticed a significant lag. Maybe I've gained performance in one area and lost a little elsewhere.
Also, I did go ahead and install APC as was suggested. Wow! Surprised and how much more quickly pages load!
Here are my system specs (also for informational value)
Mac XServe G5 Dual 2.3Ghz processors
8 GB RAM
3 7200rpm drives
LAMP on drive 1
Moodle datadir on drive 2
MySQL datadir (db) on drive 3
I haven't messed with Apache at all yet aside from changing the php.ini max post setting to allow for larger uploads. Guess I need to get in there and tweak it a little...or not?
Any observations and or feedback will be appreciated.
Sincerely
Jason
One thing I have done to help when starting my quizzes is to use a fairly long but simple password for the quiz. I have it pre-written on the chalkboard, but covered up until all the inevitable admin/roll call/questions have been taken care of. Due to different typing speeds and accuracy problems, it usually takes up to 2 or sometimes 3 minutes for all my students to hit Enter, which of course cuts down on concurrency. 
The simple ideas are the best. I like it.
We also use another practice: the teacher doesn't even show the password to students and he/she (sometimes with the help of someone else) goes to every student's computer and inserts the password, one by one. That way students will never know the password and they will start the quiz gradually.
Cheers!
Cheers!
I just want to update this discussion with some information about which path we actually followed. We are about to improve our moodle architecture and hardware (beginning of 2011) and we are waiting anxiously by Moodle 2.1.
Meanwhile we simply extended quiz form with some new fields in order teachers may provide us some information about the quiz (date/time/number of studends attending/etc). We advice teachers to fill those fields so it automatically creates and sends us an .ics file in order we can easily detect exams scheduled for the same period, or exams involving too many students. It is also useful because we are aware of peak periods and we can monitorize moodle behaviour during those periods.
Any comments or updates on this issue are very welcome.
Cheers,
susana
That sounds like a useful extension. I think it is a bit to specialist to go into the standard release, but are you able to share the code? It might be useful for other people.
Sure I can share it. It was useful for us, so it might be useful for others. But it is not a big deal... it is just a hack, not a plugin... I will compile it add a readme file with instructions and open an issue on tracker as suggested.
Hi,
Here's the link to the tracker:
http://tracker.moodle.org/browse/CONTRIB-2604
As I said before this is just a hack (a sub-form in moodle quiz form)... (you have to follow some steps I provided in a readme file in order to "install" it)
Please let me know if you have some trouble using it.
Cheers,
susana
I will dare to say what nobody else has. Are you running on a Windows server? I have seen almost exactly the levels of performance you are seeing on average "commodity" servers running Windows. I'm sure it can be done on Windows but changing to Linux gets you an order of magnitude better performance with all else being equal.
> I will dare to say what nobody else has. Are you running on a Windows server?
That's why I stressed on a "state of the art server".
That's why I stressed on a "state of the art server".
Omigawd, I once saw a system where Apache and MySQL were on Linux, but they were using a Windows OS NAS - they were getting all sorts of weird permission errors from an automated batch backup and restore block we were building for them.
We had the hardest time debugging that (remotely) b/c they kept telling us they were all LAMP, and from what they let us see (for 'security purposes' they let us see very little) everything was configured correctly. It took a guy flying across the ocean and going into the client's data center to discover that there was a Windows box in the mix, and that it was configured incorrectly.
We had the hardest time debugging that (remotely) b/c they kept telling us they were all LAMP, and from what they let us see (for 'security purposes' they let us see very little) everything was configured correctly. It took a guy flying across the ocean and going into the client's data center to discover that there was a Windows box in the mix, and that it was configured incorrectly.