Quiz Performance Issues

Quiz Performance Issues

by Colin Matheson -
Number of replies: 9

Hello,

Our Moodle site went down due to our server being pushed to high CPU and RAM (which then meant the server was using swap) which then slowed the performance and we ran out of database connections. It appears this happened because a bunch of quizzes were submitted all at once. I remember reading that quiz submissions were taxing on the server, but I thought that spacing the quiz over multiple pages could help reduce the load at the time of submission. So even though it was a 35 question quiz it was separated onto 7 pages of 5 questions, so that should have been handled 5 at a time. Am I correct in that thinking? Are there any tips for making quizzes more lightweight? Or tips for tuning the server for simultaneous quiz submission? Our server does well except for when many of our teachers give Friday morning quizzes. Thanks for any guidance,

Average of ratings: Useful (1)
In reply to Colin Matheson

Re: Quiz Performance Issues

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Which version of Moodle?

How many is "a bunch of quizzes"?

Splitting a question into pages only helps as the students move around the quiz. Submit all and finish has to touch every question however the quiz is laid out.

Which question types?

Average of ratings: -
In reply to Tim Hunt

Re: Quiz Performance Issues

by Colin Matheson -

Moodle 2.7+ (Build: 20140613)

A bunch is 100-200 being submitted in a 5-10 minute window (some of the quizzes had 35 questions, some were 5-10 questions). The server is virtual server on a blade. The server has 32 GB RAM and 4 Xeon E5-2620 cpu's (8 threads).

Multiple choice questions - quiz set to deferred feedback.

Average of ratings: -
In reply to Colin Matheson

Re: Quiz Performance Issues

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

So, worse case (5 mins / 200) is one every 1.5 seconds. I would guess that is perfectly doable with the right amount of hardware. (If you just submit one quiz attempt, at a quiet time, does that get processed in under 1.5 seconds?) I don't know if what you have is enough hardware for that scenario.

When it died, do you have any monitoring to show what it was that ran out? CPU / Memory / Disc IO / Apache workers / ... ? (This is the kind of thing that a tool like Nagios can monitor.)

Have you ever done any load-testing on your set-up? If so, how many quiz attempts could it handle in your testing?

Average of ratings: -
In reply to Tim Hunt

Re: Quiz Performance Issues

by Colin Matheson -

I will check with our server admin to see what exactly maxed out. My belief is, it was RAM and then database connections. We do have Nagios installed, but I am not sure if it is monitoring all of those sub-aspects. 

How do I measure the quiz processing time. Just start at clicking submit all and end when the results page loads?

We haven't done any load testing. What tools are there for load testing a bunch of quiz submissions?

Thanks a bunch for your help,

Average of ratings: -
In reply to Colin Matheson

Re: Quiz Performance Issues

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Timing one page loading

The crude way to measure performance of anything in Moodle is to turn on Site administration -> Development -> Debugging -> Performance info. That puts some performance stats in the footer of every page.

(You probably don't really want that. At the OU we put a little hack in the code to turn that option on when you are logged in as admin, but not otherwise.)

Note that that just shows the time taken to render the HTML, not the time that page appears to users, with all the CSS and JavaScript downloaded.

So, another approach is to use your browser's developer tools. The network tab will give you a timeline of the page loading, including all resources.

Load-testing

There are two approaches to this that I know.

The most common it to use JMeter to simulate the load. (There are other similar tools, but JMeter (https://jmeter.apache.org) is the one we & others use, and people have shared their scripts.) There is now a script-generator built into Moodle. See http://demo.moodle.net/admin/tool/generator/maketestplan.php (log in with the admin account) but as far as I can see that does not support Quiz yet. It is perfectly possible to create the JMeter script by hand. JMeter has an option to record as you go through the site in your browser. That is a good start that you then need to edit. The tricky bits are getting users logged in (use a Cookie manager and simulate completing the Moodle login form) and submitting the quiz (you need to use regexes to get the sesskey and necessary ids out of the page, so you can submit the responses.)

The other option is to to use the back-end that the Behat functional testing uses. You can use that to drive  a web browser to simulate a number of users. We did that recently and I will try to share my scripts when I get a moment. (I don't have them here.)

Profiling

You did not ask about this, but once you have found something that seems much slower than it should be, profiling can be a good way to work out what the problem is, even if the problem is with your server configuration, rather than a bug in the Moodle code. See http://tjhunt.blogspot.co.uk/2013/05/performance-testing-moodle.html. Profiling will show you if the problem is Moodle spending all its time waiting for the database, or the cache back-end, etc.

Average of ratings: Useful (3)
In reply to Tim Hunt

Re: Quiz Performance Issues

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Here is the script I said I would upload. Needs to be saved in the top level of your Moodle site, then run form the command line. To simulate a lot of load, we ran it multiple times simultaneously from several computers. The script needs to be edited to work with your Moodle site.

Average of ratings: -
In reply to Tim Hunt

Re: Quiz Performance Issues

by Colin Matheson -

This is great. Our site got hammered again today, so we will be looking into where the issue is.

Average of ratings: -
In reply to Colin Matheson

Re: Quiz Performance Issues

by Peter Halverson -

I just want to add that I don't know exactly why my Moodle site gradually slowed and finally crashed, or why optimizing the SQL database helped.

Perhaps Moodle's SQL database tables get fragmented, like a hard disk drive gets fragmented?

Maybe one of the experts can comment.

In the meantime, Hostmonster's tech support people pointed me to this article on the subject.

http://www.codeproject.com/Articles/34372/Top-steps-to-optimize-data-access-in-SQL-Server

---Peter

Average of ratings: Useful (1)
In reply to Colin Matheson

Re: Quiz Performance Issues

by Peter Halverson -

Hi,

I have experienced a gradual slowdown of my moodle site over a several month period until finally it crashed much the way you describe.

I found that optimizing the SQL database fixed the problem.  Dramatic improvement in performance.

Here are my notes on how to do it.  They apply to a Hostmonster VPS account with a cpanel interface.

  1. BACKUP the SQL database.   Do not use cpanel to do it, as I don't trust cpanel backups.  Instead do it the command line way.  (I can address how to do that separately)
  2. Login to cpanel
  3. Go to MySQL Databases
  4. Click "Repair Database" next to your database    (does not do any actual optimization)
  5. Go to PHP MyAdmin
  6. Click your database on the left side.
  7. Scroll to bottom, click "Check all" in order to select everything and click "optimize table" "with selected"
  8. Note that if you a gateway timeout error, you can optimize individual tables, or do 1/2, then the other half.

I hope this helps.  Please be careful; have a good full backup just in case things go crazy.

Peter Halverson

PS:  The problem with cpanel backups is that if the amount of data is too large, the backup is truncated without any error messages.  If that happens, you have a useless backup and you don't know it.

Average of ratings: Useful (1)