quiz module performance load questions/problems

quiz module performance load questions/problems

by Justin Haaga -
Number of replies: 4
Our institution has periodic problems with giving quizzes causing the server to max out it's cpu resources which inturn causes http requests to "timeout" until the grading is complete, typically 1-5 minutes.

Two questions:
1) What do you experience with quiz load+normal load?
2) Can we separate the quiz module onto a separate server?

Been running moodle in production for just over 6 months. Running 1.9.3 (20081029).

School-
300 students, 60 courses only present qtr, old qtrs are archived off
concurrent connections - aprx 15-30

Config is as follow:
apache2, php5, xcache (webserver/moodledata) - Xeon 2ghz quad core, 8GB mem
mysql 5.0, win 2003 server (db server) - Dual Xeon 2.33ghz, 4GB

Benchmark results are:
Moodle System Benchmarks
Note for Moodle 1.7 and above:
Disable the record cache (Site Admin -> Server -> Performance) for realistic results!
Processor performance
Function calls 1422000 1495000
Regular expression replaces over 1KB of text 13000 13000
Disk performance
16KB files read from disk (cache) 13900 13900
16KB files written to disk (cache) 4500 4500
Database performance
Get_record calls on the course table 970 1030
Insert_record calls on the course table 330 330
Update_record calls on the course table 80 80
Maximum concurrent users (approx): 77

apache config
<IfModule mpm_prefork_module>
ServerLimit 350
StartServers 180
MinSpareServers 180
MaxSpareServers 190
MaxClients 300
MaxRequestsPerChild 2000
</IfModule>

On avg a quiz has 20-40 students.

It really has to do with when the students submits the quiz for grading. If 90% of the students submitted them at the same time it causes a rush of grading. We have this problem periodically and it always is the same root cause. It's hard to predict because it depends on student completion, and how long the quiz is and how many multiple choice questions it may have.

Average of ratings: Useful (1)
In reply to Justin Haaga

Re: quiz module performance load questions/problems

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I appreciate that this isn't what you probably want to hear but we have had some clients in very similar situations.

We switched them over to Linux on the same hardware and the problem was gone. Windows seems to have a very poor threading model and it degrades horribly under any kind of load. Moodle can be quite a big load at times!

Perhaps there are some Windows gurus out there who can help tune this but, for my money, Windows is your fundamental problem.

EDIT:
I just noticed you've been talking about this is Hardware and Performance. It just confuses things (for you mostly) spreading your posting about!

http://moodle.org/mod/forum/discuss.php?d=110752
Average of ratings: -
In reply to Howard Miller

Re: quiz module performance load questions/problems

by Justin Haaga -
yes, two threads. sorry for the confusion but really had the two new question above that I want to try to get answered.

To clarify, apache2 is running on a debian-etch box. The db is on a windows 2003, mysql 5.0, box. Separate hardware that suppose to provide max performance.

I've thought about the windows DB box being a problem, but i am just not convinced.

We can outrule the DB being the problem in two ways:
1) performance monitor on the DB is low, we see the apache2 threads hogging the CPU on the linux box causing http requests to be slow
2) we have a archived site for previous qtrs running as a separate apache2 web site on the same current website, but it's db (mysql5.0) is located on the same debian machine as the current and archived websites.

The archive site is slow when the current qtr moodle instance is slow due to the cpu resources being maxxed.

Remember that I can benchmark the site using apachebench or siege and the apache2 and db is able to handle it just fine. It's something specific with the quiz module that is causing unusually high CPU utlization, or it's just naturally a hog.


Average of ratings: -
In reply to Justin Haaga

Re: quiz module performance load questions/problems

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
That's also my experience. The database is rarely the bottleneck - it's Apache. I take it you have munin or nagios (or just sat and watched 'top') to see what is going on?

If your setup is otherwise very simple, lighttpd can be worth a bash but it becomes a pain for anything more than a very basic site configuration (e.g. it doesn't work properly with mnet).

Moodle in general (and quiz in particular) is a huge lump of software. It isn't a criticism, just something you have to plan for. It can soak up RAM and land you in the dreaded swap faster than you'd expect.
Average of ratings: -
In reply to Howard Miller

Re: quiz module performance load questions/problems

by Justin Haaga -
Running top and ipmonitor(snmp like nagios) to collect the cpu stats.

I'll have to look into lighttpd.

Any ideas on separating the quiz module onto a separate server with separate resources?
Average of ratings: -