Hardware and performance

The quiz function of moodle 2.3.2 seems to put quite a strain on a server

 
 
Picture of curt bixel
The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

20 students using the quiz function of moodle 2.3.2 seems to put quite a strain on a server.  Not sure why. 

(Seperated from the thread http://moodle.org/mod/forum/discuss.php?d=211983#p924844 and renamed. Edited by Visvanath Ratnaweera - original submission Saturday, 6 October 2012, 8:32 PM)

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Well, you need to try to find out why:

  • What seems to be the limiting factor on the server, is it
    • CPU
    • Memory
    • Disc I/O
    • Network
    • ...
  • Does that load seem to be being caused by the
  • Have you tuned your server, so that it is making the most efficient use of the available memory, etc?

You need some monitoring tools, even if it is just running the top command while sutdents are attempting the quiz.

 
Average of ratings:Useful (1)
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Tim,  thanks for the response.  I have recently become interested in what optimization can be done using ONLY what is available through the admin on moodle.  

During the past few weeks, I have:

  1. Asked the advice of the sales department at my host for which package would be most appropriate for my moodle installation
  2. Paid for that package.
  3. Crashed their server.
  4. Followed the advice of the tech department at my host and learned how to install a PHP accelerator onto a server.
  5. Learned that you cannot install a PHP accelerator onto a shared server.
  6. Followed the advice of the tech department at my host and learned how to optimize MySQL.
  7. Learned that I don't really have access to the files on a shared server that I would need to access if I were actually going to be able to optimize MySQL.
  8. Learned that a shared server is probably a dumb place to try to host moodle.
  9. Verified on the moodle forum that a shared server is probably a dumb place to try to host moodle.
  10. Read everything on moodle forums about optimization possible through the moodle admin.

 

That being said, I am pretty much settled on eventually moving to either an upgraded server plan or possibly a host that specialized in moodle so that hopefully, their server already has a PHP accelerater installed and their databases are already optimized to host moodle.  

In the meantime, I am really quite interested in using this as an opportunity to learn more about what makes the moodle quiz moodle so resource intensive, and what modifications can be made here to reduce the load.  

Here is what is happening:

I have one admin (me), one course, 90 students.  When 20 of them take a quiz in my classroom at the same time, we are logging "hits" on the server at the rate of over 13,000 hits per hour.  

Details of the quiz.

  • 20 questions long
  • questions selected at random from a test bank of 50 questions
  • 10 of these questions include an image (reasonably sized)
  • Each question shown on its own page.
  • Immediate feedback set to show everything.
  • Quiz navigation block shown

Here is what I am going to try next week:

  • 20 questions long
  • questions selected at random from a test bank of 50 questions
  • 10 of these questions include an image (reasonably sized)
  • All questions shown on same page.
  • Immediate feedback set to show everything.
  • Quiz navigation block shown.

Other changes made:

  • Turned all filters off
  • Turned text caching off
  • Set the main navigation block to "show on course main page only."  This was showing on all pages by default, so I was concerned that maybe the server had to work to rebuild this every time a student checked or submitted a question.
  • Changed setting on Server session handling. – Turned “Store session information in database” off.
  • Supplied the path to "du"
  • Deleted many of the "blocks" I will never use.
  • Edited the lib/accesslib.php to increase the context cache max size from 2500 to 5000.
    • Edited lib/accesslib.php  CONTEXT_CACHE_MAX_SIZE', 5000. (It used to be 2500)
  • I was not able to find the place to change the setting referred to on the moodle forum that read as follows:
    • If there are performance problems loading course pages, check the Resource module settings. The setting resource_filterexternalpages is known to slow-down course pages and should be set to 'No' for better performance.

I would appreciate any feedback on the changes I have made as well as any other suggestions.  

If anyone feels compelled to tell me to change my hosting plan, I will be happy to listen to this advice after I learn more about the affects of different moodle admin tweeks.  

 By the way, please be gentle with your feedback, I am certainly no moodle admin or server admin.  I am just some guy trying to teach a science who has not had luck with my district's hosting of moodle, so I am looking to host my class myself.  I have certainly learned a lot so far reading all of your posts on the moodle forums.  

 

 

 

 
Average of ratings:Useful (3)
Picture of Marcus Green
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group Particularly helpful Moodlers

The quiz is probably the most demanding of all Moodle modules, and having worked on the code of a Moodle question type,  my view is that is demanding because it does an awful lot of stuff, and does it rather well. I have written my own quiz engine from scratch in the dim past and my code did a tiny fraction of the work, as I skipped a whole bunch of possibilities. 

You are right that as a general rule shared hosting will not cut the mustard. However I have had 20 students taking a quiz simultaneously on cheap shared hosting  (£50 a year) with acceptable performance. This did rather surprise me as I thought Moodle 2.0 would be the code of doom for me.

However once or twice it has slowed to unusable as you never know what else is happening on shared hosting. My guess is that turning off text caching would actually create a performance hit.

The more information you provide, the gentler we will be (grins). I didn't know if you were talking 20 users or 100 users taking a quiz. 20 Users should be very doable on a modest set of hardware. Have you got theme designer mode turned off? It would be interesting to work out if the bottleneck is the php/web server or the db. You can always run them on two different machines.

 

 
Average of ratings:Useful (2)
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Marcus, thanks for the reply.  

My largest class has 25 students, so 25 kids simultaneously taking the quiz should be about the maximum.  I did check to make sure theme designer was turned off.  My suspicion is that the problem is the repeated calls to the database.  That is why I turned off the text cache, as it uses the database.  (A little counter-intuitive, but it is possible that CPU is plentiful for my situation and the text cache just tries to push more stuff through the bottleneck)  

I will take the suggestion below and turn on debugging and see if that sheds any light on the situation.  

 

 
Average of ratings: -
Picture of Michael Haskell
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

I've seen quiz performance issues that have negatively impacted our users in 2.3.3 as well. I'll post what we experienced here soon.

 
Average of ratings: -
Picture of S Moodle
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

I've seen quiz performance issues that have negatively impacted our users in 2.3.3 as well. I'll post what we experienced here soon.

I've been running with 2.4 standard for a while now.  There's some improvement but not a drastic one for the quiz module.

I know that putting ADAPTIVE QUIZ (NO PENALTIES) is a disaster when it comes to slamming the server -- as much as I like having formative quizzes, a classfull of 30 students all pressing that CHECK button will kill any server.

 

 
Average of ratings: -
Picture of Michael Haskell
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

There's gotta be a solution for this.  Locking up at the most inopportune time (middle of a quiz) give us terrible PR and really disrupts academic use.  

I notice there are a number of row-locks in our database around the time quiz's run. (we're running cron every 15 minutes and I ses a # of row locks and the row lock wait time increases dramatically after quiz use.)

I'm contemplating upgrading to 2.4 next term to address quiz performance in particular, but if you're not seeing drastic improvements I better look into alternative solutions as well.  

Just a question, how many buffer pools and what's the size of your buffer pools in mysql?

 

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

I am really amazed that any commercial host exists that does not have a PHP accellerator already installed on their servers. That just makes no sense to me.

Given that there is no PHP accellerator, one source of load is that the Moodle quiz system is a complex piece of software. To keep the code manageable, it is broken down into lots of separate files. With a PHP accellerator, that does not matter. Without a PHP accellerator, loading and parsing all those files might have a noticable impact on server load.

There is one part of the quiz system that is less efficient than it should be. It ends up doing a few database reads to load each question, where it should probably just be doing a single read to load all the questions in one batch. This is difficult, however, becuase each type of question is different, and stores its data in a different way. Once the caching system has been added to Moodle 2.4 (MDL-25290) then it should finally be possible to lay this long-standing problem to rest.

Finally, there is one way in which quizzes just have to place more load on the server than other activities. If a user is reading a forum, then they are just reading. There is no significant new data written, unless they make a new post. With a quiz, every time the student answers a question, that data has to be processed and stored. Storing data takes more server effort than just loading data.

Having said all that, your load seems really modest, so I am surprised you are having problems.

Finally, if you are trying to undestand performance problems, the go to Admin -> Development -> Debugging, and turn on the option "display performance information in the footer" (or whatever it is called). That puts various useful metrics at the bottom of each page (e.g. server time taken to create the page, number of database queries, number of separate code files loaded, etc.) compare quiz pages to other pages, and let us know if you see anything anomalous. (You will probably want to turn that option off again once you are done investigating, so students don't see the output.)

 
Average of ratings:Useful (4)
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Got this at the bottom of the first page of the quiz.  This is when all 20 questions show up at the same time to begin the quiz.  

 

0.568524 secs

RAM: 36.8MB

RAM peak: 37MB

Included 507 files

Contexts for which filters were loaded: 1

Filters created: 0

Pieces of content filtered: 64

Strings filtered: 0

get_string calls: 1445

strings mem cache hits: 1687

strings disk cache hits: 164

DB reads/writes: 281/65

ticks: 57 user: 42 sys: 4 cuser: 0 csys: 0

Load average: 18.23

Session: 16.8KB

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Most of those numbers look OK.

Most of those numbers relate just to the current page being loaded. There is one number, "Load average", which relates to the whole sever. That is how many different processes are actively trying to run on the server. 18 is  high, unless the server has 18 CPU cores which is unlikely. It is more likely to have 2 or 4. So, that confirms that the server is under heavy load at this time.

However, the other numbers for this particular page are mostly OK. 0.5seconds to render the page is not great, but not bad. I am amazed how low the memory usage is.

The 281 DB reads is the "less efficient than it should be" bit, thought it could be worse. I have seen really badly written pages where it is in the thousands.

The 65 DB writes is the unavoidable "there is a lot of data to process and store".

 

Regarding possible solutions. It is worth trying to turn on persistent DB sessions. It is quite safe to try that. If it causes problem, you can just turn it off again, and things will go back to the way they were.

@curt bixel - turning off text caching is often a good idea. On many servers, it takes more effort to connect to the database and look in the cache again than it does to just re-do the filtering.

 
Average of ratings:Useful (2)
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

By the way, someone at the hosting service suggested I see if moodle is using a "persistent" connection.  I read somewhere that moodle does by default, but in the config file the only thing I could find was this:

dbpersist' => 0

I didn't really want to mess with it, as it seemed like one of those little commands that could pretty much tank the whole system unless you know what you are doing.  (and I really don't......)

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Note that, while I was visiting Moodle HQ a couple of weeks ago, I was able to sit down the Sam Hemelryk and implement caching of question definitions using the new caching system. This should make a bit improvement to quiz performance, but we have not actually done any testing to prove this yet. See MDL-34399 for more details. Note that this has been integrated, so this will be included in Moodle 2.4 when it is realeased.

If anyone can do any performance testing on this, that would be really helpful.

 
Average of ratings:Useful (3)
Picture of Michael Spall
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group Testers
Tim, I would like to help test performance. I have used JMeter in the past to test performance. Do you have a specific performance test that you would want run.
 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Well, any sort of test of different users simultaneously attempting a quiz, and compare performance of 2.3.x and 2.4 beta.

Right now, the details probably are not important. Any sort of test like this is better than nothing. Once we have some data, we may then want to refine things, but for now there is a vacuum to fill.

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group Particularly helpful Moodlers
Hi Curt

I'm sorry that it turned out to be quite a journey for a (non-IT) teacher!

Anyway, as others have pointed out, Moodle is a demanding piece of software, the quiz module is the most demanding component in terms of computing resources. I went through the whole discussion and realized that, except for Moodle 2.3.2 I know nothing about the platform:

"Please include as much background information as possible about your hardware, the operating system, the web server, the database server, the PHP scripting framework, etc. If your machine is virtualized, also mention the host hardware, the operating system, the virtualization technology used and the resources assigned to the guest OS. If you use a hosting service, mention either the hosting provider and the package or the resources they've allocated to your Moodle instance."
(from the intro to this forum)

Since yours is a shared (virtual) hosting, you may not be able to translate that into real hardware. In that case a benchmark would help. Here are two possibilites:
- "Performance perspectives - a little script" http://moodle.org/mod/forum/discuss.php?d=57028 (linked from the performance documentation in the intro to this forum)
- "UnixBench" http://moodle.org/mod/forum/discuss.php?d=200362

Then you need to monitor the key resources during peak use to find out what maxes out. For a choice, see "How do you monitor your Moodle server?" http://moodle.org/mod/forum/discuss.php?d=192162.
 
Average of ratings: -
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Visvanath,  thanks for your help.  Looks like I crashed their server again......

But, here is some useful info.  It may take me a bit to get the rest of the info you requested above.  By the way, I am hosted at InMotion hosting on the "business pro" package.  At least I was..... until today.... now I am on their VPS 1000 package.  

Anyway, here is a link to some good stats:

https://secure126.inmotionhosting.com:2083/awstats.pl?config=curt-bixel.com&ssl=&lang=en

And if that does not work, here are some pictures....

 
Average of ratings: -
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

By the way, I have requested that they put APC on my server as this is now possible.  They did have PHP acceleration on the shared server, but they say this will help make it even faster.

 
Average of ratings: -
Picture of S Moodle
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

As you can see the blame is on mod/quiz/attempt.php.  That script is not CPU efficient and will slam your host when all of the students press the ATTEMPT QUIZ button at the same time.

As I suggested, tell half your class to the start the quiz, wait 10 seconds, then tell the other half to start it.

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 
Average of ratings: -
Just wondering . . .
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group Particularly helpful Moodlers

We've had similar experiences.  We had tutors saying "ready, set GO" to their class who have their fingers poised over the mouse button.  

We sometimes gave written bits first so students were naturally spaced out.

We also started from scratch with students logged out which naturally spreads them out a bit.  And were liberal with an extra 5 minutes at the end as a buffer.

You could have another activity first - make a post??

I think sometimes I could hear the server groaning.

 
Average of ratings: -
Picture of S Moodle
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Yeah, the quiz module is really hard on servers.  You have to prevent all of your students from pressing the button at the same time (which is within 10 seconds of each other). 

I can lock out groups of students to prevent them from starting the quiz at the same time using classroom control software like SynchronEyes. Basically lock out the entire class, and then give only 2 people access, then 4, then 6, etc until everyone has started the quiz.

Once the quiz is started, it would be rare that all students hit the button at the EXACT same time.

It would be awesome if there was batch system which rendered a page one at a time rather than Moodle simply allowing everyone to slam the server at the same time.

 
Average of ratings: -
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Here is some more info on my server...

 

Main Domain [example.com
Home Directory /home/curtbi5
Last login from [X.X.X.X]
Disk Space Usage 1666.62 / ∞ MB
 
Monthly Bandwidth Transfer 2412.45 / ∞ MB
 
Email Accounts 1 / ∞ 
 
Subdomains 0 / 100 
 
Parked Domains 0 / 26 
 
Addon Domains 0 / 5 
 
FTP Accounts 0 / 2000 
 
All SQL Databases 2 / 50 
 
MySQL Databases 2
MySQL Disk Space 49.06 MB
Hosting package Power
Server Name ecbiz126
cPanel Version 11.32.3 (build 23)
Theme x3
Apache version 2.2.22
PHP version 5.3.13
MySQL version 5.5.24-cll
Architecture x86_64
Operating system linux
Shared IP Address [X.X.X.X]
Path to sendmail /usr/sbin/sendmail
Path to Perl /usr/bin/perl
Perl version 5.10.1
Kernel version 2.6.32-279.2.1.el6.x86_64
cPanel Pro 1.0 (RC1)  

("Cleaned" the IP addresses and the domain. Edited by Visvanath Ratnaweera - original submission Tuesday, 9 October 2012, 12:37 AM)

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
Group Particularly helpful Moodlers
Hi Curt

The stats in https://moodle.org/mod/forum/discuss.php?d=213259#p929649 are not of much use. They are good to compare external factors like the development of number of visitors/visits, which pages are called often, etc. But are averaged over a long time, an hour or more.

What I was looking for was point loads in the subsecond to few seconds region and how key parameters insidle the server, like CPU usage, RAM usage, etc. fairing. Check the last section of https://moodle.org/mod/forum/discuss.php?d=213259#p929392 again.

But in the other subthread Tim has pointed out that the Load average is far too high. It should be ideally less than the number of CPU cores. Your figure in excess of 18 means, the server is over booked. That is a well known problem with cheap shared hosting.

The "Power" and "Pro" refers to these hosting plans http://www.inmotionhosting.com/hostingplans.html, right?

Yes, a PHP accelerator might help. Report in the other sub-thread.
 
Average of ratings: -
Picture of curt bixel
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Looks like it might be interesting to run UnixBench on my site.  I have found the program, but not really any installation directions I can quickly make sense of.  

I have found these directions, but not sure where I would enter these commands, and I figure you might have to actually copy the software onto the server before typing in these commands.  Any help you have would be  much appreciated.  This is kind of fun.  

By the way, they have not installed APC yet, but they assured me that my new package, the VPS1000 (InMotion Hosting) would have no problem with my load, so today, I had all 25 students log on and take the quiz as fast as they possible could, and I started all of them simultaneously.  There were no problems.  Page load times were between 1 and 1.5 seconds for the most part.  Not lightning fast, but well below the threshold where it would be annoying to the students.  

 
Average of ratings: -
Picture of S Moodle
Re: The quiz function of moodle 2.3.2 seems to put quite a strain on a server
 

Other changes made:

  • Turned all filters off

I rely heavily on the Latex algebra filter, so I can't do that.

  • Turned text caching off

I'm not sure how that would decrease CPU time.  Go on?

  • Set the main navigation block to "show on course main page only."  This was showing on all pages by default, so I was concerned that maybe the server had to work to rebuild this every time a student checked or submitted a question.

Where can I find this setting? That's probably smart.  If PHP code is being excuted every time the page is loaded to generate this dynamic menu, that might help a bit.

  • Changed setting on Server session handling. – Turned “Store session information in database” off.

Where can I find this setting?  If the session is not being stored in the database, then less SQL is being executed, which means less CPU.  I wonder what disadvantage this does.  Is the session info stored completely in the browser's viewstate? It seems then you've saved SQL cycles, but then simply increased PHP cycles.

  • Supplied the path to "du"

Huh?

  • Deleted many of the "blocks" I will never use.

reduces php cpu from rendering HTML.  Could help.

  • Edited the lib/accesslib.php to increase the context cache max size from 2500 to 5000.

How will this help?  Esp with quizzes?

  • I was not able to find the place to change the setting referred to on the moodle forum that read as follows:
    • Edited lib/accesslib.php  CONTEXT_CACHE_MAX_SIZE', 5000. (It used to be 2500)

I have no idea what this is about.  What is the trade off?

    • If there are performance problems loading course pages, check the Resource module settings. The setting resource_filterexternalpages is known to slow-down course pages and should be set to 'No' for better performance.

I would appreciate any feedback on the changes I have made as well as any other suggestions.  

If you put all of the questions on one page, I think that would initially slam the server causing a massive CPU spike since PHP would have to generate thousands of lines of HTML at the same time.

You could tell your students to start the quiz in chunks of 5 students.  Have one group start the quiz, then wait 10 seconds, have the next group start and wait 10 seconds, etc.  Then you'd avoid the cpu spike.

big grinWhat would be awesome is if moodle had a built in staggered quiz delay, so that it would prevent everyone from being able to start the quiz at the exact same time.  You could set the quiz delay to 3 students, which means that only 3 students could start the quiz at the exact same time, with a delay of say 5 seconds.  This has to be a great idea! big grin

 
Average of ratings: -