Does this data point to where our bottleneck is.

Does this data point to where our bottleneck is.

by Rob Monk -
Number of replies: 10

We run into the same problem near the end of each semester. When we have over 100 users all doing quizzes at once the system grinds to a halt. Page loads of over 30 seconds and frustrated staff and students. Any advice on the way forward would be great.

Average of ratings: -
In reply to Rob Monk

Re: Does this data point to where our bottleneck is.

by Usman Asar -
Picture of Plugin developers Picture of Testers

Rob, what is your current setup in terms of hardware?

In reply to Usman Asar

Re: Does this data point to where our bottleneck is.

by Simon Brock -

Both the Database server and web server are running on separate VM's both have been allocated 2 CPU's and 8GB Ram

In reply to Simon Brock

Re: Does this data point to where our bottleneck is.

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Simon + Rob (I assume you work together and Simon doesn't just magically know the answer there ;)),

Could we have some further information, including:

  • what version of Moodle you're using;
  • what PHP version;
  • which PHP accelerator you're using;
  • how much memory you've given that accelerator;
  • what tuning you've done on your database server;
  • what kind of filesystem backend you're using for your VM infrastructure.

If you can additionally look at tools like top, dstat, vmstat, etc that would also help. I personally prefer dstat:

dstat -Tmcndls

Best wishes,

Andrew

In reply to Andrew Lyons

Re: Does this data point to where our bottleneck is.

by Simon Brock -

Hi Andrew,

Yes we do work together,

here is some of the information

    what version of Moodle you're using; Currently running 2.4.6+
    what PHP version;    PHP Version 5.3.3-1
    which PHP accelerator you're using;  eAccelerator 0.9.6.1
    how much memory you've given that accelerator; 16,777,144 Bytes
    what tuning you've done on your database server;  
database settings
skip-external-locking
key_buffer              = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
myisam-recover         = BACKUP
max_connections        = 500
table_cache            = 512
query_cache_limit       = 1M
query_cache_size        = 36M
long_query_time = 4
innodb_buffer_pool_size = 6144M
max_allowed_packet      = 16M
[isamchk]
key_buffer              = 16M
    what kind of filesystem backend you're using for your VM infrastructure.
our VMware setup consist or two hosts connected to SAN

 

thanks


regards,

Simon

In reply to Rob Monk

Re: Does this data point to where our bottleneck is.

by Albert Ramsbottom -

Your load average is extremely high


I processor should have a constant maximum of 1.0, OK all apache servers will peak and go to 3 or 4 but yours is showing 12.65


This suggests that you need more processors or cores to cope with the quiz when it happens. Not sure what environment you are using but if you are virtualised you can ask for more processor cores when a quiz is scheduled to take place

In reply to Albert Ramsbottom

Re: Does this data point to where our bottleneck is.

by Adam Durana -

A high load average doesn't always mean that the root cause of a system's poor performance is a lack of CPU resources.  I'm not saying this is or isn't the case here, but just that we can't know for sure either way based on just the load average data point.

In reply to Adam Durana

Re: Does this data point to where our bottleneck is.

by Albert Ramsbottom -

Looks like it in this case!

Have a look at iostat, hdparm or sar whilst this is happening, should give some useful info

 

Cheers

In reply to Albert Ramsbottom

Re: Does this data point to where our bottleneck is.

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Just to correct a common misconception, the load average is the number of items waiting to be processed over a 1, 5, and 15 minutes period (IIRC). Therefore load average is tied directly to the number of processor cores you have available. In this case 2, so anything over 2 suggests that there's a bottleneck.

It does not suggest that you need more cores, but that there is a bottleneck which is causing delays. Throwing more processor cores at the issue could make it worse. For example, if this is an IO issue, then throwing more cores could increase IO and further slow the machine down.

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Does this data point to where our bottleneck is.

by Albert Ramsbottom -

Yes no misconception here, which is why I suggested using some more advanced tools such as IOstat, hdparm and SAR

A quick look at these tools one should be able to determine where the bottleneck is, I bet 10 quid if another processor is chucked at it, it will make a huge difference, after all this is a quiz which are processor intensive.

Albert

In reply to Rob Monk

Re: Does this data point to where our bottleneck is.

by Adam Durana -

Do you have shell access to your web server that is running Moodle?  If you do, can you run the command `vmstat 5 30` on the server while you are seeing performance problems?  Post the output of the command here.  It will give us a better understanding of what is happening on the server.