Our Moodle server has had very high, in my opinion, CPU loads for a while now and I figured it's time to turn to the forums for some help. I've provided some screen shots to help with understanding the situation and below are some details of the server.
Yesterday, our Moodle server CPU load was over 90% and I was forced to reboot the machine. There was about 300 httpd's in the process list also.
Screenshot below was taken while there are only 10 users currently logged in.
Server:
CentOS 5.4
Dual Core 2.5 Ghz Processor
4 GB RAM
httpd.conf:
keepAlive Off
MaxKeepAliveRequests 100
KeepActiveTimeout 15
StartServers 50
MinSpareServers 5
MaxSpareServers 20
Server Limit 300
MaxClients 300
MaxRequestsPerChild 4000
We have change the statistics settings to keep stats for only 90 days from forever before, hoping that would bring the DB size down a bit.
I have read other forum topics and people have mentions installing eAccelerator, maybe settings up a proxy, or having the DB on a different machine. I'm not convinced this is what needs to happen as the machine specs we have are plenty for running a Moodle server, especially seeing the specs of other setups with no issues. I agree, eAccelerator would definitely help on any server though.
There has to be a configuration somewhere, but I think I'm just looking in the wrong places.
Any help would be appreciated!
Thanks!
Actually, i see no problem with the server stats on that screenshot.
Load average is excellent, memory usage is great, no swapping being done.
I run a similar spec environment on my webserver and under heavy load (quizzes with over 100 students, multiple assignment submissions), i often see load climb to over 30!! I am due for a HW upgrade though....
Regarding your post, what version of Moodle are you running?
Eaccelerator will definitely help bringing load down.
Only reboot if absolutely necessary - i.e plenty page timeouts, non-responsive actions etc.
CentOS (once tweaked properly) handles load quite nicely.
Cheers,
-n
Load average is excellent, memory usage is great, no swapping being done.
I run a similar spec environment on my webserver and under heavy load (quizzes with over 100 students, multiple assignment submissions), i often see load climb to over 30!! I am due for a HW upgrade though....
Regarding your post, what version of Moodle are you running?
Eaccelerator will definitely help bringing load down.
Only reboot if absolutely necessary - i.e plenty page timeouts, non-responsive actions etc.
CentOS (once tweaked properly) handles load quite nicely.
Cheers,
-n
Have you looked at this post. My issue ended up being eaccelerator was not really running.
Well, the screen shot I sent was from this morning where everything is working fine. Yesterday, we had maybe 30 people using Moodle, over 300 httpd's in the process list, and I had a ton of lag on command line. The server was just on it's deathbed. I ended up having to power cycle the machine.
I do not have eAccelerator installed as of yet. We will be trying that in the near future, but I wanted to take care of this actual issue before we put a bandaid on it.
I guess I will wait to see when the server is under heavy load again and try to get more SS's for you. I just thought I'd post some of the config and see if something looked wrong to any of you.
Are there any other config files I should take note of that would cause CPU usage to go through the roof if not configured correctly, as far as Moodle is concerned?
I do not have eAccelerator installed as of yet. We will be trying that in the near future, but I wanted to take care of this actual issue before we put a bandaid on it.
I guess I will wait to see when the server is under heavy load again and try to get more SS's for you. I just thought I'd post some of the config and see if something looked wrong to any of you.
Are there any other config files I should take note of that would cause CPU usage to go through the roof if not configured correctly, as far as Moodle is concerned?
Made a typo in my post.
Some of our quizzes have over 1000 students, not 100.
Some of our quizzes have over 1000 students, not 100.
I would not say eAccelerator was a bandaid. I would say a PHP cache was a requirement for Moodle.
I agree completely with Ryan. APC or eAccelerator is in no way a bandaid.
If Apache is spending too long serving requests and is unable to fulfill them as quickly as they come in, you'll see exactly the symptoms you describe - high load, many httpd processes. An opcode cache allows Apache to serve up pages without having to compile the PHP code for every request, allowing Apache to do the same work with fewer processor cycles.
Also, don't reboot the whole box if it's bogged down. That's swatting flies with a sledgehammer. Just restart Apache. It'll be much faster.
-Greg
If Apache is spending too long serving requests and is unable to fulfill them as quickly as they come in, you'll see exactly the symptoms you describe - high load, many httpd processes. An opcode cache allows Apache to serve up pages without having to compile the PHP code for every request, allowing Apache to do the same work with fewer processor cycles.
Also, don't reboot the whole box if it's bogged down. That's swatting flies with a sledgehammer. Just restart Apache. It'll be much faster.
-Greg
Maybe it isn't actually a bandaid, but I think the problem lies in that we're running MySQL off the same VM as Apache and since MySQL doesn't like to run on a VM anyway, maybe we should just move it somewhere else.
Once the database is moved over, we I'll install eAccelerator and then we should be in tip-top shape
I am attaching another SS with about 30-40 users taking a quiz from this morning between 10:30-10:35 they started... you can see the spike in CPU.
Once the database is moved over, we I'll install eAccelerator and then we should be in tip-top shape
I am attaching another SS with about 30-40 users taking a quiz from this morning between 10:30-10:35 they started... you can see the spike in CPU.
Moving the database onto dedicated hardware will definitely help but I don't think you should wait on the opcode cache, though. It really is a "turn it on and forget about it" situation. It probably took you longer to get these screen snaps than it would to get APC or eAccelerator running. There aren't any downsides and it takes almost no time at all to get it going.
As for your graph, I don't see anything that would be particularly concerning. It didn't max out the processor. It looks like it didn't even make it above 75%.
As for your graph, I don't see anything that would be particularly concerning. It didn't max out the processor. It looks like it didn't even make it above 75%.
We also had episodes of high cpu usage before we tuned OS parameters. After a considerable amount of load testing the specific moodle instance and tuning system parameters we were able to get much better performance from our server.
Send me an email. I would be interested in knowing some specifics about your install and explore tuning options. Perhaps we can post the changes made and results on this forum.
See: http://moodle.org/mod/forum/discuss.php?d=137873
Send me an email. I would be interested in knowing some specifics about your install and explore tuning options. Perhaps we can post the changes made and results on this forum.
See: http://moodle.org/mod/forum/discuss.php?d=137873