Server still running HIGH CPU

Server still running HIGH CPU

by Ryan Kelley -
Number of replies: 21

My moodle server is still running high CPU even after making several changes that have been suggested on the forums. My CPU remains high but the memory is still staying low.

We have 2200 students in our school. Right now we have 24-30 students on Moodle at a time but are going to train and require two buildings to use Moodle in the next couple of weeks.

The current setup is..

Moodle 1.9.5+ (Build: 20090923)
xampp 1.7.2 (PHP 5.3.0 MySQL 5.1.37)
Linux OS
VMware
3GB Ram
(2) Dual 3.2Ghz Processors

Performance
256M - Extra PHP memory
eaccelerator
Yes – Record cache
10 – Record cache TTL 10
50 - Int. cache max
no – memcached

My.cnf
skip-locking
key_buffer = 150M

Max_allowed_packet = 1M
table_cache = 512
sort_buffer_size = 2M
net_buffer_length = 8k
read_buffer_size = 1M
read_rnd_buffer_size = 512k
myisam_sort_buffer_size = 10M
query_cache_size = 250M
thread_cache_size = 512M
tmp_table_size = 512M
max_heap_table_size = 512M
query_cache_min_res_unit = 2M
query_cache_limit=10M
join_buffer_size=2M

I have Optimized and Repaired Tables in my database. The mysqltuner.pl show OK except for on fragmented table that will not go away.

Thank you,

Ryan Kelley
Network Engineer
Belding Area Schools

Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by Greg Lund-Chaix -
While it may not be related to this specific problem, it looks like you might be using MyISAM for your tables. I'd highly recommend switching over to InnoDB. In my experience InnoDB scales much better than MyISAM.
Average of ratings: -
In reply to Greg Lund-Chaix

Re: Server still running HIGH CPU

by Ryan Kelley -

Sorry its taken so long to reply.  I have been dealing with squirrels chewing our fiber optic lines smile

I will try the InnoDB.  I see you can use a script to do this. 

 I did see some post on this but did not try it because the Moodle Proformance Doc. is telling us to not use InnoDB.

Thanks I will let you know what I find.

Average of ratings: -
In reply to Greg Lund-Chaix

Re: Server still running HIGH CPU

by Ryan Kelley -

I changed MyISAM over to InnoDb and it seemed to help a little but today I have over 50 users logged in and the system is crashing.  My CPU has been at 100% for 40 mins.  Durning that same time my memory when upto 50% then started declining.

Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by HJWUCGA INC. -
when it looks up, do either or both a "top" or "ps aux" command. Let's see what is looking that up.

Do you have any other applications on that server?

Are there any cron jobs that are running?

Are users trying to doing a manual backup during these times?

At first glance, your ram seems to be on the low side. Try to have at least 8.

which Linux distro do you have? Did you use XAMP to install the LAMP environment or did you configure it yourself? There may be 2 apaches running?

let's start with the simple things first.

Chris
Average of ratings: -
In reply to HJWUCGA INC.

Re: Server still running HIGH CPU

by Ryan Kelley -

Thank you for helping me.  I hope this information is helpful.

when it looks up, do either or both a "top" or "ps aux" command. Let's see what is looking that up.

Please see attached
Do you have any other applications on that server?

No, the server was built just for Moodle.
Are there any cron jobs that are running?

Not that I know of. I just installed the server and loaded LAMP
Are users trying to doing a manual backup during these times?

I would not think so. Most of my users are new to Moodle and have not explored it that much.
At first glance, your ram seems to be on the low side. Try to have at least 8.

Right now I’m trying to build a 64 bit Windows Server to try it on. I would also like to load and test WLMP on it but I’m having issues getting eaccelerator working on it. I know running it on a 32 bit server is limiting me on memory but my memory is always staying low. Today was the first time my memory went over 20% and that peaked at 55%.
which Linux distro do you have?

Suse

Did you use XAMP to install the LAMP environment or did you configure it yourself?

Yes I am running LAMP at this time.

There may be 2 apaches running?

Not that I know of. When I stop the LAMP services no web pages try to load.

Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by HJWUCGA INC. -
A few suggestions:

1. Don't bother with 64-bit as Moodle is a 32bit app and putting it on a 64bit server (and best to optimize that configuration in the first place), may actually increase overhead if it's not properly designed.

2. I would not use Windows because the thread process is not exactly very good. All I heard from those using Windows with AMP install always have a problem crashing or locking up. The OS does not allocate memory processes very well. I would stay away from it. Another reason is that your test environment should be the same/mirror as your production server in every way so you can be close to 100% sure that what you tested in test should work in production. Make sure the moodle release is the same (I mean the same not just the version - ie. weekly build is the same and save it somewhere), mysql, php, SuSE version and patches.

3. I too am running SuSE (SLES 10) and there is not need for XWindows. I notice you have gnome installed ... get rid of it and just use the command line to do your configuration - that may be slowing you down.

4. login as root and see and do a "crontab -l" (minus lowercase L) to see if there are any crons running

5. Memory is a funny thing. Some apps want to use the whole lot if they know they have it so even if you have a gillion, zillion petabyte of ram, the app may try to use it all =). The bottleneck I think is your database. Try to put the database on a separate disk (SAN) if possible to offload the usage on the server's CPU.

6. I'm assuming that your moodle, moodledata, and database are all in one machine and on one disk drive, correct?

Chris


Average of ratings: -
In reply to HJWUCGA INC.

Re: Server still running HIGH CPU

by Ryan Kelley -

Hi Chris,

The only cron job I'm running is the moodle one.  I think my biggest issue yesterday was that I had over 50 users logged in all day long. Before than if I had 30 users it would be a lot.  I expect that number to keep growing.  I need to plan for 500-1000 on the system at all times.

I'm going to try to dump as much CPU at my server as I can.  I'll plan to move my moodle server out of VMware and over to its own dual quad core server with 8-16GB of ram.  I will also keep the database on VMware running on the SAN.

I'm sure I can find in on the forums but do you know how to point the moodle server to the database on another server?

Ryan

Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by Shimron Trammell -
Ryan it sounds like you are running into the same issue that we ran into last year. I've got a feeling it's not your web server but it really is your MySQL backend server. Do you have a program like phpMyAdmin installed on your MySQL server? Next time your web server starts to die go into phpMyAdmin or if you know how to do this from the console even better and click on the view processes button. It'll show you all of the queries being processed at that moment. What was happening to us was that our MySQL server even though the CPU or memory usage wouldn't be high, one query would lock out one database (we run multiple schools districts on our moodle setup) and that one database lockup would keep any new requests for that db from coming in. No big deal on the MySQL server but our web servers would get locked up with people from that district and that would eventually bring down the web server.

Another thing to look into are you watching the apache server status page? http://your-apacehsite/server-status?refresh=5. Just watch that for a while. We moved our moodle site to a super server much like you're about to do and thats when we finally started to see the MySQL issue. Now we realize we're way over power usage for our 28 school districts and we're looking to move the webserver back down to something more sensible and bring our MySQL server to a real server and out of VM. I swear I was talking to a colleague and we thought you were someone at our site going behind our back lol.
Average of ratings: -
In reply to Shimron Trammell

Re: Server still running HIGH CPU

by Ryan Kelley -

The status page on PHPmyAdmin seems to be a great resource.  I have never looked at that part of PHPmyAdmin before. 

I am seeing the fields below in red all showing that I need to add memory to them.

Innodb_buffer_pool_reads
Created_tmp_disk_tables
Opened_tables
Key_reads

Can you please tell me how many users you have on at peak loads and what your settings are for your my.cnf file?

Have you thought about using SQL instead of mysql for the database?  Not sure if it would process things better but SQL's educational pricing is cheap I wondered if anyone has tried it.

 I am not seeing where you see locked tables on the database in PHPmyAdmin. Can you please explain? 

Thank you for your help,

Ryan

Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by Shimron Trammell -
During peak loads now we're getting around 300 students using moodle but our Apache server only does max about 50 requests at a time because it's getting it's requests back from MySQL quick enough to release the apache connections and then take more. We are current re-evaluating our my.cnf file because we are in the process of moving our mysql db server from VMware to a physical box (Like literally we thought you were here with your situation lol).

Don't wanna use MSSQL just for the simple fact that we'd have to really give it some power and we want to try to make this whole setup work using open source linux based software.

From what I've been able to figure out with the locked tables it's mysql's way of protecting data. One database out of the 28 or so would run a query and that query would take forever to run. So then other queries would get be waiting for that first one to finish. Our issue ended up being that a cron job was just running at a bad time and our databases were so fragmented that it was taking forever to run. We ended up adding a cron job to clean up our databases every night. Your issue sounds like a bad query or cron job. Will know better once we can find the root of the issue.
Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by Greg Lund-Chaix -
You're running MySQL on the same virtual machine as the web server? We've had very poor luck running MySQL in virtualized environments, doubly so on the same VM as the web server. I'd recommend looking at moving your database off onto its own hardware.

-Greg
Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by HJWUCGA INC. -
Oh I did not realize that it was on VMWare. Yes, take it out to its own server (dedicated). We use to have issues on VM because it was not allocating resources as best as it could (could be our fault though for not setting it up right but that's another discussion =))

I'd put the db outside of VM as well.

It's easy to point to another server. Just setup the db and in your config.php or during your moodle install, just put in the ip of the server and the db's username and password. That's all there is to it. Make sure your connection is very reliable though because, if not, you're going to encounter problems for sure.

What we/I did was to create a separate, dedicated mount point on the SAN just for the db and moodledata respectively. The MySQL is still installed on our blade but the databases are actually stored on the SAN.

except for the SLES part, the design is the same (we decided to ditch SLES and use Red Hat instead)

http://moodle.org/mod/forum/discuss.php?d=132924#p583543


Average of ratings: -
In reply to HJWUCGA INC.

Re: Server still running HIGH CPU

by Dan Poltawski -
  1. Don't bother with 64-bit as Moodle is a 32bit app and putting it on a 64bit server (and best to optimize that configuration in the first place), may actually increase overhead if it's not properly designed.

Erm, in what way is Moodle a 32 bit app?

Average of ratings: -
In reply to Dan Poltawski

Re: Server still running HIGH CPU

by HJWUCGA INC. -
The code base is not optimized to take advantage of 64 bit features. I was considering this too but decided that it's not worth it and stick with 32 until moodle comes up with a 64 bit version
Average of ratings: -
In reply to HJWUCGA INC.

Re: Server still running HIGH CPU

by Jon Witts -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
If you use the 64bit versions of apache and PHP, then it is 64bit... Or am I way off with that assumption?
Average of ratings: -
In reply to Jon Witts

Re: Server still running HIGH CPU

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

You are absolutely right! smile

Saludos, Iñaki.

Average of ratings: -
In reply to Iñaki Arenaza

Re: Server still running HIGH CPU

by Ryan Kelley -
Thank you all for you help with this.  I think we are going to try and put this on its own server.
Average of ratings: -
In reply to Ryan Kelley

Re: Server still running HIGH CPU

by HJWUCGA INC. -
Yeah, don't install moodle on vm platform
Average of ratings: -
In reply to HJWUCGA INC.

Re: Server still running HIGH CPU

by Greg Lund-Chaix -
Actually, the Apache node is perfectly fine on a virtual machine. It's just the database server that doesn't do as well virtualized.

-Greg
Average of ratings: -
In reply to Iñaki Arenaza

Re: Server still running HIGH CPU

by Jon Witts -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Good - I thought I was going mad there! wink
Average of ratings: -
In reply to Jon Witts

Re: Server still running HIGH CPU

by Ryan Kelley -

Looks like I fixed my issue.  Attached is a graph from VMware showing the change in CPU on my server.

I think the issue was the eaccerator was not working right.  When I downgraded XAMPP from 1.7.2 to 1.6.8 eaccerator started working and my CPU dropped down to nothing.

Attachment moodle-graph.gif
Average of ratings: -