My moodle 2.3 server is running slow. There are no students currently enrolled, just a few instructors. It takes a several seconds to draw the screen and update an activity. both the web server and the database are on the same server. The server has a total of 16GB of memory and is using just over 7 GBs at the moment. It has almost 9GB's of free memory. It has 16 processing cores and their using was at its highest 8%. On the networking side, there are four 1GB NIC cards bonded together to give the server much more throughput. We have PHP 5.3.3 installed. Any ideas?.
In reply to Doris Johnson
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Doris Johnson -
In reply to Doris Johnson
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Visvanath Ratnaweera -
Hi Doris
> My moodle 2.3 server is running slow. There are no students currently enrolled, just a few instructors. It takes a several seconds to draw the screen and update an activity.
Start with simple methods like the FireBug or http://docs.moodle.org/24/en/Debugging#Performance_info.
> both the web server and the database are on the same server. The server has a total of 16GB of memory and is using just over 7 GBs at the moment. It has almost 9GB's of free memory. It has 16 processing cores and their using was at its highest 8%.
Your "server" runs a GUI, which is a big no-no. Let's leave that dispute aside, is it possible that your admins come from the GUI world and running this RedHat in a virtualized environment of their liking?
> On the networking side, there are four 1GB NIC cards bonded together to give the server much more throughput.
That tends to support my speculation. If the admins know how to "bond" interfaces in Linux, they won't be running Gnome Window Manager in their servers!
On a side note: Assuming there is a virtualization layer, ask the administrators to measure the network bandwidth between the RedHat VM and another 4 Gbit/s "bonded" machine attached to the same (network) switch.
BTW, those short "snap-shots" you've provided are not very useful, unless you can correlate them with the load on the server. What you need is monitoring as discussed in this thread: "How do you monitor your Moodle server?" https://moodle.org/mod/forum/discuss.php?d=192162.
> My moodle 2.3 server is running slow. There are no students currently enrolled, just a few instructors. It takes a several seconds to draw the screen and update an activity.
Start with simple methods like the FireBug or http://docs.moodle.org/24/en/Debugging#Performance_info.
> both the web server and the database are on the same server. The server has a total of 16GB of memory and is using just over 7 GBs at the moment. It has almost 9GB's of free memory. It has 16 processing cores and their using was at its highest 8%.
Your "server" runs a GUI, which is a big no-no. Let's leave that dispute aside, is it possible that your admins come from the GUI world and running this RedHat in a virtualized environment of their liking?
> On the networking side, there are four 1GB NIC cards bonded together to give the server much more throughput.
That tends to support my speculation. If the admins know how to "bond" interfaces in Linux, they won't be running Gnome Window Manager in their servers!
On a side note: Assuming there is a virtualization layer, ask the administrators to measure the network bandwidth between the RedHat VM and another 4 Gbit/s "bonded" machine attached to the same (network) switch.
BTW, those short "snap-shots" you've provided are not very useful, unless you can correlate them with the load on the server. What you need is monitoring as discussed in this thread: "How do you monitor your Moodle server?" https://moodle.org/mod/forum/discuss.php?d=192162.
In reply to Visvanath Ratnaweera
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Doris Johnson -
Thanks for the help. I have forwarded this information to our IT department. I turned debugging on and have a couple of screen shots.
Do these numbers look slow. I was reading a post that said Moodle 2.3 is twice as big as Moodle 1.9 and the response would be a little slower. This first one is just bring up the course. the second one is turning editing on.
In reply to Doris Johnson
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Doris Johnson -
In reply to Doris Johnson
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Visvanath Ratnaweera -
Hi Doris
You wrote:
> Thanks for the help. I have forwarded this information to our IT department.
No problem! Actually your IT people should be actively looking for a solution, here in moodle.org. Their starting point is the introduction to this (Hardware and Performance) forum.
> I turned debugging on and have a couple of screen shots.
>
> Do these numbers look slow.
Moodle looks bloated: 55 - 75 MB RAM, 255 - 583 included files, 686 - 7190(!) get_string calls, 119 - 152 DB reads. May be these numbers are normal for Moodle 2.3.
> I was reading a post that said Moodle 2.3 is twice as big as Moodle 1.9 and the response would be a little slower.
If 2.3 is double the size of 1.9 I would expect it to be more than "a little" slower than 1.9.
;-(
I got lost in the Moodle version gallop. This report http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ may help.
Coming back to the performance info, the execution times of 1.4 - 5.2 sec are too high for a 16 core machine. For simple pages, like the one you were measuring, 0.4 - 1.2 sec should be sufficient.
What I find odd is the (CPU) load average: 0.5 - 0.57. A 16 core machine when fully loaded should be able to serve 16.0. So the CPUs are mostly idling. May be they are just waiting? You didn't respond to my question on virtualization. May be the database is slow due to poor disk performance of VMs?
I asked for a whole lot of information in my previous post. Have you ignored them or do the IT people need time to answer?
You wrote:
> Thanks for the help. I have forwarded this information to our IT department.
No problem! Actually your IT people should be actively looking for a solution, here in moodle.org. Their starting point is the introduction to this (Hardware and Performance) forum.
> I turned debugging on and have a couple of screen shots.
>
> Do these numbers look slow.
Moodle looks bloated: 55 - 75 MB RAM, 255 - 583 included files, 686 - 7190(!) get_string calls, 119 - 152 DB reads. May be these numbers are normal for Moodle 2.3.
> I was reading a post that said Moodle 2.3 is twice as big as Moodle 1.9 and the response would be a little slower.
If 2.3 is double the size of 1.9 I would expect it to be more than "a little" slower than 1.9.
;-(
I got lost in the Moodle version gallop. This report http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ may help.
Coming back to the performance info, the execution times of 1.4 - 5.2 sec are too high for a 16 core machine. For simple pages, like the one you were measuring, 0.4 - 1.2 sec should be sufficient.
What I find odd is the (CPU) load average: 0.5 - 0.57. A 16 core machine when fully loaded should be able to serve 16.0. So the CPUs are mostly idling. May be they are just waiting? You didn't respond to my question on virtualization. May be the database is slow due to poor disk performance of VMs?
I asked for a whole lot of information in my previous post. Have you ignored them or do the IT people need time to answer?
In reply to Visvanath Ratnaweera
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Doris Johnson -
Sorry, I didn't hear back from our IT people until this morning. Our IT department is running a mostly microsoft shop and are only now running 2 RHEL5 servers. There is not a vertualization layer. Yesterday, one of the technical was working to make sure the bonded nic cards were working as they should. measuring the network bandwidth may be part of what he did yesterday. He did tell me they didn't have the software to measure the server load at this time. Thanks for your help and informational links. It has given them a starting place.
In reply to Doris Johnson
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Visvanath Ratnaweera -
Hi Doris
> There is not a vertualization layer.
Good to know that. In that case the performance is too low for a 16 core 16 GB machine.
> Yesterday, one of the technical was working to make sure the bonded nic cards were working as they should. measuring the network bandwidth may be part of what he did yesterday
And what was the throughput?
> He did tell me they didn't have the software to measure the server load at this time.
That's why the Gnome system monitor! There are tons of tools for Linux. Just the on-board tools in an Unix will take you much further, "wget -O/dev/null URL" for network throughput, for example.
> There is not a vertualization layer.
Good to know that. In that case the performance is too low for a 16 core 16 GB machine.
> Yesterday, one of the technical was working to make sure the bonded nic cards were working as they should. measuring the network bandwidth may be part of what he did yesterday
And what was the throughput?
> He did tell me they didn't have the software to measure the server load at this time.
That's why the Gnome system monitor! There are tons of tools for Linux. Just the on-board tools in an Unix will take you much further, "wget -O/dev/null URL" for network throughput, for example.
In reply to Visvanath Ratnaweera
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Andrew Lyons -
A handy one that I use frequently is dstat:
dstat -Tmcndls
With those options, it will give you rolling (per second) stats for:
- time (epoch)
- memory
- CPU usage breakdown
- network bandwidth utilisation
- disk bandwidth utilisation
- load average
- swap usage
You can also get per-core, and IIRC, per disk.
Andrew
In reply to Visvanath Ratnaweera
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Andrew Lyons -
Just to say, that you're extremely unlikely to get 4GB/s from a bonded link. FWIR the bonded NICs each have their own MAC and the way that LACP works means that you'll only get the max throughput of *one* NIC.
If you try and combine all four to give you 4GB/s then you've got to shard the individual packets at the one end (switch or adapter) and recombine them on the other end. The overhead of this would be far slower than just using 1 NIC to it's max.
The benefits of LACP aggregated bonding are that you get:
- resiliency - you can unplug any NIC and LACP should notice pretty quickly and IIRC won't drop packets
- max throughput is 1GB/s per NIC. If you have 4 clients, the ideal would be that the MAC address hashing algorithm directs each one to a different NIC, but this isn't guaranteed. Typically it's done by calculating which of the X NICs to use based on a hash.
Andrew
In reply to Doris Johnson
Re: Moodle 2.3 running Red Hat Enterprise - poor performance
by Simon Story -
What version of Redhat Enterprise Linux (RHEL) do you have?
An essential element for reasonable performance is an php cache, like PHP-APC
If memory serves me correctly, it is not possible to install PHP-APC and PHP 5.3.3 at the same time on some versions of RHEL.
You can have an earlier version of PHP and have PHP-APC, but that version is too old for Moodle.