Hi,
I'm running some performance tests on our institutes new Moodle 2.x site and I'm seeing significant underperformance on what I would consider pretty powerful underlying hardware.
Here's a quick summary of the architecture:
Host Server:
Dell PowerEdge R620 with
- 2 * Intel Xeon E5-2640 2.5GHz 15M Cache CPU
- 96 Gb RAM
- VMWare ESXi 5.0 (embedded image)
Virtual Machines:
2 VMs (db & web) each with 8 logical cpu cores assigned & 16 Gb RAM
Web VM running:
Apache 2.2.14 with PHP-FPM (mpm-worker)
APC
Database VM running:
MySQL 5.1
Having run the following Apache Benchmark test against the above configuration:
ab -C MoodleSession=<session id> -k -n 200 -c 20 https://moodle-web-test-apache.example.com/my/
I can't seem to get more than 3 - 4 requests per second of through put from the site. Obviously this is alarming!
With Moodle's performance information turned on, I can see that the My Moodle page loads in approx 350 milliseconds, whereas, ab reports an average of 5.5 seconds!
I've tried tweaking some of Apache's settings, to no avail. I still can't improve throughput!
If anyone has had a similar experience to mine or is seeing less than favorable performance from Moodle, please reply.
thanks,
Cathal.
I don't have a great deal of time to write a full reply right now, but a few initial comments. For starters, we need much more information to be able to help. See the top of the forum (http://moodle.org/mod/forum/view.php?id=596) for more information on background information that we need.
Other comments you should have a quick read through:
ab isn't a great tool - take a look at jmeter instead. There's a set of jmeter scripts for moodle in a report (see https://github.com/kabalin/moodle-jmeter-script-generator) which test a number of core features in moodle.
You also don't say which user you're testing with - it's never a good idea to test with the admin user as they're somewhat special and far more happens with the admin user than any other user. You should typically perform load testing with students and/or staff accounts.
You also haven't said how you've been tweaking your apache configuration - if you can, could you say what changes you've been making. Apache configuration can be a bit of a dark art and often by increasing things you think you improve performance and throughput, actually cause your system to fight for resource and thus degrade performance.
Another consideration which you may want to look at is your VMs resource allocation. From speaking to our in-house VMware specialists it's really not a good idea to over-allocate your CPUs. VMware (apparently) handles best with 1 CPU core per VM in most situations, though newer versions of VMware will happily handle 2-3 CPU cores. If you start looking at any more than this, then VMware starts to have issues scheduling your workload which causes far worse performance than you'd hope to see. I'm not an authority on this topic, but have discussed it with knowledgeable colleagues at length.
Rather than giving your web server 8 cores and 16GB Ram, consider creating multiple VMs, each with 2 cores and 4GB RAM for example, and using some form of load balancer software (e.g. haproxy). You can easily have 4 web servers balancing the load and using the same resource if you really need it.
You also haven't mentioned whether you're using a php cache - I'd highly recommend installing and configuring APC.
On the database server front, consider Postgres (though I won't start that argument right now).
Ideally we also need to know things like:
* are you using a database connection pooler?
* what OS (presumably Linux?)
* what distro?
* 32bit/64bit?
* where are your disks - e.g. are you using a SAN mounted over NFS?
* have you tuned your database server?
* etc.
Andrew
Hi Andrew,
Thanks for the quick reply. You've provided some great suggestions.
I was in fact running performance tests aginst the Admin user. Having switched to a standard user, I've seen a marked improvement in throughput; 14 reqs/sec.
To answer some of your questions:
- Apache settings
We're running with mostly default settings for Apache at the moment.'KeepAliveTimeout' is set to 4 seconds and the 'Timeout' is at 30 seconds. The mpm-worker settings are mainly default values;
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 250
MaxRequestsPerChild 0
I did up the 'MaxClients' value to 250 as I was concerned that requests were getting queued. Apart from that, Apache hands off processing of PHP requests to the local fast_cgi/PHP-FMP process. As it stands, it is configured to run fast cgi workers.
- VM Resources
Thanks for the tips here. The VMs we are running were configured to reflect the resources on the physical machines we are running Moodle 1.9 on at the moment. The intention is to gradually introduce more VMs, particularly to serve as a web cluster, and distribute load amongst these.
- PHP Cache
Yes, APC is running in this configuration.
- General operating environment
The VMs are running Ubuntu 10.04.4 LTS (64 bit)
No, we haven't begun tuning the database yet.
Yes, we will be using NFS to mount and share the Moodle application directory across the web cluster. At the moment, everything is stored locally on the VM.
best regards,
Cathal.
Obviously 12 cores at 2.5 GHz each together with 96 GB RAM serving 3 - 4 requests/sec is alarming - not to mention its impact on environment.
Apart from the forum the documentation and what Andrew said I have only a few general comments:
- people have reported repeatedly of poor database performance of virtual machines (run an 'advanced search' in this forum)
- NFS is unsuitable for Moodle session data. In a web server cluster you have to go for database sessions which are inherently slower than locally stored session data.
- Where you put moodledata is critical. Worth considering a SAN technology for a web server cluster.
And the fundamental question I always ask: If performance is your aim why you bring in virtualisation?
Hi Visvanath,
Yeah, at those specs. I was certainly alarmed at the results I was seeing too
Changing the user I was conducting tests with brought improved results.
Thanks for the tip on database performance when virtualised. I had read somewhere on the forums that the OU were running dedicated hardware for their database layer. This is something I'll definitely follow up on.
You're question is a good one! Yes, performance is the aim, and so is high availability too. The "one server, one application" model isn't cost effective anymore. Purchasing seperate hardware to fulfil each of the roles in a setup such as ours would prove too costly. Especially when you start to add additional hardware for the sake of redundancy and failover. The virtualisation model also works for us because it allows us to adapt our architecture; if we feel something doesn't work we can introduce changes without having to provision more hardware or redeploy existing.
I'd be interested to hear your views on this
Cathal.
I was hesitant to put my recent experience on this topic because it is not going to help the running discussion. Now I'm on vacation and (mobile) Internet has reached even this remote tropical paradise, I thought of jotting it down.
Yes, the customers have high demands in all fronts, performance, availability, security, ... - all at the same time! During the initial discussion, once they hear about the various levels one could have in each of these fronts, things get even worse. Then in the middle part of the discussion I start to quantify those demands to realize that there are no hard figures. Rarely I hear, "we need a four lane bridge, spanning 200 m between point X and Y, which could carry 40 T trucks on two lanes, which must stand a 50 y flood, ..." rather, "the voters said yes to a bridge across ..., it passed the council..."
Anyway, to make a long story short, rather than offering the biggest solution which the customer can afford, I finally offer them a whole series, from basic to maximum, with clear specs and costs, and most importantly, an upgrade path once the need arises.
And my experience is that a surprisingly big proportion of them stay with the (modest) plan they've started with. Though less attractive financially these are success stories for me from an engineering point of view. Also, I hope, that way I've contributed less to an ever accelerating environment disaster.