Moodle Performance Testing in AWS

Moodle Performance Testing in AWS

by Brett Cairns -
Number of replies: 5

I have been testing Moodle 2.4 in AWS as we are looking to upgrade from Moodle 2.4 and switch hosting providers. In short, the performance was much slower than our current setup and I was hoping that someone might be able to give some general advice for why that might be the case and/or how I could go about improving performance.

Profiling was completed using xhprof to keep track of specific functionality performance (as described in this helpful blog: *cannot post links for 24 hours).

The following screenshots outline the amount of time taken to open the ./question/category.php page for three different server environments. In each screenshot, the hardware resources of the environment increases and this is associated with only minor performance gains. The web server is being hosted in EC2 and the database server is being hosted in RDS.

c1.xlarge - (7GB memory, 4 x 420 GB storage)

db.m2.xlarge - (17.1GB memory, 6.5 ECU (2 virtual cores with 3.25 ECUs each), 64-bit platform, High I/O Capacity)

 c1.xlarge - (7GB memory, 4 x 420 GB storage)

db.m2.x2large - (34GB memory, 13 ECU (4 virtual cores with 3.25 ECUs each), 64-bit platform, High I/O Capacity)

 hi1.4xlarge - (60.5GB memory, 2 x 1024 SSD GB storage)

db.m2.x2large - (34GB memory, 13 ECU (4 virtual cores with 3.25 ECUs each), 64-bit platform, High I/O Capacity)

What I observed from this profiling was that at best the categories page would take just under ten seconds to load on average. Xhprof pointed out specifically that the mysql query related to the main function call was talking up the majority of processing time at roughly 4.5 seconds. Therefore, I was surprised that doubling the hardware resources for the database server and more than doubling the hardware resources for the web server had little improvement on the processing time. Of note, additional tests were ran for similar functions, with multiple runs for each function and the results were all similar.

I had heard some criticisms in general about IOPS performance for Moodle in AWS (see: *cannot post links for 24 hours) but I am attracted to the ease of use and hosting solutions provided by Amazon and therefore I would rather work out a solution if possible. Could this just be a matter of software optimization that has to take place? If yes, do you have any advice specific to a Moodle 2.4 setup in the AWS Cloud?

Here is some additional information about our Moodle extensions: PHP version 5.4.6, Apache 2.2.22, MYSQL 5.5.29, APC 3.1.13.

Any help is much appreciated. Thanks!

(Edited by Helen Foster to add images - original submission Thursday, 20 June 2013, 5:28 PM)

Average of ratings: -
In reply to Brett Cairns

Re: Moodle Performance Testing in AWS

by Grant Mucha -

Specifically that the mysql query related to the main function call was talking up the majority of processing time.

You will be hard pressed to achieve amazing performance on Amazons infrastructure.

I read you are likely already aware of IOPS and requirements for Moodle.

I have servers setup with AMD 8120 (eight core) / 32GB ram / 4x SSD with Adaptec 6405 raid controller 512mb cache in raid 10 for increased read and write as well as some added security.

I've tested and used various system in real life scenarios. I still believe one of the most important requirements is hard disk performance.

One standard 10k SAS drive has about 140 IOPS, 15k SAS is about 175 - 200 IOPS.

One standard SSD drive has about 75,000+ IOPS.

Due to the number of database transactions performed SSD drives allow Moodle thrive.

Sincerely,

Grant

Please email me the links, I will perform the same test on my servers for comparision.

 

 

Average of ratings: Useful (1)
In reply to Grant Mucha

Re: Moodle Performance Testing in AWS

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi all,

I really agree with Grant about the switch to SSD's. The largest single jump in performance of my (small) Moodle sites was when I switched from hard drives to a solid state drive.

Prior to the switch, if I monitored the cpu performance, even a single core cpu would be loafing along without pegging out at 100% when I would run tests. The first time I switched to SSD in a quad core box, I was finally able to pump enough data through the testing to peg all four cores at 100%.

My present system for me and a max of 16 students per class, uses an 8 core cpu and single ssd. Running tests, I can peg 6 cores to 100% and the remaining two run about 96%.

Typical page performance in Moodle 2.5, for students is .2 to .3 seconds, and for me when admin it is .8 to 1.2. For me as a teacher it is about .4 seconds or less.

AL

 

Average of ratings: -
In reply to AL Rachels

Re: Moodle Performance Testing in AWS

by Brett Cairns -

Hello everyone,

 

Thanks for getting back to me and giving some useful suggestions. I have completed a few more tests since my first post. I have tested a number of other types of functionality and with a number of different account types (admin, teacher, student) but noticed that the categories page for accessing system level questions was the slowest.

I also tested an AWS instance with an SSD running on the database server. This instance was noticeably faster than the instances without the SSD; however, the results were still a little disappointing.

 

Web Server: m2.2xlarge, 64-bit, 4 cores, 34.2GB RAM, 850GB storage

Database Server: hi1.4xlarge,  64-bit, 16 cores, 60.5GB RAM, 2 x 1024GD SSD storage

 

Performance here was almost 2 seconds faster than our next best instance.The MYSQL query is still taking up the majority of the processing time. The overall processing time of roughly 9.7 seconds is still longer than we would like. By comparison, on a blank installation, the categories page took around 2 seconds to load. We also completed testing by adding and removing categories and questions and noticed that had an effect on speed; however, removing questions would affect the integrity of our web site and we need to look at other ways of improving performance.

 

Here are some thoughts on our setup and testing going forward. I only connected an SSD to the database server thinking that was where the disk speed bottleneck would be. I am probably going to test it out anyways, but do you think it would be worthwhile to attach an SSD to the web server as well? Also, I had read a MYSQL performance tuning guide that stated that MYSQL performance would decrease for server with more than 8 cores. I have tested a number of instances and have not really noticed significant performance decreases for MYSQL queries on instances with more than 8 cores, but should I still be concerned about this?

Finally, would it make sense to test out an instance with multiple web/database servers, a load balancer and either a Memcache server or S3 high availability drive (Amazon solution for moodledata folder)? One of our motivations for staying with a single web server setup was that it simplified the issue of sharing the moodledata folder. However, if there are significant performance gains for multiple web server setups I think it is something that we would look into.

 

Thanks for all the help!

Regards,

Brett

Average of ratings: -
In reply to Brett Cairns

Re: Moodle Performance Testing in AWS

by Brett Cairns -

Much to my embarrassment, I realized that I uploaded the wrong image. The image below is the correct image in reference to my last post. When comparing to the faster profile in my first post, please note that the web server in this instance is less powerful and that the database server in this instance has an SSD and is more powerful. Here you can see that the MYSQL query is faster than instances without and SSD but also that there are no significant performance gains.

 

Average of ratings: -
In reply to Brett Cairns

Re: Moodle Performance Testing in AWS

by Grant Mucha -

The following screenshots outline the amount of time taken to open the ./question/category.php page for three different server environments.

Unfortunately, I have nothing to test in this area.

I'm attaching a rough profiling of one of my servers setup specifically for moodle. Below are profiles of various pages.

  Execution time CPU time Function calls Memory used
/index.php  1013.398 ms 920.057 ms 292836 20955.758 KB
/course/index.php  644.096 ms 628.038 ms 236747 17383.719 KB
/mod/scorm/view.php  720.829 ms 692.042 ms 258900 19396.406 KB
/course/view.php  952.654 ms 888.056 ms 325416 20690.656 KB
/pluginfile.php  17.841 ms 16.001 ms 4113 3003.828 KB
/index.php  1126.557 ms 984.062 ms 305764 21371.117 KB
/user/view.php  802.452 ms 780.048 ms 294178 19914.047 KB
/grade/report/grader/index.php  1939.602 ms 1884.118 ms 728265 26474.047 KB
/course/view.php  948.076 ms 880.055 ms 325239 20614.188 KB
/index.php  1133.060 ms 996.061 ms 305802 21386.063 KB

Web Server: m2.2xlarge, 64-bit, 4 cores, 34.2GB RAM, 850GB storage

High-Memory Double Extra Large 34.20 GB 13 (4 cores x 3.25 units) 850 GB 64-bit High 120 m2.2xlarge $590.40 monthly

The overall cost compared to the performance is abysmal at best

Database Server: hi1.4xlarge,  64-bit, 16 cores, 60.5GB RAM, 2 x 1024GD SSD storage

High I/O Quadruple Extra Large 60.50 GB 35 (8 cores + 8 hyperthreads) 2048 GB (2x1024 GB SSD) 64-bit Very High 1 hi1.4xlarge $2232.00 monthly

Again, performance vs cost is abysmal.

Also, I had read a MYSQL performance tuning guide that stated that MYSQL performance would decrease for server with more than 8 cores. I have tested a number of instances and have not really noticed significant performance decreases for MYSQL queries on instances with more than 8 cores, but should I still be concerned about this?

I would not be concerned. You likely would never require more than eight cores in a single server. Anything over eight cores is ideal for setting up virtual servers, gaming servers, encoding, and other processor intensive tasks. You are far better off with two 4 core servers turned to handle their designated task. ex: one web server, one database server

I think a lot of people are confused about optimizing moodle. Depending how your system is setup an average user uses about 25mb of ram. Lets say 16gb of ram might handle 500 students. In reality the cpu and ram may be able to handle this, yes with load, but the file system will not come  close.

Worth reading if your stuck with Amazon. Raid EBS

pm me for link, post was marked as spam

Quick hardward test (very basic) hdparm -t /dev/sda#

Timing buffered disk reads: 1572 MB in  3.00 seconds = 523.49 MB/sec server above

Timing buffered disk reads: 544 MB in 3.01 seconds = 180.85 MB/sec another dedicated server

I'm not sure what Amazon EC instances output but might be worth a quick check. You can also use sysbench or unixbench to perform basic tests across various hardware.

Anything else feel free to reply to the post or posts above!

 

 

Average of ratings: -