Moodle performance testing: 1.9 vs. 2.0

Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Number of replies: 25
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

At LUNS Ltd. we have done some measurements recently and I think it worth sharing our findings with Moodle community.

Having enhanced JMeter testplan generation plugin originally developed by James Brisland, we used it to run performance tests on Forum, Chat and Quiz activities in 1.9 and HEAD (2.0) versions. Each testplan included all steps required to perform the task (logging in, navigating to required page, posting in chat/forum or submitting quiz). Test was done with 10 threads (simultaneous users) and each user made 5 posts in corresponding activity (in Quiz only one submission was done). Static content retrieving (images, css) are not included in the testplan, so the time only reflects php script output rendering on the client side.

All tests were done on dev-box: Intel(R) Core(TM)2 Duo CPU E8600 3.33GHz, 8GB RAM, Debian/GNU "squeeze", Kernel: Linux 2.6.26-2-vserver-amd64. Web and database server are two different vhosts run on the same systems: Apache 2.2.16-3, php5 5.3.2-2, mysql-server 5.0.51a-24+lenny4.

Now the results. The average page performance results (mean time in milliseconds spent on each page retrieved throughout the testplan) are shown in the table below:

Moodle 1.9 Moodle 2.0
Forum

435 (σ=146)

 1450 (σ=351)
Chat AJAX 298 (σ=191) 668 (σ=586)*
Chat Accessible 438 (σ=225) 1660 (σ=503)
Quiz 471 (σ=172) 2086 (σ=742

* Navigation was slower than actual chat communication.

The same result presented on the graph:

Next table shows the summary of perfomance on some pages:


 Moodle 1.9 Moodle 2.0
1. moodle/login/index.php

780 (σ=211)

 1450 (σ=351)
2. moodle/course/view.php 476 (σ=112) 1600 (σ=158)
3. moodle/mod/forum/view.php 346 (σ=78) 1485 (σ=174)
4. moodle/mod/forum/discuss.php 349 (σ=103) 1435 (σ=158)
5. moodle/mod/chat/view.php 288 (σ=73) 1328 (σ=116)
6. moodle/mod/chat/gui_basic/index.php 312 (σ=98) 1234 (σ=257)

7. /moodle/mod/chat/gui_header_js/jsupdate.php
(moodle2/mod/chat/chat_ajax.php?action=update)

 289 (σ=124) 384 (σ=123)
8. moodle/mod/quiz/view.php 385 (σ=98) 1532 (σ=187)
9. moodle/mod/quiz/attempt.php 472 (σ=114)
 3139 (σ=141)

An image with the same data:

You may notice Moode 2.0 is not as fast as its previous version - as results suggest, on the most pages we run test with, version 2.0 is significantly slower. Generated JMeter reports can be downloaded from here (csv files).

Next, we have done a brief investigation of what makes Moodle 2.0 that slow. Profiling was done using XDebug PHP extension on the server side, while moodle/mod/forum/view.php?id=X page was open in browser. Profiling dump files (results) are avialable for download from here (cachegrind.out.tar.gz). You may view these files with any cachegrind viewer (KCacheGrind or WinCacheGrind).

It appeared that page rendering (blocks rendering in particular) causes the most significant delays.

The screen-shot below shows the function calls in Moodle 1.9 ordered by running time (the figure in the first column is 1/1000000 fraction of second, e.g. 932269 means 932 millisec.):

 

The function calls in Moodle 2.0

 

If I logged in as admin, which means there will be more items in the blocks and probably more permission checks has to be done (just an assumption, I am not that familiar with Moodle code), even more time is required:

I conclude at this point. Results above indicates that Moodle 2.0 might need some performance optimisation prior to release in order not to diminish its good reputation.

Relevant links:

JMeter test results and XDebug profiling dump files,

Enhanced JMeter testplan generation plugin,

Profiling PHP scripts with XDebug.

Average of ratings: Useful (11)
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

It is a pity that you did your detailed profiling on forum/vuew.php

In Moodle 1.9 that page did not include blocks. In Moodle 2.0 it does, so the fact that block rendering takes a long time is not such a helpful observation. I would be much more interested to see the comparison for course/view.php (which your results also show is much slower).

The extra slowness for admins seems to be almost entirely explained by block_settings->get_content. In other words building the admin tree. That is not terribly surprising to me. I think we know that is slow, but since it only affects admins, and the admin tree is complex, I think we are stuck with that.

David will be pleased to see that get_string is gone down from ~640 to ~400 whatevers-it-is per call.

Here is my attempt to break down the 1.9 numbers at the top level:

  1. 189 require_once config.php
  2. 11 require_course_login
  3. 17+ print_header_simple (in the call to navmenu)
  4. 119 forum_print_latest_discussions
  5. 41 add_to_log

That adds up to 380 out of 440, so there is a bit missing there. Probably some of the DB queries that are in the top level of view.php.

For comparison, in 2.0

  1. 179 require_once config.php
  2. ?? require_course_login (it is off the bottom of the screenshot)
  3. 420 $OUTPUT->header(); - which now handles all the blocks stuff
  4. 95 forum_print_latest_discussions
  5. ?? add_to_log (it is off the bottom of the screenshot)

That has identified ~700 of the ~940 in the 2.0 numbers.

Nice to see that config.php has actually gone down, as has forum_print_latest_discussions.

So, we have an issue with the blocks and the navigation. As I say, I would like to see the cost of building the course page in 1.9 for comparison.

What on earth is forum_serach_form doing at 82? That is crazy!

Average of ratings: -
In reply to Tim Hunt

Re: Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

Hi Tim,

> It is a pity that you did your detailed profiling on forum/vuew.php

I see what you mean. I have run profiling for moodle/course/view.php?id=X as you advised.

Moodle 1.9:

Moodle 2.0:

Moodle 2.0 logged in as admin:

Profiling dump files are avialable for download from here (cachegrind.out.course_view.tar.gz)

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Session_stub->write_close taking 112 sucks badly.

Anyway, the comparison for blocks is:

1.9: ?? for blocks_setup - I am surprised that does not show up, plus 38 for blocks_pring_group.

In 2.0??: for block_manager::load_blocks, and then 230 for block_base::get_contents_for_output ??? clearly a lot of that is setting up the contents of the settings block.

Acutally, I am a bit confused there. why aren't methods like create_block_contents - which is what calls block_base::get_contents_for_output, showing up. That must take longer than the functions it calls, so it should be there?

Ah, there is a quick win in there. We seem to be including question/editlib.php just to build the question bank navigation. That is expensive, we ought to make a question/lib.php that is cheap to include, and that can be used for external dependancies like this. Please someone file that in the tracker.

page_requirements_manager::__constrcut also seems suprisingly high to me.

And, surely html_write::attribute should not be that slow. It would be interesting to test whether it is the $value instanceof moodle_url that is killing it.

 

Anyway, this is really interesting stuff. If I could ask you to do one more thing, it would be to write really simple step-by-step instructions for how you do this profiling, so I, and the people at Moodle HQ, can do it too. I took the liberty of creating a blank docs page to get you started: Development:Profiling_PHP. Thanks.

Average of ratings: -
In reply to Tim Hunt

Re: Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

Hi Tim,

I have created a manual on Xdebug profiling as you asked. Hope it will help making profiling more popular ;))

Would be great if you guys look through it and let me know if it is clear enough and whether something has to be added or changed.

Thanks.

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Thanks. I just followed your instructions and made it work. That was quite easy for me because our dev servers already have the Xdebug extension installed. The only bit that took any time was working out where to put the phpini settings our our servers.

I hacked around with the page a bit: http://docs.moodle.org/en/index.php?title=Development%3AProfiling_PHP&diff=77813&oldid=77779. In particular, I changed the name from %s to %R, since that seems to include less of the redundant path information. I also put that first, since it seems more interesting that the time-stamp for most people.

WinCacheGrind seems to do a good job of displaying the results. So, thank you very much. You have a convert.

Average of ratings: -
In reply to Tim Hunt

Re: Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

Thanks for revising it Tim, your changes are indeed useful.

I have just came accross Easy Xdebug plugin for FireFox that has profiling control toggel button. It inserts XDEBUG_PROFILE variable into cookie data, thus making profiling enabled for any queries (post form data, ajax stuff). I have added this information to wiki page.

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Dan Poltawski -
Is it odd that we're calling 
settings_navigation->load_administration_settings()
22 times for a normal user and 241 times for an admin?
Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Chad Outten -

Interesting data. Thanks for sharing.

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

That's a really great report. Thanks for posting!

I had been expecting it would be slower than 1.9. There are some reasons why it's fundamentally slower (i.e. it does more - navigation block on every page, for instance) but also probably there is lots of opportunity for optimisation.

For a positive spin on these results, let's assume the final version of Moodle 2 will be released, for instance, on 3rd March 2011. That's exactly three years after Moodle 1.9. Conventionally, Moore's Law is taken* to suggest that computer performance doubles every eighteen months - so in three years, computer performance should have gone up by a factor of four. Even without any more optimisation, the performance differential is only about three. This therefore means that Moodle 2.0 running on the servers available in March 2011 should actually be faster than Moodle 1.9! (when it launched)

--sam

* Inaccurately, but I don't care.

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Jonathan Moore -

Great work Ruslan! This sort of analysis is really important for anyone using Moodle in a mission critical setting. Thanks for taking the time to do this.

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Bryan Williams -

Ruslan,

I'm wondering if you might like to try this on a server optimized for production Moodle use. We wouldn't mind seeing what the results look like on something other than a desktop computer. Jonathan and I are happy to provide the resources and work with you on this.

Average of ratings: -
In reply to Bryan Williams

Re: Moodle performance testing: 1.9 vs. 2.0

by Dan Poltawski -
Hi Brian,

This is what we are going to be looking at next with 'production' servers and a large number of clients generating traffic to simulate real use. We'll post the results and keep you informed!

Dan
Average of ratings: -
In reply to Bryan Williams

Re: Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

Time to share next portion of results.

This time load tests were run on production system:
Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 48Gb or RAM, Debian/GNU "Lenny", 2.6.26-2-vserver-amd64.
Two virtual guest were created, one for web server: Debian/GNU stable/testing mix, Apache2 2.2.9-10+lenny8 (mpm-prefork), php5 5.3.3-2; another one for database server: Debian/GNU "Lenny", postgresql 8.3.12-0lenny1. Measuring was run from another system, JMeter 2.3.4 was used.

Default configuration was used for postgresql, for apache some config changes were done, in particular:

<IfModule mpm_prefork_module>
StartServers 20
MinSpareServers 40
MaxSpareServers 50
MaxRequestsPerChild 20
ServerLimit 2048
MaxClients 2048
</IfModule>

The same enhanced JMeter testplan generation plugin as in previous test was used for generating performance tests on Forum, Chat and Quiz activities in 1.9 and HEAD (2.0) versions. Test was done several times with different number or threads (simultaneous users): {10, 50, 100, 200, 300}, each user made 5 posts in corresponding activity (in Quiz only one submission was done). Static content retrieving (images, css) are not included in the testplan, so the time only reflects php script output rendering on the client side.

Below are the results for different Moodle pages. Each graph shows average time required for page retrieving for different number of users. For those pages, size of output data for which was not changing through numerous requests (e.g. course/view.php was the one with constant output data size during the test, while output data size of mod/forum/post.php was growing during the test), time/size ratio was calculated. This was done to compare pure code performance irrespective to the script output data amount. Interesting enough, time/size ratio (which is in fact time in milliseconds required to generate 1 byte of data) for some scripts demonstrated that Moodle 2.0 has better performance. Unlike it was in the test with 10 users, differences in time required for fetching the page between two Moodle versions on production server is not significant.

 

Results in xml and matlab format, original JMeter reports in csv and Matlab scripts for images generation are avialable for download here.

Average of ratings: Useful (3)
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm annoyed that mod/quiz/processattempt.php is coming out as slower than teh equivalent in 1.9.

However, what are we measuring here? In 1.9, the POST to mod/quiz/attempt.php then immediately displays the next page, leaving you looking at the result of a post request, so reload and back are dangerous. In 2.0 processattempt.php finishes by redirecting back to attempt.php.

So if your graph is comparing the time for two requests in 2.0 with one request in 1.9, that is no meaningful. If the graph is just showing the time for the processattempt.php request, without the redirect, then I am really annoyed.

Average of ratings: -
In reply to Tim Hunt

Re: Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

You are right Tim, for this particular case (POST to processattempt.php) we are measuring both: processattempt.php with redirection and output of attempt.php

Though, I do not think the data is meaningless, as this reflect performance evaluated from user perspective. User will not notice pages with response code 303, for user it will look like a single page is being loaded and takes a particular amount of time, no matter how many redirects have been made ;))  Profiler might be more useful and accurate than multi-user load testing for analysing and comparing certain scripts.

As I said earlier, differences between versions are not significant in these test on production system. It is more for observations how much time page load might take with "high-density" simultaneous requests of the same pages.

By the way, forgot to mention that php-apc was also installed and enabled on tested system.

Average of ratings: Useful (1)
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Sure. (And the wording of my previous post should not be taken seriously, I was messing around.) However, redirect after POST is strongly recommended. It is the right way to do things. So it would be particularly worthwhile to profile processattempt.php, or other similar pages that should just process a POST then redirect. They should be really fast. If we find pages like that are spending time initialising the navigation, and so on, then there is some serious (but quite doable) optimising to be done.
Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Derek Chirnside -

This is a lot of Geek talk, but it is great to eavesdrop.

Here is my question: if an institution wanted to launch Moodle 2.0 in February next year and they were prepared for 1) a few glitches (as bugs get uncovered) and 2) a slightly cut down system (to solve the speed thing) is that possible?

eg.  (and I'm just guessing here) lets say we

  • shut off reporting (Just trying to think of anything that may bring an overhead)
  • Removed all blocks that could slow down load time
  • did not use site wide groups (Just thinking aloud here, maybe that is something that could add overhead . . .
  • ??

Are these speed tests quick and easy to do?

If you went in and disabled things as I've suggested, will this lead to a speed improvement?

I'm kind of looking for an apples for apples comparison.  Same box, new Moodle, similar speed (or at least not too much different)

Why I'm asking this: I know of three groups locally that are wanting to do Moodle for Februay 1.

The overhead of the change Moodle 1.9 (great version) > Moodle 2.0 (Still un-optimised) LATER in the year may be a greater pain than have a Moodle 2.0 with some hacks to improve the speed while things get optimised.

Why leap a wide chasm in two bounds?

Guys, I hope I have expressed myself clearly. THANKS

Average of ratings: -
In reply to Derek Chirnside

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Well, hopefully by February, the code will have been streamlined a bit. So, go with 2.0, but be prepared to update to the latest 2.0+ WEEKLY build from time to time.

I don't think the measures you suggest are likely to make that much difference to performance. The main problem seems to be the navigation block, and there is no way you can shut that off. There do seem to be plenty of ways that we can make that faster, so by February you should be alright.

How many users are you expecting anyway? Especially at first.

Finally, remember sam's slightly tongue-in-cheek post above (http://moodle.org/mod/forum/discuss.php?d=162045#p709461) thanks to Moore's law, Moodle 2.0 is faster on equivalent hardware than 1.9 when it was released.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Moodle performance testing: 1.9 vs. 2.0

by Derek Chirnside -

OK, thanks Tim.

Just curious: What about other blocks?  Are there any other 'slow' to execute blocks you could avoid?

Users, Hmm, low volumes of simultaneous logged in users (~4 classes of 30 max maybe). Maybe 20 classes max.  The plan is short term (ie February) trials with a few classes.  Non mission critical stuff, but still a good run through.

Average of ratings: -
In reply to Derek Chirnside

Re: Moodle performance testing: 1.9 vs. 2.0

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Well, the only really by doing profiling that you can identify which parts of the code area slow, and when we identify those, we should make them faster, rather than turn them off.

And it is not so much the navigation block, which just displays the navigation, that is the problem. It is the back-end code that tracks the navigation structure and where you are in it.

Average of ratings: -
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Gail Preuninger -

Hello Rusian,

Which MySQL database engine were you using - MyISAM or InnoDB?  Have you tested performance differences between MyISAM and InnoDB MySQL tables?  Thank you for any recommendations on which database engine to use.

Average of ratings: -
In reply to Gail Preuninger

Re: Moodle performance testing: 1.9 vs. 2.0

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

Hello Gail,

For this particular tests InnoDB engine was used. I had to mention it, sorry about that.

Regarding performance differences between MySQL engines, assiming that MySQL configuration is well-tuned,  MyISAM is slightly faster for SELECT queries. For the mixture of SELECT/INSERT/UPDATE, InnoDB is much faster. Apart of that InnoDB comes with some other advantages. The most important thing is that InnoDB supports transactions, which is quite important feature for maintaing data integrity, and Moodle already goes towards supporting transactions in the core. Also, backing up of InnoDB tables is faster and can be done without locking single tables.

So, if you are choosing between these two engines, I would recommend you to choose InnoDB, however if you are choosing a database for the new server, especially if you expect a high load, I would strongly recommend considering PostgreSQL instead wink

Average of ratings: Useful (3)
In reply to Ruslan Kabalin

Re: Moodle performance testing: 1.9 vs. 2.0

by Derek Chirnside -

with 2.0.1, have any of you guys re-run these tests?

-Derek

Average of ratings: -
In reply to Derek Chirnside

Re: Moodle performance testing: 1.9 vs. 2.0

by Séverin TERRIER -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

You mean with Moodle 2.0.2 wink

Average of ratings: -
In reply to Séverin TERRIER

Re: Moodle performance testing: 1.9 vs. 2.0

by Wen Hao Chuang -

Dear all:

Very interesting and helpful thread. Just wondering whether there is any recent update about performance comparison with the most recent Moodle 2.1.1 (as of August 31, 2011). Would love to learn more about it. Thanks!

Average of ratings: -