We are currently doing some performance profiling between moodle 1.5.4+, 1.8.4(+) and 1.9 (beta4). Just wondering if anyone have done such performance comparison (based on different moodle modules, use cases, etc.) between these three versions, and if so, is there any published online report somewhere? I'm particularly interested to know after the improvement of the accesslib.php in 1.9, how much performance gain (especially the number of db queries) 1.9 has over 1.8. Thanks!
Hi all:
We are currently doing some performance profiling between moodle 1.5.4+, 1.8.4(+) and 1.9 (beta4). Just wondering if anyone have done such performance comparison (based on different moodle modules, use cases, etc.) between these three versions, and if so, is there any published online report somewhere? I'm particularly interested to know after the improvement of the accesslib.php in 1.9, how much performance gain (especially the number of db queries) 1.9 has over 1.8. Thanks!
We are currently doing some performance profiling between moodle 1.5.4+, 1.8.4(+) and 1.9 (beta4). Just wondering if anyone have done such performance comparison (based on different moodle modules, use cases, etc.) between these three versions, and if so, is there any published online report somewhere? I'm particularly interested to know after the improvement of the accesslib.php in 1.9, how much performance gain (especially the number of db queries) 1.9 has over 1.8. Thanks!
is there any published online report somewhere?
I have taken a ton of numbers, micro-benchmarking things while I was working on it. The original threads in this forum discussing the accesslib rework and the commit messages are a good starting point. Those commit messages specially have a ton of good information of how the performance profile changes with the rework.
Probably google will help find most of those
If you are doing profiling, I would suggest
- Test at least 2 different databases, say one with 10K users and 3K courses, and one with 100K users and 50K courses. Both should have a nice mix of enrolments, and do load course content, in each course, even if it is an automated restore of the "moodle features demo".
- Once created, use vacuum analyze full ( in Pg) or MySQL's myisamchk to build good index stats for the planner.
- Make sure you have an interesting category structure -- many courses should be in deeply nested categories, and you must test those courses in comparison to shallow-placed courses.
- Measure the session file size. Sessions shrink dramatically -- and large sessions have a huge impact on load-balanced clusters.
- Test with rcache off and on. The "internal" mode is mostly useless - ensure you have eaccelerator or similar and use that.
Areas where I would test
- Login of different users - note that login has 2 requests: the POST to login.php and then the load of index.php. The initial login session is done in the POST, and then some stuff happens in that first GET, so after that, do a reload of index.php to diff the stats between the execution of index.php that finishes the session setup, and a "normal" exec of index.php
- Browsing though course categories as differnt users (specially different roles). Make sure you have one category with a few thousand courses, paginnate through it, in normal and in "edit mode".
- Course pages (course/view.php) exercise accesslib a lot. With 1.9, the first time in a user session that you load the course page, some extra info is loaded. Compare with a second load of course/view.php
- Petr has done some significant work in forum thread view. I haven't seen the numbers but I think here on moodle.org you can feel it snappier already. Forum mailouts are also much better from his work -- I merged in a mailout debugger that is useful to test that.
I also have some further patches that I will merge into HEAD later for 2.0 that speed things up in other areas (text cache, and base load), that free up the DB from unneeded traffic and give Moodle a good 20%-faster kick when under heavy traffic. Initially done to speed up usage on Oracle backends, those patches make a nice difference on Pg and MySQL too
There were significant performance improvements recently in other areas too:
- course page - recent history, blocks in general
- cron - forum mailing, mail sending, statistics processing
- course reports and statistics
- fixed major memory leak in record cache
- grader report
- etc.
It is still not finished, if somebody finds a page that is really slow, please file a bug into tracker and assign it to me (I hope I will get to performance on My Moodle page soon).
I would like to see Moodle 1.9 much faster, eating less memory and in general more suitable for large deployments and web hosting. There are very many sites still on 1.6.x, we have to make all we can to persuade them to upgrade. This includes usability work and documentation too
In reply to Martín Langhoff
Re: Performance profiling between 1.5, 1.8, and 1.9
by Martin Dougiamas -
I'm hoping to have some really interesting general numbers of this kind later this week. 
In reply to Martin Dougiamas
Re: Performance profiling between 1.5, 1.8, and 1.9
by Wen Hao Chuang -
Martin, that would be wonderful! Thanks!
By the way I'm also curious about the approach that you chose to come up with these comparison numbers. When you have time could you please elaborate this a little bit? Thanks!
By the way I'm also curious about the approach that you chose to come up with these comparison numbers. When you have time could you please elaborate this a little bit? Thanks!
In reply to Martin Dougiamas
Re: Performance profiling between 1.5, 1.8, and 1.9
by Wen Hao Chuang -
Hi Martin have you heard anything from moodleroom yet? just curious..
Haven't seen any white paper published by moodlerooms yet, but here at SFSU we did some generic internal performance comparison between Moodle 1.5, 1.8.4, and 1.9 base on three criteria:
Based on the above criteria, here are the scores that we came up with (in terms of the performance):
Moodle 1.5.4+: 72
Moodle 1.8.4: 54
Moodle 1.9: 60
A quick conclusion is that because of the ROLES implementation, in general 1.9 (performance-wise) is still not as good as 1.5, however, it is better than 1.8.4 (probably because of the rewrite of the accesslib.php)
Again these numbers are still kind of "primitive" (and might not be really representative for all areas) as someone could argue with the current weighting, but I only had 3.5 days to come up with this report (including learning JMeter!) I'm hoping that I could spend a little bit more time to polish this report so that it would be more useful for people out there who are considering to upgrade from 1.5 to 1.9, or from 1.8 to 1.9. Hopefully this is the first step and could be of some help with some folks out there..
Blessings,
Wen
- Internal performance comparison using JMeter (weight: 80%)
- Case #1: Simulating 100 concurrent users hitting our testing server, and do passive browsing with different pages
- Case #2:
- Simulating 20 users login as teachers simultaneously and
- Create resources (compose a text page)
- Creating label
- Creating forum
- Creating assignment
- another 20 users login as student, passively browsing
- Course view
- Forum
- Resource
- Assignment
- Case #3: Login as admin and going into test case course, and go to "Gradebook view", then compare the # of DB queries
- Case #4: Login as teacher and compare with long forum thread (forum view).
Based on the above criteria, here are the scores that we came up with (in terms of the performance):
Moodle 1.5.4+: 72
Moodle 1.8.4: 54
Moodle 1.9: 60
A quick conclusion is that because of the ROLES implementation, in general 1.9 (performance-wise) is still not as good as 1.5, however, it is better than 1.8.4 (probably because of the rewrite of the accesslib.php)
Again these numbers are still kind of "primitive" (and might not be really representative for all areas) as someone could argue with the current weighting, but I only had 3.5 days to come up with this report (including learning JMeter!) I'm hoping that I could spend a little bit more time to polish this report so that it would be more useful for people out there who are considering to upgrade from 1.5 to 1.9, or from 1.8 to 1.9. Hopefully this is the first step and could be of some help with some folks out there..
Blessings,
Wen
Wen, this is very odd.
I remember suggesting several things for you to try that actually focused on key pages that were problematic in 1.7/1.8 and completely bottlenecked. The performance of 1.7/1.8 is much worse that it appears there - you seem to be measuring the wrong things...
To see the difference in full effect you also need a real-life-sized database with a few thousand courses, and a few tens of thousands of users accounts.
Nico does some amazing stuff, but he hasn't worked that much on performance, or on large sites. Petr has. I have. I don't think 30/40% is right. I have measured it, and most pages have a 10% improvement, but the main thing is that the bottlenecks are gone - meaning many important pages are 100 or 1000 times faster.
This was against 1.8.2 - I would say that 1.8.4 has quite a few tricks backported from 1.9 which hide the bottlenecks a bit, but they are there, and you hit them head on as soon as you have a real-life-sized DB just by logging in. So logins in 1.8.x will take 100 times longer or more for large sites and can grind the whole server to a halt -- and create large sessions files that cause a lot of trouble. Similarly with paginating though the courses/categories list. There's a more comprehensive list of things to try in my earlier posts.
In reply to Martín Langhoff
Re: Performance profiling between 1.5, 1.8, and 1.9
by Martín Langhoff -
Heh - not just "remember" - it was this very same thread
Hi Martin, we do have a real-life-sized DB with 40,000+ courses and 61,100+ users for each testing instances, (populated with pseudo data). I guess it also depend on which area you are testing it with. For us, based on our current usage on our live site (in Site Administration block, Modules -> Activities -> Manage activities), these modules are mostly heavily used:
1. Resource (116000+)
2. Label (40000+)
3. Forum (30000+)
4. Assignment (23000+)
That's why I designed the JMeter tests focus on these specific areas. I agreed with you that when you compare again 1.8.2, with the amount of testing data that I mentioned above, it would really bring the whole server to a halt for some cases (that's the reason why we decided not to upgrade to 1.8.2 last summer). However as I mentioned, we were comparing 1.5.4+, 1.8.4 (not 1.8.2!!), and 1.9 (beta4) this time, and that was the "rough" performance result that we get.
1. Resource (116000+)
2. Label (40000+)
3. Forum (30000+)
4. Assignment (23000+)
That's why I designed the JMeter tests focus on these specific areas. I agreed with you that when you compare again 1.8.2, with the amount of testing data that I mentioned above, it would really bring the whole server to a halt for some cases (that's the reason why we decided not to upgrade to 1.8.2 last summer). However as I mentioned, we were comparing 1.5.4+, 1.8.4 (not 1.8.2!!), and 1.9 (beta4) this time, and that was the "rough" performance result that we get.
In reply to Wen Hao Chuang
Re: Performance profiling between 1.5, 1.8, and 1.9
by Visvanath Ratnaweera -
Hi
I'm interested in the database you mention. Could you give more details?
Is it suitable for versions from 1.5.4 to 1.9?
How do you benchmark such a Moodle installation?
I'm interested in the database you mention. Could you give more details?
Is it suitable for versions from 1.5.4 to 1.9?
How do you benchmark such a Moodle installation?
In reply to Visvanath Ratnaweera
Re: Performance profiling between 1.5, 1.8, and 1.9
by Wen Hao Chuang -
Hi Visvanath, I could probably do a mysql dump with the 3 DBs and send you a copy but the file probably would be really big. Here at SFSU we developed a script that would generate tons of pseudo data that we use for testing purposes, but because we still haven't found time to document the script so we haven't released it yet. I believe that someone (Nicolas??) also wrote a script that could do similar thing, if you have moodle cvs access you could find it in /contrib/tools/generators/generator.php
The script that we wrote have two versions, one for 1.5.4, the other for 1.8 and 1.9. As we also did some database-level changes (for mdl_course and mdl_user) that's why there are two different versions.
The way that we benchmark such Moodle installation was briefly described as above. Basically we popular these three databases with equal amount of pseudo data first (for 1.5.4+, 1.8.4, and 1.9. Each with 40,000+ courses and 61,100+ users). Then we use JMeter (JMeter proxy) to come up with testing scripts to simulate real traffic pattern (focus on the most frequently used areas, such as Resource, Label, Forum, Assignment), then observe the server throughput and CPU/memory/IO usage using "top" and the build-in grapher in JMeter and compare these three versions (1.5.4+, 1.8.4, and 1.9).
My next step is to write a script to profile apache error_log to find out more slow loading pages when the server is under heavy load. I will have to read more about JMeter to find out more about how to generate randomized input (such as random course ID, user ID..etc.)..
Hope this helps,
Wen
The script that we wrote have two versions, one for 1.5.4, the other for 1.8 and 1.9. As we also did some database-level changes (for mdl_course and mdl_user) that's why there are two different versions.
The way that we benchmark such Moodle installation was briefly described as above. Basically we popular these three databases with equal amount of pseudo data first (for 1.5.4+, 1.8.4, and 1.9. Each with 40,000+ courses and 61,100+ users). Then we use JMeter (JMeter proxy) to come up with testing scripts to simulate real traffic pattern (focus on the most frequently used areas, such as Resource, Label, Forum, Assignment), then observe the server throughput and CPU/memory/IO usage using "top" and the build-in grapher in JMeter and compare these three versions (1.5.4+, 1.8.4, and 1.9).
My next step is to write a script to profile apache error_log to find out more slow loading pages when the server is under heavy load. I will have to read more about JMeter to find out more about how to generate randomized input (such as random course ID, user ID..etc.)..
Hope this helps,
Wen
In reply to Wen Hao Chuang
Re: Performance profiling between 1.5, 1.8, and 1.9
by Visvanath Ratnaweera -
Hi Wen
Thanks for the information. I'm very much interested. Will have a closer look and come back.
One question come to my mind though: Have you considered "anonymizing" a real database? The problem with genarated data is that one can always question the relevence to a real situation.
Thanks for the information. I'm very much interested. Will have a closer look and come back.
One question come to my mind though: Have you considered "anonymizing" a real database? The problem with genarated data is that one can always question the relevence to a real situation.
In reply to Visvanath Ratnaweera
Re: Performance profiling between 1.5, 1.8, and 1.9
by Wen Hao Chuang -
Hi Visvanath, yes we thought about "anonymizing" a real database, but then again since real "content" will still be there (e.g. student uploaded assignments, etc.) so we decided this is not a good idea. Yes I agree with you if you could really get permission from all of your users about "anonymizing" a real database for testing purposes then of course this would be much closer to a real situation (or database).
We are in the process to come up with a really good "test case course" with lots of pseudo data (e.g. lots of quizzes, covered with all different question types..etc.), that is particularly designed for Moodle 1.9. We hope this "test case course" could later replace the one that is located at http://download.moodle.org/testcourse/ which was designed for Moodle 1.6 and is kind of old... will keep everyone updated.. stay tuned.
We are in the process to come up with a really good "test case course" with lots of pseudo data (e.g. lots of quizzes, covered with all different question types..etc.), that is particularly designed for Moodle 1.9. We hope this "test case course" could later replace the one that is located at http://download.moodle.org/testcourse/ which was designed for Moodle 1.6 and is kind of old... will keep everyone updated.. stay tuned.
Hi Wen. I'm particularly interested in your "test case course", but more from a functionality/regression-testing kinda point of view rather than a performance testing one. We'd like to build a database of courses/students/content that resembles something like our live environment that we could test against (maybe using Selenium or the like) then restore the dbase back to the original state and repeat again.
Be great if the database was a generic as possible so it was useful to other Moodle sites, but I guess there's a problem of creating something that is representative of a "typical Moodle environment"
Any threads or discussions you know of that have covered this in the past?
edit just spotted this: http://moodle.org/mod/forum/discuss.php?d=92050
Be great if the database was a generic as possible so it was useful to other Moodle sites, but I guess there's a problem of creating something that is representative of a "typical Moodle environment"
Any threads or discussions you know of that have covered this in the past?
edit just spotted this: http://moodle.org/mod/forum/discuss.php?d=92050
In reply to Wen Hao Chuang
Re: Performance profiling between 1.5, 1.8, and 1.9
by Visvanath Ratnaweera -
For this GSOC occasion I've formulated the idea at last http://docs.moodle.org/en/Student_projects#Benchmark-Framework_for_Moodle
HINT to all: Co-mentors welcome! Contact me directly.
HINT to all: Co-mentors welcome! Contact me directly.