Moodle 2.5 performance testing at the OU

Moodle 2.5 performance testing at the OU

by Tim Hunt -
Number of replies: 17
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

We have been performance testing Moodle 2.5.2 at the OU, in preparation for deploying it in December. Here is a very brief summary of what we found.

Our test script tests the most common pages our students user, something like

  • Visit course page
  • Visit some resources
  • Visit some forum threads
  • Make a forum post

On the test hardware, Moodle 2.4.3 (the version we are currently running) was giving 0.33s average page-load times.

Our initial run with Moodle 2.5.2 gave 0.41s.

The basic issue there is that there are a lot of new caches, which default to being stored on disc in moodledata. In a clustered environmant, moodledata is on a networked file system, so quite slow.

So, as expected, we had to re-configure most of the new caches to use memcache. The singificant change came from moving the pluginlist cache to memcache. That brought the average times down to 0.36s. None of the other caching changes made a measureable difference.

That is about where we are at. We are going to have to live with our 2.5.x being 10% slower than 2.4.x.

 

Note that we are using Eric Merrill's clustered memcache implementation: MDL-42071 for some of the caches. (Thanks Eric.) However, it seems to be fractionally slower than the local memcache hack we were using before. We don't know why. If we can work out what is going on there, we will post in the tracker issue. It is still much faster than non-clustered memcache for read-heavy caches.

 

(As an aside, I think our results reinforce my point that in the testing that HQ do - e.g. http://docs.moodle.org/dev/Moodle_2.6_release_notes/Performance_tests - they really need a set-up with moodledata on a different box from their web server.)

Also, I should point out that I am just posting these results. I have not done any of the work to generate them.

Average of ratings:Useful (9)
In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by David Scotson -
Hi Tim,

couple of questions for you:

1. What are you measuring to get that 0.33s, 0.41s and 0.36s "page-load" figure? I assume that it's how long the server is working for as it seems very quick for a full page load (we see 0.12 seconds just to download the average Moodle page and about 0.1-0.3s difference between "document interactive" and "document content loaded")

2. Do you happen to know if the "Performance Info" that Moodle can currently append to pages can be logged instead? (edit: answering my own question via Google: the MDL_PERFTOLOG setting seems to do this)

I was reading the "Mature Optimisation Handbook" that Facebook released and it suggested that logging that kind of info on every access in production was an important part of their optimization process (specifically Chapter 5. Instrumentation)

http://carlos.bueno.org/optimization/

It would probably be interesting to see that info for test runs like you describe. And actually, a third question, how are you scripting those? We've been using wget for similar stuff but maybe there's better tools out there?
In reply to David Scotson

Re: Moodle 2.5 performance testing at the OU

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

1. What we are measuring is exactly what we measure on our live servers as page-load time. I don't acually know that technology that is, but we call them the TTM logs. There is a rather old screen-grab here: http://learn1.open.ac.uk/mod/oublog/viewpost.php?post=12880 (The window with the graphs.)

I think what this is showing is the time recorded in the Apache logs, so basically wall time used by the PHP script.

However, we are not really interested in the absolute time, which as you say, does not necessarily mean much to users. Also, the time on our load-testing servers in not necessarily the same as on our live servers. What we are really interested in is the relative performance of Moodle 2.4 and 2.5. So, when we upgrade our live system to 2.5, will it get slower?

2. You answer yourself, but yes, we do monitor our live servers quite carefully. (E.g. from the TTM logs, you can see whch server is running cron, and when it ran. You can also see the times when someone backed up a course, or did another slow operation.)

 

3. We generate the test load using JMeter. We wrote an add-on that generates JMeter scripts for a given Moodle site. (That is, the script needs to contain URLs that contain course ids, etc., and our add-on does that.) Other people have since enhanced it. You can get it from https://github.com/lucisgit/moodle-jmeter-script-generator.

Average of ratings:Useful (3)
In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

A quick update on this.

We found some discrepancies between what the 2.4 and 2.5 JMeter scripts were doing, and when we altered the 2.5 script to be more directly comparable to what the 2.4 test was doing, the average page-load time for 2.5 went down to 0.32 seconds (compared to 0.34 seconds in the most recent 2.4 run.)

So, Yay! party! we now think 2.5 is about 10% faster then 2.4 smile

Average of ratings:Useful (3)
In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Albert Ramsbottom -

Hi

Great information here, and thanks for the update

Its good to know that 2.5 is 10% faster than 2.4, well at least for you. Still how many servers have you got to run your million or so users? tongueout

 

In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Brian King -

Hi Tim,

Are the changes you made available someplace?  On https://github.com/lucisgit/moodle-jmeter-script-generator the latest commit is one year ago.

 

In reply to Brian King

Re: Moodle 2.5 performance testing at the OU

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Sorry, don't know.

However, perhaps it would be better to standardise on what Moodle HQ are doing. David Monllao build a similar system that they used to produce http://docs.moodle.org/dev/Moodle_2.6_release_notes/Performance_tests. There is a wiki page about that, but I can't find it now.

In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Brian King -

OK, thanks.  I guess you mean http://docs.moodle.org/26/en/JMeter_test_plan_generator .  Which is great, but at the moment doesn't test anything related to quizzes, which is often an important part to test.

Average of ratings:Useful (2)
In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Wen Hao Chuang -

Hi Tim, thanks so much for this wonderful thread!

Just wondering. Did you find large mdl_log table continue to be a potential thread to overall performance in Moodle 2.4.x and 2.5.x? We have a site that will have around 42000 users and 2000 courses which we haven't decided whether it's ok to change the default site setting "Keep logs for __ days" from 30 days to 365 days (the administrator and faculty prefer to). Any thoughts about this please? Thanks!

Wen

In reply to Wen Hao Chuang

Re: Moodle 2.5 performance testing at the OU

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

We had to solve the mdl_log table long ago.

In Moodle 2.x we use an approach that that which does not require any changes to Moodle code. I am sure we have described it before in these forums, but here is a brief summary:

First we use Postgres, which is just the best database.

Then, mdl_log is actually a view which is a UNION of several real tables. I think we have one table per month. Attempts to write to the view are re-directed to the table for the most recent month, which is therefore quite small. The view makes the last two or three months of logs available for reading. Logs older than that are kept, but need to be queried manually if they are really needed. (The also get shipped off to a data warehouse for analytics.)

Average of ratings:Useful (5)
In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Wen Hao Chuang -

Hi Tim, thanks so much for the feedback!

It sounds like you were talking about how OU had to solve the mdl_log table issue. We are hosted with a Moodle partner and they are using mysql instead of Postgres. Do you think that might make a difference in terms of huge mdl_log table might bog down mysql server performance? Any experience about this please? Thanks!

In reply to Wen Hao Chuang

Re: Moodle 2.5 performance testing at the OU

by Daniel Tran -

Hi Wen,

IMHO, having a huge mdl_log may affect mysql performance if your innodb buffer pool is too small or smaller than the actual database size. You can check your db size vs. memory and figure out an acceptable time frame to keep your log before purging it.

In reply to Wen Hao Chuang

Re: Moodle 2.5 performance testing at the OU

by Daniel Tran -

Hi Wen,

IMHO, having a huge mdl_log may affect mysql performance if your innodb buffer pool is too small or smaller than the actual database size. You can check your db size vs. memory and figure out an acceptable time frame to keep your log before purging it.

In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Andreas Panagiotopoulos -

Tim, could you please be more specific?

Where can I find the postings about spliting mdl_log into views and having one of them to be read from moodle? I have the same problem and I can't find any relative post..

Thank you much!

Andreas

In reply to Andreas Panagiotopoulos

Re: Moodle 2.5 performance testing at the OU

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Here is a very quick summary. In our DB, mdl_log is a view defined like:

CREATE OR REPLACE VIEW mdl_log AS
        (         SELECT mdl_log_2014_02_12.id, mdl_log_2014_02_12."time", mdl_log_2014_02_12.userid, mdl_log_2014_02_12.ip, mdl_log_2014_02_12.course, mdl_log_2014_02_12.module, mdl_log_2014_02_12.cmid, mdl_log_2014_02_12.action, mdl_log_2014_02_12.url, mdl_log_2014_02_12.info
                   FROM mdl_log_2014_02_12
        UNION ALL
                 SELECT mdl_log_2014_02_05.id, mdl_log_2014_02_05."time", mdl_log_2014_02_05.userid, mdl_log_2014_02_05.ip, mdl_log_2014_02_05.course, mdl_log_2014_02_05.module, mdl_log_2014_02_05.cmid, mdl_log_2014_02_05.action, mdl_log_2014_02_05.url, mdl_log_2014_02_05.info
                   FROM mdl_log_2014_02_05)
UNION ALL
         SELECT mdl_log_2014_01_01.id, mdl_log_2014_01_01."time", mdl_log_2014_01_01.userid, mdl_log_2014_01_01.ip, mdl_log_2014_01_01.course, mdl_log_2014_01_01.module, mdl_log_2014_01_01.cmid, mdl_log_2014_01_01.action, mdl_log_2014_01_01.url, mdl_log_2014_01_01.info
           FROM mdl_log_2014_01_01;

CREATE OR REPLACE RULE mdl_log_delete AS
    ON DELETE TO mdl_log DO INSTEAD NOTHING;

CREATE OR REPLACE RULE mdl_log_insert AS
    ON INSERT TO mdl_log DO INSTEAD  INSERT INTO mdl_log_2014_02_12 ("time", userid, ip, course, module, cmid, action, url, info)
  VALUES (new."time", new.userid, new.ip, new.course, new.module, new.cmid, new.action, new.url, new.info);

Those definitions get modifed from time to time (new mdl_log_YYYY_MM_DD table created, and view changed to add the new one and no longer include the oldest one. The older mdl_log_YYYY_MM_DD tables are just left sitting around, making backups a pain, in case someone wants to use them for analytics some time in the future.

 

There is only one change in the code required to make this work. In get_columns in lib/dml/pgsql_native_moodle_database.php you need to change WHERE relkind = 'r' to WHERE relkind IN ('r', 'v').

Average of ratings:Useful (3)
In reply to Tim Hunt

Re: Moodle 2.5 performance testing at the OU

by Andreas Panagiotopoulos -

Thank you Tim for your analytical explanation..

Unfortunately I am using MySQL and the corresponding function at the MySQL file (mysqli_native_moodle_database.php) does not have the same WHERE statement.

So the only way may be probably to find a way to manually make this split?

 

Andreas

In reply to Andreas Panagiotopoulos

Re: Moodle 2.5 performance testing at the OU

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

The above trick is only possible in Postgres, I think. I you use another database, you will have to come up with your own solution.