A lot of people use the Performance perspective script to give a general prediction of what their Moodle setup is capable of. But more often than not, people encounter performance issues once their Moodle installation has been up and running for a while, and they cannot then run this script with live data. Additionally, the script only benchmarks the Moodle API, and ignores any script caching and web server optimisations that may be in place (not Moodle Record caching).
I notice that Moodle.org now has a JMeter page (http://docs.moodle.org/en/JMeter), however there's not much guidance about benchmarking your own Moodle installation.
So, I wondered if JMeter using Moodler's wanted to swap JMeter test scripts or ideas?
I might add you probably shouldnt stress your server too hard unless you own the hardware its running on - hosting companies would not be happy to see there servers crash because of your tests.
- Check homepage
- Check a course category
- Log in
- View my Moodle page (logged in)
- View a course (logged in)
- View a forum discussion (logged in)
- View homepage (logged in)
Which also seems to be a good check in JMeter. I'm currently in the midst of checking various server setups for Moodle, so it's a good comparison.
Fortunately I have a test box up and running, a bit of an unrealistic arrangement, but good to compare different configurations.
I'm installing Moodle for a french organization and we need to test, but we don't have much time.
Do you know where it could possible to download the jmeter scripts you're talking about? It will be so helpfull.
Thank's in advance and excuse my poor and bad english.
Actually, I would like to get a web testing framework integrated into Moodle, similar to how we have an integrated unit test framework. That way, we could start building up a body of integration tests.
Anyway, a low-tech way to easily share what JMeter scripts we have would be a good start.
In the first instance, we should have separate subfolders of /contrib/tools/jmeter where people can just dump the scripts they have made, and other people can get them and reuse them. However, that is a very low-level type of sharing, useful, but we should do more.
What more, well we need a volunteer to start organising things
- catalogue what we have and what, so we don't have 10 scripts testing posting to a forum, and no test for the glossary, say.
- work out techniques so that scripts will run easily of many different Moodle sites, and are not tied to one specific site.
- work on a standard Moodle test suite, that nicely packages up scripts for a lots of moodle with easy instructions so that anyone can run them.
What I am trying to say is that the open source magic will be most effective if everyone is contributing to a common set of tests. But, as a first step, lets just get whatever scripts we have out there in public.
I've attached the JMeter script that we recently used to benchmark various setups. The script does the following actions:
- Views a course listing page
- Views the home page
- Logs in (and is redirected to /my)
- Views a course page (whilst logged in)
- Views a forum (whilst logged in)
- Views the home page (whilst logged in)
In terms of testing methodology, we benchmarked our server with 25, 50, 75, 100, and 150 threads; leaving out the first and last where appropriate. If you then save the overall Aggregate Report, it will give you a fair idea of what the various setups are capable of.
As a complete novice in moodle (I am a Unix systems specialist) could you point out the lines in the script which will need changing?
Many thanks ......
The things you will need to change, are:
- Test Plan/Load Test
Update the Number of Threads accordingly. As above, I'd suggest 25-250 in 25 or 50 thread increments, depending on when you hit your 'ceiling'.
- Test Plan/Load Test/HTTP Request Defaults
It's very important to change the Web Server Name or IP to your own Moodle installation. Else you'll be DDOS'ing Essex's Moodle server.
- Test Plan / Load Test/Course Listing
Change /course/category.php?id=14 to an ID that your server uses.
- Test Plan/Load Test/Login
Change username and password in the Send Parameters with the Request section to a valid username/password for your Moodle setup. Note, I'd set this to a standard teacher/student, admin users have a lot more queries run for them.
- Test Plan/Load Test/HS604
Change /course/view.php?id=85 to an ID that your server uses.
- Test Plan/Load Test/HS604 - Forum
Change /mod/forum/view.php?id=4778 to an ID that your server uses.
Although the page might be returned correctly from PHP/Moodle, the assertions look for specific response codes (200) and text (which varies by page).For example, the Response Text assertion on the Home Page listener, looks for the phrase "useful downloads"; if that does not exist, the request will be considered a 'fail' by JMeter.
Also, check the View Results Tree, it will give the exact request/response from Moodle, and you can get a better idea of the failure reason there.
I've read some tutorials and successfully made a few small load tests, but the question is what should be done with the results?
- chose a course
- make the quiz
How to know that what you get as result is OKAY? or am I missing something!
Any tip would be helpful.
Thanks in advance.
1) the number of threads
2) the response time
As you increase the number of threads, you'll see the response time increase (and possibly the failure rate at high loads). The question is, what response time is acceptable?
An acceptable response range (IMHO) is <2-3 seconds; ideally you'll want to be <1s range; and <0.5s is perfect. So, by tweaking the number of threads and your hardware setup, you'll get an idea of how many concurrent users your Moodle setup can support.
It is important to know the ranges. This will help me make good decisions and get the best result of the server that we have.
I have already stress tested my moodle instance but I am not sure how to interpret the results - maybe one of you can help me with this:
Ramp-Up period: 0
||number of examples
Would be awesome if one you can help me.
Ok, so it looks like you've tested your server with 500 concurrent connections, over a 500s period, equating to 3600 'hits' on the web server (split 2/3 over index.php/logout.php).
I'd intiially suggest that you add a ramp-up period of 15-30s. 500 threads with no warmup is unlikely to happen in the real world, and a ramp-up period will give things like PHP caching time to prime itself. Generally your server will be a happier server.
Anyway, we can see from the min column that under ideal circumstances, your server will return pages in an acceptable timeframe (0.754s and 0.123s respectively). However, in the worst case scenario, your server is failing to serve ~8-15% of pages, and can take upto 20s to serve a page (max)*; this is most likely to be when all 500 threads are hitting the server at once.
*as a side note here, I suspect that you have a timeout somewhere set to 20s, since both your pages are failing after a maximum of 20s.
Finally, your average response time is 14-24s, which is fairly high.
In all, this suggests that your server cannot cope with 500 threads. You should try reducing the number of threads (at 50 threads a time; 500, 450, 400, etc), until the following happens:
- the failure rate drops to 0% (or at least <1%), which will indicate how much load your server can take before it fails completely; and then
- the maximum and average response times drop to a value that you find acceptable, I'd suggest <5s max and <2s average.
Once the above criteria are met, you'll have a number of threads that your server can handle. This roughly equates to the number of concurrent users, but YMMV depending on what your users do within Moodle.
Hope this helps,
- Log in
- View course list
- View the course
- Edit a course
- Edit a section of a course
- Add a text to the section
- Go to a Quiz
- Attempt the Quiz
- View the course
- Log out
I've got great results and tuned my server, now I'm getting an average of min 1.3 sec and max 3.4 sec.
The questions is how to share this script. I think that if I share it, people will have to change lots of things to get it running, because you have to change lots of things to be able to run the script, so has anyone another idea?
I've zero knowledge of Jmeter,
currently I host Moodle in Hosting, can I stress test it? I have to put Jmeter and install on their own server? And then what I do next? Really can't find any step-by-step hints, thanks
There are a lot of JMeter tutorials out there, it's an Apache Foundation product.
JMeter installs on your own personal computer, not the Moodle server. But if you are running on a hosted environment, you might want to speak to your host about this first; you are potentially running a Denial of Service attack on your own setup.
I'd love to see your script if you don't mind sharing. If there's things we need to change, it may help us learn more about the process anyway.
Thanks to your script, I've been able to get on the case with stress-testing our development Moodle server very quickly, so thanks for providing that. In addition it also taught me about using JMeter itself, which is valuable knowledge.
It's difficult to know exactly what numbers to try for the users/ramp-up/loop figures, but I'll try to figure out what is a 'real world' scenario for our systems and start from there.
I'm also keen to try to figure out what load I can put the server under using JMeter while still having the server visibly usable (get a colleague or two to use Moodle at the same time) to try to see what constitutes a heavy-but-still-usable load.
Finally I think I'll try to stress it to the point it starts failing to serve pages at all (as I've not got there yet) and see if I can crash it, both of which are rare occurrences in real life (to date, touch wood!).
I'm trying to post/reply to a forum thread via JMeter and ran into the following error:
"Incorrect sesskey submitted, form not accepted"
Does anyone know how to make this work? I used the regex post processor to get the sesskey after loading the homepage and it's passed to the form when submitting the reply to the forum thread.
my JMeter test does the following:
- load the homepage
- log in
- access the course
- click on forums
- view the thread
- go to reply
- post the reply (here's where the error came up)
- view the thread again
Hi,I'm struggling with sessionkey handling too. If i want to rerun the recorded test it always fails because afterlogout new session keys are needed.
Could you solve your problem?thank you in advance!
I can't help you with standard Moodle authentication, because we have a custom auth system here which really makes writing JMeter scripts tricky
Re: users/ramp-up/loop figures...
Ramp-up we defined as 30-60 seconds, this seemed to be easier on the web server rather than hitting it with n-hundred number of users instantly.
It should also give you PHP cache an opportunity to prime itself (if you use PHP caching, and indeed caching anywhere in your Moodle setup), since you're undoubtedly restarting services as you're adjusting the setup (and therefore emptying caches). There's also a chance that you have network hardware and/or server configurations which react badly to an instantaneous number of connections (ie. they consider it a Denial of Service attack). Apache has various modules which you can enable which try to avoid this, and they have their own ramp-up periods.
Finally, it also gives you a visual clue as to when your Moodle setup is starting to struggle; you can use the response-time graphs against the number of current threads, to determine this.
Loop we tried to keep JMeter running for approximately 5 minutes; although there were some tests where it was clear that our Moodle installation could cope, or was failing miserably, to the tests were ended prematurely.
Users will be your biggest variable. You're right, having a broad picture of "how high can we push this until we start to see failures?" was our methodology, and we had an 'acceptable range' in mind for general use. We roughly pushed users up in 50-user increments, covering 50 - 250 concurrent users, then refined this to 25-user chunks for the areas we were interested in (eg. 125, 150, 175).
Failures should be considered an absolute limit, as your colleagues have been using Moodle mid-JMeter, you'll see that acceptable response times come into effect much sooner. Bear in mind that your physical users will need to follow a script similar to your JMeter test script, as not to skew the results; but it's an excellent way of visibly showing the effect of high usage (especially to management!).
In the end, we concluded that our new setup (load-balanced Apache servers) would be acceptable for the usage figures we were worried about (150-200 concurrent); and in theory would scale upto about 300 concurrent.
Finally, just to add that it's important that your JMeter reflect what your users actually do in Moodle. There are documented means of using your Apache logs to build JMeter test plans, although you can also plan based on common sense (our script, posted above, uses the common sense approach). Obviously if you run a high number of quizzes though Moodle, or make extensive use of media, your JMeter script should reflect this.
I've finally gotten round to publishing my (half baked) jMeter script generator script :D
Go to my OU Site and view the Load Testing page. There you should be able to grab my script and start using it.
Sods law dictates it's not going to work for anyone, but ho hum! I'm about 80% sure it should just work providing you follow the instructions.
If there are any issues just leave a comment on the site.
Note: I forgot to say the scripts are written for Moodle 2, but shouldn't take much tweeking at all to get them working for 1.9. Just the page generation stuff needs changing (and maybe some of the error handling?)
Thanks for sharing this, its really useful - we hope to have a look at it next week
Great script, thanks a lot. Here in LUNS Ltd. we have fixed some bugs and enhanced its functionality slightly.
Now it has the following:
- Chat tests generation (both AJAX based and accessible one),
- Prettier XML output in case one want to look through it in text editor,
- JMeter loopcontroller is added to make script output more compact where possible,
- Random text generator is added.
Thanks for taking the time to look at the script. It was a basic start, and it's nice to see some people advancing it.
I have also done some work on it, funnily enough I also added in the loop controller amongst other things.
It's still not in a state to upload yet, that might take a while. I will have a look at your changes and integrate them when I get a chance.
Again, thanks for taking the time to have a look. It's nice when someone else gets use out of something I've built.
My pleasure, James.
It may also worth considering putting it on CVS, so that people could find it strightaway from http://docs.moodle.org/en/JMeter page.
Just for the record, I have published James's loadtesting generation script along with LUNS changes on github:
There were some requests to contribute, so I though having it on public repo would allow accepting pull requests easier.
For the ease of maintenance and following Simon Story's idea, I have detached the loadtesting directory from existing branch and created new fresh repo for this project having preserved all the full past commits history: https://github.com/kabalin/moodle-jmeter-script-generator It contains Simon's recent contribution as well which make it Moodle 2.2 compatible.
Regarding backward compatibility, there is pre_2.2 branch whish contains generator code for using with earlier versions.
The link http://www.open.ac.uk/blogs/XHProf/ is brocken.
Can you provide de the working link again?
-Agree to site policy
Does these in random order:
-Advanced Uploading of Files (disabled by default; make sure you set the course & assignment upload limits): Upload & download
-Chat: post a chat
-Forum: post & reply
this is my script based on Mark van Hoek one (thanks Mark), modified to do load testing of Moodle 2.5. In this script you can change some concepts like the host and path easily using User Defined Variables in the configuration of the test plan.
"User parameters" have been completed too with some variables containing the name of Moodle objets the script will test.
After you run the script all data produced by the listeners (graph, result table, result tree, etc) will be stored in a directory given in the test plan parameters.
It would be great to share scripts results, server configuration/sizing thoughts, etc...
I'm trying to write a JMeter test that logs in and looks at some pages, but even though I have a request POSTing the username and password to the login page, all subsequent requests are redirected back to /login/index.php
Can anybody give me a nudge in the right direction for getting this working? I can post the test plan I created if it helps.
Could you tell me what exactly I have to change in Mark Withheld's test plan to adapt it for my moodle installation? I've changed "Server name" everywhere but only "Goto Moodle" is correct, the rest is 100% error.
ps I'm also a newbie in jmeter and moodle.
I am new to testing , and i am trying to learn Jmeter tool . Can anybody tell me how to view the Jmeter scripts that u all people uploaded in this link http://cvs.moodle.org/contrib/tools/jmeter/ .I didn't able to understand the structure of the files from where i can view the Jmeter scripts. Please guide me ....................
Thanks in advance
Yeah, have you something like structure reviev?
I don't have something specific to that and I think fastest way for you to get it would be to record it yourself. I had to do this for some of our SCORM Package testing, and there are step-by-step instructions here:
From the Proxy Recording jMeter test plan I then merged the relevant parts over to my Moodle-script generated test plan to create my final test.