I have been trying to do some testing on my Moodle installation to estimate max number of concurrent users. Before even I found this forum, I started off using JMeter to do the testing, came to some conclusions and then came across the script "perspective.php" that has posted in this forum. When I ran that script, I got a result that was quite differnt from what my JMeter based testing showed.
A brief outline of my testing methodology using JMeter. I took a very simple test case where a student logs in, generally browses through the weekly outline of course listed for him (I had only 1 course configured), clicks on a few other links in the navigation bar and then logs out. I recorded this using the JMeter. Before recording the test case, I made sure that I did exactly the same thing once from the same browser so that browser cached whatever images and script files that are cacheable. I wanted to simulate real-life case of student accessing Moodle.
Before running the recorded test case, I restarted the Apache and MySQL servers. Then I ran the recorded case once and ignored the readings. Then I ran the recorded case with some loop count and some number of threads and noted the results. I repeated this for increasing number of threads, starting from 1, 5, 10, 20, 30 going to 40 threads. To make sure that not all the threads are trying to do exactly the same thing at the same time (an un-realistic scenario), I used the ramp-up time feature of JMeter to evenly distribute all the threads over a period of time that it took 1 single thread to complete 1 loop.
To decide whether the performance is acceptable for a given number of threads (concurrent users), I chose two criteria:
- 90% of times, the response time for any request should not exceed 15 seconds (actually 15 seconds is being too liberal)
- No request should result in an error
With these criteria and with my hardware (which I will give below), I found that at about 20 threads, the response time for some request really goes up and also some requests result in errors. So, I concluded that my configuration supports max 20 concurrent users. This was with no opcode cache. With APC enabled, this figure goes up to about 30. However, when I ran the "perspective.php" script on the same installation, it gave the figure as 69. APC, of course, does not make any difference to this figure. I know that that is only an approximate figure as mentioned in the script, still it is more than a factor of 2 difference from my test results!
Moodle Version: 2.0.3
Apache Version: 2.2.19 VC9 build with SSL support
MySQL Version: 5.1.33 (installed on the same machine)
PHP Version: 5.3.6
Hardware: 2.8 GHz Intel Pentium CPU with 2 cores, 1.5GB RAM, 300GB HD
OS: Windows 2008 R2 64 bit
I have several questions in this regard:
- Is there anything wrong in my testing methodology? Any reason why "perspective.php" results should be a better indicator than my test results?
- I noticed that very early in testing (i.e. with very few threads), the CPU utilization goes to 100%. With APC enabled, this happens after a few more threads. The memory utilization does not go up drastically with number of threads, I had 1.5 GB of memory and even after liberally allocating memory for PHP, APC, the total utilization of memory (including the Win2008R2 OS, Apache, MySQL and the works) never went about 1.3GB. This somehow seems to contradict what I have been reading in Moodle fora where most of the posts harp on memory being the key factor in deciding supported maximum concurrent users. Any explanation for this disconnect? Or is it that in MY configuration, CPU is the bottleneck and if that were somehow removed (say by using a better CPU), THEN the memory would be the bottleneck? Somehow this sounds lame.
- As mentioned above, I was tesing on Win2008 R2 (64 bit) OS with Apache, PHP, APC, MySQL all 32 bit. Would an equivalent 32 bit machine show better results?
- Would *nix OS on the exact same hardware show better results? Why? Moodle Fora seem to indicate that *nix would give better performance but I don't understand how. As I mentioned above, memory does not seem to be the constraing factor, so *nix OS consuming less memory is not going to affect anything. Unless, the system calls on Windows are much more CPU intensive than equivalent system calls on *nix OS.
- If my configuration only supports 30 concurrent users, does it mean I will need approx 33 such machines to support 1000 concurrent users? That surely does not sound right. There are many posts where people have talked about much larger number of concurrent users. So, what gives? Where am I making a mistake in calculation? I understand that 30 concurrent users refers to 30 simultaneous requests to the server. In reality, not all logged-in users would be making request to the server simultanesouly. So, if on an average, a user stays on a page for n-times more number of seconds than the time it takes for a page to load, then it is reasonable to assume that number of logged-in users supported us n * 30. Is a factor of 5 to 8 a reasonable assumption?
- Moodle 1.x used to have some configuration for record caching. That seems to have been removed in 2.0! Is anyone aware why it was removed and if it is going to come back anytime soon? Or is it that it is not required now for some reason, or it never helped?!
- When I initially enabled APC, things were fine -- i.e., I never encounterd a crash on account of APC, even with 40 threads in parallel. However, now without any change whatsoever, I notice quite a few "crashes" in php_apc.dll logged in the Event Viewer. Has anyone encountered this? Is XCache a better option?
Any help with the above would be greatly appreciated!
I've not answered all your questions as some require more answers first ;) and I'm not familiar with all areas (especially Windows).
You don't say, but is this a virtualised machine or barebones?
1) I'm not too familiar with the perspective script, but your testing methodology assumes that:
* you only have one course; and
* all users are performing the same actions at a similar time.
Typically you'll probably find that your users are doing a wide variety of tasks, and that you have more than one course. As you get to certain levels of courses, things change a little in Moodle - for example, after a point, only the groups of courses are displayed and not the actual course.
Some features of Moodle also require more power than others - historically the Chat system has always been pretty resource intensive.
2) It's difficult to give a detailed reason behind your CPU utilisation, but you don't mention whether you've tuned any of the components at all... You should ensure that at the very least, you've configured:
* MySQL; and
* Apache thread counts.
The default APC configuration is something like 30MB, which won't get you anywhere with Moodle. We have something nearer 400MB if I recall for one of our customer installations.
The default MySQL configuration is also pretty slim when it comes to memory. You should look at the MySQL tuning documents and make sure that your MySQL configuration is appropriate. Better still (In my opinion), switch to Postgres. Postgres is safer, cares about your data more, and allows for a greater degree of fine-tuning.
In Apache, if you set your thread limits too high, you'll actually cause more of a bottleneck as your machine is unable to cope with the volume of requests thrown at it. If you queue requests, you'll find that they're all served more quickly overall.
Once you've tuned your daemons, then have another look - I suspect you'll be less CPU bound and will be using more of your memory. I'm not sure about how CPU graphing under Windows works, but under Linux you can use tools like top and vmstat to determine what's using your CPU. It could be that you have high disk IO for example.
3) I assume you mean an equivalent 64 bit machine. You'll probably see better results, but only when you're giving it more than 3.5GB memory. 64 bit machines do have an inherent memory overhead over 32bit so there are times when they're not 'better'.
4) Yes. Linux would show better results (at least in my opinion). I'm not a Windows man, so I may be slightly out-of-date here, and I'm definitely biased towards Linux, but:
* PHP itself performs better on *nix than on Windows (or at least has done historically) so you get that gain for starters. I now that Microsoft did some work to try and improve PHP performance under Windows, but I believe that it still lags behind.
* Linux also typically has lower operating requirements from my experience. I can happily have a light-weight debian VM running on 128MB memory and serving web requests - I doubt that this is possible under Windows but I can't say that I've tried.
* Under Linux you have far greater control over many other components - for example the way that you mount and format your disks for different workloads (again I've not tried doing any of these things under Windows so things may have changed).
5) Here at LUNS, we've done some massive rollouts on hardware not much greater than yours. It's all about tuning. On one of our largest installs, we had 250,000 registered users, with around about 20,000 unique users visiting per day (if I recall) on a pair of Dell PowerEdge 2850s with 4GB RAM, and a quad-core 3.2Ghz Xeon running Debian Linux.
6) I'm not familliar with the record caching features I'm afraid so I can't really comment.
7) APC is better supported and is part of the mainstream php project. If you're seeing issues with it, I'd suggest filing a bug with PHP. It could be that the event viewer is actually reporting the number of times that the cache has cleared. When you fill the cache, it empties so you have to start filling it again.
As I mentioned above, make sure that you've tuned your APC cache. APC ships with an apc.php file which will give you some basic stats on what the opcode cache is up to and how much of it's memory it's using.
I hope that the above answers some of your questions. I realise it's a bit long and a bit vague but unfortunately tuning is one of those things which is very dependant on individual system configuration.
In summary though, you should really consider:
* Disk layout. Databases like lots of spindles with low latency. One 300GB disk will not be slow, but it won't be as fast a 4 or 6 spindles in a RAID 10 layout for example.
* Give your database lots of memory. Read the documentation on how to tune it and make sure you understand what each setting is. Don't just copy someone elses settings as they really won't be relevant.
* Don't give Apache too much - it's actually counter-productive
* Make sure that your PHP Opcode cache is properly tuned
All the best,
Perspective.php is not an accurate representation of anything (I think I'm allowed to say that since I wrote the original version!) - it was originally designed to show the comparative time it takes for different operations like a db query vs a function call, to help programmers get better perspective.
'Concurrent users' doesn't necessarily mean what you think it means (ie there are a zillion definitions). For example, if you have 10 students in a course and they are all using the website during the same 5 minute period, that counts as 10 concurrent users by some measures. Unless I misread your post I think you're using 'concurrent requests' which is a much tougher metric; that 10 concurrent users might only be making 1, or less than 1, concurrent requests.
A better usage measure is requests per second (or something that approximates to it). This measure also emphasises the obvious fact that you probably don't know how many requests per second your users will make - it varies a lot depending on the site. For example, interactive features like forums and quizzes will cause users to make a lot of requests compared to non-interactive ones like downloading a PDF file and then going away. Once you have some users, you can measure peak requests-per-second for your content and start load testing from there.
Thanks for posting your results.
The confusion about the "number of concurrent users supported" is to be expected. Defining a benchmark is one thing, interpreting it is something different. perspective.php has defined a benchmark, your JMeter script defines a benchmark. But nobody could so far define what a Moodle "user" does!
To make the things worse, each visitor of this forum has his own definition what the number of concurrent users means, and there is another one in the FAQ http://docs.moodle.org/21/en/Performance_FAQ#How_do_you_define_.22concurrent_users.22.3F.
The whole purpose of benchmarking is to let two systems to compete against each other - following the rules inherent to the benchmark.
Thanks everyone for the quick and detailed response. Considering that I am a newbie to this forum and to Moodle, I had not expected any response thinking that this subject must already have been chewed to death.
Andrew, thanks specifically for the suggestions. I will try them over next few days. Sam, I completely understand that the original purpose of perspective.php was different. Still, I am curious (as were others in the thread where it was posted) as to how the number of concurrent users was estimated from the benchmarking.
To answer some of the questions Andrew asked:
I am using a barebones machine, not virtualization.
JMeter just repeats the recorded HTTP request including the headers. So, I believe it exactly emulates a real browser. Sure, it does not process any CSS or JSS received but that does not affect anything.
My APC cache size was 64M. For a real production deployment of Moodle, this might be small, but for the tests which I was doing, this should be sufficient. I was only exercising a very small number of pages. APC cache size requirement does not go up with number of simultaneous users. It only depends on how much of PHP page opcodes you want to cache. The more pages or the bigger pages you want to cache, the requirement goes up. At least that is my understanding. Still, I can try increasing the APC cache size.
Most of the CPU consumption was from Apache httpd. Can't really differentiate between Apache HTTP processing and PHP script processing. MySQL was the second most active process after httpd.
My idea of "concurrent users" was those users who are actually doing something with the Moodle simultaneously. Basically submitting requests and getting responses. They all need not be doing the same thing, but doing something that generates some load on the server. So, this is a subset of users who have some Moodle page opened (logged-in or not) but not actively doing anything at that instant, thereby not generating any request-handling load on the server at that instant. For lack of any other term, lets call them "active users". This would further be a subset of users who are registered with the system. Lets call that number "registered users".
In order to figure out what kind of hardware would be required to support a given educational institute, on needs to estimate the above three for a given hardware configuration. Of the three, the "registered users" only affects the database size (though indirectly that could also affect the performance - DB query on a table with 10,000 entries would be slower than query on a table with only 100 entries). So, this is less of a problem. To estimate the max active users supported, I started off with estimating maximum number of "concurrent users". Based on this, one can estimate max number of "active users" supported by the system, assuming that these users would, on an average, spend some time on the page displayed and then click on some link or button to generate some load on the server. I realize that many factors can affect this estimation, particularly the kind of actions the users are doing (chat, doing quizzes, writing on fora etc.) but I just want a ball-park figure. So, I assumed that if on an average, users stay on a page for x seconds before clicking on a link and it takes y seconds for the server to process that request, then:
max_active_users = max_concurrent_users * (x+y) / y
the logic being that out of (x+y) seconds, the processor is busy only for y seconds.
Of this, the average y can easily be found from JMeter testing. In fact, we have a choice here ... not to increase the number of threads so much that y exceeds some reasonable figure like 10 or 15 seconds. Estimate for x can only be a guess. Depending on what the user is doing, x could be 1 to 2 minutes. So, I figure that a reasonable value for(x+y) / x could be 5 to 8.
Before I started off with load testing, I had tried to optimize Apache, PHP, APC and MySQL a little bit (admitting at the outset that I am no expert at these). Here are some crucial settings I had used (I have logging down to minumum):
APC (part of httpd.conf):
MySQL (my.ini): (All the values are standard)
- What is MySQLnd and how does one use it?
- What is the "ts" or "nts" suffix used in describing the binary distributions of PHP / APC / XCache? Is it "thread safe" and "not thread safe"?
- I do have probably more modules loaded in Apache than I really need. Would that affect server performance in any way other than just consuming memeory?
- In general, how much does virtualization affect performance? Would there be much of a difference between a barebones machine and an "equivalent specification" virtual server? I suppose the difference would in execution time of OS calls. In all other respects, they should be identical. That means as long as a process does not make any system call, it should run at the same speed on barebones machine as on a virtual machine. Any idea by what factor the OS calls are slow? And in a typical PHP program overall by what percentage the performance would be affected?
Thanks for all the help
About the definitions of "concurrent users",
> My idea of "concurrent users" was those users who are actually doing something with the Moodle simultaneously. Basically submitting requests and getting responses. They all need not be doing the same thing, but doing something that generates some load on the server.
That tallies with the FAQ: "those users for whom the server is actively doing something . It may by processing a webpage written in PHP, querying the database or simply transferring a file"
> of users who have some Moodle page opened (logged-in or not) but not actively doing anything at that instant, thereby not generating any request-handling load on the server at that instant. For lack of any other term, lets call them "active users". This would further be a subset of users who are registered with the system. Lets call that number "registered users".
Sounds reasonable. You are welcome to add those (or any other) information to the FAQ. Don't forget the different "clones" of it! (Use the same login as on moodle.org for docs.moodle.org):