I have been trying to do some testing on my Moodle installation to estimate max number of concurrent users. Before even I found this forum, I started off using JMeter to do the testing, came to some conclusions and then came across the script "perspective.php" that has posted in this forum. When I ran that script, I got a result that was quite differnt from what my JMeter based testing showed.
A brief outline of my testing methodology using JMeter. I took a very simple test case where a student logs in, generally browses through the weekly outline of course listed for him (I had only 1 course configured), clicks on a few other links in the navigation bar and then logs out. I recorded this using the JMeter. Before recording the test case, I made sure that I did exactly the same thing once from the same browser so that browser cached whatever images and script files that are cacheable. I wanted to simulate real-life case of student accessing Moodle.
Before running the recorded test case, I restarted the Apache and MySQL servers. Then I ran the recorded case once and ignored the readings. Then I ran the recorded case with some loop count and some number of threads and noted the results. I repeated this for increasing number of threads, starting from 1, 5, 10, 20, 30 going to 40 threads. To make sure that not all the threads are trying to do exactly the same thing at the same time (an un-realistic scenario), I used the ramp-up time feature of JMeter to evenly distribute all the threads over a period of time that it took 1 single thread to complete 1 loop.
To decide whether the performance is acceptable for a given number of threads (concurrent users), I chose two criteria:
- 90% of times, the response time for any request should not exceed 15 seconds (actually 15 seconds is being too liberal)
- No request should result in an error
With these criteria and with my hardware (which I will give below), I found that at about 20 threads, the response time for some request really goes up and also some requests result in errors. So, I concluded that my configuration supports max 20 concurrent users. This was with no opcode cache. With APC enabled, this figure goes up to about 30. However, when I ran the "perspective.php" script on the same installation, it gave the figure as 69. APC, of course, does not make any difference to this figure. I know that that is only an approximate figure as mentioned in the script, still it is more than a factor of 2 difference from my test results!
Moodle Version: 2.0.3
Apache Version: 2.2.19 VC9 build with SSL support
MySQL Version: 5.1.33 (installed on the same machine)
PHP Version: 5.3.6
Hardware: 2.8 GHz Intel Pentium CPU with 2 cores, 1.5GB RAM, 300GB HD
OS: Windows 2008 R2 64 bit
I have several questions in this regard:
- Is there anything wrong in my testing methodology? Any reason why "perspective.php" results should be a better indicator than my test results?
- I noticed that very early in testing (i.e. with very few threads), the CPU utilization goes to 100%. With APC enabled, this happens after a few more threads. The memory utilization does not go up drastically with number of threads, I had 1.5 GB of memory and even after liberally allocating memory for PHP, APC, the total utilization of memory (including the Win2008R2 OS, Apache, MySQL and the works) never went about 1.3GB. This somehow seems to contradict what I have been reading in Moodle fora where most of the posts harp on memory being the key factor in deciding supported maximum concurrent users. Any explanation for this disconnect? Or is it that in MY configuration, CPU is the bottleneck and if that were somehow removed (say by using a better CPU), THEN the memory would be the bottleneck? Somehow this sounds lame.
- As mentioned above, I was tesing on Win2008 R2 (64 bit) OS with Apache, PHP, APC, MySQL all 32 bit. Would an equivalent 32 bit machine show better results?
- Would *nix OS on the exact same hardware show better results? Why? Moodle Fora seem to indicate that *nix would give better performance but I don't understand how. As I mentioned above, memory does not seem to be the constraing factor, so *nix OS consuming less memory is not going to affect anything. Unless, the system calls on Windows are much more CPU intensive than equivalent system calls on *nix OS.
- If my configuration only supports 30 concurrent users, does it mean I will need approx 33 such machines to support 1000 concurrent users? That surely does not sound right. There are many posts where people have talked about much larger number of concurrent users. So, what gives? Where am I making a mistake in calculation? I understand that 30 concurrent users refers to 30 simultaneous requests to the server. In reality, not all logged-in users would be making request to the server simultanesouly. So, if on an average, a user stays on a page for n-times more number of seconds than the time it takes for a page to load, then it is reasonable to assume that number of logged-in users supported us n * 30. Is a factor of 5 to 8 a reasonable assumption?
- Moodle 1.x used to have some configuration for record caching. That seems to have been removed in 2.0! Is anyone aware why it was removed and if it is going to come back anytime soon? Or is it that it is not required now for some reason, or it never helped?!
- When I initially enabled APC, things were fine -- i.e., I never encounterd a crash on account of APC, even with 40 threads in parallel. However, now without any change whatsoever, I notice quite a few "crashes" in php_apc.dll logged in the Event Viewer. Has anyone encountered this? Is XCache a better option?
Any help with the above would be greatly appreciated!