Hardware and performance

Hi Ani,

I've not answered all your questions as some require more answers first ;) and I'm not familiar with all areas (especially Windows).

You don't say, but is this a virtualised machine or barebones?

1) I'm not too familiar with the perspective script, but your testing methodology assumes that:
* you only have one course; and
* all users are performing the same actions at a similar time.
Typically you'll probably find that your users are doing a wide variety of tasks, and that you have more than one course. As you get to certain levels of courses, things change a little in Moodle - for example, after a point, only the groups of courses are displayed and not the actual course.
Some features of Moodle also require more power than others - historically the Chat system has always been pretty resource intensive.

Also, I'm not sure how well jmeter pretends it has javascript enabled and this can lead to slightly different results in moodle too.

2) It's difficult to give a detailed reason behind your CPU utilisation, but you don't mention whether you've tuned any of the components at all... You should ensure that at the very least, you've configured:
* APC;
* MySQL; and
* Apache thread counts.

The default APC configuration is something like 30MB, which won't get you anywhere with Moodle. We have something nearer 400MB if I recall for one of our customer installations.
The default MySQL configuration is also pretty slim when it comes to memory. You should look at the MySQL tuning documents and make sure that your MySQL configuration is appropriate. Better still (In my opinion), switch to Postgres. Postgres is safer, cares about your data more, and allows for a greater degree of fine-tuning.

In Apache, if you set your thread limits too high, you'll actually cause more of a bottleneck as your machine is unable to cope with the volume of requests thrown at it. If you queue requests, you'll find that they're all served more quickly overall.

Once you've tuned your daemons, then have another look - I suspect you'll be less CPU bound and will be using more of your memory. I'm not sure about how CPU graphing under Windows works, but under Linux you can use tools like top and vmstat to determine what's using your CPU. It could be that you have high disk IO for example.

3) I assume you mean an equivalent 64 bit machine. You'll probably see better results, but only when you're giving it more than 3.5GB memory. 64 bit machines do have an inherent memory overhead over 32bit so there are times when they're not 'better'.

4) Yes. Linux would show better results (at least in my opinion). I'm not a Windows man, so I may be slightly out-of-date here, and I'm definitely biased towards Linux, but:
* PHP itself performs better on *nix than on Windows (or at least has done historically) so you get that gain for starters. I now that Microsoft did some work to try and improve PHP performance under Windows, but I believe that it still lags behind.
* Linux also typically has lower operating requirements from my experience. I can happily have a light-weight debian VM running on 128MB memory and serving web requests - I doubt that this is possible under Windows but I can't say that I've tried.
* Under Linux you have far greater control over many other components - for example the way that you mount and format your disks for different workloads (again I've not tried doing any of these things under Windows so things may have changed).

5) Here at LUNS, we've done some massive rollouts on hardware not much greater than yours. It's all about tuning. On one of our largest installs, we had 250,000 registered users, with around about 20,000 unique users visiting per day (if I recall) on a pair of Dell PowerEdge 2850s with 4GB RAM, and a quad-core 3.2Ghz Xeon running Debian Linux.

It's difficult to determine a scaling factor as you suggest largely because each page load is different. Each page can have very different load times. Additionally, different types of users have different permissions, which again changes the amount of time it can take to load a page. Different courses will have different resources displayed. Moodle does a huge amount of caching (e.g. themes, javascript, etc) and you can tune your server configuration to give better browser caching too.

6) I'm not familliar with the record caching features I'm afraid so I can't really comment.

7) APC is better supported and is part of the mainstream php project. If you're seeing issues with it, I'd suggest filing a bug with PHP. It could be that the event viewer is actually reporting the number of times that the cache has cleared. When you fill the cache, it empties so you have to start filling it again.
As I mentioned above, make sure that you've tuned your APC cache. APC ships with an apc.php file which will give you some basic stats on what the opcode cache is up to and how much of it's memory it's using.

--

I hope that the above answers some of your questions. I realise it's a bit long and a bit vague but unfortunately tuning is one of those things which is very dependant on individual system configuration.

In summary though, you should really consider:
* Disk layout. Databases like lots of spindles with low latency. One 300GB disk will not be slow, but it won't be as fast a 4 or 6 spindles in a RAID 10 layout for example.
* Give your database lots of memory. Read the documentation on how to tune it and make sure you understand what each setting is. Don't just copy someone elses settings as they really won't be relevant.
* Don't give Apache too much - it's actually counter-productive
* Make sure that your PHP Opcode cache is properly tuned


All the best,

Andrew

Perspective.php is not an accurate representation of anything (I think I'm allowed to say that since I wrote the original version!) - it was originally designed to show the comparative time it takes for different operations like a db query vs a function call, to help programmers get better perspective.

'Concurrent users' doesn't necessarily mean what you think it means (ie there are a zillion definitions). For example, if you have 10 students in a course and they are all using the website during the same 5 minute period, that counts as 10 concurrent users by some measures. Unless I misread your post I think you're using 'concurrent requests' which is a much tougher metric; that 10 concurrent users might only be making 1, or less than 1, concurrent requests.

A better usage measure is requests per second (or something that approximates to it). This measure also emphasises the obvious fact that you probably don't know how many requests per second your users will make - it varies a lot depending on the site. For example, interactive features like forums and quizzes will cause users to make a lot of requests compared to non-interactive ones like downloading a PDF file and then going away. Once you have some users, you can measure peak requests-per-second for your content and start load testing from there.

--sam

Hi Ani

Thanks for posting your results.

The confusion about the "number of concurrent users supported" is to be expected. Defining a benchmark is one thing, interpreting it is something different. perspective.php has defined a benchmark, your JMeter script defines a benchmark. But nobody could so far define what a Moodle "user" does!


To make the things worse, each visitor of this forum has his own definition what the number of concurrent users means, and there is another one in the FAQ http://docs.moodle.org/21/en/Performance_FAQ#How_do_you_define_.22concurrent_users.22.3F.

The whole purpose of benchmarking is to let two systems to compete against each other - following the rules inherent to the benchmark.

Thanks everyone for the quick and detailed response. Considering that I am a newbie to this forum and to Moodle, I had not expected any response thinking that this subject must already have been chewed to death.

Andrew, thanks specifically for the suggestions. I will try them over next few days. Sam, I completely understand that the original purpose of perspective.php was different. Still, I am curious (as were others in the thread where it was posted) as to how the number of concurrent users was estimated from the benchmarking.

To answer some of the questions Andrew asked:

I am using a barebones machine, not virtualization.

JMeter just repeats the recorded HTTP request including the headers. So, I believe it exactly emulates a real browser. Sure, it does not process any CSS or JSS received but that does not affect anything.

My APC cache size was 64M. For a real production deployment of Moodle, this might be small, but for the tests which I was doing, this should be sufficient. I was only exercising a very small number of pages. APC cache size requirement does not go up with number of simultaneous users. It only depends on how much of PHP page opcodes you want to cache. The more pages or the bigger pages you want to cache, the requirement goes up. At least that is my understanding. Still, I can try increasing the APC cache size.

Most of the CPU consumption was from Apache httpd. Can't really differentiate between Apache HTTP processing and PHP script processing. MySQL was the second most active process after httpd.

My idea of "concurrent users" was those users who are actually doing something with the Moodle simultaneously. Basically submitting requests and getting responses. They all need not be doing the same thing, but doing something that generates some load on the server. So, this is a subset of users who have some Moodle page opened (logged-in or not) but not actively doing anything at that instant, thereby not generating any request-handling load on the server at that instant. For lack of any other term, lets call them "active users". This would further be a subset of users who are registered with the system. Lets call that number "registered users".

In order to figure out what kind of hardware would be required to support a given educational institute, on needs to estimate the above three for a given hardware configuration. Of the three, the "registered users" only affects the database size (though indirectly that could also affect the performance - DB query on a table with 10,000 entries would be slower than query on a table with only 100 entries). So, this is less of a problem. To estimate the max active users supported, I started off with estimating maximum number of "concurrent users". Based on this, one can estimate max number of "active users" supported by the system, assuming that these users would, on an average, spend some time on the page displayed and then click on some link or button to generate some load on the server. I realize that many factors can affect this estimation, particularly the kind of actions the users are doing (chat, doing quizzes, writing on fora etc.) but I just want a ball-park figure. So, I assumed that if on an average, users stay on a page for x seconds before clicking on a link and it takes y seconds for the server to process that request, then:
    max_active_users = max_concurrent_users * (x+y) / y
the logic being that out of (x+y) seconds, the processor is busy only for y seconds.

Of this, the average y can easily be found from JMeter testing. In fact, we have a choice here ... not to increase the number of threads so much that y exceeds some reasonable figure like 10 or 15 seconds. Estimate for x can only be a guess. Depending on what the user is doing, x could be 1 to 2 minutes. So, I figure that a reasonable value for(x+y) / x could be 5 to 8.

Before I started off with load testing, I had tried to optimize Apache, PHP, APC and MySQL a little bit (admitting at the outset that I am no expert at these). Here are some crucial settings I had used (I have logging down to minumum):

Apache (httpd.conf):
Include                        conf/extra/httpd-mpm.conf
ThreadsPerChild                150
MaxRequestsPerChild            0

APC (part of httpd.conf):
apv.cache_by_default=        on
apc.enabled=                on
apc.max_file_size=            10M
apc.num_files_hint=            1000
apc.shm_size=                M/64


PHP (PHP.ini):
max_execution_time=            120
max_input_time=                -1
memory_limit=                512M
mysql.cache_size=            2000


MySQL (my.ini): (All the values are standard)
default-storage-engine=        INNODB
max_connections=            100
query_cache_size=            74M
table_cache=                256
tmp_table_size=                77M
thread_cache_size=            8
innodb_additional_mem_pool_size=5M
innodb_flus_log_at_trx_commit=1
innodb_log_buffer_size=        2325K
innodb_buffer_pool_size=    225M
innodb_log_file_size=        45M
innodb_thread_concurrency=    8

Some questions:

1. What is MySQLnd and how does one use it?
2. What is the "ts" or "nts" suffix used in describing the binary distributions of PHP / APC / XCache? Is it "thread safe" and "not thread safe"?
3. I do have probably more modules loaded in Apache than I really need. Would that affect server performance in any way other than just consuming memeory?
4. In general, how much does virtualization affect performance? Would there be much of a difference between a barebones machine and an "equivalent specification" virtual server? I suppose the difference would in execution time of OS calls. In all other respects, they should be identical. That means as long as a process does not make any system call, it should run at the same speed on barebones machine as on a virtual machine. Any idea by what factor the OS calls are slow? And in a typical PHP program overall by what percentage the performance would be affected?

Thanks for all the help

- Ani