Windows server: Performance on Windows

Windows server: Performance on Windows

by Ben Steeples -
Number of replies: 8
Hi all,

Has anyone managed to get their moodle setup to work reliably with 100+ concurrent users?

We're currently experiencing a performance drop-off when we have more than 50 users digging around in our moodle installation (1.9.2+). Trying to replicate it with synthetic benchmarks is proving only half-fruitful, due to the random nature of our users. However we have narrowed the bottleneck down to PHP's communication with our database.

To give you an idea of our setup(s):

Production:
Moodle 1.9.2+
Apache 2.0.63 (standard windows build)
PHP 5.1.6
- FreeTDS 0.64 (Kauffman build)
- APC
MSSQL 2005

Development:
Moodle 1.9.4+ / Moodle 1.9.2+ (running builds in parallel)
IIS7
PHP 5.2.8, running as ISAPI
- FreeTDS 0.82 (moodle.org build)
[No APC as of yet, win4pecl is currently down]
MSSQL 2005 Express

Both boxes have similar behaviour, they're fine up until 50 or so concurrent requests, and above that PHP starts stacking up requests (ultimately leading to timeouts). I've narrowed this down to FreeTDS, but there's currently very little support for Windows builds.

So, what setups are Windows users out there using? Have you found your setup to scale well beyond 50+ concurrent users, and how?

Many thanks,
Ben
Average of ratings: -
In reply to Ben Steeples

Re: Windows server: Performance on Windows

by Masimba Musa -
Hi Ben
We are having exactly the same problem. I was directed to the following link as a possible solution: http://moodle.org/mod/forum/discuss.php?d=93358. The other place to look is http://www.apachelounge.com/. We're still working through some of these options. The suggestion we've had more and more is to switch our moodle installation to a Linux box. Hope this helps.

Mas
In reply to Masimba Musa

Re: Windows server: Performance on Windows

by Ben Steeples -
Hi Mas,

Ah yes. We toyed with the ApacheLounge build last summer, but were hampered by the limited availability of FreeTDS on PHP 5.2.x; at the time of installation PHP 5.1.x on Apache's build was considered the most 'stable' setup. Now that moodle.org has it's own version of FreeTDS, it might be time to revisit things. (Although PECL4Win, which hosts many PHP caches, is currently down for maintenance.)

When I get more time in the coming weeks I'll look at benchmarking the various flavours of Apache, PHP, etc.

If it helps, our next step, once we've ruled out the Windows platform, is to move the front-end to a Linux box. The only 'must' in our spec is that the DB runs on MSSQL.

Best,
Ben
In reply to Ben Steeples

Re: Windows server: Performance on Windows

by Lewis Carr -
Hi Ben,
We run Moodle on a Windows Server and get over 2000 users logging in each day.
We can have over 200 on at any one time.
It does depend on what the users are doing, heavy forum use can slow things down a bit.
The spec of your server is very important and Memcache was a must for us.
It also pays to tune your server performance accordingly.
Currently we are running a quad core system (2.5 Ghz) with 4GB RAM. If we get any bigger we will consider load balancing.
What is the sec of your server?



Average of ratings: Useful (1)
In reply to Lewis Carr

Re: Windows server: Performance on Windows

by Ben Steeples -
Hi Lewis,

Spec is very similar to yours. 2GHz, Quad Core, 4GB. Although our main box is shared with QuestionMark Perception, so resources are limited.

Dev box is 2GHz, Eight Core, 12GB; but this is running SQL Server as well.

Thanks,
Ben
In reply to Ben Steeples

Re: Windows server: Performance on Windows

by Lee Mears -

Hello Ben,

I'm a Moodle newbie, but would be interested to know whether you were able to overcome this limitation, and also how you initially managed to isolate the bottleneck to be related to FreeTDS.

I've just started an engagement with an organisation that has deployed Moodle to a remarkably similar landscape to yours, and is seeing almost exactly the same behaviour (though the ceiling appears in this case to be 75 rather than 50).

Keen to become an active member/contributor to the community as I get my head around this stuff.

Lee.

In reply to Lee Mears

Re: Windows server: Performance on Windows

by Ben Steeples -
Hi Lee,

We've not overcome it yet, but we are seriously looking at alternatives.

To summarise, our Moodle installation would be fine with high use, having 25-50 concurrent users online for 7+ hours is not an issue for our current installation. However, if the concurrent user count goes over 50 users, and towards 75 users, we start to see serious degradation in performance; pages time out, CPU usage spikes to 100% and remains there (on all cores), and eventually Apache gives up and restarts.

To recap, our server is setup as:
Moodle 1.9.4+
PHP 5.1.6 running as SAPI module under Apache 2.0.63 on Windows 2003
PHP also runs the APC cache.
MS SQL Server 2005, via FreeTDS driver (Kroman build)

Database and Web Server reside on different boxes, both Dual-Core 2.0GHz+ 4GB RAM+ boxes.



Testing methodology was twofold, firstly using a JMeter script to emulate 'real world' Moodle usage, and secondly a simply page to repeat and time a fairly complex query directly against the Moodle database (a simple 20-line script).

What we see is:
  1. Using php_mssql is fine, no CPU spikes, and all queries return in a timely manner. You can load-test to your heart's content and the server rarely crashes or fails to return a page. Only problem is, that we can't use this driver due to the UTF-8/UCS-2 issue relating to it.

  2. Using php_dblib (FreeTDS), things start to differ. The first few dozen queries are fine, but the web server soon gets overwhelmed if you place sufficient load on it.
    We've narrowed this down to FreeTDS' single-query nature (http://www.freetds.org/faq.html#pending), and PHP's habit of reusing connections (http://lists.ibiblio.org/pipermail/freetds/2003q3/013915.html). Both lead to a single query being processed at any one time, and so high-loads cause a backup to appear. Running SQL Profiler, you can actually see that the page loads (and queries) are run in turn, not concurrently.
    This is, of course, our interpretation of what is going on - others are welcome to comment.

  3. We've also confirmed this by using a MySQL backend, and running the same tests. You get performance on-par with using the native php_mssql driver, both of which are ten-times the performance of the FreeTDS driver.
To us this seems to localise it to the FreeTDS driver and nothing else. Both the Kroman build and the Moodle build are affected by this, as they both rely on a single query-per-connection.


We're currently investigating other ways of handling our database connections:
  • We're going to look at ODBTP in the coming weels, but we've no hope that it'll improve things (we ruled it out when we intially installed Moodle two years ago).
  • We're going to see what Moodle 2.0 offers. The Moodle team appear to be working on Microsoft's new php_sqlsrv native driver, which might improve performance.
  • We're investigating load-balancing our setup.
  • Finally, we're looking at moving our database back-end to MySQL.
So yeah, that's where we're at. It'd be interesting to see if anyone else can confirm this FreeTDS behaviour (if only to prove that I'm not going mad).

Best,
Ben

In reply to Ben Steeples

Re: Windows server: Performance on Windows

by rolio rolio -

Hi Lewis,

Do you mean 200 concurrent users ?

Are you running moodle and SQL 2005 in the same box ?