Moodle 2.4 much slower than 2.3 on Windows Server

Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -
Number of replies: 22

Server info: Windows 2008R2 X64, 12GB RAM, duel Xeon E5450 3GHz. PHP 5.3.18 nts vc9. Wincache 1.2.614. SQL Server 2008 10.50.4000 cluster with 4 nodes, active/passive.  Servers are VMs on VMWare 5.

We had been running 2.3 since December on a new server we setup just for Moodle 2, and while the performance wasn't as good as our 1.9 servers, it was tolerable.  Admin user pages were loading in 3-5 seconds, and non-Admin user pages were loading in <2 seconds.  Once we upgraded to 2.4.3+, the response time was terrible, >30 seconds for admin and about 15 seconds a page for non-admins, completely the opposite affect I thought upgraded to 2.4 would have with the new caching in this version.

Through a number of changes I've been able to cut this in about half for admins and less than 1/3 for non-admins, as admin user pages are now taking 13-16 seconds to load, and non-admins 1.5-3.5 seconds depending on the page.  The major changes were moving the Moodle Data from a UNC share to a local drive, and changing the SQL server driver from freeTDS to the native MS driver.  But the performance is still dissapointing, especially for admin users, and still slower than 2.3.  Any ideas on why the performance got worse would be appreciated. 

I do have xDebug installed and can profile some pages if that would help, when I've looked at them no single thing stands out to me.  Just lots of functions that are taking >1 second that I'm guessing should not.

Average of ratings: Useful (1)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Can I cautiously check it's not something daft - you've not got the site in theme development mode or anything like that?

It's worth a quick check with YSlow just to check for an HTTP issue of some kind. Failing that you really need to monitor the server to establish what is soaking up all this time.

Average of ratings: -
In reply to Howard Miller

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

Theme develoment mode is not on, just to test it I turned it on and reponse latency about doubled.

Yslow gives an overall grade of B, items it grades low look common for others that test their Moodle 2 sites like use a CDN, put Javascript at bottom, reduce number of DOM elements, etc.  Using the developer tools in Chrome to view the network latency, almost all of the time is waiting for the server to respond, which is shown in the footer performance reports.  

Here is an example of the footer performance report in an an empty class with no content as an admin:

13.732252 secs
RAM: 13.9MB RAM peak: 14.1MB
Included 673 files

Contexts for which filters were loaded: 1
Filters created: 0
Pieces of content filtered: 1
Strings filtered: 0
get_string calls: 1050
strings mem cache hits: 0
strings disk cache hits: 1358
DB reads/writes: 83/4
Session: 6.1KB
Caches interaction by definition then store
core/string
** static persist **: 1018 / 340 / 0
cachestore_file: 340 / 0 / 0
Total Hits / Misses / Sets : 1358 / 340 / 0
 
And here is the same class as a non-admin, much faster, but still not as fast as I would like it:
3.673841 secs
RAM: 6.8MB
RAM peak: 6.8MB
Included 153 files
Contexts for which filters were loaded: 0
Filters created: 0
Pieces of content filtered: 0
Strings filtered: 0
get_string calls: 251
strings mem cache hits: 0
strings disk cache hits: 253
DB reads/writes: 48/1
Session: 10.4KB
Caches interaction by definition then store
core/string**
static persist **: 214 / 39 / 0
cachestore_file: 39 / 0 / 0
Total Hits / Misses / Sets : 253 / 39 / 0
Average of ratings: Useful (2)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by J S -

Hi

I agree with Howard.  Typically the best way to determine where slowness has been introduced is by comparing past performance with current performance metrics.  It sounds like you are familiar with your server metrics through lots of investigation otherwise I dont think would have made the UNC change on a whim.  

I would recommend:

1) try to measure the latency of your site and record the time to first byte versus how long it takes to load the page fully; if  its takes a really long time I guess that you need to take a look at your DB

2)  enable some type of slow query log and see if a query is consistently taking a long time (one bad query can way offset any MUC enhancements)

3) look to see if you are running any new blocks.  there may be something by default that was added in 2.4 that was not beneficial to your performance.  disable them possible and check performance.

4) I dont know what web server you are running, but you can do some sort of tracing of web transactions if needed.

5) Get yourself a free account with NewRelic and find out what is causing problems.

6) update moodle to the latest if you are missing any patches that could have patched up.  Its best to understand where the issue is though with the steps prior otherwise you wont have a good understanding of why it happened.

-Good luck

 

Average of ratings: Useful (1)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Benjamin,

When you're hitting the poor performance, what is the Windows performance tool showing for Disk IO? Are you IO bound, or something else (e.g. CPU, memory)?

One of the biggest new features in Moodle 2.4 was the inclusion of a feature called Moodle Universal Caching (MUC). The intention with MUC is to create a single caching system which all code can use and administrators can configure in any way that they like.

As I recall, the default MUC cache is disk storage based which could explain why your UNC change has made a difference. The disk storage is located in the moodle temp directory, which is located in dataroot/temp by default. If you're seeing high disk IO and latency, the presence of MUC could also explain why your performance is still relatively poor - depending on how your disk setup is handled for VMWare, there could be fairly high latency on disk IO.

There is some documentation on MUC at http://docs.moodle.org/24/en/Caching and you may want to consider configuring a different type of default cache - perhaps something like memcached.

In theory though, the cache changes shouldn't have any negative effect on performance but as I say, your VMWare disk configuration could have this effect.

As usual though, with any performance issue, you need to find where your bottleneck is.

Andrew

p.s. IIRC, use of the Native MSSQL driver in Moodle has been deprecated and freeTDS is always used because the native driver is not capable of handling multiple queries in certain situations.

 

Average of ratings: Useful (2)
In reply to Andrew Lyons

Moodle 2.4: Effect of MUC on other PHP caching methods

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Andrew

Does MUC make other opcode caching mechanism obsolete?

Are older methods discussed here for example, "Update on PHP-accelerators" https://moodle.org/mod/forum/discuss.php?d=168965 still valid?
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Moodle 2.4: Effect of MUC on other PHP caching methods

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

MUC is nothing to do with PHP code or PHP accelerators.

Your question is like asking 'Does the Moodle database make PHP-accellerator obsolete?". Answer: no.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Moodle 2.4: Effect of MUC on other PHP caching methods

by Matteo Scaramuccia -

Hi All,
MUC gives the opportunity to some1 PHP accelerators - which will always play, out of the box, the role of PHP opcode cachers - to play a role as cache store too (as done someway in the past for 1.x).

See MDL-37069 for details about an attempt of mine to code an XCache Store: Sam did already the (contrib) job for both XCache ad APC, but I realized the thing just when he replied to me there wide eyes.

HTH,
Matteo

1Those able to do Variable Caching.

Average of ratings: Useful (2)
In reply to Matteo Scaramuccia

Re: Moodle 2.4: Effect of MUC on other PHP caching methods

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

As long as you only have one shared cache for all nodes in a cluster, and not one per node - there's nothing in MUC to handle clearing/invalidating caches which are spread over disparate nodes (yet?).

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Moodle 2.4: Effect of MUC on other PHP caching methods

by Matteo Scaramuccia -

Hi Andrew,
correct and recently "fixed": https://moodle.org/mod/forum/discuss.php?d=222250 and MDL-38059. As you wrote, no way to manage (= coordinate) multiple separate caches at the application (= Moodle) level.

Shortly, talking about the presentation tier:

  • Single Server: boost MUC with a store based on the current supported systems (memcached and its drop-in replacement, mongodb) OR on a PHP accelerator able to to perform variable caching (currently based on some contributions)
  • Multiple Servers: boost MUC with a store based on a supported and sharable system (at this time: memcached and its drop-in replacement, mongodb), dropping options like PHP accelerators being local to the node by nature

Matteo

Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: Moodle 2.4: Effect of MUC on other PHP caching methods

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Sorry - just to clarify for anyone else coming to this topic at a later date, the documentation in config-dist.php is fixed to state that multiple and disparate caches are not supported, rather than that the feature now exists.

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

Thanks all for the suggestions so far far.  We are using IIS7.5 as our webserver. 

I've ran the SQL server profiler on the database and went through a bunch of pageas as an admin and during the time I ran it, the longest running query was 77 microseconds, so it didn't seem like there are any queries that are running really slowly.

I looked at NewRelic as an option to try fo find the bottleneck, but they don't offer a Windows client yet.

We are running Moodle 2.4.3+ from the build from last week, so I guess it's possible that the build from this week could have fixed something, but we are about as up to date on version as possible. 

The issues seem to be disk and CPU related, as both seem to be in heavy use during the delay in load times.  And the only way I've been able to decrease the delay in all of my tests the last few days was actually to kind of break the cache.  I created a store and pointed it a non-existent directory and did not check the box to automatically create it.  After doing that and pointing all of the caches at that store, the page load time for admins went from about 17 seconds to about 9-10 seconds.  Based on this, it seems that the cache is actually slowing down the page loads on the server instead of helping them, which matches the fact the 2.3.4 was faster without it. 

I had thought maybe it was 2008R2/IIS7.5 or a 64-bit issue or a VM issue, so I tried installing a fresh 2.4.3+ install on a physical 2003 32-bit server with a physical SQL server 2005 database, and the load times as an admin were almost identical to our 2008 servers at about 17 seconds per page.  When I installed the latest 2.3.6+ build and installed it on these same physical test servers, the admin pages load time was only about 7 second per page, and this is on older haredware that was not tuned at all for PHP or Moodle.  So at least from these tests, it sure seems like 2.4.3 is slower than 2.3 period on Windows servers, physical or VM.

 

Average of ratings: Useful (3)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Ryan Panning -

We are in the same boat/enviornment. Although we're still just testing the upgrade process and have not had heavy load on the 2.4 install.

Would you be interested if I made a store for MUC that uses WinCache? I would be willing to do so, and will probably need to for our enviornment.

Average of ratings: -
In reply to Ryan Panning

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

We would definitely be inerested in that, as it sounds like a promising solution for all of our issues with the MUC.  Keep us posted.

Average of ratings: -
In reply to Andrew Lyons

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

It seems I got slightly the wrong end of the stick with the Moodle MSSQL driver. Some of the functionality from the freeTDS driver has been mimiced. See MDL-37748 for further information.

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

We ended up having to switch back to the freeTDS driver.  The MS driver seems to perform a little faster overall, but while trying to perform an import/restore of a class, we would receive a timeout error.  Switch back to the freeTDS driver, and we don't see that error anymore.

Average of ratings: Useful (1)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

Update on our sitation:  After throwing a bunch of virutal hardware at it, we were able to make it useable under load, but still far from ideal.  Admin pages were taking 7-10 seconds per page to load, and non admins were taking 2-4 seconds to load. 

 

We then received Ryan Panning's new Cache store plugin for WinCache and implemented it last Thursday, and found on average it reduced page load times by about 30%!  For anyone hosting Moodle 2.4 on Windows, this is a must have to make the MUC perform correctly as compared to a slow disk cache:

https://tracker.moodle.org/browse/MDL-38987

 

Going on another suggestion, we tred a test installation with PHP 5.4, Wincache 1.3.4, and the Microsoft native MSSQL client and drivrs 3.0 in conjunction with the WinCache store.  We found this again reduced load times another 30-40%!  But we are still running into issues with restoring Moodle 1.9 classes as we did when trying the 2.0 client/driver, with larger class backup files, it just stops responding during the final restore phase.  So we are hoping to work that out in order to take advantage of this further performance.

 

I want to see if the FreeTDS drive for PHP 5.4 offered similar performance, but the links is broken to download it from the MSSQL Moodle install guide:

http://docs.moodle.org/24/en/Installing_MSSQL_for_PHP#Using_FreeTDS_on_Windows

Anyone know where we can get this?

Average of ratings: Useful (2)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by vederovan Vandos -

I want to know this too, please tell me

 

thanks

Average of ratings: -
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

I am now able to get to the FreeTDS drivers for PHP 5.4 with Firefox, for some reason IE10 doesn't like them and isn't able to connect.  The zip is over the max allowed size to attach here, but if anyone needs the non-thread safe version, I can send it to you.

Average of ratings: Useful (1)
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Benjamin St.Germain -

We have been on PHP 5.4, Wincache 1.3.4, and the FreeTDS driver for PHP 5.4 for two weeks now, with no major issues and another 20-30% performance increase.  And with the FreeTDS driver, we have had no issues with backup/restore or any other new issues over PHP 5.3.  So this seems to be the best setup in our experience for Moodle 2.4 on Windows.

Average of ratings: -
In reply to Benjamin St.Germain

Re: Moodle 2.4 much slower than 2.3 on Windows Server

by Paul Lindgreen -

We have a similar windows configuration and have had serious performance issues. Just yesterday I increased the mysql setting innodb buffer pool from 100mb ro 3gb and performance has been outstanding the last 24hr. Our zabbix reports showed a disk latency problem on the database server (seperate from the moodle webserver) so we focued our efforts on it. On a stand alone server this value can be 70% of your total ram, if the db is on your webserver I'ld start with something more conservative like 1-2gb, just make sure theres plenty left for regular server operation.

Average of ratings: -