Hardware and performance

 
 
Picture of Andrew Emmett
Poor Performance On Windows (Moodle 2.4.1)
 

We have a large Moodle 1.9 Installation (48386 Students & 825 Teachers). The site runs well and most page loads range from 0.1 - 1.1 seconds to generate the page dependning on content. We have had a pilot environment running Moodle 2.4.1 on the same hardware and are finding the performace to be really slow. It normally take between 2-8 seconds to render the page. I am doing these tests out of hours, so it not even as though there is high utilisation on the servers either. I can't really see any official benchmarks on moodle.org in relation to performance expectations, but this artcle (http://www.iteachwithmoodle.com/2012/11/17/moodle-2-4-beta-performance-test-comparison-with-moodle-2-3/) infers that the 'Time to generate a page ' should be around 0.1 seconds. I'm not sure if that is the same stat as is displayed in the footer when performance stats are turned on. I did notice that debuging was turned on as one of my staff was trying to fix something, so after turing it off, I got a bit of a performance increase, but most pages still take 2-4 seconds to load... I have been playing with many options and tried to optimise to the best of my ability, but don't really know what else to do. I'm wondering if anyone has any advice? And not wishing to sound ungreatful, please don't turn this into a 'You should try Linux' thread as thats not an option. Constructive advice only...

The setup of my environment is as follows:

  • 4 x Windows servers 2008R2 with 8GB RAM runing IIS 7.5 and PHP 5.3.20. PHP Wincache 1.1.0630.0 extension is used as PHP Accelerator
  • 2 x Windows servers 2008R2 Database server running Microsoft SQL 2008 R2. Servers are clusters with one Server Active whilst the other servers other purposes but is the failover partner incase of hardware failure
  • Database connectivity is provided using FreeTDS
  • Moodle Version is 2.4.1+ (Build: 20130118)
  • Moodle Web Directory & Moodle Data Directories are on local storage and replicated amoungst front-end servers usinf DFS-R (Temp directories are excluded from replication)
  • Front-end servers are load balanced using Windows Network Load Balancing
  • All Moodle Content is deliverd over HTTPS (I am aware of the potential performance issues regarding this, but our Moodle 1.9 installation uses the same config and is perfectly performant in my view. I have also disabled it to check and there was no difference in performance.)
  • dbsessions is set to true because we have multiple servers
  • LDAP Authentication with Active Directory is enabled
  • Enrolments are done manually so there is not latency from enrolment plugins

I have tried the follwing to optimise things further:

  • Changed server references from DNS Names to IP addresses in case DNS resolution was introducing latency (even though this should be cached)
  • Uncommented '$CFG->sessionlockloggedinonly = 1;' in my config.php
  • Uncommented '$CFG->extramemorylimit = '1G';' in my config.php
  • Uncommented '$CFG->enablecssoptimiser = true;' in my config.php

Performance info (perfdebug) output for a course with a few topics containg content

2.088613 secs
RAM: 13.2MB
RAM peak: 13.4MB
Included 694 files
Contexts for which filters were loaded: 1
Filters created: 2
Pieces of content filtered: 1
Strings filtered: 0
get_string calls: 1368
strings mem cache hits: 0
strings disk cache hits: 2101
DB reads/writes: 69/3
Session: 3.9KB

Caches interaction by definition then store
core/string ** static persist **: 1739 / 362 / 0
cachestore_file: 362 / 0 / 0 Total Hits / Misses / Sets : 2101 / 362 / 0

 Thanks in advance

Andy

 

 
Average of ratings: -
Picture of Dan Marsden
Re: Poor Performance On Windows (Moodle 2.4.1)
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Particularly helpful MoodlersGroup Translators

Wincache has some issues - try changing accellerator to see if that helps?
https://moodle.org/mod/forum/discuss.php?d=220695#p959164

also make sure theme designer mode isn't turned on.
I'd try a simple test - create a file that only includes config.php (don't include anything else except maybe an echo "test") then monitor how long that takes to load.

you could also engage a Moodle Partner to help diagnose where the bottleneck might be. (if you can find one that will help with a windows related issue)

 
Average of ratings: -
Picture of Andrew Emmett
Re: Poor Performance On Windows (Moodle 2.4.1)
 

Thanks Dan

I think I might have been very stupid. I have been trying to measure performance based on my login. I am a site admin, so it seems that being in a site administration role massively increases the time required to generate a page (400% + in my experience). When I tried the same course page as a student or teacher, the page generation time fall back from 2 seconds plus to 0.2 to 0.3 seconds. My Moodle farm is nearly 5 years old and based on Intel Xeon E5310 @ 1.60GHz (CPUMark Score 2228). When I contrast this with the benchmarking done by Frederic Nevers (http://www.iteachwithmoodle.com/2012/11/17/moodle-2-4-beta-performance-test-comparison-with-moodle-2-3/) I see he was using an Intel Xeon E5645 @ 2.40GHz (CPUMark Score 6545). Since this processor used in his benchmarking is near 3 times faster than my kit, I feel pretty confident that I'm probably getting similar performance levels. We're hoping to do a hardware refresh next year, so I think we'll probably be OK with things for now and page loads under 1 second are fine as far as I am concerned. I suppose the moral of the story is that when evaluating performance, do it as a student. Having seen other people complaining about performance, I wonder if this may be what happening to them? It might be useful to know if other installations display the same behaviour, ie a 400% increase in page load times when comparing the admin role with a non admin role.

As an aside, I have tried several different PHP accelerators and also updated my config for Moodle 2 to PHP 5.4.10 and Wincache 1.3.4 and I have to say Wincache is the most performant in my IIS based infrastructure. I tried XCache & APC, with XCache performing closest to Wincache, but I kept getting weird anomalies on the page with it. Wincache seems to be the smoothest running of all the accelerators from my experiments.

I’m not getting any errors with Wincache (http://sourceforge.net/projects/wincache/files/wincache-1.3.4/) at present, even with backups, although I suspect that this is because my moodledata directories are accessed via a local path instead of a UNC. I replicate my moodledata directory using DFS-R but was planning to move it to a UNC at some point so may still face a possible issue, but not for the moment.

Hope this help someone

Andy

 

 

 

 

 
Average of ratings:Useful (1)
I Teach With Moodle
Re: Poor Performance On Windows (Moodle 2.4.1)
Group Testers

Thanks for the follow up Andrew. You make a very good point about admins testing their site when logged in as 'administrators' and not 'students' or 'teachers'. Here is a table showing the differences for the same page, depending on the role the page was loaded with:

Difference between roles

Cheers, 
Fred

 
Average of ratings:Useful (3)
My picture
Re: Poor Performance On Windows (Moodle 2.4.1)
Group Particularly helpful Moodlers

Hello,

Thanks for the useful information shared in this thread. Yes, logging as a student reduces the average 'loading delay' for pages and resources in our Moodle site, which is hosted in a Windows 2008/IIS7 VM. However, it only reduces the delay in 1 or 2 seconds, and the number of requests, when clicking from the home page to a course, decreases from around 100 requests to 59-60. I also have a Windows 2008/IIS7 VM (see previous post: https://moodle.org/mod/forum/discuss.php?d=223952)

However, I still get loading times of around 3-3.56 seconds which I need to optimise.

We are using local pahts not UNC (I need to double check this with our technician) so I can try WinCache as a php accelerator.

I would like to discart all the possible solutions before trying the php accelerator, so I want to try what Andy suggests, uncommented the &cfg options.

Regarding this option:

'Changed server references from DNS Names to IP addresses in case DNS resolution was introducing latency (even though this should be cached)'

Can you please explain how do I change the server references ?
Do you mean in the config.php file or in the web.config file ?

We have a re-routing rule in IIS7 so that all the http requests are redirected to https, but I don't think that this is causing any latency.

Likewise we need to keep the https protocol because we use Turnitin Moodle Direct. The site was quite slow before installing Moodle direct, and I don't see any requests to the Turnitin IP address when I click on pages and courses, so I can fairly say that Moodle direct and Turnitin are not causing any additional latencies.

I would like to know if you are using fast-cgi.exe and which values you have for the following variables:

Instance MaxRequests: 10000

Activity Timeout = 600
Idle Timeout = 300
Request timeout = 600

 Keep alive is set to 'true' (it only increased the delay when I changed it to 'false'

 

 
Average of ratings: -
My picture
Re: Poor Performance On Windows (Moodle 2.4.1)
Group Particularly helpful Moodlers

I tried the standard theme on, displaying the performance information at the footer of the page.

I tried in our test server, as I cannot switch to standard theme now.

The course does not have may resources, as you can see the number of DB read/writes is very high and latency quite high (in our main site it would be around 3-5 seconds, which is still high).

 

9.504206 secs
RAM: 50.1MB
RAM peak: 50.5MB
Included 800 files
Contexts for which filters were loaded: 27
Filters created: 108 Pieces of content filtered: 14
Strings filtered: 357
get_string calls: 2202 strings mem
cache hits: 2387
strings disk cache hits: 221
Included YUI modules: 54
Other JavaScript modules: 4
DB reads/writes: 918/14
Session: 21.2KB
 
Average of ratings: -
Picture of Andrew Emmett
Re: Poor Performance On Windows (Moodle 2.4.1)
 

Hi Mari

The use of a PHP accelerator is an important part of increasing the performance of PHP. For a quick explanation:

When you access a PHP file via the web browser, the webserver has to read the PHP file from the file system and passes it to the  php framework which in turn scans through it to identify the PHP code and create a semi-compiled version of the code for execution by the PHP framework. This takes milliseconds to do, but moodle has to do this for every php file is uses. Moodle is built in a very structured way so that code can be reused, so each php file also has dependencies on other PHP files. This means that when you load a php file such as view.php it may be loading 20 - 30 other php files. As you can imagine, this adds further latency to things. You also need to remember that moodle also has to go out across the network to the database server and perform additional SQL queries. All of this contributes to the load time of the page.

A php accelerator caches semi-compiled copies of the php files in memory and when the webserver gets a request for the page again, it bypasses the compilation steps and gets the precompiled version from the memory cache. As you can imagine, this is many times faster the non accelerated process. Some caching engines also provide other enhancements that add further optimisations but I'm not really qualified to discuss those.

Its terms of the database, I'm not familair with MySQL as we use MSSQL, but I would think that this is unlikely to be the bottleneck as moodle builds its own database and indexes it, so performance is not likely to be an issue unless you database server is really busy.

I know some people have had issues with it, but wincache is working great for me. I use 1.3.4 (http://sourceforge.net/projects/wincache/files/wincache-1.3.4/) with PHP 5.3.20 for Moodle 1.9 and PHP 5.4.10 with Moodle 2.4.1 and have no issues. You fine that when you first access a page, the access will be slow, but if you the refresh the page, if shoudl be a lot faster as subsequent request will be pulled from the cache.

In regards to my 'Changed server references from DNS Names to IP addresses in case DNS resolution was introducing latency (even though this should be cached)' comment, just ignore that as I was really clutching at straws and trying to eliminate every possible issue, even though this was highly unlikely.

Like you, I have a rerouting rule in place too and have no issues. I think you can rule that out.

I am also using fast cgi with the same setting you have, so again, I doubt that's the issue.

You mention you are using 2.2.4 and my real performance measurements have been done using 2.4.1, so there might be other factors and I know that they have done work to improve performance in Moodle 2.4 (http://docs.moodle.org/dev/Moodle_2.4_release_notes#Performance_improvements)

I really suggest giving wincache a whirl to see if that makes a differece for you. In the download, you will see the wincache dll, but there is also a 'wincache.php' file that can show you what is being cached. Modify it in a text editor (notepad++ is great) to set up a username, drop it into you moodle webserver root and then load the file through the browser to see the stats.

Hope this helps

Andy

 
Average of ratings:Useful (1)
Picture of Jos Berkhof
Re: Poor Performance On Windows (Moodle 2.4.1)
 

Hi Andy,

I'm using moodle 1.9 and considering using WinCache.
I've installed it locally on my test machine and it looks to be working fine.

I'm in doubt if I need to configure the Cache Type and Record Cache in Server|Performance settings.

Did you configure that as well or did you only install the WinCache extension in PHP?

Also are there any other recommendations you can make about the WinCache settings?

Finally, did you find any issues with moodle 1.9 and using WinCache?

Thanks in advance.
Jos

 
Average of ratings: -
Picture of Benjamin St.Germain
Re: Poor Performance On Windows (Moodle 2.4.1)
 

We have used WinCache with Moodle 1.9 for 3ish years now with no issues.  You don't have to configure any Moodle cache settings, WinCache will cache all PHP by itself.  We are using an older version (1.0.1325.0) with our 1.9 servers as we haven't had any issues to need to update it.  I think the only changes we did to the default config was to increase the size of the Opcode and File cache to make sure there was plenty of room in memory.

 
Average of ratings:Useful (1)
Picture of Jos Berkhof
Re: Poor Performance On Windows (Moodle 2.4.1)
 

Thanks Benjamin for the advice

Any idea about the size of the OpCode and file Cache?

 
Average of ratings: -
Picture of Benjamin St.Germain
Re: Poor Performance On Windows (Moodle 2.4.1)
 

It looks like Moodle 1.9 only is only currently using 43MB in the Opcode Cache and 7MB in the File Cache.  But I upped our maxsize on both way over that just in case, since we have plenty of memory to accommodate it, if needed.

 
Average of ratings: -