Moodle 2 pluginfile.php performance

Moodle 2 pluginfile.php performance

by Ben Graham -
Number of replies: 16

I've tried many suggestions from this forum to try to improve the performance of my moodle 2.1 installation, but it is still extremely slow. Everything works and I don't get any errors, but literally every single request made by pluginfile.php takes well over 3 seconds when the file is not in my cache. Regardless whether a file is under 1K or 200K, or any other size, it always takes at the very least 3 seconds. Other non-moodle websites that I have running on the same server have response times of only a few milliseconds for such small files.

Surfing the moodle website is actually OK (a little slow, but acceptable), but taking courses is almost unbearable. I am using a windows server 2003 with decent hardware and I am hitting moodle with only 1 user! Can someone please tell me if this is normal due to the way moodle 2 stores files in the database? What are some of your pluginfile.php response times? (I am seeing my times with firebug extension for firefox). Also, if you have fast response times for your database files, could you please tell me how you did it?

Some of my important settings are pasted below. Please note that everything is working ok without errors. All requests come back as 200 OK, but they are just super slow. I am using MSSQL and windows, and changing to Unix and Postgre is not an option... Thanks in advance for your time and help.

Moodle 2.1 (Build: 20110701)
Extra PHP memory limit 1024M
Zend Extension  220090626 

PHP Version 5.3.6 non-thread safe
Architecture x86 
IIS 6 with CGI/FastCGI
MSSQL 2005 with FreeTDS
DBType mssql_n
dbpersist = true
Session Support enabled 
session.cache_expire 180

Some PHP ini settings:
realpath_cache_size = 200K
realpath_cache_ttl=200
memory_limit = 512M
log_errors = Off
track_errors = Off
html_errors = Off
extension=php_dblib.dll
odbc.allow_persistent = On
mssql.allow_persistent = On

wincache-1.1.0-5.3-nts-vc9-x86

Windows Server 2003R2 Standard Edition Svc Pack 2
Dual-Core AMD Opteron
Processor 1212HE
2.01 GHz 2GB Ram

The server is dedicated, not virtual, and has an extremely fast connection.

Average of ratings: -
In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Are you running a PHP op-code cache like APC or eAccellerator?

In reply to Tim Hunt

Re: Moodle 2 pluginfile.php performance

by Ben Graham -

Running wincache-1.1.0-5.3-nts-vc9-x86

It was much worse before I installed wincache, so I know that it is working. I don't know if it is just how it is supposed to work.

Over 3 seconds for each request easily adds up to over 10 seconds to render a simple page of a course with tiny images of under 1k.

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I just wanted to eliminate the most obvious possibility first.

And just to say the obvious, no, Moodle should not be that slow.

The next thing to look at is what limitation you are hitting, and it will probably be either memory or CPU, and it will be caused by either the database or the web server.

Can you monitor those while trying some page loads, to see if you can work out where the problem is.

Or it may be that something is badly configured, so there is plenty of memory in the server, but something that needs lots of memory to run efficiently is not using as much as it needs.

That is all a bit vague because I don't konw much about administering Windows. I just know in principle the kind of things you need to look for.

In reply to Tim Hunt

Re: Moodle 2 pluginfile.php performance

by Ben Graham -

Thanks Tim,

I did a very thorough performance inspection (perfmon) on the mssql processes, iis processes, processor, memory, and hard disk on my server this morning. I could not detect anything wrong and the lag seems to be somewhere in between the browser and the server. I did trace routes (tracert) to see if my browser requests were reaching a bottleneck while trying to resolve the dns and get to the server, but everything is working very fast. This leads me to believe that all the hardware and the network are ok.

I've read somewhere in these forums that moodle on iis + mssql can be expected to be slower than apache + postgresql. But, luckily you just assured me that it is not supposed to be too slow either, so I think the problem is an incorrect moodle setting or a bug somewhere (php drivers, moodle plugin conflicts, moodle scripts, etc).

Do you know of any settings that I might have overlooked that would directly affect the performance of the database access scripts in moodle?

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Some more random thoughts:

When this week's weekly 2.1.x+ build is done, you probably want to upgrade to it, since MDL-28708 and MDL-28709 are both nice performance wins. However, that just mitigates the problem of your server being unexpectedly slow, it does not solve it.

Another obvious thing to check: In the admin menu, search for 'cache' and make sure all those settings are set to the default, and not the settings you would use for doing development.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Moodle 2 pluginfile.php performance

by Ben Graham -

Cool, thanks a lot Tim. I am caching as much as possible and ajax is turned on. The moodle installation was completely manual, so I optimized the other settings for production instead of development.

I think I may have additional clues from my FreeTDSDebug.log file. My freetds.conf looks like this:

[global]
host = localhost
port = 1433
client charset = UTF-8
tds version = 8.0
text size = 20971520
dump file = c:\FreeTDSDebug.log
debug level = 1

It appears that the TDS driver is trying to convert UTF-8 to ISO-8859-1 and logging errors in the background. Any ideas on this?

log.c:190:Starting log file for FreeTDS 0.65
    on 2011-08-25 14:45:53 with debug flags 0x4fff.
iconv.c:363:iconv to convert client-side data to the "UTF-8" character set
iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
net.c:210:Connecting to 127.0.0.1 port 1433 (TDS version 8.0)
net.c:264:tds_open_socket: connect(2) returned "Unknown error"
net.c:303:tds_open_socket() succeeded
util.c:162:Changed query state from DEAD to IDLE
login.c:735:quietly sending TDS 7+ login packet
token.c:312:tds_process_login_tokens()
net.c:592:Received header .....

There are tons of other mysterious stuff like this in the actual file (about 838KB)...

Thank you so much for your help!

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Sorry, you have gone way past anything I know about here.

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

where is your moodledata located? - on a disk on the local machine? - or on some external SAN/mapped drive or something similar?

In reply to Dan Marsden

Re: Moodle 2 pluginfile.php performance

by Ben Graham -

Hi Dan,

Thanks for you reply. I have two physical local drives with fast access times.

The first drive has MSSQL, IIS, and PHP.

The second drive has the moodle files and the moodledata folder.

Both drives have plenty of space.

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Troo Don -

As you say, your hardware is not slow, so it must be a software issue. The other sites you're running on the server are based on PHP+MSSQL too? Some things you could try:
- Your FreeTDS version seems to be very old. Try a more recent one (see Installing MSSQL for PHP).
- Alternatively, try the Microsoft SQL Driver for PHP instead of FreeTDS  (see Using the Microsoft SQL Server Driver for PHP).
- Check your dbpersist setting in config.php, as for Moodle 2.X it's a part of dbarray and not a setting of its own.
- If you have the chance, try swaping IIS+Wincache for lighttpd+XCache, nginx+XCache or similar.
- Don't you have the possibility of trying PostgreSQL for Windows?

In reply to Troo Don

Re: Moodle 2 pluginfile.php performance

by Ben Graham -

Hi Troo,

The other sites are .Net + MSSQL and they are blazing fast. I was doing lots of experiments yesterday, and as a matter of fact I did try the MS SQL driver for PHP. It sounds great, but I absolutely could not get it to work after many hours of struggling. Every time I enabled php_sqlsrv_53_nts_vc6.dll my moodle installation would completely break. Is it possible to change the database drivers AFTER a successful installation, or do you think I need to install the whole moodle package again?

About lighttpdf + XCache, I never heard of these things so I will take a look. Thanks for these new ideas.

I also decided to try PostgreSQL a couple of days ago on a test server with a test installation of Moodle, but it breaks my test installation and I am still struggling. Do you know of any good documentation for windows moodle + postgresql?

Thanks for your suggestions!

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Ben Graham -

I GOT IT TO WORK!!!!

The problem was the MSSQL FreeTDS driver. I switched it to the newer MSSQL driver version 2 (released 6/28/2010: SQLServerDriverForPHP20.EXE) for PHP and IT IS BLAZING FAST NOW!

Hopefully I can save many days of aggravation for other people trying to make Moodle 2 work with Windows Server 2003, IIS6 with FastCGI, Wincache, PHP 5.3, and MSSQL 2005. Here is what I did:

Download the driver package here:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=df4d9cc9-459c-4d75-a503-ae3fceb85860

Unzip the contents and place the file php_sqlsrv_53_nts_vc9.dll in your C:\PHP\ext folder.

Now download the MSSQL 2008 native ODBC driver (even if you have MSSQL 2005, it works) from here:

http://msdn.microsoft.com/en-US/library/cc296170%28v=SQL.90%29.aspx

My server is 32bit so I downloaded the x86 package. I installed only the client (default option without any extras)

Now, in your moodle/config.php make sure the database settings are:

$CFG->dbtype    = 'sqlsrv';      
$CFG->dbhost    = 'localhost';  
$CFG->dbname    = 'Moodle';     
$CFG->dbuser    = 'MOODLEDBUSERNAMEHERE';   
$CFG->dbpass    = 'YOURPASSWORDHERE';   
$CFG->prefix    = 'mdl_';       
$CFG->dboptions = array(
    'dbpersist' => true,       
    'dbsocket'  => false,
    'dbport'    => '1433',   
);

Now edit your c:\php\php.ini to have the following extension:

extension=php_sqlsrv_53_nts_vc9.dll

Make sure any other mssql drivers are commented out, such as:

;extension=php_dblib.dll

-------------------------------------

What really did it for me was a test file that prints detailed error messages. I created a test.php in the moodle root with the following code:

<?PHP
$link = sqlsrv_connect('localhost', array('UID' => 'MOODLEDBUSERNAME', 'PWD' => 'MOODLEUSERPASSWORD', 'Database'=>'Moodle'));
if($link === FALSE) {
    echo 'Could not connect:<br><br>';
    die( print_r( sqlsrv_errors(), true));
}
echo 'Successful connection';
sqlsrv_close($link);
?>

This gave me detailed error messages which helped me to fix my problem. Hopefully these tips will save a lot of time and aggravation for other Moodle Windows users.

Thanks to everybody who helped me here!

 

Average of ratings: Useful (5)
In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Wow! who would have thought that a simple driver change would make such a difference.

Or, to put it another way, would would have though that a crappy DB driver could have as bad an impact as the old SQL server one did wink

Thank you for taking the time to report back here after you solved your problem.

If you have any more time, you might have a quick look at http://docs.moodle.org/20/en/Installing_Moodle or http://docs.moodle.org/20/en/Performance and see if there is anywhere there where it would be appropriate to add some notes (or perhaps just a link to this thread).

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Ben,

thanks for sharing the experience! And great you got it working fast! cool

Just for reference I'd point to this 2 pages in Moodle Docs, where both FreeTDS and the MS SQL Driver for PHP are explained and docummented (feel free to improve them, it is a wiki!):

Note that in the FreeTDS docs we are pointing to proper (own) builds of the library for windows (0.82+patches) because, in the past, a lot of problems and bugs were detected and, after a lot of combined work, we ended with the versions available there as optimal. I bet the version you were using (0.65) was the culprit for all your problems. I use the 0.82 daily (Mac OSX) to execute unit tests and I get them completed roughly in the same time than the MySQL and PostgreSQL ones. Also, it has been used in production environments along these years (since Moodle 1.7) and seems to work ok.

The MS Driver is a newcomer and it is supported since Moodle 2.0, that was released a few months after the driver (v1.1) added support for UTF-8. Before that was simply impossible to use it in a satisfactory way. Microsoft did the hard work and created the initial versions of the sqlsrv driver for Moodle. Since then it has been supported by Moodle HQ, and I'd say it's on pair (about maturity and stability) with the FreeTDS based one.

And that's the history of the two currently supported drivers for MSSQL databases, hope it has been illustrative.

Finally, perhaps it's a good general recommendation to encourage people to use the MS Driver if they are running the web server under Windows and leave the FreeTDS for sites running under non-MS stacks (Mac OSX, Linux, BSD...) but against MSSQL databases.

Ciao smile

In reply to Ben Graham

Re: Moodle 2 pluginfile.php performance

by Troo Don -

I'm glad you solved the issue! And thanks for posting back such a detailed solution.

lighttpd and ngnix are "lightweight" HTTP servers (I would recomend Cherokee, but its Windows builds have been broken for quite some time), and XCache is a PHP op-code cache (like Wincache). The point here is maybe these things will work better for you and as a side-effect, as they are multiplatform, you won't be tied to Windows.

How is breaking your PostgreSQL+Moodle test install? I mean what errors are you getting. My experience on administering Moodle is mainly on Linux, but AFAIK it should be as easy as installing and configuring PostgreSQL (getting an optimal config may take some time if you are not used to is plethora of parameters, but for testing the out of the box config is ok), checking the pgsql PHP module is installed and works, and performing the usual Moodle install. As you're doing a test installation you could try the latest RC of 9.1, which has some small improvements for Windows (like the pg_ctl one).