Hardware and performance

Recently someone told me about the X-Sendfile support that was added to Moodle in version 2.3 and I just wanted to publicise the feature to let others know about it.

I don't believe that it's currently well documented which is a shame, and something I'd like to improve when I have a few minutes. The current documentation that I've come across is all in config-dist.php (search for X-Sendfile).

The idea of the feature is to get the web server to serve files from moodledata rather than having it done using PHP. On the whole, this is a good thing for a number of reasons:
* php isn't particularly efficient at serving files in this way (byte-serving) - it doesn't cache files for starters (though your kernel should of course);
* while php is serving a file, it's unlikely to fork in an effective manner so you're also locking processes which could otherwise serve your users;
* your web server's primary aim is to serve files - it's designed for this and it is designed to do so in an extremely efficient manner. It's also likely to handle caching much more efficiently.

It's relatively easy to set up and at present there is support for:
* Apache2 - using the X-Sendfile header and mod_xsendfile (see https://tn123.org/mod_xsendfile/);
* nginx - using the X-Accel-Redirect header (see http://wiki.nginx.org/XSendfile); and
* lighttpd - using the X-LIGHTTPD-send-file header (see http://redmine.lighttpd.net/projects/lighttpd/wiki/X-LIGHTTPD-send-file).

We recently switched to serving the files for our main production site from moodledata (stored on NFS) to our nginx gateway server using X-Accel-Redirect. Before the switch, we had been experiencing some performance issues whereby our NFS server was sometimes being extremely slow to deliver files, which in turn caused PHP to wait whilst serving the file and we consequently saw high IO and a range of other issues affecting all users. Unfortunately the issues were usually extremely localised (affecting only some of five web nodes at a time) and short (10-30 seconds) which made them very difficult to track down accurately whilst still having a noticeable affect on end user experience.

We did make one additional change to our nginx configuration in addition to those detailed - we added configuration to ensure that our ETag headers were preserved correctly.

# Preserve the etag - X-Accel removes these
add_header Etag $upstream_http_etag;

After the switch we haven't suffered these issues any further and our performance on the whole is better. This hasn't moved the issue either - our gateway servers are seeing an increase on IO, but only very marginally. Unfortunately, I don't have any real metrics to demonstrate these improvements (sorry).

I've attached a copy of one of our munin graphs showing the change in IO before and after the switch from one of our web nodes.