General developer forum

Thanks for this comment, interesting.

By the way, I am not an expert on this but I wanted to share a few points.

First, just to correct one thing - the configuration for MUC is not actually stored in mdl_config. It's stored in a separate, rather confusing config file in dataroot. So caching mdl_config (and config_plugins please) does make sense and should be one of the quick-win things it achieves.

My main concern about caching is that there is currently no good way to support local caches in a system with multiple webservers. (Large systems where scalability is most critical always have multiple webservers.)

When caching something derived from a database query, if it is a simple database query that executes quickly and entirely from memory (most are), then a significant part of the time cost is probably the network transaction. (Webserver -> database server -> webserver.)

If the cached data is on shared storage (e.g. NFS server, or memcache), then this instead translates into... er... the same amount of network traffic. (Webserver -> shared storage server -> webserver.) It removes load from the database, which is good, but adds it to another single point of failure bottleneck (the NFS/memcache server), which is less good. These other storage servers may be faster and easier to cluster than the database but it's still not a fundamental improvement.

Using local storage (on the same hardware as the webserver) would potentially (and in our testing, does actually) achieve large performance and capacity improvements. Local disk is very fast because disk caching means that frequently-read data comes directly from RAM so there is no I/O - no network traffic, no disk - at all. (Local memcache servers may be even faster.)

Moodle currently basically works if you use local storage for most of the cache but there are two problems:

• When the cache is flushed, e.g. on upgrade, the webserver running the upgrade doesn't know how to flush the cache on all the other servers. I.e. there ought to be some way to specify that a cache is local, and provide a web service on all the machines, and have all the addresses in config, so that it can clear the cache across everything.
• I have a feeling there may be a need for extra metadata in terms of which cache systems can deliver what. For example, most of the cached data is ephemeral meaning that it can safely be stored in RAM and the system will build it up again on next request. But there may be some that isn't - either because it is actually critical data or (perhaps more likely) because it takes a very long time to calculate. Additionally, with regard to the above point, there might be a need to have certain cache backends (local caching) which only support 'flush all' type unsynchronized operations rather than being able to reliably clear a specified key across all systems. (Most cached data should be utilised in a way so that this will be good enough, because otherwise that imposes a high performance cost.)

So to summarise, for a highly scalable system you need:

• Most cached information should be stored on the web server hardware - this can scale (nearly) infinitely because you just put in more web servers if you need, and is fast because all the information is already where it's needed.
• Only cached information that absolutely needs to be shared, such as user session information or information that is exceedingly slow to calculate, should be stored on a shared cache instance. This reduces performance because of the need to wait for network traffic and could be a bottleneck, so this type of cache should only be used where fundamentally necessary.

There ought to be a way to distinguish the types of information so that you can set up a system and have it automatically do the right thing with the highest performance depending on the available caches you've set up. It kind of looks like the infrastructure for this is nearly in place but with some missing pieces.

One other note - possibly the busiest cache is the lang string cache and this caches information which in some installations (which don't use foreign languages) can be derived entirely from local disk (PHP files in the codebase). In fundamental terms, caching this on shared storage would be expected to cause a performance decrease - although PHP code interpreting is currently so slow, that isn't a given.

--sam

I am really happy that you are working on this.

I assume you are also following the Hardware and Performance forum, and this thread, where I make good on my promise above to share some of what we learned.

Speaking of those callgraphs...

This was on a site with a default install (guest and admin user only) and a single course with nothing extra added to it (caution, links are to VERY large XHProf callgraph PNGs):

1. unmodified code (current MOODLE_25_STABLE)
2. that tweak we made to remove the MD5 content verification (lines 681-685 in /cache/stores/file/lib.php commented-out)
3. with the CACHE_DISABLE_ALL constant set to completely disable MUC (which results in a ~5x slower page load)