Database cluster is running a modified version of mysql (Percona XtraDB Cluster to be exact), which uses Galera for it's innodb replication, and using the xtrabackup system for non-blocking backups and joiner transfers).
Each of the DB servers do have quite a fair amount of RAM, however there is no query caching occurring due to the inherent nature of replicated architecture, it just doesn't work.
That said, our infrastructure is under very low amounts of load due to the configuration and other caching levels on and in front of the application it's self.
The database servers are multi-master, and the load is distributed by haproxy running on the local application servers, this also serves as automated failover which takes a host out of the loop should it fail or report that it is not ready.
The database setup it's self requires a minimum of 3 nodes, either 2 data nodes and an arbitrator, or 3 data nodes, we're using 3 data nodes to reduce the risk of split brain, and allow the sites to remain operational in the event of a db server outage.
I'm not going to provide configuration files, because they've been developed for our infrastructure, and are not plug and play so to speak, plus if you dropped in our mysql config you would more than likely corrupt your own innodb files due to size changes in the ib_logfile's. Plus there's plenty of documentation around if you know where to look.
There is no local application cache being stored on the application nodes, in fact, they can be treated as ephemeral, all application cache is stored in memcache which is accessed by both servers, moodle data is accessed through a gluster cluster with some special configurations to get around temperamental behaviour (disabling overrides in apache to prevent the lookups of .htaccess on each read was a big one, as is nodiratime,noatime on the gluster server mounts).
There are only 2 gluster nodes, as bandwidth requirements grow linearly the more replica nodes you add, as you write to a gluster mount, it writes to all servers at the same time, so write 1MB in 1 second, it requires 3MB/s.
SSD's tend to make a huge difference in request latency for gluster, and also significantly speed up mysql queries (particularly writes).
At the learn moodle launch we had over 500 users on learn moodle, along with another 400 or so on moodle.org all using the same backend infrastructure, the resulting load on the servers was negligible.
Another thing that tends to help a lot is caching of static assets in nginx, specifically those stored in the data directory (which gluster would have served).
For those interested, I run Apache with mod_php, and nginx as a reverse proxy/static asset cache in front, this allows apache to do it's work, and allow nginx to handle external tcp connections freeing up apache's threads for more work.
Matt
