I have had quite a few emails regarding our Moodle Environment since I last posted that we use Galera for MySQL Replication. So here is our setup with feedback.
We currently have 3 Web Servers and 3 MySQL Servers. Eventually we will beef up the main web servers and install MySQL on top of each node so we can use auto-scaling in VMware allot easier, power down when not busy, power up when needed.
We do not use shared storage, read on to find out what we use!
Each Web Server (Ubuntu Server 12.04) has the standard Apache install tuned to our environment with eAccelerator. For Moodledata we use a GlusterFS volume that is served on each server and also mounted on each server. This allows us to replicate the Moodledata in real-time with little or no I/O penalties! Each server mounts the GlusterFS volume at /mnt/vol2/ with a moodledata folder added and configured according to the standard moodle install. To date we currently have not had any issues with GlusterFS, We use the standard Distributed Replicated Volume, One issue we did have that was not related to GlusterFS was the fstab mount documented by GlusterFS that did not work when a node had to be rebooted. We had to move the auto-mount to /etc/rc.local since GlusterFS requires networking to start first before the daemon can start.
Each MySQL Server is the standard Galera Mysql install also tuned to our environment. Since 1 of our MySQL and Web Server is located in another Datacenter many miles away for DR/Failover we installed the Galera arbitrator on the remote Web Server so it helps prevent split-brain. The only issue we had with Galera was with the split-brain situation when our site datacenter lost power and the remote node had no idea what to do. The arbitrator now tells it that it is the master since it cannot talk to the other nodes and all nodes that come online after should recieve the changes from it.
Our setup was designed for Active/Active but we currently use it for DR/Failover.
Each Server is a VM that runs on a Vsphere 5 cluster, Web Servers are allocated 1 Processor/4 Cores total 16GB RAM each, MySQL Servers are allocated 2 Processors/8 Cores total 8GB RAM each
We have a dedicated 30mb/s fiber link between the 2 sites for MySQL and GlusterFS communication.
Moodledata is 41GB in size
Moodle database is 884M in a raw mysqldump
833 Active Users, highest load around 200 or so concurrent connections
Hopefully this provides some insight.