We are experiencing periods of slowness with our Moodle 1.9.15 installation, and identifying the bottleneck is not easy. We have quite a complex setup, as we serve 15,000 students, with nearly 3,000 courses.
First, our setup:
Incoming requests are handled by a BigIP F5 load balancer. We have tried various load balancing algorithms, but in the end we settled on a plain round-robin rotation.
The F5 passes requests to one of our five web servers. These are all Linux VMs, each with 4GB of RAM, running Red Hat Enterprise Server 5.8.
The MySQL database is on a separate Linux server with 8GB of RAM.
File storage is accessed via a NFS Linux server with 8GB of RAM, connecting to an IBM XIV storage system.
When the slowness occurs, running "top" on the web VMs shows I/O wait figures up to 90% of CPU time. Normally we see wait figures 1-10% with occasional spikes to 30-40% if a big request happens. Once a slow period starts (usually between 10am and 2pm) Moodle logins and page loads can take 30 seconds or more.
We are trying to identify the source of the latency. Our Linux systems manager has been investigating, and he says that read requests to the XIV disk system show only 0-2 msec latency, but the file system VM host machine shows considerable read latency.
Is anyone else running Linux VM systems and seeing this sort of problem?