After some load tests and consulting Moodle partner experts from Uplearning we managed to find the bottleneck: NFS (of course). Using a fast Redis APPLICATION cache (sessions can go in /tmp or ramdisk) we can now handle 1000 users logging in within 30sec (rampup) doing a 40-question quiz with 1 second delay between each question. The avg page load time is now under 7 seconds.
A thing to look at when using a load test is the number of RUNNING tasks on the webserver. If a lot of processes have a D instead of R (Running) they are probaby waiting for access to a (cache) file in a slow filesystem.
So the benchmark was useful in detecting the slow "create files" score on the moodledata shared application cache. And Redis is a way to get around that and get DOUBLE throughput.
