Ouch ... 'complete crash of 128 IT systems' doesn't sound like any fun at all! Whom ever had that unforunate job obviously not really familiar with the setups of a moodle - bet they have made notes now.
I'd like to ask about the 'complete crash ...' but that's probably for off list discussion.
I did see in your profile an xKEMP??? Is that KEMP load balancing? Reason I ask ... recently got involved with an entity that used KEMP load balancer.
KEMP LB also relates to the next comment ... the idea not mine ... but many years ago a large ISD in Texas did something not in any docs of Moodle and it worked well with their overall systems setup - they had a requirement that all file systems had to be in a certain block of ip addresses ... a NFS farm so to speak.
Also ... in 3.5 ... config-dist.php there is a section that might be worth looking at:
14. ALTERNATIVE FILE SYSTEM SETTINGS
// It is possible to specify different cache and temp directories, use local fast filesystem
// for normal web servers. Server clusters MUST use shared filesystem for cachedir!
// Localcachedir is intended for server clusters, it does not have to be shared by cluster nodes.
// The directories must not be accessible via web.
// $CFG->tempdir = '/var/www/moodle/temp'; // Directory MUST BE SHARED by all cluster nodes.
// $CFG->cachedir = '/var/www/moodle/cache'; // Directory MUST BE SHARED by all cluster nodes, locking required.
// $CFG->localcachedir = '/var/local/cache'; // Intended for local node caching.
And think one could do this even with LB system ...
On the web server, the path to moodledata is on the web server ... example: /var/www/moodledata ... but contained there in there is a mount point only for filedir ... that pointed to their huge file (NFS servers).
So the caches, sessions, etc. directories were on the web server (with LB one would have to replicate that setup but doable).
And, as you've already mentioned, using Redis for caching ....
Anyhooo .... 2 cents worth ... after the fact, but ...
'spirit of sharing', Ken