Hi all,
posting here as it seems it is the good place for this topic.
We startign setting up a big VMoodle prototype on rather slow Azure virtual machines, so we were immediately concerned by setting up as fast as possible a good caching strategy.
The installation features one master Moodle and several (say 4 VMoodle independant instances). Note the issue is note related to VMoodling technique, as it would also appear with hard installed moodle instances.
First we made the common mistake to setup the standard port as 11211 for all, just relying on Memcache different prefixes to store separately our data stubs. This seems not working, and we faced with the "Memcached dead thread" issue, known when several clients from different domains are using the same Memcached process.
Then we tried to change the settings, build a Memcachd daemon manager (small Web driven service to instanciante easily dozen of Memcached daemons on distinct ports), and assign each moodle to those ports.
things appeared going well at this stage, but not for a very long time.
Configuration were fast appearing to mix and override each other.
Starting digging deep in the Memcached store code and Php Memcached documentation, here is what we found :
All seems like Memcached php implementation manages a single pool of cache servers (through addServer() or addServers()) . As Moodle gets known servers using getServerList(), Moodle reads the servers definitions from the Php pool, and does quite not care about changes in cache settings. So all moodles are starting using the same servers and processes when rolling in the distribution list.)
Amazingly, updatig the memcached instance and routing it to a fake IP had no effect on ready state, that was the start point of our interrogation.
Ideas to reproduce the case :
Install 2 moodles on te same apache
setup 2 processes memcached :
* sudo -u memcache memcached -h localhost -d -m 64 -p 11211
* sudo -u memcache memcached -h localhost -d -m 64 -p 11212
configure each moodle using each as first setup
ensure cache stores are ready
Than reroute moodle 2 to a complet faked configuration
If the case is relevant to our experience, the ready state does not fall down as the first server defined by Moodle 1 successfully binds and is in the central pool list.
If any people hove some deep insight about this memcached behaviour, comments (or solution) appreciated.
Our work in progress is to reach running hundreds of Moodle with consistant caching.....
Cheers
Valery Fremaux
Edunao CTO