2.5+ (Build: 20130524)
2X RHEL6 nodes, apache + apc, Postgresql, NFS shared, F5 Loadbalance
After upgrade from 2.3.3 i found when user is logged in the site has +- 6 sec loadtime.
I traced the problem to slow write to moodledata/cache/cachestore_file (NFS write).
Will i create inconsistencies if i mount cachestore_file in memory (tmpfs) on nodes.
Hi Johan,
I'm afraid that you categorically can not have unshared cache directories at present. This is an absolute must. There's some work in progress to improve this state of affairs, and that work is discussed in http://docs.moodle.org/dev/Server_clustering_improvements_proposal.
I'd suggest using a memcached store using the memcached plugin (I've seen some issues with the memcache plugin) as this should give you a definite boost wrt performance.
Best wishes,
Andrew
Andrew,
You mentioned you cannot have unshared cache directories, but do you know if it okay to have unshared memcached stores? For example we have 3 app servers behind a load-balancers and we are planning on installing memcached on each one and enabling each server to use the local memcached (127.0.0.1) for MUC Application caching only.
- Mike
Hi Michael,
you can't, whatever back-end you are using for the unshared MUC store.
What is the missing point? The ability to invalidate a cache entry to allow each front-end node to see the updated situation and avoid strange/unpredictable issues. If you've a cache definition and you now how it is used and the impact of not being invalidated you can try to un-share it.
An example could be the localization strings that could be managed at local level being created from the PHP files but if you update your Moodle code in each node you must take care of manually purging the store where the strings definition has been mapped into: when Moodle will be able to notify about the changes someway in a unshared cache setup everything will magically happen.
In case of adopting memcached as the back-end you could look at repcached to try to add the missing unshared cache nodes collaboration by using its own replication mechanisms which unfortunately are asynchronous, again leading potentially to unpredictable results in Moodle.
BTW there is some interesting ongoing work in this area that could open, in Moodle 2.6 and above, to a scenario different from the one described above: http://docs.moodle.org/dev/Server_clustering_improvements_proposal.
HTH,
Matteo
It's OK to have unshared application caches (directories or memcached) BUT (and this is a big butt), you (as a sysadmin) need to MANUALLY CLEAR THE CACHES at the right times (for example, every time you upgrade Moodle code).
There is unfortunately no tuning guide for this in our docs, but I am asking our developers to fill this out: http://docs.moodle.org/25/en/Caching#Cache_definition_configuration (MDL-41067)
I'd appreciate help from anyone willing to look at the MUC-related code to fill out details in those docs.
No, it isn't OK.
For example, if you put the question definition cache in an unshared place, you will need to manually clear the cache on all servers every time a teacher edits a question. Clearly that is nonsense.
You can put caches that only cache things that change on upgrade (e.g. minified JS and lang strings) on local storage, if you purge them manually.
Or you can wait until Moodle 2.6 where a new sort of cache is being introduced for these things.
I don't see how it's nonsense., Tim.
To use your example, your statement that the question definition cache would require manually clearing every time a teacher edits a question is exactly the sort of information an admin needs to make an informed decision, based on how they use Moodle.
They may know that quiz questions are not updated in a production instance, and so it might make sense to use a local cache for it for a bit of extra speed.
Oh, I agree with Tim 100 % here, adminstrators do not know the actual code that uses the caches, only the developer can decide when things can go safely to local cache or not, otherwise things start breaking randomly. The options for application caches are:
* shared cache
* local cache compatible - no action required on cluster nodes after upgrade
* local cache possible if manually purged after upgrade on all nodes
If anybody decides to ignore the shared cache requirement the setup if unsupported and things may stop working or data may be lost. Question and new course caches are good examples of areas that need shared caches because data must be invalidated after teacher actions that are not controlled by administrator.
* shared cache
* local cache compatible - no action required on cluster nodes after upgrade
* local cache possible if manually purged after upgrade on all nodes
If anybody decides to ignore the shared cache requirement the setup if unsupported and things may stop working or data may be lost. Question and new course caches are good examples of areas that need shared caches because data must be invalidated after teacher actions that are not controlled by administrator.
I've been thinking that if we are going to document this, trying to document it on the wiki is going to be very error prone.
Instead, we shoud document all the semantics of each cache (as much as possible) in the cache.php file where it is defined. If it is documented there, integrators can insist on docs before changes are integrated. We can also display the information on the cache admin screen.
I guess a good way to caputre the semantics is to use event-based cache invalidation (rather than manual cache invalidation code) whereve possible.
I also think the best solution would be for us to find a way for Moodle to reliably do purging of local caches in a load-balanced scenario. I know that it a hard problem, but if we can solve that hard problem, it just deals with the issue for all admins.
Same problem in a new 2.5.1 test installation (upgrading from 2.3.4). Performance issues (100 times slower) writing to /cachestore_file on a NFS shared. At the moment we have solved linking to another directory (in a non-NFS share), but it's not a real solution... Any advice or we have to wait 2.6?
Thanks
You would need to go the memcache route or setup an iSCSI Target to mount the shared volume if you want to stay with disk based cache. Depending on your OS you could also setup ATAoE (ATA over Ethernet) which is faster than iSCSI and NFS, performance is more like Fibre Channel and is simple to setup.
Jason
Hi All,
just to mention that if you're sharing the $CFG->dataroot folder for front-end clustering reasons i.e. not just for avoiding to install a large disk locally to your single front-end, memcached is the only supported route.
Something could change in 2.6, as mentioned above by Andrew.
HTH,
Matteo