memcached usage in clustered environment - replication questions

memcached usage in clustered environment - replication questions

by Niels Hendriks -
Number of replies: 2

Hi there,


We're migrating multiple Moodle installations which are currently on standalone VMs into a dedicated cluster. The cluster has 3 nodes and we'd like to use memcached for sessions + caching.


What's currently not clear to me is how Moodle handles the distribution of cache + session files across the nodes. From my understanding:

- I can define a pool with multiple Memcached servers. Here I will define all 3 nodes.

- Moodle will check all memcached servers in the pool and serve cache data from the first found memcached server that has the data available. Otherwise it'll be a cache MISS and the data is added to the cache.

- Moodle will periodically perform a cache flush, which is why it is required to have 2 seperate memcached instances; one for session data and one for cache data.


Now it's fine if cache data is missing as it'll simply need to regenerate the cache. However, missing sessions are of course a problem. The cluster has to be redundant. We want it to run fine with only 1 node up (out of 3). With sessions, it is possible a user is logged out of a memcached server goes down, unless the session data is replicated across all memcached servers in the pool.


It is not clear to me if Moodle handles this natively. I was able to find this URL which supposedly solved this problem in Moodle < 2.8:

https://moodle.org/plugins/cachestore_memcachedcluster

The URL mentions it is now merged in Moodle 2.8+ but I haven't been able to explicitely find this feature.


If it is not possible natively in Moodle I saw that the PHP memcache extension does support this:

http://php.net/manual/en/memcache.ini.php


However, for data caching Moodle uses the Memcached (instead of memcache) extension. Now I'm fine with installing both but if it's not required because of native support I'd rather not do so.


Could anyone clear this up for me and perhaps tell me the recommended way to manage this?

Thanks!

Niels

Average of ratings: -
In reply to Niels Hendriks

Re: memcached usage in clustered environment - replication questions

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

In your list of three points, the second one is wrong. With a multiple memcache servers, it is assumed that these will be the same servers as the web servers. Then, cache read is always from localhost (fast) and writes are copied to all servers in the pool. In Moodle >= 2.8, the multiple memcache server settings are just there when you set up a memcache store.

You are right that the memcache stores from session need to be separate.


Average of ratings: -
In reply to Tim Hunt

Re: memcached usage in clustered environment - replication questions

by Niels Hendriks -

Hi Tim,

Thanks for your response. We tried it out last week and found what you described. Good to have it confirmed.

In addition to that one other point was incorrect in my initial post. Turns out I can use the php memcached extension for both moodle caching and sessions so no need to mix it with the php memcache extension.

I have php set up now to use memcached as a session handler with all 3 ips defined as well as having memcached.sess_number_of_replicas to 3. I also enabled sess_consistent_hash though I'm not sure if that was explicitely needed. See http://nl3.php.net/manual/en/memcached.configuration.php for future reference.

In the moodle config.php I added the required options and also defined all 3 IPs of the memcached servers:
https://docs.moodle.org/26/en/Session_handling#Memcached

We now have 2 actual memcached instances running per server. The memcached sessions instance is 1GB and the caching one is 10GB, we'll monitor to see if that is enough for the sites we're running but I thought it'd be good to mention in case anyone else is looking into doing something similar.

To have multiple memcached instances on Debian 8 some additional steps were needed. This used to be easy in Debian 7 (without systemd) but there appears to be a known bug in the init script that memcached uses on Debian 8 that prevents multiple memcached config files from being read: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=784357
The solution is on the same page and goes to this url: https://moopi.uk/mod/page/view.php?id=71 where the default memcached unit file is masked and then simply copied and enabled for each required memcached instance pointing to their own memcached config files.

This appears to work for us now. Thanks for your response, Tim!


Average of ratings: Useful (1)