Hi, we have a deployment of Moodle within a Kubernetes cluster. We have multiple nodes, with the moodle application itself on every node for redundancy, an mcrouter instance on each node (again, for redundancy and to reduce latency), and a memcached server on each node. All memcached servers are reachable from the Mcrouter instances, and each application container references the Mcrouter instance on its own node as the session_memcached_save_path in config.php.
All runs fine when there is a single node, however when there are multiple nodes (and therefore a distributed memcached data store) we occasionally getting session errors being displayed (in the not-very-pretty blood-red times new roman on a pale yellow background div with rounded corners moodle error style...) - most notably when a logged in user's session expires.
Debugging gives us the following errors:
Warning: session_start(): Failed to write session lock: SERVER ERROR in /bitnami/moodle/lib/classes/session/handler.php on line 42 Warning: session_start(): Unable to clear session lock record in /bitnami/moodle/lib/classes/session/handler.php on line 42 Warning: session_start(): Failed to read session data: memcached (path: [IP_ADDRESS]:5000) in /bitnami/moodle/lib/classes/session/handler.php on line 42 Debug info: Error code: An error occurred whilst communicating with the server $a contents: Stack trace: line 82 of /lib/classes/session/manager.php: core\session\exception thrown line 794 of /lib/setup.php: call to core\session\manager::start() line 47 of /config.php: call to require_once()` line 27 of /login/index.php: call to require()
Our Mcrouter config is very simple - all Mcrouters have the same config, defining all Memcached servers as a pool, and so consistent hashing is in effect across the memcached servers. Meaning that a session which is stored on a particular memcached server should be accessible from any of the application containers (i think?) - but we keep getting the occasional miss.
I've learned quite a bit more than I intended to about memcached and hashing algorithms - we 'just'(!) want a scalable, redundant Moodle running in a cluster but it seems a little more difficult than anticipated.
A couple of questions:
- Has anybody had experience using Mcrouter in a cluster for moodle? Within Kubernetes even?
- Is this the right way to go about session caching with Memcached within a cluster, taking redundancy and scalability into account?
Please help me find a way out of this rabbit hole!