Moodle randomly logging users out on AWS

Moodle randomly logging users out on AWS

by Wathsala Werapitiya -
Number of replies: 10

I work for Kaplan International and several of our Moodle instances are randomly logging users out. This mainly seems to happen when there is a high amount of load on the servers.

We are currently running Moodle 2.6.3 and we use the Memcached Session Handler.

We've are not quite sure what's causing this to happen. However, the problem doesn't seem to happen when we use the Database Session Handler. Leading us to believe that this is something to do with Moodle's Memcached Session Handler.

We started using Memcached on an ElasticCache (single node) instance. But, when we started noticing this issue we installed Memcached locally on the Web servers and used Sticky sessions to keep the users on a single instance. This has reduced the frequency of the issue, but, it continues to affect users.

All our instances are hosted on AWS. The environment details are as follows:

Moodle: v2.6.3
Session Handler: Memcached

PHP: v5.3.28
PHP5-Memcached: v2.1
Memcached: v1.4.13

EC2 instances (web server): m3.xlarge x 4 (load balanced)
RDS (DB Server): m3.2xlarge - MySQL 5.7
OS: Amazon Linux

Has anyone else come across this issue? Please let me know if you need any more details?

Average of ratings: -
In reply to Wathsala Werapitiya

Re: Moodle randomly logging users out on AWS

by Michael Aherne -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Wathsala, have you checked that memcached isn't filling up the available memory and starting to evict data for sessions that are still in use? That would give the behaviour you're seeing. If that's the case you would have to reconfigure memcached to give it more memory (how much will depend on the number of users and the size of their session data). Unless you're running a pretty small Moodle instance the memcached default of 64Mb probably won't be enough.

If you're not already using it, it would be worth installing phpMemcachedAdmin  (https://code.google.com/p/phpmemcacheadmin/), as that gives a clear pie chart of the memory use and loads of other useful statistics.

Cheers, Michael.

Average of ratings: Useful (2)
In reply to Michael Aherne

Re: Moodle randomly logging users out on AWS

by Wathsala Werapitiya -

Hi Michael, Thanks for the advice.

I don't think it's an issue with the amount of available memory. The Memcached node has more than 10GB memory. 

I'll have a look at phpMemcacheAdmin. It looks like a good tool.

Best, Wathsala

In reply to Wathsala Werapitiya

Re: Moodle randomly logging users out on AWS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm sure there's a tracker ticker for this somewhere (which I can't find). Basically, anything that invalidates the cache also trashes the session data, hence logging out your users. 


I'll keep looking

In reply to Howard Miller

Re: Moodle randomly logging users out on AWS

by Colin Campbell -

There are a couple of related tracker tickets.  Here's one: MDL-45375

Also, although the session issue is a bigger problem, keep in mind that, if your application caches are in a single memcached instance, a purge of one will purge them all.  For example, the course contacts cache purges about one an hour (when used).  If that also purges your modinfo cache, you could get spikes of very poor performance every hour.

In reply to Colin Campbell

Re: Moodle randomly logging users out on AWS

by Wathsala Werapitiya -

Thanks Colin! This may be the cause of the problem. And, Yes, we do keep the Session store and the MUC on the same Memcached instance.

In reply to Wathsala Werapitiya

Re: Moodle randomly logging users out on AWS

by Adam Durana -

Hi Wathsala --

I don't know about ElastiCache, but Memcached has a default object size limit of 1MB.  This means if a session in serialized form is larger than 1MB, it will be truncated and corrupted.  A corrupted session that fails to deserialize could result in users being logged out.

I've seen users logged out because because of the 1MB limit, although not recently.  You may want to look through your logs to see if you can find any indication of problems deserializing sessions and also keep an eye on the size of sessions to see if they are getting close to the 1MB mark.

--Adam

Average of ratings: Useful (1)
In reply to Adam Durana

Re: Moodle randomly logging users out on AWS

by Wathsala Werapitiya -

Thanks for the reply. I think our issue might have been caused because we were using the same Memcached instance for sessions and MUC.

However, I think it's worth looking into. 

Best,

Wathsala

Average of ratings: Useful (1)
In reply to Wathsala Werapitiya

Re: Moodle randomly logging users out on AWS

by James McLean -

Yes, speaking from experience this is a major issue. It happened to us; duplicating the Memcached service and running a second instance on another port for Sessions only resolved that problem.

In reply to Adam Durana

Re: Moodle randomly logging users out on AWS

by James McLean -

Thanks for mentioning that Adam; we've recently had random logout issues on Memcached and this may explain it. We're going to investigate further today.