Slow loading/timeouts - session lock

Slow loading/timeouts - session lock

by Michael Ko -
Number of replies: 8

Some help would be appreciated. As I understand it, Moodle has a "session locking" mechanism. We're using Redis on AWS for sessions, EC2 for web servers (behind a load balancer), RDS PostgreSQL, and EFS for moodledata.

We've been dealing with this issue for some years - when a page is slow to load, and hangs for whatever reason (often when we update course modules, and a heap of course recaching/regrading is triggered) - the whole user session gets locked. 

The user is unable to open any other page in a new window/tab, and eventually receives timeouts. The only workaround is creating a new session (incognito window or different browser).

Recently, we've added in more AJAX web service calls (maybe 3-4 on some pages), to make the site appear more dynamic, and improve initial page load speeds. 

Unfortunately, in the event that an AJAX web service call hangs, the user's session is completely stuffed - there is no way for them to load ANY other page... until the "session lock" is eventually released by that request. The requests hang and eventually timeout - despite the servers being under no stress at all.

This session locking mechanism is both hogging server resources, and creating horrible user experiences.

Does anyone know a way around the session locking? We DO need to access the session (e.g. user ID) but we don't need to update it from our web service calls.

Thanks!

Average of ratings: -
In reply to Michael Ko

Re: Slow loading/timeouts - session lock

by Michael Ko -
These are the typical error messages that are filling our PHP logs:

Attachment Screen Shot 2020-04-15 at 1.22.26 pm.png
Average of ratings: -
In reply to Michael Ko

Re: Slow loading/timeouts - session lock

by Michael Ko -
And our session config from config.php:

// Session cache settings
$CFG->session_handler_class = '\core\session\redis';
$CFG->session_redis_host = 'XXXXXX.apse2.cache.amazonaws.com';
$CFG->session_redis_port = 6379; // Optional.
$CFG->session_redis_database = 0; // Optional, default is db 0.
$CFG->session_redis_prefix = ''; // Optional, default is don't set one.
$CFG->session_redis_acquire_lock_timeout = 120;
$CFG->session_redis_lock_expire = 7200;
Average of ratings: -
In reply to Michael Ko

Re: Slow loading/timeouts - session lock

by Md Prawez Musharraf -

Hi Michael,

Did you find any solution on the session lock time out issue with Redis? We are in the same situation as you have mentioned above.


Average of ratings: -
In reply to Michael Ko

Re: Slow loading/timeouts - session lock

by sayooj v v -
Same issue here:
Load balancer with two EC2, Redis server and RDS.


[Wed Nov 18 08:17:44.370742 2020] [php7:notice] [pid 32241] [client 10.0.0.198:34290] Cannot obtain session lock for sid: jkhpf6lsotojbfiv18l4s81chr within 120 seconds. It is likely another page ([pid 31870] ip-10-0-1-230:/mod/bigbluebuttonbn/view.php?id=3327) has a long session lock, or the session lock was never released.,


For us it's always the BBB that is keeping the lock.
Any solution for this?
Average of ratings: -
In reply to Michael Ko

Re: Slow loading/timeouts - session lock

by Bmfrench . -

We are also in the same boat with session locking - did you get anywhere with this?  We are also using redis and elasticache with ec2 servers and experiencing the same problems with locking.

Average of ratings: -
In reply to Bmfrench .

Re: Slow loading/timeouts - session lock

by Susan Mangan -
Not sure if you all figured this out or not but if not, this is what we did to solve the same issue on our servers:

Make sure that $CFG->session_redis_acquire_lock_timeout is the same as the php lock timout (default is 7200 I think)
Average of ratings: Useful (1)
In reply to Susan Mangan

Re: Slow loading/timeouts - session lock

by José Alberto Huerta Gonzalez -

Hello Susan, thanks for your information.

Just a question, I see on my php -i this info:

# php -i | grep timeout

hard_timeout => 2 => 2

Read timeout => 86400

redis.arrays.connecttimeout => 0 => 0

redis.arrays.readtimeout => 0 => 0

redis.clusters.read_timeout => 0 => 0

redis.clusters.timeout => 0 => 0

default_socket_timeout => 120 => 120

opcache.force_restart_timeout => 180 => 180

Which value did you referrer must be the same?

This ?

$CFG->session_redis_acquire_lock_timeout = 120;

or this?

$CFG->session_redis_lock_expire = 7200;

Thanks in advance!

Average of ratings: -
In reply to Michael Ko

Re: Slow loading/timeouts - session lock

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators
Hello,

Session locking is not unique to Moodle. Every PHP web application that uses sessions will have session locks. Moodle just makes this mechanism extendable (ie ability to use Redis). This may help explain it: https://muras.eu/2020/04/22/Session-locking-in-Moodle-part1/

The session locks are probably a cause of some other problem you have, not the source of it. Something does lock the session for a reason - you need to find out what.

Tomek
Average of ratings: Useful (1)