Moodle on cloud with Redis Cache: Unable to obtain session lock

Moodle on cloud with Redis Cache: Unable to obtain session lock

by Giuseppe Mandarà -
Number of replies: 7

Hello everyone, in various pages of my moodle, both reserved for the administrator and for public access, I get:
error/Unable to obtain session lock

My setup is as follows:
Moodle 3.9.4+ 20210211
PHP 7.2.20
Apache/2.4.6(Server MPM: prefork)
MySQL 5.7.28 (on cluster)
Redis server (for application cache) v=6.0.9
Redis server (for session cache) v=6.2.0
O.S. CentOS Linux release 7.5
shared Network File System for moodle files and moodledata no SSD

(Some configurations, as prefork, are bound by company policies)

I have a large DB (105 GB) and about 60000 users with over 2600 courses
4 virtual machine on cloud behind a V.I.P. and a load balancer, every v.m. with 12 vCPU (Intel Xeon CPU X5650 2.67GHz) and 9 GB of RAM
1 v.m. with Redis for application cache (8 vCPU (Intel Xeon CPU X5650 2.67GHz) and 8 GB of RAM)
1 v.m. with Redis for session cache (8 vCPU (Intel Xeon CPU X5650 2.67GHz) and 8 GB of RAM)
1 v.m. only for crontab every minutes (12 vCPU (Intel Xeon CPU X5650 2.67GHz) and 9 GB of RAM)
1 v.m. for MySQL DB as a Service (4 Cpu (3.2 GHz), 8 Gb ram)

Some examples on log files:
Cannot obtain session lock for sid: cphrrvj5kvc5ou11ueptoabi8p within 120 seconds. It is likely another page ([pid 4708] /course/view.php?id=123794) has a long session lock, or the session lock was never released.
Cannot obtain session lock for sid: jugh67eg4591fe4l87ngnptf2l within 120 seconds. It is likely another page ([pid 9690] /mod/lesson/view.php?id=92443) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/course/view.php?id=123456
Cannot obtain session lock for sid: d6ekd6a8mbkmj7mgpni5aq12uk, referer: https://xxxxxxxx/mod/quiz/review.php?attempt=1199680&cmid=101507
Cannot obtain session lock for sid: e7ijkt4dthsmclf4mkaromncj7 within 120 seconds. It is likely another page ([pid 1544] /admin/category.php?category=users) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/
Cannot obtain session lock for sid: 7cl4kh6e9dpmoniro6peign6oc, referer: https://xxxxxxxx/mod/scorm/player.php
Cannot obtain session lock for sid: 0ab2kt5vk0h7tr3ns296cg2ngc within 120 seconds. It is likely another page ([pid 30065] /course/index.php?categoryid=123) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/course/index.php?categoryid=123
Cannot obtain session lock for sid: lnb5hfjfd8affontfk8n9300oj within 120 seconds. It is likely another page ([pid 20917] /admin/category.php?category=modules) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/
Cannot obtain session lock for sid: emv3hrqa0b5grtu1m64jfu3ffa within 120 seconds. It is likely another page ([pid 6231] /course/view.php?id=123123) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/login/index.php
Cannot obtain session lock for sid: 9oq3tadrb5lmgp2jj709suun24 within 120 seconds. It is likely another page ([pid 11245] /report/progress/index.php?course=123321&silast=R) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/course/index.php?categoryid=123
Cannot obtain session lock for sid: eg549ic50eikmc2di4l52c2g7i within 120 seconds. It is likely another page ([pid 14670] /mod/customcert/my_certificates.php?userid=xxx&certificateid=xxx&downloadcert=1) has a long session lock, or the session lock was never released., referer: https://xxxxxxxx/mod/customcert/my_certificates.php?userid=xxx


Our config.php is as follows:
$CFG->enable_read_only_sessions = true;
$CFG->preventfilelocking = true;
$CFG->session_handler_class = '\core\session\redis';
$CFG->session_redis_host = 'xx.xxx.xx.x';
$CFG->session_redis_port = 6379;  // Optional.
$CFG->session_redis_database = 0;  // Optional, default is db 0.
$CFG->session_redis_auth = 'xyz'; // Optional, default is don't set one.
$CFG->session_redis_prefix = ''; // Optional, default is don't set one.
$CFG->session_redis_acquire_lock_timeout = 120;
$CFG->session_redis_lock_expire = 7200;
$CFG->session_redis_lock_retry = 10;
$CFG->session_redis_acquire_lock_retry = 10;

$CFG->local_redislock_redis_server = 'xx.xxx.xx.x';
$CFG->lock_factory = '\\local_redislock\\lock\\redis_lock_factory';
$CFG->local_redislock_auth = '';


On /etc/httpd/conf.d/php.conf
  php_value session.save_handler "redis"
  php_value session.save_path    "tcp://xx.xxx.xx.x:6379"
 
On /etc/php.ini
  session.save_handler = redis
  session.save_path = "tcp://xx.xxx.xx.x:6379"
 
On /etc/httpd/config_vhosts/www.xxxxxxxx_443/php.conf
  php_admin_value session.save_path tcp://xx.xxx.xx.x:6379

On /etc/redis.conf
  'appendonly yes'
  'appendfsync everysec'

Anyone have any ideas or suggestions to overcome the problem? Thanks in advance, have a nice day.

Giuseppe

Average of ratings: Useful (1)
In reply to Giuseppe Mandarà

Re: Moodle on cloud with Redis Cache: Unable to obtain session lock

by Jess Shannon -
I've just had this same error, I was using Redis on AWS ElastiCache and had set it up with clustering mode on. The Redis handler in moodle didn't support being redirected to the clustered node so always failed to store the session, and it gave that error.

Try using the redis-cli and setting a key, that's how I realised it was clustering.
Average of ratings: -
In reply to Jess Shannon

Re: Moodle on cloud with Redis Cache: Unable to obtain session lock

by TRUONG QUOC -
Hi Jess Shannon,

Can you talk in more detail about it. I also install a dedicated Redis server for moodle cluster with multiple moodle apps. I got this error also when try to open a new tab, when another task is running?
Average of ratings: -
In reply to TRUONG QUOC

Re: Moodle on cloud with Redis Cache: Unable to obtain session lock

by Jess Shannon -
If you're running a dedicated redis server then I'm guessing you're not using clustering? So your issue is probably different. I'd check that the access key you're using from moodle is able to write to the redis cache.
Average of ratings: -
In reply to Giuseppe Mandarà

Re: Moodle on cloud with Redis Cache: Unable to obtain session lock

by Alain Raap -
Giuseppe, do you still have these problems with Redis? I saw this tracker https://tracker.moodle.org/browse/MDL-68577?attachmentOrder=desc that also explains this setting for Redis $CFG->session_redis_acquire_lock_retry. Although I see in the code of lib/classes/session/redis.php that this field is already set with 100 as a default value:

    /** @var int $lockretry how long to wait between session lock attempts in ms */
    protected $lockretry = 100;


We ran into the same problem (and with nearly the same Linux stack), so I'm curious if you already found a solution. We're still searching for a solution to solve the session lock problem.

I just found this tracker, maybe this could explain the timeouts?

https://tracker.moodle.org/browse/MDL-72796

Average of ratings: Useful (2)
In reply to Giuseppe Mandarà

Re: Moodle on cloud with Redis Cache: Unable to obtain session lock

by Alain Raap -
I found these settings of Redis in my phpinfo page:

redis.session.lock_expire 0 0
redis.session.lock_retries 10 10
redis.session.lock_wait_time 2000 2000
redis.session.locking_enabled 0 0

These are (default) values of the phpredis extension, how should they be configured for Moodle (besides
the settings in config.php of Moodle)?
Average of ratings: Useful (1)
In reply to Alain Raap

Re: Moodle on cloud with Redis Cache: Unable to obtain session lock

by Alain Raap -
Our problem was solved after a network change, we don't see any session lock timeouts on the server anymore. Probably there was a bottleneck in the routing of the requests that caused these sesion lock timeouts. We couldn't find the cause, maybe a tip to keep in mind that network routing (in large organizations) might cause these kind of errors.
Average of ratings: Useful (1)