Moodle in English: Unable to run moodle beyond 250 users. (II)

Unable to run moodle beyond 250 users. (II)

by Lien Nell - Monday, August 31, 2020, 5:47 AM

We have the same problem, when more than 250 students try to login simultaneously to take a test. The message is 'Cannot obtain session lock for sid: xxx within 720. It is likely another page has a long session lock, or the session lock was never released.' Database Max Connections is set to 600. We use Redis session manager, $CFG->session_handler_class = '\core\session\redis'; and $CFG->session_redis_acquire_lock_timeout = 720;

I would appreciate some advice where the message comes from, is it a database lock or a Redis lock? As far as I can see the Moodle server or Redis server is not overloaded (<50% of CPU).

I can also add that we experienced the same limitations with $CFG->session_handler_class = '\core\session\database'; in config.php, but if I remember the error message did not mention the long session lock.

Re: Unable to run moodle beyond 250 users. (II)

by Visvanath Ratnaweera - Monday, August 31, 2020, 3:48 PM

Yours is a different problem from the OP. He is getting "The site can't be reached." => ERR_CONNECTION_CLOSED.

You need to describe your platform. Windows or Unix? Which flavour / which distribution? Moodle release? Versions of system software (web server, DBMS, PHP)? Whether a cluster or an all-in-one box?

Average of ratings: -

Re: Unable to run moodle beyond 250 users. (II)

by Lien Nell - Monday, August 31, 2020, 5:18 PM

Thank you for pointing that out. Apologies, I should have started a new thread.

We are running Moodle 3.8+ (Build: 20191212).

The platform is Azure App Service:

4.4.0-128-generic #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018 x86_64 GNU/Linux
7 GB memory
Not clustered, but scaled out to 8 instances at the time of occurrence.

Database is Azure database for MariaDB server:

MariaDB version 10.2

PHP Version 7.3.15

Azure Cache for Redis

Redis version 4.0.14
2.5 GB Cache

Average of ratings: -

Re: Unable to run moodle beyond 250 users. (II)

by Visvanath Ratnaweera - Monday, August 31, 2020, 6:51 PM

No problem. The moderator can split your post creating a new thread. In anticipation I renamed yours "Unable to run moodle beyond 250 users. (II)".

To the subject: You are on a platform which I don't know. Hopefully others can help. The information you provided were needed anyway.

Average of ratings: -

Re: Unable to run moodle beyond 250 users. (II)

by Alex Rowe - Tuesday, September 1, 2020, 6:22 AM

The default redis acquire lock timeout is 120, what was the reasoning for raising it to 720?

The error log you are seeing is when there is another request currently open for that session and it is waiting for that session lock to close before it can get another one. This could be a long running session of starting a quiz, login, report running, course backup etc. Normally it happens if a user is completing a long task in one window/tab then tries to open another tab and keep working. The session is still locked to the first tab until it completes, so the second tab gets the "cannot obtain session lock". Another option instead of window/tabs are ajax requests.

With debugging enabled, is there any other information showed? In the access/error logs, can you determine which script they were trying to access when this happens. Try searching based on IP address in the logs to find out what the user was doing.

Moodle 3.9 has some good improvements in the session locking space as it allows some scripts to use read only sessions which won't hold the session lock. It has to be enabled in the config.php file but it might be worth trying to upgrade.

Do you have the MUC and Sessions in the same redis instance?

Have the app/db/redis servers been configured/tuned at all?

With properly tuned database, redis, php etc, 5-600 concurrent quizzes should be obtainable.

Average of ratings:Useful (2)

Re: Unable to run moodle beyond 250 users. (II)

by Lien Nell - Saturday, September 5, 2020, 12:27 AM

Alex

Thank you for your advice. I have increased the timeout limit : $CFG->session_redis_acquire_lock_timeout = 15000; and it seems to solve the problem. I will confirm as soon as I have carried out a test.

Azure Cache for Redis has a link for Best practices, which recommends the following:

Configure your client library to use a connect timeout of at least 15 seconds, giving the system time to connect even under higher CPU conditions. A small connection timeout value doesn't guarantee that the connection is established in that time frame. If something goes wrong (high client CPU, high server CPU, and so on), then a short connection timeout value will cause the connection attempt to fail. This behavior often makes a bad situation worse. Instead of helping, shorter timeouts aggravate the problem by forcing the system to restart the process of trying to reconnect, which can lead to a connect -> fail -> retry loop. We generally recommend that you leave your connection Timeout at 15 seconds or higher. It's better to let your connection attempt succeed after 15 or 20 seconds than to have it fail quickly only to retry. Such a retry loop can cause your outage to last longer than if you let the system just take longer initially.

Lien

Average of ratings: -

Re: Unable to run moodle beyond 250 users. (II)

by Alex Rowe - Saturday, September 5, 2020, 6:10 AM

The default Moodle acquire lock timeout is 120 seconds (2 minutes) and should work fine in all instances.

Your original setting was 720 seconds (12 minutes) and the new setting is 15000 seconds (over 4 hours). The variable in the Moodle config file is in seconds not milliseconds.

I would be looking at other areas of you system for what is causing the issue as sessions are showing a symptom but it's not the cause.

Average of ratings:Useful (1)

Hardware and performance

Unable to run moodle beyond 250 users. (II)