Moodle in English: Clustered Moodle, bad performance with NFS and redis

We need someone help with experience with clustered Moodle to help us optimize current resources as it seems that we are currently hitting some unknown bottleneck.

For now it looks like that most of the troubles are caused at NFS.

It looks like request are waiting for file locks at shared /moodledata/sessions folder even if redis is set as session storage. Is this normal behaviour, that session file is even created when in config.php is set:

$CFG->session_handler_class = '\core\session\redis';

I have to mention that we are exclusively using Shibboleth SP for authentication, where every web node has its shibd instance and we use session Cookie-Based Session Recovery (https://wiki.shibboleth.net/confluence/display/SP3/Clustering) .

Some information from our clustered instance:

Moodle DB size: 92G

Moodledata size: 800G

Loadbalancing: BIG-IP F5

OS on all machines: CentOS 7.7.1908

DB host:

hardware:
2xCPU (10 cores),
256GB memory,
2x800GB NVMe
sql server: mariadb 10.0.38
redis server: 3.2

Webnodes:

hardware:
2xCPU(10 cores)
128GB memory
2x200GB SSD
Apache with mod_ssl
PHP version: 7.1.33
Apache: 2.4.6
Caching: Redis
Shibboleth 3.04

NFS host (for sharing Moodledata between webnodes):

2xCPU (10 cores)
256GB memory
12TB RAID6 SSD pool for storage

We can provide more technical details and informations from our monitoring systems and traces if needed.

Average of ratings: Useful (1)

Re: Clustered Moodle, bad performance with NFS and redis

by Matteo Scaramuccia - Wednesday, 18 March 2020, 5:56 AM

Hi Pavel,
the locks come from the "file locking" for MUC and probably from cron task - more about Moodle lock framework in https://docs.moodle.org/dev/Lock_API and https://docs.moodle.org/38/en/Caching#Locking.

What is the Moodle version you're running on that HW?

MDL-67052 (3.9+) offers a lock factory native for MySQL / MariaDB: you could easily backport it and try your backport into your test environment.
Otherwise, you have a "generic" DB-based lock: read more here.

Or, since you're already using Redis, give https://github.com/blackboard-open-source/moodle-local_redislock, a 3rd party plug-in, a try.

HTH,
Matteo

Average of ratings: Useful (4)

Re: Clustered Moodle, bad performance with NFS and redis

by Howard Miller - Wednesday, 18 March 2020, 3:03 PM

Been there...

My general advice is to avoid using NFS as much as possible for anything performance critical. Your simplest solution may be to move the sessions to Redis (the configuration is in config.php) and the locking to the database (config.php again - the table already exists). We have this running in 3.7 - there are improvements in 3.9 but they may not be critical to you.

Average of ratings: Useful (3)

Odg: Re: Clustered Moodle, bad performance with NFS and redis

by Martin Božič - Wednesday, 18 March 2020, 4:22 PM

Hi, I'm Pavel's teammate. Here is our current config.php for the setup - as you can see we are already using Redis for sessions:

dbtype = 'mariadb'; $CFG->dblibrary = 'native'; $CFG->dbhost = 'db.foo.bar.org'; $CFG->dbname = '*****'; $CFG->dbuser = '******'; $CFG->dbpass = '********'; $CFG->prefix = 'mdl_'; $CFG->dboptions = array ( 'dbpersist' => false, 'dbport' => '', 'dbsocket' => 'false', 'dbcollation' => 'utf8mb4_unicode_ci', ); $CFG->wwwroot = 'https://foo.bar.org'; $CFG->dataroot = '/srv/moodledata'; $CFG->admin = 'admin'; $CFG->directorypermissions = 0777; require_once(dirname(__FILE__) . '/lib/setup.php'); $CFG->emailconnectionerrorsto = 'foo@bar.org'; $CFG->tempdir = '/srv/moodledata/temp'; // Directory MUST BE SHARED by all cluster nodes. $CFG->cachedir = '/srv/moodledata/cache'; // Directory MUST BE SHARED by all cluster nodes, locking required. $CFG->localcachedir = '/srv/localcache'; // Intended for local node caching. $CFG->preventfilelocking = true; $CFG->lock_factory = "\core\lock\db_record_lock_factory"; $CFG->session_handler_class = '\core\session\redis'; $CFG->session_redis_host = 'redis.foo.bar.org'; $CFG->session_redis_port = 6379; // Optional. $CFG->session_redis_database = 0; // Optional, default is db 0. $CFG->session_redis_auth = ''; // Optional, default is don't set one. $CFG->session_redis_prefix = ''; // Optional, default is don't set one. $CFG->session_redis_acquire_lock_timeout = 120; $CFG->session_redis_lock_expire = 7200; define('CONTEXT_CACHE_MAX_SIZE', 7500);

But it seems that PHP session files in moodledata/sessions are still being used by Apache. We use Moodle version 3.7.4 (build 20200124)

Average of ratings: -

Odg: Re: Clustered Moodle, bad performance with NFS and redis

by Pavel Šipoš - Wednesday, 18 March 2020, 4:56 PM

I have to add few things:

After first post we have added "$CFG->preventfilelocking = true;" to config.php.
Thats seems to improve performance on NFS, but we don't know what negative consequences this can bring.

Also:
It looks like that reqests hang at core\session\handler::start .

Do you maybe have idea why is taking so long:

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matteo Scaramuccia - Wednesday, 18 March 2020, 10:11 PM

Hi Martin and Pavel,

But it seems that PHP session files in moodledata/sessions are still being used by Apache.

that's strange, almost a bug.

BTW $CFG->preventfilelocking = true; automatically set DB locking as the default locking system: the downside could be some sort of race condition in the Moodle Files API when saving at the same time the same file i.e. the same content regardless its filename.

The time required to handle a new PHP session is due to the DB locking.
Please try to use a backport for the MySQL / MariaDB native locking system or the 3rd party Redis locking system - $CFG->MR_SHORT_NAME is the way to prepend the "name" of your Moodle instance, when using the Redis option in a shared Moodle environment.

HTH,
Matteo

Average of ratings: Useful (1)

Odg: Re: Clustered Moodle, bad performance with NFS and redis

by Pavel Šipoš - Wednesday, 18 March 2020, 10:42 PM

So, we have one Moodle instance served by multiple webservers.
If I understand correctly, does that mean that we have to set $CFG->MR_SHORT_NAME on all webservers equally or is it meant that we dedicate name for each one of them?

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matteo Scaramuccia - Wednesday, 18 March 2020, 11:27 PM

Hi Pavel,
$CFG->MR_SHORT_NAME is used to define the prefix of the Redis locking system instance and should be shared among all the web servers belonging to the same Moodle instance.

In your case, you can ignore it and the database name will be used as the prefix: just keep care if the DB name is the same among all your environments i.e. not a unique prefix key if your Redis instance is shared among those environments.

HTH,
Matteo

Average of ratings: Useful (2)

Odg: Re: Clustered Moodle, bad performance with NFS and redis

by Pavel Šipoš - Thursday, 19 March 2020, 6:25 PM

We finally figured out why session files were created even if we had set readis as a session storage.
It was very straightfoward mistake in our configuration which cost us days of debugging to find it.

require_once call wasn't at the end of Moodle config file, as you can see in Matin's post above:

require_once(dirname(__FILE__) . '/lib/setup.php');

After fixing that, everything started to work with a blasting speed, and sessins were created at redis storage as they should.

Thank you very much Matteo and Howard for your help and for all advices you gave us.

Average of ratings: Useful (1)

Re: Clustered Moodle, bad performance with NFS and redis

by Matteo Scaramuccia - Thursday, 19 March 2020, 6:49 PM

Hi Pavel,
thanks for sharing your issue: frankly speaking, me too! I didn't notice it, just focusing on Redis related CFGs .

Please share any future further experience using Redis for the locking system too - to get rid of locks in NFS which will be now "rare" in your system, compared to the ones at the time of file sessions - , just for a reference here in the Community.

Have a nice day in these difficult times,
Matteo

Average of ratings: -

Ri: Re: Clustered Moodle, bad performance with NFS and redis

by Sergio Rabellino - Thursday, 19 March 2020, 7:42 PM

Even if i didn't had critical performance issues using NFSv4 locking, i'm now migrating my Moodle instances to redislock plugin that i believe stronger and simpler to manage than filesystem or db locking, particularly important in this epoch of elearning instant growth caused by COVID19 emergency.
Thanks to Matteo for pointing out this opportunity of performance hint.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Friday, 20 March 2020, 1:28 AM

Hey all,

with moving sessions to redis (via session_handler_class) the performance is now really good. Session locks now take between 80ms-110ms and load on NFS is practically zero. Initialize_user_session still uses DB to store the session, but inserting, selecting and updating sessions in DB is super fast (below 2ms).

Our average response time is now around 370ms and full browser render at around 1,6s.

We are planning to switch to igbinary php serializer which should bring some performance gain and lower the amount of traffic from redis which peaks at 50MB/s at the moment. We might also scrape some ms from response time.

I also attached a trace from /course/index.php to get a feeling of how it looks now.

Regarding the DB / Redis locking, I was looking at our locking table activity and it seems like we don't have much locks happening, below 20 every few minutes. So we probably won't see much gain with using redis locking for now. But that might change now that more users will migrate to Moodle given the increased performance. Will keep a close eye on the table.

Attachment Screenshot 2020-03-19 at 18.20.58.png

Average of ratings: Useful (2)

Re: Clustered Moodle, bad performance with NFS and redis

by António Godinho - Saturday, 18 April 2020, 10:32 PM

Can you please post your config right now, so I can use it and compare it with my solution?

Thanks

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Sunday, 19 April 2020, 6:57 AM

Sure!

<?php // Moodle configuration file

unset($CFG);

global $CFG;

$CFG = new stdClass();

$CFG->dbtype = 'mariadb';

$CFG->dblibrary = 'native';

$CFG->dbhost = 'xxxxxx';

$CFG->dbname = 'xxxxxx';

$CFG->dbuser = 'xxxxxx';

$CFG->dbpass = 'xxxxxx';

$CFG->prefix = 'randomprefix_';

$CFG->dboptions = array (

'dbpersist' => false,

'dbport' => '',

'dbsocket' => 'false',

'dbcollation' => 'utf8mb4_unicode_ci',

);

$CFG->wwwroot = 'https://host.example.com';

$CFG->dataroot = '/www/moodle';

$CFG->admin = 'admin';

$CFG->directorypermissions = 0777;

$CFG->emailconnectionerrorsto = 'admin@moodle';

$CFG->tempdir = '/www/moodledata/temp'; // Directory MUST BE SHARED by all cluster nodes.

$CFG->cachedir = '/www/moodledata/cache'; // Directory MUST BE SHARED by all cluster nodes, locking required.

$CFG->localcachedir = '/www/localcache'; // Intended for local node caching.

// Some filesystems such as NFS may not support file locking operations.

// Locking resolves race conditions and is strongly recommended for production servers.

// $CFG->preventfilelocking = false;

$CFG->preventfilelocking = true;

$CFG->lock_factory = "\core\lock\db_record_lock_factory";

// Redis session handler (requires redis server and redis extension):

$CFG->session_handler_class = '\core\session\redis';

$CFG->session_redis_host = 'xxxxxx';

$CFG->session_redis_port = 6379; // Optional.

$CFG->session_redis_database = 0; // Optional, default is db 0.

$CFG->session_redis_auth = ''; // Optional, default is don't set one.

$CFG->session_redis_prefix = ''; // Optional, default is don't set one.

$CFG->session_redis_acquire_lock_timeout = 120;

$CFG->session_redis_lock_expire = 7200;

$CFG->session_redis_serializer_use_igbinary = true; // Optional, default is PHP builtin serializer.

// There is no php closing tag in this file,

// it is intentional because it prevents trailing whitespace problems!

// Moodle 2.3: Increasing this from the default saved about > 1000 db queries

// on the course/index.php page for a Moodle having 1250 course categories.

// This value is specified in lib/accesslib.php, but it's OK to add a define for it in config.php:

define('CONTEXT_CACHE_MAX_SIZE', 7500);

require_once(dirname(__FILE__) . '/lib/setup.php');

Average of ratings: Useful (4)

Re: Clustered Moodle, bad performance with NFS and redis

by Alain Tchana - Wednesday, 22 April 2020, 5:53 AM

Hi Matej Zerovnik.

Thanks for sharing your config. Please I would like to know if you could share the architecture of your plateforme, meaning:

-Number of web nodes

-Number of redis nodes

-Nomber of database nodes

For each tier, the machine setup in terms of CPU and RAM size.

Also how much concurrent users you think your plateform can support.

Best

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Wednesday, 22 April 2020, 6:39 AM

At the moment, we are running with:
- 3x web nodes (40 cores, 128GB memory, SSD drives)
- 1x db/redis node (40 cores, 128GB memory, NVMe drives for storage)
- 1x NFS node (40 cores, 64GB memory, SSD drives for storage)

Hardware is too powerful for our use case, but we borrowed it from another project at the beginning of COVID.

We are peaking at around 9k concurrent users (concurrent users for us are users active in last 5min) with web nodes running at around 15% CPU and DB at around 25% (with query cache turned off), so we have some room to grow.

But there is still a lot of space for better performance or more concurrent users:
- move away from prefork to php-fpm with event workers or nginx, so we can use opcache
- web server caching for certain pages (like index.php, so lightweight DDoS attack doesn't exhaust your server resources)
- other things we will likely discover as more users got onboard

Average of ratings: Useful (6)

Re: Clustered Moodle, bad performance with NFS and redis

by Alain Tchana - Wednesday, 22 April 2020, 10:46 PM

Thanks a lot for these information.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Camilo Ravelo Durán - Friday, 24 April 2020, 8:40 AM

Thanks for sharing your information. A question. How much has the Redis server grown? I am configuring it, but I don't know how much hard disk to allocate or how much it will upload.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Friday, 24 April 2020, 7:30 PM

You can limit memory usage for Redis and configure eviction policy.

We have our redis limited to 8GB of memory (remember, redis is memory only DB), but we are also storing some data from other services, so I can't say exactly how much Moodle uses. It depends on your moodle size, so start with 1-2GB and see. You can always extend if you need without downtime (if you have free memory). You need at least 2x amount of disk space for snapshots of redis DB or more if you use AOC(??).

We see around 100MBytes/s of traffic from Redis (1Gbps) to web frontends during peak hours. igbinary helped to reduce this number, otherwise it would be even more.

Also remember, that redis is a single thread app, so if you hit 100% usage, you need faster CPU not more cores. We are seeing around 15-25% CPU usage.

Average of ratings: Useful (3)

Re: Clustered Moodle, bad performance with NFS and redis

by António Godinho - Sunday, 26 April 2020, 10:08 PM

So you have:

maxmemory 8gb
maxmemory-policy allkeys-lru

I have peaks of 500 users, you say that you have 9k. Without making any crazy math, I'll guess that 500mb could be a start for my scenario.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Monday, 27 April 2020, 4:46 AM

More than enough.

I would advise to setup some kind of monitoring and watch metrics. We are using Prometheus with Grafana, but use whatever you use for other infrastructure monitoring. You will then see if 500mb is enough or not. My guess it that it will be plenty.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Alain Raap - Tuesday, 28 April 2020, 8:44 PM

@Matej Zerovnik very interesting to read about your Moodle configuration. Maybe a tip that speeds up Redis, is to use a socket instead of TCP, don't know if that is possible for your configuration. Another tip I got here is to use /dev/shm for storing your temporary cache and localcache Moodle directories. With a clustered Moodle I don't know if it's possible to use this for all your nodes.
How were you able to use igbinary for Redis? How did you configure this?

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Tuesday, 28 April 2020, 10:12 PM

Hey there.

Thanks for the tips. I know using socket is faster, but our Redis is shared amongst multiple web frontend nodes, so we have to operate over TCP. But the speeds are acceptable for us and there are other parts to optimize where we could probably gain some ms as well.

Good idea about localcache. That could indeed go to SHM as it's not shared among other web nodes. But, TBH, I don't think we would see much gain. The folder size is small and we are running with enough free memory that OS uses quite some for fs cache/buffer. We see around 10-20 IOPS on the drives, so my guess is we are serving most of the data from FS cache. That's my guess, I didn't dig into it too much.

As far as igbinary goes, we installed php module (it might be via Composer, I'm not sure), enabled it in Moodle configuration and purged Redis database. Bam

Igbinary

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by akhmad munawar - Tuesday, 28 April 2020, 6:02 PM

Hi Mateo and Matej,

We are using db locks and redis session handler, in apache's error logs we are seeing errors for obtaining locks, such as Cannot obtain session lock for sid: 1gpg2cv4raqc0559php9turhua within 120. It is likely another page has a long session lock, or the session lock was never released.

We use cluster server, but DB server and redis serve is in the same server.

What's wrong with our config?

$CFG->preventfilelocking = true;

$CFG->lock_factory = "\core\lock\db_record_lock_factory";

$CFG->session_handler_class = '\core\session\redis';

$CFG->session_redis_host = '';

$CFG->session_redis_port = 6379; // Optional.

$CFG->session_redis_database = 0; // Optional, default is db 0.

$CFG->session_redis_auth = ''; // Optional, default is don't set one.

$CFG->session_redis_prefix = ''; // Optional, default is don't set one.

$CFG->session_redis_acquire_lock_timeout = 120;

$CFG->session_redis_lock_expire = 7200;

//Use the igbinary serializer instead of the php default one. Note that phpredis must be compiled with

//igbinary support to make the setting to work. Also, if you change the serializer you have to flush the database!

$CFG->session_redis_serializer_use_igbinary = true; // Optional, default is PHP builtin serializer.

$CFG->disablelogintoken = true;

define('CONTEXT_CACHE_MAX_SIZE', 7500);

require_once(__DIR__ . '/lib/setup.php');

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Tuesday, 28 April 2020, 10:15 PM

We are seeing the same same errors as well. There are not many of them, so it's a low priority for now, as we think it's not a showstopper for most of our users.

We might dig deeper at some point.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matteo Scaramuccia - Wednesday, 29 April 2020, 5:27 AM

Hi Akhmad,

We are using db locks and redis session handler, in apache's error logs we are seeing errors for obtaining locks, such as Cannot obtain session lock for sid: 1gpg2cv4raqc0559php9turhua within 120. It is likely another page has a long session lock, or the session lock was never released.

We use cluster server, but DB server and redis serve is in the same server.

What's wrong with our config?

That error comes out when the Redis Session Handler spends more than 120s - $CFG->session_redis_acquire_lock_timeout - to wait for a session lock i.e. there is a "page" which breaks waiting for the time required to acquire the session lock: it could be because of another PHP script which acquires and releases the same session for more time than strictly required by the code (performance bug in that code).
I'd review the number of plug-ins in the current installation to limit them to what actually required and then update each of them to get benefits from their last code version.

Your Redis cluster would be a good candidate for being stressed by the Moodle Session handling as well as as any other component involved in the code executed by the previous "page" locking that session.
An example to improve the performance? When you serve a file from the Moodle Pool Storage usually the session will be unlocked just before sending you the first byte, without the need to keep the lock until the last byte of the file will be served.

I cannot imagine anything specific but for example a timing issue (slowness) when using the Moodle Locking mechanism which could slow down the code running under the same session lock: slowness which could be due to access to any other resource, including CPU bounding code executions.
MDL-67075 will help in finding "the offending page".

Finally, MDL-58018 will help developers in reducing the pressure on the infrastructure elements.

HTH,
Matteo

Average of ratings: Useful (1)

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Friday, 1 May 2020, 6:08 PM

Thank you Matteo for these 2 tickets. We will deploy the first one, hopefully, next week to try and find the offending plugins.

MDL-58018 will also be very nice to have, specially if you have low resource hardware.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by António Godinho - Friday, 1 May 2020, 10:46 PM

Matej: What moodle version are you running?

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Saturday, 2 May 2020, 1:00 AM

We're running 3.7.5+.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Mihai Carabas - Thursday, 7 May 2020, 8:08 PM

Hello Matej,

What is your average load in the application servers (when you hit "w" there are the three average values)

Thank you,
Mihai

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Friday, 8 May 2020, 12:09 AM

It depends on the load, but we get up to 4-5 during busy hours.

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by António Godinho - Thursday, 7 May 2020, 11:01 PM

Now I have issues with the NFS share.
On the server I run a pacemaker cluster with the options: rw,sync,no_root_squash,no_all_squash

On the clients we use fstab:
IP::/moodledata /moodledata nfs _netdev,rw,noatime,sync,rsize=8192,wsize=8192,tcp,timeo=14 0 0

Do you see something wrong here?

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Matej Zerovnik - Friday, 8 May 2020, 12:12 AM

We are running without sync flag on ours.

What kind of issues do you have with NFS? We actually have very little load on the NFS server. We have Moodle code local on each server and only keep moodledata on NFS share.

Do you have any kind of monitoring on the NFS server so you can check what is NFS and server doing? Is it a IO bottleneck on the drives, network bottleneck, systemcalls,...?

Average of ratings: -

Re: Clustered Moodle, bad performance with NFS and redis

by Alain Raap - Friday, 8 May 2020, 12:52 AM

Antonio, maybe this feed can give some help:
https://moodle.org/mod/forum/discuss.php?d=362709

Average of ratings: -