The new environment has a pair of webserver VMs, fronted by a load balancer running HA Proxy. We also have a single PostgreSQL database server, which also runs a copy of Redis. Finally, moodledata is shared via NFS from a dedicated fileserver VM.
The system performs quite well for the most part, with the exception of long login times (10-12 seconds). We generally authenticate via LDAP, however I've confirmed that the login times are equally long for local, manual accounts on the server.
Sessions are shared via Redis. I've also tried a shared memcached server, but the long delays persist. I've confirmed that Moodle is talking to Redis and storing/retrieving session records there. As we're running NFS, I also disabled file locking and have tried both db and redis lock factories (https://github.com/blackboard-open-source/moodle-local_redislock) with no perceptible effect on login times.
I would appreciate any tips/pointers on where I could look next. I suspect NFS might be the issue, although we don't do anything special when it comes to dashboards or login events that might trigger a lot of disk thrashing. I'm wary of using Gluster, and Ceph is probably overkill for this environment.
Here's a copy of our config.php, with the identifying bits stripped out:
<?php // Moodle configuration file
$CFG = new stdClass();
$CFG->dbtype = 'pgsql';
$CFG->dblibrary = 'native';
$CFG->dbhost = 'xxx.xxx.xxx.xxx';
$CFG->dbname = 'moodle';
$CFG->dbuser = 'moodle';
$CFG->dbpass = 'XXXXXXXXXXXXXXXXX';
$CFG->prefix = 'mdl_';
$CFG->dboptions = array (
'dbpersist' => 0,
'dbport' => 5432,
'dbsocket' => '',
$CFG->wwwroot = 'https://servername';
$CFG->dataroot = '/data/shares/moodledata';
$CFG->admin = 'admin';
$CFG->sslproxy = 1;
$CFG->tempdir = '/data/shares/moodledata/temp';
$CFG->cachedir = '/data/shares/moodledata/cache';
$CFG->localcachedir = '/data/shares/local-cache'; // yes, it says "shares" but this is a local filesystem
// Some filesystems such as NFS may not support file locking operations.
// Locking resolves race conditions and is strongly recommended for production servers.
// $CFG->preventfilelocking = false;
$CFG->preventfilelocking = true;
$CFG->local_redislock_redis_server = "xxx.xxx.xxx.xxx";
$CFG->lock_factory = '\\local_redislock\\lock\\redis_lock_factory';
// $CFG->lock_factory = "\core\lock\db_record_lock_factory";
// Redis session handler (requires redis server and redis extension):
$CFG->session_handler_class = '\core\session\redis';
$CFG->session_redis_host = 'xxx.xxx.xxx.xxx';
$CFG->session_redis_port = 6379; // Optional.
$CFG->session_redis_database = 0; // Optional, default is db 0.
// $CFG->session_redis_auth = ''; // Optional, default is don't set one. (disabled auth for testing)
$CFG->session_redis_prefix = 'mdls_'; // Optional, default is don't set one.
$CFG->session_redis_acquire_lock_timeout = 120;
$CFG->session_redis_lock_expire = 7200;
$CFG->session_redis_serializer_use_igbinary = true; // Optional, default is PHP builtin serializer.
//$CFG->session_handler_class = '\core\session\memcached';
//$CFG->session_memcached_save_path = 'xxx.xxx.xxx.xxx:11212'; // note: 11212 used for sessions, 11211 used for app cache
//$CFG->session_memcached_prefix = 'mdls_';
//$CFG->session_memcached_acquire_lock_timeout = 120;
//$CFG->session_memcached_lock_expire = 7200; // Ignored if memcached extension <= 2.1.0
require_once(dirname(__FILE__) . '/lib/setup.php');
// There is no php closing tag in this file,
// it is intentional because it prevents trailing whitespace problems!