Cache performens test- over 800 = error 504

Cache performens test- over 800 = error 504

by Michael Jensen -
Number of replies: 8

Hey Moodlers

im finding my self in a corner. First this might be of my lack of knowhow, eventhougt i been running moodle for several years.

I have been searching the net for a long time trying to find a scalable solution in the future.

I came up with 3 VM's on ESXI (Ubuntu 20.04)

Serv 1: Moodle
Running with Nginx php-fpm 7.4
Moodledata is NFS4 linked to Serv 2
Redis (serv 2)

Serv 2: Fileserver & Redisserver

Serv 3: Mariadb v. 10.6

Im currently testing it, and came by Cache performance test (cache/testperformance.php)
And here is where i cant seem to get it done. It is generally slow when testing, all over 800 give me the 504 timeout screen. PHP-FPM

If I try on my Serv 4 (Ubuntu 16.04). currently running live Moodle with students i can do the max test without problems, it a plain install using mysql and without any extra like Redis etc.
So things in mind Serv 4 is using Apache2, could this really be it?

I have been following the guides about setting up both mariadb, redis, nginx etc. from moodle guides.

Im a bit stuck an before Im going to rearrange, I hope one of you are sitting on a solution, or a place to look for potential problem. I have been testing with the mysqltuner no "big" issues as far I can tell.

Any help would be appreciated.

Best regards
Mike




Average of ratings: -
In reply to Michael Jensen

Re: Cache performens test- over 800 = error 504

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I know it's a basic question but is your timeout in nginx (for php-fpm) long enough? My fastcgi_read_timeout is set to something like 2 hours because some processes in Moodle do take a long time.

I would also check that you have Redis and your MUC all configured properly. I didn't think that the cache performance tests supported testing of Redis? I-ve just checked - you can't test Redis, so what are you testing?
Average of ratings: Useful (1)
In reply to Howard Miller

Ang: Re: Cache performens test- over 800 = error 504

by Michael Jensen -
Hi Howard

well my Redis is "showing of" when i do test. The problem as far as I can tell is the File cache who goes as far as 24.3452 in SET when i hit test for 500. after 800 it gives me 504. Compared to the 2. server it takes 0.1641.
Fastcgi_read_timeout is se,t however you got me rethinking and i actually found out there was a mis-configuration within nginx conf.d file and in sites-available.

Memecahed is not installed but in mapping, ive choose Redis for the two top options. I think It must be something to do with "file cache" its just so slow i got the 504 problem figured out thx to your answer, however at 1000 im hitting 33.4325 SET the other server get in in 0.2456






Average of ratings: -
In reply to Michael Jensen

Re: Ang: Re: Cache performens test- over 800 = error 504

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
You should set Redis as the default cache and point everything there that supports it. You're right - NFS is just too slow to be used for the primary cache. That's the big reason for deploying Redis in the first place.

Once you have Redis working properly, the performance of the file cache is irrelevant. We already know it's going to be slow.
Average of ratings: -
In reply to Howard Miller

Ang: Re: Ang: Re: Cache performens test- over 800 = error 504

by Michael Jensen -
thx for your reply. I have been struggling with the php7.4-redis to work using https://www.digitalocean.com/community/tutorials/how-to-set-up-a-redis-server-as-a-session-handler-for-php-on-ubuntu-14-04
Although its a 20.04 ubuntu I found several with same info- somehow the system wouldn't show the test:
Output
1) "PHPREDIS_SESSION:j9rsgtde6st2rqb6lu5u6f4h83" it only shows the last "j9rsgtde6st2rqb6lu5u6f4h83" and I think this might be wrong because as stated it should be "PHPREDIS_SESSION"? anyhow can you confirm that the php-redis should be setup with the session.save_path = "tcp://10.133.14.9:6379?auth=yourverycomplexpasswordhere" etc.?
Average of ratings: -
In reply to Michael Jensen

Re: Ang: Re: Ang: Re: Cache performens test- over 800 = error 504

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I'm not sure what you're trying to do. If you want Redis to handle your sessions, the setup is in Moodle's config.php. Moodle handles its own sessions. Don't be confused with 'session cache' which is something different entirely. See config-dist.php for the correct info...


//   Redis session handler (requires redis server and redis extension):
//      $CFG->session_handler_class = '\core\session\redis';
//      $CFG->session_redis_host = '127.0.0.1';
//      $CFG->session_redis_port = 6379;  // Optional.
//      $CFG->session_redis_database = 0;  // Optional, default is db 0.
//      $CFG->session_redis_auth = ''; // Optional, default is don't set one.
//      $CFG->session_redis_prefix = ''; // Optional, default is don't set one.
//      $CFG->session_redis_acquire_lock_timeout = 120;
//      $CFG->session_redis_lock_expire = 7200;
//      $CFG->session_redis_lock_retry = 100; // Optional wait between lock attempts in ms, default is 100.
//                                            // After 5 seconds it will throttle down to once per second.
//      Use the igbinary serializer instead of the php default one. Note that phpredis must be compiled with
//      igbinary support to make the setting to work. Also, if you change the serializer you have to flush the database!
//      $CFG->session_redis_serializer_use_igbinary = false; // Optional, default is PHP builtin serializer.
//      $CFG->session_redis_compressor = 'none'; // Optional, possible values are:
//                                               // 'gzip' - PHP GZip compression
//                                               // 'zstd' - PHP Zstandard compression
Average of ratings: Useful (1)
In reply to Howard Miller

Ang: Re: Ang: Re: Ang: Re: Cache performens test- over 800 = error 504

by Michael Jensen -
Okay thx Howard.

Then i had everything running (except for the igbinary serializer) as it should, before looking at Redis as the session handler for php.
Still it took forever when testing, but this wouldn't do anything about that? as you wrote we already know nfs will be slow?

When I do performance test it will always be slow in file cache, when nfs is enable? or should Redis do something about this? because I already had everything working, even though file cache got a very high SET-time.
I was just afraid to start production if it meant the system couldn't handle the load.

Also last Question, would you do the cluster like us with Moodleserver - Fileserver/redis - DBserver or a Cluser of DBgalera with NFS Marster/Slave (1Masterserver, 1-2 Slaves)

Once more i really appreciate your time and feedback.

/Mike
Average of ratings: -
In reply to Michael Jensen

Re: Ang: Re: Ang: Re: Ang: Re: Cache performens test- over 800 = error 504

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
The file cache will always be slow if it's NFS. But it doesn't really matter because it's no longer used for anything particularly performance critical (if you've moved everything you can to Redis).

You can't test Redis performance from Moodle. On a large system, you should monitor everything you can but do watch network traffic if you can. Redis results in a LOT of network activity - make sure your network is good enough.

I've no experience of Galera or NFS master/slave so I can't really comment.
Average of ratings: Useful (2)
In reply to Howard Miller

Ang: Re: Ang: Re: Ang: Re: Ang: Re: Cache performens test- over 800 = error 504

by Michael Jensen -
Thank you, it makes sense!
Have a nice day and Christmas when time is smile
Average of ratings: -