Help with MUC on cluster enviroment

Help with MUC on cluster enviroment

by Miguel González Laredo -
Number of replies: 8
Picture of Plugin developers

Hi moodlers!

 I've found very interesting thread here: New Moodle 3.3 service super slow
but already locked. ☹️

 There we found an interesting cluster setting for our researches between Alain Raap - and  Benoît Lathière (so thanks). Concretely, it obtained performance improvement on "Application Cache" by applying Redis than NFS (shared).

 Here's my doubt: According to Moodle MUC specification this kind of cache ("Application Cache") must be shared for cluster environments, so I don't understand the improvement proposed based on local Redis for "Application Cache".  How did it be possible?

 Thanks in advace for any clarification
 Miguel

Average of ratings: Useful (1)
In reply to Miguel González Laredo

Re: Help with MUC on cluster enviroment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I'm not sure I fully understand your question.

However, I can promise you that you will have problems using NFS for the MUC in a cluster environment. NFS is (usually) orders of magnitude too slow. Using Redis is generally a straightforward solution.
Average of ratings: -
In reply to Howard Miller

Re: Help with MUC on cluster enviroment

by Miguel González Laredo -
Picture of Plugin developers
Thanks Howard, you confirm me one of the most important conclusions after navigating this amazing performance community.

Concretely my doubt is, rigth now, about one interesting solution on the locked thread (https://moodle.org/mod/forum/discuss.php?d=354191) posted by Benoit, where he said "We do that too: Memcached for sessions, Redis for application cache, in local on each node (so, no SPOF)."

I got surprised with the fact that "Application Cache" could be LOCAL for each cluster node, because Moodle MUC docs say explicitely that cache must be shared on multiserver environments (as we have).

Do you better understand my question now? We're looking for better performance on our cluster but we prefer solutions from local than add new cluster/network complexity to our system. Nowadays, for example, we already use Session Cache on Memcached locally.

Thanks for your fast answer!¡!¡
Average of ratings: -
In reply to Miguel González Laredo

Re: Help with MUC on cluster enviroment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I can tell you what we did. We installed Redis and pointed both the MUC and the Sessions at Redis (with different prefixes - I'm not sure if that is required).

As far as I can tell, although local caching is in theory possible, none of the plugins actually support it at the moment. I am no cache expert and (sorry to bring it up again) the documentation is rather patchy.

We also moved file locking to the database. My view is to shift everything you can get off NFS. It makes a difference.

Incidentally, we previously used Memcached for MUC and found that Redis performed much better - although it's not scientific as we changed a whole lot of things at the same time. Our cluster performance is now excellent - even (especially) under high load. 
Average of ratings: -
In reply to Howard Miller

Re: Help with MUC on cluster enviroment

by Miguel González Laredo -
Picture of Plugin developers
Hi again, @Howard. Any experienced advice from MUC over cluster environments is welcomed! Of course, starting from yours one. These forums help a lots!

Nowadays, we're pointing to two component:
  1.  Oracle Server (database)
  2. I/O performance and caching options.
About your case, I would like to understand the following key aspect, if it is possible: is your Redis caching system centralized/shared for all the cluster nodes or you installed one Redis instance locally inside each web node? Sorry, I deeped into MUC documentation several times, but keep having some doubts about it.

Maybe, someone with deep expertise on MUC implementation could actually help to clarify on that documented restriction (e.g. for Application Cache being restricted to shared storage)!

Lots of thanks
Average of ratings: -
In reply to Miguel González Laredo

Re: Help with MUC on cluster enviroment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
There is one Redis instance for all the nodes. This is mandatory (unless you have a session-aware load balancer and that's probably more trouble than it's worth)

I should probably point out that Oracle is not recommended as the database for Moodle. But you presumably have it working.
Average of ratings: -
In reply to Howard Miller

Re: Help with MUC on cluster enviroment

by Miguel González Laredo -
Picture of Plugin developers
Perfectly clear now! We are aware (several years ago) about Oracle support, but that is mandatory here so far. Anyway, our Oracle DBA helps a lot . I think Oracle is not the main bottleneck, but our disk storage and cache components.

We will continue to research MUC and shared / local options on it. Hope to return some interesting good news for the Community.

See you soon
Average of ratings: -
In reply to Howard Miller

Re: Help with MUC on cluster enviroment

by Chris Fryer -

Hi Howard

> We also moved file locking to the database. My view is to shift everything you can get off NFS. It makes a difference.

This is intriguing. The only documentation I can find for this feature is in config-dist.php. Are you using MySQL or PostgreSQL?

I see that MDL-67052 resolves some edge cases for Moodle 3.9. We are on the LTS track, and so are still on 3.5, but I'm interested in extracting maximum performance from our rig given the current circumstances. Can you give us some more detail on the changes you made?

Many thanks, and stay well.

Chris

Average of ratings: -
In reply to Chris Fryer

Re: Help with MUC on cluster enviroment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Documentation big grin. Very funny wink

We're using Enterprise MySQL (which I recommend if you have the funds)

Yes - the configuration has no UI and has to be made in config.php as you have noted. The table, however, is already created.

We were having problems with load spikes on our servers and we never got to the root cause. However, we're pretty sure that NFS performance had a significant part in it. Essentially, we got everything we could that was performance related off the NFS. We made a load of changes at the same time so it's not entirely scientific. The outcome was that the specific problem has gone plus we must have halved our average page load time and mostly don't see load averages on the servers that exceed 1.

It will be interesting to see if MDL-67052 makes things even better (although we have no issues with the present arrangement)

I suppose if you just take a step back... NFS tends to be a bottleneck and it just seems logical to avoid it where possible.
Average of ratings: Useful (1)