Server Clustering recommendations M2.6

Server Clustering recommendations M2.6

by Albert Ramsbottom -
Number of replies: 13

I have been having a look at the recomedations for server clustering in Moodle 2.6 here and need to clarify some of the points. It states that:

  1. Use OPcache extension on all cluster nodes. --- Why is this better than APC for instance
  2. Set $CFG->localcachedir to fast local filesystem on each node. --I suppose this is done in the Moodle config file
  3. Set $CFG->tempdir to fast local filesystem on each node. --I suppose this is done in the Moodle config file
  4. Use one big central memcached server for all shared caches that support it. -- Meaning? I dont understand this part
  5. Use local memcached instances on cluster nodes for local caches that support it. -- And this is a little vague
  6. Store user sessions in one shared memcached server. -- Does this mean that we need to have a separate memcached server as well as a shared Moodle-data folder
  7. Use fast local directory for dirroot on each cluster node. -- So for two servers, two local Moodle application folders
  8. Use dynamic cluster node management. ---??????????????
  9. Use transparent proxy servers.???????????????

 

If anyone can fill in the gaps in my understanding that would be great!

We currently run two webs with local moodle applications on both using sepaarte disks for each, with shared VM disk with Moodle-data folder using OCFS2 as a clustering service. And 1 x separate DB servers

 

Cheers

 

Average of ratings: Useful (2)
In reply to Albert Ramsbottom

Re: Server Clustering recommendations M2.6

by Albert Ramsbottom -

Ok so $CFG->localcachedir has been introduced in to Moodle 2.6

 

Why is this different than $CFG->cachedir, as both of our webs have cachedir in the local application folder so are not shared mixed

So why has the option to set the $CFG->localcachedir been included.

 

Cheers

Average of ratings: -
In reply to Albert Ramsbottom

Re: Server Clustering recommendations M2.6

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If you point $CFG->cachedir to a separate folder on each web server, then things will break. Don't do it.

(If you want a specific example, edit a question on one webserver node, then preview it with the request going to a different webserver node. However, that is by far from the most serious possible breakage.)

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Server Clustering recommendations M2.6

by Albert Ramsbottom -

interesting!

And I have never come across this and have never had anything break either!

OK we have the cache directory in the shared Moodle data folder.

But we also have local cache directories in each Moodle application folder on each web (default setup)

We have never pointed anything to anywhere so I assume that by default Moodle caches in the Moodle-data folder?

We are using Moodle 2.5.3

In the moodle config file there is an option to set the CFG->cachedir = '/var/www/moodle/cache'; (which isnt set), why is the different from the new CFG->localcachedir. Because if I was to set it then ???????????????

Just answered my own question!

The localcachedir has been set up specifically for multi-node environments, so one would leave the cachedir as is because it has to be shared and point each localcachedir to a local location because it doesnt have to be shared

wide eyes

Average of ratings: Useful (1)
In reply to Albert Ramsbottom

Re: Server Clustering recommendations M2.6

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'll have a go....

  1. Use OPcache extension on all cluster nodes. --- Why is this better than APC for instance

OpCache is natively supported from PHP 5.5 on. It's the future. APC will die out I suspect. 

  1. Set $CFG->localcachedir to fast local filesystem on each node. --I suppose this is done in the Moodle config file

Yes - as you say a 2.6 feature. 

  1. Set $CFG->tempdir to fast local filesystem on each node. --I suppose this is done in the Moodle config file

Yes - because if you don't it will try to use your shared 'data root' which will almost certainly be slow. 

  1. Use one big central memcached server for all shared caches that support it. -- Meaning? I dont understand this part

The MUC (Moodle cache) needs a shared caching area. Again, by default it will try to use your shared dataroot and this is bad because it will be slow. For shared caching (or all caching) use memcached. 

  1. Use local memcached instances on cluster nodes for local caches that support it. -- And this is a little vague

This (if I understand correctly) is advanced stuff. There are some discussions somewhere. I think the OU has something to do with this. 

  1. Store user sessions in one shared memcached server. -- Does this mean that we need to have a separate memcached server as well as a shared Moodle-data folder

Same memcached server - all set through MUC.

  1. Use fast local directory for dirroot on each cluster node. -- So for two servers, two local Moodle application folders

It just means you store the Moodle code on every front end in local storage. Don't try to keep one copy on shared storage. 

  1. Use dynamic cluster node management. ---??????????????

If I understand this properly - use proper load balancing. Don't be cheap and use round-robin.

  1. Use transparent proxy servers.???????????????

Unsure - think this might depend on your particular configuration. 

Average of ratings: Useful (6)
In reply to Howard Miller

Re: Server Clustering recommendations M2.6

by Albert Ramsbottom -

Wow thanks howard

I was a little confused bu the use cluster management suggestion as one would think that this means OCFS2 or GFS or alike

We already use transparent servers - I believe this means hide the IP address of the webs

"Store user sessions in one shared memcached server. -- Does this mean that we need to have a separate memcached server as well as a shared Moodle-data folder"

"Same memcached server - all set through MUC."

Still a little confused by this one but my understanding of this is nearly complete now - Great

I will have to give it a go on our Pre-Prod multi-node environment as I do have some Jmeter benchmarks for this cluster without any Memcached or localcaches set up

 

Thanks again Howard

Average of ratings: Useful (1)
In reply to Albert Ramsbottom

Re: Server Clustering recommendations M2.6

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Actually... just when I thought I understood this I look at the MUC admin page again and get confused. It has a criminal lack of help and documentation (anybody out there reading this??). I'll go and have a proper play before I cause more trouble.... wink

Average of ratings: Useful (2)
In reply to Howard Miller

Re: Server Clustering recommendations M2.6

by Albert Ramsbottom -

OK I wont base all my knowledge on what you have taught me then big grin

I will post any updates when I have got my head around the MUC and the Memcached part

Cheers

Average of ratings: -
In reply to Howard Miller

Re: Server Clustering recommendations M2.6

by Anna Jonna Ármannsdóttir -

I would like to thank you all for the questions and answers. I would have benefited from this discussion when I was rummaging around the MUC in a testsetup of Moodle 2.6.  After a few hours experimenting with setting up Memcached and playing around with the MUC config screen, the concepts cleared up. I would like to share what I learned here in a terse step by step guide:

  1. Determine your specific need for caches. This involves the concepts of shared cache or local cache, disk based cache or server based cache or protocol specific cache like Memcached or MongoDB. 
    1. If you have a slow shared disk, you might benefit from a local disk cache.
    2. If you have only a few users on your Moodle site, you might not want to set up a Memcached service, even if the acceleration that Memcached provides, is very significant for the user experience.
    3. If you want to do anything in your power for accelerating the site, you might consider MongoDB.
  2. Configure the caches, test them and verify them. There is room for improvement in the examples for cache configuration. This is probably the most difficult step.
  3. Install the cache support on server level. This may involve installing php modules or config for php modules.
  4. Create a cache instance in MUC here: example.com/cache/admin.php?action=addstore&plugin=memcached
    1. Give the cache instance some arbitrary name.
    2. All other fields have a question mark that can be clicked on for excellent help that tells you what to fill in, and the syntax (very important).
    3. The result should be a Configured Store Instance, with the name of your choice.
  5. The final step is to deploy the created cache instances by Editing Mappings for e.g. Language string cache in the list of Known cache definitions.
    1. While experimenting with this, I have found it a life saver to open a separate browser window, where the default setting is selected, so I just need to click on the Save button to revert the setting, just in case I lose contact with the site.
    2. Select the nam of your configured store instance from the dropdown list and click on the Save button. 
    3. Test the new cache setting. If you lose contact with the site or it behaves weirdly, try using the trick in step 1. In case of emergency you might remove the cache config file (muc/config.php) in the folder pointed to by $CFG->dataroot .  AFAIK Moodle writes a new default config file if it is missing.

I am also trying to learn more about this so please comment. smile

--
Anna Jonna Armannsdottir
Moodle admin for University of Iceland.

Average of ratings: Useful (8)
In reply to Anna Jonna Ármannsdóttir

Re: Server Clustering recommendations M2.6

by Albert Ramsbottom -

Great info Anna and thanks

I still think some points on this subject need to be clarified!

So I have decided to bump this subject once more

 

big grin

 

Average of ratings: -
In reply to Anna Jonna Ármannsdóttir

Re: Server Clustering recommendations M2.6

by Paul Lindgreen -
Picture of Particularly helpful Moodlers

re: If you want to do anything in your power for accelerating the site, you might consider MongoDB.

Can you elaborate on how to setup MongoDB with moodle by providing some screenshots of your MUC admin page configuration? Is there any downside to using MongoDB? Does mongodb need to be installed on a seperate server than the moodle webserver?

Average of ratings: Useful (1)
In reply to Paul Lindgreen

Re: Server Clustering recommendations M2.6

by J S -

There are pros and cons to each caching system available by default for MUC.  For the sake of simplicity, I'll bundle memcache and memcached into the same bucket.

So, by default I think you have as available caching stores:

1. memcache(d)

2. mongodb

3. filesystem 

 

If you are running multiple moodle sites against the same cache store backend OR have a high traffic site, memcache(d) may not be the best option.  Moodle makes calls to the memcached backend to flush the cache fairly aggressively in 2.5, and somewhat less in 2.6.  However, a flush is a flush, and if you have admins working in your moodle site your cache will be flushed quite a bit. 

MongoDB is a bit different in that it flushes cache by cache definition.  This is a lot better.  However, I was unable to get mongodb to perform at the same speeds as memcached without a lot of playing around (ramdisks, etc).  Memcache(d) just seemed to outperform mongo on all levels.  I'd be interested in hearing about peoples MongoDB setup who chose Mongo over Memcache(d).  I think Mongo is pretty cool but just couldnt get it to perform without a lot of effort.

Due to the cache flushing issues of memcache(d) and the speed difficulties of mongo, we decided to use Redis instead.  This performs at the speed of memcache(d) with the benefits of per cache definition flushes.  You can get the cachestore here: https://github.com/moodlerooms/moodle-cachestore_redis  This plugin also handles multiple moodle sites being served out of the same redis server quite well.  You will need very minimal server requirements for your redis host (1cpu,1g ram).  Only if you are serving thousands of moodle sites will you even need to breach the 2G mark.  Moodle 2.6 has a different footprint and should be profiled for your site(s) prior to settling in on your sizing.

Average of ratings: Useful (2)
In reply to Paul Lindgreen

Re: Server Clustering recommendations M2.6

by Anna Jonna Ármannsdóttir -

For testing purposes the MongoDB can reside on the webserver provided that it is not significantly loaded. In my case MongoDB was installed on a server different from the webserver. I have not gotten any further than just testing. Regarding if there is a downside to using MongoDB, I guess it is rather humongous and might be considered an overkill in small installations with few users. The performance of the caches can be tested and compared by clicking Test Performance under Cache to get a Cache store performance reporting. For this example I tested for 500 unique requests per operation. 

Store requests when used as an application cache.

PluginResultSetGet - HitGet - MissDelete
File cache Tested 5.9007 1.0184 0.0314 0.5714
Memcache Untestable - - - -
Memcached Tested 0.2245 0.2185 0.2092 0.1909
MongoDB Tested 0.5616 0.2620 0.2336 0.3095
Session cache Unsupported mode - - - -
 

 

MongoDB seems to be half as fast as Memcached when it comes to Set and Delete operations, but approximately just as fast at the Get operations. This is for the application cache. When it comes to the session cache MongoDB seems somewhat slower than Memcached, especially at the Set operations where the ratio is 1 out of 3.

Store requests when used as a session cache.

PluginResultSetGet - HitGet - MissDelete
File cache Tested 5.6237 0.9106 0.0345 0.5676
Memcache Untestable - - - -
Memcached Tested 0.1991 0.2006 0.1673 0.1561
MongoDB Tested 0.6020 0.2444 0.2380 0.2881
Session cache Tested 0.0024 0.0012 0.0005 0.0011
Static request cache Unsupported mode - - - -

 

I used Memcached as the metric for what seems to be achiveable in my setup, but if  data guarantee is needed, Memcached is disqualified because it does not support it. The ones left are File cache and MongoDB and the latter is clearly the winner, being approximately 10 times faster at Set operations and 4 times faster at Get-Hit operations and twice as fast at Delete.

Regarding screenshots, the config page is so big that I decided to print it to a pdf file. In order to get a screenshot of the actual config pages, I need to update my Moodle, seems that Moodle 2.6.0 throws an error. The PDF is attached but I had to compress it with xz to get it past the 100kB limit. The xz zipper is standard on most linux installations so it should not be a problem.

Average of ratings: Useful (1)
In reply to Anna Jonna Ármannsdóttir

Re: Server Clustering recommendations M2.6

by J S -

Why must the cache be guaranteed?  I would think that if the Moodle developers felt that memcache(d) was not sufficient, they wouldn't have made it one of the plugins available.  

Average of ratings: Useful (1)