Caching & Application Cache in Cluster

Caching & Application Cache in Cluster

by Saad Zaman -
Number of replies: 15

Hi Everyone,

I need to move to scale vertically, i have moved sessions to redis. setup localcachedir on nodes - And shared storage to NFS - However i am uncertain of the performance keeping cache on NFS, so want to move it to redis.

Can someone please guide me to how to setup Redis for application cache. Im already using it for session cache, I found the config to be set from config diff.

Currently what i have understood is, that applicationcache is stored in cachedir and by default cachedir is in moodledata - However i dont see how i can set this cachedir as Redis. 

Or is it such that when we configure redis cache store from cache stores from UI - The cachedir is not used ?

Ideally i want the moodlestorage ONLY and ONLY to be used for files. It would be great if anyone could tell me anything iam missing here

Thanks,
Awaiting reply from the awesome community

Average of ratings: Useful (1)
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Jeff White -
You cant use Redis 100% for all parts of your "caching" as cache, temp, muc (moodledata/muc),  lock, and more will still use a directory that must be shared.


  1. How to setup Redis for MUC application/session cache. Note that MUC session cache is different from the session cache you set in your config.php. 
    1. Go to cache/admin.php
    2. Click add instance on installed cache stores on the redis row
    3. Add your redis server
    4. Scroll down to Stores used when no mapping is present
    5. Click Edit Mapping
    6. Change application & session to your redis instance.

I would love it too if I could use moodledata for only filedir and some other files. Hopefully we will get there one day. 

Average of ratings: Useful (3)
In reply to Jeff White

Re: Caching & Application Cache in Cluster

by Luis de Vasconcelos -
Thanks Jeff

Note that MUC session cache is different from the session cache you set in your config.php.

Can you explain this further? How many caches are there?

What caches go in Redis and what caches don't go in Redis?

Average of ratings: -
In reply to Jeff White

Re: Caching & Application Cache in Cluster

by Saad Zaman -

Thanks a lot Jeff for this insight.

Do you have any idea of what parts of moodle rely on these ", temp, muc (moodledata/muc),  lock, " i just want to be sure , what areas of the system will degrade when i move to NFS

Average of ratings: -
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Jeff White -

Ill refer to Howard's post:  https://moodle.org/mod/forum/discuss.php?d=385374#p1553414

Average of ratings: -
In reply to Jeff White

Re: Caching & Application Cache in Cluster

by Saad Zaman -
"Some aspects of the cache system are shrouded in mystery"


Lol Looks Like We as a community agree on this :D 

Average of ratings: -
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Saad Zaman -

@Jeff What do you think , will be the consequences of keeping this UNSHARED i.e on a local node ... 

Why i want to specially move the cachedir to localnode is that -- The modules details are loaded in this directory - And i have over 1400 modules - So if i make obligatory on myself to manually clear cache after making an update - AND provided not too many concurrent users UPDATE my module / settings - Shouldnt this work ?


//     $CFG->cachedir = '/var/www/moodle/cache';      // Directory MUST BE SHARED by all cluster nodes, locking required.


I mean after all -- its CACHE -- the worst that can happen is,  a user might get old data and do future actions on it .. this seems to be only be an issue if the course content is being edited


Thoughts ?

Average of ratings: -
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Jeff White -
No idea what would happen but I do know its not a good idea to go against the the dev docs on how to set up your environment.


I think the better approach is for you to see if you can make a new cache plugin that you can use in MUC. That way will Moodle improved for everyone. 

Average of ratings: -
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Luis de Vasconcelos -

> the worst that can happen is

Quite a lot can go wrong, including users losing their sessions if, for example, you have multiple web server nodes behind a load balancer.

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Caching & Application Cache in Cluster

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Yep, corrupt cache could also result in a "white screen".
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Caching & Application Cache in Cluster

by Jeff White -
lol the white screen that says everything is fine. No errors in logs. http 200 response. 


I have seen more than once that you need to delete the caches in moodledata manually. purge_cache.php cli usually works 2/3 times. Maybe cron will gradually fix things but system outages has no time for maybe it will go away on its own. 

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Caching & Application Cache in Cluster

by Jeff White -

Sessions shouldnt be lost if you use anything other than file session. I do remember moodle always making files in moodledata/sessions even when i switch to redis but I am not seeing that anymore in moodle 3.6. 

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Caching & Application Cache in Cluster

by Saad Zaman -

I was only referring to keeping the cachedir shared - Sessions are based on different configurations. They can be on redis or a shared instance as well. 

The only motivation to keep cachedir on the same node is the access speed - as cachedir is definitely slower when kept on NFS. And even if you keep it on redis or memcache, its no doubt quicker but not as quick as accessing from local file 

Average of ratings: -
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Saad Zaman -

Jeff thanks for your help in clarifying. 

So, here is my infrastructure setup on AWS for 1000 concurrent users 

  • Redis for session (s)
  • Memcache for Application Cache
  • RDS for database
  • S3/Cloudfront for SCORM Images and JS  - So basically we have a very high usage of SCORMS, so i decided not to overload moodles pluginfile to serve this.
  • NFS for moodledata [ its another ec2 server with ssd and is storage optimized ] 
  • EC2 Instance(s) - Currently im using a 3$/hr ( aws m4.16xlarge - with 256GB RAM and utilizing 40-50% of it. Yes this is huge, ill soon reduce the process size by disabling modules and probably optimize. But we want to stay flexible as we have more users expected this month ) - Otherwise we normally use a server that costs about 1.5 - 2$ / hr which brings the bill to between 800-1500$. | Please let me know if im spending a lot keeping in mind 1000 concurrent users.
  • ELB / Autoscaling / Kubernetes : We plan to first move to a load balancer and run this ec2 behind a load balancer. This will help us to add a new instance without downtime. ( already tested on staging ). Later down the road in few months we plan to move the infrastructure to kubernetes and leverage autoscaling features of k8s. You can also try AWS Autoscaling if you dont have frequent production builds, as AWS autoscaling needs an AMI to spawn nodes on cluster
  • X-Send For quick file access + mysqlnd_ms to utilize database cluster, all reporting queries ( most of them custom ) go to the reader instance.
  • Apache on this instance : 
                       MaxClients 900
                       ServerLimit 900
                       KeepAlive Off
                       StartServers       4
                       MinSpareServers    10
                       MaxSpareServers   20
                       MaxRequestsPerChild  30

Also i plan to move from Apache to lighthttpd or nginx with fast php -


Please advice for any other improvements you think i should make.


Average of ratings: Useful (4)
In reply to Saad Zaman

Re: Caching & Application Cache in Cluster

by Jeff White -
Hi Sadd,


Just want to give some feedback/questions on your new build.

  • Why use Redis for session and memcache for application? I would just do just redis for everything. Redis is the better choice as its more up to date,  has more features like redis sentinel for HA, and you can actually restart your service without having to rebuild the entire cache.
  • NFS, as you know, is going to murder performance and it looks like you have a decent budget so I would recommend checking out GFS2. You may even get such good performance where GFS2 works better or the same as Redis caching. If that is the case then stick to using file cache for everything. My 9 node cluster has local disk performance using a IBM flavor of GFS2 but I also have a really good storage team I work with.
  • 95% of our content is SCORM  with a lot of videos. One of the best things for us to do was to offload video to a media streaming server which is fairly easy to do. Brightcove is amazing.
  • I need to check out X-Send as what your describe sounds perfect for an issue I am experiencing with reporting.
  • We are starting to move to kubernetes so I hope you keep the community up to date on your progress smile. 
  • Hope you just forgot to mention OpCache. 
Average of ratings: Useful (3)
In reply to Jeff White

Re: Caching & Application Cache in Cluster

by Saad Zaman -
  • Using memcache, because i am on version 3.1 and have made some changes in the core code. There is no redis support for cache store in that version

  • Basically, i dont have a storage team or have anyone in my team with storage / cluster experience. I plan to shift to GFS2 when i have to support another region. NFS was fairly easy to setup and it didnt impact performance too much for me. 
    As seen on community, GFS is quite a task to manage - 

  • Yea sure - will keep posting. I have tested my infrastructure on staging - I used KOPS on aws - But i think that using ECS on AWS or any other managed cluster is better.  

  • OpCache was enabled by default on PHP i was.
Average of ratings: Useful (1)