AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Lee Wilson -
Number of replies: 7

Hello,

I am running Moodle version 3.10 (Build: 20201109) as a cluster within AWS, on Centos 7 with PHP 7.4 and OpenResty 1.19.3.

Everything is running very smoothly and stable, however I am having some issues with configuring AWS Elasticache Redis to handle Application and Session cache. Although stable, I do have some slow page loads which I believe enabling the Redis cache store for application cache will help to improve. I have done other performance tweaks such as: only enabling Moodle modules I need, configured xsendfile (X-Accel-Redirect), amongst a few other things which have all had notable performance and stability improvements.

I have read various pages within Moodle Docs around caching, MUC etc including this one: https://docs.moodle.org/310/en/Caching

I have also followed the instructions on this page (https://docs.moodle.org/310/en/Redis_cache_store) for setting up the Redis cache store including the step for setting up the mappings - "Stores used when no mapping is present".

When it comes to confugring within Moodle (within Site Administration > Plugins > Cache > Configuration) the store instance is not ready to be used but has mappings are in place. The error I get is: "This store instance is not ready to be used but has mappings. Fixing this issue will improve performance on your system. Please check that the store backend is ready to be used and that any PHP requirements are met."

My connection settings (I have masked sensitive information with x and y)...


My mapping settings...


Mapping settings applied with an indication of a problem...


Description of the problem...



When I go into Site Administration > Plugins > Cache > Test Performance, it states that the "Store not ready" for Redis.


I can confirm I do have the PHP Redis module set up, and I am able to connect to the Elasticache Redis instance via a telnet connection from the EC2 instance, so there is nothing blocking the network traffic. As a test, I did set up the PHP session save handler to be Redis, and PHP was able to successfully use the instance as a session store. When viewing the monitoring section of AWS, it is possible to see the Redis store and the cache itself being hit. My web sessions are also terminated if I log in while Redis is handling sessions, and then I switch to the default "files" as a session handler. All of which further validates that connectivity is certainly ok.

I have tried numerous things to resolve this, all with no success:

  • remove everything and started again in case I missed any steps or did anything incorrect (the oldest trick in the book!)
  • stopped PHP from using Redis as a session handler in case it conflicted with Moodle
  • ensured any $CGF->session_redis_* values are not being set in case it was conflicting with what the Moodle UI was setting
  • destroyed and rebuilt my Elasticache Redis instance

I have seen a similar problem described elsewhere with people using Memcached, but this was for a local setup, not a remote store (i.e. AWS Elasticache), and the steps used to resolve elsewhere seemed irrelevant with Redis (mostly was a port issue elsewhere, I don't think that's the issue with Redis as I only have one port to use - 6379). I really cannot find any evidence elsewhere of someone else having the same issue with Redis or even Elasticache Redis.

Has anyone else seen this issue before? And does anyone have any ideas of what to do next?

Thanks, 

Lee

Average of ratings: Useful (2)
In reply to Lee Wilson

Re: AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Rex Lorenzo -
Did you solve this? I found that I needed to just have the URL and not include the port or prefix.

But I found that I am getting the error
Exception - MOVED 7209 172.24.8.49:6379

Any clue on fixing that?
Average of ratings: -
In reply to Rex Lorenzo

Re: AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Rex Lorenzo -
I found that Moodle does not support Clustered Redis yet, see https://tracker.moodle.org/browse/MDL-59196

So turning off Cluster support in Elasticache seems to work for me now.
Average of ratings: Useful (1)
In reply to Rex Lorenzo

Re: AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Lee Wilson -

Hey, thanks for looking at this.

I still havnt solved it. Being honest, I kind of parked this for a while hoping for someone to help, as I spent a lot of time on it and other tasks took a hit, so I’ve focused on other aspects of setting up my Moodle instance. 

I do remember trying without the port number but I found my entire Moodle instance would stall. I would have to go into one of the MUC cache files and set the Redis port number again in there for my Moodle instance to recover.

What you mentioned about Moodle not supporting Clustered Redis did not occur to me as a potential problem. I’ll give that a try and get back to you.

Just so I set it up correct, did you set the URL with port number via the Site Administrator section as I did in my original post?

Do you place any specific Redis config within the config.php file?

I plan on my Redis instance handling PHP sessions as well as Moodle sessions and application cache - proving its supported. I need to do some testing on this.

It’s been years since I’ve last set up Moodle, so I’m getting to grips with a lot of new stuff as well as hosting it within a clustered cloud platform rather than on premise.

Thanks,

Lee


Average of ratings: -
In reply to Rex Lorenzo

Re: AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Lee Wilson -
Sorry for the slow reply on this. I finally got round to investigating the issue further.

I stopped trying to connect to Redis in clustered mode and went for a single node Redis, and I still had no luck.

I then found this tool: https://moodle.org/plugins/tool_redis which started to give me sane error messages. And I managed to figure out that I was connecting to the Redis instance via a TLS connection which Moodle did not like. So I rebuilt the Redis instance with data encryption in transit disabled (I kept data encryption at rest enabled), and used a plain TCP connection e.g. address:port, and voila, connection successful!!

So for now, I am running a single Redis instance over an unencrypted connection. This is fine for the time being, as I have security rules and NACLs controlling private access to the Redis instance, as well as disabling public access to the instance. So in the short term this will serve me ok, but ultimately I need Moodle to be connecting via TLS.

Am I right I’m thinking that Moodle does not support Redis over a TLS connection?

Thanks,

Lee
Average of ratings: Useful (1)
In reply to Lee Wilson

Re: AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Luis de Vasconcelos -
Hi Lee.

Did you find an answer to your "Am I right I’m thinking that Moodle does not support Redis over a TLS connection" question?
Average of ratings: -
In reply to Luis de Vasconcelos

Re: AWS ElasticCache Redis for Application and Session Cache - 'Store not ready' issues

by Lee Wilson -
Hi Luis,

I did get this working and  YES Moodle does support Redis over a TLS connection. 

It's been a while, but I think the problem was that I did not input the correct host URL.

  • So in /admin/settings.php?section=cachestore_redis_settings
  • In the Test server field, the URL I have working is: tls://clustercfg.apps.xxxxxx.yyyy.cache.amazonaws.com:6379
  • And this works:




I still get Store Not Ready, but when I review the Redis instance within my AWS account, I can see it being used, and also the performance improvement of my Moodle application is significantly better. The Site Administrator section loads instantly even on a single t2.micro EC2 instance!

It turns out, the problem I had was self-inflicted. I did not include :6379 on the connection string in the Test Server field. I kept assuming Moodle would append this for me, given that 6379 is the default port number for Redis.

I hope this helps,

Lee


Attachment Screenshot 2021-12-01 at 13.12.02.png
Attachment Screenshot 2021-12-01 at 13.12.08.png
Average of ratings: Useful (1)