Moodle in English: Moodle on AWS load balancing Architecture

Moodle on AWS load balancing Architecture

by Steve Pollock - Wednesday, April 19, 2017, 9:47 AM

I have seen a number of posts on this topic, many old, including; https://moodle.org/mod/forum/discuss.php?d=254113 which provides some hints at best.

My issues is latency as I have customers around the world, and HA would be a "nice to have" as well.

Has anyone documented the architecture of a successful deployment of Moodle on AWS using EC2 web server(s) in different Regions with global load balancer and a single database instance. I can come up with a half dozen different ways to do this, but am looking for some guidance on a production deployment. Following these basic instructions for a single instance, I presume you can swap out "mysql" with one of the AWS database services like RDS. I'm looking for detail, for example how you setup config.php and the AWS services to match. I've seen everything from nfs mounts to remote mysql described, however I suspect going "native" with AWS will work out better in the long run.

Many thanks in advance,

Steve

Re: Moodle on AWS load balancing Architecture

by Howard Miller - Wednesday, April 19, 2017, 7:09 PM

Have you tried it? Are you fixing a problem that you actually have or one you think you might have?

I've used AWS quite extensively in the past for Moodle with no particular performance issues.

Average of ratings:Useful (2)

Re: Moodle on AWS load balancing Architecture

by Christos Savva - Saturday, July 29, 2017, 4:49 AM

I have setup Moodle on AWS using ELB, EC2, RDS, EFS

The setup is pretty straight forward.

A number of EC2 instances running Apache or Nginx
A classic Load Balancer to register the EC2 instances
Aurora RDS database (here you might have a problem installing Moodle 3.1 and up MDL-58931) MariaDB would be better solution
Elastic File System (EFS) that is mounted to all EC2 instances so that you can share the Moodle Core code and moodledata

The problems:

One huge problem is performance with this setup. Moodle was running slow, and by slow I mean 40-50 sec loading.

After some test we concluded that the problem was that EFS is (for some reason) is very bad for caching. We still didn't find the best solution for this but nevertheless our cluster runs ok so far, using the methods below

Use Amazon ElasticCache for Application cache. Problem: for some reason Moodle loses connectivity and automatically reverts back to Default Application Caching, making Moodle slow again. If you go to Cache Plugin and reset the Cache Store Moodle comes backs to normal
or
Use an EC2 instance as a memcached server. Problem: The same as above, but it rarely happens

Use an EC2 instance as an NFS server for caches

$CFG->tempdir = "/path/to/nfs";
$CFG->cachedir = "/path/to/nfs";

Assign localcachedir to the same local directory in each EC2 machine
```
$CFG->localcachedir = "/path/to/localcachedir";
```

I hope that the above helps. If you build your cluster and manage to solve the loading issues please let us know

Average of ratings:Useful (7)

Re: Moodle on AWS load balancing Architecture

by Sharath Chandra Kogila - Monday, August 7, 2017, 1:07 PM

Hi Christos Savva,

I have been looking for such a discussion thread if already exists as i am going to setup up Moodle on AWS exactly as you mentioned using ELB, EC2, RDS, EFS. I am worried on the EFS latency as some other posts also mentioned. Can you please help me on clear documentation on how you overcame this problem. Specifically looking for documentation which helps in configuring different cache options as you mentioned

Thank you very much

Regards,

Sharath Chandra

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Christos Savva - Monday, August 7, 2017, 6:02 PM

I don't think there is a complete documentation anywhere. I was looking for one too, but I ended up experimenting myself.

Basically what I did was

ElasticCache for caching (https://docs.moodle.org/31/en/Caching)
moodledata on EFS
tempdir & cachedir path to an NFS server (seperated EC2 instance) on the same AZ and region
localcachedir path to local filesystem of each webserver.

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Sharath Chandra Kogila - Wednesday, August 9, 2017, 7:11 PM

Thanks I will check that.

Regards,

Sharath Chandra

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Sharath Chandra Kogila - Tuesday, August 29, 2017, 6:59 PM

I almost finished multiserver configuration but am not able to understand few finer details. Any help is appreciated:

- I configured MoodleData over EFS and for this, i used the configuration variable

$CFG->dataroot = '/myefsmount/moodledata';

I have 2 more configuration variables that i did not quite understand

$CFG->tempdir

$CFG->cachedir

Are they related to Application Cache mentioned in https://docs.moodle.org/33/en/Caching#Cache_configuration_in_Moodle

I configured a Memcached service on a seperate EC2 Instance and tried to Add a Store Instance. This by default always created a Application Store. How can i add a MemCached instance for Session and Request type cache

Thank you,

Sharath

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Michael Spall - Tuesday, August 8, 2017, 1:53 AM

Christos,

When we were setting Moodle up in AWS we also found that EFS wasn't the best choice for file caching. This was while EFS was in preview mode, so things may have changed. We are using Gluster for file cache. We found Gluster to be better and faster than NFS. We are also using the same Gluster for moodledata.

We used memcached on 2 of the EC2s that were web servers. We found that better than ElastiCache. With our upgrade to Moodle 3.3 this summer, we are switching to running Redis on a separate EC2. We use separate Redis servers on the same machine for session and application. On our smaller Community instance it is already working great. We will be upgrading the main Moodle instance this week. ElastiCache has some durability built in that isn't needed. If something isn't in the cache, just generate it.

I have a fix for using Aurora for Moodle 3.1 to 3.3:

https://github.com/mspall/moodle/commit/f1d57d54aa73f73156a855678ef2b77dde91d4a2

This commit is for 3.3 but it is the same for 3.1. There is a bug in Moodle logic where if MySQL supports Compressed or Dynamic, it chooses Compressed and if it doesn't support Compressed or Dynamic it chooses Dynamic. Oops!

Average of ratings:Useful (1)

Re: Moodle on AWS load balancing Architecture

by Tawfik Daim - Saturday, October 21, 2017, 10:06 PM

Hi Christos

I went on a mission to create a stack similar to what you described, so now I have

2 EC2 instances running Apache
A classic Load Balancer to register the EC2 instances
MySql RDS database
Elastic File System (EFS) that is mounted to both the 2 EC2 instances and on it I mounted the Moodle Core code and moodledata

Succes....its working, but then I noticed the very slow performance compared to all in one AWS micro server that I have running Apche+MySql+Moodle Data, so to fix this I started with MemChacheD installation on both the 2 EC2 webservers and it significantly speed thing up (notice I'm testing with a single user just to test basic functionality).

Where I need help:

1. What I want to clarify is for your solution number 2 (Use an EC2 instance as a memcached server.), did you mean to have a new intance side by side to the Load Balanced instance of Apached? and how to configure this one ?

2. Also can I use this instance of memcached to host the moodledata on an internal SSD drive that I mount to both EC2 web servers (and later to the more EC2 webservers I plan to add to the Load-balancers based on load) - will this replace the step 4 EFS, coudl it be a faster option than EFS

3. can you please clarify how to configure "Use an EC2 instance as an NFS server for caches", is it what I try to ask about in above point?

4. Can you please clarify how to "Assign localcachedir to the same local directory in each EC2 machine"

Lastly, do you have any further recommendation from what you have done so far

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Michael Spall - Tuesday, August 8, 2017, 1:27 AM

Steve,

I have set up Moodle in AWS for Idaho State University in one region, not multi-region. We have not had issues with users across the world. We found that under a load balanced system the MUC is one potential bottleneck. I think the issues you would see moving to a multi-region set-up would be cache based and database based. You would be trading web traffic distance for cache distance, and caching issues are hard to diagnose and fix sometimes. I don't think Aurora allows multiple writers in different regions and Moodle doesn't really separate reads from writes. EFS needs to be in one region. I also think ElastiCache runs in one region.

I would get a hold of your account manager and have them put you in contact with an AWS engineer. I would consider setting up in one region and using CloudFront, Route53, and other AWS tools to help with Internet latency.

If you go multi-region, I would make sure I was using Moodle 3.3 or later and use Redis Cluster for your session cache and application cache. I have used Redis but not Redis Cluster, so I would not be the best expert on that. I would use Gluster for your MUC file caching:

$CFG->tempdir
$CFG->cachedir

We also use Gluster for moodledata.

Looking at the issues, I would push for the one region, or mostly one region, solution.

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by James Hale - Monday, August 21, 2017, 2:51 PM

Agreed. Multi-region is a huge undertaking, and nearly impossible to manage for an off-the-shelf product. I doubt even the massive MOOC providers like edx and coursera are multi-region for their core product.

We run Moodle on AWS in the us-east-1 region, even though most of our users (and our company) are in Australia. No obvious latency. Get spikes of thousands of users with no issues. You could use AWS CloudFront to cache your static assets, but your main latency is going to be the backend.

We're using AWS EFS successfully. For Moodledata. It looks like most of you guys are using EFS for your caching, which is no good. File stores are a horrible place for a cache. Use EFS for Moodledata, and Memcached for cache.

Our AWS Opsworks scripts are available for anyone who wants to try it out. https://github.com/ITMasters/moodle-on-aws-opsworks. Opsworks + RDS + EFS + S3 (Moodledata backup) + Memcached. Separate Memcached instances for session and application stores.

Opsworks is a bit of a weird cousin in AWS land, and it has a few really annoying gotchas, but it's worked fairly well for us. Deploying updates is fairly straightforward - much bigger bonus than the HA/scaling.

Average of ratings:Useful (1)

Re: Moodle on AWS load balancing Architecture

by Jesse Clark - Thursday, September 7, 2017, 12:40 AM

Hi Michael,

Would you be willing to give me an idea how many learners your setup is supporting and how many instances of what size you use to do so? Also, maybe what your peak concurrent users without performance loss is? I am trying to get some initial costing estimates together and having a bit of trouble finding information.

Thanks,

-Jesse

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Eduard TC - Tuesday, September 12, 2017, 7:28 PM

Hi all,

I solve this slow issue configuring a memcached instance with Elasticache. The moodle 3.3.1 works fine since this.

Regards,

Eduard.

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Sharath Chandra Kogila - Monday, September 18, 2017, 7:14 PM

Hi Eduard,

Thank you. Could you please advice how to configure this. The documentation for memcached - that way for any cache configuration is awfully incomplete. I successfully installed Memcached and added an instance under the section ( Installed cache stores ). I see green tick against MemCached.

I encounter the following error and am not able to proceed

. I donot see my store ready (green tick) under Configured store instances, but is see the help message "This store instance is not ready to be used but has mappings. Fixing this issue will improve performance on your system. Please check that the store backend is ready to be used and that any PHP requirements are met."

Can you please suggest the next steps after adding the Cache store instance. Also, how do i relate the configuration from Site Admin page to variables in config.php ($CFG->tempdir , $CFG->cachedir), should these be configured too.

Thanks,

Sharath

Average of ratings: -

Re: Moodle on AWS load balancing Architecture

by Eduard TC - Monday, September 18, 2017, 8:54 PM

Hi Sharath,

One time you configured the memcached node adding the instance in "Installed cache stores" (Dashboard - Site administration - Plugins - Caching - Configuration ) go to the bottom of page to the section "Stores used when no mapping is present" and "edit mappings" then in Cache administration select memcached as cache for the application. (I think this way you assign the memcached to all the possible options, I think).

Hope it helps!

Eduard.

Average of ratings: -

Hardware and performance

Moodle on AWS load balancing Architecture