We have a Moodle site running on Amazon configured via Elastic Beanstalk. The problem is, the site takes roughly 3 - 5 seconds to load each page. This is pretty frustrating to admins. I've tried resolving this by simply adding more 'ram' or 'v-cpu' onto the stack but it's not helping. I have also created a mem-cache and connected it to the site, but again, it's not really making a huge difference.
The funny part is, my local site, running on Docker is loading very quickly (1 second) in comparison, which is often the opposite in my experience. Here are the full specs:
Elastic Beanstalk - Ohio region:
Web: 2 x EC2 t2.large instances
- Moodle data is on a shared AFS
- Moodle core + modules are on the web servers directly, pushed via GIT deploys
Database: 1 x RDS t2.medium instance
- MySQL 5.7.19
- General purpose 20GB SSD
Cache: Elasticache Mem-Cache t2.small
I also have a worker instance that pings cron every 1 minute, but turning that off gives no performance gains.
We are using EdWiser UI but doubt this is the problem. Though I will test their single-sign-on to see if maybe there is some interference there.
- Have you deployed an AWS Moodle build and gotten 1-second page loads?
- ^ If so, what were your specs?
- Is there anything glaring that you see with my configuration?
Most likely you have issues with session file locking on your shared afs (what is afs?).
Reconfigure system to use memcached or redis sessions and make sure that your moodledata is on properly configured nfs server.
Thank you for the reply and apologize about the late response. I made a goof, when I said AFS I meant EFS which is Amazon's Elastic File System. On paper, it suppose to be better optimized than NFS.
This is why I'm scratching my head as to why the site is as slow as it is. It seems I've got all the right things in place.
I will try your suggestion about moving session data into mem-cached. No idea what that means but I'll google it.
I might have to try installing New Relic and seeing if it can tell me what's going on behind the scenes as I don't think Moodle has any kind of profiling capabilities. I'll report back with my findings.
-- Edit --
I moved session data into the database, and this helped. The speed is now consistently loading the pages at 3 seconds (Before it fluctuated 3 - 6+).
That's a good start. Though this is just me using the website, still not ideal if multiple users are on the system, will keep digging.
I also have the same problem with Moodle.
Setting up on the instance or container on the local hard drive is fine. But configuring in an EFS or GlusterFS, gets extremely slow. In fact, in Gluster the time takes around 10s, in EFS in 20s.
Opcache enabled. Any changed php configuration does not significantly influence time.
I already used m4.xlarge instances and the times were the same.
My current instances are t2.medium for the two nodes (eg GlusterFS) and the application instance (Moodle).
I would like to get between 3 and 6s.
Are you by chance using Edwiser UI or any of their plug-ins? If not, I can eliminate that off my checklist of things to probe. There is also S3 as another option which I will try next. Going to attempt in changing session to memcache.
I do not use Edwiser.
Switched session handling over to Memcached in the config.php file as per link (Thanks). Still about 3.75s to load. Though it's consistent now, which tells me that this was a problem, but not the problem. Not quite sure what the next step is.
Maybe I'll try disabling Edwiser UI or mess around with the EFS mount.
Not sure if this relates, but when installing Memcached I did it wrong the first time, and the site crashed. It still loaded the content and everything properly, just no theme and assets. Was getting 300ms load times.
EFS is known to have performance issues when accessing lots of small files. Most likely this is what you experience.
Just because cloud provider offers you something it does not mean it will do the work for you. So I suggest reading these documents:
And just search for things.
100% valid and I had assumed EFS would hold up, but maybe that's the bottleneck. I did play with themes and it seems certain themes are faster than others. Specifically, I did a profile in XDebug just on my local machine and noticed moustache processing taking up significant processing time.
Also to be clear, my codebase is not on EFS. The codebase is directly on the server SSD drive.
The only thing that's on EFS is Moodle data, but I don't know enough about Moodle to know exactly what it's using that folder for. If its cache, then I'd agree that would be an issue as its likely saving a lot of small files to disk.
So not sure where to go from here. I'll check out your links and maybe try S3, as I've had good success with that in the past. But if Moodle data can't be on a shared file system then that poses a lot of issues. I could simply turn this into a single application instance and scale vertically instead of horizontally but was hoping not to have to do that.
Either way, thank you so much for your help, really appreciate being able to bounce these ideas and get feedback from an expert.
We tested it about two years ago.
EFS was too slow for moodledata directory.
When using EFS, use ElastiCache together. It gave us nice speed as local disk.
Thank you Takeshi and Matteo,
I can 100% confirm the problem is EFS. Just ran an experiment where moodledata was simply on the server SSD and load times are instant. I'm using ElastiCache (Memcache) for sessions, but it doesn't appear to have any benefits in this scenario.
Will check out the provided links on optimizing NFS. I'll detail my setup when complete so anyone else looking to use AWS can benefit.
Please note that if using EFS or GlusterFS, you may find problems with Moodle locking mechanisms.
Remote and distributed file system like GlusterFS have bad performance for little file operations like Moodle locking.
To solve this, you must configure Moodle to use database for locking instead of files.
Sorry for the delay when replying ...
É verdade que o EFS oferece apenas um bom desempenho de 1 TB.
Há uma limitação em relação à quantidade de dados e troughpout de rede que é disponibilizada.
Eu praticamente desisti do EFS e estou testando outras variáveis para obter um número mais rápido de acesso.
Thanks for the help everyone. I ended up solving this by simply moving the moodledata to a separate EBS volume so it can survive a "eb deploy". Now, I won't be able to scale horizontally, so that is something I'll try to deal with in the future but as an immediate fix EBS is giving me <1s load times.
Using Elastic Beanstalk:
Webserver: 1 x EC2 t2.large instance (Can probably change to medium or smaller)
- Moodle DATA on separate EBS volume (single server disadvantage, will want to use NFS in future)
- Moodle SRC on server directly, pushed via ElasticBeanstalk
Database: 1 x RDS t2.medium instance
- MySQL 5.7.19
- General purpose 20GB SSD
Cache: Elasticache Mem-Cache t2.smallEmail: SES
Pages load instantly. Going to consider this *solved*. Thanks again for your input everyone.
Unfortunately, this is not an option for me, I need to scale horizontally.
I will follow the options of Takeshi (using ElasticCache) and Daniel (Using the database as cache instead of files.) But using GlusterFS.
I need to reach an acceptable value, around 3 ~ 5s.
I'm not using beanstalk.
In my case, as I'm migrating from a local EC2 instance to Docker, I have to use a remote file system.
Being simple in the explanation, I use Apache mounting the two volumes (html and date) and obviously creating a connection to the database.
Like I mentioned, Memcached (ElastiCache) did not work for me. As in, putting sessions in the cache and not file system did not solve the slowdown.
My hunch is that Moodledata mustache file caching is the culprit here, which can be something done on the web server locally. I wonder, if putting Moodledata just on the server (with session being in the database/elasticache) would be sufficient. More clearly stated, what parts of Moodledata actually need to be shared between servers?
Probably the user uploaded content, but I don't think that's what is causing the slowdown in this case.
I'll have to read more about it to understand.
I got some good results here.
Using GlusterFS + Local File Cache + Container
I spent two uninterrupted days testing and got a 2.41s performance as a time indicator on slow pages. I'm already satisfied with this result.
- dataroot on GlusterFS
- html on local disk
Awesome Danilo. I might try setting something like this up myself. While 2.41s is OK, it's hard to move away from the < 1s load times. Also, GlusterFS isn't really an Amazon service. And while I have no problem using 3rd party products, I'd love to stick within the cloud.
I'll try a few more things next week to see if I can speed up EFS or make EBS more bearable.
Thanks for all your findings. For those who want to try the GlusterFS route, seems like it might be OK.
To be more accurate is GlusterFS + Local File Cache + ECS with AutoScaling. All architecture in AWS.
2.41s is the slowest page load time ok, faster pages load around 1s or less.
EFS (I am AWS partner) only begins to perform with stores from 1TB, for a series of technical parameters of this service. Gluster, Freenas, or SoftNas, for example, installed in an EC2 instance has fewer technical limitations compared to EFS for small arrays. Of course it is possible to use EFS with little data stored, but for this moodle case, it does not seem to be good.
It was good to share information with everyone.
If you get good results with EFS, please tell us, if I got performance with EFS I would also use it.