memcached and redis are enabled but moodledata, which is stored on a EFS (AWS's NFS implementation) is really slow according to benchmarks.
Is there anything else I can do to make the platform usable? Right now it's crawling.
Thanks!
I'm implementing a multiple frontend moodle site but file access is horrible.
memcached and redis are enabled but moodledata, which is stored on a EFS (AWS's NFS implementation) is really slow according to benchmarks.
Is there anything else I can do to make the platform usable? Right now it's crawling.
Thanks!
memcached and redis are enabled but moodledata, which is stored on a EFS (AWS's NFS implementation) is really slow according to benchmarks.
Is there anything else I can do to make the platform usable? Right now it's crawling.
Thanks!
In reply to Red Nano
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
https://docs.moodle.org/en/Performance_recommendations
Otherwise the "knowledge base" on performance is there in the form of past discussions in a dedicated forum: https://moodle.org/mod/forum/view.php?id=596.
Otherwise the "knowledge base" on performance is there in the form of past discussions in a dedicated forum: https://moodle.org/mod/forum/view.php?id=596.
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Red Nano -
The problem is that I'm stuck with this part:
I configured memcached and redis from the web ui but I understand I must manually configure these shared directories somehow.
That's what I can't wrap my head around abfter hours of diving through documentation.
//$CFG->tempdir = '/var/moodledata/temp'; // Directory MUST BE SHARED by all clsuter nodes.
//$CFG->cachedir = '/var/moodledata/cache'; // Directory MUST BE SHARED by all cluster nodes, locking required.
I configured memcached and redis from the web ui but I understand I must manually configure these shared directories somehow.
That's what I can't wrap my head around abfter hours of diving through documentation.
Do I have to create a distributed filesystem in order to share tose dirs between the frontends avoiding the current storage backend (NFS)?
In reply to Red Nano
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
Did you visit the links I've provided? If so, what have you found there?
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Red Nano -
Yes I did! But it still seems confusing.
For instance: I've followed the NFS thread that recommends some mount options to no improvement at all.
I've reviewed the cache wiki. Configured memcached and redis.
Redis for shared sessions, and meemcached on those cache definitions that I could.
However, none of this solves the issue with slow /var/moodledata/temp and /var/moodledata/cache directories, which have to be shared between all frontends and, as far as I can understand, must be shared with some sort of distributed filesystem (GlusterFS?), in order to avoid going via NFS
What I can't still understand is why is there no clear place to configure those directories on the UI since the only way I see to do so is editing the config.php file.
For instance: I've followed the NFS thread that recommends some mount options to no improvement at all.
I've reviewed the cache wiki. Configured memcached and redis.
Redis for shared sessions, and meemcached on those cache definitions that I could.
However, none of this solves the issue with slow /var/moodledata/temp and /var/moodledata/cache directories, which have to be shared between all frontends and, as far as I can understand, must be shared with some sort of distributed filesystem (GlusterFS?), in order to avoid going via NFS
What I can't still understand is why is there no clear place to configure those directories on the UI since the only way I see to do so is editing the config.php file.
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Red Nano -
OK; It seems like memcached and redis are configured properly.
The only issue here are the disk caches.
If I configure this:
$CFG->tempdir = '{{ ramdisk_path }}/temp'; // Directory MUST BE SHARED by all clsuter nodes.
$CFG->cachedir = '{{ ramdisk_path }}/cache'; // Directory MUST BE SHARED by all cluster nodes, locking required.
$CFG->localcachedir = '{{ ramdisk_path }}/localcache'; // Intended for local node caching.
The infrastructure works smootly.
Problem is, tempdir and cachedir MUST be shared between the frontends.
Now, how to achieve this? GlusterFS
And more importantly, how to do it so I can implement it on an elastic environment?
Finally, should I decide not to share those directories, can/should I work around this by setting sticky sessions?
The only issue here are the disk caches.
If I configure this:
$CFG->tempdir = '{{ ramdisk_path }}/temp'; // Directory MUST BE SHARED by all clsuter nodes.
$CFG->cachedir = '{{ ramdisk_path }}/cache'; // Directory MUST BE SHARED by all cluster nodes, locking required.
$CFG->localcachedir = '{{ ramdisk_path }}/localcache'; // Intended for local node caching.
The infrastructure works smootly.
Problem is, tempdir and cachedir MUST be shared between the frontends.
Now, how to achieve this? GlusterFS
And more importantly, how to do it so I can implement it on an elastic environment?
Finally, should I decide not to share those directories, can/should I work around this by setting sticky sessions?
In reply to Red Nano
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
Hi
You are moving too fast for me. I'm trying to follow.
First thing, your original sub-thread https://moodle.org/mod/forum/discuss.php?d=424148#p1708527 :
> $ dd if=/dev/zero of=/moodle_ramdisk/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 0.0411606 s, 2.5 GB/s
> $ dd if=/dev/zero of=/var/moodledata/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 30.6057 s, 3.4 MB/s
You can't compare RAM to disk storage, whether local or network mounted. Still 3.4 MB/s is broken. Here are three measurements for comparison; first one from a high-end VPS, second one from a state-of-art dedicated server, SSD drive and the third from the same server SATA drive:
# dd if=/dev/zero of=/var/www/moodledata/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.0804138 s, 1.3 GB/s
# dd if=/dev/zero of=/var/www/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.10178 s, 1.0 GB/s
# dd if=/dev/zero of=/var/www/data/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.105108 s, 998 MB/s
You must ask your server provider what you are doing wrong. If they can't bring the data rate up to 125 MB/s, which is 1 Gbit/s, the likely bandwidth of the LAN, you have to forget them. (I assume a network file system is a must for your project.)
Second one, your first answer https://moodle.org/mod/forum/discuss.php?d=424148#p1708539 :
> I've followed the NFS thread
Which one? The link please.
> that recommends some mount options to no improvement at all.
Those recommendations are? Where? What were your earlier options? What were your corrected options? What was the data transfer rate before and after?
> I've reviewed the cache wiki.
Same thing. Where is the link?
> Configured memcached and redis.
> Redis for shared sessions, and meemcached on those cache definitions that I could.
How exactly?
> However, none of this solves the issue with slow /var/moodledata/temp and /var/moodledata/cache directories,
The same 'dd' result 3.3 MB/s?
> which have to be shared between all frontends
See the comment on "cache wiki" above.
> and, as far as I can understand, must be shared with some sort of distributed filesystem (GlusterFS?), in order to avoid going via NFS
References please!
You are moving too fast for me. I'm trying to follow.
First thing, your original sub-thread https://moodle.org/mod/forum/discuss.php?d=424148#p1708527 :
> $ dd if=/dev/zero of=/moodle_ramdisk/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 0.0411606 s, 2.5 GB/s
> $ dd if=/dev/zero of=/var/moodledata/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 30.6057 s, 3.4 MB/s
You can't compare RAM to disk storage, whether local or network mounted. Still 3.4 MB/s is broken. Here are three measurements for comparison; first one from a high-end VPS, second one from a state-of-art dedicated server, SSD drive and the third from the same server SATA drive:
# dd if=/dev/zero of=/var/www/moodledata/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.0804138 s, 1.3 GB/s
# dd if=/dev/zero of=/var/www/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.10178 s, 1.0 GB/s
# dd if=/dev/zero of=/var/www/data/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.105108 s, 998 MB/s
You must ask your server provider what you are doing wrong. If they can't bring the data rate up to 125 MB/s, which is 1 Gbit/s, the likely bandwidth of the LAN, you have to forget them. (I assume a network file system is a must for your project.)
Second one, your first answer https://moodle.org/mod/forum/discuss.php?d=424148#p1708539 :
> I've followed the NFS thread
Which one? The link please.
> that recommends some mount options to no improvement at all.
Those recommendations are? Where? What were your earlier options? What were your corrected options? What was the data transfer rate before and after?
> I've reviewed the cache wiki.
Same thing. Where is the link?
> Configured memcached and redis.
> Redis for shared sessions, and meemcached on those cache definitions that I could.
How exactly?
> However, none of this solves the issue with slow /var/moodledata/temp and /var/moodledata/cache directories,
The same 'dd' result 3.3 MB/s?
> which have to be shared between all frontends
See the comment on "cache wiki" above.
> and, as far as I can understand, must be shared with some sort of distributed filesystem (GlusterFS?), in order to avoid going via NFS
References please!
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Red Nano -
First thing, your original sub-thread https://moodle.org/mod/forum/discuss.php?d=424148#p1708527 :
> $ dd if=/dev/zero of=/moodle_ramdisk/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 0.0411606 s, 2.5 GB/s
> $ dd if=/dev/zero of=/var/moodledata/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 30.6057 s, 3.4 MB/s
You can't compare RAM to disk storage, whether local or network mounted. Still 3.4 MB/s is broken. Here are three measurements for comparison; first one from a high-end VPS, second one from a state-of-art dedicated server, SSD drive and the third from the same server SATA drive:
# dd if=/dev/zero of=/var/www/moodledata/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.0804138 s, 1.3 GB/s
# dd if=/dev/zero of=/var/www/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.10178 s, 1.0 GB/s
# dd if=/dev/zero of=/var/www/data/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.105108 s, 998 MB/s
You must ask your server provider what you are doing wrong. If they can't bring the data rate up to 125 MB/s, which is 1 Gbit/s, the likely bandwidth of the LAN, you have to forget them. (I assume a network file system is a must for your project.)
--> I know I can't compare both systems but still, speed is quite slow: No wonder moodle can't perform properly.
The backend is AWS EFS which is a NFS "flavour" - modified by amazon.
Second one, your first answer https://moodle.org/mod/forum/discuss.php?d=424148#p1708539 :
> I've followed the NFS thread
Which one? The link please.
--> This thread: https://moodle.org/mod/forum/discuss.php?d=310501#p1242382
> that recommends some mount options to no improvement at all.
Those recommendations are? Where? What were your earlier options? What were your corrected options? What was the data transfer rate before and after?
--> I don't have the numbers right now, but no difference at all.
> I've reviewed the cache wiki.
Same thing. Where is the link?
--> https://docs.moodle.org/311/en/Caching - If I understand correctly, configuring cache from moodle UI does not include disk cache or modifying what gets stored under /var/moodledata/cache, so after configuring memcache and redis following this doc, I understand there's no further improvement to be gained from here.
> Configured memcached and redis.
> Redis for shared sessions, and meemcached on those cache definitions that I could.
How exactly?
--> Moodle administration -> Plugins -> Cache -> Redis: Added a redis instance and a memcache instance, then defined at the bottom of the same page for the dafault definitions.
> However, none of this solves the issue with slow /var/moodledata/temp and /var/moodledata/cache directories,
The same 'dd' result 3.3 MB/s?
--> Yup, same result.
> which have to be shared between all frontends
See the comment on "cache wiki" above.
> and, as far as I can understand, must be shared with some sort of distributed filesystem (GlusterFS?), in order to avoid going via NFS
References please!
--> https://docs.moodle.org/37/en/Performance_recommendations#Hardware_configuration
"If your 'moodledata' area is going to be on relatively slow storage (e.g. NFS mount on to a NAS device) you will have performance issues with the default cache configuration (which writes to this storage). See the page on Caching and choose an alternative. Redis is recommended. Using GlusterFS / OCFS2 / GFS2 on a SAN device and Fiber Channel could improve performance (See more info on the Moodle forum thread, NFS performance tuing )"
I tried sharing /var/moodledata/cache and /var/moodledata/temp between the frontends by mounting a remote NFS share (Not EFS this time, but NFS) and the overall speed improved on the moodle benchmark: From the first screenshot, I reduced the write times from that horrendous "40" to "8", which is still far from the recommended values (and the platform is still slower than what it should)
In short: I see lots, like TONS of documentation that date from 3 to 10 years in age, some chunks of the documentation are a mess, vague and don't really give a true solution to the issues.
I don't think I'm the only one that is deploying a distributed moodle setup in AWS and yet, there's no definitive write-up on how to do so properly.
I'm kind of desperate and don't know how to go ahead and finish this depluy and have spent way to many hours trying to improve it to no success...
I really appreciate your help: Please don't give up on me.
> $ dd if=/dev/zero of=/moodle_ramdisk/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 0.0411606 s, 2.5 GB/s
> $ dd if=/dev/zero of=/var/moodledata/testfile bs=1M count=100
> ...
> 104857600 bytes (105 MB, 100 MiB) copied, 30.6057 s, 3.4 MB/s
You can't compare RAM to disk storage, whether local or network mounted. Still 3.4 MB/s is broken. Here are three measurements for comparison; first one from a high-end VPS, second one from a state-of-art dedicated server, SSD drive and the third from the same server SATA drive:
# dd if=/dev/zero of=/var/www/moodledata/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.0804138 s, 1.3 GB/s
# dd if=/dev/zero of=/var/www/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.10178 s, 1.0 GB/s
# dd if=/dev/zero of=/var/www/data/testfile bs=1M count=100
...
104857600 bytes (105 MB, 100 MiB) copied, 0.105108 s, 998 MB/s
You must ask your server provider what you are doing wrong. If they can't bring the data rate up to 125 MB/s, which is 1 Gbit/s, the likely bandwidth of the LAN, you have to forget them. (I assume a network file system is a must for your project.)
--> I know I can't compare both systems but still, speed is quite slow: No wonder moodle can't perform properly.
The backend is AWS EFS which is a NFS "flavour" - modified by amazon.
Second one, your first answer https://moodle.org/mod/forum/discuss.php?d=424148#p1708539 :
> I've followed the NFS thread
Which one? The link please.
--> This thread: https://moodle.org/mod/forum/discuss.php?d=310501#p1242382
> that recommends some mount options to no improvement at all.
Those recommendations are? Where? What were your earlier options? What were your corrected options? What was the data transfer rate before and after?
--> I don't have the numbers right now, but no difference at all.
> I've reviewed the cache wiki.
Same thing. Where is the link?
--> https://docs.moodle.org/311/en/Caching - If I understand correctly, configuring cache from moodle UI does not include disk cache or modifying what gets stored under /var/moodledata/cache, so after configuring memcache and redis following this doc, I understand there's no further improvement to be gained from here.
> Configured memcached and redis.
> Redis for shared sessions, and meemcached on those cache definitions that I could.
How exactly?
--> Moodle administration -> Plugins -> Cache -> Redis: Added a redis instance and a memcache instance, then defined at the bottom of the same page for the dafault definitions.
> However, none of this solves the issue with slow /var/moodledata/temp and /var/moodledata/cache directories,
The same 'dd' result 3.3 MB/s?
--> Yup, same result.
> which have to be shared between all frontends
See the comment on "cache wiki" above.
> and, as far as I can understand, must be shared with some sort of distributed filesystem (GlusterFS?), in order to avoid going via NFS
References please!
--> https://docs.moodle.org/37/en/Performance_recommendations#Hardware_configuration
"If your 'moodledata' area is going to be on relatively slow storage (e.g. NFS mount on to a NAS device) you will have performance issues with the default cache configuration (which writes to this storage). See the page on Caching and choose an alternative. Redis is recommended. Using GlusterFS / OCFS2 / GFS2 on a SAN device and Fiber Channel could improve performance (See more info on the Moodle forum thread, NFS performance tuing )"
I tried sharing /var/moodledata/cache and /var/moodledata/temp between the frontends by mounting a remote NFS share (Not EFS this time, but NFS) and the overall speed improved on the moodle benchmark: From the first screenshot, I reduced the write times from that horrendous "40" to "8", which is still far from the recommended values (and the platform is still slower than what it should)
In short: I see lots, like TONS of documentation that date from 3 to 10 years in age, some chunks of the documentation are a mess, vague and don't really give a true solution to the issues.
I don't think I'm the only one that is deploying a distributed moodle setup in AWS and yet, there's no definitive write-up on how to do so properly.
I'm kind of desperate and don't know how to go ahead and finish this depluy and have spent way to many hours trying to improve it to no success...
I really appreciate your help: Please don't give up on me.
In reply to Red Nano
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
That was an explosion of topics considering that the initial question was, "Why is my MoodleBench result is so bad?" Rather than getting entangles, lean back and take your bearings.
1. Rather than asking why I can't do this, tell where you want to go.
Examples: I want to launch a million user Moodle site. Or, 100 Moodle sites of 10 k users each. And with numbers of the actual usage. Say, up to 10 k requests (user clicks, client Js connections) within a window of 1 minute. Also, the distribution of those requests, forum posts, file downloads, file uploads (how big), number of candidates of synchronous MC exams, group interactions (chat, gamification), interactive activities (lesson, H5P).
Depending on the answer you might have to cluster or not.
2. In case a cluster is must, why neither the Moodle developers nor the community doesn't tell you how?
This site, moodle.org, is about the standard Moodle, its official plug-ins and the mobile app: https://download.moodle.org/. Our subject is the first one, the standard Moodle. It is just a collection of scripts, PHP and JS, a single directory tree, not packages any other way than simple zip archives and a Git repository. That's it.
The world is free, yes Moodle is Free and Open Source Software, to run it in DIY style. Its requirements are modest, and very clear: "Moodle is primarily developed in Linux using Apache, PostgreSQL/MySQL/MariaDB and PHP (sometimes known as the LAMP platform). Typically this is also how Moodle is run," https://docs.moodle.org/311/en/Installing_Moodle#Requirements.
It goes further, "although there are other options as long as the software requirements of the release are met." That is where you are. You need to know your platform. If it is very much different, you might have to look for its own community. For example https://discuss.moodlebox.net/ for Moodle on Raspberry Pi. Of course, there is no harm sharing your experience about other platforms here, provided there are others. As you are already experiencing there are not much AWS support here. You might want to try A new "Large scale Moodle deployment" support group.
1. Rather than asking why I can't do this, tell where you want to go.
Examples: I want to launch a million user Moodle site. Or, 100 Moodle sites of 10 k users each. And with numbers of the actual usage. Say, up to 10 k requests (user clicks, client Js connections) within a window of 1 minute. Also, the distribution of those requests, forum posts, file downloads, file uploads (how big), number of candidates of synchronous MC exams, group interactions (chat, gamification), interactive activities (lesson, H5P).
Depending on the answer you might have to cluster or not.
2. In case a cluster is must, why neither the Moodle developers nor the community doesn't tell you how?
This site, moodle.org, is about the standard Moodle, its official plug-ins and the mobile app: https://download.moodle.org/. Our subject is the first one, the standard Moodle. It is just a collection of scripts, PHP and JS, a single directory tree, not packages any other way than simple zip archives and a Git repository. That's it.
The world is free, yes Moodle is Free and Open Source Software, to run it in DIY style. Its requirements are modest, and very clear: "Moodle is primarily developed in Linux using Apache, PostgreSQL/MySQL/MariaDB and PHP (sometimes known as the LAMP platform). Typically this is also how Moodle is run," https://docs.moodle.org/311/en/Installing_Moodle#Requirements.
It goes further, "although there are other options as long as the software requirements of the release are met." That is where you are. You need to know your platform. If it is very much different, you might have to look for its own community. For example https://discuss.moodlebox.net/ for Moodle on Raspberry Pi. Of course, there is no harm sharing your experience about other platforms here, provided there are others. As you are already experiencing there are not much AWS support here. You might want to try A new "Large scale Moodle deployment" support group.
Sorry, No, I don't have live serves on AWS and other "public clouds". Happened to come across this talk after a random search on the Web:
Cloud infrastructure for high concurrency
Jordi Molina, Antoni Bertran
MoodleMoot Global 2019 | November 20 | Perth Room - Afternoon Part 2
(first 20 min, acoustics is bad, but excellent sub-titles)
- Slides https://moodle.com/wp-content/events/mootglobal19/Cloudinfrastructureforhighconcurrency.pdf
- Catch phrase: "Clean up and watch out for your bill!" 14:15
P.S. The quoting style I used earlier is the one which was common in Usenet. It goes like,
===
>>> What you wrote in the second previous post
>> What I wrote in my previous post
> What you wrote in the post I am referring to
What I am saying now
===
Not that you don't have to follow. But if you adopt it, it is easier for the reader if you follow the same pattern. See https://en.wikipedia.org/wiki/Posting_style#Interleaved_style.
P.P.S. I am reluctant to talk to people in "hiding", I mean the pseudonym. My intention of the first reply was to point you to the correct forum, not more. I hope, from whatever information you give in your next reply, somebody will come to the rescue.
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
(timed out)
This is where you need to "Clean up and watch out for your bill!"
This is where you need to "Clean up and watch out for your bill!"
Basically, this is my issue:
# Test to ramdisk
ubuntu@ip-192-168-80-59:~$ dd if=/dev/zero of=/moodle_ramdisk/testfile bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0411606 s, 2.5 GB/s
# Test to moodledata on EFS/NFS
ubuntu@ip-xx-xx-xx:~$ dd if=/dev/zero of=/var/moodledata/testfile bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 30.6057 s, 3.4 MB/s
What I don't get is how do I share the cache directories between all the frontends... glusterfs?
# Test to ramdisk
ubuntu@ip-192-168-80-59:~$ dd if=/dev/zero of=/moodle_ramdisk/testfile bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0411606 s, 2.5 GB/s
# Test to moodledata on EFS/NFS
ubuntu@ip-xx-xx-xx:~$ dd if=/dev/zero of=/var/moodledata/testfile bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 30.6057 s, 3.4 MB/s
What I don't get is how do I share the cache directories between all the frontends... glusterfs?
Moving to Hardware and performance forum...
I haven't read all this (TL;DR) but there's a fundamental point - don't try to use NFS for anything performance critical. It's too slow. It's always going to be too slow.
- Use Redis for the cache (configured in admin settings)
- Use Redis for sessions (configured in config.php)
- Use the database for file locking.
- Use Redis for the cache (configured in admin settings)
- Use Redis for sessions (configured in config.php)
- Use the database for file locking.
In reply to Howard Miller
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
Howard
Great summary, short and sweet.
Mein tries to convey a fundamental question before the fundamental point: the question whether clustering is necessary. And the fact that Moodle does not owe anybody a ready-made cluster solution.
Great summary, short and sweet.
Mein tries to convey a fundamental question before the fundamental point: the question whether clustering is necessary. And the fact that Moodle does not owe anybody a ready-made cluster solution.
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Howard Miller -
Moodle is remarkably simple to configure in a cluster. But you end up with lots of bits and you need to understand all those bits. You now probably have cache servers, NFS, load balancers, remote databases, blah blah blah. And you now need to properly understand all this stuff. That's the hard bit. The Moodle bit is easy.
EDIT:
I was amused to see you getting pulled up for not using proper Usenet quoting. The last time I heard that I still had hair 
Eric Raymond will be here soon
In reply to Howard Miller
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
Ha, ha, ha! Another hilarious one! Apparently I have missed the right time for retirement.
;-(
Seriously, now the "clustering in the cloud" type of questions multiplying, see https://moodle.org/mod/forum/discuss.php?d=424130#p1708520 for example, I wonder whether there should be a link in the forum header pointing to the Telegram chat https://t.me/large_scale_moodle of the specialists at 'A new "Large scale Moodle deployment" support group' https://moodle.org/mod/forum/discuss.php?d=414040 - like the link in the General developer forum https://moodle.org/mod/forum/view.php?id=55 pointing to the Telegram Moodle Dev Chat room https://telegram.me/moodledev.
;-(
Seriously, now the "clustering in the cloud" type of questions multiplying, see https://moodle.org/mod/forum/discuss.php?d=424130#p1708520 for example, I wonder whether there should be a link in the forum header pointing to the Telegram chat https://t.me/large_scale_moodle of the specialists at 'A new "Large scale Moodle deployment" support group' https://moodle.org/mod/forum/discuss.php?d=414040 - like the link in the General developer forum https://moodle.org/mod/forum/view.php?id=55 pointing to the Telegram Moodle Dev Chat room https://telegram.me/moodledev.
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Howard Miller -
I start to get a bit cynical now.... "The Cloud" is just marketing BS for somebody else's computer. I've used AWS in the sense that I had a few Ubuntu Server images on there. However, they like to give the impression that they have some sort of dark magic going on and give everything weird names to make it worse. However, you can clearly set up some Linux web servers, a cache, a database and an NFS share from their menu the same as you can anywhere else. They're not reinventing the wheel.
I'd rather keep the discussion here, personally, if nothing else in an attempt to demystify this stuff.
In reply to Howard Miller
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
I understand you fully, it is hard not to be cynical. You say AWS is Ubuntu, possibly because you have the background, you lived through going backwards Debian, GNU, other Linux distros and to the root Unix. For those who talk "cloud" AWS is not all that. You can't even explain, because they talk a different language, of things like droplets, buckets and elastics. I assume it is the AWS marketing language. I presume the "Large scale Moodle deployment support group" talk that. (Well, don't ask me how they communicate with the Azure group, or Google Cloud or ..)
I wasn't happy with the split. See my replies in https://moodle.org/mod/forum/discuss.php?d=414040. Now it is there, why not make use of it? But you're the boss, I can conveniently look away when visitors descent from the clouds. No problem at all, seriously.
Less seriously, look at this, just broke today:
I wasn't happy with the split. See my replies in https://moodle.org/mod/forum/discuss.php?d=414040. Now it is there, why not make use of it? But you're the boss, I can conveniently look away when visitors descent from the clouds. No problem at all, seriously.
Less seriously, look at this, just broke today:
New mystery AWS product ‘Infinidash’ goes viral — despite being entirely fictional
Reg chats to developer whose joke that mere mention of a new prod would appear in job ads came true and spawned books, songs, forks, cryptocoin, and more
https://www.theregister.com/2021/07/05/infinidash/
https://www.theregister.com/2021/07/05/infinidash/
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Howard Miller -
You can ask for a "vanilla" Ubuntu or CentOS image on AWS if that's what you want. You don't have to use all the fancy stuff.
From everything else I've seen, it's all just common server applications that you can spin up but they give them mad names to add a bit of mystique. Don't get me wrong, it's handy to be able to get a Linux server and a MySQL server set up in two minutes with a couple of clicks.
From everything else I've seen, it's all just common server applications that you can spin up but they give them mad names to add a bit of mystique. Don't get me wrong, it's handy to be able to get a Linux server and a MySQL server set up in two minutes with a couple of clicks.
Thanks to both of you for the help. 
FIrst of all, I just implement stuff. There's a client who wants to implement clustering for their moodle environment, so that's what I did: Got myself into AWS ecosystem, know the bits and parts, how to make them work together but there's something I'm nissing from moodle here.
- Memcached is configured for cache (I can try redis as you suggest)
- Redis is configured for sessions
- I could implement database for file locking
But there's still a very important bit which is cache and temp directories, which must be shared across fronetends.
And why do I obsess wih this part, you might ask: Well... one of the ways I know to test the overall platform's speed and performance is the screenshot's test.
I try to get all greens and the only ones showing red are the scores related to (as the same menu / screenshot says) "the moodle temporary folder", which is one of the folders that must be shared between frontends.
That's why I think here's the issue. And I can only see two ways to fix this.
- Clustering (glusterfs)
- NFS shares
- EFS shares (Which are painfully slow because they athrottled depending of the EFS space occupied - so this option is discarded)
From the remaining two options, both give me similar performance scores (Tried glusterfs on a previous test half a year ago)
Here is the point in which I can't go forward and wanted to now how other people have implemented this part so I can follow the example and try myself because this is the last bit that is preventing me from finishing the deployment and I have already been spending a lot of hours trying to improve it to no gain but the one mentioned earlier, when I shifted from EFS to a dedicated NFS server instead for those temp and cache directories)
PS: I still have hair, so I did not live usenet myself :P Sorry if my markdown / format is wrong.
FIrst of all, I just implement stuff. There's a client who wants to implement clustering for their moodle environment, so that's what I did: Got myself into AWS ecosystem, know the bits and parts, how to make them work together but there's something I'm nissing from moodle here.
- Memcached is configured for cache (I can try redis as you suggest)
- Redis is configured for sessions
- I could implement database for file locking
But there's still a very important bit which is cache and temp directories, which must be shared across fronetends.
And why do I obsess wih this part, you might ask: Well... one of the ways I know to test the overall platform's speed and performance is the screenshot's test.
I try to get all greens and the only ones showing red are the scores related to (as the same menu / screenshot says) "the moodle temporary folder", which is one of the folders that must be shared between frontends.
That's why I think here's the issue. And I can only see two ways to fix this.
- Clustering (glusterfs)
- NFS shares
- EFS shares (Which are painfully slow because they athrottled depending of the EFS space occupied - so this option is discarded)
From the remaining two options, both give me similar performance scores (Tried glusterfs on a previous test half a year ago)
Here is the point in which I can't go forward and wanted to now how other people have implemented this part so I can follow the example and try myself because this is the last bit that is preventing me from finishing the deployment and I have already been spending a lot of hours trying to improve it to no gain but the one mentioned earlier, when I shifted from EFS to a dedicated NFS server instead for those temp and cache directories)
PS: I still have hair, so I did not live usenet myself :P Sorry if my markdown / format is wrong.
Ok - I think you're really confused here.
You said you had memcached configured for the cache. Great. I don't know why you would use both memcached and redis when you can use redis for the whole lot - but let's skip that.
...but then you start to talk about cache and temp directories "which must be shared". You just said you have memcached configured for the cache. Do you or don't you?
What do you think you mean by 'temp directories'? By default, Moodle creates a temp folder in 'moodledata' (NFS to you). It's not performance critical - don't worry about it.
I don't know what glusterfs and so on is, so I can't comment on that (well I do, but I don't know anything about it). Also bear in mind that you are getting hung up on a performance report that some random dude wrote. It's neither official or part of core Moodle. If you have set up Moodle properly then it should be fine. All that matters is page load time. Have you run the built in performance report just in case you're missing something stupid (we all do it).
You can quote however you like. As long as we understand you. No old-skool Unix Geeks here
You said you had memcached configured for the cache. Great. I don't know why you would use both memcached and redis when you can use redis for the whole lot - but let's skip that.
...but then you start to talk about cache and temp directories "which must be shared". You just said you have memcached configured for the cache. Do you or don't you?
What do you think you mean by 'temp directories'? By default, Moodle creates a temp folder in 'moodledata' (NFS to you). It's not performance critical - don't worry about it.
I don't know what glusterfs and so on is, so I can't comment on that (well I do, but I don't know anything about it). Also bear in mind that you are getting hung up on a performance report that some random dude wrote. It's neither official or part of core Moodle. If you have set up Moodle properly then it should be fine. All that matters is page load time. Have you run the built in performance report just in case you're missing something stupid (we all do it).
You can quote however you like. As long as we understand you. No old-skool Unix Geeks here
In reply to Howard Miller
Re: How do I improvbe file read / write on NFS (EFS)?
by Visvanath Ratnaweera -
A note on Moodle Benchmark https://moodle.org/plugins/report_benchmark. It is what the name says, _a_ benchmark. Not a load test https://en.wikipedia.org/wiki/Load_testing. Sure you get a set of numbers, but there is no formula to translate them to numbers what Moodle administrators want to know, see https://moodle.org/mod/forum/discuss.php?d=393752. The practical use for me is if Moodle Benchmark brings warnings you have to take them seriously. But no warnings does not necessarily mean a highly scalable site. For details read the code, it is not complex.
The less helpful message is, this forum is a true knowledge base, if you take the trouble to go through it.
In reply to Visvanath Ratnaweera
Re: How do I improvbe file read / write on NFS (EFS)?
by Howard Miller -
...I have no idea where the "acceptable limits" came from. Says who?
NFS is fine with Moodle as long as it doesn't do any caching or user sessions.
With Redis being used for the MUC Application cache, MUC Session cache and PHP session storage, the moodledata dir on NFS only really is used for file downloads.
There have been lots of reports of EFS being slow on AWS as you get throttled depending on everyone else using their storage. Running your own NFS on AWS was a recommendation I have seen previously.
With Redis being used for the MUC Application cache, MUC Session cache and PHP session storage, the moodledata dir on NFS only really is used for file downloads.
There have been lots of reports of EFS being slow on AWS as you get throttled depending on everyone else using their storage. Running your own NFS on AWS was a recommendation I have seen previously.
Ok, so there's something I'm really getting wrong here which is the part of the shared directories.
I will have anothe rlook to the cache config page on my setup and see if I have everything configured as it's supposed.
Thanks!
I will have anothe rlook to the cache config page on my setup and see if I have everything configured as it's supposed.
Thanks!
So, I found the culprit.
I made the error of trusting this benchmark as the definitive metric to check if my infrastructure was correctly configured but I was so wrong.
So, I did as you suggested and used memcached for both caching and sessions.
Login took around 120 seconds, which was very odd. After that delay, the site seemed to work properly.
Benchmark aside, I took some days investigating this delay until I found this thread: https://moodle.org/mod/forum/discuss.php?d=408647
And bingo... as the thread's OP, I'm migrating an already working configuration.
Full disclosure: I'm an infrastructure guy and my knowledge in moodle is not null but it's not that great either.
So, once I identified the external database that was configured and checked that indeed, it was pointing to a local IP adress from the old environment, I only had to disable it and now everything is up and running.
No glusterfs, no nfs for shared sessiond, just plain old slow EFS.... no more config file definitions for cache and temp dirs..... and the platform is a wonder in cluster configuration.
Thanks for the guidance to everyone that contributed.
I made the error of trusting this benchmark as the definitive metric to check if my infrastructure was correctly configured but I was so wrong.
So, I did as you suggested and used memcached for both caching and sessions.
Login took around 120 seconds, which was very odd. After that delay, the site seemed to work properly.
Benchmark aside, I took some days investigating this delay until I found this thread: https://moodle.org/mod/forum/discuss.php?d=408647
And bingo... as the thread's OP, I'm migrating an already working configuration.
Full disclosure: I'm an infrastructure guy and my knowledge in moodle is not null but it's not that great either.
So, once I identified the external database that was configured and checked that indeed, it was pointing to a local IP adress from the old environment, I only had to disable it and now everything is up and running.
No glusterfs, no nfs for shared sessiond, just plain old slow EFS.... no more config file definitions for cache and temp dirs..... and the platform is a wonder in cluster configuration.
Thanks for the guidance to everyone that contributed.
"...made the error of trusting this benchmark as the definitive metric..."
I usually think it's pretty low to say "I told you so"... but....