Set up Moodle for 4000 users on Linux for my university

Set up Moodle for 4000 users on Linux for my university

by Tien Do -
Number of replies: 11

Thank you for your very detailed answer! Could you please tell me how to set up Moodle on Linux for my university (about 4000 users, max clients at the same time is about 500-800 users). I have 4 physical local server machines, which can be used for Moodle serving. Which way is better for installing Moodle - on 1 machine or on several machines? ( 1 for web, 1 for MySQL, 1 for moodle data).

Thanks in advance!

Average of ratings: -
In reply to Tien Do

Re: Moodle on Linux for my university (about 4000 users, max clients at the same time is about 500-800 users).

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

Hi

This is a different question from the discussion https://moodle.org/mod/forum/discuss.php?d=422441#p1703728 (watch the new subject line) and warrants a new thread. Your starting points are the documentation linked to the header of this forum and the "knowledge base" of similar past discussions.

You haven't provided any specifications of the machines. If they are state of the art machines, I would start with one, serving both the web server and the database server, and closely monitor the server load always extrapolating how long that machine can serve. (Think of the environment damage by excessive power consumption.) Once it becomes necessary, shift the database server to a second machine. It is unlikely that a server cluster is necessary for the numbers you mentioned.

Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Moodle on Linux for my university (about 4000 users, max clients at the same time is about 500-800 users).

by Mary Cooch -
Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators
Just to inform you both I have split this discussion from the earlier discussion on Scalablity, so no need to start a new thread, and thanks for pointing it out Visvanath.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moodle on Linux for my university (about 4000 users, max clients at the same time is about 500-800 users).

by Tien Do -
Yes, I'm confusing between vertical and horizontal scaling. I know that it is much simple when run both moodle and SQL on 1 machine but maybe in the future we need to scale up.
Average of ratings: -
In reply to Tien Do

Re: Set up Moodle for 4000 users on Linux for my university

by Usman Asar -
Picture of Plugin developers Picture of Testers
Tien, get the information on hardware specifications of all the machines down to chip-set level, (if they are different), else we'll see what upgrades are required on existing ones that can assure handling of 1000 concurrent users.
Average of ratings: Useful (1)
In reply to Usman Asar

Re: Set up Moodle for 4000 users on Linux for my university

by Tien Do -
Thank you for your reply! Here is my hardware specs:
- 2* Fujitsu RX2540 M2:24 - Xeon E5 2650 V3 / 2.3Ghz / 10 core - 20 threads/ 32Gb DDR4 / 2 * HDD SAS 10k 1.2TB Raid 1/
- 2* PowerEdge R340 - Xeon E-2124 3.3Ghz/ 8GB Ram / 1TB SAS + 1TB HDD 7k2 rpm
Are they enough for 600-800 concurrent users? I'm going to install Moodle 3.9 on Linux server 18.04 with MySQL/MariaDB. Is it worth updating to Linux 20.04 or 21.04 and Moodle 3.11? I need stability.
Thanks in advance
Average of ratings: -
In reply to Tien Do

Re: Set up Moodle for 4000 users on Linux for my university

by Usman Asar -
Picture of Plugin developers Picture of Testers
Tien, both types of boxes are perfect at their specifications, Fujitsu being older but more powerful, Dells are lesser powerful but are newer so can support Intel Optane memory as well. Only downside is they both are on SAS HDD's while database and webserver requires higher throughput and lower latency for which SSD's are better.

Will you be able to get drives changed/added? if you can add two additional 1TB SSD on RAID-1 on any of the fujitsu machines and put O/S, Web Server, Database, Moodle files and cache/temp files on SSD's and keep moodledata alone on HDD's that will surely handle 1000 users concurrent. if you can get RAM upgraded to 64GB that will be perfect, but 32GB is good enough as long as you allocate at least 12GB to Database server alone.

regarding Linux, I have absolutely no idea but 90%+ of the moodle are running on Linux so better ask community for suggestion of version. You can use any latest version of moodle with that hardware I just mentioned above. as of MariaDB 10.2+ and MySQL 8, I think they are not drop-in replacements anymore, so you'll be choosing either, out of which MariaDB is used by moodle so MariaDB is recommended.

you can even add one DELL machine with drives changed to SSD in RAID-1 to be used as database server alone, for expand-ability, and use memCache to reduce database latency.
Average of ratings: Useful (1)
In reply to Usman Asar

Re: Set up Moodle for 4000 users on Linux for my university

by Tien Do -
Thanks for your reply. That Fujitsu still has 16 more free slots for HDD/SSD. And it has 2 CPU so total it has 20 cores - 40 threads.
I will upgrade it to 64GB Ram, 2 SSD sata 3 in Raid-1, and keep HDD for moodle data as you advised.
But our internet speed is only 500MBs total for university and local network speed only 1GB/s. Could you please tell me how to forward all connections inside the university to use local network instead internet to save internet bandwidth? (We use a Mikrotik switch to apply all rules for internet and draytek for load balancing ).
Average of ratings: -
In reply to Tien Do

Re: Set up Moodle for 4000 users on Linux for my university

by Usman Asar -
Picture of Plugin developers Picture of Testers
Tien, you hadn't mentioned before having 2S platform,  though E5 comes in 2S scalability but not all servers with dual CPU sockets make use of that and instead uses a terminator CPU on 2nd socket for future upgrade. for your dual socket system, NUMA just entered the picture complicating the architecture of how a CPU talk to the RAM.

let me explain in easier way, on a dual/quad socket system, each CPU is allocated with its set of memory, by default O/S gets priority to use first CPU and its RAM, and if there is enough headroom then rest of the programs (say MariaDB) will be allocated space from first CPU's RAM and then once its full, rest to second CPU's RAM and so on, now if O/S has to allocate something and its running out of space on first CPU's RAM, what it will do is, swap out MariaDB's RAM allocation to 2nd CPU - This is where latency tax penalty comes in.

Now it depends on your BIOS, see if there is setting of RAM allocation, set it to "interleave" <<-- this is one way and will prevent premature swapping of database to 2nd CPU's RAM. Other possible way to counteract is turning NUMA Off all together if the O/S allows this. As far as I know, older MySQL/MariaDB required configuration in their conf files to set processor affinity, newer versions MAY have this settings in built to sense NUMA and configure itself for that architecture. You can read about it more at this MariaDB page HERE. I am afraid I am not database guru to assist in this further, Ken Task may step in to assist further.

As for your clients to use local traffic rather than internet, I am not sure how to set it at hardware level (i.e switches, routers etc - excuse my limited knowledge of networking) but one way is to set host file pointing domain to local IP, so you'll be adding this entry to your host file

www.mymoodle.com  192.168.2.100 <--- this will be your local IP allocated to the moodle server.
Average of ratings: Useful (1)
In reply to Usman Asar

Re: Set up Moodle for 4000 users on Linux for my university

by Tien Do -
I got your point. So I will add RAM for only CPU 1. How does it sound?
In BIOS, in Memory Configuration I see a line Memory Mode with 3 options: Normal, Mirroring and Sparing
On another line stands NUMA with 2 options: Disabled and Enabled.
There is no "Interleave" setting for RAM.
Should I disable NUMA and which mode for Memory I need to choose?
Average of ratings: -
In reply to Tien Do

Re: Set up Moodle for 4000 users on Linux for my university

by Usman Asar -
Picture of Plugin developers Picture of Testers
Tien, adding more RAM to one CPU and leaving other will lesser wont work, as it's not just the RAM, you have to consider CPU L1/L2/L3 cache's as well, as RAM can be increased and decreased, what will you do with cache's?

Keep memory mode to Normal, as others (Mirroring and Sparing) are for mission critical circumstances, Memory mirroring is as same as creating RAID 1 on storage, and in Sparing mode, one stick will be left spare until other memory module fails and spare will take over (memory level redundancy). but NUMA Enabled and Disabled is something that can work, so naturally you will set NUMA to be Disabled.

When I mentioned interleave, every server manufacturer has different BIOS options, where Fujitsu has enable/disable option, it should work.

As you'll be adding RAM, make sure all modules match and they are in correct banks, as I have seen buffoons from Dell sending servers with memory installed in wrong banks. there must be two colored banks, start populating the ones closest to CPU (e.g. if black one is nearest to CPU, then populate all black ones on either sides, unless every bank is populated in your machine. secondly all modules MUST match in terms of speed and latency.

Lastly, you mentioned having additional slots on server, are you referring to SATA connections or PCI slots? though SATA SSD's will work, but if you can add additional NVMe RAID card just for Database and rather than choosing SATA, get NVMe drives and keep database on that. The E5 platform unfortunately is not capable of booting through NVMe protocol drives, else I would have recommended NVMe boot drive and database, as SATA has latency of 30-100 microseconds while NVMe has just a few microseconds, and in database application latency does play its role.

Just to keep you informed, why I recommended moodledata on HDD's (so in case you have to justify your institute or have knowledge of what and why you're doing), when cron runs and your courses are backed up, it's moodledata folder that gets temporary copies made of the backups before saving it at your desired location, so for example if your courses are for 100GB, it will be writing 100GB of data on drive before creating backup file, and doing it everyday means by month end you'll have written approximately 3000GB (3TB approx) data already, SSD drives have a downside of number of writes, a standard SSD has life of 200TBW (Terabytes Written), datacentre SSD's (which you'll be getting) have much higher TBW's, but in the end they have life, so I recommended moodledata folder on HDD's so your drives can last longer. BUT do remember to route temp and cache folders from moodledata to SSD drives (due to latency). 
Average of ratings: Useful (1)