We are just planning to install Moodle and are looking to expand our student base rapidly over the next several years. Could any of you that have 5000 students or more please weigh in on your experiences with your architecture, and what you would propose as the preferred moodle architecture (we are using Postgres by the way) for both performance, system availability and scalability?
Thanks!
Tim
This wiki page has plenty of hints for your kind of question:
http://docs.moodle.org/en/Performance
Martin L has described the very large setups in NZ, which use a farm of webservers (you can start with two, and scale if needed) and a single very powerful postgres box. At UCL we're using a webserver plus a failover webserver, and a single MySQL mirrored by a slave MySQL as failover. Our student base is smaller than OPNZ's but still we're allowing for thousands of users.
In Glasgow they have dozens (literally) of separate Moodle installations, so the thousands of students are spread out over a large number of servers. This strikes me as an expensive route though...
http://docs.moodle.org/en/Performance
Martin L has described the very large setups in NZ, which use a farm of webservers (you can start with two, and scale if needed) and a single very powerful postgres box. At UCL we're using a webserver plus a failover webserver, and a single MySQL mirrored by a slave MySQL as failover. Our student base is smaller than OPNZ's but still we're allowing for thousands of users.
In Glasgow they have dozens (literally) of separate Moodle installations, so the thousands of students are spread out over a large number of servers. This strikes me as an expensive route though...
At what point does the multi-tier architecture start to degrade (is it fair to estimate that in number of students?)
As far as the Glasgow example, how do they ensure availability if each server is a dedicated, self-contained moodle server partitioned for performance by a maximum number of courses? Also, what maximum number of courses/students do they have per server to maintain good performance? (how many courses/students do they have that they require dozens of servers?)
As far as the Glasgow example, how do they ensure availability if each server is a dedicated, self-contained moodle server partitioned for performance by a maximum number of courses? Also, what maximum number of courses/students do they have per server to maintain good performance? (how many courses/students do they have that they require dozens of servers?)
The Glasgow example may not be the best thing to point to. The only reason they have lots of separate servers is because different departments set up their own independent ones, and/or wanted different setups. It's not generally necessary.
I don't understand your main question. What do you mean about the multi-tier architecture degrading? There are no hard and fast rules about the maximum number of concurrent users a single db or a single webserver can handle. No hard and fast rules at all...
I don't understand your main question. What do you mean about the multi-tier architecture degrading? There are no hard and fast rules about the maximum number of concurrent users a single db or a single webserver can handle. No hard and fast rules at all...
Now hosting multiple moodles on once cluster is interesting: it would certainly give overall better general performance than having separate servers for each installation, but you would have to some clever tweaking of the database to prevent too much swapping (or install a helluva lot of RAM!).
LAMP LVS clusters / true mosix beowulfs have near-linear scalability, which is why they're the typical model of choice for general large-scale web hosting, and is confirmed by my limited experience on our small cluster (4 identical nodes running mysql, apache, pvfs2 on gentoo; LVS proxied).
That's one of the beauties of it; in theory, to scale "out" you simply plug in more boxes.
LAMP LVS clusters / true mosix beowulfs have near-linear scalability, which is why they're the typical model of choice for general large-scale web hosting, and is confirmed by my limited experience on our small cluster (4 identical nodes running mysql, apache, pvfs2 on gentoo; LVS proxied).
That's one of the beauties of it; in theory, to scale "out" you simply plug in more boxes.
Note that Beowulf and similar "computing cluster" architectures are great at job throughtput but suck big rocks through thin straws when it comes to interactive jobs.
In other words, they work great for big batch jobs -- heavy number crunching. Setting up a new job is costly and slow, but once it is setup it goes fast (if it can be parallelised). Web apps are made out of lots and lots of tiny jobs that need to be done quickly. What end users perceive most is not so much performance but latency. And in a computing cluster latency is awful.
LVS is a radically different thing from Beowulf in that sense. We use LVS in the Moodle cluster. It is great -- some layers you can just plug in more boxes. _Some_ layers though. Disaggregating the RDBMS is hard, hard, hard hard. So the winning combo, IMHO, is a really powerful DB server machine, tuned heavily, and cheap pluggable Apache/PHP boxen, using LVS.
In other words, they work great for big batch jobs -- heavy number crunching. Setting up a new job is costly and slow, but once it is setup it goes fast (if it can be parallelised). Web apps are made out of lots and lots of tiny jobs that need to be done quickly. What end users perceive most is not so much performance but latency. And in a computing cluster latency is awful.
LVS is a radically different thing from Beowulf in that sense. We use LVS in the Moodle cluster. It is great -- some layers you can just plug in more boxes. _Some_ layers though. Disaggregating the RDBMS is hard, hard, hard hard. So the winning combo, IMHO, is a really powerful DB server machine, tuned heavily, and cheap pluggable Apache/PHP boxen, using LVS.
The two architectures on the table are:
1. a cluster of separate, self-contained Moodle servers such that each server would handle both web traffic and database requests for it's specific courses. Each server would be addressible by a separate URL or host name, therefore no "cluster latency" should exist. When the courseload/traffic gets to be too much, a new moodle server would be deployed and courses from the taxed server(s) would be published to the new server for the next term.
2. a load-balanced pool of web servers all writing to a master db server, which in turn replicates to a pool of read DB servers which is load-balanced for requests from the web servers. If Master write DB server fails, promote slave to master and replicate off of that. Seems this would address availabilty and optimize performance for students, but at some volume of student activity the write DB server would start to be overburdened (using traditional hardware).
Questions...
a. in architecture 1, what could be done to increase availability... can the Moodle servers be configured as backup for each other, perhaps in a ring, such that if a server fails, one of the others can be promoted to serve Moodle for the failed server until it is repaired?
b. in architecture 1, will we run into limitations because some students would be enrolled in courses that are hosted on separate servers?
c. in architecture 2, can Moodle differentiate DB reads from DB writes so that a write (master) can be kept separate from a load-balanced pool of read servers? (We are using Postgres, so DB cluster is not an option)
1. a cluster of separate, self-contained Moodle servers such that each server would handle both web traffic and database requests for it's specific courses. Each server would be addressible by a separate URL or host name, therefore no "cluster latency" should exist. When the courseload/traffic gets to be too much, a new moodle server would be deployed and courses from the taxed server(s) would be published to the new server for the next term.
2. a load-balanced pool of web servers all writing to a master db server, which in turn replicates to a pool of read DB servers which is load-balanced for requests from the web servers. If Master write DB server fails, promote slave to master and replicate off of that. Seems this would address availabilty and optimize performance for students, but at some volume of student activity the write DB server would start to be overburdened (using traditional hardware).
Questions...
a. in architecture 1, what could be done to increase availability... can the Moodle servers be configured as backup for each other, perhaps in a ring, such that if a server fails, one of the others can be promoted to serve Moodle for the failed server until it is repaired?
b. in architecture 1, will we run into limitations because some students would be enrolled in courses that are hosted on separate servers?
c. in architecture 2, can Moodle differentiate DB reads from DB writes so that a write (master) can be kept separate from a load-balanced pool of read servers? (We are using Postgres, so DB cluster is not an option)
I would not recommend option 1.
Answer to question c: No, I'm afraid not. Moodle uses a single ADODB connection, for both reads and writes. Plus various bits of code assume that as soon as something has been written, it's readable (which can't be assumed in the setup you suggest).
Answer to question c: No, I'm afraid not. Moodle uses a single ADODB connection, for both reads and writes. Plus various bits of code assume that as soon as something has been written, it's readable (which can't be assumed in the setup you suggest).
Martin,
When you speak of a really powerful DB server, how powerful are we talking here, without going overboard?
Is there such a thing as too much RAM on a Linux box?
Is a 64bit opteron going to give performance boosts over 32bit architecture?
Brgds
Brett
When you speak of a really powerful DB server, how powerful are we talking here, without going overboard?
Is there such a thing as too much RAM on a Linux box?
Is a 64bit opteron going to give performance boosts over 32bit architecture?
Brgds
Brett
Close your eyes and see...
A 4 or 8 or 16 way Opteron, running a custom-built 64-bit linux 2.6.16 or newer kernel. PostgreSQL and probably MySQL too have explicit optimizations for AMD64 architecture as it's really popular for DB servers -- and there are some optimisations in the kernel for memory access too as the acrhitecture for these babies is NUMA. Add UW-SCSI RAID-1 split in 2 or 3 sets of disks (keep your transaction log _physically_ separate form data storage!). Use carefully considered filesystem type and options for each partition.
Our OS of choice is Debian Sarge or Ubuntu Dapper but that's for ease of management. If you've done the above well, any modern 64-bit linux distro will do.
This will need 4GB or more to run well -- Memory is crucial though, make sure that when you pick the type and _arrangement_ of the memory you know what you are doing (consider NUMA). We often get our DB servers custom built, Sun's Fire boxes seem to be well designed though.
Remember to work on your kernel and Postgres (MySQL!) tunables to make sure it is well used.
And when you get to this you discover that you have to spend a _lot_ in a tape backup kit that can keep up with the beast!
(Feel free to edit and add to the wiki
)
A 4 or 8 or 16 way Opteron, running a custom-built 64-bit linux 2.6.16 or newer kernel. PostgreSQL and probably MySQL too have explicit optimizations for AMD64 architecture as it's really popular for DB servers -- and there are some optimisations in the kernel for memory access too as the acrhitecture for these babies is NUMA. Add UW-SCSI RAID-1 split in 2 or 3 sets of disks (keep your transaction log _physically_ separate form data storage!). Use carefully considered filesystem type and options for each partition.
Our OS of choice is Debian Sarge or Ubuntu Dapper but that's for ease of management. If you've done the above well, any modern 64-bit linux distro will do.
This will need 4GB or more to run well -- Memory is crucial though, make sure that when you pick the type and _arrangement_ of the memory you know what you are doing (consider NUMA). We often get our DB servers custom built, Sun's Fire boxes seem to be well designed though.
Remember to work on your kernel and Postgres (MySQL!) tunables to make sure it is well used.
And when you get to this you discover that you have to spend a _lot_ in a tape backup kit that can keep up with the beast!
(Feel free to edit and add to the wiki
Gulp......
Maybe we'll just look at hiring Catalyst if things ever get that big!
<Brett scurries back under the desk from where he came>
Cheers mate!
Brett
Maybe we'll just look at hiring Catalyst if things ever get that big!
<Brett scurries back under the desk from where he came>
Cheers mate!
Brett
Yes,
having 50 or so Moodle installations suits us, but it is more about the culture here than anything else. It is a pain from the administration point of view.
If you can get away with a small number of sites (or even 1) then that is the way to go.
Having said that, it works well for us, but we would like to reduce the number.
having 50 or so Moodle installations suits us, but it is more about the culture here than anything else. It is a pain from the administration point of view.
If you can get away with a small number of sites (or even 1) then that is the way to go.
Having said that, it works well for us, but we would like to reduce the number.