Hello there,
My district (K-12) is purchasing 2 Dell 1U servers that will function as application servers. In addition we are purchasing 1 Dell 2U (a beast of a machine, might I add) to function as the dB/NFS server. On our testing servers we have utilized TCP load balancing using the Linux Virtual Server Project (www.linuxvirtualserver.org) and we are using Ldirectord to "touch" a server to see if it is responding. We are using the masquerading method to do things. I'm wondering if anyone else is doing TCP load balancing, and what their experiances are with it.
Thanks.
--Richie
Mesa Public Schools
Mesa, AZ
Yep, the Moodle Cluster in NZ uses LVS + keepalived with great results. There are additional tricks to get the cronjobs running "load balanced" (we use dotlockfile), and quite a bit of tweaking to get NFS just right.
Don't forget to set LVS to do session affinity, or you'll see a lot of competition for the same session files. You might want to increase the directory hashing in moodledata/sessions -- if you do that you'll also want to do your own garbage collection there, as PHP's internal session GC can't deal with hashed directories.
Good idea to go high-end on the DB server. Very latest kernels on it recommended, as well as heavy disk tuning, separate disk spindles for the DB transaction log, carefully considered FS options, CFQv3 IO scheduler, etc.
Don't forget to set LVS to do session affinity, or you'll see a lot of competition for the same session files. You might want to increase the directory hashing in moodledata/sessions -- if you do that you'll also want to do your own garbage collection there, as PHP's internal session GC can't deal with hashed directories.
Good idea to go high-end on the DB server. Very latest kernels on it recommended, as well as heavy disk tuning, separate disk spindles for the DB transaction log, carefully considered FS options, CFQv3 IO scheduler, etc.
Martin,
Thank you for your suggestions, we will take them into consideration. I had heard that you were using TCP load balancing and LVS seemed to be the best option. We are actually modeling our Moodle installation after you guys. We're also using your config file that allows multiple Moodle subdomains without duplicating code.
I think we are going to utilize database sessions to handle the sessions, what are your recommendations on that?
Once again, Thanks
--Richie
Thank you for your suggestions, we will take them into consideration. I had heard that you were using TCP load balancing and LVS seemed to be the best option. We are actually modeling our Moodle installation after you guys. We're also using your config file that allows multiple Moodle subdomains without duplicating code.
I think we are going to utilize database sessions to handle the sessions, what are your recommendations on that?
Once again, Thanks
--Richie
Cool 
In terms of DB sessions, I think it may be easier to setup, but it won't be as fast, and will put a higher load on the DB server. Also, Postgres won't be the best option for DB sessions backend, as it tries too hard to be reliable (ACID) and that's overkill for session data.
If you are really expecting a lot of traffic and want to use DB sessions, consider patching the code to work with a 2nd DB server doing only sessions, possibly using MySQL with HEAP tables. It will by a 'lossy' but blazing fast DB server.
In terms of DB sessions, I think it may be easier to setup, but it won't be as fast, and will put a higher load on the DB server. Also, Postgres won't be the best option for DB sessions backend, as it tries too hard to be reliable (ACID) and that's overkill for session data.
If you are really expecting a lot of traffic and want to use DB sessions, consider patching the code to work with a 2nd DB server doing only sessions, possibly using MySQL with HEAP tables. It will by a 'lossy' but blazing fast DB server.
Martin,
How would you recommend going about setting up session affinity? Is this just done by increasing the persistance value in ipvsadm? If so, what would you recommend increasing it to?
Thanks,
Richie
How would you recommend going about setting up session affinity? Is this just done by increasing the persistance value in ipvsadm? If so, what would you recommend increasing it to?
Thanks,
Richie
Martin, how are you mirroring your moodledata directory (and for that matter the web root) across the web servers? You mention NFS from which I'd gather that it's shared from some central storage; what's the best method for achieving distributed storage instead, mirrored across each web server for redundancy and performance, given our farm doesn't have a suitable SAN or other centralised storage?
Ideas I'd looked at for mirroring partitions over TCP included InterMezzo/InterSync and RAID 10 ing them (software using NBD), using GFS for locking.
Thanks!
Ideas I'd looked at for mirroring partitions over TCP included InterMezzo/InterSync and RAID 10 ing them (software using NBD), using GFS for locking.
Thanks!
NFS is your friend. Access to moodledata isn't that aggressive, so NFS copes pretty well. If you have a SAN, that may be faster (if configured correctly). But NFS is ok if tuned correctly and used with newish Linux kernels (v2.6.14 or newer) and using the in-kernel server daemon.
Note however that the web root is _not_ on NFS (that'd be suicide from a performance/reliability point of view). And moodledata must _never_ be inside web root -- this is a basic security tenet. Web root is kept in sync using a combination of GIT, rsync and shell scripts.
We use RAID 10 on the server that serves NFS. GFS may be useful, I havent tested it. Intermezzo looked pretty green and not geared for performance when I looked at it.
Note however that the web root is _not_ on NFS (that'd be suicide from a performance/reliability point of view). And moodledata must _never_ be inside web root -- this is a basic security tenet. Web root is kept in sync using a combination of GIT, rsync and shell scripts.
We use RAID 10 on the server that serves NFS. GFS may be useful, I havent tested it. Intermezzo looked pretty green and not geared for performance when I looked at it.
Thanks again, just one further question; how well would Samba (SMB Client) perform in place of NFS for mounting/serving moodledata (our only suitable shared storage is running Windows) ?
There are several NFS implementations for Windows, including Microsoft's Services For Unix packages (I seem to remember last version is free as in beer).
Saludos. Iñaki.
Saludos. Iñaki.
Hi Rory,
If you haven't already, look at DRBD - http://www.linux-ha.org/DRBD/FAQ and http://wiki.linux-ha.org/DataRedundancyByDrbd.
SVN (for tools, benchmarks,etc.) - http://svn.drbd.org/drbd/
Viorel
If you haven't already, look at DRBD - http://www.linux-ha.org/DRBD/FAQ and http://wiki.linux-ha.org/DataRedundancyByDrbd.
SVN (for tools, benchmarks,etc.) - http://svn.drbd.org/drbd/
Viorel
Thanks; however DRBD is only a HA not load balancing system - only one active (as in read-write) node at a time unfortunately.
Correct, but talking about filesystem HA, how do you achieve it with only one NFS server?
I'm really looking for something that will cluster the filesystem across multiple nodes for both enhanced performance and redundancy, sort of RAID 1+0 over TCP. There is already a Windows DFS "cluster" on-site that does this (hence the questions about Samba - I'm not sure whether the MS Unix Services will serve a DFS and I'd trust Samba over Microsoft's implementation of NFS), but I'd rather have a linux-only setup for Moodle (
).
DRBD wasn't ideal as IMO it "wastes" a node when the cluster is fully operational; since Intermezzo isn't ready for production use, I'm thinking of giving PVFS (Parallel Virtual File System Version 2) a go which seems to do this and more, unless you can provide any more good ideas
.
Thanks everyone for all the help!
BTW The Performance pages in the Moodle wiki are still lacking on the topic of clustering...
).
DRBD wasn't ideal as IMO it "wastes" a node when the cluster is fully operational; since Intermezzo isn't ready for production use, I'm thinking of giving PVFS (Parallel Virtual File System Version 2) a go which seems to do this and more, unless you can provide any more good ideas
.
Thanks everyone for all the help!
BTW The Performance pages in the Moodle wiki are still lacking on the topic of clustering...