Best practices for mirrored production environment

Best practices for mirrored production environment

by Daniel Wolff -
Number of replies: 10

Greetings all,

First off, a big thank you to this community for helping me get Moodle up and running on a CENTOS Lamp Stack. It was my first time doing a full install and many of you, especially Ken, helped address my concerns.

I was wondering if others have experience or know of any best practices for setting up a three machine (four machine including disaster recovery) of moodle on LAMP. What I've envisioned with our IT department here is a sandbox/dev machine, then two mirrored production servers, and a fourth disaster recovery machine. The mirrored production servers exist in separate locations with the hope that if one went down the other would kick on and traffic would be redirected there.

Our IT department was planning on syncing the databases together - would that be sufficient? I would think you would need to sync both /moodle/ and /moodledata/ in order to ensure that plugins, themes, etc would be replicated. Should we be syncing other things as well?

Do you folks see any red flags in trying to do this?

Thank you all for your time and consideration.

Regards,

Dan

Average of ratings: -
In reply to Daniel Wolff

Re: Best practices for mirrored production environment

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

Unless you tell us what you want to achieve, nobody can tell you what you are doing is right!

Do you expect a huge flood of users the very first few months? Or, is Moodle so business critical that you need a second (perhaps third) hot fail-over to take over without users noticing anything? Or, are your people do such a heavy course development work that it would slow down the production server?

(I know, it is everything. But we can't talk about everything at the same time. Definitely not in a single forum thread!)

P.S. I saw the previous discussion "First time install of moodle on CENTOS 6.5" https://moodle.org/mod/forum/discuss.php?d=269952#p1164328. But Ken hasn't earned a single Karma (Rate as useful) point?
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Best practices for mirrored production environment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Yep - do you *really* need a failover VLE?

Moodle was not designed to work like this and it will be very difficult to even get close

Average of ratings: -
In reply to Howard Miller

Re: Best practices for mirrored production environment

by Daniel Wolff -

In regards to the failover VLE I believe our need is high. Our environment will be supporting learners that have a limited time to complete course work for graduate credit. If it is incredibly difficult we can look at other solutions but my naive understanding is that this common practice at large schools/universities that are running moodle, or at least they have some form of mirrored production.

Average of ratings: -
In reply to Daniel Wolff

Re: Best practices for mirrored production environment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Don't confuse load-balanced Moodle sites (common) with failover Moodle sites (not so). There is a completely different emphasis. The former gives you some protection in the form of multiple web servers (if properly configured). However, there is always a single point of failure (or points). I think it makes more sense to concentrate on rapid recovery than some over-complex clustering arrangement which only introduces further points of failure. Good backups, spare hardware (i.e., so that failed hardware can quickly be replaced) and a properly clustered database cover a high percentage of what you need. 

Average of ratings: Useful (2)
In reply to Howard Miller

Re: Best practices for mirrored production environment

by Daniel Wolff -

Howard,

I hear what you are saying about rapid recovery and it is one of the avenues we are exploring. If you are able, and have the time, can you elaborate on how a load-balanced Moodle site differs from a failover in terms of architecture? Through reading on this site (and others) about high availability, the load balancing seems to be the "easy" part. Moodle seems to be quite different from alot of the applications our IT department often supports so I want to make sure I am illuminating any assumptions that we should be analyzing.

Regards,

Dan

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Best practices for mirrored production environment

by Daniel Wolff -

Visvanath,

Thank you for your interest. Yes, Moodle is incredibly business critical that we would want a hot (or easy manual) failover to take over if something "bad" happened to the main machine/install. I expect about 2000 users/month when we make the switchover (it is what our current install is handling). I don't expect the course development work to slow down the production server. Please let me know if I can clarify anything else.

Regards,

Dan

Average of ratings: -
In reply to Daniel Wolff

Re: Best practices for mirrored production environment

by Usman Asar -
Picture of Plugin developers Picture of Testers

Dan,

in usual high availability scenarios, several web servers fetch data off one database server, whilst you can replicate database in Master-Slave configuration in case if Master fails, you can safely switch to slave meanwhile master is recovered (but this isn't automated switch over, rather a fail safe strategy), alternatively you can do Master-Master replication, so if changes are made to one database they are reflected in other as well.

With ken assisted you to reach this point, you may want to speak to Tim Hunt, as they have set up 8 web servers fetching data off one database with master-slave configuration. If it would have been a WIMP stack, I would have been in a better position to assist, but my hands are bound on LAMP stacks, which most of the people on the forum have extreme capability for.
Average of ratings: Useful (2)
In reply to Usman Asar

Re: Best practices for mirrored production environment

by Daniel Wolff -

Usman,

Thank you for your reply. This is the setup we are currently considering (the Master-Slave configuration). One of the discussions we are having is what should be synced between Master/Slave (or even a Master/Master setup). We are going to sync the /moodledata/ directory and the database, but what about the /moodle/ directory? One of our IT folks is suggesting that we just apply any themes/updates etc to the /moodle/ directory manually in both places. This raises concerns for me because I am afraid that doing so might cause some sync issues between the mirrored environment. I can't claim that these concerns are based on anything I've read (I can't find much online that is recent in regards to this issue, but I may not be searching the right terms) but the logic of it feels a little funny which is one reason why I am reaching out to the moodle community to see if others have similar experience.

Regards,

Dan

Average of ratings: -
In reply to Daniel Wolff

Re: Best practices for mirrored production environment

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

How often do you expect to update your Moodle program directory? 

If you are using something like Git to maintain any changes (and you should be) then the jib becomes as difficult as 'git pull' on each front-end. 

Average of ratings: -
In reply to Daniel Wolff

Re: Best practices for mirrored production environment

by Usman Asar -
Picture of Plugin developers Picture of Testers

Dan, Master/Slave replication is meant to be for database server only, but can as well be applied on web server, moodle folder rarely gets changed (unless upgrading or adding plug-in), consider this as read-only content, but moodledata folder does change rapidly as it holds files, images, uploads and cache etc. In a linux environment I am with limited knowledge in terms of how to achieve that, but for Windows initiating a Distributed File System manage to keep specified folders/drives synced across all servers, but this only makes sense when setting up a load balanced scenario as incoming connections are distributed as per balancer rules, it will be much easier if you set up rapid disaster recovery, rather high availability (load balancing).

Average of ratings: Useful (2)