Does anyone have experience setting up Moodle in a high availability scenario with multiple servers for failover & load balancing, using RAID, as well as a PHP accelerator? I'm using Mac OS X server on a single XServe, but I'd like to architect a much more robust system that scales with our increasing enrollment. I've read there are issues with MySQL in a RAID scenario (have to use MySQL MaxDB) as well as performance issues with OS X server and MySQL performance (http://www.anandtech.com/mac/showdoc.aspx?i=2436&p=6). We currently have a few hundred simultaneous users and performance appears to be okay for now. Thanks! John
Hi,
While someone else is asking, I'll extend this question a little and suggest a possible configuration. I've been looking to set up redundant moodle servers using:
Gavin
While someone else is asking, I'll extend this question a little and suggest a possible configuration. I've been looking to set up redundant moodle servers using:
- 2 Linux Servers
- PostgreSQL on both using pg_pool to keep in sync
- Apache 2/PHP[45] on both servers
- Moodle's database sessions
- NFS share of the data directory mounted from one to the other
- A round robin or some such load balancer to share queries between the two servers
- share the PHP cpu/ram load between boxes
- allow one server to drop and the moodle service to keep running
Gavin
I don't know about pgpool, since I don't run postgres, but what I have been pursuing is
Linux HA to control the failovers (looking at active/passive failover note you could do one box sql one web and do a sort of active/active with this without a load balancer)
Mysql replication for syncing up the dbs (although Martin L says this isn't the best
since there are no gaurentees of the time to sync)
And drbd to sync the moodledata folders
In terms of just scaling up splitting the sql and apache components across two databases should do it. From there switch to dbsessions and you can start adding multiple front ends as long as you use nfs or some other network protocol to provide a shared moodledata folder. I believe you have to do some file locking setup too.
Linux HA to control the failovers (looking at active/passive failover note you could do one box sql one web and do a sort of active/active with this without a load balancer)
Mysql replication for syncing up the dbs (although Martin L says this isn't the best
since there are no gaurentees of the time to sync)
And drbd to sync the moodledata folders
In terms of just scaling up splitting the sql and apache components across two databases should do it. From there switch to dbsessions and you can start adding multiple front ends as long as you use nfs or some other network protocol to provide a shared moodledata folder. I believe you have to do some file locking setup too.
Come to think of it, yes nfs would not be useful for this, drbd looks to be the think to use for that.
I'm not sure how much better it'll perform with the mirrored database but I'd like to have the redundancy.
Thanks,
Gavin
I'm not sure how much better it'll perform with the mirrored database but I'd like to have the redundancy.
Thanks,
Gavin
There are a couple of configuration problems to solve there. Even then I think your common NFS volume is going to be the bottleneck. How about replacing that with a SAN or NAS?
In either case, this common storage is going to be a single point of failure.
Haven't had a chance to try any of those things though.
In either case, this common storage is going to be a single point of failure.
Haven't had a chance to try any of those things though.
Hi Gavin,
A few comments on that proposal:
1) PGPool is good for load balancing / connection pooling, but it isn't a real solution for replication since there are scenarios where your databases can get out of sync (e.g. asynchronous power failure). We use Slony for PostgreSQL replication, but it won't be multi-master until Slony II is eventually released.
2) DNS round robin is no good for failover. The Linux 2.6 kernel virtual servers is what you want. It's relatively simple to configure, and it will typically have your dead machine being ignored in under 10 seconds.
3) The shared data directory is as problematic (and necessary) as the shared database - there aren't any file systems that do multi-master replication either. We regularly rsync to the DB replica, but necessarily there's the possibility of some loss in the event of that server failing.
Since the file / db server is necessarily a bottleneck it's better to move what you can away from it which means the web servers and load balancing. On the other hand these machines can be cheap clones built from a standard image. It's not essential for functionality to have the load balancing separate from the web server, but we do it that way because it gives better security.
We're also using DB sessions, but there are some plans I've heard to separate that out in the next few months into some sort of shared memory daemon. I forget the exact technical details unfortunately, but it will give improvements in performance over database sessions while still coordinating the sessions correctly across multiple front-ends.
Regards,
Andrew McMillan.
A few comments on that proposal:
1) PGPool is good for load balancing / connection pooling, but it isn't a real solution for replication since there are scenarios where your databases can get out of sync (e.g. asynchronous power failure). We use Slony for PostgreSQL replication, but it won't be multi-master until Slony II is eventually released.
2) DNS round robin is no good for failover. The Linux 2.6 kernel virtual servers is what you want. It's relatively simple to configure, and it will typically have your dead machine being ignored in under 10 seconds.
3) The shared data directory is as problematic (and necessary) as the shared database - there aren't any file systems that do multi-master replication either. We regularly rsync to the DB replica, but necessarily there's the possibility of some loss in the event of that server failing.
Since the file / db server is necessarily a bottleneck it's better to move what you can away from it which means the web servers and load balancing. On the other hand these machines can be cheap clones built from a standard image. It's not essential for functionality to have the load balancing separate from the web server, but we do it that way because it gives better security.
We're also using DB sessions, but there are some plans I've heard to separate that out in the next few months into some sort of shared memory daemon. I forget the exact technical details unfortunately, but it will give improvements in performance over database sessions while still coordinating the sessions correctly across multiple front-ends.
Regards,
Andrew McMillan.
Andrew thanks for sharing the information, most informatitve. I am wondering if you can answer a few details.
1) What type of ratio of backup database/file server to front end do you get?
2) How many drives and how large do you put in your RAID 10 sets on the database servers?
1) What type of ratio of backup database/file server to front end do you get?
2) How many drives and how large do you put in your RAID 10 sets on the database servers?
Hi Jonathon,
We usually only get one local database/file server and one backup against around two to five web servers.
If we want more D/R than that we duplicate the whole cluster in another city and replicate to that as well. Failover to a secondary location is much slower, since it generally involves DNS timeouts, but it covers a whole class of failure possibilities in one act.
The best price performance for us on SCSI drives recently has been with 147G SCSI and RAID-10 across 8 or more of those gives excellent performance - the more drives you can put in a RAID-10, the better the performance you will get, of course.
I think that the 300G drives are just crossing the 147G cost-per-GB now, so next time we'll probably be getting those.
Regards,
Andrew McMillan
We usually only get one local database/file server and one backup against around two to five web servers.
If we want more D/R than that we duplicate the whole cluster in another city and replicate to that as well. Failover to a secondary location is much slower, since it generally involves DNS timeouts, but it covers a whole class of failure possibilities in one act.
The best price performance for us on SCSI drives recently has been with 147G SCSI and RAID-10 across 8 or more of those gives excellent performance - the more drives you can put in a RAID-10, the better the performance you will get, of course.
I think that the 300G drives are just crossing the 147G cost-per-GB now, so next time we'll probably be getting those.
Regards,
Andrew McMillan
nice to meet you Andrew McMillan,
We are finding the steps to install one local database/file server and one backup with two web servers.
We are using windows 2008 + apache +php.
It was no problem as we install first moodle node, but we have problem as we install the second. Do the installation step in install second node is the same as the first one?
any steps need to be done before install the second node. thanks
Thanks Andrew, this is good stuff to know. I'm in the early stages of reading Karl Kopper's "Linux Enterprise Cluster" which goes into some of this, though I haven't got that far yet 
I hadn't seen Slony but will take a look, thanks. I gather dbbalancer isn't really a solution to this or you would likely have mentioned it (given that you appear to be maintaining the debian package).
Everyone else seems to be using multiple db servers with multiple web servers in front of them. I really don't have this budget. Am I crazy to try and replicate both the db and web server on just two machines? Long term I'd like to create a web server cluster with several different apps (PHP, Tomcat, Perl) which might pull in an extra machine or two.
Gavin
PS thanks also for whereami!
I hadn't seen Slony but will take a look, thanks. I gather dbbalancer isn't really a solution to this or you would likely have mentioned it (given that you appear to be maintaining the debian package).
Everyone else seems to be using multiple db servers with multiple web servers in front of them. I really don't have this budget. Am I crazy to try and replicate both the db and web server on just two machines? Long term I'd like to create a web server cluster with several different apps (PHP, Tomcat, Perl) which might pull in an extra machine or two.
Gavin
PS thanks also for whereami!
Hi Gavin,
Slony seems to be the only real replication answer for PostgreSQL. Unfortunately it is not yet synchronous replication, but that is probably less of an issue for Moodle than it is for other things. Now that PostgreSQL has 2 phase commit functionality that a fully synchronous replication can solution can be built but it could still be a year away.
DBBalancer is pretty much defunct. I haven't been able to contact the upstream for a couple of years and I've marked it as "obsolete" in Debian. Fortunately there is a well-maintained replacement called PGPool which does the same job only better.
The problem with DBBalancer & PGPool is that it doesn't cope well with situations like a backend database going down. It _could_ be used in a model where all write transactions went to a master database (which was then replicated) and all reads were pooled but you would need to know that was happening when you were building the application, and it would not achieve the database independence that Moodle currently has. PGPool actually has a mode for this kind of operation where all queries that are (a) not in a transaction and (b) begin with the word "SELECT" are load balanced, but all other queries go to a designated master. Even so, something as simple as 'SELECT nextval('my_sequence');' has the potential to screw things up for you.
The reasons for having multiple DB servers and multiple web servers are about both redundancy and performance, and the architecture you choose will relate to which of those has the greatest weight. Not to mention the cost issues!
As Martin Dougiamas said in one of his posts: "get as much hardware as you can afford". I would add "buy it as late as possible to get the most bang for your buck".
I don't think you're crazy to try and do things on two machines, but if I had two machines for running Moodle I would tend to divide them with one as (primarily) the web server, and with the other as (primarily) the database server. You can still have them with a failover set of each other's requirements but it is a simpler division of functionality and it gets things in the right shape for expanding the cluster further as more funds or hardware become available.
Regards,
Andrew McMillan.
PS. Glad you like whereami
Slony seems to be the only real replication answer for PostgreSQL. Unfortunately it is not yet synchronous replication, but that is probably less of an issue for Moodle than it is for other things. Now that PostgreSQL has 2 phase commit functionality that a fully synchronous replication can solution can be built but it could still be a year away.
DBBalancer is pretty much defunct. I haven't been able to contact the upstream for a couple of years and I've marked it as "obsolete" in Debian. Fortunately there is a well-maintained replacement called PGPool which does the same job only better.
The problem with DBBalancer & PGPool is that it doesn't cope well with situations like a backend database going down. It _could_ be used in a model where all write transactions went to a master database (which was then replicated) and all reads were pooled but you would need to know that was happening when you were building the application, and it would not achieve the database independence that Moodle currently has. PGPool actually has a mode for this kind of operation where all queries that are (a) not in a transaction and (b) begin with the word "SELECT" are load balanced, but all other queries go to a designated master. Even so, something as simple as 'SELECT nextval('my_sequence');' has the potential to screw things up for you.
The reasons for having multiple DB servers and multiple web servers are about both redundancy and performance, and the architecture you choose will relate to which of those has the greatest weight. Not to mention the cost issues!
As Martin Dougiamas said in one of his posts: "get as much hardware as you can afford". I would add "buy it as late as possible to get the most bang for your buck".
I don't think you're crazy to try and do things on two machines, but if I had two machines for running Moodle I would tend to divide them with one as (primarily) the web server, and with the other as (primarily) the database server. You can still have them with a failover set of each other's requirements but it is a simpler division of functionality and it gets things in the right shape for expanding the cluster further as more funds or hardware become available.
Regards,
Andrew McMillan.
PS. Glad you like whereami
Hi Andrew,
We, here at AU, are in the process of setting up a PostgreSQL/Slony-I solution but ran into schema replication issues - refer to http://cbbrowne.com/info/slonyintro.html section "1.5.Current Limitations".
Since Moodle upgrades might modify the DB schema I was wondering what approach did you use?
Thanks in advance!
Viorel
We, here at AU, are in the process of setting up a PostgreSQL/Slony-I solution but ran into schema replication issues - refer to http://cbbrowne.com/info/slonyintro.html section "1.5.Current Limitations".
Since Moodle upgrades might modify the DB schema I was wondering what approach did you use?
Thanks in advance!
Viorel
We moved on to 8.1 and use log shipping. Works like a charm, but both Postgres installs must be compiled _exactly_ the same.
Google for "postgres pitr" and you'll find the right documentation page in Postgresql.org
Google for "postgres pitr" and you'll find the right documentation page in Postgresql.org
Thanks very much for your reply, Martin - that points me into the right direction.
We ran a failover system at the Aust Moodle conference and ran a one day workshop on how to set one up. It involved 2 load balancers 4 web servers and 2 database / file servers to provide a rugged system as possible. The Database servers also have RAID. At this stage the system is relaible, though it does not have a large number of users attaching to it.
I also run each standalone server in a RIAD config with an rsync'd failover backup (identical machine).
It is not an easy sysetem to setup, and not cheap, however it does negate the need to have a tech available on site in the event of a system failure. The reqiuirement of not having to employ a technical support person was the finacial payback for the investment and ensuring a QoS meets Australian Standards for delivery service.
There has been a reasonable amount of development undertaken in the area over the past couple of years, and many lessons learned.
My basic system would look something like dual (identical) machines each with RAID, then use replication to sync the MySql database and a regular rsync script for the file areas. Simple(ish), so it will probably work
My other observation, is that it's only the VLE. Hardly life and death, do you really need all this hassle and complexity. Sods Law says that you go to all this trouble and then your internet connection fails. There's always a single point of failure somewhere!
Howard you make a good point. Any redundancy needs to be balanced across all the things that can fail. It is easy to over spend on a single area of redundancy and neglect others completely. I good aproach is to prioritize the likeliness of a failure and start with the most probable. Hence why server hardware tends to have redundant powersupplies and hard drives as these are the most likely points of failure. Keep in mind that a really redundant setup can do more than just help prevent unexpected failures. It also allows for any type of system upgrade to be performed without down time.
One of the interesting things about building redundant clusters with off the self hardware is that with the right software and planning you can do some very cost effective setups. For example we just purchased two 1U servers with 4 drives each. We are testing using these in a cluster with drbd for mirrored storage and heartbeat for failover. We are going to use raid striping on the four drives and mirror the contents across the network. We are using SATA drives, but since we are stripping them we should still have pretty good performance on them. A single more traditional server with redundant power supplies, raid controller, and scsi drives would have cost as much as the two servers and would still have single points of failure in the RAID controller and motherboard.
Rsync is very handy and is great for backups, but it isn't really a real time tool. drbd allows you to keep two partitions mirrored in closer to real time across the network. I haven't put it into production yet, but it looks like a pretty cost effective alternative to a raid shelf or SAN.
You touch on one of the best reasons to host at a data center. Redundant Internet connections. Not to mention redundant power systems, deisel backup generators, etc. Compia recently release a guide for data centers that goes into great depth about all the factors you should account for if you want true redundancy.
One of the interesting things about building redundant clusters with off the self hardware is that with the right software and planning you can do some very cost effective setups. For example we just purchased two 1U servers with 4 drives each. We are testing using these in a cluster with drbd for mirrored storage and heartbeat for failover. We are going to use raid striping on the four drives and mirror the contents across the network. We are using SATA drives, but since we are stripping them we should still have pretty good performance on them. A single more traditional server with redundant power supplies, raid controller, and scsi drives would have cost as much as the two servers and would still have single points of failure in the RAID controller and motherboard.
Rsync is very handy and is great for backups, but it isn't really a real time tool. drbd allows you to keep two partitions mirrored in closer to real time across the network. I haven't put it into production yet, but it looks like a pretty cost effective alternative to a raid shelf or SAN.
You touch on one of the best reasons to host at a data center. Redundant Internet connections. Not to mention redundant power systems, deisel backup generators, etc. Compia recently release a guide for data centers that goes into great depth about all the factors you should account for if you want true redundancy.
The truth is that individual organisations' needs will vary wildly.
For example, moodle is used more substantially from within our network then from the outside (I must go and verify this with proper stats). So, use of a data centre would add the 'net connection as a point of failure for internal users. Not to mention killing our download bandwidth.
In an ideal world we would have redundant 'net connections and tonnes of bandwidth of course.
If you find yourself running several apache web servers for different tasks (eg your VLE, main web server, etc), the idea of making a redundant web server cluster can be attractive.
Gavin
For example, moodle is used more substantially from within our network then from the outside (I must go and verify this with proper stats). So, use of a data centre would add the 'net connection as a point of failure for internal users. Not to mention killing our download bandwidth.
In an ideal world we would have redundant 'net connections and tonnes of bandwidth of course.
If you find yourself running several apache web servers for different tasks (eg your VLE, main web server, etc), the idea of making a redundant web server cluster can be attractive.
Gavin
Good point. Redundant connections at the data center doesn't do you any good if you only access it from inside your organization and you only have a single connection from the organization to the Internet.
I would like to, but I'm running it on an XServe and some Apple reps told me to run MySQL on RAID (Apple's version is XServe RAID), I would need XSan. They said in that configuration, I would pay a huge MySQL performance penalty just to have the reliability of RAID. Anyone out there running Moodle on XServes using XServe RAID/XSan?
thanks,
John
thanks,
John
John,
My 2 cents: we researched HA Win systems that had dual everything from UPS, to server (RAID& auto failover to backup server), to switches, load balancers, and even telcos/internet providers from 2 different companies.
The killer was not the hardware, but the software cost if memory serves. Think it was about $14K USD for each server. The original (overly lofty) goal was five 9's, but on re-exam this was too extreme as we don't realistically have traffic 12 hours out of every day and very little on weekends.
The secret to HA systems is not looking at w Windows environment - the O/S licensing costs will bury most projects, but review Linux solutions. There are a number out there.
For providing basic failover a Linux solution works well, particualrly when one considers what level of failover you actually need. For any institution delivering on-line, then a small Linux solition takes the heat off the systems administrator and they can be built quite cheaply - just takes time in installing and configuring everything, but you have that with any solution. 1 RU units are an ideal option and affordable.
A typical small configured system - 2 load balancers, 3 web servers and 2 database/file servers. The key advantage is that if a single unit fails, there is a backup. If one was to provide backup to this system, then a single spare 1 RU unit that could be interchanged as required.
Doable, affordable and provides oy peace of mind that in the event of a system failure, toye environment does not stop. That said the backend to this simple system, is your internet connection, but at least you can sleep at night comfortably in the knowledge that you have provided the most robust and economical solution without breaking the bank, and your system admin costs may reduce as a bonus.
"A typical small configured system - 2 load balancers, 3 web servers and 2 database/file servers."
Specs on that "small" configuration are exactly what Im looking for. I currently run Moodle on one XServe with MySQL on that same box. Id like to setup a system similar to the one you mentioned, but am unsure of the technical steps (running MySQL on RAID drives, etc.).
Ive read OS X Server has issues with scaling MySQL, so Im sensitive to that. Im not married to XServes, although they would have the most in-house support.
I heard there was a Moodle conference that had sessions specifically on setting up high availability scenarios. Does anyone know of a Moodle book that discusses designing and setting up Moodle like this? Our program is growing rapidly and I want to make sure the hardward and software support.
Thanks!
Specs on that "small" configuration are exactly what Im looking for. I currently run Moodle on one XServe with MySQL on that same box. Id like to setup a system similar to the one you mentioned, but am unsure of the technical steps (running MySQL on RAID drives, etc.).
Ive read OS X Server has issues with scaling MySQL, so Im sensitive to that. Im not married to XServes, although they would have the most in-house support.
I heard there was a Moodle conference that had sessions specifically on setting up high availability scenarios. Does anyone know of a Moodle book that discusses designing and setting up Moodle like this? Our program is growing rapidly and I want to make sure the hardward and software support.
Thanks!
As much as I would like to recommend OS X, I can't. It is not just a problem with MySQL performance, apache/php seems to suffer from the same issue. Performance under little or no load will be fine, but once you go into the 25-50 concurrent connection range and beyond throughput crashes with the result being the system running 1/10th the speed of what the same hardware will do running Linux (yes you can run linux on an Xserve if you prefer this hardware).
I suggest fast drives for the box running MySQL and fast processors on the front end units.
I suggest fast drives for the box running MySQL and fast processors on the front end units.
That's funny. I have been running on a dual G5 with OS X 10.4 and a gig of ram. On exam days we have seventy-five students hitting it with no lag. To tell you the truth, I think the Apple techs are just not on the ball so I do it myself. It's not that I can't afford something hunkier, but that load is pretty low compared to the transaction processing load I saw at my last job. We used to have the kind of setup MacMillan is talking about to run transactions for Radio Shack, Walmart, and Walgreens prepid phone card activations. Maybe I should run some fibre next door and see if it slows things down?
Bob can you share how you did your setup? And you are sure these are 75 concurrent users? What software packages/versions are you using? I would love to have a stable and decently performing OSX installation guide.
Here is a link to the Ars Technica article that identified the performance issues, I mentioned. Some have disagreed with there results. but I haven't been able to come up with a decent configuration after working on about 3-4 OS X different OS X servers.
http://www.anandtech.com/mac/showdoc.aspx?i=2436&p=6
Here is a link to the Ars Technica article that identified the performance issues, I mentioned. Some have disagreed with there results. but I haven't been able to come up with a decent configuration after working on about 3-4 OS X different OS X servers.
http://www.anandtech.com/mac/showdoc.aspx?i=2436&p=6
Hi,
Martin Langhoff has talked about the setup for The Open Polytechnic at a few conferences, I believe. He's on holiday until the end of the month though, so you won't see his response here and you'll have to take my pale imitation
Firstly, we run our main Moodle cluster using PostgreSQL, because we strongly believe that it is a more robust and scalable database solution than MySQL. There is a tool in contrib for migrating from MySQL to PostgreSQL for any who are interested.
The hardware we use for PostgreSQL for the Moodle cluster (and, indeed for many of our non-Moodle PostgreSQL databases such as the NZ Electoral Roll) is 2- or 4- way Opteron systems. Opterons give two big benefits (1) they are 64 bit native, so increasing memory gives real benefits to the DB server and (2) the hypertransport bus is very efficient for processor intercommunication, which is why Opterons leave EM64T (Intel) servers in the dust. We have maybe twenty of these kinds of systems, some have been built from motherboard and CPU up, some are white box from a local server manufacturer, some are HP and some are Sun. All of them are reliable and run well with Debian Sarge and/or Ubuntu Breezy. We've been using Opteron systems as 64 bit database servers for over two years. Disk is, of course, also important. We're pretty cost conscious and at this point we're managing just fine on RAID-10 U320 SCSI subsystems.
Each cluster has two database servers, with the second as a replicated failover.
For the web servers we use fairly straightforward systems, generally Intel-based, and all running Debian Sarge. They mount shared file space off the database / file servers.
In front of the web servers we have a pair of load-balancing firewall systems. These are (naturally running Debian Sarge and we use the Linux 2.6 virtual server functionality to do the load balancing. A heartbeat arrangement controls which one is actually doing the work.
We target a spend of around NZD$10k for the database servers and NZD$2k for the web servers and load balancers (USD$7k & $1.4k perhaps). That pricing has been remarkably consistent as the "value for money" points for the two types of hardware for several years now, although the actual hardware has got significantly better. It is rare that price is no object, hence our tendency to base the choice on 'what can we get for X' rather than on specific CPU, memory, etc. For the prices here you get a very functional and available Moodle cluster for NZD$30k (USD$21k?) plus setup work.
In the coming year we'll be doing a lot more work with Xen and other virtualisation stuff so I expect we'll be bringing that learning back to our Moodle cluster setups in due course too.
Regards,
Andrew McMillan
Martin Langhoff has talked about the setup for The Open Polytechnic at a few conferences, I believe. He's on holiday until the end of the month though, so you won't see his response here and you'll have to take my pale imitation
Firstly, we run our main Moodle cluster using PostgreSQL, because we strongly believe that it is a more robust and scalable database solution than MySQL. There is a tool in contrib for migrating from MySQL to PostgreSQL for any who are interested.
The hardware we use for PostgreSQL for the Moodle cluster (and, indeed for many of our non-Moodle PostgreSQL databases such as the NZ Electoral Roll) is 2- or 4- way Opteron systems. Opterons give two big benefits (1) they are 64 bit native, so increasing memory gives real benefits to the DB server and (2) the hypertransport bus is very efficient for processor intercommunication, which is why Opterons leave EM64T (Intel) servers in the dust. We have maybe twenty of these kinds of systems, some have been built from motherboard and CPU up, some are white box from a local server manufacturer, some are HP and some are Sun. All of them are reliable and run well with Debian Sarge and/or Ubuntu Breezy. We've been using Opteron systems as 64 bit database servers for over two years. Disk is, of course, also important. We're pretty cost conscious and at this point we're managing just fine on RAID-10 U320 SCSI subsystems.
Each cluster has two database servers, with the second as a replicated failover.
For the web servers we use fairly straightforward systems, generally Intel-based, and all running Debian Sarge. They mount shared file space off the database / file servers.
In front of the web servers we have a pair of load-balancing firewall systems. These are (naturally
We target a spend of around NZD$10k for the database servers and NZD$2k for the web servers and load balancers (USD$7k & $1.4k perhaps). That pricing has been remarkably consistent as the "value for money" points for the two types of hardware for several years now, although the actual hardware has got significantly better. It is rare that price is no object, hence our tendency to base the choice on 'what can we get for X' rather than on specific CPU, memory, etc. For the prices here you get a very functional and available Moodle cluster for NZD$30k (USD$21k?) plus setup work.
In the coming year we'll be doing a lot more work with Xen and other virtualisation stuff so I expect we'll be bringing that learning back to our Moodle cluster setups in due course too.
Regards,
Andrew McMillan
Humm, I'm interested in this tool, but can't find it in cvs:/contrib. Where should I look for?
Saludos. Iñaki.
That's my fault. I did all the work on converting several MySQL moodle installations to PostgreSQL and people think I have a script for the job.
Actually I don't have one. But here is how to do that manually:
- Do an installation of the same copy of the moodle on your MySQL database on postgres.
- Dump the postgres database with only schema. There are two ways of doing this, one is to truncate every table, the other is use pg_dump/pg_restore to produce a schema-only copy.
- Do a mysqldump with option -c/--complete-insert and -t/--no-create-info. This will produce a data-only dump with full insert syntax to reduce problems.
- Remove all backticks "`" from the mysqldump file.
- Search for "CONCAT(firstname,' ',lastname)" and replace them with "firstname||' '||lastname". They are log displaying formats in table prefix_log_display.
- Pop the data-only dump into your postgres database. As it is using complete INSERTs this could take quite a while.
The final step. Reset all the sequences in the new postgres database. I have actually wrote a block of php code for the PostgreSQL UTF-8 migration process to do that, but for some reason it is not needed anymore so I lost it (doh!)... To do this in a linux command line, I give you a one-liner to use.
PGPASSWORD={password} psql -qAt -U {username} -c "\ds" {database} | cut -d '|' -sf 2 | sed "s/^\(\(.*\)_id_seq\)$/SELECT setval\('\1', \(SELECT MAX\(id\) FROM \2\), true\);/" | PGPASSWORD={password} psql -U {username} {database}Replace {username}, {password} and {database} with your own ones.
Hmmm... Martín L told me there was 'something in contrib' for doing that, I thought.
Of course it's possible that he meant postgresql/contrib rather than Moodle/contrib
In any case, I see that Patrick has now posted some instructions, and I'm sure he could convert that to a shell script in short order.
Regards,
Andrew McMillan.
Of course it's possible that he meant postgresql/contrib rather than Moodle/contrib
In any case, I see that Patrick has now posted some instructions, and I'm sure he could convert that to a shell script in short order.
Regards,
Andrew McMillan.
Andrew thanks again for sharing this information.
You mentioned in this post, or perhaps another in this thread, that you also use the database servers as a storage point for your frontends files. I am assuming moodle data. Two questions
1) What mechanism/softwarae/filesystem are you using to coordinate multiple machines accessing the same moodle data folder. There seem to be lots of options out there for this setup, so I am curious if you have found one to be superior over the others.
2) What are you using to keep the moodledata files in sync between the primary and backup servers? Obviously replication works fine for the sql components, but what about for regular files?
You mentioned in this post, or perhaps another in this thread, that you also use the database servers as a storage point for your frontends files. I am assuming moodle data. Two questions
1) What mechanism/softwarae/filesystem are you using to coordinate multiple machines accessing the same moodle data folder. There seem to be lots of options out there for this setup, so I am curious if you have found one to be superior over the others.
2) What are you using to keep the moodledata files in sync between the primary and backup servers? Obviously replication works fine for the sql components, but what about for regular files?
ok everything said to this point is excellent so let me just describe our set-up
2 centos server sharing virtual ip running the apache/php stuff
each server is connected to each other using serial connexion for heartbeat and STONITH
each server is connected to the other server UPS via serial
each server is connected to the other using a dedicated 100Mbits ethernet link
All files and the DB goes into the NAS that is very redurant on its own
this is almost a textbook exemple of implementing linux HA, is it trus HA ? no because it will only work for a limited time and we dont have generator or a complete second setup in case a earthquake or a flod at the datacenter. BUT its a very solid solution that will get trough most mishap that will hapen in normal life
if i was unclear on anything just ask this is a very breif description for a setup that took time to get running just like it should
2 centos server sharing virtual ip running the apache/php stuff
each server is connected to each other using serial connexion for heartbeat and STONITH
each server is connected to the other server UPS via serial
each server is connected to the other using a dedicated 100Mbits ethernet link
All files and the DB goes into the NAS that is very redurant on its own
this is almost a textbook exemple of implementing linux HA, is it trus HA ? no because it will only work for a limited time and we dont have generator or a complete second setup in case a earthquake or a flod at the datacenter. BUT its a very solid solution that will get trough most mishap that will hapen in normal life
if i was unclear on anything just ask this is a very breif description for a setup that took time to get running just like it should