Hardware and performance

HI Hugh,

Great to see your work in this area.

At the 2005 moodle Moot in Adelaide I ran a workshop on Enterprise Moodle, where I showed how to build a Moodle Cluster. We created a eight server configuration which we later demonstrated to the main conference, showing the system continue to work as we "removed" each computer from the cluster - in this case by removing the network cables.

I had based this on a research project I undertook at the Waiariki Institute of Technology ( Rotorua NZ ) where I work part time.

The load balancing and web server management is not too difficult, as you have found.

The area that I came to struggle with was the persistent storage - ie database, and in the end I felt that the answer was in a practical compromise. Elsewhere on this site I have discussed the issues and fallacies of trying to achieve a high level of availability - or "nines" as they call them.

For the database, my recommendation is now that you provide the best hardware platform that you can - dual psu's, fans ( the things that move and wear out ), quality gear, and raid to give better performance and to handle HDD failures ( at last one at a time ) without having to take the server down.

Then provide a smaller ( or same if you have the budget <g> ) backup Db server that runs a simple master - slave replication. If you need instant failover then you can place another couple of load balancers between the web servers and the db servers. ( you need two else you introduce another single point of failure ) I've tried to implement an effective bi-directional replication mechanism that is easy to setup AND maintain, but have found it to be elusive so far.

When you put this all together you end up with a lot of boxes to support. My effort now is creating a two and/or three (physical) server configuration using virtual machines - effectively allowing everything to run on just one box, but ideally on the 2 / 3. This I hope will provide an effective solution that the average IT guy can get his head around. Being Linux based the configured images can then be distributed ready to go.

In a practical sense, trying to achieve 100% uptime is a fruitless task, as there are so many other issues that can affect the perceived QOS of your system.

The basics of good hardware, good network connectivity, good ISP, experienced and motivated support staff ... always apply, and can achieve almost all the desired results for the most. What we are trying to achieve here is perhaps get to that extra level we call excellence!

I'll follow your work with interest.

regards

gary 

 

 

 

 

Hi Hugh,

Nice job with the research. You've compiled a lot of separate threads and discussions into a really great summary.

Let me clarify a few things from our perspective at SFSU and my experience.

1) Performance. This is why we use replication. Our major concern is making sure our 30,000 some odd users can all use the system during finals week. This is why we switched to INNODB and threw a whole bunch of RAM at the system. This got us the best performance. We use replication as our backup and statistics system. It is fail over but it primarily is used for us to run statistics on without impacting the live site. It can be used as failover but the few times we've had problems with the main database it has been just minutes to fix the live database which is less time than it would take to implement the failover. I'll also say if you implement INNODB correctly you should be ACID compliant meaning you have little to no risk of loss of data and extremely high reliability/upditme.
http://proquest.safaribooksonline.com/0596102461/I_0596102461_CHP_15_SECT_7

2) DRDB. I don't have a lot of experience, mostly just my own tinkering and reading, but from everyone I've talked to and heard from this is the best for High Availability. If automatic failover and high availability are your concern I'd go here. Do make sure you have enough hardware to run it as my understanding leads me to believe that there can be major performance bottle necks in moodle especially in high SQL write times such as high concurrent use of the moodle quiz.

3) Crashing/table corruption. Corruption is not a problem if your ACID compliant. Under myisam, I had tables crashing left and right about 2 years ago. Now with INNODB, I've never had a table crash. Even if you do crash under query replication, standard in 5.0 but to change in 5.1, the query would not replicate to the slave until it is finished on the live leaving your replicated database in tack. (I haven't tested that last part). A safer bet will be in 5.1 when row based replication gets implemented.

In terms of tuning there is a lot you can do in the my.cnf file. I don't use the out of the box my.cnf but it is always a good starting. Assuming you are using dedicated hardware for the database some basic items:

1. make sure you have the right RAID with a battery backup write cache.
2. All RAM should be allcollated to mysql with maybe half a gig reserved for the system.
3. Disable any engines you are not using: innodb ect. (Can't disable myisam as it is required for the mysql table)
4. Memcached is great for mysql performance. (actually all caching is)
5. Innodb is easy to configure for memory... throw it all at the innodb_buffer_pool
6. Myisam is a little more tricky but lots of good information on it online.
7. Use jmeter or some other similar load testing software. Have it throw everything it can at your install and see how you do. Tweek as necessary based on this info preferably before you go live.
8. MONITOR MONITOR MONITOR. Use MRTG or Cacti or Mysql Enterprise or something to actively monitor your database for performance changes. Much easier to do this from the start than to try and go back and look at 'show extended status' and try to figure it out after the fact.

Hope some of this helps if anything is unclear or I'm wrong about please let me know.

Cheers.

You seem to have covered most/all the points of debate. if your performance needs are one server or less, ie one server total for apache and database combined. I like DRBD in combination with a virtualization technology (I'm fond of openvz) to contain all the complexity of the setup, IE if one server is fast enough to keep up with your load, then there is a lot to be said for doing the HA via DRBD and Heartbeat. No special parts and pretty good fail over for hardware based issues and you only need two off the shelf even low end servers to pull it off. Combine this with some form of backup software that supports snapshots on a decent interval and you have some backup for sql problems that replicate between the two hosts.

As you point out there are certain classes of problems that this doesn't protect againts and I think things get much murkier when/if you need to scale up to multiple numbers of servers. There are some pretty good articles out in the mysql community discussion the relative merits of DRBD vs binary log replication. For example if I remember right with innobd based tables you can have a long startup time after a fail over for DRBD solutions while certain database checks are performed. On the flip side binary replication doesn't have a gaurenteed convergence time for when the two units will be in sync. Additionally, the syncing method is not multi-entrant so I have read reports of the database slave having to be much bigger hardware wise so it can keep up with the master with only one cpu core.

There is actually a third posibility with mysql replication 'tricked' into doing dual-master replication (using different auto-increment values). This is useful for a rapid switch to a 'slave' server.