Rahim,

do you know what the usage pattern will be? Will your users be using all of moodle, or only the more static things (resources). Server load is very dependant on the moodle module used and the number of concurrent users.

So basically you should think about the number of users, the number of courses is not of your primary concern database and apache/php wise (but it may be disk space wise). Do you have an idea of the number of concurrent users and the number of users in total? Knowing this figure can help us support you better.

Mysql replication, especially multi master, is a rather complex thing and you should really know what you are doing here, to avoid user (admin) errors. More info on : http://dev.mysql.com/doc/refman/5.0/en/replication.html

I would advise one big database server (lots of fast disks, loads of ram, fastest cpus you can pay) and multiple webservers sitting behind a loadbalancer or using some kind of balancing mechanism. One big fileserver for the data and you are set. Keep in mind that you only get higher performance doing this, not higher availability. If you need higher availabilit, you should think about making things redundant (drbd block devices, mysql master/slave failover).

This is my opinion on these kinds of setups, maybe others have differen ideas and I'd love to hear about them, too.