Hey Matthew
I would suggest monitoring your setup to know how much overhead you have on your system before making any decisions. One persons 5000 user moodle instance could be as active as a 1000 person moodle instance and vice versa. Once you get some good metrics, you will then know at what rate you can add courses/users.
Load balancing benefits you in scalability and redundancy. The drawbacks, a more complicated environment. Ex: shared filesystems, ensure code is consistent on each, etc
Opcode caching is probably one of the biggest and easiest wins in terms of performance...oh and...a fast DB server. I'd make the following recommendations based on my experience:
- get off of mysql 5.1.51 as soon as possible and move to 5.5. there are some serious linked lists issues
- throw ssd behind the db
- use APC or ZendOptimizerPlus