Hi,
while I don't have experience with large Moodle installations specifically (ours is rather small), I do have some experience with large web-based applications. I'd have some comments on what you wrote.
First, on a rough scale, the bandwidth seems reasonable to me - I would rather suspect that you end up more on the low end (20 MBit/s rather than 50 MBit/s). Of course this very much depends on what your users are actually doing. You might want to have a look at HTTP data compression - this would increase the load on your web servers, but would make a considerable difference for users on slow connections, and also save bandwith on your side.
Second, I would strongly advise that you use a dedicated ("hardware") device for load balancing, even if it is a very basic model (for example, Cisco
CSS 11501). The problems that you will be facing are not so much with bandwith, but rather with a high number of parallel HTTP connections, particularly if your users are on slow connections. Plan for 2 connections per active user, at the very least. Hardware load balancers are designed to be very efficient in handling this high number of connections (they would all need to go through one device), while usual operating systems are not.
Third, I do not think that the Ethernet switches are a bottleneck. Even one (good) GE switch would suffice.
Fourth, what system are you going to use for database clustering? This can be a very challenging point.
Fifth, are you planning for a resilient system (i.e., should hardware faults of any one server be tolerated)? The single LDAP server seems a single point of failure.
As to the specific server sizing for Moodle, I don't have much experience. But, as a general rule, design your system so that more capacity can be added at any time. This will not be much of a problem at the web server / application server level; again the database can be much more challenging.
Best wishes
Henning
