The advantage of having everything on one server is that the network is not involved when data is loaded from the database, so database access is quicker. So on a system with a small number of users, everything on one server will be fastest. The disadvantage is that you are limited by the amount of hardware you can get in one machine, and if that one machine crashes, your Moodle site is down.
The advantage of multiple servers is that you can have much more hardware (or certainly more hardware for less money). You can then have lots of load-balanced web servers. That scales much better with lots of simultaneous users. The disadvantage is that each database access then incurs network latency, so for single users, things are a bit slower. Also if one web server falls over, the others can keep your site up. With the database, you can use master-slave replication with fail-over, if you want to remove the DB server as a single point of failure.