If you are talking about a 2 server setup, 1 of each. Then the best place is on the Web Server, setup RSYNC for backups to the DB server if you do not use a 3rd party solution, use APC for opcode cache and configure the MUC using the plugin, https://moodle.org/plugins/view.php?plugin=cachestore_apc to add all other cache's into APC.
Now if you have 2 web servers and 1 DB Server, For best performance put /moodledata on a shared LUN using OCFS2 for concurrent read/write. My best opinion if you cannot have a shared LUN is to use GlusterFS for the raw data storage like the contents in /moodledata/filedir and put as much of the cache's in memcache. To make it easy for a novice, install Couchbase on all nodes and configure the MUC accordingly and since Couchbase is a drop-in replacement for memcache it will be automatically distributed and highly available.
Thanks,
JASON