Since someone rattled my tree! ...
The other discussion may not apply to all setups ... keeping some moodledata on web server and some directories that needed to be fast ... like cache/sessions, etc. ... and using remote storage for filedir ... where moodle stores backups/user uploaded files/content. Kinda depends upon where/how hosted and what one can do in overall environment/virtualization.
Any remote storage involves networking ... if network hickups and filedir is a mount point to a directory on something providing that remote storage, moodle kinda crippled.
Example of hosting ... Rackspace offers storage devices (concept like a remote drive) ... OP choose size and has to set up the mount ... like /mnt/data/. That can be attached or de-tached ... moved to another server within the same data center.
Filedir could be in /mnt/data/filedir ... of course, if there is any hickup in the data center .... there's issues with moodle ... can login cause sessions is in moodledata on web server, but bascially cannot do anything involving files that are stored in /mnt/data/filedir/
That muddy the waters any? :|