The downside to moving from a dedicated setup on your own hardware and network to the AWS platform is that you will notice that there is a speed difference. Though AWS is very resilient, your network connections between EC2, ElasticCache, RDS, etc., is not tuned to be optimized like you can do in your own dedicated setup. If you look at the EC2 instances you will notice that they have a rating on network performance, with the larger the system the better the network performance.
Definitely do not move the moodledata folder to S3 as you will see a definite slow-down. Currently, until Amazon EFS becomes available, the best choice for a shared storage system is to use GlusterFS. Again, network performance will be the critical factor.
In the interest of sharing best practices it would be great to hear about your experience as you set this up and what you find in terms of performance and the balancing of it. Currently I am only using an EC2 connected to RDS.