Hi Biel,
In Moodle 2.3 a feature was added to lighten the load of serving files. Andrew wrote about it here: https://moodle.org/mod/forum/discuss.php?d=217355 .
Regardless, if it slow with only a few users, it's gonna be slow anyway. A good virtual server will be an improvement.