I am creating Podcasts currently for Deeside College in North Wales, these are the same issues i am confronting, i will only offer students to download podcasts from discussuion forums. In Deeside College there are over thousands of students on their database, however, i concluded that our intranet server, in which moodle is, might suffer from the increased requests to download podcasts and it may crash. It has in the past.
I would predict if i did succeed, our server would incurr reduced performance, and more downtime. At least it is what probably would happen.
Do Not worry padawan! Itunes U already do it at Universities and it is very successfull as students download lectures, lessons and critical talks.