Sometimes when the Youtube site is running slow, then the video doesn't load and it looks like the block is empty. Anytime you are pulling something from another site, there is a possibility that your site will be impacted.
With Youtube you just don't see the video...with things like rss feeds, it can drag your site to a halt.
Steve