You can create your own Categories, Sub-categories & Courses list and have them display in the Frontpage as you want them to. It just takes a lot of time to do it that's all. So yes it is possible to hide subcategories on the frontpage until they are moused over, or clicked on.
As for a sub-category within a course, I don't think so. The way Moodle works is that Courses need a Category not the other way round. So I think the answe to this is still no....but I may be wrong.
Besides' this isn't really a theme related question, although your 1st question needs extra styling if you choose to do what I said. But for your 2nd question you might be better asking in another Moodle forum and ask the same question, but don't make it theme specific. You will probably get a better answer, as my forte is not in course design.
Mary