There is no feature in the released Moodle to do what you want. There are a set of changed and new files in a zip file called "Moodle_Maximum_Enrolments_1.8.3.zip" near the end of the following forum thread: http://moodle.org/mod/forum/discuss.php?d=50063#p363123 which provide the feature I think you want. This works for Moodle 1.8.3 . A quick perusal of the changes for Moodle 1.8.4 indicate it may well work for that version also at the cost of reverting some minor bug fixes.
Alan.