Yes, maybe this can be reviewed in a future version. At the moment, the group scheduling in Scheduler is triggered only by the "Group mode" setting of the activity. Maybe that's an "abuse" of the group feature... just as well as using "one group per grouping" to restrict access to only one group is in a sense an abuse of the group feature (cf. MDL-42979, MDL-20944).
However, my feeling is that this would involve quite a bit of changes in the code. One might also think of controlling the facility to schedule "for a group" with a capability, so that one could allow only certain students to schedule an appointment for their group.