That sounds like a valid an useful use-case. However, I am afraid that the current Moodle versions do not support that. From what I can see, the script group/index.php does not respect the course group mode. So even with the Separate groups mode and without the accessallgroups capability, you Non-editing teachers need the "managegroups" capability which in turn gives them access to all the groups and their members on this screen.
One possible option could be a custom plugin that implements only that part of the groups management UI where group members can be defined.