Hi there. A couple of thoughts:
Restrict Access only does that: Restricts Access. But students still see the activities of other groups; they just can't click on them.
you completely hide the Restrict access settings (by clicking the eye icon) then they shouldn't be able to see the activities of other groups at all.
If you remove the permission "moodle/site:accessallgroups" from the teacher role then they will only see the groups they have been put into. See the docs questionHow do I restrict a teacher to view only information about the groups that they are in?