I've encountered the same limitation with the capability mod/quiz:viewreports now. Despite moodle/site:accessallgroups being set to false, the user is able to view the quiz reports for all users across all groups, not merely his own group.
This is at odds with the official Moodle answer to "Restricting Teachers to viewing their own students grades ONLY", which states that with separate groups and moodle/site:accessallgroups set to false, teachers should only see the grades of their own group.
Did this never apply to mod/quiz:viewreports or did this break in the mean time? Or is there something special about teachers (i.e. default teacher permissions) that leads to differing results?