To confirm the situation here, if an activity/resource is restricted to a group (or grouping) then a user will able to access it if ANY of the following are true:
a) They belong to the group!
b) They have 'access all groups', which lets you access all group-restricted content, regardless of whether you belong to the group.
c) They have 'view hidden activities', which lets you access any activity even if it is hidden (due to 'show/hide' status or due to 'restrict access' conditions).
If none of those are true then the user should not be allowed to access the activity - depending on how the restriction was set up this either means they should not see it at all, or else it means they should see it, but not as a link, and with a message underneath explaining that they can't access it.
Note: I wrote this from memory but I can't think of another reason other than those three why people should be allowed to access it.
Anyhow so I'm fairly sure this works in general (including in Moodle 2.8.x, which is the version we currently use on our live systems here). However, if anybody can reproduce a specific problem with detailed steps (for testing, it's often best to start with a new blank test course and just add the one user/group/resource so that the set up is simple) then please file it as a new issue in the Moodle Tracker.
--sam