If they are able to see the "View all responses" tab as well as the "Your responses" tab then they have the ability to see all of the users' responses. Check for the "mod/questionnaire:readallresponses" capability. If this is set for the user context on the questionnaire, then remove it and see what happens.
If they are only seeing the "Your responses" tab, then the averages they are seeing should only be for their responses.