Hi Irith
I think you are asking how to prevent users in a group from posting to a forum... if you are talking about the messaging system please let us know ;)
You could use Groupings (if enabled at site level) to determine which groups CAN access and view the forum. For example if you have three Groups (A, B and C) and you only want Groups B & C to be able to post to that forum you can create a grouping, assign Groups B & C to it then select that grouping in the forum activity - if you tick the 'Available for group members only' then Group A cannot even see the forum on the front page of the course.
I hope this helps
Cheers
Teresa