The setting you are looking for is "Allow site-wide messaging" (messagingallusers), which you can find in Site administration > Messaging settings. Disabling that option will limit users so they can only view other users in their courses.
Security and privacy
Messaging system only sitewide or restricatble?
This discussion has been locked so you can no longer reply to it.