AFAIK, the messenger module itself is a plugin, so you have hit on a possible solution that might be easier than it seems.
IANAD, but have some programming experience, so I think I would take the current messenger plugin break it down, and likely add in some code that identifies the recipient of a message. If the sender has a Student Role and the receiver has a Student Role, sending does not occur. You have to be able to work out how to get the Role of both Sender and Receiver, I suspect the Sender will be easy, but the Receiver, that might be tricky. I am not suggesting this is the way to do it, but it is one way. I would likely try it first as this seems, on first look perhaps, to involve the least amount of change to a core plugin. Sorry I can't help, but there are a lot of others here who can.
You might want to start a thread in the Developer's General forum, when you have some idea of the code required to make it all work and need some help. Good luck!
[EDIT] Thinking about it a bit more, you may not have to change the plugin as it is, or much at all. Except for adding in code to find the Roles and to allow or block a message, I don't think much other code will need to change at all. I suggest you keep full documentation of what you do, so when you get it working, you might want to present it for adding to Core so someone else might take it to ensure it continues to work as Moodle upgrades to newer versions.