Hi Steve,
The reason why changing the site:sendmessage capability to prohibit for the student role didn't work is because the capability can only be applied in the
system context, and most likely your student role is applied in the
course context (i.e. users are enrolled as students on particular courses).
Messaging may be disabled completely via
Administration > Security > Site policies. Alternatively, you can limit messaging to specific users by changing the site:sendmessage capability for the authenticated user role from allow to not set, then create a new role with site:sendmessage allowed and assign this to selected users in the system context. Please see the
blogger role documentation for a similar case, where blogging is limited to specific users.