I've got a nearly identical setup requirement. We have guest access open on most courses because we want most users to be able to view content in nearly all courses. We have a small subset of users that aren't supposed to be able to view courses in a particular category. I put in an override permission for the role that is supposed to be blocked, within the category in question. The override is: moodle/course:view PROHIBIT. I am sure I had this working before I upgraded the site from 2.6 to 3.0, but now it looks like this override is not having the desired effect. Users with the role are still able to view courses within the category in question.
I wonder if something changed.