There really is no way to do this, but to just get the one group to not SEE the courses/categories on the FrontPage, then this method had worked for me in the past when I had to rig something up like you're asking about. Doesn't prevent the 'access' bit you are looking for, and it's cumbersome, but hear me out:
Lets say you have 2 managers (ManagerA and ManagerB) and you only want them to see their respective categories (ie. CategoryA and CategoryB). For the managers to see only a certain category, you'll need to hide both categories (but not the courses within). Then duplicate the Manager role* per manager and append the name (ie. ManagerA, ManagerB). Now use these roles instead of the original Manger role to assign both managers at the site level. So instead of all your managers sharing 1 site role, they will all have their own (identical) role.
So at this point, your managers will still be able to see all the hidden categories. Now, navigate to the specific Category(s) you want the that manager (ie. ManagerB) to NOT see and click on Permissions. Change 'See Hidden Categories' to Prevent for each manager role you don't want to see that category on the Front Page. Now, when on the Front Page, ManagerB will not see the CategoryA.
*If ManagerB does a course search, then in the course search results they'll still see (and be able to enter) courses that were in CategoryA, so consider using some CSS to the hide the course search button.