As far as I'm aware, there's no custom filter option to sort by category. The category name is displayed for each course you're enrolled in, but that's pretty much it...
However: if you'd use course custom fields, you'd be able to filter one selected course custom field and value...