You are starting with the resources and adding what course they belong to.
You need to start with the courses, and look for those that do not have resources. In addition, the GROUP BY course seems to only shows those with resources. The below will show you what you wish.
SELECT COUNT(l.id) COUNT, l.course, c.fullname coursename
FROM prefix_resource l INNER JOIN prefix_course c ON l.course = c.id
WHERE c.category IN (12,9,17)
GROUP BY course
ORDER BY COUNT DESC
Alternatively, the below will list just the courses which have no resources.
SELECT c.id, c.fullname coursename
FROM prefix_course c LEFT JOIN prefix_resource l ON c.id=l.course
WHERE l.course IS NULL
Cheers
Peter