I think I'm doing something similar to what you are after in my filter_units plugin, though I'm using a filter as opposed to the horrorshow of overriding frontpage_available_courses().
`/course/lib.php` has a method called `course_filter_courses_by_customfield` which you could use like this:
$category = core_course_category::get($COURSE->category); // or whatever
$pool = $category->get_courses(array('recursive' => false, 'summary' => false));
$courses = course_filter_courses_by_customfield($pool, 'courseofstudy', 'Math');
You might have to call it twice to modify your course listing when there are multiple custom attributes you need to match.