I would suggest you use asciimathml, which makes this much easier to manage
One reason I advise installing via met.php is to avoid conflicts with other moodle scripts, such as quiz.js.
you can find info on setting fonts inside the asciimathml script and on the asciimathml home page
I tested that quickly yesterday and although tex filter otherwise can use font sizes
- \tiny
- \scriptsize
- \footnotesize
- \small
- \normalsize
- \large
- \Large
- \LARGE
- \huge
- \Huge
it did not seem to work in quizzes. But like Marc said AsciiMathML works in quizzes if it's rendered from theme (meta.php) and not from filter. One problem with font sizes is that if we use image fallback scripts (for IE) we need to use for example Mimetex, MathTex or some other renderer to produce those font sizes as images instead of html, CSS or MathML. It's not simple anymore.
And we have several versions of those AsciiMathML scripts that may confuse people - "my scripts", "your scripts" and "their scripts"... 
I can't comment algebra filter because I have not used it
I could test this better tomorrow - the input/output is clearly different if you use for example Mimetex vs LiveTex
Mimetex (cgi) does render them correct
how about filter here in moodle.org - almost correct (teTeX and TeX filter):
$$\tiny X \scriptsize X \footnotesize X \small X \normalsize X \large X \Large X \LARGE X \huge X \Huge X$$