Moodle in English

MathJax does it automatically and changing font-colors or sizes is easy - with Tex filter it's more difficult.

You could first go to Site administration > Plugins > Filters > TeX notation and set Density to 100.

Are you using \frac or \dfrac ( = \displaystyle\frac) ?

Wait a minute - your site using Mimetex fallback, isn't it? (No Latex distribution installed) Then changing Density has no effect...

It's a sort of "bug" because Mimetex fallback is using \Large command in all output. If you open file filter/tex/pix.php with an some text editor like Notepad++ and change line

                $texexp = '\Large '.$texexp;

to

                $texexp = '\small '.$texexp;

or simply delete or comment out that line

                \\$texexp = '\Large '.$texexp;

and purge all caches from  Site administration > Development > Purge all caches
to make that change take action
and delete all images from your moodledata sub folder filter/tex
and refresh your browser you should see all Mimetex images smaller

You can use also check commands from http://www.forkosh.com/mimetexmanual.html or try to switch to MathJax filter or add MathJax tags to Additional HTML - that makes using Latex formatting easier