Yes, you can easily add styles to style select list with init code of tinymce (in this case theme/math/editor1t.php or editor1s.php)
There is a line
content_css : "<?php echo $CFG->themewww .'/'. current_theme() ?>/css/content.css",
which means that tinymce looks for styles from folder theme/math/css/content.css
and I have defined there two styles: span.AMedit and span.JeMedit
try to add for example line:
p.test1 {color:red;}
to content.css and you should see in your style select list a new style "test1" that adds p tags with style "color:red"
And if you have different themes with different configurations (different init code for tinymce) you can set each theme or each configuration to use different styles - simply change the location (path) of file content.css (here content.css is loaded from current theme of moodle - it can be set to one absolute path as well)
If content.css is not used you can also use tags like
theme_advanced_styles : "Header 1=header1;Header 2=header2;Header 3=header3;Table Row=tableRow1",
in init code but it is easier to use and edit file content.css.
Note also that http://korpelainen.net/mathplugins.zip has the latest tinymce included - read http://moodle.org/mod/forum/discuss.php?d=123806 - but some plugins are modified for moodle and for example media plugin is not included - you can download the full package of tinymce from http://tinymce.moxiecode.com/ and read documentation from http://wiki.moxiecode.com/index.php/TinyMCE:Index