I agree with Mikael that MathJax is a good option for this kind of cases (and otherwise).
Usually the first option to find more info is to set debugging to ALL from Site Administration > Development > Debugging and go back to texdebug.php
If there is no return code and if latex/dvips/convert binaries are executable the problem is often in other (security) settings of server. Some functions of php may be disabled in php.ini (you can see them in php info - disable_functions is usually empty...) or some other setting/apache module etc - or missing permissions for the folders - might prevent creating images to moodledata subfolders like ../filter/tex or .. /temp/latex
Check also server (apache) logs if debugging does not give more details - the reason could be some "environmental bug" as well - see for example http://tracker.moodle.org/browse/MDL-33616