Hi Polaris,
I think that Mathjax is able to cope with Korean UTF8 as long as the correct fonts are available. From your screen image it seems that you have the TeX notation filter enabled and are attempting to use it with the TinyMCE Mathslate plugin. Mathslate requires Mathjax to be configured and available to work. The easiest way to do that would be to enable the MathJax filter plugin.
Cheers,
Daniel