On my site using the TeX notation filter (MathJax disabled) with PNG I get:
which looks okay to me. All my other settings are default. Are your formulas displayed like this? Or worse?
Are there any TeX sequences you want me to try or anything I can share about my config? I have pdfTeX 3.14159265-2.6-1.40.21 (TeX Live 2020) on CentOS 7 with Moodle 3.8.