An IT specialist would tell you to put your site in debugging mode (https://docs.moodle.org/en/Debugging) and might be able to diagnose the problem from the reported messages. You can also see what happens through trial and error by gradually changing the question.
Imho, maybe the problem would be more with JSXGraph than with the Formulas question with which I have never encountered such a problem. I suggest you submit your problem to Albert Wassermann (email@example.com), responsible for the JSXGraph project, who I am sure will be happy to help you. Do not hesitate to communicate with him.