I think most people doing serious maths in Moodle tend to use add-on question types such as STACK (https://moodle.org/plugins/qtype_stack) or Formulas question type (https://moodle.org/plugins/qtype_formulas). You might want to take a look at those.
However, that is no reason not to improve the core calculated question type. I don't know exactly the right way to do it, at least not without further thought. You are certainly asking the right questions: thinking about how best to fit in with the existing code is certainly the way to start. Another thing I would encourage you to do early on is to find whatever unit tests already exist for this code, and think about what tests you should add for the new functions. That would be a good way to define exactly what it is you plan to implement.
I will try to have a look at the code myself, tomorrow, to see if I can give a more helpful answer.