I do not think this kind of thing is supported by the current quiz module. Even if it were supported, there would be difficult issues on which format the students would submit their supporting work in. You could get something along these lines by combining a quiz with an assignment, but I doubt that this would work well in practice.
An alternative would be not to have multiple choice answers, but rather answers which are symbolic mathematical expressions. This is currently supported by the Moodle webwork module. This requires the installation of another web
server software called WeBWorK which evaluates symbolic mathematical expressions. You can take a look at the following example:
https://webwork2.math.ohio-state.edu/moodle1_4/course/view.php?id=66&username=guest Click on the Print WeBWorK... and Do WeBWorK...online links.
There is a similar module called AIM, but as far as I know, it has not been publicly released. You might ask Gustav Delius, who is very active on this site for further details. Unlike webwork, it uses a proprietary component, Maple, which you would need to purchase.
Gustav is also leading a project called "Serving Mathemtics" (search these forums) which is intended to provide more options along these lines, but I expect that you will have to wait a year or more for this to bear useable fruit.
