(The third and final point I wanted to make.)
I want to explain how we decided to handle this in OpenMark, the other OU online assessment system. Just to remind you, OpenMark has been being used for high-stakes assessment since Autumn 2005, but it has also been used for formative quizzes, e.g.
https://students.open.ac.uk/openmark/s205.ayrf/) Naturally, we have had a number of instances of faulty questions only being detected after the quiz went live to students, and what we originally implemented for question versioning has let us deal with each problem that has arisen, I am inclined to base whatever we do in Moodle on what is currently done in OpenMark.
(From this point on, I will refer to OpenMark as Om.)
In Om, questions are files on disc (actually, they are
Java .jar files that also contain
XML). They have a file name like ayrfs205.reactions.q01.1.3.jar. That should be broken down into two parts, the question id 'ayrfs205.reactions.q01' and the question version '1.3'. There is no good reason why the version numbers have two parts, it would work just as well if they were simple integers. Anyway, they are totally ordered, so 1.0 < 1.1 < 1.2 < ... < 1.9 < 1.10 < 1.11 < ... < 1.100 < ... < 2.0, etc.<%1%0%>
An Om test (or word for quiz) is defined as a list of question ids, so for example, ayrfs205.elements.q01, ayrfs205.elements.q02, ..., ayrfs205.reactions.q01, ayrfs205.reactions.q02, ... (The definition is actually stored in an
XML file on disc.) Once a student has attempted the quiz, you are not allowed to edit the test definition file.
When a student first sees a particular question as part of a quiz attempt (Om tests are one question per page, so this may be some time after the attempt starts, it is the time the student first navigates to that page) the system looks to find the latest version of that question, and then records which version of the question this student received as part of this attempt. Then, for the remainder of that attempt, and forever afterwards (for example when a teacher reviews the attempt) the same version of the question is used.
This 'once you have started, you always get exactly the same version' is very important for Om, because some of our questions are highly adaptive. For example, imagine a Cloze questions which starts of with shortanswer boxes, but if a student answer incorrectly in one box twice, it gets replaced with a dropdown menu with restricted options. Obviously, that only works if you can replay the exact sequence of interactions that the student had with the question. If not, the sequence of responses stored for that student in the
database might no longer match the options available in the question they saw.
Anyway, this lets up update a faulty question in a formative quiz, and any students who do it after the update see the repaired question.
The only other thing you can change in Om is the weight of each question, used when calculating the overall score. This can be changed later, so when a faulty question is found in a summative test, then, unless it is something trivial like a spelling error which will have disadvantaged students who got the bad question, we don't change the question, we just zero-weight that question so that it does not contribute to the final score.
(In Om, for summative tests, we normally don't show students their score until after the close date, so they are not aware their score 'changed'.)
The one thing that this set-up does not let you do is change a question to have an answer that used to be marked wrong now be considered correct, and then re-grade the quiz. That is a useful feature in Moodle (at least for non-adaptive quizzes), and one we would need to retain.