Quiz

Pierre > When a quiz is started and there is a first attempt from a student we should not allow any question changes.

I'm afraid I'll have to disagree on this matter. This is related to a recurring discussion we are having in this forum about using quiz questions for learning purposes versus for testing purposes (see this discussion, this other one, etc...).

Since I only use the quiz module for self-evaluation and learning purposes, I need to be able to edit questions even after students have attempted quizzed, and ever more so after than before. This is part of an iterative process of developing computer quizzes, where new versions of the questions are based on the students' actual responses. So for me any mechanism which would prevent me from editing questions actually used in a quiz from the moment students have started taking the quiz would ruin my teaching strategies and is out of the question.

On the other hand I do understand Pierre's arguments, when quizzes are used solely or mainly for testing purposes. May I suggest the following solution: if a quiz's parameter Adaptive mode is ON, then it means that quiz will be used for learning purposes and teacher should be allowed to edit questions at any time; if adaptive mode is OFF then the quiz is being used for testing purposes and its "integrity" should be preserved.

Joseph

I think that the requirements are actually much more complicated that Pierre's list, as can be seen by the response. What is good for some sorts of quiz is not good for others, and Moodle should be capable of supporting all educational uses of quizzes. What we currently have, while a bit haphazard, is actually quite a good compromise, so I would rather not start changing things until we have worked out a really good plan, then we should change it once and for all.

(I'm going to make some other point too, but to ease following discussions, I will put them in separate posts.)

Actually, with the tools available in Moodle 1.9, and the right processes around the production of deployment of quizzes, you can do a lot. Here is what we are planning to do for high-stakes quizzes at the OU (or exams and assessment department have been involved in developing this, and they are hard people to satisfy).

1. There will be a formal process to manage the production of high-stakes quizzes. We even have a workflow block (one day we will will get around to putting it into contrib) that we will use to help manage this process. In our block, a workflow is a sequence of tasks, someone is responsible for doing each one, and when they have finished a task, the system emails the person responsible for the next task to tell them it is now their turn.

2. Using the new question bank, which lets you have questions an categories private to one particular quiz, we will have a rule that a high-stakes quiz can only use questions from that quizzes private bank. (This has to be checked manually - at least at the moment.) This is because ...

3. Once the quiz and been created and tested, we will override all the roles on the quiz, to remove all the capabilities to edit all the questions belonging to the quiz, or change any of the quiz settings. (Again, this has to be done manually, at the moment.)

If you do all that, then actually you are very close to what Pierre is suggesting.

Actually, there is currently a bug that prevents 3. being done - that is MDL-11421. However, I am going to fix that soon if no one else does.

(The third and final point I wanted to make.)

I want to explain how we decided to handle this in OpenMark, the other OU online assessment system. Just to remind you, OpenMark has been being used for high-stakes assessment since Autumn 2005, but it has also been used for formative quizzes, e.g. https://students.open.ac.uk/openmark/s205.ayrf/) Naturally, we have had a number of instances of faulty questions only being detected after the quiz went live to students, and what we originally implemented for question versioning has let us deal with each problem that has arisen, I am inclined to base whatever we do in Moodle on what is currently done in OpenMark.

(From this point on, I will refer to OpenMark as Om.)

In Om, questions are files on disc (actually, they are Java .jar files that also contain XML). They have a file name like ayrfs205.reactions.q01.1.3.jar. That should be broken down into two parts, the question id 'ayrfs205.reactions.q01' and the question version '1.3'. There is no good reason why the version numbers have two parts, it would work just as well if they were simple integers. Anyway, they are totally ordered, so 1.0 < 1.1 < 1.2 < ... < 1.9 < 1.10 < 1.11 < ... < 1.100 < ... < 2.0, etc.<%1%0%>
An Om test (or word for quiz) is defined as a list of question ids, so for example, ayrfs205.elements.q01, ayrfs205.elements.q02, ..., ayrfs205.reactions.q01, ayrfs205.reactions.q02, ... (The definition is actually stored in an XML file on disc.) Once a student has attempted the quiz, you are not allowed to edit the test definition file.

When a student first sees a particular question as part of a quiz attempt (Om tests are one question per page, so this may be some time after the attempt starts, it is the time the student first navigates to that page) the system looks to find the latest version of that question, and then records which version of the question this student received as part of this attempt. Then, for the remainder of that attempt, and forever afterwards (for example when a teacher reviews the attempt) the same version of the question is used.

This 'once you have started, you always get exactly the same version' is very important for Om, because some of our questions are highly adaptive. For example, imagine a Cloze questions which starts of with shortanswer boxes, but if a student answer incorrectly in one box twice, it gets replaced with a dropdown menu with restricted options. Obviously, that only works if you can replay the exact sequence of interactions that the student had with the question. If not, the sequence of responses stored for that student in the database might no longer match the options available in the question they saw.

Anyway, this lets up update a faulty question in a formative quiz, and any students who do it after the update see the repaired question.

The only other thing you can change in Om is the weight of each question, used when calculating the overall score. This can be changed later, so when a faulty question is found in a summative test, then, unless it is something trivial like a spelling error which will have disadvantaged students who got the bad question, we don't change the question, we just zero-weight that question so that it does not contribute to the final score.

(In Om, for summative tests, we normally don't show students their score until after the close date, so they are not aware their score 'changed'.)

The one thing that this set-up does not let you do is change a question to have an answer that used to be marked wrong now be considered correct, and then re-grade the quiz. That is a useful feature in Moodle (at least for non-adaptive quizzes), and one we would need to retain.

Although teachers are more or less responsible for their courses contents and can even add new question types, Moodle.org is responsible for the actual core question types.
The Moodle responsibility extends also to the students using Moodle quizzes for their grading.

I propose that at least we add an automatic warning message when the students review their question results if the question has been modified since their attempts has been done (or closed).
We left to the teacher to add an explanation of the question modifications purposes ( in the question text or as a general comment on the course site).

Pierre

P.S. This could be extend to all questiion types or only to the actual core question types.