"Out of Sequence" while taking quiz

Re: "Out of Sequence" while taking quiz

by Tim Hunt -
Number of replies: 4
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

This is a very specific error, in that it is only triggered in one place in the code, to detect and avoid a particular problem.

The potential problem is that as students interact with a question, it changes state. You can see that when you review a quiz attempt. Under each question is the grey response history area. That has a number of rows in it.

In normal use, what happens is that the student loads one page a quiz with some questions on. They do something, then they submit that page, and the new responses they have given get processed.

The problem comes when they try to do something more complex. For example suppose you

  1. Open the quiz in the web browser on your computer.
  2. Open the same quiz on your mobile phone (or in another tab, or another browser, on your computer).
  3. Answer some questions in the first browser, and click Next or Check or something.
  4. Go to your mobile, which still contains the questions as they were before the action you too in 3. above, and try to perform a different action here.

Now, Moodle is going to get some data that relates to how the questions were in steps 1. & 2., and which does not account for the changes caused by action 3. That (potentially) makes no sense, and has to be rejected. (Think, in particular, about a question behaviour like Interactive with multiple tries.)

In order to detect this situation, we add a marker to each question in the HTML. For example, if you hunt around in the HTML at https://github.com/moodle/moodle/blob/master/question/engine/questionusage.php#L744, you will find

<input type="hidden" name="q4:1_:sequencecheck" value="3">

This indicates that the first question in this quiz attempt has previously had 3 actions performed on it (which you can see in the response history table). In another action is performed on this question (which could only be manual grading by the teacher since this is a finished quiz attempt) that value will be included with the submitted data, and before anything is processed, Moodle will check in the database, and verify that there are still only 3 actions for this question, before processing the manual grading as a 4th.


Of course, with any mechanism like this that is designed to detect problems, you have to worry about false positives. That is, the mechanism erroneously rejects a submission that is OK. However, this code has been there since Moodle 2.1 and I don't recall anyone else having problems until now.

But, what could be going wrong here? and how might we be able to diagnose it?

It is really hard to think of what might be going wrong. The mechanism is simple and robust.


Places to look for clues:

  1. Are there any other errors in the PHP error log that might be related?
  2. Are there any JavaScript errors showing up in the web browser of any student who experiences this?
  3. Can you look at the the Moodle logs, and the Apache web logs, to try to see what the students who hit this problem were doing immediately before it happened?
  4. Can you reproduce it yourself?

I am struggling to think of any way that error could be triggered (other than through the kind of weird behaviour that we are intentionally trying to prevent as explained above). I have to vague ideas:

  1. Could it be that students are double-clicking on the Next button? Now, actually, I already thought of that long ago. You may have notice that when you click on the Next button, all the buttons on the quiz page are immediately disabled, so a second click is ignored. Except that, if your web browser is really slow, and you click really fast, you might be able to beat it. (This is why I suggest looking for JavaScript errors.)
  2. Could it be that some firewall, or web proxy, or cache, is getting in the way and somehow causing the HTTP request to get duplicated? I don't really believe this one, but if it was happening, you might be able to see it in the Apache logs.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: "Out of Sequence" while taking quiz

by Melissa Benson -
Thank you for the response!

It does make sense when it's explained but what doesn't is there is no obvious "thing" that students are doing! What's weird is that it seems to be in bunches..where there is an issue in one period but the afternoon there were no issues. I'll check to see if we can get any logs.

Would a different quiz set up make any different? Like have 1 question per page or something? Right now it's a few questions on 1 page, then hitting next goes to the submission page.

Another note from the teacher
Interestingly, I noticed that nearly everyone had the same answers erased. Meaning, after they pressed "continue" it brought them back to their attempt with only about 3 of the 16 or so answers completed. As I mentioned, I had ALL students, even if unfinished, hit next to try to save their answers as I have done with other students in the past. However it gave them the error too, so when it brought them back they also had 3 or so answers filled in and the rest blank. So to me, it almost seems as a timed error somewhere. Like an error happens or some sequence happens on the server or out in "moodle-land" so nearly everyone is effected at the same time. And whatever answers were entered at that time are saved and everything else is not. Anyway, I don't know anything just throwing it out there.

I asked how many students and if they were all hitting next at the same time
I would say with that quiz there were max 50 students taking it at roughly the same time. And, they would have been clicking next at roughly the same time but they were all over the map enough that not EXACTLY. But probably many of them within the same 5 minute span.

I'll reference your post to the teacher so he may be able to watch out for double clickers smile
In reply to Melissa Benson

Re: "Out of Sequence" while taking quiz

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Questions per page should not matter.

Intermittent bugs are a pain. Somehow you have to get from "sometimes this breaks" to "this always breaks when ... happens", but working out the ... can be really hard. I hope you can track this down.

In reply to Melissa Benson

Re: "Out of Sequence" while taking quiz

by Robert Drmola -

Hi,

  after one year of using Moodle, we have probably the same problem.


Student don't use back or other navigation button in browser

It is random error, not every test, not for every student

It is generated only in "Essay" type question

After pushing "Next" button, student is returned to the same question, and sametimes is the form blank, sometimes is filled with only part of student answer, sometimes is the answer fully saved. 

We have this problem for about two months. This error wasn't generated before, and we tested about 200 students per week.

Moodle wasn't upgraded about one year. Browsers are Google Chrome.


In reply to Robert Drmola

Re: "Out of Sequence" while taking quiz

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I know it is not easy, but you need to do the diagnosis work to go from "It is random error" to "In exactly this situation, then the bug happens." Since this is only happening on your server, you are really the only person who can do that. Once we know the specific cause, then we can fix the bug. Until then, we are stuck.