Distubing behavior in Random Question spontaneously(?) switching to different question pool

Distubing behavior in Random Question spontaneously(?) switching to different question pool

by Sharon Strauss -
Number of replies: 4
Picture of Testers

Hello,

I have a very strange and disturbing Moodle issue I am trying to figure out. It is hard to describe well, as I am not entirely sure what is going on, but I'll do my best.

I have a Moodle course that has a little over 90 quizzes, all of which grab randomly from 90+ question bank categories. Quiz #1 grabs randomly from question bank category #1, etc.

The instructor contacted me, because when he went to update quiz #73 (a.k.a. Q73), he noticed that it was grabbing questions from category #74. Then he looked at an student attempt made to that quiz earlier in the summer and saw that the attempt indeed used the correct pool of Cat #73. At some point, the random question pool changed--although the log does not show anyone even looking at that quiz between the student attempt and the instructor going to update quiz #73. Meanwhile quiz #74 started to pull from some questions from banks #73 and some from banks #75. Again, attempts show that quizzes #73 and #75 were correct earlier in the summer, but there are no logged changes to those questions to account for this.

This is on Moodle version 3.2 In trouble shooting, we restored to a version just before we did a minor upgrade from 3.2.3 to 3.2.4 to see if that might have somehow been a cause. The quizzes were correct, just before the upgrade. However, the instructor then looked at an almost identical course site he used last year's on Moodle version 3.0 (it is still up, on a separate server). He noted that some quizzes on that had also switched on the Moodle 3.0 site.

Looking at the log, it seems that there may be a correlation with these changes and edits of the quizzes in the rows above or below the corrupted quiz. The instructor wrote, "By using 'review attempts' I discovered that the change happened between Nov 13 and Nov 15 of 2016.  All the attempts from Nov 13 or earlier correctly had the three random Q74 questions, and all the attempts from Nov 15 and later had the incorrect mix of Q73/Q74/Q75 (these are easy to spot because there was only one question in the Q73 set [so it wasn't really a "random" question], and it contains a picture of an IR spectrum).  I looked at the logs for activity in the Q74 quiz, and there was nothing at all on November 14, and nothing suspicious (i.e. no activity by me or any other person with teaching privileges; just attempts by students) on November 13 or November 15."

Although I did not see any log activity for KQ74 either, I did find that the instructor edited Q73 on November 13 and November 14.  Strangely, I found similar activity in the neighboring quizzes around time frame they seem to have become corrupt. However, I have yet to be able to replicate this issue, even in a newly restored version of this particular class.

I am totally stumped on how to proceed with this, but having quiz questions spontaneously change is highly disturbing. I welcome any insights or suggestions you may have.



Average of ratings: -
In reply to Sharon Strauss

Re: Distubing behavior in Random Question spontaneously(?) switching to different question pool

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

To help you work out what is going on, let me explain the database structure here.

On of the key tables is quiz_slots. This links quizid to questionid, so tells you which questions Moodle thinks are in the quiz, and for all the question in the quiz, the slot column tells you the order.

For random questions, the questionid in quiz_slots will point at a 'question' in the question table with qtype 'random', and then the important thing there is what the category is. When a student starts an attempt at the quiz, the quiz attempt will be build by picking a real question from the same question category to go in its place. (And, not relevant here, but the question text for the random question will be 0 or 1, which is the setting for whether to include subcategories.)

So, the symptoms you see could be caused by the quiz_slot entry changing to point to a different random question. Or it could be that the random question was edited to move it to a different category. If you look at the raw data, you should be able to tell which.

There are two known issues that might be relevant:

  • MDL-36088 - Adding/modifying questions to/in the question bank is not logged - I suspect this one. Just because there is nothing in the logs, you can't conclude this wasn't a change by a human.
  • MDL-45851 - Duplicated quizzes reuse qtype_random (and should not)

Other than that, this part of the code has not changed much in a long time, and is widely used, which means it is likely to be reliable. Also, you kind-of explore the possibility that did the DB corrupt the rows, but that is another really unlikely hypothesis.

You have my sympathy that you are only discovering now something which went wrong last November. That long time delay never helps with diagnosis.

Hopefully the above clues are some help in diagnosing this. When you know more, get back to us and we may be able to help further.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Distubing behavior in Random Question spontaneously(?) switching to different question pool

by Sharon Strauss -
Picture of Testers

Tim,

Thanks for the detailed response! That is helpful.

I'll look into those and a couple of other ideas we have and post back if I have anything new to share.

Average of ratings: -
In reply to Tim Hunt

Re: Distubing behavior in Random Question spontaneously(?) switching to different question pool

by Sharon Strauss -
Picture of Testers

Hi Tim,

I think you nailed it and solved the mystery. Unfortunately, that does not solve the problem. The logging does not concern me so much, but MDL-45851 is big problem since it makes quizzes using random questions untrustworthy. It looks like you have been working on that for years (THANK YOU), but nobody is currently assigned assigned a lead on this.

Do you have a suggestion for getting that bug back on the radar? It is one thing to have minor bugs languish for years, but bugs that create unstable behavior should not be left in permanent limbo.

-Sharon

Average of ratings: -
In reply to Sharon Strauss

Re: Distubing behavior in Random Question spontaneously(?) switching to different question pool

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I have not been working on that issue for years. I have been ignoring it for years. There are only so many hours in the day.

The only guaranteed way to progress the issue is to create a patch that is self-evidently right, with tests to prove it.

Average of ratings: -