Automatic check for duplicated questions

Automatic check for duplicated questions

by Giacomo Ciani -
Number of replies: 13

Hi all,

I'm rather new to Moodle, and only recently I started administering tests created as Moodle quizzes. I'm encountering a few issues, and I will post them in different threads so that they can be addressed independently.

Let me add that Moodle is administered by my university's IT services, and I have no control over the installation (I may be able to request some modification/plugin installation, but I'm not sure of the answer). I also can't fine any information on the Moodle version.

First issue I have is with the calculated multiple choice question type, which is the main question type I use. It is indeed very powerful. However, one big issue I'm encountering is the possibility (not that remote) that for a given set of random parameters, two calculated answers give the same value (or two values that look the same once rounded to the requested number of significant digits/decimal places). Of course this can be avoided by a (very) careful choice of formulas in the answers and parameters space, or by manually checking the answers corresponding to each random set of parameter. Both approaches are very time consuming and tiresome.

On the other hand, it would seem easy to program Moodle so that to check for duplicated values in the answer resulting from a given random set of parameters, and discard it (or display a warning) if that's the case. Am I missing something? Does this functionality exists? If not, can it be implemented?

Thanks

Giacomo


Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
(Your IT services should be able to tell you the version of Moodle that you are using.)

As a reference, I gave this presentation at a couple of MoodleMoots this year. 
MoodleMoot2019 (Global and Mountain), "Using Calculated Multichoice Questions"

When you have Calculated Multiple Choice create the datasets, you should always look them over for duplicates. If you see a duplicate, you can manually change one of the input variable values and Moodle will then use the new value.

As far as I know, there are no "automatic" checks. And there could be situations where you would want duplicates, perhaps because one set of input values are really desired to appear more frequently.
Average of ratings: Useful (1)
In reply to Rick Jerz

Re: Automatic check for duplicated questions

by Giacomo Ciani -
Hi Rick,
thanks for your answer, and for the link to the presentation, which is very useful.

Given that you confirm that there is no automatic check, I guess I should move to the tracker and submit a feature request there. Manually checking 100 datasets for each question I create (and I create 30 for each test) is not very practical, and extremely time consuming. Automatic checks, on the other hand, are what computers are good at... smile

Let me also point out, for the sake of clarity, that I wasn't referring to duplicated set of parameters, but to duplicated values in the calculated answers for a given set of parameters. This can happen because two different formulas give exactly the same number for particular values of the variables, or because the two calculated values are close enough that when rounded they look the same.

Quick example: What is the work needed to lift of h meters a weight of mass m kg.
Answers:
A: {m}*{h}*9.81 (correct)
B: {m}/{h}
C: pow({m},2)/{h}*g

If a random set of parameters is generated in which, cy chance, h = sqrt(m) (for example m = 4, h = 2), then answer A and C will both formulas will result in the same value, and there is no way for the student to know which one to pick.
Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Okay, I better understand your situation. I can give, perhaps, a simpler example.

Assume that the question is "What is the area of this rectangle?"
The correct equation is length x width.
The incorrect equation is length + width.

However, if length and width both equal 2, both of these equations produce the same result.

So you want Moodle to "flag" you when two equations produce the same result, given some randomly generated input values.

Well, interesting request. My guess is that you don't want Moodle to fix the problem. This would be up to the teacher. But you do want Moodle to show you ("flag") that your equations have produced two (or more) identical answers.

Actually, I understand what you mean. I think that I was once caught by this kind of situation. So, I have learned to scan the answers and manually look for these problems.

Maybe Tim Hunt is watching this conversation. He's the Moodle Quiz Engine guru, and a good guy. I wonder if he would think that flagging these situations, maybe with a red color, is easy to program.

Sure, you can create a Tracker item for this (after searching to see if it already exists.) This is a good way to get your idea noticed.

In my own case, I typically create questions with less than 25 or so "sets" of answers, so a manual scan is not that hard. And I wonder if Moodle did flag the dataset with duplicates, maybe with a red color, if it would really save much time since you would still need to look at the results? But sure, color or some other kind of mark (since color by itself might not meet "accessibility" guideline for colorblind folks) might be handy.
Average of ratings: -
In reply to Rick Jerz

Re: Automatic check for duplicated questions

by Giacomo Ciani -
Yep, you got it right, and your example is simpler than mine.

Two comments:

1) I don't know if I completely agree that Moodle shouldn't correct the answers (where "correct" here means simply generate a new random set). One possibility: add a checkbox on the dataset generation page that says "avoid duplicated values". If that's checked, for every new dataset generated, Moodle checks if it results in duplicated answers. If it does, the dataset is discarded and a new one is generated. personally, I would use it all the time.

2) Even if one decides to not let Moodle "autocorrect" the dataset, there are plenty of ways of highlighting the problem in a way that is noticeable without scanning the datasets. For example, when new datasets are generated, if there are problematic ones a pop-up a window could appear
pointing to their number
Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Maybe so, but for #1, how does Moodle know that the instructor did not intend to have two correct answers? And the issue is with the person who created the questions. There could be situations where Moodle will not find a good dataset, and how many tries should Moodle attempt?

For #2, Moodle is generating random values correctly. The error is with the equations.

I am not saying that this can't be done. But I am wondering about the logic that a programmer would have to go through. If you make a feature request, try to provide a flowchart of the logic.
Average of ratings: -
In reply to Rick Jerz

Re: Automatic check for duplicated questions

by Giacomo Ciani -
Rick, I guess you know by now that I like debate, but this time I don't really understand your objections.

1) not sure what having "two correct answers" has to do with "having duplicated answers", which is what I am talking about. And Moodle will know the intentions of the instructor, because he/she would have checked the "avoid duplicated values" option I'm proposing. Besides, if you think about it, there is really no real-life case in which you may want duplicated answers in a multiple choice calculated questions (if you can find one, please let me know).
Also I ond't understand what you mena by "the issues is with the person who created the questions". If the issue is not wanting to spend an hours performing a mechanical check that a computer can do in a fraction of a second, well, I wouldn't call it an issue. If the issue is having chosen equations that can yield duplicated values for some combinations of the input parameters, believe me: avoiding this situation for a moderately complex physics exercise with parameters varying in a reasonable range is not a trivial mathematical problem.

2) I never said Moodle was in error. Nor I think it is correct to say that the equations are not correct. In fact, I don't think there is an error anywhere here. But there is something that can be done in an easier, faster and more reliable way, which is generating datasets that do not result in duplicated answers. Moodle's job is not (only) generating random numbers (one line of code in any common programming language is good enough at that), but it is (in this context) to help create and administer viable and formally correct quizzes. Any task related with this activity that can be performed by a computer is in principle in Moodle's scope, and the one I'm proposing is one of these (and an easy one, even).

As for the logic, the details can be discussed, but a baseline is pretty trivial. If the option to avoid duplicate answers is checked:
A) set a counter C = 1
B) generate a set of parameters S
C) Check if S results in visually identical answers (once rounded as requested)
D) if is does not, S is good ad gets added to the dataset
E) if it does not:
F) if C is smaller than a given threshold (say 100), go to B)
G) if C is equal to the threshold, display a message ("Unable to find an unambiguous set of parameters in 100 trials. Please check your answers and/or adjust the parameters range") and abort the operation.

No black magic... and I bet that for a reasonably designed questions E) will happen a few times, but G) very, very rarely,
Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Giacomo, you are correct about #1. I wasn't making sense.

No, Moodle isn't in error. We are brainstorming a better approach.

Relative to the logic, maybe so. Maybe the message can show itself, but then we (the creator) will have to scan the 100 trials and see what is going on. There might be a clue about why we are getting duplicates. It should probably be left up to us what we want to do. We could, for example, delete all (as we can right now), or delete the last 10, and just regenerate 10. Alternatively, if we did let Moodle regenerate all, and if we didn't give Moodle enough flexibility, Moodle might be in some kind of infinite loop.

I am not sure if we are debating this, rather just discussing a variety of methods and logic about your idea. 😀

Average of ratings: -
In reply to Rick Jerz

Re: Automatic check for duplicated questions

by Giacomo Ciani -
Hi Rick,

I'm not a native speaker and maybe my use of the word "debate" is a bit off-target. "Discuss" sounds goo to me smile

As for the logic, let me explain better. My (basic) proposal:
- you ask Moodle to add N sets of parameters
- for each one of them Moodle goes through the decision process outlined in my previous message. So 100, or whatever, in that case has nothing to do with N, it is just how many timea you want Moodle to keep trying, to see if you are lucky and find a set that works (in the sense of not giving duplicates). It is your "timeout" to avoid infinite loops. The higher that number, the lower that chances of a fail.
- in the event of success, Moodle adds a set of parameters and repeats the procedure with the next set.
- in case of failure, it shows the error message and stops wherever it is.
- or, it stops when it has generated N sets.

I'm fairly confident that only two things will happen in rel life:
- the instructor mad a mistake in writing the equations, and two of them, although apparently different, always yield the same result: in this case the error comes up already when Moodle tries to generate the first set.
- the equations/paramters range are such that the equations, although not identical, can sometime produce the same result. The above procedure should just be able, with very, very high degree of success, to reject the "accidental duplicates" and just go on.

So, in practice, I expect that the instructur gets an error immediately, or never. However, the procedure should be robust in any case. It's true that I can think about ways of constructing (quite artificial) cases in which the duplicates happen not 100%, of the times, but often enough that Moodle may be successful in generating a few datasets and then fail later on. The procedure is however robust also in these cases: Moodle will save the sets it was able to generate before incurring into the error.
Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

I think that you are to the point where it is now worth it to create a Feature Request in the Moodle Tracker system.

With the calculated multiple-choice question, the Moodle developers chose to create x number of datasets as a last step in the process and then left the burden of reviewing the dataset to the instructor.  At times, I have wondered why Moodle doesn't create the variable values "on the fly?"  Why specific datasets.  However, as I continue to use this question type, I have come to appreciate why the specific datasets are created... so that you are not caught with any surprises.  So I like the current method.

As I have mentioned, I have been caught with the exact problem that you describe, so when I review my datasets and answers, I look for duplicate answers, and immediately fix them by typing in new values.

Having some form of automatic check could be useful.  I am not the developer, so I am not sure how easy or hard some of this programming can be.  But I think that we have identified some techniques.

Moodle is the only major LMS with calculated multichoice questions, a very powerful question type.

Average of ratings: -
In reply to Rick Jerz

Re: Automatic check for duplicated questions

by Giacomo Ciani -

Hi Rick,

I share your considerations about the "variables on the fly" vs "pre-generated datasets". Both approaches have pro and cons... so in the end it's a matter of taste. As yuo may have understood, I'm one that prefers to spend more time in setting up the automation and less time manually reviewing the result, but in this case it's not obvious which approach is better, and I'm perfectly fine with the pre-generated. Of course, we can still try to sneak-in some automation at generation time, like we are trying to do here smile

Just for your curiosity, my reluctance for manual review partially comes form the fact that the Italian university system requires me to provide 5 attempts for students to pass my exam. With the number of students taking the exam averaging to 100 and the quiz comprising 30 questions, it is a huge amount of answers to review!

I'll put in a feature request. I hope it'll get at least your vote smile

Thanks for the discussion

Giacomo

Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Your idea has merit.

When you get the feature request created, come back to this forum and let us know.

Average of ratings: -
In reply to Rick Jerz

Re: Automatic check for duplicated questions

by Giacomo Ciani -
I've created a feature request on the tracker:
https://tracker.moodle.org/browse/MDL-68033

Initial response from the developers (Tim Hunt in particular) is not encouraging, as what I'm proposing it seems more complicated than anticipated. But discussion is (I hope) ongoing.
Average of ratings: -
In reply to Giacomo Ciani

Re: Automatic check for duplicated questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Giacomo, I think that you are getting a sense of how Moodle, and idea generation works. In your Tracker item, Tim Hunt has provided some interesting issues about your idea.

Incidentally, Tim is a really good guy, one of the most open programmers that I have come across. He's really sharp. But he also listens. So even if your idea doesn't go anywhere right now, Tim keeps all of these ideas in his head so that when appropriate, he can integrate them into other ideas.
Average of ratings: -