"Shuffle all but last" option in multichoice

"Shuffle all but last" option in multichoice

i le Luca Bösch -
Number of replies: 28
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Dear community

I stumbeled upon an old issue – was reminded of it, at least – when the other day a lecturer talked to me about a 'other reasons' option he uses to introduce in his multiple choice questions.

The options are shuffeled and he wanted to preserve that one to appear as the last.

That reminded me of https://tracker.moodle.org/browse/MDL-35236 – «Quiz module, multiple choice, would like "shuffle all but last" choice».

What I would like to know here: is there really the need for this?

It would be great if you could express yourself on this one. Thanks in advance!

Best,
Luca


(To be honest, that lecturer then wanted to have 'other reasons: …………' paired with an short answer input field in an MC option. I declined.)

Average of ratings: Useful (1)
In reply to Luca Bösch

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

What I would like to know here: is there really the need for this?
It would be great if you could express yourself on this one.

As always, here is my point of view as a user.

It is often said that a feature would be implemented if the demand is sufficient. In reality, it seems that other factors, equally important if not more important, affect the implementation of a given feature. These factors are whether a developer has the time to implement the feature, based on his/her workload and the technical difficulty in implementing the feature, practical feasibility and usefulness of the feature and whether the developer thinks himself that the feature is relevant or not.

Probe other developers and users to know if they think a new feature is relevant should not be used mechanically, i.e. no vote, no implementation, but rather as one of the decision criteria. When in doubt about a feature, the survey can be used simply to obtain opinions.

Quiz forum users will not express themselves for a feature that does not attract their attention. This does not mean there is no need. If, instead of asking on this forum, you google "all of the above and none of the above for multiple choice questions", you get 144 millions results. In other words, this feature must be relevant. The lack of response in this forum simply means that there is no objection to the implementation of the feature.

In summary, the feature is not technically difficult to implement, as you point out in the tracker, it is easy to show that the feature is used universally in multiple choice questions, and no one has objected. The last criterion is your judgment.

In my opinion, this feature is certainly useful and I think you should implement it. In the meantime, your lecturer and other users can use the script I mentioned in my previous post. ata

In reply to Luca Bösch

Re: "Shuffle all but last" option in multichoice

i le Ewout ter Haar -

I think this would be very useful. Just this semester I had to deal with a lot of questions with "none of the above" distractors. Rewriting them to "none of the other alternatives" still gets you weird results when this option is the first one.

Sure, "none of the above" is not a very good distractor. But we have to deal with normal educators, not professionals in educational assessment.

In reply to Ewout ter Haar

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

I googled again this time with "multiple choice none of the above distractor" and checked the top twenty results:

  • University of Waterloo, Canada
  • The Canadian Journal for the Scholarship of Teaching and Learning
  • Vanderbilt University, USA
  • The Journal of Experimental Education
  • North Central College, Naperville, USA
  • University of Toronto, Canada
  • Brigham Young University , USA
  • University of Tasmania, Australia
  • University of Manitoba, Canada
  • University of Texas, USA
  • Journal of Applied Research in Memory and Cognition
  • University of New Brunswick, Canada
  • University of California, USA
  • etc.

They all discourage the use of "none of the above" and "all of the above". So I changed my mind! I now think that Moodle should also discourage their use and therefore not implement the feature. The little script I wrote would suffice for their use, hopefully, very limited.

See also this forum discussion ↗.

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Interesting! I have been using "none of these are correct" for quite some time, because I have always preferred to shuffle answers. I have also noticed that some of my textbook authors have begun using my language, too.

The "all of the above" and even "all of these are correct" has never worked out well for me, so I get rid of these.
In reply to Luca Bösch

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Luca,

I thought about it, did further research and changed my mind again.

There are actually serious references that use NOTA (None Of The Above).

I looked at the SAT and ACT tests. In language tests, "No change" is used regularly in first position. In math tests, "I would be guessing" is used in last position.

This can be done if the order of the choices is not random. However, it would nice to be able to fix the position of certain choices when the order is random.

I also noticed that the "Cross out" option is available and that some examples are in study mode.

So here is my suggestion:

  • Implement the fixing of the position of choices in random order mode.
  • Implement a "Cross out" mode.
  • Implement a "Study" mode.

In my (latest) opinion, these implementations are urgent, necessary and must be done in core, not in a plugin.

I made all these features in java script because I do not know yet how to do them in php. I could do it one day, but for the moment, it would be nice if you could do it. ata

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

"None of the above" valid example
Ref.: College Board
Javascript by Dominique Bauer ↗


"I would be guessing" always in last position
Ref.: Khan Academy
Javascript by Dominique Bauer ↗


"Cross out" / "Bring back"
Ref.: Khan Academy
Javascript by Dominique Bauer ↗



"NO CHANGE" always in first position
Ref.: Act English practice test
Javascript by Dominique Bauer ↗


"Study mode"
"Independent scrolling of text and question"

Ref.: College Board
Javascript by Dominique Bauer ↗
Javascript by Dominique Bauer ↗


For all question types:
Prevent moving forward in sequential mode
Ref.: Khan Academy
Javascript by Dominique Bauer ↗

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

In my opinion, Moodle should not exclude the possibility of placing NOTA (None Of The Above) in the last position in a random order, but leave it to the discretion of the teacher to use NOTA.

There are many examples of using NOTA in several sectors (academic, administrative, etc.)

American Optometric Association

Texas Department of Public Safety

Government of India - Ministry of Corporate Affairs

Government of Canada - Natural Ressources

and many other examples on https://www.google.com/search?q=allintext:+none+of+the+above+filetype:pdf.

By the way, it seems possible to lock positions in CANVAS:


In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

The multiple-choice question is certainly one of the most used for all tests, in all sectors and in all countries.

Despite this, Moodle does not invest enough in its development. Compared to that of other LMS, Moodle's multiple choice question is no longer up to par.

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
I do like this concept of being able to lock the position of any choice, not only the last. Yes, this is a case where another (competing) LMS has a feature that is better than Moodle, and Moodle should do things better. I just voted for MDS-35236.

Since this feature might take a little development, I wonder if it should be proposed in the Moodle User's Group as a project? Your thoughts, Dominique? (Maybe you already proposed this.)

Until "locking" is implemented, I still think that "All of these are correct" or "None of these are correct" would be the way to phrase these choices, which then makes their position less of a factor.
In reply to Rick Jerz

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Hello Rick,

Thank you for voting for this issue ata (The link to the Tracker is actually MDL-35236 ↗)

I think you already mentioned Moodle User's Group. It looks cool and may be a better place than this forum to promote the idea of fixed choices in random-ordered MCQ.

I do not know what to think of Moodle. On the one hand, the software is distributed free of charge and, on the other hand, given its high popularity, I expected it to be up to snuff, at least with regard to the multiple-choice question.

In my opinion, the features of fixed choice, crossing out, study mode, independent scrolling and preventing moving forward should be considered basic by Moodle for the MCQ and should already have been implemented.

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Dominique, my school is using Canvas, had been using Desire2Learn, and before that, I used Blackboard and WebCT. For what I do in my courses, I am sticking with Moodle. Canvas does not have a question bank, and so it bundles questions into a course (unless it has recently changed.) It also has very limited "import" abilities. But there are some things that Canvas got right, as you pointed out.

In your Canvas example, is that a "multiple choice" question type?  The reason why I ask is that in my version of Canvas, I do not see a multiple-choice question display as you have shown.  And I think my school is using the latest version of Canvas.
In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Thomas Korner -
Picture of Plugin developers Picture of Testers
Dear Dominique
Thank you very much for these interesting ideas. We at ETH Zürich invested some resources in the MC-qtype. In fact, we ended up in splitting it into 2 qtypes: Multiple true false (mtf) and single choice for several reasons (in addition we've developed a specialised question type called kprime). The answer option NOTA can be fixed with mtf in a nutshell. For example in your example above, you would ask:
***
The fundamental frequency of a piezoelectric crystal used in ultrasonics is a function of:
a) its thickness (true/false)
b) velocity of sound in the crystal material (true/false)
****

In the single choice (sc) we have implemented a cross-out option with a variety of grading options.

Both of them are in use for several months now with great success.

You find all of them in our github (and we are in the process in submitting both to moodle.org):
Single Choice - https://github.com/ethz-let/moodle-qtype_sc
Multiple True False - https://github.com/ethz-let/qtype_mtf

Best regards from Zürich
Thomas Korner
In reply to Thomas Korner

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Hello Thomas,

I agree with you when you say in your github:

The MC question type as it exists in Moodle is not very ideal, since you can only configure questions which are not very well designed regarding best practices of MC questions.

I hope your SC question will be an improved version of the Moodle core multiple-choice question. That is, it will incorporate all the functions of the latter plus the new features you have developed.

I think you meant:

The answer option NOTA can be fixed with SC in a nutshell.

Please consider that all choices (at least the first, before last and last) should be able to be fixed.

Your MTF question seems like a great idea! Again, I hope your MTF question will be an improved version of the Moodle core true or false. That is, it will incorporate all the functions of the latter plus the new features you have developed.

Best regards from Montreal ata

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Luca Bösch -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Let me add something here again.

Those are the question types mentionned by Thomas Korner in a screen grab.

You see that Multiple True/False can implicitely let you do "All/None of the above" (then all options are true/false) without mentioning this verbally.

The Single Choice then has the "Cross out" mode.

I'd like to remind you not to extend the base question of this topic too much. That's whether the "Shuffle all but last" option in multichoice should be introduced.

Discussion about cross-out, fixing of the position of any option choice (not only the last) and not meeting features other LMS have are valid but probably should be lead in another context. I encourage you to open "New function" or "Improvement" type Moodle Tracker issues or insert new topics in this forum.

Having said that: the proposed feature ("Shuffle all but last") has been programmed and the code is ready. Should it be integrated?

(Side note to Dominique Bauer: your Javascript solution sure works but it has the backlash of going and comparing with the displayed string of the chosen option. How does this behave if you have multiple questions with the same option on the same quiz page? Also, multilangual questions would probably struggle with this.)

Best,
Luca

In reply to Luca Bösch

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Hello Luca,

You see that Multiple True/False can implicitly let you do "All/None of the above" (then all options are true/false) without mentioning this verbally.

The MTF is not Moodle's MC core question. Locking the position of any choice should be integrated in Moodle's MC core question.

The Single Choice then has the "Cross out" mode.

The Single Choice is not Moodle's MC core question. The "Cross out" mode should be integrated in Moodle's MC core question.

I'd like to remind you not to extend the base question of this topic too much. That's whether the "Shuffle all but last" option in multichoice should be introduced.

As Thomas said, « The MC question type as it exists in Moodle is not very ideal... ». Your proposed feature ("Shuffle all but last") is insufficient to make it ideal. Extending your base question was relevant.

Having said that: the proposed feature ("Shuffle all but last") has been programmed and the code is ready. Should it be integrated?

No, that's not enough. Locking the position of any choice should be integrated.

(Side note to Dominique Bauer: your Javascript solution sure works but it has the backlash...

My java scripts are not "solutions", but merely workarounds to overcome the lack of features, in the core questions, asked by users and for which they have so far received no response from developers. fa'anoanoa

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
It seems that there is a need to review what kinds of features should be included in core T/F and MC question types. And it's an interesting issue whether some of the plugin question types should be part of core, or remain as plugins. I am watching this discussion and making some comments, and letting you folks know that I have appreciated this discussion.

At some point, maybe someone can summarize the desired features and put them into a feature request. It seems to me that this could be a good MUA project proposal, but I certainly welcome the Moodle developers to simply improve TF and MC questions.  

One thing that we need to keep track of is to retain the ability to import questions from other places (such as a Blackboard export) into Moodle. Right now, Moodle does a good job importing T/F, MC, Essay, and maybe a few other common question types. If the current MC question type is improved, how will this affect both importing and exporting questions? (I am not a programmer, but I sense that import/export might take a little care.)
In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Thomas Korner -
Picture of Plugin developers Picture of Testers

Dear Dominique, dear Luca

Thanks Luca for the assist fa'aivi-le-mata

Yes, they are not core qtypes, but the installation is straight forward and we've developed a 4-way-script to batch convert mc questions into sc/mtf and vice versa. 

If there is a broad agreement we would be very happy to integrate our developments into the mc qtype (our code is open source too). But, we realized that the mc qtypes serves many different scenarios, some of them contradicting. Probably I have to be more concise: MC as it is in core at the moment has some issues with our usage as summative online assessment. Therefore we decided to develop and publish a compatible new qtype for those who are interested or have the same issues.

Best regards

Thomas


In reply to Thomas Korner

Re: "Shuffle all but last" option in multichoice

i le Samuel Witzig -

Dear Luca, dear all

Working together with Thomas Korner at ETHZ, I would like to point to one (already mentioned) argument: From a didactical point of view, the option “none of the above” should only be used in exceptional circumstances in MC-questions*. The option “all of the above” should not be used in any case*. I know that these options are frequently used, as Dominique Bauer mentioned. However, a frequent use of these options does not mean that they are good options in MC-questions that should be used from a didactical point of view. I therefore do not see a necessity for the option “shuffle all but last choice”, since I think that the quiz module should not support bad exam habits.

One argument more against the setting: The quiz has already a lot of settings to choose from. So please let’s not add another setting if it is not really necessary or even supports bad exam habits.

* See e.g. Krebs (unfortunately only in German) on page 18 at  https://www.iml.unibe.ch/attachment/7/download/mc_anleitung.pdf

Best,

Samuel

In reply to Samuel Witzig

Re: "Shuffle all but last" option in multichoice

i le Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
I totally agree with Samuel's arguments. He rightly points out that the didactic point of view should always prevail.
I would add that there is a logical flaw in both the "all/none of the above" options. After all, in a multiple choice type of question (whether single or multiple answer sub-type), the respondent is (pre-)supposed to make a choice between the options offered. Offering a further choice of all/none of the options is counter-intuitive and in my opinion defeats the whole purpose of the exercise.
Unfortunately, in highly competitive exams mainly relying on MCQs, the aim is not so much to test the respondents knowledge as to ask tricky questions in order to eliminate as many candidates as possible. Maybe the all/none options are useful then. But this is not didactics.
In reply to Joseph Rézeau

Re: "Shuffle all but last" option in multichoice

i le Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Samuel, Joseph,

The decision whether to use "all/none of the above" in a formative or sommative quiz should be left to the teacher. The recommendations are to avoid using it, not to never use it. The implementation of this feature would not be encouraging but accommodating its use, which is justified in many cases. Anyway, its use will continue to be widespread because you can still use it in an ordered MC.

Luca,

Your original post was about having an "Other reasons" option locked in last position, in a shuffled answer choice MC quiz,... further paired with a short answer field, which you declined (!), but that is very common in administrative questionnaires.

Although it can be argued that "Other reasons" is identical  to "None of the above", it shows that another formulation and, why not, another use of a choice locked in the last position in a random order MC would be a nice feature for users. In fact, a little research shows there are cases where you want to lock any choice, for example "No change" in first position in English tests and "I am guessing" in last position in Math tests, and other examples with both before last and last answer choices locked in position.

Rick,

I took the Canvas example from the Canvas Instructor Guide, on page 2765, available at https://s3.amazonaws.com/tr-learncanvas/files/pdf-guide/CanvasInstructorGuide.pdf (note that this file is 170 MB).

In reply to Dominique Bauer

Re: "Shuffle all but last" option in multichoice

i le Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Dominique, I will attach what I see when I try to create a MC question in Canvas. My screens seem to match this documentation.

https://community.canvaslms.com/docs/DOC-12884-415241477

Beware: Canvas spends its efforts selling its product, and not making product improvements or maintaining documentation.  ata

Actually, Canvas documentation is pretty bad.  That link that you gave me is far too big to display while I currently have slow Internet.  And when I ask my school's support folks Canvas questions, they usually direct me to the Canvas Community.

Attachment MC Canvas.jpg
In reply to Samuel Witzig

Re: "Shuffle all but last" option in multichoice

i le Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Samuel, you have made a convincing argument. You have me agreeing with you.

First, I agree with your assessment of "All of the above." This has never made sense to me.

"None of the above" holds some merit, but only when it is the last choice. I prefer "None of these are correct" which doesn't make it position-dependent, meaning that answers can be shuffled. But its problem is that a multiple-choice question is seeking a correct answer, and "None of the above" implies to not seek a correct answer, except it is a possible double negative.

My current textbook authors, in their question banks, seem to be getting away from "All of thee above." I haven't seen this being used for a while. However, they still use "None of the..." but seem to be changing the wording to something similar to what I suggest. When I do see these questions as "None of the above," I change them to "None of these are correct."

So I can accept the logic to not build into Moodle, bad practices.

However, I do appreciate that Thomas provides a viable solution for those who want a little more flexibility, albeit as a plugin. And it wouldn't bother me if the Moodle developers did decide to build some of these suggestions into core.
In reply to Luca Bösch

Re: "Shuffle all but last" option in multichoice

i le Hugo Ribeiro -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
HI All,

So once again I arrive late... Nonetheless would like to add something.
While NONA or similar may not be very correct from a pedagogy point of view I reckon there is a strong point for this requirement. If you happen to have negative marking there's no way to unselect an option because of radio buttons. This leads to a problem:
In a first glance student selects option A and moves on. After a while goes back to the question, and isn't fully sure and does not want to take a chance (negative marking). He's unable to respond null again.
Because of this we suggest adding an extra option like "I do not answer", that ideally should be located always in the last position.

Thanks
In reply to Hugo Ribeiro

Re: "Shuffle all but last" option in multichoice

i le Thomas Korner -
Picture of Plugin developers Picture of Testers

Hi

Agreed on that. IMHO negative marking isn't correct from a pedagogical point of view too (e.g. good described here: www.sciencedirect.com/science/article/abs/pii/S0191491X13000254). But letting this out of scope, I would recommend going towards an untackling option rather than an "I do not answer" - option. Reason: If you answered and want to go back, you would have then the visual mark "unsolved question" rather "solved question". This gives you the opportunity to come back directly to your answer later in the exam.

Best regards

Thomas

In reply to Thomas Korner

Re: "Shuffle all but last" option in multichoice

i le Hugo Ribeiro -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Hi Thomas,

Exactly. In fact, the usage that I explained is a workaround to the system as it is, and has a couple of [important] downsides: like you pointed out, this is just another plain distractor in the question. Also, this usage introduces noise in the quiz/question statistics.

So, a formal and technical way of the student go back to unsolved state would be preferable, however shuffle all but last would be an improve for those who use all/none of the above or "I do no answer" because of negative marking (and yeah, neither are the best approach pedagogical)

kind Regards,
Hugo
In reply to Hugo Ribeiro

Re: "Shuffle all but last" option in multichoice

i le Luca Bösch -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

I'd like to remind you all that since Moodle 3.7 the 'clear my choice' function is there.

So, being able to set a 'undecided' option by choosing an arbitrary one, which should always be last is no valid argument any more.

Best,

Luca


In reply to Luca Bösch

Re: "Shuffle all but last" option in multichoice

i le Hugo Ribeiro -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Thank you Luca! That's what happens when we stay on LTS version...

You are right, no valid argument.
Then I guess it depends if Moodle should promote best approaches, or simply cover teacher's needs. In our case, we advise against the usage of such distractors whenever we can, but we are not always successful...

Cheers,
Hugo