Negative grading practically

Negative grading practically

by Miro Babinsky -
Number of replies: 16

Thank you for introducing negative grading from version 2.1. I want to use quiz for multiple choice - multiple answer questions without any limits of number of correct options (answers). I mean for example with 4 options I can have 1,2,3,4 but also none correct option.  It is now possible to set the marking for that with one exception: NONE. If no option is checked then it is not possible to submit (student will get: "Please select at least one answer." , which is probably good for all other instances. I am just suggesting to add an option in Editing a Multiple choice question which may sound like: "allow no answer". Then instead of "Please select at least one answer." something like "Are you sure you want to submit with no checked option?" may apear (just to be shure that question was not skipped). 

Some people may think this is making sense only if in “One or multiple answer” is set to “Multiple answers allowed”, but it may be good also in Single answer mode for possible Confidence-Based,  Certainty-Based Marking (I am just guessing).

Thanks.

 

Average of ratings: -
In reply to Miro Babinsky

Re: Negative grading practically

by Avrila Klaus -

I've taken lots of multiple choice tests where "none of the above" was an option.  I think that would help with your problem because it would give the students something to select to keep Moodle happy.

Average of ratings: -
In reply to Avrila Klaus

Re: Negative grading practically

by Itamar Tzadok -

Statistically, 'none of the above' and its conterpart 'all of the above' prove to be "spoilers" in the sense that in most of the questions that contain one of these options, that option is the correct answer. Consequently these options are considered bad multiple-choice practice. Moreover, with these options you cannot shuffle the options (unless the question type allows for "sticky options").

The "allow no answer" suggestion may pose a problem for multiple attempts with instant scoring, since users could simply submit the first attempt before attemting any question, and "allow no answer" questions would be marked correct. But then again that's just an optional setting and if available may be used with caution.

smile

Average of ratings: -
In reply to Itamar Tzadok

Re: Negative grading practically

by Avrila Klaus -

"Statistically, 'none of the above' and its conterpart 'all of the above' prove to be "spoilers" in the sense that in most of the questions that contain one of these options, that option is the correct answer. Consequently these options are considered bad multiple-choice practice."

That's true when they are applied only or primarily as needed.  However, if they are applied universally, their spoiler effect is negated because the right answer isn't always, or probably even usually, going to be "none of the above" (or "all of the above").  If that option is always there, the students won't be able to tell, just from it being there, that it's the right answer.  (On the other hand, to mess with their heads, a test could be written so that it is always the right answer...kind of like when way too many answers in a row are B...bonus points for head-messing-with if it's a statistics test.)

As for shuffling, writing the answer as "none of these" takes out the reference to "above" which would legitimately be confusing if some of the answers were below.

Of course there probably are ways of doing what you want more invisibly, relying on technology instead of question design.  My preference is to use question design because changes to the technology have the possibility of breaking in a future upgrade.

Average of ratings: -
In reply to Avrila Klaus

Re: Negative grading practically

by Itamar Tzadok -

"none of these" rings a liar paradox bell. Should it be "none of the others", or "only me", or "pick me"?

I haven't yet seen a multiple question with 'all/none of the above/these' which was such that none of these options couldn't be dismissed as a lazy shortcut. But that's me and perhaps only me ...

Cheers smile

Average of ratings: -
In reply to Itamar Tzadok

Re: Negative grading practically

by Nick Anthis -
As I was reading through this thread, something crossed my mind. If the answer can be none of the choices (and not using none of the above as a choice), then how can you differentiate between a student skipping the question or selecting nothing at all? I guess it could be by the "submit" button, but it's just ripe for error.
Average of ratings: -
In reply to Nick Anthis

Re: Negative grading practically

by Miro Babinsky -

I was suggesting just to allow scoring/marking the aswer as correct also when none of options is correct (I am not suggesting to add last option none of all above). Sure that students have to be instructed that this could also be considered as the right choice and their decision should be confirmend not to be confused with empty submission.

Average of ratings: -
In reply to Miro Babinsky

Re: Negative grading practically

by Itamar Tzadok -

Given a question with none of the options correct, is there a significant difference between this question and the question negated such that all the options become correct? smile

Average of ratings: -
In reply to Itamar Tzadok

Re: Negative grading practically

by Jean-Michel Védrine -

I have no references on hand (I am on summer holidays in the country without all my books on MCQ wink) but I seem to remember studies show the percentage of correct answers is not the same in both cases.

Average of ratings: -
In reply to Jean-Michel Védrine

Re: Negative grading practically

by Miro Babinsky -

Computation is easy: with m options there are 2m combinations if all combinations are allowed (from zero to m correct options). None correct means just one option down: 2m - 1. In one university here in SLovakia there using such kind of tests as an entry exam. It is possible to do everything in Moodle. Without modification I am suggesting this is not possible to trasfer such testing in MOodle. 

Average of ratings: -
In reply to Miro Babinsky

Re: Negative grading practically

by Itamar Tzadok -

I'm not sure there is even this 1 difference in the computation. In both cases there are 2m permutations, and one of them, the empty (or unchecked) set, is already presented, so you can either submit the presented or try one of the other 2m-1.

There is a clear practical difference on the user side though. In the all-correct the user has to actively tick all the options for getting it right. There may also be a psychological difference but I don't know what it would amount to. I always prefer to require more than less work from the user (offering the user an opportunity to get creative by searching for shortcuts!). So if there is no psychological effect which cannot be adjusted by some training I would dismiss the none-correct type as an unnecessary complication. smile

Average of ratings: -
In reply to Itamar Tzadok

Re: Negative grading practically

by Miro Babinsky -

Hi Itamar,
You are saying the same as me regarding number of options (maybe you are right that these are permutations not combinations): the difference is one, there are stil planty option wihtout this one, but that does not mean that I have to reject just this one option. All what I am proposing is a simple change the Editing form which will allow me just this - to submit with no option correct.

I read about None at Kehoe, Jerard (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9). Retrieved July 10, 2011 from http://PAREonline.net/getvn.asp?v=4&n=9  . Adopted form Testing Memo 4: Constructing Multiple-Choice Tests -- Part I, Office of Measurement and Research Services, Virginia Polytechnic Institute and State University, Blacksburg, 
Take a look into part Writing option Poiint no 3, which explaining that it has some psychological effect.  You are right that it seems now as an "unnecessary complication", but to have totally correct (not possible to guess just by chance) options, than NONE should be only in 1/2m cases/questions/items. For m = 4 it means that you should use none in one of 16 questions. For m=5 in one of 32 etc. 

Average of ratings: -
In reply to Nick Anthis

Re: Negative grading practically

by Jean-Michel Védrine -

This is a real problem with MCQ when mulpiples answers are allowed.

And this is something I have some trouble explaining to me french colleagues teachers new to MCQ authoring.

In fact if you wrote an MCQ question where all, some or none of the answers can be true, it's  the same this as if each answer has 2 radio buttons "True" and "False" but with a very big difference : you can't select none of the 2 buttons.

So it's not possible to skip the question.

This is not a concern if no negative marking is involved, but if the question can result in a negative mark this is a big problem.

The only possible solution that I know is to provide a "I don't know" answer associated with a 0 mark. But the student wanting to pass the question must not forget to check this one ! Experience prove that this is unrealistic !

This is why I tell them it's better not to use negative marking when multiples answers are allowed. But most of the time they don't listen to me .

Average of ratings: -
In reply to Jean-Michel Védrine

Re: Negative grading practically

by Miro Babinsky -

These are two different things:

MCQ with multiple option is sensless (at least in my field) without negative grading. If student checks all options he will get more points/marks even with penalty (if applied) then as a student who will try to check only some options. You are right: what I am proposing is exactly the same as to have "2 radio buttons" at each option even with the option to choose none of them. Not checking means False, checking mens True.

Second thing is the button "I do not know". If there would be actions which i described above there is no need for this. But to implement a concept of Confidence-Based,  Certainty-Based Marking (look at http://www.ucl.ac.uk/lapt/laptlite/) or simplier formula scoring: (R. Frary http://www.ncme.org/pubs/items/ITEMS_Mod_4.pdf) it would be usefull to have such a button. 

Average of ratings: -
In reply to Miro Babinsky

Re: Negative grading practically

by Jean-Michel Védrine -

Maybe I should have explained more the adventure with my colleague teaching french juridic system (very complex matter !).

She made multichoice questions with several answers allowed but with an all or nothing marking AND negative marking (the system used was not Moodle) !!!

So with n choices there were only one answer with a positive mark and (2^n -1) with a negative mark and no possibility to say "I don't know" !!dead

Then she came to me and says "Why do I have so many 0 ?" big grin

Most non scientifics teachers here don't have any notion of probability theory they want negative marking to prevent guessing and not knowing how to ajust positives and negatives marking they realise only after their exam is done that it was either too easy or too difficult.

Average of ratings: -
In reply to Jean-Michel Védrine

Re: Negative grading practically

by Miro Babinsky -

I am new here (not an English speaker or an English teacher) but you inspire me to response without much checking: Just yesterday we had a visit of some French teacher lady from Grenoble. I can understand your todays worries...

Back to Moodle: there si a documentation of calculation of grades/points on http://docs.moodle.org/dev/Quiz_statistics_calculations It is not mentioned there that:

  1. For MC-MR (multiple response is better then multiple answer) The positive grades have to add up to 100%
  2. No1. is not valid for negative grades
  3. In vesion 2.1 and lower the total grade for one item is always positive. In version 2.2 it is working properly (as I would like to)
  4. It is not clear with negative grading (now I mean when Default Question Grade in version 2.1 or Default mark in version 2.2 is set negative) which options are correct. Now the options which summ up to +100% are considered correct.. which gives total negative sum.
My last comment: Now it is possible do set Default Question Grade in version 2.1 or Default mark in version 2.2 to decimal but not for me. We are using different decimal seprator "," which was interpreted not correctly. I tried also "." Maybe because of the language/or other setting it was always rounded or maybe it is just a bug (I tried to export/ change DQG and then import questionbank without success)
Average of ratings: -
In reply to Avrila Klaus

Re: Negative grading practically

by Jean-Michel Védrine -

Hello Avrila,

A few years ago, a lot of exams here in france were using your "none of these" option (in french "Aucune des autres réponses n'est correcte") even for MCQ with only 1 answer allowed.

I think the goal was to avoid the student thinking that the right answer is always one of the propositions and they only have to "choose".

I observe that the trend in the last 2 years is to no more include this option for MCQ with 1 answer allowed and thre are less and less MCQ with several answers allowed in exams.

A strong argument for me against MCQ with several answers allowed is that adaptation of IRT (Item Response Theory) to this case is rather difficult. But very few french teachers care about IRT or such things ...

Average of ratings: -