Mdl 2.3.4 Certainty Based Marking, Too harsh?

Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Douglas Broad -
Number of replies: 10

I just tested CBM in a quiz.  The table of results seemed a bit harsh to me. Given a question weight of 1, the actual grades are shown below.

Answer Not Very Fairly Certain Very Certain
Correct 0.33 0.67 1.0
Incorrect 0 -0.67 -2

Is the negative -2 weight an error or by design?  If by design, why so harsh compared with the rest?  Seems unbalanced.  I would have thought a -1.0 or -1.33 would have been more balanced.  A quiz of one question gotten wrong but confidently right would yield a grade of -200 percent.  How would that average with other quizzes?

Would anyone else like the ability to control the minimum overall grade for a quiz under these situations?

Average of ratings: -
In reply to Douglas Broad

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Douglas,

I agreee and fail to see the logic of this -2 grade.

However I must say that in my whole career as a teacher (44 years) I have never ever had the opportunity to give negative marks and the concept of negative marking escapes me. So I am probably not the best judge in this matter.

Now, regarding the concept of CBM, I consider it interesting but it can easily be found at fault/illogical. Let's take a very simple example.

Q.- A butterfly is an insect. (Correct answer is "Yes").

case#1 Student answer: Yes. Not very certain. Grade: 0.33.

case#2 Student answer: No. Not very certain. Grade: 0.

One might argue that if the student is "not very certain" of their answer, whether that answer is Yes or No, i.e. correct or incorrect, they should get the same grade. In other words, if the CBM system penalizes those correct answers which are qualified by the respondent of "not very certain", then it should - symetrically - give a bonus to those incorrect answers which are qualified by the respondent of "not very certain". Thus, in case#2, the student should get a grade of 0.33 for admitting that they were not quite certain that their (incorrect) answer was the correct one.

Joseph

PS.- Hey, Joseph, How certain are you that your reasoning is correct?

s

Average of ratings: Useful (1)
In reply to Douglas Broad

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

It is by design. When devising the CBM marking scheme, you need to make sure that the numbers give the students the right incentives to be honest when self-evaluating their certainty. (I guess I am using the word incentive in the way an economist / game theorist would.)

The graph in this middle of this page "When should I use the different certainty levels?" explains it: http://www.ucl.ac.uk/lapt/. It is important that the three lines for 'Expected score, given how sure you are that you are correct' overlap like that, so there is a range of scores where each one is highest.

Remember, the goal here is to help students learn to better self-evaluate the reliability of thier knowledge.

Of course, the specific numerical values could be changed. You could use scores

Answer Not Very Fairly Certain Very Certain
Correct 2.33 2.67 3
Incorrect 2 1.33 0

if you prefer (just add 2 to everything), but I think it is less clear. This is an area where there is a difference between the abstract maths of the incentives, and the psychology of now humans react to numbers, give different answers. Really, you are asking a good question. The kind of question that a teacher should ask about how the pedagogic choices they make will affect students.

Average of ratings: Useful (3)
In reply to Tim Hunt

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

@Tim,

Thanks for the link to http://www.ucl.ac.uk/lapt/; now I have food for thought.wink

Joseph

Average of ratings: -
In reply to Tim Hunt

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Douglas Broad -

Tim,

First, thanks for the link.  Its nice to get the official explanation for the scheme.  As I was reading it however, even when I was looking at the chart explaining the rationale, I couldn't see the justification for the -2 (my number) penalty for confident.  The gain of 0.33 over fairly did not seem to balance the loss of 1.33 for the wrong when confident.

Second, I don't see how the grade adjustment you propose (adding 2) could be accomplished in Moodle.  I would probably want to add between 0.5 and 1.5 but I can't see how in Moodle.

Third, this ability seems to require the whole quiz to be graded this way.  I know this complicates matters extremely but I wonder whether it is more appropriate to be applied on a per question basis.

Fourth, if we don't adjust (add 2), what would we do if the entire quiz had a negative score?  Would we throw it out or average it together with other quizes.  Somehow a course grade of below 0 wouldn't make sense to me.

I still need to experiment to see how this works with cloze questions and some of the other unusual question types before I will feel comfortable using it.  I would really like to use it but may limit it to a small percentage of quizzes initially.

Average of ratings: -
In reply to Douglas Broad

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Sorry I was not clear. The "add 2" thing could only be done by editing the code.

The important thing to remember is that you should only be using "Very Certain" if you are. In practice, you should be seeing very few -2s (but you should see them occasionally) otherwise you are not judging your certainty correctly.

I don't think you should use CBM with

  • Cloze questions, or
  • constructed response questions (e.g. shortanswer), or
  • questions with partial credit.

Reserve it for multiple choice / true false. There, it is a great way to get students to think more, and acknowledge where they are guessing.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Douglas Broad -

Thanks Tim. That clarifies most of it.  However, since the CBM method can be applied only at the quiz level, that would limit all the questions on that quiz to one of the suggested formats.

Would be nice to know how many people are using it and how well it has worked.  I will look on the link you provided to learn more.

Average of ratings: -
In reply to Douglas Broad

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

The other point to make is that we really need MDL-32188, and I still have not finished it.

Average of ratings: -
In reply to Douglas Broad

Re: Mdl 2.3.4 Certainty Based Marking, Too harsh?

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Points re CBM. Sorry I'm slow to respond, but I don't often check this forum.

(1) CBM is OK when the Q asks for one response that will be marked right or wrong according to clear criteria. Numerical or short answers are OK if numerical and spelling tolerances are defined. E.g. What is pi (to within +/-1%)? or What is the Italian spelling of the capital of Italy?

(2) Partial credit isn't really a problem if you like it. E.g. you might give 50% credit for answers pi=3.1 or "Rome" to the Qs above and explain why they don't get full credit. At C=3 these answers would then get 1.5 marks, while answers 2.718 or Milano would get -6. NB the core Moodle CBM may not mark partial credit correctly.

(3) I don't like the idea at all of making all CBM grades positive by adding twice the max penalty! The normal CBM grades (setting max=3, to avoid fractions) are 1,2,3 for correct and 0,-2,-6 for incorrect. If you make them all +ve by adding 6 you get: 7,8,9 for correct and 6,4,0 for incorrect. Not knowing an answer at all (e.g. omitting an answer or guessing at C=1) would give you 6 or 7 marks. All rather bizarre and counterintuitive!

(4) What's wrong with negative grades? The natural 'zero' for knowledge is saying you don't know, or that you're just  guessing. "I'm sure Milano is the capital of Italy"  is worse than ignorance. In medicine or engineering, confident misconceptions can kill people. In a summative test they deserve to be treated worse than acknowledged ignorance or uncertainty, and in a self-test they should be a wake up call to pay attention and get things straight. Students don't mind negative marking if they have the option to avoid or minimise it by acknowledging they are unsure. That's what CBM is all about: rewarding the acknowledgement of uncertainty.

(5) You can include things like  cloze Qs in a Moodle CBM quiz, but they won't be presented using CBM, and to avoid confusion it's probably best not to.  In principle one could ask for certainty about each component of the Q (as is done in LAPT for multiple response Qs) but it's probably best to split the Qs up if you want to use CBM with this kind of material.

(6) How many people use CBM? Would be interesting to know in Moodle. As Tim says, it isn't really fully/properly implemented yet. In LAPT (see www.ucl.ac.uk/lapt ), there have been on average about 300 student self-test sessions per day throughout the year for many years (mostly students from London).  

Tony Gardner-Medwin

Average of ratings: -