Confidence based marking / Certainty based marking (CBM)

Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -
Number of replies: 22

Is anyone developing or using CBM with Moodle, or have tried to?  Is anybody currently interested in it?

Since 1994 there have been posts about Confidence based marking / Certainty based marking (CBM) of True / False and multiple choice answers.  This scheme seems to prevent wild guessing paying off, promotes students' reflection upon their knowledge, and has many benefits if done properly.  There are good reading papers and presentations about it at http://www.ucl.ac.uk/%7Eucgbarg/pubteach.htm

My group would be interested in using it.  The Moodle forum posts seem to have dried up about a year ago.

e.g.

http://moodle.org/mod/forum/discuss.php?d=6443

http://moodle.org/mod/forum/discuss.php?d=21532

http://moodle.org/mod/forum/discuss.php?d=12161

The LAPT people (http://www.ucl.ac.uk/lapt/) have years of experience in implementing CBM both formatively and summatively, and in studying its effects. 

They aparently have a way of linking Moodle to their own web based CBM system, with certain restrictions.  They seem keen to help implement CBM in Moodle directly, rather than passing messages between Moodle and LAPT's system as can be done at present.

I expect that the more demonstrable interest there is from the Moodle community, the more resources the LAPT team can pour into helping us. 

Is anyone out there interested in it?

Cheers,

Sacha

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
I found that very interesting. I'm not generally too interested in the points on tests. But this approach teaches an important attitude towards the scale from Certain knowledge to delusion.

I checked out the latest Moodle forum thread you gave a link to. I didn't notice any one mention the option that should be possible in Moodle even now:

A True-False question could be a setup as a radio button MC question with 6 options:
True-Certainty c=3 mark 3 if correct {Justified CERTAINTY}
True-lower Certainty c=2 mark 2 if correct
True- low Certainty c=1 mark 1 if correct
False-low Certainty c=1 mark 0 if correct
False-higher Certainty c=2 mark -2 if correct
False-high Certainty c=3 mark -6 if correct {DELUSION}
(no answer would be 0 also)
where I followed the the point scheme from http://www.ucl.ac.uk/~ucgbarg/tea/lhl_argm.pdf

For questions with more alternatives (that TF) that can result in a very extensive answer array.

Possibly JRs regular expression might be able to be used for more "open" questions where one would append a certainty to the answer, and that that part of the string could connect to the point.

It would of course be interesting to have a built in "Confidence" option for all question types.

AND All questions should have a "challange" option, to use Dennis Daniels term.
Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -

Hi!

You're quite right - that would be a good way of doing it, without any modifications.  It never struck me!

However, you're also right that there could be formatting problems with larger question types.  'My' authors seem to be keen on MCQs, so we'd be looking at these large question arrays.

Like you say, it would be good to have these for all question types!

You said,

"Possibly JRs regular expression might be able to be used for more "open" questions where one would append a certainty to the answer, and that that part of the string could connect to the point.
[snip]
AND All questions should have a "challange" option, to use Dennis Daniels term."

I wonder if you could expand on these points a little?  I'm not familiar with either (I'm very new to Moodle), and I'm not quite sure what kind of questions you're referring to... numeric response?  Short text answers? - things that can be automatically marked?

Regards,

Sacha

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
Last night I remembered something that would make my suggestion not work, without modification. Moodle does allow negative points on MC questions to allow the possibility to prevent people just filling everything. But, though I haven't checked that, I believe that the total mark for a question may not be below 0, so negative values are saved as zero. Has anyone checked if negative values can be saved for any question types? It seems like that would be a rather simple change that could be made to make CBM questions easier to make.

What one could do is
  1. Set a base level for the CBM question as 6 pts
  2. include a "I don't know" option with 6 pts
  3. set the other options I mentioned above on a range from 9 down to 0
Then a person having grades on a question with <6 thinks he knows something but is actually wrong.

I'll get back on challenge and reg exp later.


Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -

Challenging a question


I feel like I would like to easily get feedback from the people on Internet that read my pages and even more so from pupils that are doing a quiz question I've written.

I've written a general feedback page and Java script that I use to let people easily give me feedback (including automatically: URL with query, and various info about browser version, screen size etc). I haven't found a way to get that into Moodle. (I can't get it at all into pages created with editor inside Moodle (which strips out most JavaScript). And pages I make outside and put into Moodle give me URLs that lose a lot of info because they are based on PHP sessions).

Most of my stuff is in Swedish but I've made an English version here.

Why challenge a question?

If I were to give you the True-False question:

I have stopped beating up my mate. True False

You ought get the feel of wanting to "challenge" a question!

The question can be wrong or unclear. The judgement of answers can often be based on simplified views that will fit most students but some that know more details about a field (maybe even more than the teacher) should be given an opportunity to give THEIR answer and motivation for it.

Dennis Daniels has described a possible path for using a feature like this in Moodle to have the pupils do the hard part of making the tests. The teacher makes some relatively quick questions that the fastest students do first AND CHALLENGE. The teacher can than easily refine the questions and the "expected answers" and their feedback by using these challenges.

He has a lot of video connected to Moodle, like this one
http://video.google=dennis+daniels+moodle+question
In this one he mentions the "challenging" concept in the middle.

Regular Expression Questions

Early this year I had "Christmas" twice the same week! I discovered MOODLE which would import SCORM packages quite well, which was what I was looking for, and it had question types with feedback for different expected answers. The same week Joseph Rézeau announced his new "regular expression question module". It was exactly what I had been hoping I would make or find.

Regular Expressions are a powerful pattern recognition tool that many programming languages use. He connected it to the Moodle Short answer question type. It's not the easiest thing in the world to get into but it can be made to do amazing things if you create a series of steps like:
  • does the answer contain this expected mistake (including misspellings)
  • is this neccesary part missing
  • are they in the right order
  • Is it basically completely right with the exception of small details like punctuation, CAPSsmalls
Each step/condition/recognized pattern is connected to guiding feedback. (and differentiated grade if that's your interest. JR and I are most interested in questions as teaching tools.)

I think that's enough to let you imagine the learning process that can be built into one question!

The reg exp module is not included in the standard Moodle package (unfortunately) but Moodle modulated construction makes it relatively easy to install.

To make a CBM question one might be able to add "Write the certainty after your answer with C1=very UNcertain; C2=Pretty certain or C3=very certain, like this:
myanswer C2

Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -

Jeff, Hi!

Regular Expressions

Aha - thanks for the extra info... that makes much more sense.  Yes, I see that the regular expression stuff would be very powerful indeed, and could really be used to do CBM using short answers.  That would mean framing all questions as short answers, but that's not so hard is it?  At least for pick the best answer type MCQs - it should work great!

It sounds like you have some experience with this!  I'm not a programmer, so I think I might find it hard to design the appropriate 'regular expressions' (is that the right phrase?) to process the answers.  Do you have any first ideas about what code would work for answers that were a number, then a space then C1 C2 or C3 (upper or lower case), then check for the correct answer and then apply the appropriate score? 

I suppose it would be hard to get it to ask for the answer to be repeated if the confidence rating was missing, but easy to give the answer a score of zero?

For pick all that apply type MCQs it would be a lot more effort to answer - and potentially error prone - by typing in confidence levels.  It would only be a bit better to point and click - you'd be less likely to miss giving a confidence rating... the way I've been thinking about it lately is that a "pick all that apply" type question is really lots of T/F type questions stuck together.  If you pick items b and c as applying, of course you should apply a certainty rating.  However, that also means you've decided that items a, d, and e don't apply - and really you should apply a certainty rating to those too. 

So, even with point and click, a "pick all that apply" question would end up with a largish grid of possible responses, all of which have to be considered properly.  That's the point of CBM of course.

With all this extra effort, source of potential mistakes in responses, and screen real-estate taken up - are "pick all that apply" good questions for CBM?

Challenging Questions

That's a very neat way of developing questions.  Test them on live students!  Ebarassingly the soundcard on my PC's not working - so I can't listen to the video, much as I'd like to.

It sounds rather frustrating you can't get your javascript feedback working with google.  I suppose if you make extensive use of the regular expression patch, students could use an escape string in the short answer, after which the 'answer' would be split off and considered to be a comment... 

e.g.  Your question:

What's 2+2?  

(If you have a comment on this question, type "/comment" then your comment in the same line as your answer)

Student's answer:

3  /comment This is too easy!

it's not as good as the javascript, and I suppose  you could as easily embed a comment email address in the question... 

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Hi Jeff and Sacha,
Here is my modest contribution to this thread. Using my regular expressions question type in Moodle 1.6, one can do the following.

1- Challenging/commenting questions

Animals
Question text: Name an animal in 3 letters, with vowel a in the middle.
You can add an optional comment (or a question) after your answer, using the ! separator.
Sample answer:
cat !That was too easy.
Answer 1: the [bcr]at
Feedback: Correct Grade: 100%
Answer 2: the [bcr]at\s?!.*
Feedback: Correct. Thanks for your comment. Grade: 100%
Answer 3: --the.*
Feedback: Missing definite article! Grade: 0%
Answer 4: .*!.*
Feedback: Your answer is probably wrong, but thanks for your comment. Grade: 0%

Sample student answers:

Student answer #1: the rat !that was dead easy...
Feedback: Correct. Thanks for your comment.
Student answer #2: dunno
Feedback: Missing definite article.

2- Confidence indicator

Animals (extinct)
Question text: Which (one) of these animals is extinct? the cat, the mammoth, the tiger, the elephant. Please add an indicator of your confidence after your answer. From C1 (not sure); C2 (maybe); C3 (absolutely certain).
Sample answer: the cat C1

Answer 1: the mammoth\s*[cC]3
Feedback: Correct. Indeed, the mammoth is quite extinct. Grade: 100%
Answer 2: --.*[cC][123]
Feedback: You forgot the Confidence indicator. Grade: 0%
Answer 3: the mammoth\s*[cC]1
Feedback: Why do you doubt that the mammoth is extinct? Grade: 33.33%
Answer 4: the mammoth\s*[cC]2
Feedback: Why are you not quite sure that the mammoth is extinct? Grade: 66.66%
Answer 5: .*\b(cat|tiger|elephant)\b.*
Feedback: That's not an extinct animal. Grade: 0%

Actually we should be able to give negative scores for wrong student answers associated with a high confidence indicator. e.g.

.*\b(cat|tiger|elephant)\b\s*[cC]3 -> Grade -66%

.*\b(cat|tiger|elephant)\b\s*[cC]2 -> Grade -33%

.*\b(cat|tiger|elephant)\b\s*[cC]1 -> Grade 0%

Unfortunately questions cannot have a negative score in Moodle at present, so the negative scale of scores is not available in SHORTANSWER not in REGEXP question types.

Use at own risk.black eye The attached screen shot shows how, with careful use of regexp expressions to analyse the student's answers and provide meaningful (automatic) feeedback, plus manual, human comment by teacher (plus student's challenge/comment facility), a kind of pedagogic "pseudo-dialogue" can be established.

Joseph

Attachment image-0000.jpg
Average of ratings: Useful (1)
In reply to Joseph Rézeau

Re: Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -

Hi Jef and Joseph!

Thank you both (and Joseph for giving me a such a head-start on the regexp coding!) - this is all looking *very* promising.

I hope to have a working Moodle server next week, so I can start experimenting.  I may well have lots of questions for you!

Best regards,

Sacha

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
It would actually be easy in reg exp questions (if you are allowing feedback during the quiz) to get the question to say "You MUST include a confidence value C1 .. C3 at the end of your message."

NOTE If you want to use "virtual negative points" on a question in MOODLE you must set a baseline like 6 = "zero" = dunno. (If it is true that Moodle doesn't save a negative total for a question).

If you were to give us a sample question or two with expected answers and wished points, maybe someone (probably JR or me) will try and see how that might be done. (Are you out after numerical answers? What do you teach?)

Here is a Moodle forum thread on the challenge idea:
http://moodle.org/mod/forum/discuss.php?d=46084

Here is a page that JR has about reg expressions:
http://perso.orange.fr/joseph.rezeau/eao/developpement/expandRegexpToString.htm

You can try out the Reg exp question type itself and some other stuff) here:
http://rezeau.info/moodle/course/view.php?id=2
I think "moodler moodler" works as a login.

I see that JR was writing at the same time I was. (I was wondering if he was ill, when he didn't "surface" when I mentioned his Reg Exp! smile )
Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Jeff > I was wondering if he was ill, when he didn't "surface" when I mentioned his Reg Exp!

Ha ha! dead sleepy black eye

Average of ratings: -
In reply to Joseph Rézeau

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
I decided to finally install reg exp on my new 1.7.

When I try to follow the instructions I don't find this place

.shortanswer .feedback,

where I'm supposed to do this
.regexp .feedback, //ADD THIS LINE

Is it unnecessary in 1.7 or ???

is this:
4- There is another problem because the <title> tag for the new regexp question type is created through \lib\questionlib.php:
function print_question_icon($question, $editlink=true, $return = false) {
// returns a question icon
global $QTYPES, $CFG;
$namestr = get_string($question->qtype, 'quiz');
One has to add to \lang\...\quiz.php this line:
$string['regexp'] = 'Regular expression short answer'; //JR
just for the page title? In thjat Case I don't think I'll bother with that.
Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Yes, the CSS changed in 1.7 in a way that made that bit of CSS unnecessary.
Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
I was thinking of toying some more with using the Reg Exp to address "question challenge" and CBM but would need a lot of repeated code that would have to be entered in the interface so I was thinking maybe it would be better to prepare outside MOodle and Import it.

I thought I would ask if it was possible, but decided to try first. I have M 1.7 but none of the export formats worked for RegExp, so I guess that situation is still a pain. Do you (JR) have any hacks that make import/export of RegExp possible?

If one does a backup of a course will RegExp questions (and CLOZE) survive a restore?

I remember Tim H asking about the actual definition of The GIFT format. Did he get any answer? When I search with Google I just wind up back in Moodle.

I'm also wondering about the SCORM and IMS interactive question definitions. It would be good to know if they have already "invented the wheel" or maybe need broadened horizons. Anybody know anything about that?
Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Jeff,

I have not yet checked my Regexp question type against current Moodle 1.7 stable but will do so shortly. If you have only dumped the regexp folder inside your moodle 1.7 \question\type folder and not changed the \question\format\xml\format.php file, it is "normal" that you cannot export/import the regexp questions you created.

As far as I can see, all you have to do for the import/export of regexp to work is to uncomment all the lines concerning this question type in file \question\format\xml\format.php.

Joseph

PS.- Use at own risk and not on a production server but on a test server...

Average of ratings: Useful (1)
In reply to Joseph Rézeau

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
I tried taking away all the \\ in reg exp sections and exported and re imported (as Moodle-XML, I suppose that's the only supported format) and it seemed to work fine.

I'm enclosing the php file I used (the part of the filename before "format.php" should be removed before using. (I'm on 
moodle 1.7 (2006101007))

Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -

That marking scheme seems like the best solution possible, without modification.  I'm in touch with the CBM people at UCL, and will ask them what they think. 

From what I've read, the scoring is a critical thing to get right.

Thanks for pointing this out by the way!

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Jeff Forssell -
I've just exported, as Moodle XML, 3 questions where 2 and 3 are Reg Expression quesions.
2. Is basically JRs CBM suggestion (without neg points, which misses the target)
3. Is my version of it with the points changed so 6pt is a virtual 0, 9 = +3 and 0 is a virtual -6
There's also some sloppy explanation about what it's about. (I also noticed now that I had planned to make "the" unnecessary in the answer, but didn't follow through sad )

Question no 1 was an unsucessfull attempt to give hints with a javascript collapse/expand that I often use outside Moodle, but it gets stripped out in the HTML editor in Moodle.

JRs challange Reg Exp example could be built upon to even give HINTS actually. (But the amount of repeated code is a bit daunting). Other possibilities I'm thinking of is OverLib popups that seem to work in Moodle. I'm wondering if the YUI javascript library (which in now included in the Moodle standard library) could be used, but I can't find any examples of how it can be used inside moodle.
Average of ratings: -
In reply to Jeff Forssell

Re: Confidence based marking / Certainty based marking (CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

For examples of use of Certainty-Based Marking, see the UCL site at www.ucl.ac.uk/lapt for a full range.  If you want Qs where everyone will (I imagine) have varying degrees of certainty, try www.ucl.ac.uk/lapt?ukgov , or for mathematical examples www.ucl.ac.uk/lapt?medmath . There's a big variety of Q types at www.ucl.ac.uk/lapt?bmat1 .

We can straightforwardly link exercises at present from within a Moodle course, passing out the username to LAPT, so that the student-specific records are then stored outside Moodle.  You could try this with http://moodle.ucl.ac.uk using the Guest login (if you can get the UCL Guest login to work - it doesn't always!). Go to 'About Moodle' and 'Comparing WebCT and Moodle' and 7: Interoperability, LAPTlite options. We haven't yet mastered how to return grades to within Moodle so that they are stored alongside a student's internal Moodle data.

A major question is whether it is really worth trying to embed CBM within Moodle's own Q types and structures. There would be various pros and cons of this: direct functionality vs the work and maintenance needed to provide a high level of usability. How would this compare with an 'open architecture' solution, jumping back and forth between independent software structures?  One factor is that LAPT is designed for really important fast feedback to students, primarily as a learning or formative self-assessment tool. It achieves this with client-based javascript, which means it is intrinsically relatively insecure. We don't use it for high-stakes tests unless properly invigilated (we mainly use OMR cards) - but then any online high-stakes test does really need invigilation.

Any views?   Tony Gardner-Medwin, UCL

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -

Linking to LAPT vs. Native Moodle CBM

By the way - LAPT also has a "question challenge" feature - a "View / Make Comments" button. 

I can vouch that LAPT is lightening fast.  LAPT is really good!

However, I think if CBM can be built into Moodle somehow it will instantly get a wider audience than it might otherwise.  People will download moodle and it will already do CBM.  The first step in this is making some kind of hack or module to existing releases, to demonstrate it's useful and good, and can be done in Moodle.

I think people also may feel a little cautious about the overhead of having to learn two systems - moodle and lapt, rather than just one, and also having to learn about how to interface between them.  

My vote would be to aim for developing native moodle CBM in one of the 1.6.x releases, because they have working conditional access / activity locking hacks, which I rather want as well.  At the moment it doesn't look promising for activity locking until moodle 2.0.  Anyway...

Aesthetically - there appears at first sight to be a jump from Moodle to LAPT.  Moodle uses CSS, so in theory its questions should integrate well with the site's visual design.  Does LAPT?  How would one go about matching LAPT quizes' presentation to that used in their moodle site?  In once sense it's a trivial matter to ignore - but others it's not - some people use Moodle to create paid for courses, and I suppose need to brand everything as their own.  Also, there may be accessibility issues? - for example Oxford allows students to select high contrast and large text presentations of it's moodle materials. 

(you can see this at http://openmoodle.conted.ox.ac.uk/ you have to register - which is free- then go to the Getting Research Published: example unit )

Negative marking

I am going through the forum threads on negative marking in Moodle (because CBM relies on negative marking to motivate students to rate their certainty appropriately).  I've read that Moodle will not allow a negative score on a test, and one of the reasons for this is because teachers "rely" on it not being able to. (will see you on the other side of the threads - there are quite a few - I may be some time!)

I would have thought that it makes more sense for students to end up with a final test score of less than zero if they have been receiving feedback after each question (which are scored and fedback before going to the next question - like a Moodle 'lesson') that they have scored -6 points on that question. 

It seems this is what LAPT does - it gives a raw 'mark' which can be negative, but it also gives a percentage correct, and other indices.

For example these are the score information that I got from doing a demo course in LAPT:

-------------------

Score Summary:

    20 responses: 25% correct, Total CBM marks: -39

Breakdown:
    5 responses at C=1 :   20% correct (=1 answers)     Target: < 67%
    6 responses at C=2 :   33% correct (=2 answers)     Target: 67%-80%
    9 responses at C=3 :   22% correct (=2 answers)     Target: 80%-100%

and

TOTALS: Q's done=20, Marks=-39 (25% correct, %A=0%, CBS=0% : see explanation ) [T=2.1 min]  

------------------

The LAPT marking system is very sophisticated...  it introduces concepts like

%A -  "% Above chance"... (see the "see explanation" link), and

CBS - "Certainty Based Score", which has a range from 0% to 100%.

Moodle's quiz score is analagous to the CBS I think.

It looks like if we're going to put CBM into Moodle natively, we may have to be able to cite several different 'scores' or 'marks', as well as enable negative marking?

I wonder how neccesary Prof. Gardner-Medwin has found these various scores?

Sacha

 

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

For explanation of the UCL CBM scores and their rationale, see http://www.ucl.ac.uk/lapt/laptlite/sys/lpscoring.htm . [The basics are explained on the homepage:  http://www/ucl.ac.uk/lapt . ]

To look at an anonymised demo sample report of student scores (derived from practice for the biomedical admissions test BMAT, run by Cambridge Assessment), see http://www.ucl.ac.uk/lapt/demo/reports.htm 

By 'embedding' I meant making mods to Moodle code, so as to introduce CBM as a smoothly working option within the existing Q&A structures within Moodle (and maintaining this as these structures may evolve). This would be really good, but as someone quite unfamiliar with Moodle code I have little idea of the scale of work that may be involved and the hurdles that may be encountered. I would certainly be happy to discuss these issues and help with the work if it seems feasible. Dan Stowell did some basic work on this in 2004, but he says that Moodle structures have changed sufficiently snce then that his code would probably not be a great help. Our continuing development since then has been on the independent software.

Tony Gardner-Medwin

Average of ratings: -
In reply to Sacha Brostoff

Re: Confidence based marking / Certainty based marking (CBM)

by Sacha Brostoff -

So, after a bit more reading, I see that the CBM marking scheme we've all be talking about (Table 1.) is 'optimised' for T/F questions, which have a random chance of being answered correctly 50% of the time.  The three confidence levels C1, C2 and C3 are set so that they cover the range between 50% and 100% fairly evenly.

Table 1.  Motivating CBM scheme for T/F questions

C1
(<67% certain)

C2
(67 to 80% certain)

C3
(>80% certain)

Score if Correct

1

2

3

Score if Incorrect

0

-2

-6

Prof Gardner-Medwin has also devised a scheme for MCQ questions, where the proportion of correct answers due to chance is going to be lower than 33% - possibly a lot lower depending on how many possible answers you have.  This other scheme (Table 2.) uses a wider range of certainty, so covers the range of certainty more than the first scheme.

Table 2. Motivating CBM scheme for Single Answer MCQ questions?

 

C1
(<50% certain)

C2
(50 to 75% certain)

C3
(>75% certain)

Score if Correct

1

2

3

Score if Incorrect

0

-1

-4

The good Prof finds that since students can adapt to both schemes, and that for the purposes of simplicity you might as well stick to the first marking scheme for all question types.

Ref:

Gardner-Medwin, A.R. Confidence based marking - towards deeper learning and better exams.  In Bryan, C. and Clegg, K. (eds) (2006) Innovative Assessment in Higher Education, Taylor and Francis Group Ltd, London.

Negative Marking

I've come across a post (http://tracker.moodle.org/browse/MDL-5324) where someone claims to have hacked Moodle 1.6 to allow negative marking, and gives the code snippet to insert and lists where it should go...

[Can anyone verify this?  I would try, but my moodle server's still being turned into a moodle server!]

Tim Hunt explains a bit further down the page his view of why negative marking is not yet part of Moodle:

"Yes, I know this is an important feature for a lot of people. However the quick and dirty hack is not sufficient for two reasons:

1. Multiple-response multiple-choice questions.
2. When the quiz allows multiple attempts at each qustion using the penalty-factor system.

If someone can come up with a patch that covers all cases reliably, then I would be delighted to check it in."

Average of ratings: -