Hi,
We love the idea of getting students to indicate their confidence for the answers they are supplying. However, in our situation having them penalised by up to 2 marks is not appropriate - the need for them to be accurate in our situation is not as great as in a medical situation where CBM was developed for. Is there a setting to adjust this or is it hard coded somewhere? If so - does anyone know where?
Thanks,
Lael
Your reference to a penalty of 2 suggests you may not be using Moodle 2.6, or the code patches available to improve CBM for earlier versions. See the documentation at http://docs.moodle.org/26/en/Using_certainty-based_marking and patches available at www.tmedwin.net/cbm . The all-important feedback to students and staff is much improved.
Actually, students rarely suggest that the penalties are too harsh. (Staff are much more inclined to suggest this!). It is really important though that they see feedback with the proportions they are getting correct at different C levels. Each student has lots of questions they understand and can answer confidently with less than a 1 in 5 chance of being wrong. These are the questions they are being asked to identify with C=3 (risking a -6 penalty if they actually are wrong) and the need for a biggish penalty is mathematically unavoidable if they are to be properly motivated to acknowledge uncertainty. It's a mild criterion compared with what one obviously expects of medics for decisions in critical situations - the stakes are often much, much higher and 80% correct wouldn't be nearly enough in these situations. But CBM helps to start them thinking about the principle that it is a good thing to identify uncertainty, and to look for proper justification before you express confidence.
No, there aren't settings available for changing the CBM rewards and penalties. Tim Hunt and I decided this was probably best for reasons of both policy and simplicity, and indeed it would be rather complicated to try to change them in the code. My recommendation would be to persevere with the simple 1,2,3 : 0,-2,-6 scheme and if you find this creates problems please tell us what kinds of students and questions, and what kinds of problems.
Hi Tony,
Thanks for your response. ok - so it got worse in 2.6? It went from -2 to -6!?
I understand that for medics in a critical scenario that is the case. But for a programming student... the program won't compile, or they have to look something up. It is not as high stakes and certainly doesn't need to be so draconian. I understand the desire to encourage the student to acknowledge uncertainty accurately - I'm not sure the issue is as significant in a high school course such as the ones I teach.
In your experience, what is the minimum number of questions that you would use in a CBM based test? It seems like such a large penalty results in a fail very quickly unless you have significant numbers of questions (greater than 20 or 30 at a guess).
Thanks!
Hi Lael -
Look at the documentation. The change of scoring in 2.6 was just a matter of scaling: certainty expressed as C=1,2,3 now gives marks 1,2,3 if correct and 0,-2,-6 if incorrect. In earlier versions the default marks in the unpatched code were 0.33,0.67,1 or 0,-.67,-2 unless you altered the weight of every question to x3. Integer marks make CBM simpler and more intuitive. More fundamental though is the inclusion of proper feedback in 2.6 (or with the patches), and presentation of the scores in a more sensible way than just as a % of maximum possible marks (see the long discussion here).
The way to deal with students who don't feel confident enough to be sure they are giving a correct answer is to encourage them to start with C=1 (1/0 marking) and change to C=2 when they are a bit confident (2/-2 marking) without too big a risk and with the prospect of doubling their marks if correct. When they come across Qs where they are really sure (which of course they will do), then they can use C=3 (3/-6 marking). This will encourage them to check answers and think more about them. -6 means you've claimed to be sure of something incorrect - a reality check and wake-up call. Immediate feedback is very important when starting to use CBM. Feedback at the end shows how well a student is using the options, and final scores include a bonus for effective use, added to the conventional accuracy. Typically, university students give about 40% of their answers at C=1, 20% at C=2, 40% at C=3, depending of course on how difficult the Qs are. There's nothing special about learning medical science - just lots to learn, and there will always be some bits you know and some you don't. It's important to acknowledge and get right the difference.
CBM actually increases the statistical reliability of final scores by up to a factor of 1.5 or more. This means that you don't need so many Qs to get scores that correlate well with student ability. It's important however, before students use CBM in summative (or otherwise important) tests that they get plenty of practice in self-tests for their own benefit, so they get the hang of it. 20 would seem a reasonable test length, though luck can obviously still play a significant part. I'm more interested myself in using tests to encourage constructive thinking and self-assessment than for pass/fail issues - though I acknowledge these are necessary. Lots of issues and discussion are covered in my publications here .
Tony, thanks for the fantastic response. The integer representations do make it easier to understand. I'll need to have a play with it today to see the difference in practice. Is the value of the question considered 1 or 6 for possible marks?
I think the problem we were confounded with was that someone could potentially answer all questions correctly with a low certainty and fail. Or in a test with a small question bank say... 10 questions - a single confident but incorrect question would result in a fail as well (as there are not enough questions which results in the large penalty being a disproportionate percentage of the total marks available). Hence my question about what minimum number of questions you would think would allow CBM to 'work', and my request to be able to alter the penalties to be less harsh.
Does that sound to be how it is intended to work and is there anything I'm missing?
I'll start looking at some of the publications. Thanks!
I don't have experience school teaching, but would be interested to hear how people get on using CBM in school, or if they would like to participate in some research on its use. It always seems to me that 1st year students at uni would have benefited from CBM earlier - because when you ask them a question in class they often tend to answer based on an immediate idea, without thinking much.
The consequences of one or two confident errors in small tests are maybe not as bad as you fear. Suppose (for simplicity) a student gives 5 responses at C=1 and 5 at C=3 in a test of 10:
1 wrong at C=3 and 2 wrong at C=1 mean the student has a conventional accuracy of 70%, a CBM bonus (for using CBM well) of 1% and a resulting CB accuracy of 71%.
2 wrong at C=3 and 1 wrong at C=1 mean the student has accuracy = 70%, CBM bonus (for poor CBM judgment) = -7% and CB accuracy = 63%.
If you want to overlook poor CBM judgment as with student 2, you can always 'pass' on the basis of either accuracy or CB accuracy. But this student has something to focus on: twice doing worse on things s/he was confident about than on things s/he knew were dicey.
I didn't understand the issue about the 'value of a question' being 1 or 6. I think it is always best with CBM to keep the same 'value' for each Q to give marks 1,2,3 for correct answers. This is the default weight (=1) in Moodle 2.6 or with the patches in earlier Moodle. Moodle sometimes calls this weight the 'max mark', though with CBM it is really a 'nominal mark', since a mark up to 3 times bigger can be awarded. Unfortunately Moodle may somewhere still have a bug that truncates grades to the sum of the 'max marks' when it makes up a table, but the Quiz Report Plugin that you must install for CBM will show things correctly.
