Rationalising grading options (adaptive, feedback, negative marks, CBM)

Rationalising grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Number of replies: 21
Picture of Particularly helpful Moodlers Picture of Plugin developers

The grading options under 'update quiz' seem potentially confusing, and need anyway to be adapted to accommodate CBM, hopefully in 2.0. I suggest the arrangement below, shifting 'Grading Method' to 'Attempts' and renaming it 'Final grade based on..',  and combining 'Adaptive Mode' and 'Penalty Scheme' into a single multi-option switch called 'Submission & Grading'

mark schemes

The grading options would all allow negative marking of individual Qs except the last, adaptive mode, which would apply penalties for multiple submissions only down to a mark of zero.  A new facility (apart from the addition of CBM) would be option 3 (immediate f/b), allowing a student to submit answers one by one (as with adaptive mode) to get immediate feedback, but without the option to resubmit in the light of feedback.

Any comments?  I think this should cater fine for those wanting negative marking. My inclination would be to truncaate the total score on a test to a minimum of zero, though negative total scores are of course rare in any case. NB I've omitted the current combination of adaptive mode ignoring penalties set on individual Qs, though that could of course be added as an extra option 'Adaptive mode without penalties' if anyone thinks getting the answer right on the third attempt is worth the same mark as getting it right first time!

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalising grading options (adaptive, feedback, negative marks, CBM)

by Ray Lawrence -
Tony I think the last item needs to be kept as an option.
Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Florian -

Tony I think the changes address the need for negative marking. I think that when using CMB the CBM score should be an output of the submit and grade selection. The CBM score should only increase if the number of correct and certain responses on the quiz increases. With this option, if a student tries to game the quiz then the CBM score will be very low. Only if the learner is certain and correct should they receive a passing mark. This is similar to the scoring of the KFI system. It worked effectively in our tests at Douglas Count Schools.

Your thoughts?

Average of ratings: -
In reply to Tim Florian

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
The changes we I am planning to do to adaptive mode for Moodle 2.0 already change these parts of this forum, so this thread may become obsolete - or I may decide that what is discussed here is the best way to do it, and copy it.
Average of ratings: -
In reply to Tim Hunt

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

OK fine. I'll ignore the form for now. But it would be helpful to get straight any proposed changes in the structure of the control variables in the code (at present binary switches 'adaptive' and 'penaltyscheme').

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Basically yes, can we change it from Adaptive mode yes/no and Apply penalties yes/no to your Submission and grading setting.

Can we define some constants at the top of mod/quiz/locallib.php:

QUIZ_MODE_BATCH_SUBMIT 0
QUIZ_MODE_ADAPTIVE 1
QUIZ_MODE_ADAPTIVE_NOPENALTY 2
// add any others you need.

Don't bother with the database update. I'll do that.
Average of ratings: -
In reply to Tim Hunt

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

I'm working at present with the following control options for the marking & submission schemes. Please say if this interacts/conflicts strongly with other code changes. The aim is to use several bits in optionflags to directly control the code. Only bit 1 (ADAPTIVE) is being used at present. The PENALTY bit replaces the functionality of the present 'penaltyscheme', which itself is used to represent directly the choice amongst the various options, for which the control bits are set in mod/quiz/lib.php. ('penaltyscheme' could better be renamed 'markscheme', but since it is in the database, this seems maybe unwise). I've put code into restorelib.php to set these variables correctly when backups lacking new options are restored into new code.

Tony G-M

// values of penaltyscheme
define('QUIZ_MODE_BATCH_SUBMIT', '0');
    // optionflags set to 4 = QUESTION_NONEGMARKS
define('QUIZ_MODE_ADAPTIVE', '1');
    // optionflags set to 7 = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE | QUESTION_PENALTY
define('QUIZ_MODE_ADAPTIVE_NOPENALTY', '2');
    // optionflags set to 5 = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE
define('QUIZ_MODE_BATCH_WITH_NEG', '3');
    // optionflags set to 0
define('QUIZ_MODE_IMMED_NO_NEG', '4');
    // optionflags set to 12 = QUESTION_NONEGMARKS | QUESTION_IMMEDFB
define('QUIZ_MODE_IMMED_WITH_NEG', '5');
    // optionflags set to 8 = QUESTION_IMMEDFB
define('QUIZ_MODE_IMMED_CBM', '6');
    // optionflags set to 24 = QUESTION_IMMEDFB | QUESTION_CBM
define('QUIZ_MODE_BATCH_CBM', '7');
    // optionflags set to 16 = QUESTION_CBM

//Component bits of optionflags
define('QUESTION_ADAPTIVE', 1);
define('QUESTION_PENALTY', 2);
define('QUESTION_NONEGMARKS', 4);
define('QUESTION_IMMEDFB', 8);
define('QUESTION_CBM', 16);
define('QUESTION_NONEGFINAL', 32); // (not used at present but could control whether overall final grades are allowed to be negative) 

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I'm feeling pretty tired and ill today, so I don't really feel confident judging this, but on the whole it looks right.

I would change one thing. It seems silly to say things like

define('QUIZ_MODE_ADAPTIVE', '1');
// optionflags set to 7 = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE | QUESTION_PENALTY

Why not just do

define('QUIZ_MODE_ADAPTIVE', '7');
// = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE | QUESTION_PENALTY

that is, make the constants be the correct setting for optionflags. They are all distinct numbers.

Average of ratings: -
In reply to Tim Hunt

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Fair enough Tim! 0-7 is the sensible ordering on the form, so was the starting point - as indeed you suggested. But in retrospect the ordering of presentation of the array of options can probably just be determined by the sequence in which they're written down. This anyway only concerns code round modification of the quiz form and restoration, while the more important point is the use of independent bits in optionflags throughout, which I don't want to conflict with others' plans. Extra bits can of course be added for other controls.

Sorry it sounds like the onset of real November is taking its toll on you Tim. We're celebrating Andrew Huxley's 90th birthday tomorrow, so no rush for any other comments!

Tony 

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Hi Pierre! I'm replying here to your post today at http://moodle.org/mod/forum/discuss.php?d=90013 because the relevant info is discussed in this thread. Controlling negative marking and CBM options shouldn't I think be a matter of new Q types, because they can apply to any Q that for which answers are definitely right or wrong. For odd reasons I did occasionally have to alter code in specific Q type routines, but not so as disable their normal functions in any way.  So the CBM implementation at www.ucl.ac.uk/lapt/moodle19/moodle (login as a teacher with userid=mdlteacher password=mdl19) can simply switch a quiz set up with a variety of Q types between conventional and CBM (with or without immediate feedback) under 'update quiz'. Notes about this, and the implemented Q types are at http://www.ucl.ac.uk/lapt/moodle19/moodle/mod/forum/discuss.php?d=3 

The proposal from Tim & myself in this thread is to have mostly independent control bits that govern the following: Whether to use ADAPTIVE MODE, with or without PENALTY; whether to allow NEGMARKS on a Q, whether to give IMMEDIATE FEEDBACK, whether to use CBM, and whether to set a minimum overall score of ZERO. I'm afraid I haven't had time to work on this since November, when I got completely frustrated trying to make use of CVS. But the coding issues aren't complicated (though they have to be in core rather than as plugins). You should see what comments you have on the functionality in the modle19 site above, which is based on a rather old (8/07) version of 1.9.

Tony

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Pierre Pichet -
"to have mostly independent control bits that govern the following:"
and in the other post you set the bit order.
// values of penaltyscheme
define('QUIZ_MODE_BATCH_SUBMIT', '0');
// optionflags set to 4 = QUESTION_NONEGMARKS
define('QUIZ_MODE_ADAPTIVE', '1');
// optionflags set to 7 = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE | QUESTION_PENALTY
define('QUIZ_MODE_ADAPTIVE_NOPENALTY', '2');
// optionflags set to 5 = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE
define('QUIZ_MODE_BATCH_WITH_NEG', '3');
etc.
Could you define more clearly for people like me that are in a learning processwink, a truth table that list all the real possibilities.

Pierre
P.S. This will help me solve correctly or suspend work on MDL-12406
Average of ratings: -
In reply to Pierre Pichet

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Is this the kind of table you are asking for, Pierre? The defined codes are the sum of the bit values for each of the options treated as 'true'. I don't think this handling of presentations and marking interacts a great deal with what you are doing in  MDL-12406 does it? In myMoodle19 demo site I have an identical demo quiz set up using the codes 0, 8, 16, 24 as I recollect. The negative marking people are asking for 0 and 8 to be available in core, which is easily set up and has been done by a number of people independently in one way or another. Tony

define('QUIZ_MODE_BATCH_WITH_NEG', '0');  // = 0
   // batch submission, using negative marking if specified for individual wrong answers
define('QUIZ_MODE_BATCH_SUBMIT', '4');     //  = QUESTION_NONEGMARKS
   // batch submission, ignoring negative marks that may be specified for individual wrong answers
define('QUIZ_MODE_ADAPTIVE', '7');     //  = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE | QUESTION_PENALTY
   // adaptive mode as at present, using set penalties but not allowing mark on a Q to be <0
define('QUIZ_MODE_ADAPTIVE_NOPENALTY', '5');  //  = QUESTION_NONEGMARKS | QUESTION_ADAPTIVE
   // adaptive mode as at present, ignoring any specified penalties, not allowing mark on a Q to be <0
define('QUIZ_MODE_IMMED_NO_NEG', '12');   // = QUESTION_NONEGMARKS | QUESTION_IMMEDFB
   // answers marked immediately (as in adaptive) but not allowing second attempts. Negative marks for wrong answers ignored.
define('QUIZ_MODE_IMMED_WITH_NEG', '8');  // = QUESTION_IMMEDFB
   // answers marked immediately (as in adaptive) but not allowing second attempts. Negative marks used for wrong answers if specified
define('QUIZ_MODE_IMMED_CBM', '24');   // = QUESTION_IMMEDFB | QUESTION_CBM
   // answers marked immediately (as in adaptive) but not allowing second attempts. Uses CBM.
define('QUIZ_MODE_BATCH_CBM', '16');  //  = QUESTION_CBM
   // batch submission, using CBM.

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Pierre Pichet -
Thanks,
Probably, but I have to study it to be sure that it is as clear in my head than in yourssmile.

For example, it is not clear (for me...) that a question valid as a normal question i.e
  • one with a 100% grade for a response and 50% for a response "not so bad" .
  • ...
can be handled correctly for all those settings.
If no, how do we warn the teacher that his question is not compatible with the actual quiz setting ( for example the actual cloze for CBM if I understand well your comments on your site)

Pierre

P.S. MDL-12406 is not directly implied actually but could as this will interfere with the lower (in the quiz editing page) settings of feedback display.


Average of ratings: -
In reply to Pierre Pichet

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers
one with a 100% grade for a response and 50% for a response "not so bad" .

What I did technically in this situation with CBM is simply to give the CBM mark for a correct answer (1,2 or 3) multiplied by the set factor (as in my Q6 in the demo for incomplete answers, or Q2 if you click 'dog'). Cloze Qs at present are simply handled without CBM even if it is set, because I haven't tried to implement a way to ask for certainty about each of the separate decisions made by the student; the best option might be to use popups to avoid cluttering the screen. I have no experience with Cloze.

 

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Pierre Pichet -
Thanks,
I need to study all this, experiment and come back later.

Pierre
Average of ratings: -
In reply to Pierre Pichet

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Florian -
Pierre or Tim, I would like to put the CBM score that is generated in the grade book so that I can download it in excel for analysis. I would like it to go next to each raw score that a student receives when taking a quiz. Can you guide me to the code that I need to modify for this and I would appreciated any guidance you can provide with this project.
Average of ratings: -
In reply to Tim Florian

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
A single activity can have multiple columns in the gradebook. The code that needs changing is in mod/quiz/lib.php, where it calls quiz_grade_item_update and grade_update.

I am not sure exactly what you need to do to get multiple columns. Try looking at the other modules (particularly assignment) and asking in the general developer forum.
Average of ratings: -
In reply to Tim Florian

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Are you trying to download scores for attempts already done, or for attempts in the future? The latter might be easier to deal with, though both are in principle possible. CBM grades for existing attempts would have to be recalculated from question data, as happens in 'review'. What Moodle version are you using? My code for 1.9 didn't put CBM grades into the gradebook, since I understood the gradebook structure was to undergo changes before Christmas. Perhaps it is stable now and worth tackling.
Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Florian -
Tony I am using the same version you are as I downloaded my moodle from your site. This month I am starting to migrate the CBM mods to the latest version of 1.9 beta so that I am using the most current version. As for the gradebook, I want to be able to have the CBM score print to the gradebook as well as downloading the resulting score to excel. Any collaboration/help/guidence on this from those of you who know moodle code better than I would be helpful.

Tim
Average of ratings: -
In reply to Tim Florian

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

I can tell you how to put the CBM score into the gradebook INSTEAD OF the raw score. Would this be OK?  I don't want to do things that require changes to the database structure at present, and Moodle (in 1.9 anyway) only stores a single float number as a grade.

If you are not using partial credit for any of your answers, then there is another way. I can make the grade be something like 53.027 which would mean CBM=53 and Number Correct=27 (max=999!), which you can separate in Excel. Would this be OK?

I think an eventual solution to this issue will require either an addition of extra fields to the gradebook database, or making the grade field be a string instead of a float, which could then accommodate either simple grades, like 16.5 or arbitrary complex comma separated grades, like 27,53 or 85,A- etc. 

Average of ratings: -
In reply to Tony Gardner-Medwin

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tim Florian -
The CBM instead of the raw score is just fine. Can we notify the user what the CBM score is out of? Such as a 5 question quiz has a possibility of a CBM score of 15. Is that information available some place for the learner?

I am not using partial credit so the second option is usable as well. I want the students to see their grade as a CMB score and use the negitive marking to get them to self-regulate their learning.

Have you looked at updating your moodle to a newer version? I have this weekend blocked out to start the upgrade of CBM to a new 1.9 Beta. If I can add the CBM score in the gradebook my live would be complete (my school life any way).

Thanks Tony

By the way I have been accepted as a presenter at a conference in Colorado this summer using CBM in Moodle. I will keep you up to date as I find out more information. Your posts of conversations about CBM on the UCL site are helpful when preparing for discussions on how CBM works and what it can do for lerners.
Average of ratings: -
In reply to Tim Florian

Re: Rationalizing grading options (adaptive, feedback, negative marks, CBM)

by Tony Gardner-Medwin -
Picture of Particularly helpful Moodlers Picture of Plugin developers

CBM isn't really the focus in this thread, but I'll try and answer. Students who gamble inappropriately with CBM, by expressing certainty when they aren't sure, will definitely lose out. The important thing with a CBM scheme is that they gain by expressing uncertainty when they aren't sure (more than by just not replying or pretending there aren't problems). This is about training students to discriminate between reliable and less reliable answers. Answers in a quiz will usually consist of a mix of things you're sure and unsure about, and if you judge reliability correctly your score will always (on average, or expectation) increase. But of course as Tim Florian wants, you only get a really good score if you are correctly confident in answers to a good proportion of the Qs. Tony G-M

Average of ratings: -