Quiz

Is it necessary for Grade Reports to truncate grades to their nominal maximum values?

 
This discussion has been locked because a year has elapsed since the last post. Please start a new discussion topic.
argm
Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Plugin developers

When using CBM behaviour in 2.6, 2.7, grades reflect the total number of marks, which can be up to 3 times the sum of the weightings of individual Qs (i.e. maximum marks without CBM). This is handled fine when you look at results using the analysis of CBM data for a quiz with CBM (in Attempt Reviews or with the (essential) teachers' Quiz >Results > CBM Grades plugin). However, crude display of grades on all the quizzes in a course often show incorrect CBM grades because the code truncates the grades at their nominal maximum, which is inappropriate for CBM.

I suspect this truncation in the code (for example using the function bounded_grade() in /lib/grade/grade_item.php ) is actually unnecessary without CBM and damaging with it, so should perhaps be removed from the code. Does anyone see why it should be kept? If (other than with CBM) the grade is higher than the sum of maximum grades on all the Qs it would appear that something must be wrong elsewhere, and so it wouldn't in any case be appropriate just to truncate the grade.

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Core developersGroup Documentation writersGroup Particularly helpful MoodlersGroup Plugin developers

This is really a gradebook issue, and Martin D has said that there will be some work done on the gradebook in Moodle 2.8. A few weeks ago there was a meeting about that in Los Angeles (not very convenient) but I don't know what was decided there. Hopefully someone will write it up at some point.

This reminds me of the huge debate there was about 'extra credit' which is something Americans, however, I don't recall exactly what was done there, but I think it was the addition of the 'Unlimited grades' option on Admin -> Grades -> General settings. Does that option do any good in this case?

I am being pretty useless here. Does anyone know more.

 
Average of ratings: -
Picture of dawn alderson
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
 

Hi,

so,

In mathematics and computer science, truncation is the term for limiting the number of digits right of the decimal point, by discarding the least significant ones.

I suppose...if there is any option that involves ease for practice for both teacher and learner....might that be the best way to go....(rounding up-is always a definition of ease for me as practitioner) if no effect on that.....am standing on sidelines....with a hot cup of cocoa.

D

 
Average of ratings: -
argm
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Plugin developers

Thanks Tim.

Yes, setting 'Unlimited grades' in the Admin -> Grades -> General settings does cure the problem. Searching code, this seems  the only place it has any effect - in the function bounded_grade(). Unfortunately the default for the switch is off, and as a global switch it is only admin accessible. It ought really to be 'on' by default since it messes up presentation of CBM quiz grades. Having it on always (or at least by default) would mean it would no longer override people who try to give manual grades above the nominal maximum - but surely that is a matter for in-house rules about how you grade, just like any other guidance about how to grade manually. NB In a large university it can be very hard to get global settings changed (or even plugins installed), however common sense they are.

Last week I see someone else was thrown by this default setting of unlimited grades (? changed since 2.4) in a different context: https://moodle.org/mod/forum/discuss.php?d=262334#p1136638 .


 
Average of ratings: -
Picture of dawn alderson
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
 

oh I am scratching my head over this...having thought more about it.  I always tend to need to picture examples from practice-in my head...to unpick concepts. Here goes:

'extra credit'- massive thought cloud above head...I am  not sure what this means in terms of UK HE-because credits are fixed credits for modules pre-determined at programme development-then pass quality control/ and then final fixtures of module credits put in place and eventual award calculated within those fixtures-sunny and hot here might not be making sense, apols.

Now, extra credit in another context might mean-could be off kilta-that A* award for GCSE exams....where the pupils can gain the extra credit from sitting the whatever they have to do to top-up from an A to A* (am talking practice in class-course work tracking of marks from completion of higher papers and whatnot, say).

This then would appear to suggest, extra credit is about accommodating a stretch of the boundary-marks/grades, which would be down to the teacher's discretion to place their informed judgement in terms of the marking criteria/rubrics, and in turn need to input marks with no restrictions.

Now I could (probably!) be wrong. If anyone else can think of an example that presents an argument for the override-argument/change-gosh that would be helpful. 

Itch now scratched, however, I think!  smile

D

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Core developersGroup Documentation writersGroup Particularly helpful MoodlersGroup Plugin developers

It might be better to move this discussion to the Gradebook forum.

If you want to propose the question "Can we change the default value of this setting" then really you need to know the answer to the question "What bad side effects, if any, would there be if we made that change?"

I am afraid that I don't know.

 
Average of ratings: -
Picture of dawn alderson
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
 

that is exactly where I was wandering- aimlessly-...in my thinking

"Can we change the default value of this setting" then really you need to know the answer to the question "What bad side effects, if any, would there be if we made that change?"

D

 
Average of ratings: -
argm
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Plugin developers

I've added a note about the need to set 'Unlimited grades' to the CBM Documentation  ]. If you don't set this, CBM gradebook entries will show in reports limited to one third of their maximum possible values. That's just a result of how it seemed best to Tim and me to handle the complexities of dealing with CBM marks - we made the nominal mark for a correct answer be the CBM mark for a correct answer at low confidence (C=1) so you can get up to 3 times this. The best interpretation of CBM marks is through the tailored CBM reports in the quiz domain. But it would be totally confusing also to have gradebook reports that show students who have 100% and 35% of the maximum mark as having the same grade - through unnecessary truncation.

I agree of course about the need to assess potentially bad side effects of a change in the default. Also this needs to involve the gradebook forum folk. But it's primarily an issue affecting (CBM) quizzes. As far as I can see, turning on 'unlimited grades' will only affect display of other grades if manual graders have chosen to assign grades above their nominal maximum: these would come to be shown as assigned, no longer limited to the nominal maximum. As I said, this issue of whether extra credit is allowed should surely be a matter of in-house policy - perhaps varying between different courses and different quizzes. It surely makes no sense to use the software to over-ride 'extra credit', because graders and grades not conforming to house rules should stand out for discussion, not just suffer truncation (I stick to my Latin verb: 'cut the top off', though I don't actually mean it to apply to the graders!).

If any quiz people see bad side effects of allowing grades to exceed the nominal maximum, do say - before it transfers to gradebook people. But note that this is NOT about the desirability of 'extra credit' or making A into A*. It's about whether the software should over-ride whatever extra credit people have consciously chosen to do (like give an extra 5 marks to an outstanding essay when the nominal maximum is 50).






 
Average of ratings: -
Picture of dawn alderson
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
 

Hi,

thanks for this, really useful to have the detail.  Please do forgive me, I have a couple of points in reference with the last post Tony, they may be points that are obvious.

1. 'I've added a note about the need to set 'Unlimited grades' to the CBM Documentation '

this tells me that we are 'only' talking about the CBM quiz, and I agree it does sound complex, not least in respect of the gradebook alignment-so it seems to me.

In view of point 1. I think I the get the following:

'It's about whether the software should over-ride whatever extra credit people have consciously chosen to do (like give an extra 5 marks to an outstanding essay when the nominal maximum is 50).'

...in as much as I assume this is an example but, not a plan for any impact on the assessment of essays? Yet, I find this conflicts somewhat: But it's primarily an issue affecting (CBM) quizzes

2.'I agree of course about the need to assess potentially bad side effects of a change in the default.'

I think, this maybe related to Point 1.

General points:

- What is the estimated %/chance/probability of the stduent being able to score 100%? I think, if this is low then does this type of quiz belong in gradebook...might it be a reflection tool alone'?

-I think equity issues can arise from issues such as the following:

'like give an extra 5 marks to an outstanding essay when the nominal maximum is 50'  Reason: it suonds as though this invites the award of marks without any underpinning reasoning...

hope helpful

Dawn


 
Average of ratings: -
argm
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Plugin developers

@Dawn -  Perhaps I've led you to think this is more complicated than it is. Issues like whether to allow extra credit should be debated in a local educational context. Moodle is here to facilitate what people think is good practice, not to dictate what that should be.

Here are the downsides of having  "unlimited grades" off (the current default):

  • CBM quiz grades appear in gradebook as mostly nonsense. Nearly everyone shows as 100% because they are limited to and compared to 1/3 of the actual maximum possible mark. They are stored OK but displayed wrong.
  • If people have good reason to use gradebook override facilities to award grades for exceptional assignments above the nominal maximum, then these won't show.

Then there's the downside of having  "unlimited grades" on:

  • People can use gradebook override facilities to award marks for exceptional work above the nominal maximum, which may be contrary to local policies.

It seems a no-brainer to me, in favour of having "unlimited grades" on by default. They should only be set 'off' if local policy is never to allow exceptional marks and never to use CBM. But I may have missed some way in which there are other technical implications of this option. 


 
Average of ratings: -
Picture of dawn alderson
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
 

Hello Tony,

Appreciate you taking the time to respond back, thanks. I am now going to let you have my concluding thoughts.

1. When you say: issues like whether to allow extra credit should be debated in a local educational context

-I think this is a redundant idea for Education systems in the UK...I tried to map a GCSE example and I have since found out the 'local educational context-argument' does not count in that instance either.

2. When you say: Perhaps I've led you to think this is more complicated than it is. Issues like whether to allow extra credit should be debated in a local educational context. Moodle is here to facilitate what people think is good practice, not to dictate what that should be.

-I didn't think you were being dictatorial smile

Tony, I think this discussion has moved on somewhat in terms of simply listing the difficulties/downsides of changing the default of unlimited grades and I will now outline the reasons:

-It is apparent that the CBM quiz is out of synch with the way in which gradebook works.E.G. To use gradebook, as we would for any other formal assessments, we need to use a mark-range, and I mean utilise the 'whole of the mark-range'. To be clear, all students should have an equal chance of achieving the highest top band grade, and that band should be used. The CBM quiz is out of synch with that because it appears terribly difficult for all students to have a first attempt at assessment and pass with a 100% score across the quiz....now if you think I am wrong with that...I will listen, of course.

-Thus, I think the beauty of the CBM quiz is that it is a self-testing tool that aides reflection GREAT! I do not think the CBM quiz belongs in gradebook and I do not think therefore the current default should be changed. One more reason for that follows, when you say:

'It seems a no-brainer to me, in favour of having "unlimited grades" on by default. They should only be set 'off' if local policy is never to allow exceptional marks and never to use CBM. But I may have missed some way in which there are other technical implications of this option'.  I think that is a big risk.

I really do hope this is helpful.

Cheers,

Dawn

 
Average of ratings: -
argm
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Plugin developers

The best way to look at CBM performance is by considering both Accuracy (% correct) and CB Accuracy (=Accuracy + CB Bonus) side by side, as described in the CBM Documentation  and shown in 2.6+ (Attempt Reviews and the CBM Grades plugin). A positive bonus indicates successful identification of reliable and unreliable answers. Good students will get close to 100% more easily with CB Accuracy unless they have confident wrong ideas. The statistical reliability of CB accuracy data is much better than conventional data and it is a better predictor of conventional accuracy on other Qs than is conventional accuracy itself. So the proper CBM measures are pretty sound and easy to use. CBM is of course not part of exams like GCSE, but I'm sure things would be better if it were. That's why I'm so concerned to have CBM working properly in Moodle and not showing stupid things.

I agree with Dawn on one point, that the Moodle gradebook grades, which are essentially the scaled sum of marks on all the questions, are not the best way to summarise CBM performance. This is because 80% accuracy typically goes along with an average CBM mark that is around only 50% of its maximum. This doesn't reflect any inadequacy of the students, but the fundamental way in which CB marks cannot be a linear function of accuracy. It would be better if changes could be made in Moodle code so that the gradebook showed CB Accuracy rather than the scaled mark totals. But so far as I can see this would be hard to achieve given the way the code is set up.

People using CBM to help their students should be using the proper CBM report tables in the plugin, not the gradebook. However, the gradebook entries do reflect the total of CBM marks - which is what the students see accumulating with immediate feedback. This is a simple measure, and the higher the total the better the student is doing. The limited problem I'm trying to tackle is simply to make the gradebook reflect these totals accurately, instead of mistakenly cutting them off at 1/3 of their maximum values and making it appear that anything above this value deserves 100% credit.
 
Taking a wider perspective, here are 3 different ways of handling the problem, in my order of preference:

  1. Make the gradebook show values for CBM quizzes that are the specified maximum grade * the CB accuracy.
  2. Make the gradebook not show grades for CBM quizzes. I'm not keen on simplistic gradebooks, but the omissions would be a bit weird.
  3. Continue showing grades as calculated at present, but revealing the true values, as with "unlimited grades" turned on.

(1) seems hardest and (3) easiest. Having looked at the code I don't see reason to think there is a risk involved in (3), though of course I would like someone else familiar with gradebook code and usage to look at both this issue and at how difficult (1) might be instead.
 

 
Average of ratings: Useful (2)
Picture of Joshua Bragg
Re: Is it necessary for Grade Reports to truncate grades to their nominal maximum values?
Group Testers

First, I've only played with CBM fairly briefly at the end of this year.  With that said, I really do think that #1 should be the priority.  Those scores are most helpful to both students and teachers.

Rather than fighting institutional momentum and Moodle momentum with #3, it seems to be a better option to find a way to implement #1.

That and get the reports in Moodle core.... They really are invaluable.

 
Average of ratings: -
argm
Correction of the problems with the presentation of 'Grades' for CBM quizzes
Group Plugin developers

I've given in to the wise logic of Joshua's vote for solution #1 to the problem of inappropriate and truncated 'Grades' being shown for Quizzes that use CBM.

That has meant tackling some code that was rather obscure to me, but that I think I have mastered OK. You can download zips (from here for 2.6 and from here for 2.7)  with original and new versions of the altered files (including the CBM grades plugin that really has to be in core code). I would much appreciate anyone who could check or comment on the relatively minor mods to the code, and their usefulness. I very much hope these can go into 2.8. They may have to remain as patches for 2.6 and 2.7 - after any necessary corrections. They don't affect functionality when CBM is not in use. I've put them onto my own website rather than as attachments here, so improvements can be easily incorporated without confusion.

Tony GM 

Notes on the changes:

................................................................................................................

1. Add a link to the 'Help' button for Behaviours under Quiz - Edit settings to link to information about CBM.
This just requires addition of a single line as follows to /moodle/lang/en/question.php
     $string['howquestionsbehave_link']= 'qbehaviour/deferredcbm/certainty';

2. When CBM behavours are set, calculate the Moodle 'Grade' as the maximum grade for the Quiz, multiplied by the 'Certainty-Based Accuracy' (= Accuracy +  CB bonus). At present the summary 'Grade' is calculated in a way that is confusing and usually exceeds the nominal 'maximum' grade for the quiz, and is liable to truncation in displays. This involves:
    * A new grade calculation function in
             /moodle/question/engine/questionusage.php
    * Clarification & changes of display items in grade displays (only effective when quiz CBM behaviours are selected), in:
             /moodle/question/behaviour/deferredcbm/behaviourtype.php
             /moodle/question/behaviour/deferredcbm/renderer.php
             /moodle/mod/quiz/view.php
             /moodle/mod/quiz/review.php
             /moodle/mod/quiz/report/cbmgrades/report.php
             /moodle/mod/quiz/report/cbmgrades/cbmgrades_table.php
    * Minor changes & additions to language descriptors in:
             /moodle/question/behaviour/deferredcbm/lang/en/qbehaviour_deferredcbm.php
             /moodle/mod/quiz/report/cbmgrades/lang/en/quiz_cbmgrades.php

3. Include the CBM grades plugin as core code rather than as an add-on. Everybody seems to agree this plugin is essential when using CBM, and there is now an additional reason, to ensure that the CBM Grades displays are consistent with code elsewhere displaying Grades.


 
Average of ratings: Useful (2)
Picture of Joshua Bragg
Re: Correction of the problems with the presentation of 'Grades' for CBM quizzes
Group Testers

Is there a tracker issue for all of this yet?  (Changes to gradebook score and CBM reports added to core)

If I have a chance to try this out of my dev site in the next two weeks before school starts then I will... but that's doesn't seem likely.

 
Average of ratings: -
argm
Re: Correction of the problems with the presentation of 'Grades' for CBM quizzes
Group Plugin developers

I don't know that anyone has set up a formal tracker about this, though there have been various discussions in the forum and related to the CBM grades plugin. Trying out the code just involves checking that the original versions of the files in the zips are the same as you are using, then copying  in the new files, in a single operation. You can always copy back the original versions to restore.

 
Average of ratings: -
Picture of Joshua Bragg
Re: Correction of the problems with the presentation of 'Grades' for CBM quizzes
Group Testers

Well for this to move forward for 2.8 as you'd suggested, I would think that someone would have to make two tracker items.  One would be to include the CBM reports in the standard distribution and the other to make these gradebook changes...

 
Average of ratings: -
argm
Re: Correction of the problems with the presentation of 'Grades' for CBM quizzes
Group Plugin developers

Let's see what Tim thinks, because he's going to have to approve (or improve!) what I've done. I'm not familiar with the tracker process.


 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Correction of the problems with the presentation of 'Grades' for CBM quizzes
Group Core developersGroup Documentation writersGroup Particularly helpful MoodlersGroup Plugin developers

Trying out the code would be so much easier if the changes were in git. Joshua has already pointed out the need for a tracker issue for this. The process is summarised on http://docs.moodle.org/dev/Process#New_feature_development

I want to hear what other quiz users think.

 
Average of ratings: -