Inaccurate Calculations in Simple Weighted Mean (SWM)

Inaccurate Calculations in Simple Weighted Mean (SWM)

by Bob Puffer -
Number of replies: 10

Moodle core gradebook 1.95 or greater
Try out this scenario (my actual scenario in a production course site):

  1. course category set to SWM
  2. three subcategories (for organizational purposes) set to SWM
  3. gradebook set to aggregate non-empty grades only
  4. 1st category has four grade items marked (so far) representing 85 points.
    2nd category has three items marked (so far) representing 50 points
    3rd category has one item marked (so far) for 90 points.

The wrong total will be computed (real, percentage or letter) because...

If you leave the category totals at the default (probably 100) SWM takes the percentage correct for three categories valued at 100 and averages them, disregarding the "imputed weights" of 85, 50, and 90.

Potential Solutions?

  1. Sum of grades, except it doesn't exclude ungraded items, just takes the sum of grades and is therefore unusable for translating to accurate percentage or letter
  2. Change the  category total to represent the actual total points of the grades marked so far, except you have to mark all your grades and change the total in one fell-swoop, probably making the gradebook unavailable to students in the mean time and you have to do all this each time you mark another grade. You'd also need to go through and exclude (and possibly later re-include) all those students who haven't turned in their grades yet.
  3. Alternatively to #2, you could use Weighted Mean of Grades agg method for the course category and assign weights to each subcategory that represent the proportioned weights of the marked items to-date.  Like #2, all three weight proportions would slide every time a new grade is marked.
  4. Disallow the display of the course grade until all grades have been marked and implement #2 or #3 above to accurately weight the subcategories' contributions to the final grade.  A disappointing workaround at best.  Hope you have your version patched because most instances still hide the whole gradebook when the course total is hidden.  Some will also claim that "Hide totals if they contain hidden items" might get us some leverage here, but it really won't (IMO).
  5. A calculation for the final total?  Give this a try but don't waste too much time on it before you remove it from the list of candidates -- you'd be revising the calculation as often as #2 and #3 above.

Moodle's help documentaiton's explanation for SWM is:

"The difference from Weighted mean is that weight is calculated as Maximum grade - Minimum grade for each item. 100 point assignment has weight 100, 10 point assignment has weight 10.
A1 70/100, A2 20/80, A3 10/10, category max 100:"

As we have now found out, this of course does not actually hold true for nested SWM categories because we're locked into that category total that needs to move around.

A REAL SOLUTION:

The rawgrademax field in each grade_grades record for category or course itemtypes should be maintained as the "maximum earnable".  This would be the total of grademax for child items marked so far if aggregate only non-empty grades is on, and the total grademax for ALL child items if aggregate only non-empty items is turned off.  Consequently the finalgrade field in grade_grades would then represent the accurate number of points EARNED.  With this valuable information available the grade reports can calculate accurate category and course totals under all circumstances.

I've put in a tracker item MDL-24554.  Please vote, vote early and vote often (under assumed names).

In reply to Bob Puffer

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Bob Puffer -

A bit more research has led me to another possibility though only slightly less ugly, in my view.

You can "Aggregate subcategories" for the main course category which will accurately weight the points.  We turn this off and force it to be hidden from instructors because of the ENORMOUS trouble caused when its used in the wrong places.  In fact, this is the only time I've found a right place for it.

In reply to Bob Puffer

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Robert Russo -

The issue that most people fail to see in SWM is that when the subcategories get normalized to their base 100 grade, the weights are already pre-baked in, so an average is all you need in order to calculate the correct grade.

If you'd like the final grade for each subcategory to reflect the actual number of points earned, use SUM for those subcategories. This way the category total for each subcategory will be less confusing, mathematically. This will not work with excluded items.

Another (simpler) way to deal with this and have all the benefits of SWM is to make each subcategory total to be the sum of the max grades for each item. This way the grades will not normalize and the weights more apparent. Do this when creating the item, to avoid confusion from students.

 

But leaving the total as 100 still results in a course total that would be the same grade based on their 3 different subcategory totals as their normalized values are the same.

Here's the math:

65/85 = 0.764705882352941 = 0.764705882352941 X 85W = 65pts

38.23529411764705/50 = 0.764705882352941 = 0.764705882352941 X 50W = 38.23529411764705pts

68.82352941176469/90 = 0.764705882352941 = 0.764705882352941 X 90W = 68.82352941176469pts

If you SWM them to a course total, you get:

(65 + 38.23529411764705 + 68.82352941176469) / 225 = 0.764705882352941 X 100 = 76.4705882352941

 

If you normalize those to 100, you get:

76.4705882352941 + 76.4705882352941 + 76.4705882352941 / 300 = 0.764705882352941 X 100 = 76.4705882352941

76.4705882352941 = 76.4705882352941

Whew.

In reply to Robert Russo

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Elena Ivanova -

ok,  a little bit more on this issue from our corner of the world.

We see/understand the SWM math that Robert describes, but most of the time this is not what instructors expect in the situation that Bob mentioned, even the advanced ones.

Instructors want to have the SUM way of calculating the total, but at the same point of time they also want to be able to use SWM features to:
- utilize  "Aggregate only non-empty grades"
- have a different number of grade items per person
- exclude students

p.s. My personal concern is SWM discrepancy for  students who are excused/excluded on the assignment vs those who get points.
E.g. - Get a category with SWM with 2 items. Do not use Aggregate only non-empty grades.
- Exclude one student from item 1, and give 100% to second one. The first student will get less points for category total.
- I believe it is unfair to him. I would expect him to get the same amount of total points as the second one, because I "excused" him.

In reply to Robert Russo

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Bob Puffer -

I'd take issue with your statement:

"If you'd like the final grade for each subcategory to reflect the actual number of points earned, use SUM for those subcategories."

What most people fail to understand is that with SWM the points ARE the weight.  You shouldn't have to use SUM in order for the weight to be accurately calculated in.  I believe I fully understand what takes place in SWM and it can be (IMO) accurately described (as can any other weighted method) as:

"weight" (in the case of SWM, "points") * percentage correct = contribution towards the container (category) total

In order to represent the container total as a percentage (or letter based on percentage) the total must be normalized to 100.  SWM with nested categories normalizes THE WEIGHT to 100 which forces the SWM selected method into becoming a Mean of grades.  My issue is that the gradebook forces a SWM into a Mean of Grades by not updating the "weight" of the container... leaving it at 100.  This is an error.  Plug ths numbers I've provided into a clean gradebook and see if you come up with something other than Mean of grades for the total unless you manually change the category totals (everytime a new grade is marked).

In reply to Bob Puffer

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Elena Ivanova -

Bob, we have tried to "aggregate with subcategories", but that also gave us a wrong total.

So my suggestion was your number 1 and 2 combined: replicate 1.8 behaviour and do things manually using SUM:

- Create 2 big categories, one for  graded items, another for non-yet-graded.
- Set course total to Weighted Mean. Assign 1 and 0 weights respectively.
- In the "graded category, created sub-categories, set them to  SUM aggregation.
- Once the new item from "non-graded" category is graded, move it manually to the needed sub-category in the "graded" area.

In reply to Elena Ivanova

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Bob Puffer -

Five years ago we had all our teachers trained to use two categories, "ungraded and graded".  They were so happy when we told them they could stop doing that.  And the point of all this is... they were able to stop because (IMO) the only correct solution to this problem is to accumulate points so that SWM aggregations are always based on the "Simple weight", i.e., Points.  The laegrader and laeuser reports do that.  This isn't a plug for using those reports, this is strong encouragment for changing the Moodle core gradebook to do likewise.  Then we can largely do away with the laegrader and the laeuser reports (if we can also incorporate frozen names AND frozen column headers into core).

In reply to Bob Puffer

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Brian Holt -

I'm looking for seom clarification on this.

I have 5 categories, each weighted a bit differently (5% here, 10% there) but they add up to 100%.

So the course is set to 'weighted mean', and within the individual categories are simple weighted means.

The problem is that moodle is calculating the total grade by the "percent of percentages" and not the "percent of points earned."  I know this because I was able to reproduce the scores using an openOffice spreadsheet.

 

So, for instance, when I calculate the score for one of the categories for a particular students, she earns a total of 2% towards her total grade.  The category is weighted 5% of overall grade, and she earned 40 points out of 70 in that category.  5% of 40 points is "2". That's how I get the 2% towards the course.   But the way moodle does it, she earns a total of 2.8%  40/70 = 57.1%  5% of 57.1 equals 2.8.

As far as I know, all of the assignments in all of the categories are based on a 'maximum grade'.

 

 

In reply to Brian Holt

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Bob Puffer -

Assuming a 1.95 or greater version of moodle:

  1. WM calculates a "contribution" (easier term to understand and translate) based on the assigned weight * the percent earned of the total earnable
  2. SWM calculates a "contribution" in precisely the same way except the "implied weight" of each grade item is the total points earnable.  Because the total of the "implied weights" rarely equals 100, the total earnable (weight) is scaled to 100 and likewise the percentage "earned" is scaled accordingly.
In reply to Bob Puffer

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Brian Holt -

I'm finding this thread enlightening.

http://moodle.org/mod/forum/discuss.php?d=117394

I've never seen such pushback to a total over 100%.   We can't measure 100% accurate in physics, sure as hell not in the classroom.  This might be a dealbreaker for me and moodle, will have to go back to Angel, the dark side.  Maybe I'll go back to letter grades.

In reply to Brian Holt

Re: Inaccurate Calculations in Simple Weighted Mean (SWM)

by Bob Puffer -

Yes, that discusssion was a lot of fun and provided an excellent example (there are several) of what happens when non-users take charge of the user needs and determine when something is needed.  The ultimate product would only work for a short time before it blew the grades up by 1,000%.  I don't know anyone who's courageous enough to turn on "grades over 100%" anymore.  There were a lot of core people involved in that and I wish a similar discussion could be held over the 2.x file system (but with better outcome).