Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Bob Puffer གིས-
Number of replies: 35

Wanted to make this information available to those interested:

High side:

  • two large changes:
    • Grader report fashioned similar to LAE Grader where item headers and student info columns never leave the screen from UCLA (better than LAE Grader)
    • Natural weights allowing aggregation methods to be removed from the Setup (Cats and items) screen, display of natural weights allowing adjustment by teacher.
In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Thank you for providing a summary.

For those of us who don't want to go and read dozens of tracker issues, would it be possible to provide more of a readable narrative on what the proposed changes are.

https://moodle.org/mod/forum/discuss.php?d=262467 is an example of the kind of thing I have in mind. (Though, in an ideal world, there would be more pictures there.)

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Bob Puffer གིས-

Okay... I'm going to begin the process of showing all what Moodle intends to do with the gradebook for 2.8. Its a good thing I get to edit as long as I want in this forum because this post will stretch across many days.

There are two major projects being undertaken; redesign of the user interface for the Grader report and a thing called 'Natural weighting'.

Grader User Interface Redesign

This is a prototype of a redesign coming out of UCLA. It includes fixed headers and footers and numerous filtering mechanisms making it work well with very large classes. You will also note that there are 'Quickedit' links for each column and each row, using 'Quickedit' from LSU. It is not intended that editing would be done directly in this form. See http://docs.moodle.org/dev/Grader_report_2014 for more information on this project.

UCLA Prototype Redesign of Grader Report

Natural Weighting

This picture shows the Categories and items screen (soon to be renamed to the 'Setup' screen). 

Luther College/ UMN Natural WeightsNOTE: the absence of an aggregation drop-down. In short, natural weights derives the weights of the relationship between maximum points for items contained in the same container (category or course). This operates just like Sum of grades but is fixed to exclude ungraded and hidden items. Providing a means by which weights can be easily overridden fills out the remaining aggregation method requirements, just much easier and much more easily understood by the common teacher. The following is from this link 

http://docs.moodle.org/dev/Natural_weighting#Admin_settings 

which you can access for more information:

How natural weights meets the requirements of the aggregation types

Sum of grades
Moodle determines the "natural weights" for this by default based on point values.
Mean of grades
Set all weights equal to each other. Doesn't matter what the number is, as in legacy behavior Moodle will automatically adjust so the sum equals 100%. However, in this implementation the group wants to have a pop-up warning the user the weights don't add up to 100% and they will therefore be adjusted uniformly to equal 100%.
Weighted mean of grades
Adjust weights as desired. No change from current behavior.
Simple weighted mean of grades
Leave all weightings as is, no change necessary. Same as Sum of grades.
Mean of grades (with extra credits)
Extra credit has a natural weight that takes the total container (category/ course) weight above 100% (hence, 'extra'). Extra credit is available at all times no matter what scheme of weighting is chosen.
Highest 
User would input the value '1' for 'Keep highest' in the category
Lowest 
Of doubtful use. Not possible using natural weights, but it can also be done by a calculation into a new item.
Median / Mode of grades
Not possible using natural weights, but it's doubtful anyone uses these anyway so perhaps this can be deprecated entirely.

Less major changes:

Gradebook configuration settings and admin settings (initially marked 'critical by Martin D, Michael d marked it minor MDL-46117)
  1. Hide email in grader report without having it unavailable elsewhere
  1. Preconfigured setting packs
  2. Recover grades default should be set to 'Yes'
  3. UMN has added additional navigation
      1. Report item order
      2. Import order
      3. Export order
  4. Would like to add a sort order for scales
  5. Adding a toggle for view hide multiplicator and the offset (similar to toggle for keep lowest)
  6. Remove the aggregate including sub-cat (forever)
  7. Scales
    • Include scales in aggregation - do not make default, disable b/c assignments allow an integer vs. a pulldown menu that can be set at a max by admin
All of the gradebook default settings - at the admin level - should be forcible and hideable.
  • Similar to what is done for the site admin > Grades > Grade category settings - ex: Drop the lowest
  1. Re-organize the user report admin settings
  • Have a 2 column format 'available and selected' with add remove arrows, to determine which settings are on or off.
  • Each report needs a cog that takes you straight to those settings instead of having a settings tab with all report settings. In the case of reports that have both preferences and settings, these would be combined on a pop-up similar to the markups below.
Grader Report Preferences pop-upCategories and items settings
  1. Delineate between reports settings - let the user know who the settings effect. (ie. my preferences, and user settings)
  1. Dropdowns that contain 2 or 3 items should be radio buttons or a checkbox depending on the options. If a yes/no, checkbox
  2. Default values should be called 'checked' or 'unchecked' instead of Default:Yes or Default:No unless the values are 'Yes/No' (see attachment)
  3. Add the calculation setting to administrative settings giving them the ability to make it available or not available.
  4. Settings and My Preferences Tab -- Combine.
  5. Consistency among all admin settings

NOTE: Numbers off, Atto editor at work in a real world

User Report Enhancements

  • Extra credit clearly marked
  • Dropped/ kept clearly marked (category settings and which items are dropped/ kept)
  • Calculated weights column (not weights input that likely have little to do with the grade)
  • Contribution column which shows how much this item /category contributed to the container grade, helping the end-user easily see how the final grade was derived
User report enhancements
Enhancements to Import

University of California, Santa Barbara has greatly improved on the import/ export process (see diagrams):

Grade Import

Grade Import

Grade Import

Grade Import

Grade Import

... more to come

In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Let me start by saying that http://docs.moodle.org/dev/Grader_report_2014 looks great. (But good luck to whoever has to implement all that མིག་ཁྱབ་)


However, but http://docs.moodle.org/dev/Natural_weighting has me baffled, and a bit worried.

As a mathematician, the word 'natural' leads me to expect a certain thing, probably what Moodle currently calls Simple weighted mean. Anything else is not natural in that sense, so in what sense is the word being used?

Anyway rather that getting into a semantic argument about one word, perhaps it is better to just ask "What are you proposing to do here?" Is it not possible to give a two-paragraphs summary that is comprehensible? The wiki page is hugely long. I know it goes into a lot of deep implementation details, this is necessary, and fine, but only after you have provided a brief and comprehensible explanation of what you are intending to do.


Some comments on the detail of the spec, which may not mean anything since I don't get the big picture:

Database columns:

  1. grade_items.extracredit - since Moodle gradebook already handles extra credit, why is this new column necessary? If it is necessary, why is there not some old column that is being removed now that this exists.
  2. grade_items.weight - given that we already have a column called aggregationcoef, why do we need this?
  3. grade_items.weightoverride - how is this different from weight?
  4. grade_grades.weight - I can't for the life of my see why this column would be necessary.

Hidden items:

  1. I am worried about the sentence "Sum of Grades aggregation must be fixed to exclude hidden item". What is the effect of this change? Can you give some examples? I OU staff can see hidden grades, and want to see them included in aggregations, before students can see them.

Calculation examples:

  1.  simple mean - "User should adjust all weights to be the same, i.e., 25%" - why? Are you really saying that for something that Moodle currently handles automatically, now we will expect users to do a lot of manual work? What if you have a nastly number, like 17, of grade items? I really hope that I have misunderstood the spec.
  2. Similar comments where other aggregation methods seem to require manual work from users.
  3. What happens (to the weights) when a graded activity is added to or removed from the course in the middle of students sudying it?


P.S. but number MDL-46117 which you link to above has nothing to do with the gradebook. What is the right link?

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Aha! There was a better overview, but it was deleted: http://docs.moodle.org/dev/index.php?title=Natural_weighting&curid=5158&diff=45879&oldid=45788

I guess it may have been deleted because it contained some unwarranted assumptions, like "It is hardly a leap to conclude a setup screen that displays the natural weights for all items and categories with the ability to easily adjust those weights, if desired, would be great step forward in ease of use." - I have to disagree with that.

Also, what makes you think that 'aggregation' is incomprehensible to teachers, but 'grade weighting' is simple for all teachers to understand?

Wikipedian Protester

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Bob Puffer གིས-

Code for the Grader report has been supplied by UCLA. There are some tasks left beyond that code.

Toward the bottom you'll find the summation to which refer in your followup post. I thought it was good but as you say, shoulda left off the presumptive remarks. I'll try to respond to both your posts here.

  • The database columns for weight and extra credit are replacing aggregationcoef which is used for both, and I'm pretty certain you would agree is poor design. We offer extra credit wherever it is needed and therefore could't the same column representing both. You're right in suggesting we should eliminate aggregationcoef but cannot do that while we are still using legacy code where natural weights is not being used.
  • grade_grades->weight is required because weights for individual students are calculated differently based on what items have been graded for them or what might have been excluded for them
  • everybody at workshop agreed that Sum of weights is broken. Its doesn't respect the setting of hidetotalsiftheyincludehiddenitems, a setting by which you'd be allowed to accomplish your desired functionality setting it to SHOWREALTOTALSINCLUDINGHIDDENITEMS. Sum of grades doesn't respect aggregateonlynonemptygrades but purports to do so, though now with a warning. Sum of grades doesn't allow for extra credit which many instructors want
  • virtually nobody at the conference or at a followup user group had any aggregation methods turned on other than SWM, WM and Sum. The point here is transparency with much greater being accomplished by having the instructor do exactly what they want and get their expected result rather than understand what simple mean, means. I guarantee you, many do not.
  • My previous comment may apply to the need to do work to accomplish the other methods. 
  • If an instructor uses a terrible pedagogy like adding grades to a course in the middle the items come in with zero weight if all the other weights have been manually adjusted. If weights are still natural they would be recalced for that container. This was the consensus of Martin D, Michael D, Mark McKay and myself during a hangout this past Monday. Naturally, nothing has changed in the User report to prevent the (bad) teacher from only displaying points, hiding weights, etc. 
  • 2nd post
  • I can agree with you disagreeing with my presumptive remark about it being easier, though we had total consensus at the workshop and for a 'far-fetched' idea like this one, I think unanimity is quite an accomplishment.  I wish you had been able to attend.
  • I don't think that 'grade weighting' is easy for teachers to understand. Most have no idea about their strategy and how to accomplish it. I don't find anybody who disagrees with me that 'aggregation' is almost unintelligible for teachers outside Math and Computer Science but you could start the club. Between this spec and the 'contribution' column in the User report we hope to make it easier for teachers to see the effects of various actions and easier for students to understand how their final grade was arrived at. 

I'd like to continue the discussion, have others join. I've had to sell this idea to terrible skeptics and am willing to go overtime to provide anybody with explanations of difficult to understand concepts.

Sorry about the bad link, mdl-46177 (transposed).

===Background philosophy===

;The Moodle 1.95 was a giant step forward but it did some things fundamentally flawed. Those involving weights and grade category calculation (otherwise known as 'aggregation') are detailed below.

:- Running accurate points were not maintained for category and course totals 

::- It was assumed that the final 'Grade' (or points) total for categories would always be 100. If that were really the case then there would be no reason to have a percentage as these values would be exactly the same.

::- The only aggregation method that did maintain accurate point totals was Sum of grade. But it had substantial flaws (to this day). It did not exclude ungraded items (though it purported to do so) and it did not exclude hidden items, regardless of the setting for hidetotalsiftheycontainhiddenitems.

:- It made a gigantic leap of misplaced faith that even a small percentage of teachers would understand what aggregation means and even fewer would take the time to commit to memory what each aggregation method produced for results

;Grading isn't simple but grade weighting is. A teacher either designs their course with graded items in points-proportional value to other graded items or they choose to override those 'natural weights'.

;All of this distills down to two different methods, 'grades based on points' or 'grades based on adjusted weights'. 

;It is hardly a leap to conclude a setup screen that displays the natural weights for all items and categories with the ability to easily adjust those weights, if desired, would be great step forward in ease of use.

In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

Bob, hi

it appears that a lot of thought and hard work/time and effort has been spent on this, so many thanks to all concerned for sharing the detail thus far.

On that note, detail, there is a lot to wade through. The problem with this is that I am struggling to keep an eye on the big picture at the same time. It would be easy for me to say 'it all sounds fabulous'-and leave it at that, but I think there is value in having many voices involved with this, including the UK, as there are certain conventions, language and practices in place for marking/assessment and generating awards based on the aforementioned, across education systems.

I would like to caution at this juncture, any attempts to transplant one system, as a generic approach, into moodle. Of course, I am not saying this is the case-but the US system is quite different in many ways, including processes for assessment/language etc to that of the UK for example.   

I would also like to pick up on a point or two with regard to a series of assumptions in your post here:

'We offer extra credit wherever it is needed'

Is this the same case globally?

'The point here is transparency with much greater being accomplished by having the instructor do exactly what they want and get their expected result rather than understand what simple mean, means. I guarantee you, many do not'

How valid is this statement-on a global scale?

'I don't think that 'grade weighting' is easy for teachers to understand. Most have no idea about their strategy and how to accomplish it. I don't find anybody who disagrees with me that 'aggregation' is almost unintelligible for teachers outside Math and Computer Science but you could start the club'.

-I strongly disagree.

A suggestion: how about including a choice for teachers in terms of the format they would like to use? The lingo I know: means/aggregates/marks/awards.....never used points before except when considering the number of points students enter University with (UCAS points-but this has nothing to do with grading/gradebook matters).  As for the whole thing with natural weighting...I am still not getting it-so sorry about that.

Cheers,

Dawn


In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར
The meeting

I wish I had been able to attend, but I was not going to travel 1/3 of the way around the world for a 1-day meeting. We live in the 21st century, so you could have reached out to people you particularly wanted to hear from on Skype, Hang-out, Jabber, IRC, ... but you didn't.

Then, you took months to write up what you discussed so that anyone else knew. I completely empathise with being too busy, but you have to acknowledge that it is not good.

Now it seems lots of code is being written, before anyone has really commented. This is worrying.


Aggregation methods

I cannot defend the names currently given to different aggregation methods in Moodle. They are mostly completely incomprehensible without reading the docs. This is not good. But, that does not immediately mean that the only approach is to change to explicit weights. I am not going to comment on whether the average teacher can understand weighted sums. I have a PhD in maths. I cannot imaging not understanding. Has anyone consulted a lot of teachers of non-numerate disciplines about this?


DB columns

Weights for individual students may be calculated differently based on what items have been graded for them, but that does not necessarily mean that this dervied data should be stored in the database. I guess this is a performance trade-off, which you are probably best placed to judge if you are writing the code.

I hope that you are writing good comments for all the columns in your install.xml file. Please add those comments to the document, and please list the columns you plan to remove when everything is done.


What to people actually use now?

"virtually nobody at the conference or at a followup user group had any aggregation methods turned on other than SWM, WM and Sum." - why are we taking the memory of a small number of people who happened to attend certain meetings. Why not get real data?

When I had to make a decision about the future of a question type, I asked people to run some queries on their database to get some real evidence: https://moodle.org/mod/forum/discuss.php?d=175930 - the response was overwealming, and unforunately showed that my guess was wrong, and so I had to do the thing I did not want to do.

Can't you do the same here. Make some DB queries that show us what people acutally use in the gradebook, then ask people to run them and send you the results? I'm happy to run any such queries against the OU's databases.


Modifying a course in the middle

This is not terrible pedagogy. It is one of the principles that Moodle is founded on. http://docs.moodle.org/27/en/Pedagogy#Social_Constructionism_as_a_Referent Point 5. "A learning environment needs to be flexible and adaptable, so that it can quickly respond to the needs of the participants within it."

Just because your proposal does not handle something, do not just dismiss it as "Terrible pedagogy".

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Bob Puffer གིས-

Hi Tim, seems there's a lot of misunderstanding here so I hope my responses can help to sort things out.

The meeting

Actually the meeting ran two days and fortunately we had some folks from around the world in attendance. The logistics of pulling together the meeting quickly in physical form were so overwhelming that we chose to focus on preparation of materials so a lot of work was done before the meeting and didn't put an effort into providing distance participation. We did have one request from UMass and unfortunately had to turn it down.

Aggregations

I and many of the others in attendance had consulted large portions of their faculty and found tremendous similarity among non-mathematical professionals of finding the concept of 'aggregation' incomprehensible.

DB Columns

I don't think I can comment much on this because I'm not writing the code for this but providing a a working prototype. I, like you have to prioritize what gets my attention so I will certainly consider your suggestions if there is time.

What do people use

I think you misunderstood my statement here. This was from an administrative standpoint, other aggregation methods were not made available to instructors so querying the DB to find out what people use would provide the same information.

Modifying a course in the middle

I rescind my remark about terrible pedagogy but maintain that our system does handle it as well or better than the current gradebook. North American higher education is so driven by grades that the is very little room for a use a 'constructivist pedagogy'. Students don't tolerate surprises and appeal to the department head and/ or Dean frequently involving what they consider to be unfair grading practices.  

Martin has traveled around to the Moodlemoots and talked about this very change and has had very warm reception. I've not had a single person say this is an unwanted change, be it faculty or instructional designer. The experience of my colleagues from the gradebook workshop who meet bi-weekly regarding the gradebook changes has been expressly the same. 

Few final notes, I hope you noted that this is a 'phased' approach, we're not taking away legacy behavior until a critical mass of support can be established. Also, I certainly appreciate your PhD in Mathematics, wish I'd had one while working on all this stuff.


In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

Hi Bob-am not Tim དགའ་འཛུམ་  oh I suppose I need to declare I am an expert of sorts-shall stop there-tis different for a female.

Anyway. My comments:

RE:

Aggregations

I and many of the others in attendance had consulted large portions of their faculty and found tremendous similarity among non-mathematical professionals of finding the concept of 'aggregation' incomprehensible.

I THINK THIS IS NOT TRANSFERABLE ACROSS THE GLOBE.

Modifying a course in the middle

'I rescind my remark about terrible pedagogy but maintain that our system does handle it as well or better than the current gradebook. North American higher education is so driven by grades that the is very little room for a use a 'constructivist pedagogy'. Students don't tolerate surprises and appeal to the department head and/ or Dean frequently involving what they consider to be unfair grading practices.' 

THIS MAKES NO SENSE TO ME WHATSOEVER-APOLS, BUT I HAVE NO GIFT WRAP PAPER HERE.

'Martin has traveled around to the Moodlemoots and talked about this very change and has had very warm reception.'

MARTIN WILL ALWAYS PUT FORWARD IDEAS FOR INNOVATION ACROSS THE FORUMS-THAT IS WHY WE ARE HERE......OS-NATURE-OF-WELL-DONE PRACTICE!

cheers,

Dawn


In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

The meeting:

Is it really not possible to get a list of attendees at the meeting and their affiliations? Is that an unreasonable request?

Why is this meeting not listed at http://docs.moodle.org/dev/Developer_meetings ?

Aggregations:

Really it is better to talk about weights. Forget about the past. What evidence do you have that people will understand what you propose to do in the future?

What do people use:

The fact remains, you have a UK-centric view. I don't care whether policy is set by admins, or individual teachers. Let's get hard data from around the world. Work out some SQL queries to return aggregate data, and get some people at Moodle partners to run them across many sites. If you could get Remote Learner (USA + Canada + UK), Catalyst IT (NZ + Aus + UK), Puknui (Aus + Far East), +1 south-american partner, plus one other Asian partner, you have hard data from around the world.

MoodleMoots:

I was at the UK Moot as recently as April, including the developer day. Martin did not talk about gradebook weighting at all. He just said that something would happen to the gradebook. The first I knew about this was reading the spec when I got back from holiday on Monday, 6 weeks after the meeting, and days before work started.

Final notes:

No, I had not piecked up this was a phased approach until minutes ago, catching up on the recording of the developer meeting  . If you actually want to understand what is being proposed, your best bet is to watch that from about 1:30:00 to about 1:38:00.

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Martin Dougiamas གིས-
Core developers གི་པར Documentation writers གི་པར Moodle HQ གི་པར Particularly helpful Moodlers གི་པར Plugin developers གི་པར Testers གི་པར

The Gradebook working group is not a developer meeting.  It was a user meeting to brainstorm and collate features.  These are documented in MDL-44673 (specs are still being improved).

In reply to Martin Dougiamas

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

right-yes.

That data needs some scrutiny. Happy to be involved. I would like to suggest for the future if/when we have time...XY for some help and Tim-if you have time-for the hard sums....let us get down to it eh. And a pub too-some thinking for the mo- tis all.

D

In reply to dawn alderson

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Jason Hardin གིས-

As an FYI Moodlerooms did run the following query counting the number of courses using different aggregation methods across all of our clients. We used that data during the discussion on aggregation methods along with what each institution had from running the query on their own instances.


select aggregation, count(id) from mdl_grade_categories group by aggregation;-- need to parse the results based on the constants define('GRADE_AGGREGATE_MEAN', 0); //sum all grades in the category and divide by the number of grade items

define('GRADE_AGGREGATE_MEDIAN', 2);

define('GRADE_AGGREGATE_MIN', 4);

define('GRADE_AGGREGATE_MAX', 6);

define('GRADE_AGGREGATE_MODE', 8);

define('GRADE_AGGREGATE_WEIGHTED_MEAN', 10); // manually add weights to grade items

define('GRADE_AGGREGATE_WEIGHTED_MEAN2', 11); //simple mean - take the sum of points earned from all items in the category and divide by the total of all possible points in the category

define('GRADE_AGGREGATE_EXTRACREDIT_MEAN', 12); //mean with extra credit support

define('GRADE_AGGREGATE_SUM', 13);


We have clients in all of the markets that Tim mentioned, but our clients are mostly US higher ed, k12 and pro ed.  If it would be useful I can anonymize our data to share, I only mentioned the data in the meeting but did not share it beyond comments. 

I would recommend admins run that query on their own Moodle instances it would be great to see what the OU comes back with for aggregation method usage.

Remote learner also had multiple employees in attendance. 

In reply to Jason Hardin

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

That's great.

Probably more useful than the full data would be a summary of what it shows. Do you remember the interesting points?

If it would help, I could look through the anonymised data and write a summary. I am not sure that would add much to the debate at this point, so take me up on that offer or not, as you like.

I'll run the query on the OU database on Monday. I asked our most experienced gradebook admin to look at the spec, and he had no concerns for two reasons:

1. The final computation of course grades at the OU is done by our student record system, which combines grades from both Moodle and other sources like exams using some very complicated rules.

2. We do some calculations in Moodle sometimes, to prepared data to send to the SRS, but in that case, anything complicated is done with the calculator icon.

Regarding the proposed UI changed in the grader report, he thought those looked very promising.

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

J&T

running queries sounds like a plan-great!

So, some data analysis of sorts will now be considered-two greats-opening up a wider view of things!

I want to expand a bit on another comment I made...so will do so-higher up this thread-next.

D

In reply to Jason Hardin

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

OK, here is the OU data:

"Custom calculation";5
"Mean of grades";2152
"Simple weighted mean of grades";909
"Sum of grades";4

And here is a souped-up version of Jason's query that I used. It does two things. Coverts the aggregation int to a string, and also looks to see if there is a custom calculation, which would take precedence over the aggreation if present. (I hope I got that join right).

SELECT CASE WHEN gi.calculation IS NOT NULL THEN 'Custom calculation'
            WHEN gc.aggregation = 0 THEN 'Mean of grades'
            WHEN gc.aggregation = 2 THEN 'Median of grades'
            WHEN gc.aggregation = 4 THEN 'Min of grades'
            WHEN gc.aggregation = 6 THEN 'Max of grades'
            WHEN gc.aggregation = 8 THEN 'Mode of grades'
            WHEN gc.aggregation = 10 THEN 'Weighted mean of grades'
            WHEN gc.aggregation = 11 THEN 'Simple weighted mean of grades'
            WHEN gc.aggregation = 12 THEN 'Mean of grades (with extra credits)'
            WHEN gc.aggregation = 13 THEN 'Sum of grades'
            ELSE CAST(gc.aggregation AS CHAR(10)) END AS aggregationtype, count(gc.id)
FROM mdl_grade_categories gc
JOIN mdl_grade_items gi ON (gi.itemtype = 'category' AND gi.iteminstance = gc.id)
	OR (gi.itemtype = 'course' AND gc.depth = 1 AND gi.courseid = gc.courseid)
GROUP BY aggregationtype
ORDER BY aggregationtype

I was trying to understand whether the large number of 'Mean of grades' categories was singificant, and I don't think it is. As far as I can see, $CFG->grade_aggregation is not used anywhere in the code, which must be a bug.

So, when new grade categories are created, the default will be 0 = GRADE_AGGREGATE_MEAN, the first item in that dropdown menu, and also the default for that column in the database definition.

I cannot, at the moment, find where we create the top-level grade_category for a course when a new course is created.

UPDATE: Wow! not only is $CFG->grade_aggregation unused now, a quick run of

git log -S '->grade_aggregation'

shows that the only change since 2009 that is at all related to this is when Petr converted the unit tests from Simpletest to PHPUnit.

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Toni Soto གིས-

Tim,

I don't know if this can help but I ran your query in our small Moodle and it shows:


Custom calculation 2
Max of grades 1
Mean of grades 19
Simple weighted mean of grades 168
Sum of grades 3
Weighted mean of grades 5

Toni
In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Bob Puffer གིས-

Interesting, I never recall running across the variable, $CFG->grade_aggregation. Also interesting that none of those aggregation types used allow user-input weights. Since we've turned off all agg types except SWM, WM and Sum our categories default to WM.

In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

Hi,

Forgive me, but can I sum up? It will help me to understand and maybe others who are not clear, maybe དགའ་འཛུམ་ 

Some role-play, I know all this is spelled out for 2.8 adjustments-cool.

I am teacher A and we are using Moodle 2.3 across our three-form entry school

I am teacher B and we are using BB across our Further Education college, we are thinking about better gradebook options in our LMS and switching to Moodle.

I am teacher C and we are using Moodle 2.7 looking to upgrade to 2.8 when it is out.

All teachers use an electronic gradebook so that teachers can manage their marking/grades and student assessment in order to share details with the students and send the required detail to the main central admin system (Foxpro/SITS/etc).

So, Jason H stated -[although] ' We have clients in all of the markets that Tim mentioned, our clients are mostly US higher ed, k12 and pro ed.' Nevertheless according to Jason this command is in place meaning a focus on mean of grades:

select aggregation, count(id) from mdl_grade_categories group by aggregation;-- need to parse the results based on the constants define('GRADE_AGGREGATE_MEAN', 0); //sum all grades in the category and divide by the number of grade items

Tim's data= "Mean of grades";2152

And we had an update from Tim, which stated

'So, when new grade categories are created, the default will be 0 = GRADE_AGGREGATE_MEAN, the first item in that dropdown menu, and also the default for that column in the database definition.'

Toni also produced data which aligned with the mean of grades popularity and weighted mean of grades

Mean of grades 19
Simple weighted mean of grades 168



And weighted mean of grades is the default according to Bob in his setting- Bob: Since we've turned off all agg types except SWM, WM and Sum our categories default to WM.

I am guessing then that any new code to adjust/introduce for 2.8 will just be for 2.8.

This also suggests that institutional teachers/practitioners/admins/Techies will be able to adjust the settings or if need be the code to suit their purposes, if they wish to upgrade/move from BB to 2.8 i.e. if they do not want WM as the default ( I am of course, wrongly assuming this to be the case-based on Bob's setting-yes?).Because, those three teachers above may rely on mean of total grades alone...but can still have this option and of course the data here-suggests it to be a popular feature.

Or, are there wider implications when core code has been fiddled with...

Cheers,

Dawn

In reply to dawn alderson

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

Bob Puffer གིས-

I could be missing something but it seems the only way mean of grades can produce accurate final or category grades is if all items have the same number of points. If that is the case,  mean of grades acts just like SWM. The other possibility is that points are irrelevant in this scenario and the only thing the teacher is interested in is the mean of the percentages, which is what Mean of grades does. It seems hard to believe there isn't a desire frequently to give more weight to one grade over another, which cannot be accomplished through Mean of grades.

The default aggregation method will be a site-wide setting.

In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

Bob-

The other possibility is that points are irrelevant in this scenario and the only thing the teacher is interested in is the mean of the percentages, which is what Mean of grades does.

we are on the same page...


དགའ་འཛུམ་ དགའ་འཛུམ་ དགའ་འཛུམ་


In reply to dawn alderson

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA

dawn alderson གིས-

Here, I want to expand on the gradebook survey.

I understand XY has done some serious hard work with this-I could be reading this wrong-in terms of the person at HQ overseeing the survey-but I would like to think it was XY's hard work.  Now, there is data there-that not only shows potential for descriptive stats...as we can access to date, but it seems to me that there is an oppt to mine that data-approx 800 responses-I think....that involve not only choice answers, but also REASON FOR ANSWERS in the qual data.  I think, XY has done a great job-the figs are clear. I think there is potential also to consider correlations across questions (Tim-;)  and there is also an oppt to mine the data for qual support across such quant considerations........I think it is essential to value user voice.......

hope that, explains my thinking for now......list of to do-s as long as my arm at the mo-as I am sure you all have-but food for thought, perhaps.

cheers,.

D 

In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Martin Dougiamas གིས-
Core developers གི་པར Documentation writers གི་པར Moodle HQ གི་པར Particularly helpful Moodlers གི་པར Plugin developers གི་པར Testers གི་པར

For those worried about the larger changes I can reassure people that:

  • they are much better thought-out than the docs show in their current state
  • the bigger ones will be put up as prototypes on the prototype site for further feedback before they get into core.  It is very hard to really understand them without the prototypes, and so I think some of the discussion now is not helpful for anyone.
In reply to Martin Dougiamas

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Thanks Martin.

In case it is not clear (and I am sorry for not making this clearer sooner).

  • It is really, really great that something is now being done to tackle the long-standing issues in the gradebook.
  • I am really grateful for the people doing the work, and taking such care about it.
  • Ultimately, it is more important that good changes get into 2.8, than that anyone outside the working group understands what is going on now.
  • I would like to make a contrition to this that is helpful, if I can. (Though any contribution I make is likely to be tiny compared to what others are doing.)

To repeat what I said above, Martin gave a nice clear 8-minute summary of the proposals from about 1:30:00 to about 1:38:00 in

I think I now understand what is being proposed, and now that I do, most of the things I was worried about were unfounded. I look forwards to seeing the prototypes.

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

dawn alderson གིས-

I see, yes I did watch this video close to the event, and now it has a wider context and thus appears more meaningful to me...the focus in those 8 mins then appears to be on GUI of gradebook.

Very clear now about the choices that will be in place for teachers....that was not clear in the forum thread-across both of them and therefore contrasted somewhat with Martin's words on the vid-for me anyway.

One thing, Tim, I think you have made several valuable points across both threads and while you say I have reflected and think otherwise or whatever you stated in the last post...I still think there is a lot of value in your contributions.

So, two things remain for me, of which I would like to include here:

1. always a very good idea to have working groups-but do remember not everyone is truly represented in such groups-and certainly not all practices, as highlighted in the missing UK element across the working-group-which in turn has presented a series of questions and discussions.

2. RE:    MDL-46175.   I sincerely hope this will maintain a balanced view in its evolution-balance in terms of code-chats and innovation- with- the notion of being fit for purpose for many practitioners.

cheers,

Dawn


In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Bob Puffer གིས-

I added a section to the DevDoc for Natural Weights that walks through the functional logic of what's being proposed. Tim I would greatly appreciate if you could look it over closely because you have strong knowledge of the gradebook and a PhD in Mathematics. Its the logic I used to program the working prototype which will be released as soon as we can get it loaded to UMN's servers.

In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Well, I read it. Quite carefully. But only twice, which is not enough for me to understand it.

A particularly tricky issue is dropping lowest or highest with unequal weights. (I don't know how that is supposed to work with the current aggregation methods either.) If I am understanding correctly, items 4.1 & 5 under Functional calculations covers that. That seems to make sense, though I would not want to claim I fully understand it.

The bit I currently can't understand is the formula in Item 3.1 (which also related to Item 1). Sorry.

Apart from that, it all looks quite carefully thought out, and I can't see anything else obviously wrong.

Presumably a key part of all this is going to be to come up with a good set of test cases that can be turned into unit tests. The 4 calculation examples given in the document seem like a good start. Related to that will be structuring the code so that these complex calculations are as isolated from everything else as possible. Hopefully that will make it clear what inputs can affect the calculations and which cannot, and will also make it easier to test the calculations without needing lots of set-up.

Have you got a typo in Example 3? "place the value 20 on the Attendance category (or item) line." Should that be 10?

When I was writing the specification for the quiz statistics report (http://docs.moodle.org/dev/Quiz_statistics_calculations#Notation_used_in_the_calculations ) I made a section listing all the different numerical quantities, and trying to explain what they were, to try to make the following explanation easier to understand. I don't know if you think it is worth doing that for this spec too? (Also note that MoodleDocs has TeX installed if you want to type-set equations.)

I'll try to look again at some point, to see if I understand any more then.

In reply to Tim Hunt

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Bob Puffer གིས-

You've come upon the only issue we didn't have 100% consensus at the gradebook conference -- what happens when multiple drops are done in a category with unequal contributions. I would have sworn nobody could convince me it wasn't straight-up, lowest contributions but I  am now convinced and the general feeling was we should not allow multiple drops in categories with different contributions. Otherwise the calculation is very complex. The current gradebook algorithm doesn't handle dropping correctly in most cases, assumes all are worth the same and drops the lowest percentages. 

Formula in 3.1 takes into account adjusted weights and must remove those from consideration for the weight of the unadjusted items. Since each container has to have a weight of 100 that's why 100 - adjusted weights is used.

I think the idea of explaining the numerical quantities is good. Fortunately the gradebook is quite a bit less complex than the quiz. I'll work on that.

We did have a typo, which I corrected, thanks.



In reply to Bob Puffer

Re: Results of the 2014 Gradebook Workshop in Long Beach, California, USA - UPDATED

Aaron Volk གིས-

Its been asked on other discussions about having a function to email grades to a parent email, without having to enroll them as a mentee and have them check grades.

Any thought to adding this to the gradebook update?  I have no idea if this is a really complex thing or not, but most other gradebook programs have this function.