Ability to rate feedback effectiveness

Ability to rate feedback effectiveness

by Meg Goodine -
Number of replies: 6

We would like to extend the ability of the workshop module to provide an additional phase to allow learners to rate the effort or usefulness of the feedback provided by their peers, and have this rating contribute to each reviewer's grade.

Can any workshop experts out there advise how difficult this would be to customize?

Thanks for any ideas.

Meg

Average of ratings: -
In reply to Meg Goodine

Re: Ability to rate feedback effectiveness

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

There is a plan that reviews can be commented by authors (using the standard comments feature available at other places in Moodle). But this will not affect grading scheme.

WRT your question, it's really hard to answer. It's definitely not a trivial customization. You would also have to specify in details how the "rating contribute to each reviewer's grade" actually behaves.

Average of ratings: -
In reply to David Mudrák

Re: Ability to rate feedback effectiveness

by Meg Goodine -

Thanks for your reply. Here is our scenario: students are assigned 3 peers to assess (anonymously- which can be done now) and then rate the effort or helpfulness of the feedback they received. The grade for the assignment is based on the evaluation of your work and the quality of the feedback you provide to others. So the ratings an individual receives on the effectiveness of the feedback would make up part of the grade. Does that make sense?

Meg

Average of ratings: -
In reply to Meg Goodine

Re: Ability to rate feedback effectiveness

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

Thanks Meg. Indeed, the concept itself makes perfect sense. But still I can't see how exactly the rating would affect the received grade for assessment.

Let us imagine the following scenario:

  • Alice submits her essay.
  • The essay is allocated to Bob, Cindy and Dan for assessment.
  • All three reviewers fill the assessment form (say a rubric)
  • The teacher switches the workshop into the grading evaluation phase and (which would have to be added to the current workshop code), Alice has a chance to see the reviews and rate them.
  • Now, let us say that Alice fully accepts Bob's assessment, partially agrees with Cindy's assessment and completely disagrees with Dan's review. How would she express this? Using a scale? Or percentage "acceptance" level?
  • And then, how exactly would this "acceptance" influence grades that Bob, Cindy and Dan get for the assessment? Let us imagine that the teacher uses the "Comparison with the best assessment" evaluator. That would calculate grades, for example: Bob 100%, Cindy 100% and Dan 42%. Now, how these grades should get amended? I am asking for some formula or exact description.

It sounds like an interesting feature, really. But we need it specified into a deeper details.

Average of ratings: Useful (1)
In reply to David Mudrák

Re: Ability to rate feedback effectiveness

by Meg Goodine -

Thanks David.

The model we are trying to support here uses a rubric and a 5 point scale for both the peer assessment of submitted work (assessment 1- 50% of the grade) and assessment of feedback ( assessment 2- 50% of the grade)

Currently this translates 0-5 into a percent when displaying a final mark.
0 = 0%
1 = 20%
2 = 40%
3 = 60%
4 = 80%
5 = 100%

So in your scenario above, Bob would receive 100% from Alice, Cindy would receive maybe 40% from Alice, and Dan would receive 0. These values would have to be calculated back to form part of the Bob, Cindy, and Dan's  assessment 2.

If feels so "almost there" but maybe I'm just an optimist. Thanks for any further thoughts. Really appreciate your assistance.

Meg

Average of ratings: -
In reply to Meg Goodine

Re: Ability to rate feedback effectiveness

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
Right. So it's really like a task for a custom Workshop grading evaluation subplugin. Unfortunately, the Workshop architecture does not provide functionality for these subplugins to interact with students (that is to let them fill the "feedback" assessment form) at the moment :-S And I can't promise you I would work on it anytime soon as currently I'm allocated to other projects in Moodle HQ. I'll be happy discussing the suggested coding approach with anybody who would want to catch this opportunity.
Average of ratings: -
In reply to David Mudrák

Re: Ability to rate feedback effectiveness

by Meg Goodine -

Thanks, David. We may be able to fund some of this work so if there is interest and someone can provide an estimate, I'd be happy to take it forward from my end.

Meg

Average of ratings: -