Attached, please find description of a method how the new Workshop 2.0 will randomly allocate submissions for peer-assessment. This is a complete rewrite of the current code with some new features.
If you find a minute, please go through the slides and check the algorithm. Don't worry - you don't need to be a programmer to understand how the method works.
Any feedback, comments, ideas or eventual contra-proposals are welcome. Let me remind you that the described method is just one of at least two planned allocation method to be shipped with Workshop (random and manual).
TIA
d.
The random allocation algorithm to be used in Moodle 2.0 Workshop module
by David Mudrák -
Number of replies: 7
In reply to David Mudrák
Re: The random allocation algorithm to be used in Moodle 2.0 Workshop module
Thanks for your work, it looks very powerful.
Comparing this worshop version with the previous, I realize something very different: for to be REVIEWER is not necessary to be AUTHOR? Because, in the current version, you can't review works, if you don't have any submision (you don't play...).
I think that this is a good step forward.
And comparing too, I can see that a group can work as an 'individual' REVIEWER, but at last, only a person of this grup is the 'real' REVIEWER. My question is: when a group is a REVIEWER, all this group will see the same work? Because, if the work is only assigned at a single person of this group, it's not easy that the revision and evaluation will be as the result of the analisys of all this group.
Seeing your slide 15, if the first red square can't see the black's work, if this work is only assigned at the second square (here in black), I can't understand how this revision can be considered a revision from the group workload red...
Sorry if my explanation is not correct enough, and thanks another time.
Jordi
Comparing this worshop version with the previous, I realize something very different: for to be REVIEWER is not necessary to be AUTHOR? Because, in the current version, you can't review works, if you don't have any submision (you don't play...).
I think that this is a good step forward.
And comparing too, I can see that a group can work as an 'individual' REVIEWER, but at last, only a person of this grup is the 'real' REVIEWER. My question is: when a group is a REVIEWER, all this group will see the same work? Because, if the work is only assigned at a single person of this group, it's not easy that the revision and evaluation will be as the result of the analisys of all this group.
Seeing your slide 15, if the first red square can't see the black's work, if this work is only assigned at the second square (here in black), I can't understand how this revision can be considered a revision from the group workload red...
Sorry if my explanation is not correct enough, and thanks another time.
Jordi
In reply to Jordi Voltas i Aguilar
Re: The random allocation algorithm to be used in Moodle 2.0 Workshop module
by David Mudrák -
> REVIEWER is not necessary to be AUTHOR
Correct. This is new in Workshop 2.0. There are two possible ways how users can become reviewers without their own submission:
No, I am afraid you did not get it correctly. Even in visible groups mode, every reviewer acts as individual. The group workload is just an hypothetical factor that helps the Workshop to allocate submissions. Image group as a colour of the square, nothing more, nothing less. In Separate groups mode, only shapes with the same colour may be connected. In Visible groups mode, the allocator tries to pick squares from all groups in a balanced way.
The main goal of the whole procedure is to prevent situations, where (thanks to the blind Randomness) one reviewer has 6 submissions to review while his peer has nothing to do. The students would not like such rules
Thank you very much for considering this.
d.
Correct. This is new in Workshop 2.0. There are two possible ways how users can become reviewers without their own submission:
- Either they simply have not managed to submit their work in time. When the submission period is over, there may be (and very often are) some student without submission (maybe they were ill or whatever). The teacher can decide if such students can participate in peer-assessment period. They can't receive no grade for submission but still they can get a grade for the quality of their assessment.
- Or it is the teacher's intention. Teacher can use capabilities to say which users will submit and which will assess. They can but do not need to be the same users. As I have posted recently - I can imagine an online course with 100+ students and with a dozen of tutors (non-editing teachers). Workshop can assign student's submissions to be reviewed and assessed by tutors, allocating submissions randomly, in balanced way etc. So, the "peer-assessment" is technically a special case where the sets of authors and reviewers overlap.
No, I am afraid you did not get it correctly. Even in visible groups mode, every reviewer acts as individual. The group workload is just an hypothetical factor that helps the Workshop to allocate submissions. Image group as a colour of the square, nothing more, nothing less. In Separate groups mode, only shapes with the same colour may be connected. In Visible groups mode, the allocator tries to pick squares from all groups in a balanced way.
The main goal of the whole procedure is to prevent situations, where (thanks to the blind Randomness) one reviewer has 6 submissions to review while his peer has nothing to do. The students would not like such rules
Thank you very much for considering this.
d.
In reply to David Mudrák
Re: The random allocation algorithm to be used in Moodle 2.0 Workshop module
by Stephan Rinke -
Hi David,
thanks for this. It is always enlightening to see how much thought has to come into the development of moodle activities.
I have gone through your slides and now feel I understand the allocation process. I have not found anything that does not make sense or that would not allow me to use the workshop in the way I have been using it in the past.
Thanks again for all your work. I'm really excited about the Workshop 2.0!
Stephan
thanks for this. It is always enlightening to see how much thought has to come into the development of moodle activities.
I have gone through your slides and now feel I understand the allocation process. I have not found anything that does not make sense or that would not allow me to use the workshop in the way I have been using it in the past.
Thanks again for all your work. I'm really excited about the Workshop 2.0!
Stephan
In reply to Stephan Rinke
Re: The random allocation algorithm to be used in Moodle 2.0 Workshop module
by Mary Cooch -
I appreciate and understand your explanation too and am also grateful to you for all your work on this.
In reply to David Mudrák
Prevent visible groups from "self-assessment" in random allocation
by Gunther Kreuzberger -
Hopefully, this thread is not already closed. So, firstly let me thank for the clarifying slides and al the work done on the random allocation algorithm.
Having seen your slide I still have one question. would it be possible to prevent groups from somehow "self-assessment", i.e. would it be possible to organise the random allocation in a way that submissions from members of one group can not reviewed by members of the same group but receive reviews by members of the other groups?
Regards, G. Kreuzberger
In reply to Gunther Kreuzberger
Re: Prevent visible groups from "self-assessment" in random allocation
by David Mudrák -
Hi Gunther,
that is a very interesting idea indeed! I can see a pedagogical benefit of such behaviour. May I ask you to record a new feature request in Moodle Tracker (component Workshop, me as the assignee) for this? I am going to work on the allocator to fix issues reported by Morgan Harris (MDL-25523 and MDL-25526) so I could add this option, too.
TIA
that is a very interesting idea indeed! I can see a pedagogical benefit of such behaviour. May I ask you to record a new feature request in Moodle Tracker (component Workshop, me as the assignee) for this? I am going to work on the allocator to fix issues reported by Morgan Harris (MDL-25523 and MDL-25526) so I could add this option, too.
TIA
In reply to David Mudrák
Re: Prevent visible groups from "self-assessment" in random allocation
by David Mudrák -
Hi Gunther,
I just implemented this as a new allocation option "Prevent reviews by peers from the same group" displayed at the allocation form if the workshop is in the visible groups mode. I am going to submit this patch into the next weekly build. See MDL-25982 for the progress. Thanks for the idea!
I just implemented this as a new allocation option "Prevent reviews by peers from the same group" displayed at the allocation form if the workshop is in the visible groups mode. I am going to submit this patch into the next weekly build. See MDL-25982 for the progress. Thanks for the idea!