Not all students allocated peer reviews

Not all students allocated peer reviews

by Thomas Lancaster -
Number of replies: 11

I'm currently using the Workshop module to allow students to assess each other's work for a simple assignment, for which the deadline for marking is tomorrow. The students have all been asked to review 5 documents, plus their own, out of 88 students who submitted work.

However four students have been to see me today. Three of them have only been allocated their own work to mark. Another has just been asked to mark one other assignment, plus his own.

I've checked a random sample by logging on as different students and at the moment those are the only four like that I can spot, however there may be more. Is there any reason why this is so, or could it be a bug? Also, is there any easy way to find out which submissions each student has been assigned to review? I have stated in the assignment specification that any student who does not complete all their reviews will not receive any marks. I had thought that this would be as simple as to check that they had all completed six reviews, although with this problem this will not be the case. This is quite a concern, so I'd appreciate any thoughts about what is going on. I asked three of the students when they submitted the work - they all said it was the day before the deadline, which could be coincidence or could mean something? I wonder if it could tally with the time when I moved the assignment from stage 2 to stage 3, say the 30 minute editing period, where some assignments are missed?

One unrelated question for next time I run this assignment - can the anonymous marking process be turned off. I need student to know whose work they are marking, since a proportion of the marks are based around students assessing each others contributions to the forums. At present we've been having problems with students who have not put their name in their submission.

Two other questions that would make administration much easier. Is there an easy way for me to upload a revised version of a student document, (say to add their name), other than hunting through the files to work out which it is and doing a manual replace? Also, is there an easy way to allow a student to edit a review after they have submitted it, or to edit it for them (say when a factual error has been found)?

Thanks,
Thomas

Average of ratings: -
In reply to Thomas Lancaster

Re: Not all students allocated peer reviews

by Ray Kingdon -
(Only a partial answer) Try upping the "Over Allocation" option to 1 or even 2. Normally the workshop gives out the submissions in a balanced way, so if each student has three submissions to mark, then each submission is only used three times. This sounds sensible but in practise can cause a bit of a bottleneck in giving work to students who want to mark work (having already given the work to students who don't want to mark work). Setting the Over Allocation to one or two frees up some "extra" submissions by allowing the workshop to give out a submission four (or five) times rather than the minimal three (in my example case here). In the case of a large group (as is yours) setting it to 1 should do the trick.

(The option is there because in some cases it's essential to have the submissions marked in a balanced way, or so I was told.)

Average of ratings: -
In reply to Ray Kingdon

Re: Not all students allocated peer reviews

by Gustav W Delius -

We have experienced the same problem that Thomas has reported: Not all students have been assigned the full number of assessments. That implies also that some pieces of work have not been assigned to the full number of assessors. Why not? The workshop has closed two days ago and is now in the assessment only stage. (This is with the new version of the workshop module).

Average of ratings: -
In reply to Gustav W Delius

Re: Not all students allocated peer reviews

by Ray Kingdon -
What's the Overallocation Value?
Average of ratings: -
In reply to Ray Kingdon

This forum post has been removed

The content of this forum post has been removed and can no longer be accessed.
In reply to Deleted user

Re: Not all students allocated peer reviews

by Ray Kingdon -
Richard, that's true. That was a kludge. In the latest versions of Workshop this option (and others) have been moved into the module's options. So in these versions, Overallocation is changed be simply clicking on the "Update this Workshop" button. Some changes are for the better...wink.
Ray
Average of ratings: -
In reply to Ray Kingdon

Re: Not all students allocated peer reviews

by Gustav W Delius -
The overallocation value was 0. I don't see why overallocation should be necessary. After all we are after the deadline.
Average of ratings: -
In reply to Gustav W Delius

Re: Not all students allocated peer reviews

by Ray Kingdon -
True upping the overalloaction value won't help now. But it would have helped during the assessment phase. If you don't want a strictly balanced allocation of work then setting this value to 1 gives the allocation process a bit of "slack" and eleviates the "Not all students allocated peer reviews" problem. smile
Average of ratings: -
In reply to Ray Kingdon

Re: Not all students allocated peer reviews

by Gustav W Delius -

I haven't made myself understood. Let me try again differently: I set up my workshop so that each student student should be asked to do 3 assessments. So I assumed that each piece of work would be sent out to three students in such a way that each student would get three pieces of work. That would work out perfectly. However for some unknow reason this did not happen. Some pieces of work were allocated to less than 3 markers and the necessary consequence of that is also that some markers did not get 3 pieces of work to mark.

Average of ratings: -
In reply to Gustav W Delius

Re: Not all students allocated peer reviews

by Ray Kingdon -
You're right, of course, you're right, I agree with you, I really agree with you...

However, if you sit down and actually do it the process kinda gums up. You find that one or two of the submissions you allocated to Tom and Dick should have gone to Harry as poor Harry only has two submissions to access as the "spare" submission is his own. And Harry's submission should have gone to Tom or Dick.

It probably could be avoided if the allocation process was not "sticky", once a submission has been allocated to Tom it stays allocated to Tom. There is a reason for that. In the early days the module used to do the allocation more on the fly and if the work was not assessed the student would see a new set of submissions to assess each time they went into the Workshop. Of course what I was told happened was that students were jumping in seeing if they recognised their mates' work, if not they would jump out and jump back in... Sort of cherry-picking the work they wanted to assess. Fendish these students you know evil.

I really suspect that a better default value for the Overallocation option would be 1 rather than the current 0 (given this "sticky" constraint).

Ray

Average of ratings: -
In reply to Ray Kingdon

Re: Not all students allocated peer reviews

by Gustav W Delius -

Ray, thank you very much for explaining that problem with allocation to me.

I usually don't need the phase where submissions and assessments happen at the same time. I am happy with allocating assessments after the submission due date has passed. In that case the problem with allocations could of course be avoided if the algorithm is clever enough. Is it?

Average of ratings: -
In reply to Gustav W Delius

Re: Not all students allocated peer reviews

by Ray Kingdon -
I wouldn't have thought so. It's works on a "give this student 3 (say) pieces of work to mark (if possible) NOW!" basis when the student goes into the workshop. Once those pieces of work have been allocated they stay allocated (for reasons explained above). There is obviously no way of knowing the order in which the students will visit the workshop. When the Overallocation level is zero, I think the potential for gumming up is always present because there is a possibility that the work from the last one or two students to visit the workshop might be amongst the remaining pieces of unallocated work. It's unlikely but nevertheless a real possibility. In that case the students would be allocated 2 pieces rather than 3 to mark. That would be avoided if the overallocation level is set to 1. Our "last students standing" would then end up with 3 pieces of work and one or two of the submissions would be marked 4 times and their work only twice.

A job for Better Algorithms 'R' Us then. Possibly but I suspect there will always be a degree of unbalance as not all the students will do their full allocation of assessments for all sorts of good and not so good reasons. The current algorithm is robust and it works OK within the limits discussed here and really I'm in no mind to change it.

Ray
Average of ratings: -