- Asynchronous workflow - Students may progress through workshop at own pace
- Iterative grading - Student has choice to accept grade or revise and resubmit based on feedback
- Peer review evaluation - students review student reviews to improve review quality
I fully understand the use cases and the need for this. And I really like the concept of "pools" in your proposal. It's important to realize there is a trade-off for other current workshop features though. Firstly, the level of allocation randomness is significantly lower. And the randomness has been seen as an important prerequisite for an objective assessment. In other words, if there were three mates participating in the same workshop, they can make an agreement to submit their submissions at the same time, either very early or very late. Then there is a big chance that their submissions will be allocated within their close circle only. Second, there are edge cases that are not covered well in your proposal. Imagine a student who pops into the Workshop when the mainstream has finished it already. She finds her submission in the peer-review pool but she may never get an assessment as there is simply no other peer to come.
Believe me, I would really like to support less-formal scenarios of Workshop usage. They all can work well unless you require grades. If workshop was used just to get a feedback (via Comments strategy, for example), everything seems easy. But if students are to obtain a grade, and the grade depends on others' input, things become much more complicated. We need an interaction and the interaction requires synchronized / controlled process.
Thanks for your review, I appreciate your time. Here are my thoughts
"Randomness is lower" - I agree, though are there some situations where this is acceptable? I can think of two where this may be the case:
- The number of students is huge, they do not know each other, the stakes for the grades are not that high, and the level of asynchronicity is high. In this case I am imagining corporate education. The workshop could be part of a course on leadership training. Students (professionals) simply need to pass the course, the actual grade is irrelevant. Students could enter from any division within the company or may be from different companies and start the program at any time. There are no cohorts. In this situation, the opportunity and motivation for collusion are low.
- There is a significant penalty for "late" assignment. This is for a cohort environment. Perhaps it is "recommended" that an assignment be completed by a specific time so students don't fall behind. However, missing this date results in losing points. This keeps the majority of people turning it in at the same time. Any benefit from colluding to turn it in late are offset by the penalty. My experiences is that students that are not good students typically don't turn homework in early. The possibility for colluding is there still, but the benefits of allowing a more flexible submission schedule may outweigh the disadvantages
Perhaps another solution is to do the allocation in "batches", perhaps based on time or count. i.e., allocate only after x # of submissions so that the randomness is acceptable, or batch only once a week, so there is a literal pool available and not just the "next person in line".
"Imagine a student who pops into the Workshop when the mainstream has finished it already. She finds her submission in the peer-review pool but she may never get an assessment as there is simply no other peer to come."
I have put a lot of thought into this one, it is a serious issue. I have an OK solution and am excited to hear if anyone can do better. This is my thinking: The allocation method must be 3 reviewers per paper, rather than 3 papers per reviewer. This would mean that some people would not get all their papers to review at once. There would need to be some way to indicate that there is a new paper to review when one arrives. The system could then give a grade for the paper when all three papers have been graded, and give points to the grader when papers have been graded.
Papers put out for grading would need to be able to go "stale" and be reassigned after a period of time to ensure all papers got graded a minimum 3 of times. If papers are not scored 3 times after a period of time, a grade is computed on the existing scores. Another option could be to allow people to grade additional papers for extra points if they would like to if a paper does not successfully get enough reviewers.
The system could then give a grade for the paper when all three papers have been graded, and give points to the grader when papers have been graded.
Sorry, what do you mean by that? Would be Alice's submission graded somehow (how?) right after she assesses three others' submissions allocated to her?
be reassigned after a period of time
But that would also mean that the already calculated grade would change. I don't think that is a common use case.
I really appreciate your creative ideas and solutions. But I must admit that at the end, it all just sounds too heavy and complicated for me. Let us try to find (suboptimal if necessary) solution that would not add extra complexity to this (already complex enough) activity module. Thanks for the discussion!
Thanks for the feedback David, here is some clarification
"Would be Alice's submission graded somehow (how?) right after she assesses three others' submissions allocated to her?"
- no, When Jenney's paper has received 3 scores (from 3 different reviews), it's grade would immediately be computed, when Alice had reviewed a paper, she would immediately get credit for reviewing that paper
"But that would also mean that the already calculated grade would change"
- no, there is no calculated grade on the submitted document until it has been reviewed 3 times or a reasonable period of time has passed
"it all just sounds too heavy and complicated for me. Let us try to find (suboptimal if necessary) solution that would not add extra complexity"
I agree, I appreciate the dialogue. The more we beat up the ideas, the better ideas that survive
The end goal is to create an asynchronous workflow. Perhaps it would be useful to pull the problem out from the solution for a fresh look.
The fundamental difference between asynchronous and synchronous workflow is that triggers for workshop phase changes would need to follow the individual rather than cohorts.
These are the events I think we could tap into
- Individual document submission
- Individual review submission
To those events we need to attach these events
- distribution of document for review
- record review
- Give submission score
- Give review score
Cases I can think of that we need to handle include
- One or more of the reviewers don't do the review
Did I miss anything?
I actually think that the current design of the tool is opening up and becoming flexible enough to handle this, or at least heading in that direction, is that right David?
Well, yes. I hope so - if the time permits. I must admit that further Workshop development is not on my urgent TODO list. But yes, I have a plan to move from "whole workshop in a phase" concept to "each submission in a state" model. That is, it would not be Workshop that moves from phase to phase. It would be about submissions workflow (draft -> submitted -> peer-reviewed -> graded -> published eventually) and hopefully about revisions/iterations too.
Who knows, maybe the time will show that we can actually extend the Assignment module to support peer-reviews and the whole Wokshop would become sort of obsolete...
"Who knows, maybe the time will show that we can actually extend the Assignment module to support peer-reviews and the whole Wokshop would become sort of obsolete..."
I actually like that idea, this is how Canvas does it, except that it does it very poorly (not anonymous, no peer grading)
I hope this can move up in priority for Moodle. No one does peer review as well as Moodle, and I think peer review will become more and more important as MOOC technology improves. We need ways to grade massive amounts of work that are not limited to multiple choice questions.
I realize this is a bit bleeding edge right now but it won't always be and I would like to use it now and be on the cusp of the innovation, helping to guide it.
Thanks David- would love to see this happen (either the submission workflow, or the extension to the assignment module as an optional feature)
I would love to see asynchronous workflow in the workshop, and Jared's proposal looks good. I am currently faced with abandoning the workshop because of it's lack of flexibility in this regard. I am running a differentiated course and have students on different projects at different times. Switching back and forth between phases is prohibitively cumbersome. I need either peer assessment in the "Assignment" module or asynchronicity in the workshop. I'm considering moving to a Google Docs-based approach as my only alternative.
-Bill, teaching HS science in CT