Allowing overlapping phases: discussion and suggestions

Allowing overlapping phases: discussion and suggestions

by Elizabeth Dalton -
Number of replies: 8

In this issue: MDL-27238 the suggestion is made to allow more flexibility of which phase students are in. David has suggested:

What is doable is to introduce a new phase covering both submission and assessment phase. I still resist to allow any on-the-fly calculation of the final grades because they rely on all submissions are available.

Still, there are many open questions that must be answered prior any actual development can start. Most notably:

  • How should submissions allocation behave? I can imagine a new allocator type ("instant" for example) that would automatically allocate the submission immediately after it is saved and marked as final by the student. Note this automatically enforces that a submission can be reviewed by a student that has not submitted anything yet (otherwise, the first submission could not get any reviewer).
  • How to improve the submission workflow and keep it still easy to understand for students. The author has to mark (tag) the submission as "ready for assessment". But then, maybe on the initial feedback from the reviewers, the student has to have an option to switch the status back to "work in progress" (when no other assessment is possible) and then submit it for assessment again. Do we expect that the submission is re-allocated in this case?
  • In other words, I believe it is possible to allow submission + assessment phase overlapping. But to calculate the grades, both submitting and assessing must be finished.
  • Most significantly, the allocation of submissions seems to be a tricky one:
    • When the first student submits their work, there is nothing else to be allocated to them for assessment.
    • When the second student submits their work, there is only one work to be allocated. Should it be allocated or not? Students might easily abuse this behaviour to "choose" their own reviewer, instead of having them assigned randomly.
    • As other students are submitting their work, the pool of submission is getting bigger and bigger and the allocation may seem to work. Still we have pretty unbalanced situation as early submissions are already assigned for assessments and hence are unlikely to be assessed by students who submit their own work later.
    • At the end of this time scale there is another problem emerging. The very last submission must be allocated to someone. But by that time those students may think that their task was already done. In other words, students never know if there is some submission to be allocated for them yet or not. Until the phase is closed, they never know the expected amount of work to do ("Mr Boerman, I already did all assigned peer-reviews last week. I did not know that a new submission can appear").
I will follow with a post with my suggestions. David asks that we work through these issues in this forum to make sure the needs of the community are taken into account.
Average of ratings: Useful (1)
In reply to Elizabeth Dalton

Re: Allowing overlapping phases: discussion and suggestions

by Elizabeth Dalton -

Here are some of my suggestions regarding David's questions:

  1. As David suggests, set a lower limit for starting the review allocation process. For example, require at least 3 submissions before allocating any reviewers. Prior to that point, display "Insufficient submissions to start allocation process" to students, instead of a grade. This deals with several points of concern that David raised.
  2. This limit should not affect self-assessment, which should take place immediately upon submission, with an opportunity for the student to update their own work before allocation to others. If students are going to self-assess, it should be done as quickly as possible so they can have the material fresh in their minds, and they should be able to correct errors before letting the process go further. I would be reluctant to allow the student to continue to revise between this point and the completion of the required number of peer reviews.
  3. An option to restrict allocation to those students who have been given a grade of at least n for that assignment would be helpful. The instructor would assess the first couple of students, then they would be able to assess others. This allows a "snowball" effect of more and more potential assessors. This addresses another of David's key points. Yes, later submitters will be allocated to other later submitters for assessment, but the assessors are validated in this process, so there should be less of an impact on the quality of assessment.
  4. Set a lower limit of number of peer assessments before displaying a grade. If there have not been enough peer reviews yet, display "grading incomplete" or similar message to the student, rather than a grade for the assignment. Obviously, this number could be set to 0 for teacher-only assessments or 1 if the student should be able to see partial grades.
  5. Set the number of required assessments per student, and display to each student the number of assessments they have completed whenever they check the Workshop (or even better, in the student view after the link to the workshop). Allow the instructor to set a due date after which no more student assessments will be allocated (the instructor will have to decide whether to assess stragglers at that point). This addresses David's last point.
  6. Finally, there is an ongoing discussion of how to handle resubmissions within the Assignment module [MDL-36804]. Ultimately, I think it would be helpful to merge Assignment and Workshop modules, but in the meantime, some kind of audit trail of submissions needs to be maintained, and possibly multiple rounds of resubmission and peer review should be supported in Workshop, with an instructor option to maintain reviewers in consistent groups (probably using the Groups feature) or re-allocating in each round of revisions. This is consistent with supporting good writing process by encouraging multiple drafts of major works with review and revision between each draft (i.e. what I usually think of as the "workshop" process.)
Average of ratings: Useful (3)
In reply to Elizabeth Dalton

Re: Allowing overlapping phases: discussion and suggestions

by Jeus Perez -

We are using a moodle course for handling submission and assessment of the degree dissertations in our Faculty. We are dealing with 450 dissertations per year and 300-400 tutors grading them (some of them reviewing 2-3 dissertations).

In Moodle 1.9 we used the activity "assignment" but  this implied that tutors should have teacher/editor permissions and they could have a look at all dissertations and assessments with no privacy for the student-tutor relation.

Later we used a "database" with impossible requirements for visualisation, so that tutors (a modified student role) could only see their own contribution to the database.

This year we are trying to use the workshop. The idea is that students submit their dissertations to the workshop. Tutors (a copy of the student role with another name) are allocated manually to review their corresponding students. And results will be easily obtained by giving a 100% value to the peer review scores (=tutors' grading). In such a situation, in which dissertation submissions are not synchronous, it would be interesting to have both phases -submission & assessment- opened at the same time.

A related problem is that you cannot allocate students-tutors unless students have made a submission. This is logical for automatic allocations, but for manual allocations it would not be necessary. It would be practical to make manual allocations even when the students have not made any submission (yet).

More flexibility among phases (allocation, submission, assessment), particularly in combination with manual allocation would be welcome.

Average of ratings: Useful (1)
In reply to Jeus Perez

Re: Allowing overlapping phases: discussion and suggestions

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

Hi Jeus

Thanks for the suggestions. Yes, I am aware of this issue. There is an idea floating back in my mind that might address this. The easiest way to enable allocation of submissions prior authors actually start on them, we could implement a "draft submissions generator". That is a code that simply creates empty submissions (so they get the id) that can be then allocated. Such a draft could be even be made from a template defined by the teacher (whereas submission templates is another feature I'd like to see in Workshop so we would hit two birds with one stone here).

Average of ratings: Useful (1)
In reply to David Mudrák

Re: Allowing overlapping phases: discussion and suggestions

by Miro Iliaš -

Dear David,

strict phases in the workshop are good idea. But as mentioned above, I would welcome some more flexibility in the workshop phases.

As an extension of the current workshop (or maybe of an assignment?) I would be glad to have so called reviewed workshop. Each student, who submits something, would be immediately assigned given number of anonymous referees with right to return submission back to the working (unfinished) version. Referees are other students in the same group, does not depend whether they have submitted their works or not. The student will have to do improvements in his work according to his reviewers and only after that "phase" the teacher would grade his work (and referees for their judgement as well).

The main idea lies in implementing referees suggestions into students works to make them better, and also in mutual learning from each other (referees learn from student, student is taught lesson from referees).

Average of ratings: -
In reply to Miro Iliaš

Re: Allowing overlapping phases: discussion and suggestions

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

Hi Miro

strict phases in the workshop are good idea

Well, they seemed to. The experience of real teachers using the Workshop in real classes says it does not work well all the time, as you say.

Yes, I am aware of the problem with this very long feedback loop in the Workshop module. To make the learning process effective, the feedback loop must be very short. The most valuable feedback is the immediate one that the student get minutes or even seconds after the work is submitted (as their brain is still tuned up for the task). This is not the case of the Workshop module unfortunately. I am gathering various suggestions like your one and thinking about some bigger improvements in the code, should the time and gods allow me to work on the module in a near future...

Average of ratings: Useful (1)
In reply to David Mudrák

Re: Allowing overlapping phases: discussion and suggestions

by Jared Chapman -

I have very strong feelings about this and have proposed different solutions from time to time. I am experimenting with a peer review tool called SWoRD out of University of Pittsburgh. They have a feature called soft deadlines. The way it works is you set the submission due date in the past. When a student submits a paper, the system then allows them to request a paper to review. There is an algorithm that selects the paper that has been waiting longest with the fewest number of reviewers. After one review is complete, the reviewer can then request more papers to review.

In my dream world, this is how workshop would work. Rather than being driven by dates, it could be driven by events. 

  1. Event one: paper submission
    1. Result: student can request papers to review (if none are available, or if there are too few in the system to prevent collusion or ensure anonymity, the user may need to come back later)
  2. Event two: the student has reviewed all required papers and their paper has been reviewed the required number of times
    1. Result: a grade is generated

Some possible ideas additional ideas include:

  1. Allow people to request additional papers to review for extra experience, and possibly extra points
    1. This could help ameliorate the problem of some students not doing their reviews
  2. Dates could still play a role as a soft "best if done by" deadline to help keep the class on roughly the same pace
    1. I would assign a small amount of bonus or penalty point for doing the assignment early or late, but let the students manage their own time.
  3. The interface would allow students to petition grades and request instructor comment

I imagine using this in large university classes where it would be too difficult to assign and grade numerous large writing assignments without peer review. In this system I get the best of both worlds. The students get quality writing assignments and detailed feedback from peers using a rubric and comments. If they are unhappy with their grade or would like an "expert" opinion on their writing, they have complete access to the instructor's feedback.

I use this on assignments where there are no real right answers, it is just graded on completeness and polish. I envision a day where I create a MOOC with thousands of students in it, all starting and finishing the class at different times, having a rich and engaging experience that would be impossible otherwise.

Average of ratings: Useful (2)