Workshop

Hi Peter, we have a faculty member who has published on peer review (Dr. Elizabeth Eschenbach: Web Based Forms for Design Team Peer Evaluations) and had a tool written years ago in java, which no longer functions on our servers.

We've started working on a way to provide a peer review tool, based on the one that Virginia Tech wrote as discussed here:
http://www.manastungare.com/publications/powell_2006_online.pdf

The VT tool has some nice features: the rubric interface works with the quiz question interface, it has a pretty straightforward interface, and is set up to allow reviewing of any activity or resource in the course (though this has to be implemented by extending a class in the code).

Anyway, below are my review and design notes, with Dr. Eschenbach's criteria in bullets:

Michael Penney
LMS Project Manager
3/6/2006

Virginia Tech Moodle Peer Review Module Report & Development Proposal

Last week I tested, debugged, and reviewed the Peer Review module from Virginia Tech. It had a few bugs that needed recoding and I made one of the interface changes suggested by the original authors.

Unfortunately it is built to work with a newer version of the assignment module than the one we are using on the current main HSU Moodle installation, so we will not be able to run it on the main installation until we upgrade the assignment module this summer.

I set it up on our test installation, and we could copy the course list and send the students over to the new installation if you would like to try using it for a current course. The first required change as outlined below would be to

Measured against Dr. Eschenbachs criteria, the module meets the criteria below:

Peer Review

• Faculty member builds a questionnaire for each student to complete.
• The questionnaire has text boxes and/or numerical responses for students to complete.

The current module allows rubrics to be created which each student can fill out when reviewing another students work. These can contain multiple choices, true false, and open-ended questions.

Students can also send a file back to the reviewed student, either with or in lieu of the rubric.

• Each student has to review a subset of students in the class, perhaps including self.

The instructor can assign a number of students to review each student's work. Peer review can be anonymous or not.

• All peer reviews of a single student's work are added together randomly and anonymously in a single document that is made accessible to the student after faculty review.

This requirement is only partly met: reviews can be anonymous, and are available to the reviewed students, however they are available immediately after being reviewed, there is not currently a way for faculty to approve a review before the student sees it. There are functions for sending out email notifications after a review has been graded, so we may be able to use these functions to withhold the reviews from students until after faculty review.

It does not meet these two requirements:

• If possible, simple stats of numerical scores are provided and the student can see self assessment separately from peer assessment

Students cannot self assess and there are no overall statistics.

• Faculty can review and edit inappropriate answers

This is not supported in the current version, reviews become available as soon as students complete them.

Required changes:
These are the required changes for this module to meet Dr. Eschenbach original request:

1) Make it self contained (remove dependencies on assignment) OR make it an assignment plug-in. Making it self contained will make it able to run independent of assignment versions, but more importantly it will simplify the process of creating and using a peer review (as it will not require an assignment be set up first).
2) Add optional ability for instructors to control posting of reviews.
3) Add optional ability for students to self assess.
4) Make module stats available in a single page, anonymous for students and optionally anonymous for instructors.

From the VT teams list of changes:

a) Provide better feedback to instructors for new reviews regarding review status; provide intra-page navigation links to reviews received.
b) Improve email functions
c) Better comment code.

Currently it is not clear which students have been reviewed. This should be part of step 2, to show instructors a list of reviews that have been submitted.

Backup and restore code (currently rubrics are not backed up).

If folks message me I can send them a demo login.