Gradescope-type plugin

Gradescope-type plugin

by George Ionita -
Number of replies: 12
Hi there,
 
In the past 2 years at ETH Zürich we conducted online exams in Moodle using standard Moodle questions, STACK questions, and also classical open questions that students need to solve on paper.
 
One of the main issues lies with the grading of the open questions. The number of students is increasing, but the number of TAs remains constant.
 
One tool that caught our attention for handling paper-based exams is Gradescope. I know this can be integrated with Moodle, but because of legal issues and data protection, this is not a possibility for us. I also know that one can do manual grading for scanned exams via the Quiz or Assignment module, but this does not offer the functionalities of Gradescope.
 
Are there any plans for developing Moodle in this direction? Are there other universities which share this desire?
 
Thank you,
George Ionita
Average of ratings: -
In reply to George Ionita

Re: Gradescope-type plugin

by Joseph Thibault -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

George,

What features specifically from gradescope are you trying to replicate?

Average of ratings: -
In reply to Joseph Thibault

Re: Gradescope-type plugin

by George Ionita -
Joseph,

Some of the features are the following:
- scanning and uploading paper-based assignments;
- creating and modifying rubrics while grading, which allows for consistent grading across all submissions;
- annotate student submissions directly, providing feedback, corrections, or suggestions;
- multiple graders being able to work together on grading a single assignment to ensure uniformity by applying the same rubrics across graders.

George
Average of ratings: -
In reply to George Ionita

Re: Gradescope-type plugin

by Ewout ter Haar -
I just had a quick look at Gradescope, an edtech service that was bought by Turnitin a few years ago. Some features, like scanning and uploading paper-based assignments, seem difficult to implement and may be out of scope for Moodle. But some features looked interesting and may inspire a moodle developer:
  • the automatic grouping of submissions with similar answers, which then allows the teacher to give feedback to a few groups, for all students in the group, instead of re-using feedbacks for each individual student. This would make an interesting "manual feedback" extension for Quiz. 
  • the interactive creation of rubrics: the advanced grading methods of moodle require the teacher to define the rubric ahead of time. The gradescope demo showed the creation "on the fly" of a rubric while grading assignments. I thought this mimicked my own grading process very nicely, the criteria, feedback and points more or less come to you while you grade the first few students. This would be an interesting advanced grading plugin.

Average of ratings: -
In reply to Ewout ter Haar

Re: Gradescope-type plugin

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
"Are there any plans for developing Moodle in this direction?"
I feel confident that Moodle HQ will not be developing in that direction. (I don't in any way speak for them)
Average of ratings: -
In reply to Marcus Green

Re: Gradescope-type plugin

by Brett Dalton -
Picture of Moodle HQ Picture of Particularly helpful Moodlers

There are no plans to build the scanning parts of the Gradescope solution at this time.  That said there is some work being done that will longer term be beneficial

  • creating and modifying rubrics while grading, which allows for consistent grading across all submissions;

Rubrics will be reassessed at some point.  The grader work we are doing will improve the presentation of them but a reexamination of existing rubric workflows is being considered further down the track given they are a heavily used.  It would be great to understand more what the gaps here are compared to Gradescope.

  • annotate student submissions directly, providing feedback, corrections, or suggestions;

Improving our current pdf processing and annotation tools are definitely in scope for the current assignment improvement work.

  • multiple graders being able to work together on grading a single assignment to ensure uniformity by applying the same rubrics across grader

This is one I am keen to understand more.  There are several scenarios where multiple graders in assignments would be beneficial.  Moodle has some of this type of functionality but it's not well utilised, I suspect it is hard to use but I haven't dug into it yet.  Again if you have experience with the existing functionality it would be great to understand the gaps better.

Thanks.

Average of ratings:Useful (2)
In reply to Brett Dalton

Re: Gradescope-type plugin

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Brett, at Moot DACH in 2023, Luca Bösh and I spent some time digging into the Grading API and the question system, to see if it was possible to use rubric grading with essay questions in the quiz. We were forced to conclude that the way Rubrics in Moodle were designs (or, strictly, the 'Advanced grading API') is too limited so this is impossible. If anyone wants to understand the technicalities of why that is, please reach out to one of us.

(The fact that https://github.com/a-camacho/moodle-qt-rgessay exists does not really prove this is possible. It proves someone has done some extremely clever hackery around the limitations of the advanced grading system, but in a way that would not be apporopriate for Moodle core.)
Average of ratings:Useful (2)
In reply to Tim Hunt

Re: Gradescope-type plugin

by Brett Dalton -
Picture of Moodle HQ Picture of Particularly helpful Moodlers
Thanks Tim, thats good to know. I suspected it would be a non-trivial amount of work to consider changes in this space, hence knowing we would need to do a full assessement going in. When we get closer to looking at Rubrics I'll get the team to reach out.
Average of ratings: -
In reply to Brett Dalton

Re: Gradescope-type plugin

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Having worked extensively with Rubrics/Advanced grading methods over many years, it is something I have found hard to like, so re-work would be of greate interest.
Average of ratings: -
In reply to Brett Dalton

Re: Gradescope-type plugin

by Antonia Bonaccorso -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Thanks @Brett Dalton for insights on the plans of Moodle HQ on this matter. Highly appreciated.

I would like to comment on the use of rubrics: For them to be widely used in my opinion they must work for quiz questions, too. This is where teachers at our University ask for rubrics mostly.

Average of ratings: -
In reply to Antonia Bonaccorso

Re: Gradescope-type plugin

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Changes like this to the Quiz are quite large projects. Recent changes have been made possible by consortiums (consortia?) of Universities clubbing together to fund changes. 
Average of ratings:Useful (1)
In reply to Marcus Green

Re: Gradescope-type plugin

by Germán Valero -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Can you please provide some examples of these changes made by consortiums (consortia?) of Universities clubbing together to fund changes ?
Average of ratings: -
In reply to Germán Valero

Re: Gradescope-type plugin

by Antonia Bonaccorso -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Hi Germán

I know of two projects.

One is the integration of the Safe Exam Browser Deeper Integration that simplifies the configuration of Safe Exam Browser via Moodle quiz. You may find further details here: https://blogs.ethz.ch/letblog/2020/06/03/deeper-integration-moodle-and-safe-exam-browser-take-their-relationship-to-the-next-level/

The other one is the changes in the Moodle question bank implemented over the last four years starting with the release of Moodle 4.0. this project is still ongoing. You may follow the history of this project via the following tracker issues
- https://tracker.moodle.org/browse/MDL-70329
- https://tracker.moodle.org/browse/MDL-72752
- https://tracker.moodle.org/browse/MDL-76204
- https://tracker.moodle.org/browse/MDL-77627
- https://tracker.moodle.org/browse/MDL-78877
- https://tracker.moodle.org/browse/MDL-81401
and finally further ideas may be found here https://tracker.moodle.org/browse/MDL-75775

Best,
Antonia
Average of ratings:Useful (1)