Adaptive/Dynamic Comparative Judgement in Moodle

Adaptive/Dynamic Comparative Judgement in Moodle

by Matt Bury -
Number of replies: 2
Picture of Plugin developers

Hi Moodlers,

I've been looking into a new strategy for grading students' work, especially longer-form writing, which tend to be time-consuming to do in predominant practices such as using marking schemas & rubrics.

Adaptive/Dynamic Comparative Judgement is a strategy by which graders are presented with students' submissions, 2 at a time, & then asked to rate which one is better. An algorithm controls which essays are paired for judgement so that they can be ranked in order of quality from highest to lowest. This brings a few advantages over schemas & rubrics:

  • Unlike schemas & rubrics, it isn't necessary to comprehensively define criteria & descriptors beforehand;
  • Since criteria & descriptors don't have to be strictly defined, graders are free to award higher marks for unexpectedly excellent work, i.e. that doesn't meet the criteria as expected but nonetheless gives an excellent answer, solution, response to the assignment;
  • If students practise comparative judgement themselves, it gives them a more concrete & comprehensible impression of what is expected of them in their work (unlike trying to "decode" criteria & descriptors);
  • It saves graders a lot of time;
  • It's at least as valid & reliable as schemas & rubrics;
  • It tends to produce fairer grading outcomes.

It's only in recent years that the necessary supporting IT & software for comparative judgement has become feasible for the wider education community, and it's starting to make an impact in some schools & institutions on a small scale.

Comparative judgement can also be used by students for learning, i.e. getting students to judge each others' submissions, so that they see many versions of an essay or answer & think analytically & critically about why one is better or worse than another, & also translate that to their own essays/answers in order to see how they could improve their own work.

I've searched the Plugins directory & I'm already aware of the Workshop module & similar plugins for peer-review submitting. However, I haven't found anything that closely matches comparative judgement: Despite the research evidence in its favour, it isn't commonly practised.

I believe that if comparative judgement were included in Moodle & more widely adopted, it could save graders a lot of time, produce fairer grading, allow for "unexpected excellence," & be a generally very useful & productive learning & teaching tool.

The Wikipedia entry on adaptive comparative judgement lists some prominent examples of it in action: https://en.wikipedia.org/wiki/Adaptive_comparative_judgement

What do you think? Would you/your teaching staff like to spend less time on grading & make it fairer? Would this be feasible at your school/institution? How well do you think it would be received by staff & students?

Thanks in advance,

Matt

P.S. Here's a bibliography list of research papers on comparative judgement that I've found so far:

Benton, T., & Elliott, G. (2016). The reliability of setting grade boundaries using comparative judgement. Research Papers in Education, 31(3), 352–376. https://doi.org/10.1080/02671522.2015.1027723
Bisson, M.-J., Gilmore, C., Inglis, M., & Jones, I. (2016). Measuring Conceptual Understanding Using Comparative Judgement. International Journal of Research in Undergraduate Mathematics Education, 2(2), 141–164. https://doi.org/10.1007/s40753-016-0024-3
Daal et al. - 2016 - Validity of comparative judgement to assess academ.pdf. (n.d.). Retrieved from http://www.tandfonline.com.libezproxy.open.ac.uk/doi/pdf/10.1080/0969594X.2016.1253542
Daal, T. van, Lesterhuis, M., Coertjens, L., Donche, V., & Maeyer, S. D. (2016). Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice, 0(0), 1–16. https://doi.org/10.1080/0969594X.2016.1253542
Hardy, J., Galloway, R., Rhind, S., McBride, K., Hughes, K., & Donnelly, R. (2016). Ask, answer, assess: Peer learning from student-generated content Judy Hardy, Ross Galloway, Susan Rhind, Karon McBride, Kirsty Hughes and Robyn Donnelly. Edinburgh, UK: University of Edinburgh. Retrieved from https://www.heacademy.ac.uk/system/files/ask_answer_assess_1.pdf
Jones, I., & Inglis, M. (2015). The problem of assessing problem solving: can comparative judgement help? Educational Studies in Mathematics, 89(3), 337–355. https://doi.org/10.1007/s10649-015-9607-1
Jones, I., Swan, M., & Pollitt, A. (2015). Assessing Mathematical Problem Solving Using Comparative Judgement. International Journal of Science & Mathematics Education, 13(1), 151–177. https://doi.org/10.1007/s10763-013-9497-6
Kolhe, P., Littman, M. L., & Isbell, C. L. (2016). Peer Reviewing Short Answers Using Comparative Judgement. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 241–244). New York, NY, USA: ACM. https://doi.org/10.1145/2876034.2893424
McMahon and Jones - 2015 - A comparative judgement approach to teacher assess.pdf. (n.d.). Retrieved from http://www.tandfonline.com.libezproxy.open.ac.uk/doi/pdf/10.1080/0969594X.2014.978839
McMahon, S., & Jones, I. (2015). A comparative judgement approach to teacher assessment. Assessment in Education: Principles, Policy & Practice, 22(3), 368–389. https://doi.org/10.1080/0969594X.2014.978839
Pollitt - 2012 - The method of Adaptive Comparative Judgement.pdf. (n.d.). Retrieved from http://www.tandfonline.com.libezproxy.open.ac.uk/doi/pdf/10.1080/0969594X.2012.665354
Pollitt, A. (2012a). Comparative judgement for assessment. International Journal of Technology & Design Education, 22(2), 157–170. https://doi.org/10.1007/s10798-011-9189-x
Pollitt, A. (2012b). The method of Adaptive Comparative Judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300. https://doi.org/10.1080/0969594X.2012.665354
Seery, N., Canty, D., & Phelan, P. (2012). The validity and value of peer assessment using adaptive comparative judgement in design driven practical education. International Journal of Technology & Design Education, 22(2), 205–226. https://doi.org/10.1007/s10798-011-9194-0
Tarricone and Newhouse - 2016 - Using comparative judgement and online technologie.pdf. (n.d.). Retrieved from http://educationaltechnologyjournal.springeropen.com/track/pdf/10.1186/s41239-016-0018-x?site=educationaltechnologyjournal.springeropen.com
Tarricone, P., & Newhouse, C. P. (2016). Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability. International Journal of Educational Technology in Higher Education, 13(1), 16. https://doi.org/10.1186/s41239-016-0018-x

Average of ratings: -
In reply to Matt Bury

Re: Adaptive/Dynamic Comparative Judgement in Moodle

by Gustav W Delius -

Hi Matt, I am surprised your post has not had more feedback. I only learned about comparative judgment as an approach to peer marking when I ran into Ian Jones yesterday. I find the idea very compelling. He suggested using https://www.nomoremarking.com/ . But being able to do it within Moodle would be so much nicer! Have you experimented more with comparative judgement since you made your post?

Average of ratings: Useful (1)
In reply to Gustav W Delius

Re: Adaptive/Dynamic Comparative Judgement in Moodle

by Matt Bury -
Picture of Plugin developers

Hi Gustav,

No, I haven't. It would require a lot of time and effort to develop and maintain plugin(s) that would effectively support Adaptive Comparative Judgement, i.e. it'd take some kind of on-going institutional support to make it happen.

Matt smile
Average of ratings: -