Automated Grading for Essay type questions

Automated Grading for Essay type questions

by Saiyam Kohli -
Number of replies: 10

Hi,

I was wondering if adding automated grading for Essay Type / Short answer questions has been considered. Also, is there any work being done in this regard?

Actually, I developed a system to grade English essays as part of a computer science course. For the past few months I have been thinking about improving the system and making it more usable. Today I came across moodle at slashdot.org and I think its a really great platform to promote learning and I think such a feature would be a good addition to moodle and so decided to inquire here and see what others feel about it.

How such a system would work:

When the teacher is setting up the quiz they would also provide some sample answers. Generally these answers would cover a range of scores. For example if a question has a maximum score of 10 points. The teacher will provide 4-5 sample answers: one answer for range 0-3, one for 4-5, one for 6-7, one for 8-9 and one for 10 points. These answers will then be used to grade the student response. Typically, the more sample answers the teacher provides the more accurate the system will be.

The system can also be setup to provide a range of score instead of an absolute score. This way the students get an instant feedback about their performance and the final grade will be given by the teacher when they grade the student response. This also ensures that the system does not result in giving incorrect grades to students.

A note: The system I developed was to grade GMAT essays on a scale of 0-6. So, I know it works pretty well for long Essay type questions. I have not tested it for shorter answers. But I am confidant that it can be extended to work with shorter answers as well. Please note that such a system is already used by ETS to grade GMAT essay responses.

Please let me know if this is something the moodle community would be interested in.

Regards,

Saiyam

Average of ratings: -
In reply to Saiyam Kohli

Re: Automated Grading for Essay type questions

by Joseph Thibault -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Saiyam, for my company I would be interested.  We've evaluated the ETS solution (which actually can be used for essays beyond the GMAT question types) but have not had any successful implementation.  Testing has provided us "ok" results but additional QA is definitely needed.  Their system requires 100s of essays front loaded in order to create a quality automation process.

So, I would be interested in whatever project you have, even for simple feedback (for early drafts) as it could provide very useful information to students early on without teacher/facilitator interaction.

Moodle or other LMS integration is obviously a plus.

Average of ratings: -
In reply to Joseph Thibault

Re: Automated Grading for Essay type questions

by Itamar Tzadok -

Joseph, what do you mean by "actually can be used for essays beyond the GMAT question types"? Do you think that such a system can be used for evaluating, say, university level research papers? smile

Average of ratings: -
In reply to Itamar Tzadok

Re: Automated Grading for Essay type questions

by Saiyam Kohli -

Edit: I am sorry but it seems I was mistaken. On further reading it seems they have predefined list of prompts. Its mentioned in there catalog.

http://www.ets.org/criterion/higher_ed/about

Yes, I think it can be used to evaluate 'university level research papers'; Here is a link to their service page:

Regards,

Saiyam

Average of ratings: -
In reply to Saiyam Kohli

Re: Automated Grading for Essay type questions

by Itamar Tzadok -

Thanks for that. In one of Tim Hunt's post there should be a link to an article discussing an automated system for evaluating short answers that is developed and used at the OU. Search and you may find it interesting. smile

Average of ratings: -
In reply to Itamar Tzadok

Re: Automated Grading for Essay type questions

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Here's the reference to the article mentioned by Tim Hunt.

Butcher, Philip G. and Jordan, Sally E. (2010). "A comparison of human and computer marking of short free-text student responses". Computers and Education, 55(2), pp. 489–499.

Download submitted article full text from The Open University’s repository of research publications and other research outputs here.

Average of ratings: -
In reply to Joseph Thibault

Re: Automated Grading for Essay type questions

by Saiyam Kohli -

Joseph,

Any such system would require some graded answers to begin with. The more graded inputs it has to work with the better it will perform. That is why initially it would be better if the system just generates a range of scores and then refines the scores when more essays are available.

Also, Please keep in mind that the solution I worked on would certainly be inferior in quality to ETS as they do this day in day out and have very intelligent people working towards solving this problem, where as I developed this system over a couple of months as part of a class project. However it would be free smile

Regards,

Saiyam

Average of ratings: -
In reply to Saiyam Kohli

Re: Automated Grading for Essay type questions

by Doug Loomer -
Just yesterday I was talking to one of our teachers who said she would start using Moodle when it could auto-grade her essay questions. This would be a really, really great addition to Moodle.
Average of ratings: -
In reply to Doug Loomer

Re: Automated Grading for Essay type questions

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Douglas,

I do hope that a teacher wanting to use Moodle for her teaching would have many other - more pedagogically sound - motivations than getting a system to auto-grade her essay questions.wink

Joseph R.

Average of ratings: -
In reply to Joseph Rézeau

Re: Automated Grading for Essay type questions

by Doug Loomer -

I would hope so too, but you can't make some people eat ice cream.

Average of ratings: -
In reply to Saiyam Kohli

Re: Automated Grading for Essay type questions

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Saiyam, question types in Moodle are plug-ins, so you should be able to make a new question type, based on the essay question type, that does what you want. There is a tutorial here: Development:Question_type_plugin_how_to but it is not very good. The quiz forum http://moodle.org/mod/forum/view.php?id=737 is a good place to ask for help.

Joseph has already posted the link to Phil and Sally's paper. The PMatch algorithm described in that paper is now described online at The_OU_PMatch_algorithm. The reason it is described there is that we are going to be developing a question type using that over the next few months. You will see that PMatch is a fairly simple pattern matching system, though different from regular expressions. PMatch seems to work quite well up to responses of about 20 words. So a question type using your algorithm would also be worthwhile, as would a regular expression question type.

Average of ratings: -