Automatic Free-Text Assessment on Moodle

Automatic Free-Text Assessment on Moodle

by Richard Klein -
Number of replies: 10

Hi All,

I have just finished honours in computer science at Wits University in South Africa. My thesis was on the automated assessment of free-text paragraph responses using statistical natural language processing techniques - similar to what some of the search engines use. It compares textual submissions based on hidden semantic structures that span all the submissions. I've implemented and tested such a system and we've got very promising results. If the system is configured properly we get correlations of between 0.8 and 1.0. These are very high - humans that mark the same papers generally correlate at around 0.8.

It is very accurate if configured properly - which is difficult -  the next step in our work will be to automatically generate configuration files based on features of the input. Once (if) that is done correctly, the system could prove to be a highly useful tool on Moodle.

The system is currently written in Python. We think that we'd like to keep it as a separate application that can be used anywhere as plugin. The system takes input from stdin, presents a small set of the inputted submissions to stdout for manual grading. When enough submissions have been manually marked to accurately represent the semantic structures that have been found, the system will automatically mark the remainder of the submissions and write a csv file with the awarded marks.

I administer Moodle in a number of schools in my area and would like to see this feature implemented.

Could someone please point me in the right direction to get started implementing an interface between the two systems and also if anyone is interested in being a part of such a project please let me know.

Regards,

Richard

Average of ratings: -
In reply to Richard Klein

Re: Automatic Free-Text Assessment on Moodle

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Richard,

Your venture sound very similar to the one described in this discussion. The person to help is Tim Hunt.

Automated Grading for Essay type questions

ATB

Joseph

PS.- I have taught for 44 years, been involved in Computer Assisted Language Learning for 25 years and cannot help but being highly skeptical of any claims of automated/automatic essay/text assessment/grading.wink

Average of ratings: Useful (2)
In reply to Joseph Rézeau

Re: Automatic Free-Text Assessment on Moodle

by Richard Klein -

Thanks I'll look into that section as soon as possible.

We're in the process of publishing a paper on what we have so far, when its done I'll post the reference to it.

I understand your skepticism, thats ok... I think that it'll take a long time for these kind of systems to prove themselves and I don't think that the problem is solved. The systems themselves could even just be used as moderators in the mean time if you dont trust it to actually award the final marks.

Bare in mind that our system is looking at content (no style assessment yet) so it would only ever be appropriate for technical submissions where content is the only important thing....

Our system is nice that once given some accuracy threshold, it will continue to ask the user to manually mark until it understands enough. In the worst case, it understands nothing and the the user ends up marking everything manually, what's important is that the system can tell when it doesn't understand. When it does mark, it marks accurately. *Assuming we have configured it properly... aka... when the next stage of research is done*

I suppose we can only wait and see... I think that ultimately, the problem is to a large degree solvable if given the correct training data...

Average of ratings: -
In reply to Richard Klein

Re: Automatic Free-Text Assessment on Moodle

by Itamar Tzadok -

Richard, the skeptical point is often missed. There are no doubts that the system can mark accurately when it marks. It is the manual grader from whom the system is supposed to learn how to mark that can never be trusted! smile

Average of ratings: Useful (2)
In reply to Itamar Tzadok

Re: Automatic Free-Text Assessment on Moodle

by Richard Klein -

Haha, yes that's true... I suppose that the alternative, however, is having the untrustworthy manual grader marking everything...  :P

This way you have him marking as little as possible - then the system consistently applies what he has done... this is at least much more detectable than if he just marks badly on a handful

Average of ratings: -
In reply to Richard Klein

Re: Automatic Free-Text Assessment on Moodle

by Joseph Thibault -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Richard, I'm course director for Straighterline.com and we would be very interested in being part of such a project.  Let me know anyway we can be involved.  We have already short answer type questions and the capacity to quickly create the answer keys and test in live student situations.

Sounds like a great opportunity.

-Joe

Average of ratings: -
In reply to Joseph Thibault

Re: Automatic Free-Text Assessment on Moodle

by Richard Klein -

Hi Joe,

Thanks for your interest. Your site seems cool.

At the moment we are still in very early stages. As I mentioned when the paper is published and presented I'll have some more time to work on this and I'll be able to share more details. Until the automatic configuration is complete, the system is difficult to use without large training sets. Feature detection for configuration will hopefully solve this problem.

At this stage I need to start looking into how the Moodle framework works and how to write plugins to use the program. I've got no experience with that at the moment.

How big are the classes that you generally deal with? small = 20-30 or large = 200+ or very ridiculously large?... The larger the classes the better the system works.

This system works by comparing the student submissions to one another, so ideally, for testing we need a large number of student submissions, with someone to mark a set of them that's selected by the automated system - we select the submissions that best represent the overall set (based on meaning) then get those marked and mark the rest based on those sets.

If you have people/skills/knowledge who are able to help with the actual plugin development, that would be great - then I could spend more time on developing the actual backend program. I dont really mind who does what, I'm just excited to get this working in some school environments where we can see how it performs in real life.

I'm still in discussions with the university on releasing the source code as GPL or something similar. I don't forsee this being an issue.

Average of ratings: -
In reply to Richard Klein

Re: Automatic Free-Text Assessment on Moodle

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

This does sound like an intersting project. Keep us informed in the quiz forum about how you get on.

Average of ratings: -
In reply to Tim Hunt

Re: Automatic Free-Text Assessment on Moodle

by Richard Klein -

Sorry about the lack of updates here - my msc is raping all my time :/ 

I'm still keen on getting this done - maybe sometime over the holidays

Average of ratings: -
In reply to Richard Klein

Re: Automatic Free-Text Assessment on Moodle

by John Stewart -

Dear Richard,

I'm working on very much the same problem, so it would be great to intersect.  Is your systems LSA-based?

Best,

jds

Average of ratings: -
In reply to John Stewart

Re: Automatic Free-Text Assessment on Moodle

by Richard Klein -

Hi John, 

Yes its based on LSA, but follows a different approach to the usual method of using corpus of knowledge. I find corresponding submissions based on differences in the LSA structure and then get the user to mark only a few representative samples - ie. outliers first, then representative samples from clusters of meaning. 

Here is the publication based on my work:

http://dl.acm.org/citation.cfm?id=1999793

Regards

Richard

 

 

 

 

Average of ratings: -