Wanted: a tool to automatically assess submitted code

Wanted: a tool to automatically assess submitted code

by Martin Dougiamas -
Number of replies: 6
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
I have a customer who is looking for a new tool to help with assessing C++ code from students - this is what it needs to do:
  • Teacher hands out sample code (one file) with errors in it, as well as a (hidden) correct example.  The errors are fairly small and simple.
  • Student finds errors in some of the lines
  • Student submits corrected code
  • Moodle compares submitted code line-by-line against the correct example and gives the student a grade.

Now, believe me, I know how difficult this could become.  Ideally, one would use a complete C++ parser to somehow compare the code on a functional level and avoid differences that might be caused by comments or different formatting, but the customer believes the changes will be small enough that text matching could be used.  This is good because I had a search and couldn't find any sort of C++ parser that would be suitable.

This could be implemented as a new module, as an assignment type, or even a new quiz question type.  With my Moodle Manager hat on I would prefer to see it implemented as some sort of standard and generic solution for any fixed-type code or text but this is not essential.

Personally, I don't have time to work on this particular job, or a staff programmer (yet!) so I'm really advertising here for people who might be interested in doing this work.  If you have some ideas about this sort of thing, feel free to post here, but if you're really interested in doing this work please contact me privately and I'll put you in touch with the customer to talk about costs etc.
Average of ratings: -
In reply to Martin Dougiamas

Re: Wanted: a tool to automatically assess submitted code

by John Lacey -
yes, if the errors are kept strictly line-by-line it makes things somewhat easier...the customer could also upload a code example file with line numbers and ask a series of multiple choice quiz questions or simple fill in the blank that would be quicker to implement.
Average of ratings: -
In reply to John Lacey

Re: Wanted: a tool to automatically assess submitted code

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

I have given a little thought to this with reference to allowing people to run/test/evaluate Java code. Assuming the compiler puts out decent error messages it ought to be possible to analyse the messages on the server and wrap it in HTML to give feedback. I was concerned that it could incur security errors, but if it only compiles it rather than runs it, perhaps  it is not such a problem.

Average of ratings: -
In reply to Marcus Green

Re: Wanted: a tool to automatically assess submitted code

by Philip Norton -

At university I remember having to write programs that had specific outputs which were computer assessed (with 500 students in a course, it is not exactly easy to do this manually!).

We wrote JAVA files that were all in a specific directory structure, these were then all compiled, executed and the output piped to a specific file. If the output matched, you got 100%. If not, sorry for you. There were a number of different alternatives...

Philip

Average of ratings: -
In reply to Philip Norton

Re: Wanted: a tool to automatically assess submitted code

by Ray Kingdon -
Locally we use Coursemarker. It's designed to automatically mark (smallish) programming tasks. It uses a compiler to produce the output from the student's code. (We use it for first year Java but it's language independent.) Runs under Windows or Linux. We ran under Windows last year, it was OK but required a bit of "nursing". It's being run under Linux this coming Academic Year and hopefully it will be a bit more "lights-out". I think the problem was endless loops in student code which Windows didn't (or couldn't) stop.

The Nottingham people may be interested in doing some integration work. They're certainly actively developing/supporting the product.
Average of ratings: -
In reply to Martin Dougiamas

Re: Wanted: a tool to automatically assess submitted code

by Gustav W Delius -
A team led by David Livingstone at Kingston University has just won a JISC grant to develop an "Automated system for assessment of programming". This sounds like just what your customer wants. The tool will be ready by next March.
Average of ratings: -
In reply to Martin Dougiamas

Re: Wanted: a tool to automatically assess submitted code

by Jill Kaminski -

Maybe I'm not understanding the application, but I don't think that simple text matching for these assignments wouldn be all that difficult. I'd be interested in pursuing it, Martin. You can contact me directly.

I know a computer science teacher who has a high school student who wrote something much more involved for evaluating and testing submitted Java code. They even integrated it with a plagarism checker. It's called JAMtester, and can be found at http://www.jamtester.com/.

Jill

Average of ratings: -