Hello,
My name is Arkaitz Garro and this year I participate in Summer of Code, developing a new module for Moodle.
I have started with some specifications, but Nicolas and me prefer to get us much feedback as possible from the community.
The project wiki: http://docs.moodle.org/en/Student_projects/Automated_grading_of_programs
Thanks.
Epaile, Automated grading for Computer Programming Assignments
by Arkaitz Garro -
Number of replies: 15
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by David Horat -
Good luck with the project! 
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by Brian Jones -
Hi!
I work in a Computer Science dept., and I created a new assignment type that is geared toward our department's needs, and it uses a mix of Moodle API functionality and one or two out-of-band methods for accomplishing this task. However, I would *love* to see a solution that was more integrated into Moodle. There are a couple of things that I have been thinking about doing that might make our lives easier as both developers and site administrators, so maybe you can work some of these ideas into your project:
0. DO NOT ASSUME SMALL CLASS SIZES. This is, imho, the cardinal sin of a great many moodle modules. I am constantly hacking things to make things more usable for class sizes of around 150 students. It's not a lot of work to write code that works as well for big classes as small ones - you just have to be in that mindset at the time
1. Compiling and executing code using a web front end incurs quite a bit of risk, both in terms of security and performance. What if my moodle site is on the same server as some other internal site(s)? What if it's not, but it hosts 30 different classes? If 2 classes have the same due date for a programming assignment, I could be looking at a *lot* of concurrent users compiling and executing random bits of code on my web server! One solution to this would be to develop an XMLRPC plug-in to moodle that can shuffle things off to a remote (and very simple) XMLRPC server for compilation/execution. The server would then report back the results. Note that in some cases, the XMLRPC server may not be *really* "remote". It may just be a virtualized host that is isolated to protect the main server from malicious code or badly behaving code.
2. Instead of forcing zip or jar files down our throats, please allow the option of specifying the names of the files required for the assignment, and allow multiple file uploads in the upload interface. It's easier to force the student to upload files of a certain name than it is to debug broken zip or jar files, which involves emails back and forth asking the student what platform they're on, what version of zip they used, blah blah blah. Not having to use these tools to submit assignments also takes a possible excuse away from students who just want to use it as an excuse for why their submission was late. If you want to use zip on the *server* side of things, well, that's a tool that is known and controlled by the administrative team, and so is a bit more acceptable.
3. Allow unsubmit/resubmit, and remember to give the end user *lots* of feedback. For example, if a student resubmits a file of the same name as one that's already submitted, make sure there's a timestamp next to the file name so they can *see* the system acknowledge that the file is newer than it was
4. Give profs the option of requiring the student to submit all required files before enabling the ability to compile/execute. Also, if the student *has* submitted all required files, they should be allowed to do a test run via the web interface so they can be sure that their code compiles not only on their own machine, but on the machine doing the grading, which may be a different platform/architecture.
5. Don't just create pretty moodle errors - output the *actual* compiler/execution errors so that students can see what's going on, and they have something to send to their profs that is more meaningful than just "compilation failed". Error handling in this context is something of a challenge. Historically, I've been using a page that just shows the user the output of the compilation/execution, errors or not.
6. Professors will undoubtedly want access to be able to read the code submitted by students to try and troubleshoot, and to help them more easily develop these type of assignments. Make it easy for professors to find/access/read/edit source code. Right now, I use the course's "Files" interface, but I've had to hack that interface to name the files and directories after the courses and student names instead of using numbers that mean nothing to the professors. By the way, note how this is all made much more difficult if the files are submitted in zip files.
7. *Please* make the "Assignments" index page more easy to access. You shouldn't have to click an assignment, and *then* click the "Assignments" link in the breadcrumbs at the top of the resulting page
While you're at it, make that page more useful by adding the ability for profs to delete assignments or add a new assignment from that page instead of forcing them back to their course's homepage, and forcing them to turn editing on to create an assignment.
8. Again, please try to keep in mind that some courses have upwards of 100 users.
I hope this helps. There's a bunch of old code you can look at in the 1.5.x 'contrib' area. The assignment type I wrote is called "uploadmulti". Nowadays, you wouldn't need all that cruft - I'm working on how this can be done using the new "upload" assignment type, which already allows multiple file uploads, and includes many improvements that I'm sure will save me on the order of hundreds of lines of code to do what I need. It'll take some catching up on how Moodle works now to implement it, though.
There's an email in my profile if you'd like to chat - but I'm in the forums quite a bit as well, so if the conversation could benefit others, please reply to me here.
Good luck!
I work in a Computer Science dept., and I created a new assignment type that is geared toward our department's needs, and it uses a mix of Moodle API functionality and one or two out-of-band methods for accomplishing this task. However, I would *love* to see a solution that was more integrated into Moodle. There are a couple of things that I have been thinking about doing that might make our lives easier as both developers and site administrators, so maybe you can work some of these ideas into your project:
0. DO NOT ASSUME SMALL CLASS SIZES. This is, imho, the cardinal sin of a great many moodle modules. I am constantly hacking things to make things more usable for class sizes of around 150 students. It's not a lot of work to write code that works as well for big classes as small ones - you just have to be in that mindset at the time
1. Compiling and executing code using a web front end incurs quite a bit of risk, both in terms of security and performance. What if my moodle site is on the same server as some other internal site(s)? What if it's not, but it hosts 30 different classes? If 2 classes have the same due date for a programming assignment, I could be looking at a *lot* of concurrent users compiling and executing random bits of code on my web server! One solution to this would be to develop an XMLRPC plug-in to moodle that can shuffle things off to a remote (and very simple) XMLRPC server for compilation/execution. The server would then report back the results. Note that in some cases, the XMLRPC server may not be *really* "remote". It may just be a virtualized host that is isolated to protect the main server from malicious code or badly behaving code.
2. Instead of forcing zip or jar files down our throats, please allow the option of specifying the names of the files required for the assignment, and allow multiple file uploads in the upload interface. It's easier to force the student to upload files of a certain name than it is to debug broken zip or jar files, which involves emails back and forth asking the student what platform they're on, what version of zip they used, blah blah blah. Not having to use these tools to submit assignments also takes a possible excuse away from students who just want to use it as an excuse for why their submission was late. If you want to use zip on the *server* side of things, well, that's a tool that is known and controlled by the administrative team, and so is a bit more acceptable.
3. Allow unsubmit/resubmit, and remember to give the end user *lots* of feedback. For example, if a student resubmits a file of the same name as one that's already submitted, make sure there's a timestamp next to the file name so they can *see* the system acknowledge that the file is newer than it was
4. Give profs the option of requiring the student to submit all required files before enabling the ability to compile/execute. Also, if the student *has* submitted all required files, they should be allowed to do a test run via the web interface so they can be sure that their code compiles not only on their own machine, but on the machine doing the grading, which may be a different platform/architecture.
5. Don't just create pretty moodle errors - output the *actual* compiler/execution errors so that students can see what's going on, and they have something to send to their profs that is more meaningful than just "compilation failed". Error handling in this context is something of a challenge. Historically, I've been using a page that just shows the user the output of the compilation/execution, errors or not.
6. Professors will undoubtedly want access to be able to read the code submitted by students to try and troubleshoot, and to help them more easily develop these type of assignments. Make it easy for professors to find/access/read/edit source code. Right now, I use the course's "Files" interface, but I've had to hack that interface to name the files and directories after the courses and student names instead of using numbers that mean nothing to the professors. By the way, note how this is all made much more difficult if the files are submitted in zip files.
7. *Please* make the "Assignments" index page more easy to access. You shouldn't have to click an assignment, and *then* click the "Assignments" link in the breadcrumbs at the top of the resulting page
8. Again, please try to keep in mind that some courses have upwards of 100 users.
I hope this helps. There's a bunch of old code you can look at in the 1.5.x 'contrib' area. The assignment type I wrote is called "uploadmulti". Nowadays, you wouldn't need all that cruft - I'm working on how this can be done using the new "upload" assignment type, which already allows multiple file uploads, and includes many improvements that I'm sure will save me on the order of hundreds of lines of code to do what I need. It'll take some catching up on how Moodle works now to implement it, though.
There's an email in my profile if you'd like to chat - but I'm in the forums quite a bit as well, so if the conversation could benefit others, please reply to me here.
Good luck!
In reply to Brian Jones
Re: Epaile, Automated grading for Computer Programming Assignments
by Martin Dougiamas -
About (7), there is an "Activities" block for that.
In reply to Martin Dougiamas
Re: Epaile, Automated grading for Computer Programming Assignments
by Brian Jones -
Thanks Martin. I believe the problem our faculty had with that was they were using the Social format, which forces you to have something called a "forum", and they didn't want to use the forum except to post news to the course's homepage. However, there's no way to remove the "forums" link from that box.
Also, Martin - I edited mod/assignment/index.php to put "X" delete markers next to the assignments in the listing, and made them viewable only if isteacheredit() Of course, restricting access to this probably changes from 1.6 (where I initially implemented it), but putting the markers there works in 1.8. Here's the diff of mod/assignment/index.php. My changes are on the left.
95,99c98
<
< /// ADDED BY BKJONES@GMAIL.COM - this creates the "delete" link in the assignment listing.
< $del = "<a href=\"{$CFG->wwwroot}/course/mod.php?delete={$assignment->coursemodule}&sesskey={$USER->sesskey}\"><img src=\"$CFG->pixpath/t/delete.gif\"></a>";
<
< /// I (bkjones@gmail.com) edited this slightly to add the delete icon for assignment listings.
---
>
101c100
< $table->data[] = array ("$del $printsection", $link, $type, $due, $submitted, $grade);
---
> $table->data[] = array ($printsection, $link, $type, $due, $submitted, $grade);
103c102
< $table->data[] = array ("$del $link", $type, $due, $submitted, $grade);
---
> $table->data[] = array ($link, $type, $due, $submitted, $grade);
If you find this useful I'll post the new role-enabled mechanism for checking the role before showing these markers when I figure out roles (I'm just getting around to porting my code in preparation for the coming September). Of course, if someone more experienced already knows how to do this, feel free to shoot me an email or post the relevant details
Also, Martin - I edited mod/assignment/index.php to put "X" delete markers next to the assignments in the listing, and made them viewable only if isteacheredit()
95,99c98
<
< /// ADDED BY BKJONES@GMAIL.COM - this creates the "delete" link in the assignment listing.
< $del = "<a href=\"{$CFG->wwwroot}/course/mod.php?delete={$assignment->coursemodule}&sesskey={$USER->sesskey}\"><img src=\"$CFG->pixpath/t/delete.gif\"></a>";
<
< /// I (bkjones@gmail.com) edited this slightly to add the delete icon for assignment listings.
---
>
101c100
< $table->data[] = array ("$del $printsection", $link, $type, $due, $submitted, $grade);
---
> $table->data[] = array ($printsection, $link, $type, $due, $submitted, $grade);
103c102
< $table->data[] = array ("$del $link", $type, $due, $submitted, $grade);
---
> $table->data[] = array ($link, $type, $due, $submitted, $grade);
If you find this useful I'll post the new role-enabled mechanism for checking the role before showing these markers when I figure out roles (I'm just getting around to porting my code in preparation for the coming September). Of course, if someone more experienced already knows how to do this, feel free to shoot me an email or post the relevant details
In reply to Brian Jones
Re: Epaile, Automated grading for Computer Programming Assignments
by Arkaitz Garro -
Hi Brian!
Good replay! I see you have much more experience than me. Some comments about your ideas:
0. This module is geared toward our department's needs too, where the number of students is about 100. I keep this in mind.
1. I talked about this with Martin. And yes, it is a risk, both in terms of security and performance. The idea is allow the compilation/execution in other server, if possible. Be sure that students will submit their works at last moment!!
2. You are right. Furthermore, uncompress zip, jar.. files is more work load for server.
3. I agree.
4. Great idea!
5. I agree. I will add the option to email compile errors to teachers.
6. Do you mean allow teachers to read/edit files into Moodle interface or provide a link to external editor?
8. I will keep it in mind!
Thanks again for your reply. I will work in this and more ideas theese days.
Good replay! I see you have much more experience than me. Some comments about your ideas:
0. This module is geared toward our department's needs too, where the number of students is about 100. I keep this in mind.
1. I talked about this with Martin. And yes, it is a risk, both in terms of security and performance. The idea is allow the compilation/execution in other server, if possible. Be sure that students will submit their works at last moment!!
2. You are right. Furthermore, uncompress zip, jar.. files is more work load for server.
3. I agree.
4. Great idea!
5. I agree. I will add the option to email compile errors to teachers.
6. Do you mean allow teachers to read/edit files into Moodle interface or provide a link to external editor?
8. I will keep it in mind!
Thanks again for your reply. I will work in this and more ideas theese days.
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by Brian Jones -
Hi!
As for item 6, I think that allowing access through the moodle course's "Files" interface (clicking the "Files" link in the course's administration menu) is just fine, but you'll notice if you go there that the Files interface names things using the numeric id's for users, courses, and assignments. Since profs should never have a reason to know the numeric ID of a user or assignment, this makes it pretty difficult for them to use, even *outside* the context of this module. I have written a hack that causes moodle to use human-readable names instead of numeric IDs to name these files and directories, but I took some heat for that when I suggested that moodle do this by default, so you'll want to check with others on this. Perhaps the thing to do is let moodle name things using numeric id's, but map them to names in the front end interface
I'll keep an eye on this thread - and please keep in touch about the progress and any questions you may have, or even if you want a hand or want to bounce ideas off someone
As for item 6, I think that allowing access through the moodle course's "Files" interface (clicking the "Files" link in the course's administration menu) is just fine, but you'll notice if you go there that the Files interface names things using the numeric id's for users, courses, and assignments. Since profs should never have a reason to know the numeric ID of a user or assignment, this makes it pretty difficult for them to use, even *outside* the context of this module. I have written a hack that causes moodle to use human-readable names instead of numeric IDs to name these files and directories, but I took some heat for that when I suggested that moodle do this by default, so you'll want to check with others on this. Perhaps the thing to do is let moodle name things using numeric id's, but map them to names in the front end interface
I'll keep an eye on this thread - and please keep in touch about the progress and any questions you may have, or even if you want a hand or want to bounce ideas off someone
In reply to Brian Jones
Re: Epaile, Automated grading for Computer Programming Assignments
by Arkaitz Garro -
Hi Brian,
Another way to access students submited files is through submissions list of an assignment. At this moment, is not possibly to edit or rename this files from here, but if we add this possibility, it will be much easier access to files, without remembering courses and assignments IDs.
Another way to access students submited files is through submissions list of an assignment. At this moment, is not possibly to edit or rename this files from here, but if we add this possibility, it will be much easier access to files, without remembering courses and assignments IDs.
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by Brian Jones -
By the way - it would be nice if you could make execution/compilation options that have to be enabled by the prof when the assignment is created. For our systems, we don't execute the students' code. We just allow the student to press a button to see that their code compiles after they've uploaded so they can be confident that their code didn't get mangled during the upload or something. I don't foresee our faculty ever enabling "execute" functionality for our assignments, though the it would be nice to have that capability (one never knows what next semester will bring).
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by Björn Lakeberg -
Hello Arkaitz,
I'am a student in computer science and at present on my graduation as bachelor. This is my project that is very similiar to yours: http://moodle.org/mod/forum/discuss.php?d=80232
I tried to test Epaile. But there was no access over CVS on anonymous login. I don't want to load each single file over the web CVS system.
Epaile on actual is the alpha version? How looks the further development?
Nevertheless is it possible to test the actual version and become access?
Kind regards from Germany
In reply to Björn Lakeberg
Re: Epaile, Automated grading for Computer Programming Assignments
by Süreç Özcan -
Hello Arkaitz, Björn and other Moodlers,
@Arkaitz: Is it able to use Epaile yet in Moodle 1.8.2+? Are you planning to offer it as a module or will I have to download it from the web cvs system as Björn says?
@Björn: Did you get any further with your implementation or/and the use of Epaile?
I am also a student in computer science and am planning on developing a similar module to both of yours (for programminng courses) only that I would like to integrate it as a question type.
This way students would type in their code directly into Moodle, compile, post it and receive feedback through e.g. JUnit before or after posting it. The code shall be automatically checked through JUnit to save correcting time for the teacher. As far as I understood you are using DOMJudge instead (I will need to compare each of them before deciding which one to use).
At the same time this way this type of question could be integrated into an exam with a time limit together with other questions and question types.
Any information will be great,
Süreç
@Arkaitz: Is it able to use Epaile yet in Moodle 1.8.2+? Are you planning to offer it as a module or will I have to download it from the web cvs system as Björn says?
@Björn: Did you get any further with your implementation or/and the use of Epaile?
I am also a student in computer science and am planning on developing a similar module to both of yours (for programminng courses) only that I would like to integrate it as a question type.
This way students would type in their code directly into Moodle, compile, post it and receive feedback through e.g. JUnit before or after posting it. The code shall be automatically checked through JUnit to save correcting time for the teacher. As far as I understood you are using DOMJudge instead (I will need to compare each of them before deciding which one to use).
At the same time this way this type of question could be integrated into an exam with a time limit together with other questions and question types.
Any information will be great,
Süreç
In reply to Süreç Özcan
Re: Epaile, Automated grading for Computer Programming Assignments
by Süreç Özcan -
Any help to my post from "Thursday, 24 January 2008, 03:52 PM" yet?
In reply to Björn Lakeberg
Re: Epaile, Automated grading for Computer Programming Assignments
by Arkaitz Garro -
Hi again!
Maybe is too late, but I've been very busy this last months... sorry.
@Björn: you can download epaile from here: http://download.moodle.org/plugins/mod/assignment/type/program.zip
@Süreç: Epaile is implemented for Moodle 1.9
Leave your questions here or write me at arkaitz.garro@gmail.com
Maybe is too late, but I've been very busy this last months... sorry.
@Björn: you can download epaile from here: http://download.moodle.org/plugins/mod/assignment/type/program.zip
@Süreç: Epaile is implemented for Moodle 1.9
Leave your questions here or write me at arkaitz.garro@gmail.com
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by Anurag Agarwal -
Hi,
I have downloaded the program.zip and unzipped the file under
moodle/mod/assignment/type
I have moved the domjudge to top directory and was able to install it successfully. But I don't see the assignment_judgehost option in the assignment/setting.
Is there something I am missing here.
Regards,
Anurag.
I have downloaded the program.zip and unzipped the file under
moodle/mod/assignment/type
I have moved the domjudge to top directory and was able to install it successfully. But I don't see the assignment_judgehost option in the assignment/setting.
Is there something I am missing here.
Regards,
Anurag.
In reply to Arkaitz Garro
Re: Epaile, Automated grading for Computer Programming Assignments
by Ilya Minkin -
Does anybody use this system in his/her work? Is it stable/secure enough to be used in the real course?
In reply to Ilya Minkin
This forum post has been removed
The content of this forum post has been removed and can no longer be accessed.