New question type: Junit

New question type: Junit

by Süreç Özcan -
Number of replies: 30
Hello Moodlers,

I have developed a new question-type called sojunit and want to share it with the community.
I have tested and debugged as much as I could, but if you should find any other bugs, let me know please!
I want to thank Tim Hunt for his great help while implementing this question-type!

The last development has been done on a linux-server but it should work on windows as well.
Since the uploading file has a limitation you need to download JUnit3.8.2 and unzip and copy junit.jar into the sojunit-folder attached.

Note:
In the question-type there is a section that includes 'Source Code' and 'Selected Source Code'. This won't work without installing the Source-Code-question-type.
I have been thinking about creating quizzes for computer science classes and created a new architecture for this purpose with following idea:
In computer science classes the usage of source codes is essential. Corresponding to a given Source-Code-question-type (which I am still working on) which simply displays a nicely syntax-highlighted source code you can create a following group of questions. So the idea is to separate the source code from the questions.
You may display a Source-Code with its corresponding questions on one page and make a page break before the next group while organizing the questions for a quiz.
You may reference from this single question e.g. to special code lines of code fragments of the displayed Source Code to ask questions. Be creative! wink
Maybe this feature matches more for for example the multichoice or regexp question types (which I will post soon). But I wanted to enable this also in this question type.

The following is a copy of my Readme.txt-file inside of my plugin - this way you can get an idea of what it does:

For computer science courses this question type allows a student to type source code for a given interface in Java.

In case the student response compiles the response gets graded automatically. This way the teacher can save time for corrections. Otherwise the teacher needs to grade the student response manually (in Results-tab -> Overview-tab -> click on a students attempt -> a questions "Make comment or override grade"-link) in order to give partial scores instead of none.

After the quiz has been taken the given source code can be seen nicely syntax-highlighted in the gradings. In the quizzes "Results->Item analysis" the compiler or execution output get displayed for each students attempt. This way you can get a better overview about the problems of your students in coding.

There is also a security manager which checks for non-proper student-responses. This way you should not worry about the student messing up the system.

General Quiz settings

It is required to leave the quiz in Adaptive Mode in order to get the compiler outputs displayed. In dependence of the Adaptive Mode used you need to leave "Penalty factor" to "0" in question-settings. This way the student won't get penalties subtracted when pressing the "Submit"-button in order to see if his/her code compiles.

For an "exam" and "exercise with grading" in quiz-options set e.g. "Attempts allowed" to "1 attempt" and "Grading method" to "Last attempt". This way the student can send in their final answer once using "Submit all and finish"-button but because of the Adaptive Mode they can "Submit" their code several times for compilation tests.

If several Junit-questions used in a quiz - for usability reasons it is recommended to put one question on a page at a time using "page breaks". It is also possible to put several Junit-questions on one page though.

The idea of which output to display at what time for which test type is as follows:

compiler output:

  • Types: exams, exercise-sheets with grading and exercises without grading:

    gets displayed always

execution output:

  • Types: exams, exercise-sheets with grading:

    Gets displayed allways after the quiz has been taken.

    Settings: In "Students may review" in quiz-options uncheck in "Immediately after the attempt" "Feedback" so that the execution output does not get displayed and the student does not get extracted or demotivated while taking an exam. In the "exercise-sheets with grading" the student should not get to see this until after the quiz.

  • Type: exercises without grading:

    Display "immediately after the attempt" to student.

    Settings: In "Students may review" in quiz-options in "Immediately after the attempt" leave "Feedback" selected in order to display also the execution output after an attempt to the student.

Given code for student

Make sure the class and method names used in "Given code for student" are identical with the ones in your test class you want to use for this question. Otherwise the student response won't be executed and you will have to grade each student response manually.

The class names need to be non-public in order to allow several classes to be used in one JUnit-test-file.

Load test class

In order to get an automatic grading for this question you need to create and implement a JUnit test-file. This file needs to match exactly the class and function names used in the source code written in the "Given code for student" HTML-editor.

The number of your implemented test cases need to match the number in "Default question grade". For each test case the student can get 1 point. If you have for example five test cases in your test-file you should set the "Default question grade" to "5". If the response of the student passes all test cases he/she will get 5 points out of 5. Otherwise each failed test case will subtract 1 point of the possible.

Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Of course, if there was better documentation for people trying to write new question types, Süreç probably wouldn't have needed so much of my help. I will try to find time to update http://docs.moodle.org/en/Development:Question_type_plugin_how_to one day, I promise. I just don't know when.

I think this is a really great idea for a question type. It is doing something that properly harnesses the abilities of a computer to engage the students, rather that more boring multiple choice questions. This is what online assessment should be about.
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit

by Süreç Özcan -
Here you can get the plugin for the Source-Code question type (or a better understanding to it) and if you want more - beginning from here: by Süreç Özcan - Monday, 11 August 2008, 11:24 AM
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Süreç Özcan -
Hi Tim and other Moodlers,

At the moment I am testing my Junit-question-type with some students using the student-role. Beneath some other bugs we have found 2 bugs of which I don't know how to fix them. I am hoping you can give me some hints!

1.
A student takes a quiz with e.g. three Junit-questions. After pressing the "Submit all and finish"-button the student can view his given answers (in the quiz-settings I set the gradings to last attempt - therefore this should be shown - as far as I understand). Each question displays the compilation or execution output below the student-response. When a student has made e.g. 8 attempts for a question, arbitrarily it adds an additional 9th attempt with the execution-output: Class not found "XXX.java".
Using the Submit-button while taking a quiz this error message didn't get returned - instead it gave the correct execution output in correspondence to the used JUnit-test-file.
I don't understand why this happens. In my test e.g. it happens for the first two questions, the third gets correct.
Whereever this error-message gets displayed, the student gets full marks for this question, independent whether he got all, just some or none.

When a student types in answers provoking compiler-error-outputs this problem doesn't seem to occur.

Do you think this has to do anything with the compare_responses-function from questiontype.php?


2.
After taking a quiz the student gets all the questions displayed. Each question displays the questiontext, the student-response (which in this case is his source code) and the compiler- or execution-output.
The student-response and the output have a background coloring.
Now when a student wants to look into another attempt of a question a new window opens. Everything is correct, but the background colorings are not there.
In "questiontype.php"-file in "print_question_formulation_and_controls"-function this was implemented. I think it needs to be implemented somewhere else as well - what do you say?
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
2. is just a bug. MDL-16219. I guess no one notice before. Should be quick to fix.

1. Is probably explainable, but may require some changes to your code. Basically, when the student does 'Submit all and finish' then an additional EVENT_CLOSE gets sent to each question that is not already in a closed state. You probably need to make sure that if the question has already been graded, and you get an EVENT_CLOSE, you don't actually regrade it. Working out exactly what needs to change in your question type will involve some serious study of question_process_responses, which I don't have time for right now. I will try to have a look later, if you can't work it out on your own.
Average of ratings: -
In reply to Tim Hunt

Re: New question type: Junit - debugging

by Süreç Özcan -
1.
Could you tell what I need to do in order to fix it at the moment in 1.9.2?

2.
I think I have managed to get the last responses' output displayed correct now.
What I couldn't manage and don't understand yet is the grading. It still gives wrong gradings. (you may take a look at the attachment)

I added the following in the grade_responses-method below the line "$state->raw_grade = 0;":

if (isset($state->responses['_compileroutput'] ) and (QUESTION_EVENTCLOSE == $state->event)) { //new
// copy of original code from else-part (I think I need almost everything, since the "$state->raw_grade" gets calculated through the executionoutput, which I get only if I get the compilation informations)
// I have commented every appearance where "$state->responses['_compileroutput']" and "$state->responses['_executionoutput']" get set.
// I have replaced the line "$state->raw_grade = $question->maxgrade - $numfailures - $numerrors;" with:
// If submitted then compare against last graded responses, not last given responses in this case
if (question_isgradingevent($state->event)) {
$state->raw_grade = $state->last_graded->grade; // use the last_graded grade - otherwise it returns wrong gradings (max/max)
} else {
$state->raw_grade = $question->maxgrade - $numfailures - $numerrors;
}
}

else{
//original code
}

3.
What doesn't work now (I don't know if it worked beforehand):

Case 1 - correct:
I have several Junit-questions in a quiz on one page. Pressing the "Submit all and finish"-button without giving any answer shows the points for each question 0/X correct.

Case 2 - wrong:
I have several Junit-questions in a quiz, each on one page using page breaks. Pressing the "Submit all and finish"-button without giving any answer shows the points for each question -/X. I have compared this scenario with other question types and it should return 0/X instead. Do you know what I have to add in my grade_responses-method?

Note: I have just repeated case 2 and now it shows scores only for that question I have pressed the "Submit all and finish"-button page. The other questions don't get the scores displayed at all. Trying to "comment or regrade" a question which has no grading displayed, I change it from 0 to 1 point. When I save this I get a message that it can't find an attempt for this question yet. But when I close this warning it displays gradings with 1/X. (Between the first and second trial of case 2 I have changed the grade_responses-method. Maybe this takes account!?)
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
1. The fix is http://cvs.moodle.org/moodle/mod/quiz/reviewquestion.php?r1=1.16.10.5&r2=1.16.10.6.

Will look at the rest later.
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Süreç Özcan -
Due to some changes in the sojunit-folder since the last uploaded zip-file from the first post of this discussion topic I post the most current version (before I made the changes in questiontype.php mentioned above).
I think this version should work although no sourcecode-question-type has been installed.
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Süreç Özcan -
Hi Tim,

I have adapted your fixes.patch from the conrtib-entry to my plugin (manually, since I couldn't get the patch working properly).

It seems to work now - thanks a lot! It is a good idea to separate the configs into a config.php-file.

I don't understand the temp-folder issue yet though. Deleting it in the end of the grade_responses-function returns the could-not-find-file-error. Removing it it works. But the temp-folder would get really big after a while - don't you think? It would be a pity to empty it once in a while manually.
Maybe it is possible to give it another try by deleting the $temp_file at the very end of the attempt procedure. In one of the last function calls I could override it the way it deletes the temp-files (of the last attempt). What do you say? If it sounds possible - do you know where I could do this? I have looked into the attempt.php-file, but am not really sure.
In a second thought - maybe this way after everything seems to work fine, but going back to the grades would output the same error - I am not sure.
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I am completely baffled by the deleting the temp files problem too. I still think that logically the code you had before should have worked. But experimentation showed that commenting out that delete statement fixed the problem.

I'm afraid that it is going to take some more debugging work to discover exactly what is going on, and so devise a scheme for safely cleaning up the temporary files. Perhaps the solution is to change the name used for the temp dir to include some different ids, so the folders are more unique, and so deleting one does not mess up other other questions in the same attempt. I am pretty sure the problem is not with the quiz or question bank, but with things that your question type is doing, so I think in the first instance it is up to you to try to solve it. If you get really stuck, I may be able to try to help.
Average of ratings: -
In reply to Tim Hunt

Re: New question type: Junit - debugging

by Süreç Özcan -
ok, I will see what I can figure out (hopefully all by myself!)
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Süreç Özcan -
Hi Tim,
I guess I will need your help here...

1.
First I have tried to generate a randomly numbered folder below
$cfg_dataroot . '/so_junit_temp/' . $state->attempt . '/' . $state->question
This didn't make any difference.

2.
Later I have figured out that deleting only the temporarily created files, everything still works fine.
So I thought it would be a good idea to remove all the folders below \moodledata\so_junit_temp-folder only if they are older than e.g. 10 days.
I have been trying quiet long now with many different examples I could find by googling, but I must be doing something wrong - and I don't get it.
But yet I am not sure if this will be the solution to the problem. What do you say?

Here an example of one of my trials:
function rm_recursive($filepath) {
print_object($filepath);
if (is_dir($filepath) && !is_link($filepath)) {
if ($dh = opendir($filepath)) {
while (($sf = readdir($dh)) !== false) {
if ($sf == '.' || $sf == '..' ) {
continue;
}
print_object('1');
print_object($filepath.'/'.$sf);
if (!rm_recursive($filepath.'/'.$sf)) {
print_object('2'); //it never gets here and the page won't be redisplayed :o(
throw new Exception($filepath.'/'.$sf.' could not be deleted.');
print_object('3');
}
print_object('4');
}
closedir($dh);
}
return rmdir($filepath);
//$seconds_old = 864000; //10 days = 864000s
// if( @filemtime($filepath) < (time()-$seconds_old) ){
// return @rmdir($filepath);
// }
}
return unlink($filepath);
}
and calling this at the end of the grade_responses function with:
//at the moment I have tried it only with one sub-folder below ($cfg_dataroot . '/so_junit_temp/' . $state->attempt';),
//just to make sure not to delete the so_junit_temp-file as well - in this case the commented time comparison would
// never work, because the current attempt is never older than 10 days
$del_temp_folder = $cfg_dataroot . '/so_junit_temp';
$this->rm_recursive($del_temp_folder);
Since it doesn't display the quiz-attempt page again, I could not try out the commented code at the end of the function. Any ideas how to get it working properly?

3.
At the end I thought it would be easier to just use something like this for linux:
shell_exec("rm -rf so_juni_temp-path/*");
somehow combined with "find . -mtime 10" which means: find everything that is older than 10 days from the current directory (I am not sure if it should be from the current directory or from one below the current though).
(Something similar would be needed in case the question type has been used on Windows)
What do you say?

Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
If deleting just the files works, but deleting folders causes problems, then why not just put all the files in one folder, and put all the various ids in the file name? Would that work.
Average of ratings: -
In reply to Tim Hunt

Re: New question type: Junit - debugging

by Süreç Özcan -
Great idea, Tim! Thank you very much.
It took me a while but I guess it is working now. I have two more testers today and tomorrow. When they find no bugs corresponding to this issue I will post the new questiontype.php here.
No, maybe it is even better to wait until I have completed every other found bugs and suggestions. At the end I will post a final version.
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think you should post the code sooner rather than later. If you ever make some code where you have found and fixed every bug, then please tell me how you did it wink
Average of ratings: -
In reply to Tim Hunt

Re: New question type: Junit - debugging

by Süreç Özcan -
ok, then I will put a temporary questiontype.php-file here.
I will be off for a short break tomorrow so that I will work on probable debugging next week.
And to your comment about finding and fixing every bug - I guess you know what I mean! big grin
Of course I am talking about all the bugs that will be found - but there could and would be sure some bugs not found yet....
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging - teacher role

by Süreç Özcan -
Hi Tim,

at the moment I am testing my Junit-question-type with some students using the teacher-role and have the following question:

In Results -> Item analysis: All the student responses' outputs are listed. All varieties are displayed red. Is it possible to display those outputs that give full marks in blue color as it does with other question types?
In this case a full marked response would look like
"output: ,output: ... OK (3 tests)"
with 3 changing in dependence of the number of test cases.
Would this be more work than effort to implement? Could you give me some hints, please?
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - debugging - teacher role

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I must admit I know very little about how the Item Analysis report actually works. And it is being replaced in Moodle 2.0. So I am afraid that you are on your own here.
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version

by Süreç Özcan -
Hello Moodlers,

so here is the final version for the junit-question-type after several manual tests and debuggings.

Note: You will still need to put the junit.jar-file into the sojunit-folder in order to get it working properly.

Enjoy!
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Nice work.

By the way Süreç, you might like to know that for Moodle 2.0, it will be possible for question types to have settings screens like Administration > Modules > Activities > Forum. For a question type it would be Administration > Plugins > Question types > JUnit, and you could let the administrator put the paths to Java in there, rather than having to edit the code.

Hmm. That reminds me, I must document how to do that. MDL-16558.
Average of ratings: -
In reply to Tim Hunt

Re: New question type: Junit - final version

by Süreç Özcan -
Thanks Tim!
Also for the the information about the interesting new feature in Moodle 2.0!

By the way: Do you know when they plan to get Moodle 2.0 released as a stable working version?
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I don't think anyone knows. The official line is, as ever, 'when it is ready'. There is an online developers meeting tomorrow. Perhaps after that we will have a clearer picture how much needs to be done.

The aspiration is to release it early next year. I think with luck it will be released before next (northern hemisphere) summer.
Average of ratings: -
In reply to Tim Hunt

Re: New question type: Junit - final version

by Süreç Özcan -
hi hi hi - it is fun to read the way you write - thanks for the information!
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version - architecture

by Süreç Özcan -
In the following next three posts (including this one) I want to add some architectural information about this question type that may be needed to understand the plugin better.
I want to start with the database table diagram. For this question type the only table I have added is question_sojunit (in the plugin download itself you can read what each field is for).
Attachment db_fragetyp_junit.jpg
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version - architecture

by Süreç Özcan -
I have been using the plugin template prepared from Tim Hunt. This is how I have extended it (you may compare with the general template structure here):
Attachment Template_structure_junit.jpg
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version - architecture

by Süreç Özcan -
and finally the question type architecture part especially for this question type. You may compare with the general question type architecture diagram at the link at the post above.
Attachment uml_klassendiagramm_junit.jpg
Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit - final version - architecture

by Chen ZhongWei -
Thanks Süreç, it is a wonderful module. And I am wondering is it possible to be rewrited for C/C++ programming questions?
Average of ratings: -
In reply to Chen ZhongWei

Re: New question type: Junit - final version - architecture

by Süreç Özcan -
Thank you Chen! smile

yes - I think it should be not that hard to do so.

Is there something equal for JUnit (for Java) for the programming language C/C++?
You would have to adapt the calls for compiling and executing in c/c++.

Maybe there would be some smaller changes as well, but the main part would be adapting the stuff written above.

Cheers,
Süreç


Average of ratings: -
In reply to Süreç Özcan

Re: New question type: Junit

by Pushpender Garg -

What if junit test file is dependent upon some external jars. What I mean is test file cannot run on its own but i also need to provide additional jars on classpath. Is there a way for that? 

Average of ratings: -