Moodle in English: junit-question-type

I have designed a db-structure for my new JUnit-question-type.

The idea of the question type:

The teacher gives a question text e.g. Please implement the factorial-function of the Factorial-class below.
Then he adds the interface of the source code needed to implement the factorial-function for the student (naming the classes and methods as they will be used in the corresponding junit-test classes to checked with each other).
Next he can load needed junit-test classes he has prepared for this question.
Now he can give partial scores to each needed test class (if a loaded junitTest.java-file has 5 test cases he types in 5 for this file - one point for each test case).
The "default question grade" is the sum of all partial-scores set for each used junitTest-file.
Finally he can choose the display option whether to
1. do not display test class results to student at all (e.g. during an exam)
2. display test class results to student before posting his answer (e.g. for a quiz taken as an exercise)
3. display test class results to student after posting his answer (e.g. for a quiz taken as an exercise)

So I decided to use following tables and fields:
table: question_so_junit
field1: givencode (the code given by teacher to be filled out by student)
field2: displayoption (display test results to student? 0=no; 1=display results before posting answer; 2=display results after posting answer)

table: question_so_junit_testclass
field1: testclassname (the name of the test class needed for this question loaded by teacher )
field2: testclasscontent (content of loaded test class file - maybe not needed)
field3: partialscores (score to achieve for this question - one point for each testcase of testclass)

table: question_so_junit_testoutput
field1: testoutput (execution output of test cases of student response)
field2: partialgrade (partial grade the student receives for this testclass output (calculated through field testoutput))

table: question_so_junit_studentresponse
field1: studentid (foreign key to user table?)
field2: studentresponse (source code response of student)
field3: compileroutput (compiler output of student response)

So there can be several JUnit-classes used for one new junit-question-type, but on testclass belongs to one junit-question.
There can be as many testoutputs and with it partialgrades depending on the number of students that took the junit-question, but one testoutput for each student of one test-class.
There can be as many student responses as students have taken the question, but each student response belongs to one junit-question.

So my questions are:

Is there something I forgot to think about in general or should do different?
Does the db-structure make sense to do what I want to do?
What I didn't understand yet is if I will need e.g. in the "question_so_junit_studentresponse"-tabe the studentid (as a foreign key from user-table?) or if the question-module is built up the way that it knows through the session whose response belongs to whom. Or do I need to add the studentid into the "question_so_junit_testoutput"-table as well?

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Wednesday, 14 May 2008, 4:46 AM

First, I have to say, brilliant idea for a question type. This is what online assessment should be about. None of these feeble multiple choice questions, but instead use the computer's processing power to really test the student's understanding.

Now, before I comment on the the database design, can I just check: are you really sure you need to allow for multiple test classes? What does that let you do that cannot be done with a single class? I suppose multiple test classes may be easier to manage for people creating questions, but they make designing the question type harder, so it worth really being sure you need that extra complexity/sophistication.

OK, so on to the database tables. The stuff on the left, the question definition is very good. Just a few quibbles:

A. the column should be called question_so_junit.questionid (not question_id) to match the Moodle coding conventions. (And strictly speaking that column should not be part of the primary key, because the id column already uniquely defines the row.)

B. Since, rows in the question_so_junit table are associated with a unique row in the question table - (the way I think about it is that question_so_junit is an extension to the question table, that is from my limited knowledge of object-relational mapping and how you deal with object inheritance on the relational side) - it seems more natural to me to link to question_so_junit_testclass directly to the question table, and not via the other table.

C. I don't think you need the testclasscontent column. I assume that teachers will upload the test classes to the Moodle server (in course files?) as .jar files, so what you do need, in either question_so_junit or question_so_junittestclass as appropriate, you need a jarfilename field, or similar.

So, I think question_so_junit_testclass should just have columns id, questionid, testclassname, jarfilename, partialscore

Oops, I have just spotted another point. Moodle coding guidelines say sub-table names should be plural, so question_so_junit_testclasses, etc.

On the right of your table, it is more problematic, because it does not really fit with how Moodle works. Now, the problem here is that I (even as official quiz maintainer) don't really like the way that Moodle stores student responses. So I would be the first to agree that what I am about to advise sounds like rather strange advice in places. However I may one day change some of the core quiz tables to fix the problems, is which case, you will find it better to have followed my advice.

OK, so the thing is that question types are really not supposed to make their own tables for storing student responses. (For example, there is no support for backing up and restoring the contents of such tables). Instead everything should be stored in the question_states table.

Now, when a student submits an attempt at a question, what they entered initially gets pulled into an associative array $action->responses by question_extract_responses(...) (Look at line 225 of mod/quiz/attempt.php http://cvs.moodle.org/moodle/mod/quiz/attempt.php?annotate=1.141#l225), then later in question_process_responses(...), which in turn calls the grade_responses(...) method of the question type, that data gets transferred to $state->responses.

Now, if necessary, during your grade_responses method, you can add extra things to this array, so you could, for example, do:

$state->responses['compileroutput'] = ...

So far, so good. Now we get to the silly bit. When it comes to storing $state->responses in the database, what actually happens is that this array has to be converted to a string, that is then stored in the question_states.answer column. The way this happens is that your question type has to implement the save_session_and_responses and restore_session_and_responses methods to convert the responses array to and from a string.

Fortunately, there is an easy way to do this. You should steal the implementation of these functions from my Opaque question type: http://cvs.moodle.org/contrib/plugins/question/type/opaque/questiontype.php?revision=1.2&view=markup. They fairly directly convert the associative array to a string and back.

(In the future, what should actually happen is that the core question processing code in questionlib should take care storing the responses array without bothering the question types, probably in a qestion_state_resoponses table with columns id, questionstateid, key, value. The only reason I have not done this is that the database upgrade script would be a nightmare to write and test, and take forever to run on large sites.)

Finally, if I can make a boring point about security: Have you though about what happens if a student types something like Runtime.exec("rm -rf /"); into their answer?

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Wednesday, 14 May 2008, 9:32 PM

general info: I use Moodle 1.8.2+

I will check if I really will need several test cases or just one. What makes it complicated though... the grading and usage of several tables?
I was thinking to load up different junit-test.java-files in case the teacher given interface has several classes to be implemented. But it should be possible to add all test cases into one test.java-file as far as I understood - right?

A. I have been using DB-Designer4 to draw the ER-diagram which automatically created the syntax of "question_id". I am aware of using "questionid" when creating my xml-file instead. As soon as I set a relation between two tables the program creates the needed foreign keys itself (in its predefined syntax) and with it it wrote e.g. the "question_id" as a key. In the xml-file I will only define this field as a foreign key. Good though that you mention this to make it clear for me again!

B. So you mean that I should move the ["question_so_junit" - "can_have_testclass" - "queston_so_junit_testclass"] - relation to ["question" - "can_have_testclass" - "queston_so_junit_testclass"]?
What about the "can_have" - relation then?

C. Yeah, I thought to let them upload the test classes (.java-files not .jar-files - why do you think of .jar-files?) in course files. Then I will need for each student a temporary folder with the student-answer-java-file and the test.java-file(s) of the teacher. That is why I was thinking of saving the studentid-foreign-key into the so_junit_studentresponses-table.
Do you think I wouldn't need this and do it differently without the need of the studentid?
You are right, I might remove testclasscontent from the testclass-table. I wasn't sure if I will need a database entry although I will load the test classes to the Moodle server.

To your last point about security... well yeah, I know this is a big lack. If I have time I will think about this issue as well. For the beginning I would like to get it working. The teacher knows his students who registered into Moodle and would not like to do such stuff - but this is easy to say. ;o)

New db-structure in case only one test-file is loadable for one question:
Maybe I don't need the score-field anymore (renamed partialscore to score since there can be only one score with one junit-file) anymore. I could just use "Default question grade", couldn't I?
If I allow only one junit-file to upload I can move the so_junit_testclasses table fields into the so_junit-table.
Then there can be only one testoutput and grade (renaming partialgrade to grade then) for one compileroutput. So I can merge both tables into one as well.
I deleted studentresponse due to your great explanation, since it is kept in question_states-table.

So there are two new main questions:

Would I leave the "can_have_studentresponse"-relation like you can see in the pic or would I delete this relation and put it between the question-table and so_junit_testoutputs-table with an 1:n-entity?
What about the studentid-field (foreign key) - can/should I delete it?
1. I thought I need it in order to match the fields used in this table to a student so that I don't lose track of who got how many points.
2. And because of the point mentioned below C.
3. How can I match one students-response with his compilation and execution output?
4. If the question_states-table creates an entry for each student should I be able to take care of that in the grade_response-function?
5. Which function would it be I would have to compile and execute the given sources at?

Average of ratings: -

Re: junit-question-type

by sam marshall - Wednesday, 14 May 2008, 10:26 PM

I agree with Tim that this question type is a really cool idea.

I think there are maybe problems with it - for example, if the idea is that the system can mark it by checking the student code passes the test cases, that's good, but that doesn't cover other aspects of 'human' marking such as making sure everyone didn't copy each other and checking non-code aspects such as comments. I'm sure you have already thought of that however.

(Incidentally there are automated systems for checking people didn't copy code, don't know if these are experimental or actually available. And I guess you could check comments maybe using something like Eclipse's compiler ECJ - open-source, very good, I use it in a non-work project that uses java for scripting - which, I think, has options to give warning if javadoc is missing? Obviously that wouldn't actually check the comments are useful.)

About using uncompiled source files all the time, even for test classes - that's ok I guess but you probably should make sure it checks that the test class compiles at the time when the question is edited (not just when it is run).

About security - I don't know how to do it off-hand but it should be possible, when you run the class under test, to specify a SecurityManager that disallows everything (same as for applets).

By the way if I were using this question type for an advanced class I would like to do things such as testing performance - i.e. give people something to do which will take like 10 seconds to run using the most obvious algorithm, but can be done in under a second with a more suitable one, and include that time test in the JUnit test. Might be issues if the server were under load at the time... but that would be pretty awesome. Could even give people more marks the faster they got it.

--sam

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 15 May 2008, 5:10 PM

Thanks for your encouragement!

So far I haven't thought much about the interesting issues of copying and comments. Since I have a limited time to implement this new feature I guess I could at least take it into the TODO-List for future extensions! Same as for the performance testing which I had been thinking of also but leaving it by side now.

Very good point about compiling the source files while editing the new question. I will keep that in mind!

About security I will see what I can do - I don't know much about that part - so time will show if I have time for that.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 15 May 2008, 5:33 PM

corrections to
1.: ... between the question-table and so_junit_studentresponse-table with an 1:n-entity?
(I had uploaded another file but Moodle didn't load it correct I guess - in my current ER-Diagram the queston_so_junit_studentresponse table is called question_so_junit_outputs (not even testoutputs anymore.). I am asking this because you were criticizing the right side of the db-structure in your above comment. What is the problem/advantage if I leave it like this?

2.: 1 and 2 belong together as well as 3 and 4.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Thursday, 15 May 2008, 6:20 PM

There is nothing terribly complicated about allowing multiple test classes. It just means a few extra database tables, and a bit more code. It would be a pity to do that extra work if it was not necessary.

One advantage that multiple test classes would bring is if you wanted different setUp and tearDown methods for different test. Although it is still possible to do that manually within one test class.

.jar files or .java files. It does not really matter. I suppose the thing to do is whatever is most convenient for teachers. Anyway, I have not thought about this and you have, so you are probably right.

With only one test class, you don't need score. There is already $question->defaultgrade, or, more importantly, the grade of the question in a particular quiz which, in your grade_response method, will be available as $question->maxgrade.

I still think you should not have the so_junit_studentresponses table. Instead, in grade_response, store compieroutput and testoutput in the $state->responses array, so they get stored in the question_states table (which is already linked to userid - a better name than studentid, what about teacher previews?).

And the grade should be stored in $state->raw_grade. (I know, that does not follow the naming conventions. Its not my fault!)

Compiling and running the java code, you will need to use the functions documented here: http://php.net/manual/en/book.exec.php

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 15 May 2008, 9:34 PM

Alright, I have also talked with my prof about the JUnit-file number issue and he can't think of any other case as well where he would really need several JUnit-files. So I will keep it simple! But I still would like to know what setUp and tearDown methods would do?

What about teacher previews?

About the studentresponses-table - alright, I think I got it now. I don't really need to create a new field for compileroutput and testoutput since I can store additional information in the $state->responses array in the question_states table.

To the compiling and running issue... well yeah, the link that you have sent is also very interesting but I was talking about the Moodle-function in which all of this stuff should happen in questiontype.php-file.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 16 May 2008, 3:33 AM

If your question type is implemented properly, teacher preview should just work.

Compiling and running, should happen in the grade_response method.

By the way, if you have not found it yet, http://docs.moodle.org/en/How_to_write_a_question_type_plugin should tell you everything you need to know about writing a new question type. Unfortunately it is not complete yet, so you might like to read it, and it might help you, but in other places, you might be able to update it to make it more useful.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Tuesday, 10 June 2008, 12:39 AM

Hi Tim and other Moodlers,
I can get state->responses['compileroutput'] and so on but I have following problems:

In all situations the student shall be able to get compilation results to his code-response. Therefor I implemented a compile-button to take care of this. Now the problem I have is that I don't know how to get the code compiled through my button. It works when the quiz setting allow unlimited attempts -> clicking on submit-button -> clicking on compile-button. I think this works because the data is saved through the submit-button in the database. Should I implement a copy of grade_responses-function where only compilation takes place? Would it be possible to save data in the database then or would I still have the same problem? The compilation results shall be displayed in the results views as well.
After saving a student_response neither student_response nor compilation_outputs are displayed anymore. Once I could see the student_responses - before using state->responses['compileroutput'] in print_question_formulation_and_controls-function.
Logged in as a student I can't see the grades of the students at all navigating directly to the "Grades". Instead I receive many notices e.g. Trying to get property of non-object in D:\oezcan\workspace\moodleSue\moodle\grade\lib.php on line 717 Do you have an idea why? Navigation through quizzes->"JUnit-Test"->Review works.

Note: At the moment you will need to delete the temporary created folder before each new compilation/execution manually to find at \moodledata\"course_id".

I have attached my current question type version. As I said the compilation and execution happens in grade_responses-function and the displaying of compilation (and later execution if feedback is on) in print_question_formulation_and_controls-function. The outputs in the colored areas are still from print_question_grading_details-function.

Kind regards,
Süreç

so_junit_06_09.zip

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Tuesday, 10 June 2008, 12:48 AM

due to limited uploading size... I have removed the needed junit.jar for this question type and thought to post it separetly but it is still to big with its 118kb. Please download JUnit3.8.2 and copy junit.jar into my so_junit-folder above.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Tuesday, 10 June 2008, 9:02 PM

and you will have to set "$_path2java" to the bin-path in which javac.exe is

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Tuesday, 10 June 2008, 11:19 PM

using sessions I have left problem 1 and 3 and get rid of problem 2.
(in this version I have also implemented the feedback - you may have a look at the attachment; again without junit.jar)

so_junit_2008_06_10.zip

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 13 June 2008, 2:17 PM

Some comments on the code:

You don't really need the text_to_js_string function. I think addslashes_js from the Moodle libraries does the same, so you should use that.

Instead of using the HTML editor, why not just use a plan <textarea>?

I don't think the compiler files should go in the course files area. Instead, I think they should go somewhere like moodledata/temp/so_junit/$attemptid/$questionid

I think the code would be simpler to understand if you made separate methods like store_students_code_in_file, compile_code, run_tests, rather than just putting everything inside grade_responses (but I realise that the current code is work-in-progress). Oh, I see, you have defined some functions like that, but you have defined them inside the grade_responses function. That may be leagal PHP syntax, but it is very confusing.

There is already an rm_recursive functions in the Moodle libraries. (In fact, I think there may be more than one!) So use the existing one. (It actually has a different name.)

You have got to stop using so many global variable. You should never need to use globals to get data into a function, you should always pass the necessary information as arguments. And you should use the return value to pass everything back. In PHP is is very easy to pass back complicated data by returning an array or an object.

I realise that I have not answered the real issue yet, but it is getting late, and I need to go to sleep now, and I don't yet understand the code. I'll try to look again soon.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 13 June 2008, 11:52 PM

Thank you very much for the comments on my code!

1.
I have replaced text_to_js_string with addslashes_js. Seems to work properly.

2.
I decided to use the HTML-editor in order to get nicely syntax highlighted output after the student has made an attempt.
Therefor I am using the geshi-plugin. Good that you reminded me - I have put a README_syntHighl.txt-file for this issue into my module.
I allways wanted to ask though: Do you think I could use a plain textarea for the teachers (while creating this question type) and students (while writing their response) inputs and only use a hidden HTML-editor later for displaying the response syntax highlighted? At the moment this works fine and I will see if I will change it. Would be nicer though, because the students might focus more at their response in a textarea than in an html-editor.

3.
Why do you think moodledata/temp/so_junit/$attemptid/$questionid would be a better place to put the files?
Do you suggest putting the files into $questionid because this way several junit-questions in a quiz will be saved seperate without writing on each other?
I changed the place for the files as you have suggested. I delete the files (later automatically) including '/$attemptid'. Or should I delete also the so_junit folder?

4.
In order to make the code understand better I have added create_test_file-function and create_student_response_file-function. I put my all my newly created functions out of grade_responses-function. Thank you very much for this advice! Those are little but important things for good programming!

5.
I replaced the function rm_recurse with remove_dir defined in Moodle lib. Thanks!

6. I would like stop using the global variables. But in the cases that are left I can't do so without changing the proper file outside of my junit-module. This would mean that I would save the returns in parameters in the code where the complete flow happens. I would like to keep doing stuff only inside of my module. Tell me if I am mistaking and PLEASE give me a small example e.g. for $_compileroutput_junit.

I have attached the new version of my so_junit-module (again without junit.jar).

I am looking forward to receive answers for my issues above as well...

so_junit_06_13.zip

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Saturday, 14 June 2008, 9:53 PM

I am now back in Milton Keynes, after a 20 hour journey from San Francisco (I think that means it was a good journey.) I can answer your two easy questions:

2. I think it should always be possible to use a plain text area for input, and syntax-highlighted HTML for output.

3. I am not sure that path is exactly the best choice. The key part of my alternative suggestion was temp instead of course. Apart from that, you need the rest to be unique, so no two students attempting the quiz at the same time will overwrite each other's files.

I am afraid looking at the latest version of the code will have to wait for tomorrow, or whenever my brain catches up with which time-zone it is meant to be in.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 27 June 2008, 12:16 AM

Some more comments:

I have now updated the Opaque question type in contrib, so if you think it will help to see how that works, the code is now available.

It occurs to me that the way I have organised the code there, with a separate locallib.php that deals with talking the Opaque protocol, and questiontype.php that implements the Moodle questiontype API using the library, might work for you. You could move the code for compiling and running Java to another file - but it might not be worth it.

By the way, Moodle has a nice function print_object, that is like print_r, but does the echo '<pre>' bit for you, and also applies appropriate styles. You may find it convinient.

In grade responses, you should not do

if($question->options->displayoption == 0){

You should always do the grading work when that method is called. Whether the outcome is displayed or not gets decided later.

You should not be using global variables for returning data from the compile method. In compile you can do:

return array($compiles, $compileroutput);

and then when you call it, do:

list($compiles, $compileroutputstudent) = $this->compile($compilation_file);

similarly, you should not be using global variables to get data in. Just add more parameters to the method, if necessary.

The same comment applies to the execute method. Cleaning up those two methods would make grade_responses easier to understand.

Sorry, I have run out of time again. I will try to look again at print_question_formulation_and_controls this evening.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 27 June 2008, 4:24 AM

Further comments:

Looking at the editing form, I would not use addFormRule. I would just do any necessary validation in the validation() method.

And I still think it would be better to use textareas for people to input code. You can still ensure that it is nicely formatted on output.

I really think that this question type will work best in adaptive mode. If you do that, you don't need to worry about a separate compile button. You can just make it so that if the compilation fails, you don't add on any penalty.

Hmm. OK so having spent a lot of time looking at the code, and I found that the best way for me to understand it was to rewrite it. I don't know if that was helpful or not. Anyway, attached is the code after I finished hacking around with it. Note that I got rid of display.html, and instead just did all the output in print_question_formulation_and_controls.

Also, I am working here on a Mac laptop, and I have screwed up my PHP install on this laptop, so I have not been able to test the code at all. I am sure it is full of bugs, but hopefully if those bugs are fixed, it will then work.

Anyway, can you have a look and see if any of it makes any sense, and if (when) it doesn't, get back to me with more questions, or contact me via Skype chat.

so_junit_th.zip

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 27 June 2008, 7:17 PM

Yes, I definitely messed it up. I misinterpreted how the givencode and testclass fields were supposed to work. Sadly our development servers here don't have javac installed, so I can't do much more testing.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 27 June 2008, 8:35 PM

At the moment I am trying to fix all the bugs in your code in order to see if that is what I want. When you say that you misinterpreted how the givencode and testclass fields were supposed to work - do you say that your code won't work at all? Or what do you mean when saying this? Should I go on fixing your code or rather not because it won't be worth it?

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Wednesday, 2 July 2008, 7:00 PM

Hi Tim,
I have worked through your code which looks very clean - thanks.
It compiles and executes now.

What I don't understand is following (see attachment):
I have created a new Junit-question type and added it to a new quiz.
Quiz settings: students may review - everything checked; 1 attempt allowed, adaptive mode.
When I go to Preview-tab to try it out I have two scenarios for compilation (3. and 4.) and execution (1. and 2.) each:

1. typing code as response with correct compilation but inducing a failure in response
in order to receive execution output
-> then press "submit"
-> the execution gets displayed
-> correct
2. typing the same code as in 1.
-> then press "submit all and finish"
-> the student response-code and execution gets displayed
-> correct
3. typing code as response witch induces a compilation output
-> then press "submit"
-> the compilation output gets displayed
-> correct
4. typing the same code as in 3.
-> then press "submit all and finish"
-> student response-code and compilation output - both don't get displayed
-> wrong!!!

so_junit_08_07_02.zip

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Wednesday, 2 July 2008, 7:38 PM

Here are some example files to try it out. I will put them somewhere in my module later.

example_files.zip

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Wednesday, 2 July 2008, 8:22 PM

Note: Please remove 'public' in Factorial.java-file in order to allow several classes to be used in one JUnit-test-file.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Wednesday, 2 July 2008, 10:28 PM

OK, here is a version with a few changes to questiontype.php, styles.css and lang/en_utf/so_junit.php.

This seems to work for me in Moodle 1.9, and I can't see any significant differences between 1.9 and 1.8 that would cause problems.

so_junit_tjh.zip

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Wednesday, 2 July 2008, 11:56 PM

On Windows with MySQL it still doesn't work.
Just to make sure I have used all changes and maybe this is not a bug... here my last version with your changes.

so_junit_07_02_compilationissue.zip

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 3 July 2008, 7:14 PM

Hi Tim,

I have just finished installing Moodle 1.8.5+ and tried out case 4.
This time the student response gets displayed correct, but the compilation output is still not displayed. mixed

What do you suggest to do next?
Maybe we will need to figure out which change is responsible for letting display the student response. But the compilation output display must relay on something else then...

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 3 July 2008, 8:00 PM

Hi Tim,
three more question you may help me with:

1. Is it a good place for temp-folder in "\moodle\so_junit_temp\" or better in "\moodledata\temp\so_junit_temp\"?

2. Would it be better to delete the temp_folder at the end of grade_responses-function up from "$cfg_dirroot . '/so_junit_temp/"? This way this folder wouldn't get big. On the other hand the response and outputs get stored in the database (as far as I understand) so that it would be unnecessary to keep the same data twice. What would you suggest?

3. How to use validation-function without addFormRule? That is why I did it this way. What is the problem with addFormRule? Could you give an example for example for testFileSelected-function, please!

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Thursday, 3 July 2008, 9:39 PM

1. You should never write files under \moodle from within Moodle itself. It must be under \moodledata, and \moodledata\temp\so_junit_temp seems like a good choice within there.

2. It would be good to delete the temp folder when it is no longer needed. However, only delete the specific folder you created for that attempt, becuase another student may be attempting another question (or even the same) question at the same time. (Basically, at the start of grade_responses, you do

 $temp_folder = $cfg_dirroot . '/so_junit_temp/' . $state->attempt . '/' . $state->question;
 $this->mkdir_recursive($temp_folder);

So I think you should delete $temp_folder and everything it contains at the end of that method.

3. There is not really a problem with addFormRule, it is just not used very much elsewhere in Moodle, whereas validation is used everywhere, so this is really a question of style. Also, I don't understand addFormRule becuase I have never used it.

Oh, validation is another of those things that was fixed in Moodle 1.9. Also, I don't think I have ever done a form that needs to deal with file uploads so I am guessing a bit, but in 1.9 it would look roughly like this:

 //test if a file has been selected, otherwise reload the page and show a message
 function testFileSelected($data, $files){
 $errors = array();

 $testclassname = $data['testclassname'];
 $testclassnamefile = $files['testclassname'];

 if (/* Some test of $testclassname or $testclassnamefile*/ true) {
 $errors['testclassname'] = 'Please select a test class file!';
 }
 // Other tests if necessary ...
 return $errors;
 }

 function validation($data, $files) {
 $errors = parent::validation($data, $files);
 $errors = $errors + $this->testFileSelected($data, $files);
 return $errors;
 }

Hmm. That is probably not very helpful. Perhaps in 1.8 look for another form that deals with file uploads, and see what they do there. (How about the user profile that lets people upload images? No they don't seem to do any validation there!)

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 3 July 2008, 11:47 PM

1. yeah, that is what I thought
2. good thought - thanks!
3. Yeah, I couldn't find any other example in Moodle, that is why I tried it out with an example from the internet. I guess I will use it again the way I had it even though it doesn't match the Moodle-style. At least this way it works for now.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 4 July 2008, 5:13 PM

Does the following sound to be going to work this way with my understanding of "students may review"-settings in quiz settings?

The idea of what (compileroutput, executionoutput) to display at what time is as follows (works in adaptive mode and this way I would probably not need a separate selection in the teachers' editing form):
compileroutput: display always (independent of the selection of feedback)

=> interesting for exams, exercise-sheets with grading to turn in weekly and exercises without grading

executionoutput: if feedback is selected

1. display "after quiz is closed" -> teacher and student can see executionoutput after the quiz in gradings

=> exam, exercise-sheets with grading to turn in weekly and exercises without grading (will the last case be possible to be seen this way?)

2. display "immediately after the attempt" -> student can practise

=> exercises without grading

For the exercises without grading - case it may be better to set in quiz settings "unlimited attempts" - right?

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 4 July 2008, 9:59 PM

I think I have managed realizing it now this way:
compileroutput: display always (independent of the selection of feedback)
=> interesting for exams, exercise-sheets with grading to turn in weekly and exercises without grading
executionoutput: display allways after quiz taken (also in grades? -> test)
=> exams, exercise-sheets with grading to turn in weekly
display "immediately after the attempt" -> to student only if feedback is selected in quiz options
=> exercises without grading

The one thing that bothers me now is that "the grading details from the last graded state" get displayed in question/type/questiontype.php when using the Submit-button. At this stage I don't want to show the students their grade (called "Scores" in quiz-settings).
Unchecking all "Scores" in the quiz settings still show the scores for one submission in adaptive-mode.
Is this a bug and should only be shown when one of the "Scores" has been checked in the quiz settings? If it is not a bug - can I change this for my question type in my question type module-folder (if yes how) or will I need to change it in question/type/questiontype.php?
(e.g. with a separate if-branch (or would there be a more elegant way?):
if (question_state_is_graded($state->last_graded) and (($question->qtype == 'so_junit') and (!isset($_POST['resp540_submit']))) )
{//do not display grades when "Submit"-button is pressed but when "Submit all and finish"-button is pressed}

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Sunday, 6 July 2008, 7:04 AM

It is not a bug, in that when Gustav Delius implemented adaptive mode, they chose to make it work that way. However, it is debatable whether that is how it should work.

You should be able to change it for your question type by overriding the print_grading_details method.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Monday, 7 July 2008, 4:34 PM

Hi Tim,
I just got confused, because I didn't check the grading "immediately after attempt" and still got grading shown.
Oh yeah, that is right - I just need to override the named method above - very easy - great and thanks!

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Monday, 7 July 2008, 6:24 PM

Hi Tim,
I have a couple more questions and would appreciate your help!
1.
So far I had left following methods like they were given through the question type template. Can I simply delete them or will I need to leave them like suggested through the question type template?
create_session_and_responses
get_correct_responses
get_all_responses
get_actual_response

2.
In editing quiz view when wanting to "Preview" an already created Junit-question "Type" in the "Action"-column the wrong $COURSE->id is used (in my case it should be "6" but is "1"). Therefore it can't find my test-file in the $course_files_directory and doesn't compile. Do you have the same problem or any ideas why this happens?
In the quiz-preview or when taking the quiz it is working properly.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Monday, 7 July 2008, 9:10 PM

1.
create_session_and_responses - You only need this one if you have some initialisation to do. At the moment, the question types that use this are the ones that do randomisation. You don't need it, so you can delete it.
get_correct_responses, get_all_responses, get_actual_response - You will need these if you want the reports, particularly the Item Analysis and Detailed Responses reports to work.

2.
Are you still in 1.8? If so, I think this may be because question/preview.php just calls require_login(), rather than require_login($courseid, ...). If that is the case, it is a minor bug. File an issue in the tracker, and I will fix it. If that is not the cause, I will think again.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Monday, 7 July 2008, 10:56 PM

1.
aha, thanks! I will work through the methods to get working the reports.

2.
Yes, I am still using 1.8.2+.
You were right, in question/preview.php there is written require_login();
But I don't get it working properly - this way e.g. I get back the error-message. Do you see what I am missing?

$courseid = optional_param('courseid',PARAM_INT);
if (! $course = get_record("course", "id", $courseid)) {

error("Course does not exist!");

}
require_login($course->id, false);

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Monday, 7 July 2008, 11:37 PM

The problem is that there is no &courseid=123 parameter in the URL, so optional_param won't find anything.

In 1.9, the require_login call is moved to the if (empty($quizid)) bit lower down, like this.

 if (empty($quizid)) {
 $quiz = new cmoptions;
 $quiz->id = 0;
 $quiz->review = $CFG->quiz_review;
 require_login($courseid, false);
 $quiz->course = $courseid;
 } else if (!$quiz = get_record('quiz', 'id', $quizid)) {
 error("Quiz id $quizid does not exist");
 } else {
 require_login($quiz->course, false, get_coursemodule_from_instance('quiz', $quizid, $quiz->course));
 }

However, as we have found, there is no $courseid variable here in 1.8. Instead you would need to move this if statement down to where it currently says $quiz->course = $category->course, and use $category->course insteadof $courseid.

I think the attached patch does the right thing. Can you apply it?

1.8_q_preview.patch.txt

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 11 July 2008, 11:16 PM

Hi Tim,
finally I have installed my junit-module successfully on the institutes Moodle 1.9 Beta 4 version. (Attached the last version that should work on Windows and Linux operation systems)

My first notices:
1.
In the Quiz-Preview and after a student has taken a Quiz (means: After pressing "Submit all and finish"-button) the compiler_output and student_response is still not displayed (execution_output still works fine). In Question-Preview it gets displayed correct though!!!

2.
a)
I have created a Quiz with e.g. 3 Junit-questions (displaying each on one page). I don't understand the sequence when the following happens exactly, but you may try out a little - e.g. Give a response to the first question, press Submit-button; give response to the second and third questions as well and pressing the Submit-buttons (or not). Then go back to the first question and submit it again. Sometimes going back to the third question the already made response gets lost. Sometimes the response get lost after using the "Submit all and finish"-button.

b)
Displaying all three questions on one page seems to keep the responses, but after e.g. giving response and submit to the third question, then to the second + submit, then to the first + submit, then changing the first + submit -> removes the output of the second question.

Does this has to do with a function I may have forgotten to implement or does this seem to be a general quiz-issue?

3.
a)
In Quiz-Preview and after a student has taken the quiz - the grades don't get displayed (correct, partially correct or incorrect) for any question at all. Is this meant to be like this?
b)
In Question-Preview it gets displayed (but in case of 'incorrect' incorrect is not highlighted).

sojunit_08_07_11_moodle19beta.zip

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Thursday, 17 July 2008, 11:39 PM

We talked about 1) on Skype. It seems to be an addslashes/stripslashes problem.

2) The sequence of actions can be a bit strange until you are used to it. You need to look in the mdl_question_states table as you work through the quiz (ORDER BY attempt, question, seq_number). You will see that there are a number of rows for each question, with different values in the event column. Those event numbers come from the top of /lib/questionlib.php. The important ones are 0=open, 2=save, 3=grade, 6=closeandgrade.

3) Should depend on the quiz settings.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 18 July 2008, 12:13 AM

2) Does this mean that I will have to live with these "inconsistent" behaviors or can/should I make changes in the source code (I hope to receive access to phpMyAdmin soon in order to follow your description above for a better understanding)?

3.a) No - I have checked everything in the quiz-options->Review options. So there must be another reason why this doesn't work. It must be something special to my junit-question-type since it is working for other question-types. So I must be missing something in a method I guess - but where?

3.b) This one seems to be a general bug also for other question types (in question-preview as well as in quiz-preview).

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 18 July 2008, 3:42 AM

2) What I am really saying is when I am trying to understand weird goings on in the quiz and questions, watching what goes on inside that database table is one of the most useful ways to pin down the problem. I would suggest that you make a quiz with some questions of your type, and some shortanswer questions (I think shortanswer is probably the most similar of the standard question types to your one) and do the same actions to each question, and see if you get the same sort of states in the database.

That should help you find the inconsistency. There is no reason why it should be inconsistent.

3a) I'll have another look at your code tomorrow. I'm sorry I had to rush away earlier today.

3b) Ah, I think this is MDL-12772.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 18 July 2008, 9:15 PM

3.b) Ah, good to know - thanks. In my local installation I had been using standardlogo-theme. Looks like this bug is only in standard-theme.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Thursday, 24 July 2008, 11:33 PM

Hi Tim,

2)
I have created a quiz with 5 questions (order: junit1, short-answer1, junit2, short-answer2, junit3) tried out to follow the database entry changes. I have realized following:

seq_number-field:

1. for short-answer the seq_number of e.g. short-answer1 increases only when I submit exactly that question. For junit the seq_number of e.g. junit1 increases with each submit made, independent of which question has been submitted.
2. submitting a question without changing the response the seq_number in short-answer doesn't change. For junit it increases (=>can this have to do sth. with the compare_responses-method you mention in "by Tim Hunt - Friday, 18 July 2008, 04:41 PM"?)

event-field:

once a regexp question has been submitted it keeps on staying at event 3 (=grade) when submitting other questions whereas in a junit question once it gets event=3 and as soon as other submits for other questions are made it changes to event=2.

The junit-outputs which are generated through the use of the submit-button disappear after any other submit-usage for other questions. Does this tell you what needs to be done to get the junit-question working properly? Maybe solving this would also remove the similar issues when the questions are on one page each, instead of all on one page - like it was for this anlysing?

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 25 July 2008, 2:42 AM

Ah, the base class compare_responses will not work, because it compares the $state->responses array, and we have added the '_compileroutput' information, and so on, there. You need a version of compare_responses that just looks at the student's input. The implementation in the shortanswer question type might give you some ideas.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 25 July 2008, 7:43 PM

yeah, great - I think now it is working properly - yipieee.
I have written the following:

function compare_responses($question, $state, $teststate) {
if (isset($state->responses['']) && isset($teststate->responses[''])) {
if ($state->responses[''] == $question->options->givencode){
return true;
}
else{
return strcmp($state->responses[''], $teststate->responses['']) == 0;
}
}
return false;
}

As you can see I didn't need to compare only the responses without the compiler and execution outputs, because here they are not in the $state->responses[''] or in $teststate->responses[''] included.
But later I have realised that I need to check if it is really a student response or only the givencode of the teacher so that initially the event-field won't set 2 for all junit-question-types.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 18 July 2008, 8:00 PM

3.a) The reason the grading details are not showing up is becuase you have overridden the print_question_grading_details method. If you delete that method from your class, so it uses the default implementation instead, then the grade shows up just like for other question types. (I have not look at the details of how your p_q_g_d differs from the standard one, so I don't know if there was a good reason for your changes - I just found that this seemed to be the cause.)

I'll post more about 2) later.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 18 July 2008, 9:13 PM

3.a) The only change I have made to the print_question_grading_details-method is that I changed:

if (question_state_is_graded($state->last_graded) )
into
if (question_state_is_graded($state->last_graded) and (isset($_POST['finishattempt'])) )

This way I wanted to avoid that the grades are shown after using the submit button (e.g. in an exam or graded exercise) before handing in the answer.

I don't see why this shouldn't work - do you have any idea?

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 18 July 2008, 10:15 PM

People may go away from the quiz, and come back later. So it is no good relying on transient information like $_POST. When deciding what to display on-screen, you should only rely on what is in the database, which really means what is in $question and $state.

You can probably achieve what you want by changing question_state_is_graded to question_state_is_closed, but I am not sure it makes sense to do this in just one question type. Surely you want all question types to work in the same way in your quizzes?

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Friday, 18 July 2008, 10:51 PM

Using question_state_is_closed does, what I want - that is good.

At the moment I didn't need to use a quiz in adaptive-mode so I guess this setting will be special for the use of JUnit-question types.
Of course it would be nice and better to have the possibility to mix them with other question types, but then the students could see if their responses are correct or not and I would not like this in e.g. a graded-exercise.

You had suggested to use the adaptive-mode in order to get rid of the extra compile- and execute-buttons and use the Submit-button instead - do you remember what I am talking about?

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Saturday, 19 July 2008, 3:30 AM

Yes, I do remember making that suggestion. It has contributed to some thinking I have been doing.

At the moment, the way individual questions work - that is adaptive or non-adaptive mode, is controlled by the quiz. (Well, it is controlled by a setting in the quiz, and implemented in questionlib.php). I am becoming more and more convinced that this should instead be controlled by the individual question types. For example, adaptive mode does not make sense for essay questions. Having some sort of compile button would make sense for your question type ...

On the other hand, it is quite nice that you can re-use the same question in both adaptive and non-adaptive mode for most of the core question types, and it would be a pity to lose that. However, we have done some work at the OU that lets you provide additional feedback for questions in adaptive mode, and I would like to incorporate that into Moodle 2.0. Entering that additional feedback is only relevant to questions that will be used in adaptive, and just clutters up the editing interface for non-adaptive questions.

There must be a good solution to all this. I am still thinking.

Average of ratings: -

Re: junit-question-type

by Süreç Özcan - Monday, 21 July 2008, 11:09 PM

Seems to be a more general thinking issue...

At the moment the junit-question-type makes more sense to be used in adaptive mode, because of the usage of the submit-button. Actually a compile-button would be a copy of the current submit-button used in adaptive mode.
I think I will leave the question like it is at the moment - but would it be possible to implement a compile-button at all or is there rather a limitation due to the current architecture - or would it be just too complex to do?

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Tuesday, 22 July 2008, 1:37 AM

I think it would be possible, but quite difficult, to implement a compile button at the moment. If I get time to do all the improvements I would like for Moodle 2.0, then it would become much easier.

Average of ratings: -

Re: junit-question-type

by Tim Hunt - Friday, 18 July 2008, 10:41 PM

2) The key thing to keep in mind is that whenever you do anything with a page of the quiz, it has to process everything on the page, to make sure that the students responses to all questions on the page are saved.

What happens is that in attempt.php, it looks at the data that was recieved, and sorts it into the stuff that belongs to each question ($actions = question_extract_response(...). Each of these actions has a -> event field that will be one of QUESTION_EVENTSAVE, QUESTION_EVENTSUBMIT or QUESTION_EVENTCLOSE. (There is also QUESTION_EVENTOPEN, which is used when a quiz attempt is started.)

QUESTION_EVENTCLOSE is used when 'Submit all and finish' is pressed.

QUESTION_EVENTSUBMIT is only used in adaptive mode, when the student does Submit on a particular question, or Submit page.

QUESTION_EVENTSAVE is used for all other cases, that is, when the user navigates to another page, or when they do Save without submitting, or for the other questions on the page when they submit just one question in adaptive mode.

$action->event affects how question_process_responses updates $state. What is perhpas a little suprising is that question_process_responses actually calls qtype::grade_responses for all types of events. This is because you may want to do some processing of the students responses after they have been received, and before they are stored in the database. So actually the grade_responses method might better be called process_responses. Of course, on a SAVE event, the quiz does not do anything with the score computed (although it is stored in the database).

The other thing to note is the bit where question_process_responses calls qtype::compare_responses to see whether the the response has changed from the previous state. If the response has not changed, then actually it avoids repeating calculations it has already done.

So to summarise, whenever the quiz is submitted you get an action for each question on the page. This is processed by question_process_responses, which always calls qtype::grade_responses whatever type of action it receives - unless it can detect that the responses are identical to some previous responses it processed.

Average of ratings: -