Penalties for cloze questions in interactive mode

Penalties for cloze questions in interactive mode

by Catherine Berry -
Number of replies: 7

I've already posted about some problems with scoring of Cloze questions in interactive mode here. I think I now understand the problem a little better, so I'm starting a new thread in the hope that someone can help.

I think there are problems with how penalties are applied to Cloze questions in interactive mode. I have used a question with 2 parts, each with 3 marks and a 33.333 penalty. So correct on the first attempt scores 6, correct on the 2nd attempt scores 4 and correct on the 3rd attempt scores 2. The problem occurs when only one part is right. If an attempt with one part correct is submitted at the second attempt, I think that this attempt should be scored out of 4, 2 marks for each part, and so should score 2 + 0 = 2. However, what seems to happen is that the correct part scores 3 but then a penalty of 2 is applied, giving 1 mark. If it is the 3rd attempt, I think that it should be scored out of 2, 1 mark for each part, and so should score 1 + 0 = 1. Instead, it takes a penalty of 4 from the 3 marks for the correct answer, giving 0.

My knowledge of php is quite limited, but I've had a look in the code. I noticed that in question\behaviour there is a folder called interactivecountback, as well as one called interactive. The file interactivecountback\behaviour contains an explanation of what it does which appears to be the behaviour I am looking for (although the example is for a matching question). I'm not sure whether this behaviour would work for Cloze questions as well, and I haven't been able to locate the code which calls the appropriate behaviour.

Can anyone help?

Average of ratings: -
In reply to Catherine Berry

Re: Penalties for cloze questions in interactive mode

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Catherine,

I suppose you are using Moodle 2.1 (or 2.2)?

Unfortunately you do not say which question type are the 2 sub-questions in your Cloze question (Multiple Choice or shortanswer or matching, etc). I suggest you attach your Cloze question to your reply, so we can test it out and understand better your problem.

Joseph

Average of ratings: -
In reply to Joseph Rézeau

Re: Penalties for cloze questions in interactive mode

by Catherine Berry -

Sorry - I'm using Moodle 2.2.2.

The sub-questions are shortanswer, although I've tried numerical as well, with the same results. Here is the question:

The solution of the simultaneous equations

3x - y = 7

2x + 3y = 1

is x = {3:SA:=2}, y = {3:SA:=-1}

Average of ratings: -
In reply to Catherine Berry

Re: Penalties for cloze questions in interactive mode

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Your konwledge of PHP does not seem to be that limited. I am impressed with the detective work you have done so far.

As you have spotted, a good example is qtype_match, though some of the necessary bits of code are hidden in the base class.

So, have a look at https://github.com/moodle/moodle/blob/master/question/type/questionbase.php#L631 and https://github.com/moodle/moodle/blob/master/question/type/match/question.php#L241

So, you are definitely going the right way, and if you want more help, please ask.

Can I also point out that it it is quite easy to write unit tests to verify the expected behaviour. For example https://github.com/moodle/moodle/blob/master/question/type/match/tests/walkthrough_test.php.

Average of ratings: -
In reply to Tim Hunt

Re: Penalties for cloze questions in interactive mode

by Catherine Berry -

Thank you Tim. I am probably going to convince you of my limitations now...

I've had a look at the two files you refer to. Then I looked at the type\multianswer\question.php file and optimistically tried changing line 42 'question_graded_automatically' to 'question_graded_automatically_with_countback'. This resulted in an error:

PHP Fatal error:  Class qtype_multianswer_question contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (question_automatically_gradable_with_countback::compute_final_grade) in D:\inetpub\wwwroot\resources_2.2\question\type\multianswer\question.php on line 266

Having browsed around a bit, it looks as if in most question types the penalty is dealt with within the appropriate behaviour.php file in question\behaviour. But for matching questions, it is used in the compute_final_grade function within the matching\question.php file. So I guess something similar would be needed within the multianswer\question.php file - but I have no idea how to do this.

Average of ratings: -
In reply to Catherine Berry

Re: Penalties for cloze questions in interactive mode

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Well, you are still on the right track. The point is that in normal situations, the behaviour does all the calculations involving penalties, but in the more complex case like matching and multianswer, then the question type has to do the final calculation a the end, and that is what the compute_final_grade method is supposed to do.

So, what you need to do is

1. change 'question_graded_automatically' to 'question_graded_automatically_with_countback', as you already did.

2. Copy compute_final_grade from qtype match, or somewhere, and change it so that it does the right thing for multianswer.

Working out what is 'the right thing' is the heart of the problem smile

To give some clues, if you look at grade_response, then it basically loops over all the subquestions, and gets them to grade the relevant bit of the response. I wonder if it is possible to do something similar in compute_final_grade? E.g. http://pastebin.com/T8iy34YJ. There are two comments in there explaining (very cryptically) what that will not work, but something like that ought to be the right way to do it.

Average of ratings: -
In reply to Tim Hunt

Re: Penalties for cloze questions in interactive mode

by Jean-Michel Védrine -

Hello Catherine,

I remember having learned a lot of things when I looked at the compute_final_grade method of the oumultiresponse question type because grade computation uses history of student responses. I am not quite sure it will help you, but maybe you can have a look at it ?

You can find it here : https://github.com/moodleou/moodle-qtype_oumultiresponse/blob/master/question.php

You need to look at the grade_computation method because the real work is done here.

Average of ratings: Useful (1)
In reply to Jean-Michel Védrine

Re: Penalties for cloze questions in interactive mode

by Catherine Berry -

Thanks Tim and Jean-Michel.

I've had a bit of a re-think, and done a bit more investigation. It seems to me that this problem applies to any question for which responses can score part marks, not just Cloze questions. The problem is that the penalty is being subtracted from the score gained from the question, rather than being used as a multiplier.

I found that in question\behaviour\interactive\behaviour.php, the function adjust_fraction (line 211) contains the line

$fraction -= ($totaltries - $triesleft) * $this->question->penalty;

If a question has 6 marks, and there are 3 tries with penalty 1/3, and the student scores full marks on the 3rd try, then $fraction is initially 1 (for full marks) and the effect of this line is to subtract 2/3 from 1, giving 1/3 and so the question score is 2. However, if the student scores half marks on the 3rd try (which could be getting part of a Cloze question right, but it could also be obtained from any response that gains half marks in any type of question), then $fraction is initially 1/2 and the effect is to subtract 2/3 from 1/2 giving a negative number - the next line sets this to zero.

I have changed this line to

$fraction = $fraction * (1 - ($totaltries - $triesleft)* $this->question->penalty);

In the scenario above, if the student gets full marks on the 3rd try then the effect of this line is to multiply 1 by 1/3, giving 1/3 and so the question score is 2 as before. If the student scores half marks on the 3rd try, the effect is to multiple 1/2 by 1/3, giving 1/6 and so the question score is 1, as I think it should be.

Of course, this doesn't address the 'countback' idea that is covered by matching questions, but I'm not too concerned by that.

Average of ratings: -