Problems with cloze question scoring in adaptive mode

Problems with cloze question scoring in adaptive mode

by Catherine Berry -
Number of replies: 5

I'm trying out Cloze questions for the first time, in Moodle 2.1.

Here's my question:

The solution of the simultaneous equations

x + 3y = 5

3x - y = 5

is x = {3:SHORTANSWER:=2} and y = {3:SHORTANSWER:=1}.

I have it set up with two hints and adaptive mode with penalty of 33.333%, so that the question has a total of 6 marks reducing to 4 on the second attempt and 2 on the third.

If I get one part right and the other wrong and just press 'Submit' without using 'Check' first, it gives 3 marks as expected.

If I get one part right and one wrong, and press 'Check', hovering over my answers tells me that I have 3 marks for one part and 0 for the other. If I then decide to submit it anyway, it gives me 1 mark and 'Partially correct' - but on the review hovering over still says 3 marks and 0 marks.

If I get one part right and one wrong, and press 'Check' and then have another go, press 'Check' again (still one right and one wrong) and then submit without another try, I get 0 marks and Partially correct - again hovering over on the review gives 3 marks and 0 marks.

If I give both answers correct at any stage, the marking is correct - it gives 6 marks if it is the first attempt, 4 marks if it is the second attempt and 2 marks if it is the 3rd attempt.

So there seem to be two issues here - one that it isn't giving the correct marks for partially correct answers, and secondly that the hovering over behaviour doesn't reflect how many attempts there have been - it always says 3 marks for a right answer even when 3 marks won't be awarded because it's a second or third attempt.

Have I done something wrong, or is there a bug?

Thanks.

Average of ratings: -
In reply to Catherine Berry

Re: Problems with cloze question scoring in adaptive mode

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

@Catherine, are you testing this question on its own (Preview question) or inside a Quiz?

I have tried to apply your scenario but do not get the same findings. Tested in the Preview question window (teacher).

a) If I get one part right and the other wrong and just press 'Submit' without using 'Check' first, it gives 3 marks as expected.

OK

b) If I get one part right and one wrong, and press 'Check', hovering over my answers tells me that I have 3 marks for one part and 0 for the other. If I then decide to submit it anyway, it gives me 1 mark and 'Partially correct' - but on the review hovering over still says 3 marks and 0 marks.

No, it gives me Marks for this submission: 3.00/6.00 and Partially correct.

c) If I get one part right and one wrong, and press 'Check' and then have another go, press 'Check' again (still one right and one wrong) and then submit without another try, I get 0 marks and Partially correct - again hovering over on the review gives 3 marks and 0 marks.

No, it gives me Marks for this submission: 3.00/6.00 and Partially correct.

If I give both answers correct at any stage, the marking is correct - it gives 6 marks if it is the first attempt, 4 marks if it is the second attempt and 2 marks if it is the 3rd attempt.

OK

In conclusion, I do not see the problem you are reporting.

Joseph

EDIT.- Forgot to mention I'm using  Moodle 2.1.3+ (Build: 20111209)

Average of ratings: -
In reply to Catherine Berry

Re: Problems with cloze question scoring in adaptive mode

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Note that there were some bugs in adaptive mode that have now been fixed (Thanks Henning), so you should upgrade to Moodle 2.1.3, or 2.1.3+ - perhaps even wait until next week when the fix for MDL-29772 is available.

Average of ratings: -
In reply to Tim Hunt

Re: Problems with cloze question scoring in adaptive mode

by Catherine Berry -

Tim, thanks for the information. I'll upgrade at some stage and see if that fixes the problem.

Joseph, I tried both on its own and within a quiz. If you are using a later version of 2.1 then that would suggest that one of the bugs that Tim mentions is causing the problem. Thanks for taking the time to check it out.

Average of ratings: -
In reply to Catherine Berry

Re: Problems with cloze question scoring in interactive mode

by Catherine Berry -

I have now upgraded to 2.2.2 but I am still having problems with strange scoring of Cloze questions in interactive mode (sorry, I put adaptive mode in my earlier post, which could explain Joseph's different results).

I have a quiz containing three questions, similar to the one in my earlier post, each with two hints.

In the screenshot, I've attempted the first question once, the second question twice and the third question three times, in each case getting one part right and one wrong. I pressed Check after each attempt.

Hovering over the right answer in all cases says it gets 3 marks (even though where there has been more than one attempt, 3 marks shouldn't be available).

Here's the summary:

and here's the review:

These should be getting 3 marks, 2 marks and 1 mark respectively.

Hovering over the right answer again says 3 marks in all cases, but the wrong answer says 'Not answered'.

I then attempted the quiz again, but this time I didn't use the 'Check' button for the answer which I was going to submit. So for the first question I didn't press the Check button at all, for the second I pressed it for the first attempt and then entered a second attempt and didn't press it, similarly I didn't press it for the final attempt at the third question.

Here's the page:

Here's the summary:

and the review

So this gives different marks - the marks for the first question are right but not for the second and third.

So there seem to be several problems:

  1. Hovering over answers gives the full amount of marks, regardless of which attempt it is - this is misleading as it tells a student they have 3 marks even for a third attempt which would score 1. In adaptive mode there is at least mention of the penalty.
  2. Sub-questions are marked as 'Not answered' when they have been answered incorrectly (I've seen this mentioned on other threads too).
  3. It seems that a question is not considered to be complete unless all possible attempts have been used (unless the question is completely correct). Students may quite legitimately decide not to take another attempt which would carry a penalty.
  4. The behaviour is different according to whether or not the 'Check' button is pressed after an attempt.
  5. Marks awarded are wrong!

Sorry for the very long post - but I wanted to try to make the issues clear.

Thanks.

Average of ratings: -
In reply to Catherine Berry

Re: Problems with cloze question scoring in interactive mode

by Catherine Berry -

Just to add - having looked at Adaptive mode as well (in which it talks about penalties), I think I can see what is causing some of the problems. In the second example, I think a penalty of 2 marks has been applied to the whole question in the second attempt, so 3 - 2 = 1 mark, and then a penalty of 4 for question 3, which would give -1. My intention is that each part of the question has 3 marks, so on the second attempt each is marked out of 2 for each part and on the third attempt it is marked out of 1 for each part. It's the way the penalties are applied that seems to be the problem.

In the first example, I think it assumes that because 'Check' has been pressed, a second attempt has been made, even though 'Try again' hasn't been pressed. So it is applying the penalty of 2, and then on the second and third questions the marks are negative.

Average of ratings: -