Should we change the adaptive behaviour in 2.1?

Should we change the adaptive behaviour in 2.1?

by Tim Hunt -
Number of replies: 11
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Henning has made some improvements to how adaptive behaviour displays the scores in MDL-29772, and we are definitely going to apply those changes to Moodle 2.2.

The question is, should we make these changes in Moodle 2.1?

On the one hand, the new behaviour is definitely better.

On the other hand, it is a change of functionality, and we are not meant to change functionality on stable branches.

To put it another way, we could have an argument about whether this counts as a 'bug fix' or a 'feature change'. Would any of you who use adaptive mode like to comment?

Average of ratings: -
In reply to Tim Hunt

Re: Should we change the adaptive behaviour in 2.1?

by Richard Lobb -
Picture of Plugin developers

If I implement the one liner given in MDL-29772 I get the following result on a question with a 25% penalty that asks "What's half of 8", when I enter successively 3, 4, 5:

Incorrect
Marks for this submission: 0.00/1.00. This submission attracted a penalty of 0.25.
Total penalties so far: 0.25

----------------------------------------------------

Correct
Marks for this submission: 1.00/1.00. With previous penalties this gives 0.75/1.00.
Total penalties so far: 0.5

----------------------------------------------------

Incorrect
Marks for this submission: 0.00/1.00. With previous penalties this gives 0.75/1.00.
Total penalties so far: 0.75

----------------------------------------------------

At very least the code should check if the current answer is correct, rather than multiplying the penalty by the try-number. Moreover, the behaviour if a student continues to try different answers after having entered a correct answer needs to be considered, too. I use a subclass of adaptive behaviour in my Python  programming question plug-in, and there it makes perfect sense for a student who has found a cumbersome but technically correct solution to try a shorter  more elegant solution. It doesn't make sense to see the penalty clocking on upwards in such a case. So perhaps the "Total penalties so far" line should be suppressed once a student has entered a correct response? With that change, I'd be happy to see the change retrofitted to 2.1.

-- Richard

In reply to Richard Lobb

Re: Should we change the adaptive behaviour in 2.1?

by Itamar Tzadok -

Something doesn't look right in the logic of this adaptive mode. What happens if the student submits and finishes after the third attempt? If the student looks only at the end result s/he may get the impression that marks have been awarded for an incorrect answer.

At any rate, I suppose any format would be confusing one way or the other. Building on Richard's example, my suggestion would be:

Incorrect
Current mark for this question: 0.00/1.00
Marks for this submission: 0.
Penalties for this submission: -0.25 (accumulated: -0.25).
----------------------------------------------------

Correct
Final mark for this question: 0.75/1.00
Marks for this submission: 1.0.
Penalties for this submission: 0 (accumulated: -0.25).

----------------------------------------------------

Incorrect
Final mark for this question: 0.75/1.00
Marks for this submission: 0.
Penalties for this submission: 0 (accumulated: -0.25).

smile

In reply to Itamar Tzadok

Re: Should we change the adaptive behaviour in 2.1?

by Richard Lobb -
Picture of Plugin developers

I quite like that Itamar. But perhaps have "Penalty for this submission..." rather than "Penalties..."? I also wonder whether the "Marks for this submission" line should come before the "Final mark ..." line, though I can see benefits both ways.

A further problem (and this is where my head starts spinning): the quiz allows you to set different grading methods: Highest grade, Average grade, First attempt, Last attempt. Your proposal looks great for "Highest grade" but I think you'd need different feedback patterns for the other types of grading. And in the case of "First attempt" grading, there seems to be a fundamental inconsistency in the design: it doesn't make much sense to talk about "penalty for each attempt" then, does it?

-- Richard

In reply to Richard Lobb

Re: Should we change the adaptive behaviour in 2.1?

by Itamar Tzadok -

Should definitely be "Penalty for ...". As for the place of "Marks for this submission" I think that overall the mark of the question would be more important for most quiz takers and so should be at the top.

I don't actually use the per question adaptive mode so I'm not sure how it functions under the different grading methods and what would be regarded as a quiz attempt. It is weird and I'm not trying to understand it. My standard approach, which may also be called adaptive, is quizzes opened for a whole week, unlimited attempts, each attempt builds on the previous one, and quiz score shown. So students have plenty of time to keep trying until they get everything right. (It would have been nice if the a new attempt which builds on the previous could show which previous responses were correct; perhaps it does that in 2.1; I'll find out next term). I also use all-or-nothing in all multichoice cases to counter the temptation to systemtically guess your way to the correct answer. smile

In reply to Richard Lobb

Re: Should we change the adaptive behaviour in 2.1?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Sorry, I just assumed that everyone already knows how the tracker / moodle development process works. 

That one-liner is not the point. You need to be looking at this change: https://github.com/bostelm/moodle/compare/master...MDL-29772

In reply to Tim Hunt

Re: Should we change the adaptive behaviour in 2.1?

by Richard Lobb -
Picture of Plugin developers

Sorry about the wasted bandwidth there. The new code is, as you say, definitely an improvement.

>>To put it another way, we could have an argument about whether this counts as a 'bug fix' or a 'feature change'. Would any of you who use adaptive mode like to comment?

I use adaptive mode almost exclusively and I'm inclined to call this a bug fix rather than a feature change.

Richard

In reply to Tim Hunt

Re: Should we change the adaptive behaviour in 2.1?

by Oleg Sychev -
Picture of Core developers Picture of Plugin developers

I don't like the idea  of moving this back for 2.1 unless really necessary, sorry.

I'm thoroughtly exhausted by rewriting Preg question type for 2.1 and providing necessary behaviours for hinting. Last thing I need right now is to deal with changes in adaptive behaviour and looking whether I should update my behaviours accordingly - and it seems I should (so better give this at least a minor version, not just weekly release - but having to support two minor releases of 3d party behaviours ... oh my sad  ).

And overall "tries" scheme is bad since it isn't scale well when adding other penalty (cost) types.

You could see more robust version of adaptive based on "totalpenalty" behaviour variable instead of just "tries" on http://code.google.com/p/oasychev-moodle-plugins/source/checkout (warning - repository using Mercurial instead of Git), look for /question/behaviour folder.

In reply to Oleg Sychev

Re: Should we change the adaptive behaviour in 2.1?

by Henning Bostelmann -
Picture of Core developers Picture of Plugin developers

Just wanted to add one point, in case there are misunderstandings around: The intent of MDL-29780 is not to change the way in which penalties are computed, or the way in which the backend question behaviour works. (This certainly shouldn't be done in a stable branch, if at all.) The intent is to improve the way in which the information is being displayed to the user. In parts it is certainly bugfixing (see point (3) in the tracker entry) and in parts it may be a new feature rather than fixing a bug - that's a bit up to interpretation.

In reply to Henning Bostelmann

Re: Should we change the adaptive behaviour in 2.1?

by Oleg Sychev -
Picture of Core developers Picture of Plugin developers

Yes, Henning, I looked at changes and was able to see that you don't change way in which penalties are computed. But the matter is more complex.

My point is exactly opposite: if anyone (like me or Joseph Rezeau) did change the way penalties are computed in backend behaviours, they should change their behaviours accordingly, or total penalty would be showed wrong. That leads to the mess, which is much easily sorted out by having them in different major versions.  I'm sorry if I look like an enemy of the progress wink - I know you position well, I've been in it myself - but this is real problem.

Unfortunately the penalty computation in adaptive mode isn't designed well and doesn't scale when you add another types of penalties. But this problem isn't easily solved because it requires major DB upgrade. Thought problems like this convince me more that it is actually necessary.

P.S. I didn't have anything against fixing stuff like MDL-29780 in Moodle 2.1  Just don't add total penalty display to it.

In reply to Oleg Sychev

Re: Should we change the adaptive behaviour in 2.1?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

It sounds like we are reaching a consensus that we should ask Henning to do a bug-fix-only change to deal with MDL-29780 for 2.1, and apply the full changes to 2.2+ only. Thanks everyone for your input.

In reply to Tim Hunt

Re: Should we change the adaptive behaviour in 2.1?

by Oleg Sychev -
Picture of Core developers Picture of Plugin developers

Tim, is it correct to calculate total penalties right in the renderer? And don't use adjusted_fraction method too. 

I expect calculating something to be a job of behaviour (or question), not renderer. And separate function, which could be overloaded if you want to change model without view or view without model. Calculating total penalty in the renderer, as it's done in these changes, violate MV architecture.