Moodle in English: STACK Question tests: Modified behaviour?

STACK Question tests: Modified behaviour?

by Martin Kraska - Wednesday, August 26, 2020, 12:26 AM

Hello, I seem to get problems when running the question tests of from STACK version 2018080600 (that's the version ID from the question test page) in STACK version 2020061000 (this was updated in the course of switching from Moodle 3.8 to Moodle 3.9).

I have test cases with partially empty inputs, with expected score empty and answer note set to NULL. This happens if there are more answer variants for some inputs than for others to test.

In 2018080600 this worked (tests passed)

In 2020061000 such tests fail.

For example, the following test seems to have returned answer note NULL and now returns False on empty input.

ATAlgEquiv(EMPTYANSWER, true) = [0, "ATAlgEquiv_SA_not_logic."];

It seems that handling of empty answer is test-dependent now. This requires attention in question tests for inputs where I actually don't intend to test anything.

Is there a way to de-activate the test for particular inputs in a question test?

questions-T-B2-MB-TM2-Hauptträgheitsmomente-20200825-1825.xml

Re: STACK Question tests: Modified behaviour?

by Christopher Sangwin - Thursday, August 27, 2020, 12:54 AM

I've had a look at your question.

You have "allowempty" in the input S_Kreis, so when you test this case you don't get a "NULL" (empty) answer, but you get the special tag EMPTYANSWER passed to the PRT. So, we have actually fixed a previous bug in the testing procedure and you can now test the case of "allowempty". I think you should change the expected test outcome here!

Average of ratings: -

Re: STACK Question tests: Modified behaviour?

by Martin Kraska - Thursday, August 27, 2020, 4:35 AM

Thanks for the feedback. I changed the expected outcome.

There remains another test fail with empty input S_phi (and other empty inputs without the option allowempty).
I don't actually want to test the behaviour with empty S_phi. Yet the test fails due to penalty. Expected penalty is (default) and actual penalty is empty.

In the previous version I could get along with empty input if I just specified NULL as answer hint.

Now it seems that I need to specifiy some nonsense input ('none' for example) and to specify score and penalty and set the answer hint to F.

Do I really need to follow this bloated worklfow or is there a smarter way? I don't want to touch all my question tests without having this clearly asked ;)

Average of ratings: -

Re: STACK Question tests: Modified behaviour?

by Martin Kraska - Sunday, August 30, 2020, 1:33 AM

here I have another example with 4 question tests (because at least one of the 6 inputs has 4 variants to test)

Usually I just test if the non-empty feedback branches can be triggered. This allows me to review a formatted version of all text fields of the question.

Test 1 is ok because all input have the correct values specified.
Test 2 is ok because I added "none" as dummy for inputs I don't care about. Yet I had to add score, penalty and answer hint for them. For deeper response trees this might be non-trivial, because you have to guess where the tree stops with dummy input. This can be done but is useless work because in fact this is a dummy entry.
Test 3 and 4 are unmodified. They worked in the previous version but don't work any more.

I wish it could work as in the previous version: Empty input, empty score, empty penalty and answer hint set to NULL won't make the test fail.

Can anything close to this be obtained in the new version?

If you have 10 input fields and want to test 5 answer variants for just one of them, how is this done efficiently?

quiz-T-B2-MB-TM2-Hauptspannungen-20200829-1917.xml

Average of ratings: -

Re: STACK Question tests: Modified behaviour?

by Martin Kraska - Monday, August 31, 2020, 5:52 PM

After revisiting the documentation my impression is that either the doc isn't up to date any more or there is a bug in the question test logic. I hope for the latter because I like the behaviour described in the doc (as it was in the previous version) more. Therefore I filed an issue at github.

Average of ratings: -

Mathematics tools

STACK Question tests: Modified behaviour?