Quiz

Tony, hi

Am not convinced about that. How about you post a practice example or two...I will get that- much easier than the football example.  I think Tim said earlier that a rationale remained absent, to be convinced. I think that is fair enough. Itamar and Tony-I am open to the reasoning behind your thoughts, but I am not reading it at the moment.  

D

I have never used the quiz for summative assessment. I have only ever used it for low stakes exercises. My students used to ask me "can I take the test again or would that be cheating". I would sometimes reply, "you call it cheating, I'll call it learning". 

My students would be annoyed with repeated questions, they would generally feel they were having their time wasted, they would want new ones, they would want some alternative angle.  Because of this, after a few tries I never used random questions at all, which is a pity as it is a useful concept.  Preventing duplication of random questions is not something specific to the use of quiz for summative testing.

Sorry, Tony. I don't follow your arguments "Recognition of a question as at least partially familiar is not the same as getting the answer right"

So, surely it is good to prevent unfamiliar questoins for as long as possible, and that is what this changes does. It ensures that no question is repeated until all questions have been seen at least once.

I only gave one example before. Let us consider a second example:

Suppose we have a pool of 100 questions, and a quiz that randomy selects 10 questions each time the student attempts it.

With the Moodle 2.8 code:

1. In the first attempt the student gets 10 questions at random.
2. If they attempt the quiz a second time, then the get another selection of 10 questions at random. Therefore, we would expect that on average the student sees one question they have already seen in the first attempt.
3. And, if they make a third attempt, they will see on average just under 2 questions that they saw in attempts 1 & 2.
4. ...
5. By their 10th attempt, they are likely to only be seeing 4 questions that they have not seen before, and 6 that they have.

With the proposed code

1. In the first attempt the student gets 10 questions at random from the full set of 100.
2. In the second attempt, the student get 10 of the 90 questions they have not seen yet.
3. In the third attempt, the student get 10 of the 80 questions they have not seen yet.
4. ...
5. In the 10th attempt, they get the 10 questions they have not yet seen.
6. If they make an 11th attempt, then of course, they have seen all the questions once, so we are back to a completely random choice of 10 from 100. And so, on average, one of the questions will be the same as in the 10th attempt.
7. And a 12th attempt would be 10 questions from the 90 the student have only attempted once.

In this case, and given the students are likely to only make a few attempts, not 10, then I think it is fair to describe the difference as subtle. Before the student may have sene perhaps 3 repeated questions out of 30, and now they don't see any repeats.

And, despite having read all the discussions, I still thing it is entirely fair to describe this as an improvement for almost everybody.

Well I am not going to sod off now-Am I...given I have been trying to follow here.

So, Itamar when you say:

This is a major change in core functionality of one of the most important core modules in Moodle. Are you suggesting that you are aware of this to be a major change because of the high level of usage across the user-base? And you know this-right? 

And, when you say...

Since you have the power to push such a change the burden of proof is on you and you should be much more sensitive to objections than to support. Itamar, really now, it has taken us some time to get to a decent rationale-really it has, and I honestly think default would not bloody well harm things, but I do now see a reason for optional-as I stated in my last post.  

And...when you say

There are probably far more people using this module and this functionality than people participating in all of Moodle forums and in the Moodle tracker together.  Oh come on...you are not comparing like with like there, those of us here are familiar with the quiz and know its affordances/functionality and are asked talk about it among colleagues.  

OK, when you say this:

You won't hear from them until their instructional strategies break after upgrade. Then you might see MDL-5234X Unforce unique/unseen questions in quiz retakes, with at least 9 votes and 13 watchers. How about you expand on that? I think that would be helpful all round. 

As for this para:

 But given your current attitude, you are likely to dismiss such MDL-5234X and its 9 votes as not having sufficient support.I can appreciate that adding this feature as an option requires more work and that you, just like everyone else, is very busy. I suspect that at the end of the day this is the only real reason for not adding the feature as an option. With such core functionality, this is not a good reason at all.

I would say....bordering on a flaming attack....and the moderators are slow around here...I have been mod-ed for far less Itamar!  

$qubaids = new qubaid_list(array());

Dawn,

I think this is a minor change. (I thought that when I first posted, and I still think that despite all the comments.)

I am really suprised at the response this thread has got. I though I was just posting about a minor technical change, so no-one could later complain that they had not been given the chance to comment, but I was wrong about the strength of feeling other people have about this. Many things in other parts of the code go through the tracker without anyone in the forums being aware until the next Moodle version is released. I try to do better than that.

Hey, all

For the record, this thread has frustrated me no end-and yes women get frustrated too and cross and all the stuff that goes on with men when things are frustrating-we even swear!!!!!!!! Gosh how terrible is that eh.  We also know how to express empathy skills when someone has worked hard-only to have their work/ideas questioned at the last minute....yes-that is right empathy-not emotional subjectivity blah...

And Tim that post is persistent:

  Itamar, did you see the above request for data? I would still be interested to see it.

and made me laugh, sorry Itamar...and I needed to laugh!

So, I want to build an argument based on what has gone before...it seems there is now an overwhelming majority for this Tim, great. I do think there is one small point left, in my view, could be wrong, and I will present that next in my points.

"The moodle quiz engine is already the best that I can find"

I've done the research, it's the best, free or paid for.

So your set up is essentially the same as Itamar's.

I have the same questions about real data, and I would be interested in your answers too.

• How many questions in the bank?
  
• How many questions in each practice quiz?
• How many questions in the final exam?
• The instructional design is clearly set up to give students to do the practice quizzes many times, but if you look in teh quiz reports, how many attempts do they actually make?

I have learned an important lessons from hanging around OU academics

Another interesting point is that your use of quizzes would not be allowed at the OU. Our rules say that you cannot use the same questions in formative and summative assessments. However, the questions in the practice quizzes can be very similar to the ones in the summative quiz. E.g. performing the same calculation, but with different data.

Also, we would never build a summative assessment with a set of questions pulled completely at random from the whole question bank. We would want to know that the questions cover all the key topics, so each question would come from a separate category of similar questions. 

If you really want to ensure that students don't simply write down all the answers to all the questions, do the same thing. Make the practice quizzes pull from one pool, and reserve some separate questions for the final exam. Probably a bit more work, because you need more questions overall, but more robust.

Thanks to Ray, a long way above, for catching a potential bug;

He is right, and I have now fixed this in my code.

As Rick says here, it is not obvious that it is a bug. That is, if you use calculated questions, for example if you have one calculated questions that provides 10 variants of a simple addition problem, then you might like to add several copies of that to your quiz.

Unfortunately, because of the details of how shared data-sets (a good feature) are implemented, that won't work, all the repeated copies would show exactly the same variant.

Obviously, that is not what we would like. I hope that can make ths work one day, but it was not possible as part of this change. (There are even cases for adding multiple copies of a non-internally-ramdomised question to the quiz. E.g. after every Moodle-graded question add a the essay question "Reflect on the process of answering the last question, and write a sentence about what you think you learned.")

"Adding something to the UI to make it optional is what Itamar suggested and seems a small price to pay for the maintenance of backwards compatibility and retention of functionality that would otherwise be lost."

I think such a UI feature should be well hidden from the average user as the Quiz interface is quite crowded enough.

>  Does anyone have experience of students being pleased they get a chance to do the same question again on a later attempt at a quiz.

Yes, with me being the student.  Think flash cards.

It is good when I need to remember the answer to that question.  Much like flash cards, I suppose.  I kept going through the quiz as practice until I remembered the information.  This is of course useful primarily for things that need to be memorized, rather than for these that only need to be understood.  One recent example is one that needed to be both understood and memorized - the normal forms of relational databases.  I already understood WHY transitive dependencies should be removed; repetition helped me memorize that's THIRD normal form, not second or fourth.

But Tim, surely you can't deem this a bug. As I understand it, the code has always performed a straightforward function exactly as described. The new code does something different and trickier - certainly better for some purposes, but incapable of doing some things people may like, which the old code permitted.

I gave an example in my link for Dawn. I considered the principle so valuable I gave it a particularly simple implementation in my software (a command to select m at random out of each n Qs in what follows). In the example, each Q is randomly selected from a pair of similar Qs with different answers. This forces careful thinking about the issue at stake in the Q, when a similar Q but not necessarily identical Q is seen on a repeat attempt. The example that Dawn said she got wrong was:  "Is the underlined word in this sentence ('effect' or 'affect') correctly spelled?"). A random pick out of 2 could be set up in 2.8, but in 2.9 (as planned) a student would see different versions of the Q that would alternate in successive attempts. The almost irresistible instinct is then simply to think "I remember I thought it was OK last time and got it wrong, so this time it must actually be OK"). Just the sort of thing we don't want students to do. So far as I can see, you couldn't get round this under the proposed changes.  I have no idea how many uses people make of this kind of functionality at present in Moodle, but since it is in principle useful, I can't see the point in suddenly taking it away.

A little PS, it IS still random.

Suppose I'm getting dressed, and my wife is sleeping so I leave the light turned off.  

I want to wear blue socks, but can't see whether the socks in the drawer are blue or black.

I RANDOMLY choose two socks from the drawer and carry them over to the night light.

I see that they are not blue.

I return to the drawer and RANDOMLY choose two more socks and check them.

I keep doing this until a find two blue socks.
After I get the blue socks, I return all of the black socks to the drawer.

Note it would be silly of me to put the black socks back in the drawer before randomly choosing another sock to check.  It's smarter to randomly choose from the socks that might be blue - the socks still in the drawer.

Note because it is dark and I can't see the socks, I would be RANDOMLY choosing socks from the drawer.  The fact that I'm not also including the socks already in my hand doesn't make it non-random.  It's randomly selected from the drawer , not from all of the socks in the whole world.  If you say "it's not random because you don't include the socks picked previously", that's precisely the same as saying "it's not random because you don't include your neighbor's socks".  It is random - a random selection from the socks remaining in the drawer.  The fact that other socks exist in other countries doesn't matter.

The new implementation of random questions is the same.  It's absolutely random, Tim is just changing the pool from which one is randomly chosen.

question:

Does the SoD allow for that?

Am playing....long night!