fun puzzle?

fun puzzle?

by Jamie Pratt -
Number of replies: 5
I am working on a new statistics report for Moodle 2.0. One of the statistics calculated is Random_guess_score :

This is the score that the student would have got by guessing randomly. It depends on the question type. For types like shortanswer, it is 0 - or the score associated with answer '*', if there is one.

For multiple choice questions (including matching, truefalse, etc.) it is the average score over all the possible choices.

(There should probably be a method in the question type class to compute this.)



This turns out to be quite a complex calculation. I thought I might invite some community participation since I'm finding working these out rather time consuming. I have worked out methods to calculate this statistic for all other question types but the matching question.

The short answer matching question is obviously similar to the matching question type. There are N drop down boxes each with N possible answers to select. Assuming the student is clever enough to not select the same answer twice I worked out that the average grade for a random guess is likely to be 1/N (out of 1).

For the matching question type thought there might be M drop down boxes and N extra answers. I worked out that if there are factorial M possible combinations of answers if there are no extra answers. Ie :

For M = 3 combinations of answers is 3 * 2 * 1 = 6
For M =4 combinations of answers is 4 * 3 * 2 * 1 = 24
For M =5 combinations of answers is 5 * 4 * 3 * 2 * 1 = 120
etc.

In all the factorial(M) possible combinations then I worked out factorial(M) out of the factorial(M) * M drop down answers will be correct. So this means that there will be an average grade of 1/M.

But what if you take into account the extra wrong answers?

I worked out that if there is one extra answer then I think there are :

factorial(M) * (M+1) possible combinations of answers.

But after that I start getting confused.

Oh!

I got it, there are most probably :

factorial (M + N) / factorial (N) possible combinations.

But there are now more than factorial(M) drop downs in the right positions out of the possible combinations * the number of drop downs.

Does anyone follow what I've written so far?? And can you suggest a formula for the random guess score for a matching question with extra possible answers.
Average of ratings: -
In reply to Jamie Pratt

Re: fun puzzle?

by Jamie Pratt -
Darn. I think I will just assume for now that the student is stupid enough to pick the same answer for each drop down box. Then there are simply M+N possible answers to select for each drop down box and only 1 answer is right for each box. So the average grade is 1/(M+N).

If anyone comes up with an answer for the above that is properly reasoned out then please let me know. I guess the random guess score will only be slightly better if the student doesn't pick the same answer in several drop downs.
Average of ratings: -
In reply to Jamie Pratt

Re: fun puzzle?

by Pete Horne -
I believe you were on the right track with looking at the permutations when describing the random guessing.

Let's assume there are 5 questions and 5 answers. Since the questions have a definite order, then the answers must match that order. Then by the fundamental counting principle there are factorial(5) possible permutations, only one of which is correct. This gives a 1/120 chance of guessing the correct order. What is assumed here is that each answer is removed from the list once it is used.

If the answers are not removed, then for each question there remain 5 possibilities. This means the number of possible guesses, including repeated guesses is 5^5 = 3125. For example, a student could guess A, A, A, B, A. (Wouldn't you know it, question 4's answer was A and the student guessed B.)
By blindly selecting answers, some of which are repeated, the chance of guessing is 1/3125.

As you mentioned, the student can defeat this and get one correct answer by guessing the same answer every time. In this case the chance of guessing the correct answer reduces to the probability of guessing one correct problem, 1/5.

Since you are trying to determine whether or not it is likely the student guessed or not, you need to consider the largest possible probability, 1/5. Scores at or below this value should be considered to be guessing.

Of course this all goes out the window if the student knows one or more right answers. Then the process would have to be considered for the remaining answers.

With extra answers, becomes a problem of a distinct order (permutation) of N elements taken R at a time, where N is the number of possible answers, and R is the number of questions.

For no repeats, this probability is:
1/(factorial(N)/factorial(N-R))

The denominator of this represents the number of possible ordered exist for the answer set.

If repetition of answers is allowed, then then number of possible choices is N^R.

Since there is no guarantee the person will select the correct answer, the guessing probability becomes:
(factorial(N)/factorial(N-R))/(N^R)
Number of valid orders/Number of repetitive guess orders




Average of ratings: -
In reply to Pete Horne

Re: fun puzzle?

by Jamie Pratt -
Thanks for the input Pete.

Hmm.

Remember there is not only one correct answer that scores a grade. The student gets partial grades for getting the answer right in any of the drop down boxes. And what we want to find is the average grade for a random guess.

There is a set of subquestion and answer pairs. And then there are extra wrong answers. For each subquestion the student can choose from any of the answers or any of the wrong answers.

So the correct average grade for the question if repeats are allowed is 1/(totalnumberofanswers). Which will give us a grade out of 1 for the whole question. This is because the odds of getting each subquestion right will be 1/(totalnumberofanswers) each subquestion contributes a grade weight of 1/totalnumberofsubquestions. So the sum of the average grades for the whole question will be 1/(totalnumberofanswers).

Assuming that the student knows enough not to repeat his answers then the formula becomes more complex and involves factorials.

I did not follow all of your reasoning Pete. I hope I explained the problem a little better this time.

I think 1/(totalnumberofanswers) will be fine for calculating the random guess score but would be interested if someone could solve the problem.

It is a difficult thing to discuss in a forum though I think. It is the sort of thing that I would imagine would be a lot easier to explain on a blackboard in a meeting room with more knowledge of the background of all the participants in the conversation.


Average of ratings: -
In reply to Jamie Pratt

Re: fun puzzle?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
You need to remember that not all of the answers need to be distinct. For example, in Moodle, you can have a matching question where there are 10 statements, but only two things to match them against (e.g. true/false). In that case RGS = 0.5.

So I think the correct forumula to use is 1/(num choices in dropdown).
Average of ratings: -
In reply to Jamie Pratt

Re: fun puzzle?

by Pete Horne -
Jamie,
Okay, I might not have understood the structure of the questions. I was thinking of a two column matching arrangement with the first list being static and the answer choices in the second list. The second list would be a series of drop down boxes for the first list (same possible answers in each). As you said, having a chalkboard would probably make it a little easier to figure this out.

Am I understanding correctly, based on Tim's reply, that a typical question might be:

[Question]List the three closest planets to the Sun:
[Answer] Three drop down boxes, each listing the nine (yes I'll include Pluto, because that's the way I was brought up) planets.

Therefore, the correct answer set would have Mercury, Venus, and Earth in any order, but each planet only could occur one time for full credit. That is, Earth, Earth, Earth would receive one third credit while Mercury, Earth, Mercury would receive two thirds credit, etc.

If order is not important, we need to look at combinations to obtain a probability, but if order is important we have to look at permutations.
Average of ratings: -