What I think is a major flaw in way partial credit is graded

What I think is a major flaw in way partial credit is graded

by K P -
Number of replies: 10

This only applies in adaptive mode.

Ok, I only recently discovered this and I am not sure how many of my students figured it out but remained quiet about it.  I only found out because an honest student brought it to my attention.

First Flaw:  Question is 1 mark.  Penalty set at 0.5.  They get the question wrong the first time and they get 0.5 marks.  If they try again and get it wrong, they get 0 marks.  This means that the student is guaranteed at least 0.5 marks as long as they only guess/attempt once.  I think that on the first wrong answer attempt, that they get 0 marks and if the attempt again and get it right, they get 0.5 back.

Second Flaw: This one is pretty major.  Lets say I answer wrong, so as mentioned above, I get 0.5 marks (assuming question is 1 mark and penalty set at 0.5).  I try again and get it wrong now I am down to 0 marks.  I then go back and put my original first attempt wrong answer, it gives me back the 0.5 marks.  I repeat this pattern as many times as I like until I find the right answer.  I am guaranteed at least half credit and I can answer as many times as I like.  Once I was informed of this issue, I went back and looked at some of the individual student responses and found that quite a few times a student got half credit for a question and answered it as many as 15 times by using the method I described above.  It would be better I think to have an option to limit the number of attempts per question to stop this.  I don't know about anyone else, but I don't look at every single individual student question and hand grade each one.  If I wanted to do that, I wouldn't use Moodle at all to give quizzes/tests.

Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by Chris Collman -
Picture of Documentation writers
Humm, I thought the whole point of an adaptive quiz was the learning process, rather than the score. I suspect this is another case of the Moodle feature being used in a manner that was not part of the initial vision.

Of course Lesson has the max attempts but not the short cut step of a Quiz penalty. While to set up an 'adaptive' lesson is a lot of work, you would not have to worry about students cheating on rules that you have in your head but which Quiz can not follow smile

I realize this is the Quiz forum but let me present an example.

In a Lesson you can control the flow (jumps-links) associated with each answer and use "duplicate questions". For example: Create 3 questions in this order: Q1, Q1w, Q2. In Q1 correct answers are scored (say 2 points) and go to Q2. In Q1 wrong answers are scored (say 0) and go to Q1w, this appears to be an identical question. A correct answer here is scored (say 1) and they are sent to Q2. A wrong answer in Q1w can just jump back to "this page" which is Q1w and receive no credit. By setting max attempts at 2, the student who tries two wrong answers on Q1w, will automatically be sent to the next page without a score. The next page is Q2. I would set the max attempt to 1, and if the student missed Q1w, would tell them the correct answer in the feedback or send them off to a remedial page in another part of the lesson, which would have a link back to Q2. That is adaptive.

Lesson is not a one size fits all and has it's own special little frustrations. Quiz reports, regrading and item analysis have no equal in Lesson.

Chris


Average of ratings: -
In reply to Chris Collman

Re: What I think is a major flaw in way partial credit is graded

by K P -

I do understand the purpose of adaptive mode.  I also understand that Moodle was created by developers outside the US where ideas of education and how to assess learning is different.  I taught in the UK for several years and understand that the quiz on here was probably more intended as a revision exercise, even though I believe GCSEs are not used in Australia (where Moodle was "born" right?).  Assessments in the US don't work the same way.  Most districts still require a hard test grade to show "learning" even though the score doesn't necessarily show learning. 

I just joined a professional learning class where one of the "good" test methods was to offer students a second chance on each question so they can catch a stupid mistake but everyone agreed that on an official assessment of the material, you can't just have them keep going because at some point learning stops and guessing begins. 

I have done the adaptive lessons before and you are right they are time consuming, more than I am willing to spend creating a 20 question test.  I also don't have the option of the question bank.

Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by Chris Collman -
Picture of Documentation writers
Kameron,
I apologize for my choice of words if they sounded mean.

In my training environments, we use Quiz as an Exam, what I believe some call a summative assessment. They have no or little feedback with each answer and students must pass with a minimum score. I always get the impression that this use of Quiz is in the minority, but perhaps it is so simple that nobody has issues using it for this purpose, so no squeaky wheels.

I am sure you are 3 steps ahead of me in understanding how to balance your class, your school, your district, your state and our federal expectations regarding student progress. I shiver listening to my spouse discuss these hoops as a school counselor.

I guess all I was trying to say is that on some levels, Moodle is also a balance of many ideas and concepts. Which as you explained, is something you already appreciate.

Best wishes in finding a solution to creating better ways of doing things in your class and with Moodle.

Chris


Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by Itamar Tzadok -
but everyone agreed that on an official assessment of the material, you can't just have them keep going because at some point learning stops and guessing begins.

Interesting agreement. When the question the student has to solve is constructive and composed of multiple steps and multiple ways to proceed from one step to the next guessing is not an option and you can let them keep going forever if they really want to. If you add some feedback they may realize at some point that it may be more productive to start working systematically and work out the solution. And then there is learning. The only problem is that developing such questions is no easier then solving them. How many instructors/institutions are willing to make the effort? smile
Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I am confused by your description because adaptive mode does not work that way.

If you get the question wrong first time, you will get 0.

More specifically, in the scenario you give:
  • The grade for the first try will be the raw grade for that answer.
  • The grade for the second try will be the raw grade for that answer - 0.5
  • The grade for subsequent tries will be 0.
  • The final grade for the question is the max of the grade for all the tries you make.

On unlimited retries, at the OU, we don't like that either, which is why the rewrite I am currently working on Development:Question_Engine_2 introduces a new mode 'Interactive mode' which limits the number of tries to whatever you specify when you create the question.
Average of ratings: -
In reply to Tim Hunt

Re: What I think is a major flaw in way partial credit is graded

by K P -
Chris: No problem. I forget that Moodle has other uses than just "school" and forget about the corporate/training uses. I can completely see how what you first described works very well for you.

Tim: I am not sitting at home trying to reproduce the problem and can't. But I know during the attempt, I have looked over a student's shoulder and saw the half credit on first attempt of the question. Going back now and looking at random grades, I can't see the issue in the final grade. So, I am lead to the conclusion that this is a misunderstanding by both myself and the students of the marks information during the attempt, but at the end it is working itself out. Is there anyway to turn off the marks/penalty information on the student end but not turn off the red/green/yellow answer colors?


Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by Chris Collman -
Picture of Documentation writers
I am glad my words were not a big issue. Continued success in your problem solving. Chris
Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
On the quiz settings form, the review options for 'Immediately after the attempt' also control what feedback is displayed during the attempt in adaptive mode.
Average of ratings: -
In reply to Tim Hunt

Re: What I think is a major flaw in way partial credit is graded

by K P -
It seems that an upgrade to the lastest daily build fixed the issue. 
Average of ratings: -
In reply to K P

Re: What I think is a major flaw in way partial credit is graded

by K P -

Well I thought the lastest build fixed it but I think I have determined the real culprit.  Today it seemed to happen again during a test and looking at it closely and more students seem to ask about it.  I discovered it had to do with the marks the student was being shown differing from the marks I was seeing on the results screen.

I posted a follow up here http://moodle.org/mod/forum/discuss.php?d=65166#p648128 in a different topic that more relates to what I am seeing.

Average of ratings: -