Proposed quiz usability tweaks

Proposed quiz usability tweaks

by Tim Hunt -
Number of replies: 23
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

There has been moaning here in the recent past about how many clicks it takes students to attempt a quiz, which is not always appropriate, particularly in informal uses of the quiz.

I am not completely unsympathetic to this, and during the Hackfest at the Edinburgh MoodleMoot, I grabbed Stuart Lamour and Bas Brands, who often go on about usability issues in the forums, and we went through the quiz UI from the student point of view, to see what could be done. Wow! that was 2 months ago now. Since then, I have had this scrappy bit of paper on my desk, and been meaning to post it. Here is the answer:

Quiz UI outline

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Grrr! I did mean to explain that diagram. (What, you mean it is not perfectly clear wink) but instead I just discovered MDL-46069 after typing for about 20 minutes. sad I will explain at some  point.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

OK, short version:

Note, the reason I am posting is so that you can comment about whether these are good ideas. Please do.

  1. At the start of the quiz, only show the "Are you sure you want to start this quiz now" if the quiz has a time limit. At the moment, it is shown if there is a time-limit or a limited number of attempts, but there is only possible harm in starting an attempt if there is a time-limit.
  2. If the quiz has a password, this is currently shown after the Are-you-sure. (Though other checks like https://moodle.org/plugins/view.php?plugin=quizaccess_honestycheck are shown on the same page as the password, if both are enable.) Merge the time-limit Are-you sure onto this page, so there is never more than one page to click through.
  3. And, make that combined page a JavaScript thing in all cases, like the current 'Are you sure' message.
  4. At the OU, we have some nicer styling of the quiz navigation panel. Get that in to Moodle core.
    OU quiz navigation
    (Note, I promised to do that once before, several years ago, but it never happened.)
  5. For now, at least, we are going to leave the Summary page, and the 'Are you sure' when you click "Submit all and finish".
  6. After that, when (if) the student gets to the review page. We should change to to show all the questions on one page by default. (Perhaps only if the quiz has fewer than 30 questions, or some limit like that?)
  7. Instead of the 'Finish review' link at the end of the review, we should instead put 'Back to course' and Re-attempt quiz' buttons right there. There would be no direct link back to the quiz information page.
  8. Also, we should always show a 'Back to course' button on the quiz information page.
  9. I wonder if it would be better to move the quiz 'Overall feedback' from the top of the review page to the bottom?

Average of ratings: Useful (4)
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If you are wondering what I think, I am pretty sure 1.-5. are all good ideas. I was not sure about the others initially, but the more I have thought about it over the last two months, the more sensible they have seemed, but I am still not sure.

I'm sorry it took me so long to post these. I would really like to know what you think. If we do this, it would be nice to do it in time for Moodle 2.8 in November. UI changes like this would only go into a major release like that.


Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by ben reynolds -

I'm good for 1 - 6 (6 will cure an annoyance more than a usability issue).

9 I'm less certain about, but I think that's because I have become accustomed to the overall feedback at the top. Moved to the bottom, it becomes a summary after I've reviewed all my answers, so that makes narrative sense.

Thanks for the warning about Firefox and atto.

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Joshua Bragg -
I agree and like 1-6.

7 would be a step backwards for teachers (like me) who have some quizzes that are deferred feedback where some feedback is only available after closing that allow multiple attempts.  For example, I let students know which ones they missed but I don't give the correct answers until after they close.  If a student has two attempts and they go back after closing to review the answers then they will have to re-enter the quiz (or use the breadcrumb trail, haha yeah right) to get back to see the other attempt.

8 - A back to course button used to be pretty standard across Moodle in the 1.9 days (or so I seem to remember).  Putting it back in the quiz would seem out of place with everything else.

9 - I don't have a strong opinion on this one.  It seems slightly more logical to have it with the grade/attempt information at the top though.
Average of ratings: Useful (1)
In reply to Joshua Bragg

Re: Proposed quiz usability tweaks

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

7. In the table at the top of the review page, there are links to review your other attempts at this quiz (if you have other attempts). Probably another bit of UI, like the breadcrumbs, that is too subtle for anyone to discover or use.

9. Another option would be, repeat that whole review info table at the bottom of the page if the quiz is long enough (> 10 questions?)

Actually, I suppose a better way to think about it is this: If you are a teacher grading a paper test, do you write the big fat F and your overall comment at the top of the first page, or at the end?

Average of ratings: Useful (2)
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Joshua Bragg -

The links to other attempts are only present in the teacher view.  Students don't see that little bit.

I write the big fat F in the top right corner of the page and do overall comments immediately underneath.

Average of ratings: -
In reply to Joshua Bragg

Re: Proposed quiz usability tweaks

by dawn alderson -

Hey Joshua,

not going to argue the toss with you on that for sure. I do want to outline my reasons for suggesting feedback at top....grade at the bottom however....reasons follow:

1. have spoke to many students over the years...both undergrad and post-grad and been both of those type of student myself....both ftf and online learning envs.

2. I conclude from that, when the feedback is at the top of the scroll and the grade at the bottom...there is more chance of the feedback at the top-being read..........the words chosen by the teacher can often signal what is to come...as a student...and having talked with students-they know this.

3. When both are at the top...same section.....more often than not........I could be wrong.......it is more of a quick scan at the grade and off to the pub!

Yet, the F score...is the one person who needs to really....read that feedack to avoid a trail of such marks/grades and avoid the inevitable.............do those feedback sections get read.....probably...but the best way is with guidance with the tutor.....and on...and on.......

Hope helpful.

Dawn 

Average of ratings: Useful (1)
In reply to dawn alderson

Re: Proposed quiz usability tweaks

by Joshua Bragg -

Makes sense to me.  With that said, the overall feedback option isn't really a detailed set of feedback that we can guide students with.  What you're suggesting really makes sense though on paper assignments where rich feedback is given.

Average of ratings: -
In reply to Joshua Bragg

Re: Proposed quiz usability tweaks

by dawn alderson -

Hi,

oh my, this- I think is a whole new thread...not in quiz forum though eh.

When you say: feedback option isn't really a detailed set of feedback that we can guide students with' I struggle with that idea, to be honest, but I shall shut up about it for now ;)

Paper feedback-eh? Does that still happen then! Only joshing with you Joshua-I understand your point.

Shame that there still does exist the mindset, which includes a demarcation between online/electronic feedback and paper feedback....(the latter perceived as affording more depth) I disagree with that perception, although over time I suspect practice will change markedly and the balance will be more evident, perhaps.

And, I still stand by my view, in that the grade/mark at bottom of page might be better placed for learning purposes. I am glad the plan to keep written feedback at the top of scroll is a popular one.

Cheers,

Dawn


Average of ratings: -
In reply to dawn alderson

Re: Proposed quiz usability tweaks

by Ray Morris -

Most of those match up with the feedback I've received from users.  #6 and #9 I've not heard anything about or have any opinion. One of our local customization is adding "Back to course" on pages where it might be reasonable for a student to return to the course at that point. I fought that a bit, but others in my organization convinced me to do it, and now I think they were right.

The end of the quiz does seem to have a lot of clicks between answering the last question and being completely done, heading back to the course page. Anything to reduce that is good.

Eliminating "are you sure" in cases that don't have bad consequences is a great idea. We don't want to train users to ignore those messages by popping them up too often. Besides, I clicked "Start Attempt", so yes, I want to start an attempt. 

I'm glad everyday objects aren't like Windows^H^H^H^H^H^H^H computers.

"Are you sure you want to unlock the car door?"
"Are you sure you want to open the door?"

"Are you sure you want to get in the car?"

"Are you sure you want to start the car?"

"Are you sure you want to put the car in reverse?"

"Are you sure you want to get on the highway?"

"Warning - you are late for work!"

 tongueout

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Phil Butcher -

1-5 and 7-8 are fine.

6 I'm less sure about. It certainly needs an extra refinement that if there are questions brought in through opaque then it should remain at one question per page for default (because we know that multiple OpenMark questions on a page can fail to render properly).

9 I'm not in favour of.

Average of ratings: Useful (1)
In reply to Phil Butcher

Re: Proposed quiz usability tweaks

by dawn alderson -

actually....very good point. I would be inclined, if it were me-to keep the feedback at the top (that is what needs to be read-and is often not!) and the grade at the bottom...in line with the same SOP.....that can often exist across assignment feedback templates for Ed per se (course, I am talking about my experience here in UK)...so might be a narrow view...or not smile

D   

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

There seems to be a surprising amount of consensus about this. People are happy with all the proposed changes except 9, which almost everyone thinks is bad.

Actually, 9. is something is added to my forum post at the last minutes, not one of the changes that I have been thinking about for a while, so I am not so surprised it turned out to be a bad idea.

Because there seems to be consensus, I created MDL-46090 to track this work. (I also added a couple of related accessibility issues there, because they seemed relevant and I might fix them at the same time.)

To answer some of the questions raised:

6. Phil: issue with qtype_opaque not working with show all on one page. I don't think we can let this hold back improvements to the quiz. Instead, we need to fix qtype_opaque. (I wonder how hard that is?)

7. Joshua: difficultly of navigating to review of other attempts, in quizzes with multiple attempts and not immediate review. I am not sure how much of an issue this is. Maybe make inter-attempt links available to all users, not just teachers?

7. Dawn: getting rid of back to info link at the end of review is bad. Maybe, but on balance I would still rather remove it. Once I have coded up the change, we can see how it looks before the change is actually added to Moodle.

7. Tim Lowe: Only show Re-attempt quiz button if that is possible: of course.

8. Joshua: back to course link on quiz info page, does that go against the Moodle standard? Yes, that is my concern, so I am still unsure, but quite a lot of people have asked for it over the years.

I hope that is everything.

Average of ratings: Useful (2)
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Joshua Bragg -

More thoughts on #7:

Perhaps the reason why the inter-attempt navigation has not been traditionally shown to students is that there are so many review options that it would be difficult to rationalize when to show them and when not to.  Having a link to an attempt that students are not allowed to see would be very confusing.  Having a list of attempts where somethings are not linked would be just as confusing.

One of the nice items on the quiz info page is that it shows when the review will become available to students.

Perhaps an alternate option would be to:

  • If there has only been one attempt so far only give a return to course and reattempt quiz buttons.
  • if there have been multiple attempts so far then also add in a "see my other attempts" button.  That would take them back to the traditional quiz info page.
Other concerns with this particular proposal:
#7 seems to encourage students to immediately reattempt the quiz.  If I give a timed formative assessment to my students and they don't do well the first time then I don't want them to immediately reattempt the quiz.  I want them to stop, review, study, and then reattempt.  

Also, unless we're talking about adding warnings about how many attempts are remaining to the are you sure dialog then I'm sure I'll get some complaints about students losing track of how many attempts they've used.

The more I've thought about that particular option, the less I like it.
Average of ratings: -
In reply to Joshua Bragg

Re: Proposed quiz usability tweaks

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Well, I won't rush to implement this one. I will go on thinking about it.

I will just note that any situation where we show a user a link that just takes you to an error saying "You are not allowed to see this" is a serious bug. It should just not be done, and I would not do it here. It is clearly possible to display this properly, since we display the right links in the table on the quiz info page.

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Mathieu Petit-Clair -
Picture of Core developers Picture of Moodle HQ Picture of MoodleCloud team Picture of Plugin developers Picture of Testers

I had a lot of discussions with coworkers (course creators, mostly) and this list of changes, mostly 1-3 and 7-8, would make them very happy. 4-6 would be appreciated as well, but are not as critical. Good stuff!

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by dawn alderson -

I need to finely tune my brain for this Tim....back after some zzz......s.

Dawn

Average of ratings: -
In reply to dawn alderson

Re: Proposed quiz usability tweaks

by dawn alderson -

Hey,

have taken the liberty to order the points in terms of what I see as priorities for the user...now I hope there is laughter there....because if you really think about that...how could you have expected me to have done anything other than that!  smile

Here you go..(and yes, that F needs to placed very gently across proceedings-agree)

1. At the OU, we have some nicer styling of the quiz navigation panel. Get that in to Moodle core.

(Note, I promised to do that once before, several years ago, but it never happened.)

2. I wonder if it would be better to move the quiz 'Overall feedback' from the top of the review page to the bottom?

3.After that, when (if) the student gets to the review page. We should change to to show all the questions on one page by default. (Perhaps only if the quiz has fewer than 30 questions, or some limit like that?)

4. Also, we should always show a 'Back to course' button on the quiz information page.

5.Instead of the 'Finish review' link at the end of the review, we should instead put 'Back to course' and Re-attempt quiz' buttons right there. There would be no direct link back to the quiz information page. SHAME, I THINK THAT WOULD BE USEFUL.

The rest are all linked together/or involve no change-as far as I can see.

D

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Tim Lowe -

Hi Tim,

(Was eventally allowed to reply after first resetting my forgotten password and then realising that when I'm not I need to "enrole in the course" really means "join the community"! ) 


I'm happy with all of these, except 9.

In 7, I assume the "Re-attempt quiz" button would only be displayed if re-attempts were permitted!


Best wishes

Tim




Average of ratings: -
In reply to Tim Lowe

Re: Proposed quiz usability tweaks

by Mary Cooch -
Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Tim Lowe - where did you see it say "enrol in the course"?

Average of ratings: -
In reply to Tim Hunt

Re: Proposed quiz usability tweaks

by Stuart Lamour -
Picture of Plugin developers

for creation of complex form stuff - analysing the workflows of survey monkey and google forms could be a nice user friendly source of inspiration.

Average of ratings: -