Quiz

 
 
Tim at Lone Pine Koala Sanctuary
Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

I have mentioned in several recent threads that the OU would like to make several changes (we hope improvements!) to the quiz edit page, and the structure of quizzes. Here are the changes we are proposing

1. Breaking up a quiz into section

The sole purpose of this is to let you put section headings into the navigation, to break up large quizzes. You can see what we mean by going to https://students.open.ac.uk/openmark/s205.ayrf/ and clicking the Begin link. The quiz is in 5 sections.

I know some people would like to be able to compute a separate score for each subsection. I'm sorry but we are not going to implement that at this time. However, these changes should make it easier to implement that later.

2. Question dependencies

Suppose you have a sequence of questions that guide a student through a calculation one step at a time. Then, the text of question 2 might inevitably give away the answer to question 1. Therefore, we want the ability to say, you can't see question 2 until you have answered question 1. (This idea works best with a behaviour like Immediate feedback, so a student can submit one question and get the feedabck before starting the next question.)

3. Sequences of question variants

The idea here is that you might have a sequence of related questions, all about the same thing. For example:

i. Description containing a picture
Q1 What is in the name of the animal in the picture?
Q2 What colour is the animal?
Q3 Which continent is the usual home for the animal?
Q4 Complete the following sentence to describe the picture.

and suppose you have several different pictures, and you want each student to get a different randomly selected picture, and then the corresponding follow-up questions. (In this example, the question text for Q2-5 is identical, but of course the answer matching rules are different, so there are different versions of those questions for each picture.)

4. Repeating a question in-situ

This is aimed at formative quizzes, where we want to encourage students to keep trying until they can get 100%.

At the moment, you can let students have repeated attempts at the whole quiz, but then they have to re-attempt all the questions, and the can only do that after they have finished all the questions in their first attempt.

Of course, we have the "Each attempt builds on last" option, but if you turn that on, then the second attempt will use exacty the same questions as the first attempt. At the OU, with practice quizzes, we provide several randomly selected versions of each question, to give students more practice.

So, what we want is that when a student has finished a question in an attempt, then they see a Re-start question button (probably in the grey information area to the left of the question). If they click that button, then they can try the question again within the same attempt, and if it is a random question, they will see a different variant. (Again, this is an idea that works best with a behaviour like Immediate feedback or Intereactive)

[Note, this sounds like a really crazy requirement. At least, it has to me. However, we have had this feature in our OpenMark system for a long time (as an option) and students really like it. You can see it here: https://students.open.ac.uk/openmark/s205.ayrf/. Answer question 1 and get it wrong. Then click next. Then click back to question 1. Note you are now looking at a different variant of the same question, and you can have another go.]

5. Modernising the quiz edit page

The UI here is quite clunky. E.g. having to click up and down arrows to move questions one space at a time. Since this page was made, there have been major improvements to the course edit page (to give just one example). We should take the best ideas from there to make the quiz editing easier.

Also, we think we can eliminate the difference between the Editing quiz and Order and paging tab in the UI, and make a single screen that does everything that both of those do, while at the same time being simpler to look at. (This will require more use of JavaScript, but our facilities for doing JavaScript in Moodle have been getting better.)

I guess you don't need to believe we can do that until we have shown you the UI mock-ups wink (I will believe we can do it once I have drawn the UI mock-ups big grin)

Full specification

Note that all the above features will be optional. You can turn them on when and where you want them.

The full specification (which was written for an internal audience by my colleague Phil Butcher, so excuse the OU-specific stuff) is here:

https://docs.google.com/file/d/0B_W3H1TnOxK-WHpzdk1SYktiT0k/edit?usp=sharing

I think that is a good, clear, summary of the goals of what we are trying to achieve, in more detail than I have given above. It contains some suggested user-interface mock-ups. Some of which I like, and some of while I want to try to improve on.

There is not yet anything written about how this will be implemented. I need to write that and I am unlikely to have time soon. I particularly need to think about, and write down, how we are going to change the database structure to store all the necessary information.

Comments please!

I hope you agree with us that these are a useful set of additions to the quiz.

Past experence shows that when we post a spec in the forum, normally someone can point out something stupid we should have thought of, or some way to improve on what we are proposing. So, please do not hold back in telling us what you think. smile Thank you for considering this.

 
Average of ratings:Useful (9)
Picture of Don Hinkelman
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

Thanks, very much Tim. I think it is a wonderful situation for Moodle that the development impetus and testing ground for new features is the vast OU, which is probably the largest or one of the largest universities in the world.  These changes are obviously well-needed, well-discussed and not just good-idea brainstorms.  Some comments:

  1. Sections: Yes, we will use sections immediately. And yes, we need to calculate grades for each section (we can wait for that). For example, our EFL placement test has a listening section  and a reading section and students need to know their scores for each and then compare again at the end of the term.
  2. Question dependencies: For us, this is actually more important than activity dependencies.  This will take time for teachers to learn how to make paths of thinking, but it gives training and learning a new far deeper dimension
  3. Sequences:  This is intriguing and new ground for me.  It will take time to develop and explore.
  4. Repeating in-situ:  Again intriguing.  I have to try it to understand it.
  5. Quiz edit page:  All welcome ideas.

Missing?  Well there are some higher priorities for us at SGU which are probably outside the scope of your plans.

  1. Team collaboration in the question bank:  The question bank is very hard to use and to share questions among teachers. We are learning pitfalls and workarounds, but this needs work on.
  2. Drag-and-drop ordering and matching question types:  These are the most-used question types amongst our teachers, because students love them, they are fun to make, and they can combine many separate questions into one.  Extremely helpful for language learning.  A pity that these are outside of core, and need reworking with each new Moodle version.
  3. Backup and restore of question bank in courses:  Lots of problems in this area and just saw a little action from Moodle HQ on this.
 
Average of ratings:Useful (1)
Tim at Lone Pine Koala Sanctuary
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

That you for your feedback. Of the extra issues you point out:

1. and 3. are really related. Backup and restore is so hard to get right because of shared questions, so we really need a better model of sharing. However, that is a separate problem. One step at a time.

2. Yes, I am strongly inclined to add the drag-and-drop question types to Moodle core. The main reason I hesitate is that they do not work reliably on phones and other touch-screens, but it is close.

 
Average of ratings: -
Just wondering . . .
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

Quick side comment Don: you said "A pity that these are outside of core, and need reworking with each new Moodle version" - I was sure there was a tracker item for this, but I can't find one. Maybe wrong search terms. I agree about this this question type.  Is it worth a suggestion for i to be in core?

-Derek

 
Average of ratings: -
Picture of Joshua Bragg
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

My thoughts:

Sequences of variants:
Yes!  I've been using Cloze questions to do this.  I've already expressed my interest in the Combined question type to do more of this also.  I guess I'm confused about why you would need both of these options?

Repeating questions in situ:
I'm not sure that I'm seeing this behavior in the example link you sent.  I've tried the first question and keep getting oxygen each time I revisit that question after I've exhausted all my hints.

With that said, I think that's a really interesting option.  I can see why students love it.  I really think this option should be worked into the Question Practice module too.

How does grading work with this option?  Are there penalties associated with the restart?

Modernizing Quiz Editing:
Yes!  I was doing some training recently and was struck by how much drag and drop has infiltrated so much of Moodle, but not the quiz interface.

Quiz Sections:
I understand why the OU doesn't want to the development on the score by subsection since you all clearly don't need it.  With that said, there have been a lot people to ask for the subscore feature.  If this gets incorporated, there will be a good bit of confusion on why the subscore option is not included.  

Item Analysis/Statistics:
This isn't in your list but I remember you saying that it was on the list at OU at some point.  What's the status on including more options on the reports for showing more answers than just the final answer on adaptive (and interactive?) behaviors?

Overall, these are really great additions.

 

 
Average of ratings: -
Picture of Sanjeev Kumar
Re: Proposed changes to the quiz edit page from the OU
 

This is very good for us (moodle user) thank you One vote from my side.

Two things more

If it can be Section wise Navigation also changed like because if I have hundred Questions in my quiz then It shows lengthy.

In Question every question per page is good but if we can add the facility to show all question one page then it will great reason is same if 100 questions and so students can make plan which he/she wants to answer because time is running.

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Thanks for you comments.

Using combined question types to sequence variants is just a hack. It forces students to answer all the parts at the same time. Also, there will always be questions types that can't be combinable, probably the more advanced and interesting ones. That is why we need proper sequences.

Yes, something seems to be wrong with the randomisation in s205.ayrf, or it may be that it is set up differently to how I thought. Anyway, I am sure you get the ides of how it should work. As for how the restarts affect grading, that is a detail that is still to be decided. We probably need options. In most of the places we use it, we are trying to encourage students to do more practice, so I think we just take the grade of their final attempt.

I know lots of people want score by section. If you look at the full spec, you will see we considered it, but then cut it to keep the scope manageable. When I am thing about the database design, I want to try to include the necessary places to store the information, even if we don't have time to build the UI.

The quiz reports improvements are being worked on by Jamie Pratt now, but he has only just started. Looking good for Moodle 2.6 though.

 
Average of ratings:Useful (1)
Just wondering . . .
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

From Tim: "I know lots of people want score by section. If you look at the full spec, you will see we considered it, but then cut it to keep the scope manageable. When I am thing about the database design, I want to try to include the necessary places to store the information, even if we don't have time to build the UI."

+1, if we cannot get scoring by section now, it would be great to get it.

The scenario we need is like this: lots of topics that require a passing score (maybe 4/5 of questions correct) and I've just taken to having lots of small quizzes.  To be able to have a quiz with several sections and to selectively retake sections would be great.  But I know it is a big ask.

-Derek

 
Average of ratings: -
Picture of Ray Morris
Yes please, yes please, how about the same question
Group DevelopersGroup Particularly helpful Moodlers

Tim and OU are staying quite busy with improvements, aren't you.

We've had requests for sections (with subgrades) and for question dependencies, so it would be great to see those implemented. In fact we're doing hackish sections right now, creating a Topic with multiple quizzes in it, then using gradebook to create a combined grade.

 If they click that button, then they can try the question again within the same attempt,

> and if it is a random question, they will see a different variant.

If they want to retry the question, it would be a very nice option if they didn't get a different, random question.

Random questions solve most of the backup and restore issues, which is the only way to import / export.  If you have a development server and a production server, every quiz needs to be exported from development and imported on production, so as far as we can see all quizzes pretty much need to be all random questions in such an environment.  That changes this:

>  If they click that button, then they can try the question again within the same attempt,

To this:

>  If they click that button, then they can try a different question within the same attempt

 

 
Average of ratings: -
Picture of Michael Weston
Re: Yes please, yes please, how about the same question
 

Hi Tim,

Is there an ETA on the sections functionality for quizzes?  All of my quizzes are over 100 questions long and they normally contain questions that pertain to 5 or 6 different topics (sections).  I don't have a need to separately grade each section.  

From a navigational standpoint, having the navigational block display the various section headings would make navigating between questions a whole lot easier.

 

 
Average of ratings: -
Just wondering . . .
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

This is looking good Tim.  We are just dabbling at using quizzes here, and I'm not sure I understand everything.

The UI question: scores of questions on a page is always a big ask.  What about an option to edit, preview, develop at the section level?  Maybe a little more power there to do things could be possible?  

Good luck.

-Derek

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

This is getting closer to happening. Colin, Mahmoud and I started discussing how we should implmement this earlier today. (We don't have the answers yet, but we understand the question better.)

Anyway, I really just wanted to say I had created MDL-40987 to track this work.

 
Average of ratings:Useful (1)
Cochesaurus
Re: Proposed changes to the quiz edit page from the OU
 

Hi Tim,

I'm really happy that sections are being proposed!!

And is just exactly what I was asking the other day about a 'random quiz based on categories' (but I couldn't explain it well in my topic). According to that topic of mine, Categories are like Sections.

Returning to the idea, I think it would be nice if like other people commented, the sections can be graded or have a score, and also have the option to display the sections in order (like in the example) or display the questions for a given section randomly in the quiz. Some other things to add might be:

  • Create pool of questions that can be used for a specific section (a section may fetch all or a subset of, optionally random questions)
  • I'm thinking out loud, maybe create a sub-section that will have the purpose of throw sticky questions to the quiz (e.g. in a sub-section, add questions about a difficult topic, that will require more effort and time from the student to answer. Configure that only one or two of this questions appear in the quiz)
  • At the time of submitting the quiz, show the score obtained for each section, just to make the student study more and focus in that areas (sections)

Thanks for considering this.

 
Average of ratings: -
at the OU (Perry building)
Re: Proposed changes to the quiz edit page from the OU
Group Developers

HI,

Please find attached the mockups for proposed changes in quiz editing page and quiz navigation. The quiz navigation is self explanatory.

On Quiz editing page and Add icon provides following options:

  1. Add a section heading: This will trigger a Moodle form with one text field.
  2. Add questions from question bank: This will trigger question bank as a popup window.
  3. Add a new question: Will work as the current “Add a question” button
  4. Add a random question : Will work as the current “Add a random question” button
  5. Add sequenced question: Similar to random questions chosen from a category, but the questions are grouped. i.e. they have to remain in sequenced after repagination.

Section heading

A quiz section heading is the text that used in quiz navigation for inserting a line break into quiz navigation and grouping the buttons. It is useful to break down quizzes with large number of questions into sections. Therefore, we are going to make the section heading collapsible in order to enhance drag and drop experience for quiz editors.

Paginating the quiz

Depending on quiz setting on the left-hand-side a vertical curly brace indicates the page number and there is a link icon (orange chain on the left)  under each question which is set by default as follows:

  1. Vertical: There is more than one question on a page and the current question is not at the end of the page.
  2. Horizontal: The current question is at the end of the page or there is only one question on the current page.

Clicking on the above icon allows users to repaginate.

Question dependency

Quiz authors can describe dependencies between questions. For instance, in this mockup the question 2 depends on question 1. i.E. the students have to answer question 1 befre being able to see question 2 (BTW, this is not reflected on my quiz navigation mockup). However, on quiz editing page the link icon on the right hand-side shows that effect.

 

Drag and drop

We are going to use drag and drop technology similar to the course page (2.6).

In terms of drag and drop technology, we could consider the section as a container so that, when a section is moved the entire content of that section move with it. However, it would be more flexible, if we did multiple drag and drop. That way, we do not have to treat a section as a container and we still could:

  • Move entire section including section heading
  • Move multiple questionstogether

 

I hope that other information on the mockups are self explanatory.

comments please!



 
Average of ratings:Useful (1)
Logo CCCC
Re: Proposed changes to the quiz edit page from the OU
 

The following point is perhaps not directly relevant to the current initiative.

However, I am still hoping that one day Moodle will enable markers to do their work without having to navigate through a course to individual quizzes, assignments etc. So when constructing your database structure and linkages, please provide for the possiblity of accessing any question/quiz data from outside the course by addressing the database item.

 
Average of ratings: -
Picture of John Rodgers
Re: Proposed changes to the quiz edit page from the OU
 

In the history of the quiz module, the last great change, from a  pedagogical standpoint, was the introduction of the 'adaptive mode' which took the quiz module from an assessment tool to a learning tool. Smaller milestones have been the implementations of flexible feedback.

These are improvements I have wished for for a long time and would represent a new milestone. The 'next other thing' I have wished for would be quiz-instance level control of the value of a question penalty. Setting the value at question authoring time doesn't make much sense unless you can be sure of the context in whaich the question is deployed.

 
Average of ratings: -
Picture of Marcus Green
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

Could you elaborate on your idea for quiz-instance level control of the value of a question penalty, it sounds interesting.

MG

 
Average of ratings: -
Picture of John Rodgers
Re: Proposed changes to the quiz edit page from the OU
 

Currently I use 1.9. Perhaps things have changed since then.

When a question is authored, there is a penalty specified on the question which is applied to each submission in  adaptive mode. The default is 0.1.

I often repurpose questions based on student context. Sometimes a penalty of 0.1  is appropriate, particularly for new material, but sometimes 0.5 would be better for example during consolidation activities. It would make sense to be able to control this at the quiz level, since it tends to be deployed closer to the intended context of the question.

 

 

 
Average of ratings: -
Picture of Marcus Green
Re: Proposed changes to the quiz edit page from the OU
Group Particularly helpful Moodlers

I'm still not quite following the concept. Are you saying that if you re-use a question in a different quiz (a.k.a. a different context), you might want a different penalty value?). Ahh I think you want to set (or override?) this at the quiz level rather than the question level?

 
Average of ratings: -
Picture of John Rodgers
Re: Proposed changes to the quiz edit page from the OU
 

Yes.

 
Average of ratings: -
Picture of Joseph Rézeau
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Particularly helpful MoodlersGroup TestersGroup Translators

John "Currently I use 1.9. Perhaps things have changed since then."

Oh yes things have changed... dramatically. I would certainly advise upgrading to a more current Moodle version (currently 2.6) to take advantage of those dramatic changes improvements.

Joseph

 
Average of ratings:Useful (1)
Picture of Aurelio Gallardo
Re: Proposed changes to the quiz edit page from the OU
 

First of all, thanks for your hard work!

Forgive my poor English. I'm not a native speaker and still find it difficult! smile

A couple of upgrading proposals:

  1. Please, just a little, insignificant, copy icon at quiz edit page! I find heavy slow entering a question edit mode to copy the question. Doesn't it appear at edit course page for resources/assignments edition?
  2. Even I thought to program it, yet little time to do it! Why not a calculated question accepting fractions as answers?

Thanks again!

 

 
Average of ratings: -
Pierre Pichet
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Particularly helpful MoodlersGroup TestersGroup Translators

Hi Aurelio,

"Why not a calculated question accepting fractions as answers?"

Simply to keep it simple as the decimal fraction is the mostly used fraction to translate the binary number representation that is handled by computer languages as PHP on which Moodle is  build.

Pierre

P.S. If the students can answer with a fraction, the teacher could also define the answer as a fraction. Is 2/4 will have the same grade as 1/2, etc.

 

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Sorry folks, but we have just taken stock of where we are with 2 weeks to go to the Moodle 2.7 code-freeze, and this is not going to be ready to go into Moodle 2.7. You are going to have to wait another 6 months for these new features. Hopefully that will give us time to make it really good. (And, we are not expecting it to take the whole 6 months smile).

 
Average of ratings: -
Picture of Darren Cocco
Re: Proposed changes to the quiz edit page from the OU
 

This is unfortunate to hear.

Is this being developed in a public repository or an internal only OU repo?

 
Average of ratings: -
Picture of Cris Fuhrman
Re: Proposed changes to the quiz edit page from the OU
 

Hi Tim,

I just saw this thread (my comments are late, but it seems that's OK). 

Here's my feedback on specific points (if I didn't mention a point, it's because it seems straightforward and/or I support your comment in [brackets]):

2. Question dependencies

It's a great idea, but please consider defining dependencies in questions as opposed to the quiz editor. For example, a student who doesn't understand what a metaphor is won't be able to answer a question about providing a metaphor that relates to an idea. Therefore, questions (several, not just one) validating a student's understanding of what is a metaphor should be linked (prerequisites) to questions asking students to provide metaphors.

In some knowledge modeling schemes (I tend to use MOTPlus because I have the software), this is simply a relation between principles. I suggest if you implement this feature, it be integrated into questions in the question bank. The dependencies should be defined at the question level, and the quiz editor should allow instructors to construct quizzes using the dependencies (as you describe).

A user could optionally specify that a question "depends on" another, or is a "prerequisite to" another. Here's a domain model (not a database model) describing the problem:

Optional inter-question relationships

I created this using PlantText.com if anyone is curious.

3. Sequences of question variants

I'm not sure how common the need of the feature is that you describe. I would just make in the question bank a category called "recognize X" (where X is the picture) and create a random question in the quiz that takes the N variants in the category. 

But there is an idea for question variants that I saw in a MOOC on Coursera I took last year. Quizzes were set up with 3 attempts, with the best score counting as the grade. This means you got some of the same questions again with the following attempts. Interestingly, there were variations in the questions. One way to vary was a question A that is logically negated as question ^A as follows:

Question A Logical inversion of Question A

A. Which are colors (multiple answers):

  • red
  • blue
  • hot
  • yellow

^A. Which are not colors (multiple answers):

  • yellow
  • hot
  • red
  • blue

I wasn't sure if the instructor of the MOOC created these variants, or if the software did it. But I realize it could be done in software. There are possibly other patterns. Probably a question would have two stems, the positive form and the negative form (this would be hard to support automatically for all languages, as negation has syntactic rules that differ). 

Note that these kinds of automatable variants could help in the problem described in #4. 

5. Modernising the quiz edit page

Agree that this needs improving. I support aiming for consistency with one editor as you suggest. There are several areas where lack of consistency causes me extra cognitive load as a user of Moodle in big courses:

  • Moving things around in the course editor (activity layout) is different than moving things around in a quiz.
  • Granting an extension for a quiz is very different from granting an extension to a homework (this doesn't happen often, but when it does, I lose time unless I wrote the procedure down because I have to Google the videos that show how to do it).
  • Drag and drop in activity layout doesn't require clicking "submit", but it does when you upload files. I stopped uploading files to Moodle for this reason and use Google Apps with a shared folder because it's many fewer clicks and truly "drag and drop."
  • "Edit" (the gear icon) means different things in different contexts - it might mean the "content" of an item, or the "properties" of another. I always have to look at the menu on the left to know.
  • etc.
 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Proposed changes to the quiz edit page from the OU
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Thanks for you feedback and suggetsions.

I think what you describe under 2. is not a quiz. It is some sort of educational activity using computer-marked questions, but at least in Moodle's terms it is not a quiz. (Sounds a bit like DIAGNOSYS, see http://www.staff.ncl.ac.uk/john.appleby/diagpage/diagindx.htm.)

3. Also is something other than what we are currently doing.

Regarding 5. Yes we are basing the new quiz interface heavily on the idioms established by the course page editing.

 
Average of ratings: -