Quiz-Questions development:questions for Tim

Quiz-Questions development:questions for Tim

by Pierre Pichet -
Number of replies: 24

To the quiz community,

Tim will return from a  short but real holidays i.e. without his laptop.

We expect that there will be some sort of "What's next".

So I open this forum issue so that YOU can put here your main questions or priorities on quiz-question development.

This could help Tim setting the priorities.

As en example :

Users love multianswer ( cloze, formula etc.) as a rapid way to build an almost complete quiz. 

How far should we go to give to the enclosed questions all the properties of standard questions?

 

Pierre

Average of ratings: -
In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Steve Turley -

Thanks for the invitation to provide input!   From my standpoint, the following priorities would be good (in this order or priority).

  1. I think the first thing we need to do is make the new questions format more stable, improve the developer and user documentation, and fix the bugs.  For me, the inability to use calculated or numerical questions in adaptive or interactive mode is pretty serious.
  2. Add an ability to format quizzes for nice printing.    Even something as simple as reformatting an HTML page so that it would be more easily printed form a browser would help a lot of us who would like to give paper copies to our students or reviewers for various reasons.
  3. I agree with you Pierre that the ability to bundle multiple questions together like cloze or formula questions would be great to have as properties of standard questions.  There could be several ways to do with this that would all be great enhancements
    1. adding a calculated question type in cloze questions
    2. adding a formula-like capability to have more than one questions linked together as part of a single question
    3. having more flexible kinds of inputs in calculated questions (again like formula).  It would be nice to be able to restrict the random numbers to have a particular spacing (every 1 or every 0.5 for instance).  It would also be nice to have random non-numerical values coupled with numerical ones.  As an example, I have used formula questions to ask questions about the physics properities of various random elements.  The random element symbol can be tied to its atomic number, binding energy, decay modes, etc.
  4. Generalize the interactive with multiple tries type in a way that would eliminate the need for adaptive questions.  This would simplify the interface for users.
    1. Allow users to specify how many tries independent of the number of hints (including the possibily of working with no hints).
    2. Give students the option to retry the question with a hint (for a penalty) or without a hint (for no penalty or a reduced penalty).
  5. Provide an interface to make quizzes with more of a lesson-type format.  The following features would be nice:
    1. Allow a richer description question types which could be target as instructional text rather than question introductions.
    2. Have checkpoints in quizzes where students have to meet a certain standard up to that point in order to proceed.
    3. Be able to bundle sets of contents and quizzes together in a way that looks like a self-contained lesson to the students.
In reply to Steve Turley

Re: Quiz-Questions development:questions for Tim

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Steve, here are my comments on your "wish-list".

1.- first things...

  • there is already quite a lot of documentation available for developers, and, as Tim often says, the best way to understand how the question engine works is to have a look at the scripts themselves, and the comments.
  • fixing bugs : of course ! It helps if people who spot bugs a) report them in the tracker; b) hint at a possible fix. The new development process of Moodle makes it a) more reliable but b) lenghtier to get bugs fixed, even when providing a simple fix. See e.g. MDL-28259 where I provided a simple fix (a couple of lines) but has remained un-fixed for a month. Of course we must acknowledge the fact that Tim's time is limited and he has to prioritize the bugs.
  • Since I have no use for numerical & calculated questions I will not comment here.

 

2.- I keep being amazed at this recurrent wish for a "print feature" for quizzes. To me, wanting to print quizzes that are available online is "going back into the past".

3.- & 4.- Yes, it would be nice if the cloze/multianswer question type was "fully pluggable", and not limited to a certain number of standard question types. See this discussion for Tim's answer.
Again, I have no use for "atomic number, binding energy, decay modes, etc." so will not comment.

5.-"Generalize the interactive with multiple tries type in a way that would eliminate the need for adaptive questions." I am not sure I understand what you mean here and certainly do not agree that "adaptive" question mode should be removed. On the contrary, it is for me the most important feature of Quiz.

6.- Hints

6.1- "Allow users to specify how many tries independent of the number of hints (including the possibily of working with no hints)." By "user" I suppose you mean "teacher". And I agree with your proposal. But this should be argumented with concrete examples.

6.2.- Why not? but again, should be argumented with examples.

7.- Make Quiz more like Lesson...

I am not sure this is a "reasonable" proposal. We regularly have discussions in these forums where teachers would like some Quiz/Questions features to be implemented within the Lesson activity and ... vice-versa. However, I am of opinion that the Quiz and the Lesson activities are very different types of activities, each with its own strengths and weaknesses, and it would be better to improve both, as separate activities. The Quiz activity (and the Questions bank) has been tremendously improved thanks to Tim's indefatigable energy.approve Unfortunately, for quite a few years, the Lesson activity has lacked a maintainer and all significant development of that module has stopped, only partial bug-fixing is happening.sad

In reply to Joseph Rézeau

Re: Quiz-Questions development:questions for Tim

by Steve Turley -

Joseph,

I appreciate you taking time to respond to my previous post.  Here are a few clarifications.

1. first things...

  • Good point on the documentation.  I haven't looked at the code itself (which I should).  I only went to the development wiki and noticed it was somewhat behind.  The user documentation also needs some work explaining some of the new choices.  Clicking ? on "How questions behave" doesn't explain those options very well yet, for instance.  (Perhaps this is someone else's assignment besides Tim's.)
  • I understand on the bugs.  The particular one I mentioned is already in Tracker.

2.  (print feature) I keep running into situations where students or course reviewers would like to see my quiz questions during a period when the quiz is not currently open.  In a student role or guest role, this option isn't available to them.  Here are some examples that have come up in my personal teaching.

  • A student didn't complete an online homework assignment by the deadline.  Because of this they didn't get credit, but wanted to see a copy of the questions so they could use them to study for the exam.
  • A campus administrator or committee is reviewing my course.  They wanted to see samples of assignments I give students to assure themselves that my assessments are aligned with my published learning objectives.
  • A colleague (who did not use moodle) asked for a copy of some of my quizzes for adaptation in a similar course.

5, 6.  Interactive with multiple tries type: Tim has created a new question behavior that seems quite powerful called interactive with multiple tries.  It is similar to the older adaptive mode except that students are presented a new hint for each successive try at an answer.  When no more hints are available, the student is given no more tries.  I agree with you that adaptive questions are a powerful and important feature in moodle.  My suggestion wasn't to eliminate them, but rather to consolidate these two types.  Since they are so similar, it seems like this would improve maintenance and reduce complexity for users.  If the new interactive with multiple tries questions had an option box to not display the hints (with everything else the same) they would be essentially the same as the old adaptive questions.  To generalize them further, they could give the students an option to either ask for a hint (with an associated penalty) or try again without a hint.

You requested an example.  Here is one:

Question (calculated): Muons at rest in the laboratory frame have a half-life of 1.6 microseconds.  In an experiment with high energy muons going close to the speed of light, they are observed to have a half-life of {hl} microseconds.  How fast are the muons travelling in meters per second?

Correct Answer: 3e8*sqrt(1-pow(1.6/{hl},2))

Hint 1: Recall that time dilation is responsible for the difference in lifetimes for the muon in the two different reference frames.

Hint 2: The relativistic factor gamma is the ratio of the half-life in the moving frame to the half-life in the rest frame.

Hint 3: The relativistic factor beta is related to gamma by the relation gamma = sqrt(1/(1-beta^2))

Hint 4:  The speed v is related to beta by the relation v=beta*c, where c=3*10^8 is the speed of light in m/s.

One student tries the question and checks to see if their answer is right.  They are very confident about knowing how to do this question but get the answer wrong.  Upon checking their work they discover a simple error such as keying the numbers  incorrectly on their calculator and want to resubmit the answer without additional assistance.  Upon submission, they discover their answer is correct.  An appropriate score in this case would involve a small penalty for being sloppy using the calculator, but reward the student for being able to answer the question without additional assistance.

A second student is very uncertain about how to approach the problem.  They come up with an answer but are quite uncertain they even have the correct approach. When they check their answer, they find that the answer is wrong.  They check their work and can think of no way to do things any differently.  They decide it is worthwhile to try the question again after getting some assistance in terms of a hint.  After they see the hint, they recognize how to solve the problem and solve it correctly.  They get partial credit for the question, but less credit than the student who arrived at the correct answer without needing the hint.

A third student tries multiple random guess in solving the problem, hoping some combination of the supplied parameters might give him or her the correct answer.  Eventually, they exhaust the available credit and get no points for this question.

Presently, adaptive questions (if they worked for calculated questions) wouldn't work because they only give one hint (feedback) no matter how many times a student has attempted a question.  The interactive with multiple tries questions gets closer to working.  Students can get more and more hints as they need them on successive tries.  A student who doesn't require multiple tries can get more credit than a student requiring hints to answer the question.  However, this question type still lacks the ability to allow a student who doesn't require a hint to try answering again without using the hint which he or she may not need.

7.  What I was suggesting here was a way to take advantage of Tim's great work by being able to bundle existing moodle components activities and resources together in a bundle, similar to what a lesson does.  The linking and branching in lessons need to be present, but the chance to simply give students a linear progression through set of activities related to the same topic seems helpful.  Let me give you an example to illustrate what I mean.  Let's say I wanted to give prepare an online activity for my students to help them understand Newton's Second Law.

  • They start by clicking on a link in moodle which says Newton's Second Law.
  • Next, they are asked to take a pre-assessment quiz (using Tim's quiz module) to see how well they understand the Second Law already.
  • If they get above a certain score on their quiz, they don't need to proceed further and get credit for mastering the concept.
  • If they get below that score, they are asked to read a resource (HRL, web page, file, wiki, ...) dealing with the first aspect of the concept.
  • Next they are given a mastery quiz (using Tim's module again) to see if they understand the first part of the idea.
  • If they have done well enough on the quiz to move on, they are allowed to do so.  Otherwise, they are asked to go back to the resources about the idea and review them (or choose other resources).
  • After they have demonstrated mastery of this first concept, they are given another.
  • After they have finished all concepts, they are given a final assessment on Newton's Second Law.

Many of the ideas in my field of physics are vertical.  Students who are used to using the web to randomly search for information often don't recognize the need to proceed sequentially through course material in order to be able to build on earlier concepts when presented with later ones.  This kind of approach helps guide them in the kind of learning they need to master this kind of vertical material.

The building blocks for building this kind of "lesson" are mostly already there in moodle.  With completion tracking I think we can set up the check points as I've outlined above.  We can then use existing moodle modules for resources and activities.  We're just lacking the ability to bundle a set of activities together in a nice unit where we can tell the students to complete an entire activity and give them a single grade for that unit when it's done.

In reply to Steve Turley

Re: Quiz-Questions development:questions for Tim

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Steve,

Many thanks for your detailed reply to my remarks re your original post. I need some time to "digest" it all, and especially to gather my arguments "in defense of the Adaptive mode", especially in the light of Pierre's quotation of Tim, in this discusion, re Adaptive vs Interactive.

ATB,

Joseph

In reply to Joseph Rézeau

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

Joseph,

Don't worry, I think that Adaptative and Interactive should be maintained as they are 2 ways to do something similar but not identical.

I quoted Tim to help the discussion. wink

The problem with the Hint is that you have to guess the most "popular" student error.

This is more easy than to code the hint from the student response as done in adaptative in your favorite question type.  

 However adaptative and hint are somehow trying to do in a quiz what is more of the lesson domain

Pierre

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Steve Turley -

Pierre,

I'd like to comment on your last statement.  If you think of quizzes as a summative assessment of student learning, then you're absolutely right.  However, many of us use these as formative activitivies which are part of the learning process as well as part of the assessment process.  At our university, the quizzes are mostly used for homework problems where students are getting practice learning how to perform tasks that they are tested on later.

I agree that it is very challenging to guess what kinds of hints will be most useful to students when they make a mistake.  In multiple choice questions, the combination of hints and specific feedback can help tailor the hint to the error the student made.  On calculated and numerical questions it is almost impossible to give students custom hints based on their particular errors.

That said, I really like the direction both these questions are going.  I think they are moving towards being powerful tools to engage students, encourage them to think about their errors and learn to correct them, and reward them for being persistent.

Please don't misinterpret my suggestion of combining Adaptive and Interactive questions as suggestion the functionality of either should be lost.  Rather, I was suggesting that their separate functions are similar enough that perhaps it might be clearer and easier to have one flexible question type which gives one the option of doing what one can do with either of the separate times now.

Steve

In reply to Steve Turley

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

Hi Steve,

It is sometime "bizarre" how a chemist can be more at ease with chemistry formulas than with pedagogical terms.

In the real world, I am using Moodle quiz essentially as homework and mostly on calculated so that the student could master the different quantitative aspects of analytical chemistry.

In the standard paper exams, I use mainly questions that were included in their moodle quiz.

I try to set as answers the most common faults so the grading and the feedback can guide the student to the right answer. Classical adaptative is very usefull in this context and probably more useful than the hint process.

But this is quite specific to numerical data.

As far as I understand, there is no fundamental disagreement wink

Pierre

In reply to Steve Turley

Re: Quiz-Questions development:questions for Tim

by Jeff Forssell -

"On calculated and numerical questions it is almost impossible to give students custom hints based on their particular errors."

I don't find that true. The possibility of having a specific numerical interval, I have felt, made giving specific response for numerical answers much easier than for open text answers in other subjects. 

I have collected common mistakes (like radius used when diameter should have been used) and made specific responses to help correct them. Answers in an interval can of course be arrived at through different routes, so the feedback should be carefully worded "An answer like the one you gave could be gotten by doing .....   in which case you should be  aware that .....  . If you got that answer in another way, please tell us!"

But what I would really like is that there be a student feedback to teacher channel from each question. (A description of how that could be) That could be used, for example, when the student gives an answer that doesn't meet any of my prepared "right and wrong" alternatives which could get the feedback:

We didn't recognize your answer as one we expected. If you didn't just make a wild guess, could you please describe how you got the answer that you submitted? Thank you!

If it seems like an error (or alternative RIGHT answer!) that other students might make, I can make appropriate feedback for that question.

I would also like the possibility of having numerical questions evaluate an answer expression, like sqrt(1/(1-2.5E8/3E8)), which is often the ability we are looking for (even if it is also good if they also can punch it in to their calculator and round off properly ). This has the added value of letting the teacher see HOW the student was thinking, which values he put in. Then I can often make suitable feedback without additional feedback from the student. 

I have a numerical wishlist (but many things have been already addressed in newer Moodle) here.

In reply to Jeff Forssell

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

"it is almost impossible to give students custom hints based on their particular errors"

Here hints referred to the "new engine" Hint which is defined at he question level

"The possibility of having a specific numerical interval, I have felt, made giving specific response for numerical answers...so the feedback should be carefully worded"

Here you are writing about "classic" feedback.

So finally we seem to agree wink.

Pierre

 


In reply to Steve Turley

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

Just a little comment about calculated and interactive. The hint is shown (Moodle demo ) but we should add the possibility to put the {x} and or an equation result {={x}/2} as in question text or feedback.

Such an improvment should be set in the tracker if not done.

Pierre

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

As a reminder :

Interactive

This will replace adaptive mode. This is the model that has been used successfully for several years in the OU's OpenMark system. The OU has also modified Moodle to work like this, but because of the way the quiz core currently works, I was not happy just merging the OU changes into Moodle core. In a sense, this whole document grew out of my thoughts about how to implement the OU changes in Moodle core properly.

In the existing adaptive mode, after the student has clicked submit, the question both shows the feedback for the last answer the student submitted, while also letting the student change their answer. This can lead to weird results. For example create a numerical question 2 + 2 = 4. Attempt it. Enter 4. Click Submit. Enter 5. Click Save. You will see a page that seems to say that your answer is 5 and it is correct!

In Interactive mode, the question is either in a state for the student to enter their answer, with a Submit button; or it is showing feedback for the student's previous attempt, with a Try again button to get back to the first state.

The other difference is that students are only allowed a limited number of tries at each question (typically three). When they submit their third try, or when they submit a correct answer, that question is finished and they must go onto the next one.

from

http://docs.moodle.org/dev/Question_Engine_2:Overview#Interactive

Pierre

P.S. In the actual 2,1 the number of tries  = number of hints + 1 

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Jean-Michel Védrine -

Hello Pierre,

Thanks a lot for starting this.

The first think I want to say now that my production 2.1.1 website is online and that students are here (first training quiz has not started but I have spend a lot of time adding hints to old questions and modifying them to look better in interactive behaviour mode) is that the quiz "experience" is a lot better in QE2 than it was before. Unfortunately in the upcoming weeks I will use it with "new" students so they will not be able to compare with QE1 but I hope they will like it too.

IMHO it's time the question bank get some love, having working a lot creating questions in the last days I think this is now the weakest point.

Import/export formats should be improved too, apart XML and gift nothing is really working here now !

About new questions types, maybe it would be better to fist let some time pass before thinking to new types. New OU questions types are not really released now, so when ddwtos, gapselect, ddimagetoimage, varnumeric(set), pmatch will be largely available and people can play with them, maybe they will have a better idea of what else they want.

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Oleg Sychev -
Picture of Core developers Picture of Plugin developers

Well, my priorities are quite known to the Tim, but maybe they get support there:

1. get convenient two-click interface to move question categories for any distance (MDL-20251) - these moving by one step at a time is a nightmare!!!

2.  get random question select from unseen questions for this student, not duplicating them (MDL-6340)

3.  get a way to disable HTMLEditors for feedback in question editing to speed up the process

4.  a good way to construct one-page quizzes: turn summary page off if everything is answered etc

5. versioning for the questions, so that editing the question don't affect data from previous attempts

6. a good set of dehavours for hinting and related penalties

In reply to Oleg Sychev

Re: Quiz-Questions development:questions for Tim

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

I vote for Oleg's #3 : disable HTML editors when editing questions which do not require them.

In reply to Joseph Rézeau

Re: Quiz-Questions development:questions for Tim

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Oh, and while we are on the topic of the questions editing interface, just remembered an old request of mine: Re-ordering and insert Answers tool needed for Questions editing... dating back to Feb. 2007.

At the time Tim answered "That is a good idea, but a fair bit of work, so don't hold your breath." Tim, I've been holding my breath for 4 1/2 years now.clown

Joseph

In reply to Joseph Rézeau

Re: Quiz-Questions development:questions for Tim

by Jeff Forssell -

For us that use question feedback for specific detailed guidance and who are  continually refining it as the questions are used, this feature would be extremely valuable. An interface like the one for reordering questions in a quiz would work fine. That even has some advantages over an AJAX drag around type.

EDIT: I just tried (using FF 6.0 on XP sp3) clicking on the word interface after submitting this. A rather lengthy term definition box came up with two bad characteristics:

most important: It extended below the window so some text was not visible. It was not moveable!

less important: The text was certer justified.

Is that the same for everyone?

In reply to Jeff Forssell

Re: Quiz-Questions development:questions for Tim

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Jeff "EDIT: I just tried (using FF 6.0 on XP sp3) clicking on the word interface after submitting this. A rather lengthy term definition box came up with two bad characteristics: most important: It extended below the window so some text was not visible. It was not moveable! less important: The text was certer justified. Is that the same for everyone?"

That is one of the problems with the new way that Glossary definitions are displayed in Moodle 2. Please re-post to the Glossary forum where I'll be happy to provide more info.

Joseph

In reply to Joseph Rézeau

Re: Quiz-Questions development:questions for Tim

by Jeff Forssell -
Weird nr 2

Today the word "interface" is not linked at all!
I will go to the glossary forum.

EDIT: But in this post it is?!?
In reply to Joseph Rézeau

Re: Quiz-Questions development:questions for Tim

by Oleg Sychev -
Picture of Core developers Picture of Plugin developers

Actually we started work on "Re-ordering and insert Answers tool needed for Questions editing... ", doing a bit of complex things, but some problems left, mainly with HTMLEditor and File Picker moving, which require some more Javascript tinkering...

I may share a patch if anyone want to complete it. I myself lack time to do it now.

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

How do we develop editing multianswer question types.

In the Cloze case, many users are not at eased with the cloze syntax so

  1. a  separate (external) shortanswer, numerical, multichoice editor can be used to generate the equivalent cloze syntax that can be copy and paste at the convenient place in the questiontext
  2. the question text can be edited by a modified HTML editor to do the same process.
  3. the multianswer_edit form decode answer function can be improved so that the individual questions can be edited using the standard moodle form display.
There are pro and cons for each solution among them is how and where the validation of the individual questions is done.
Pierre
In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Steuard Jensen -

Just in case this thread gets extra attention, can I put in a request to fix the nasty session timeout dataloss bug that I talked about in my recent post and that's described in bug MDL-11972? Everyone has their own priorities, but if I were running the Moodle project, a single bug report half as disasterous as the one in that bug would immediately mark the issue as very severe, and when it's as easy to hit as it appears to be I'd have elevated it to "blocker" status years ago. (I have been *personally* hit by that behavior twice in the past week alone while editing course pages, and a lot of my students lost lengthy quiz sessions before I figured out what was going wrong this week.)

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

Tim,

The 2,0 and 2,1 implied important code changes for question types that necessitate an important task for all those that supported the question types. The task was surely greater for contributed plug-ins given that few people have your competence.

I think that contributed plug-ins that have a working 2,1 version should be considered as a special class that could be recommended to Moodle administrators. 

The recent survey on questions showed that core questions were much more used than contributed ones.

 

Is there any major changes in quiz or general moodle structure that are planned that could necessitate the same amount of work for question-type apart from the building of general functions that will facilitate the building of new question types?

Pierre 

In reply to Pierre Pichet

Re: Quiz-Questions development:questions for Tim

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Interesting descussion, which I don't have time to respond to in detail now. I will just note that my list is at http://docs.moodle.org/dev/Goals_of_an_online_assessment_system#A_roadmap_for_computer-marked_assessment_in_Moodle, but I have not updated this recently.

In reply to Tim Hunt

Re: Quiz-Questions development:questions for Tim

by Pierre Pichet -

As usual, you are always a step ahead wink.

And your conclusion is always true:

"One thing we can say for certain with a system that is as rich as the Moodle quiz, which so many teachers are interested in pushing to its limits, is that no matter how good we make it, there will always be more ideas about how to make it even better. Looks like I won't ever be out of a job smile--Tim Hunt 13:37, 11 October 2009 (UTC) "

Pierre