I have just written some more stuff on the docs page Development:Goals_of_an_online_assessment_system.
The new bits start at section 2 (The components of an online assessment system) which is a copy an paste of the answer I gave here to a question that was asked by Chris Collman recently.
Section 4 (A roadmap for computer-marked assessment in Moodle) is probably the most interesting bit. It is my attempt to summarise what I think are the major things we need to address in future. Although I call this a roadmap, there is not time-scale for any of it. As ever, things will only happen as and when someone has time and inclination to do something.
Naturally, I hope this provokes discussion. I already know what I think. I am much more interested to know what you think.
The Moodle quiz must always be a tool for teaching, not a toy for software developers, and I am sure I have forgotten some things, so please let us know your ideas.
I think this is a great document and rightly calls for us to avoid complacency.
I would add one item to the development roadmap: question difficulty/level.
As a language teacher I am constantly seeking to assess ability generally (i.e., not the knowledge of course content). It would be a great tool if we could rate question difficulty and have moodle keep track of how high a student could go. If a student answers a question correctly, then moodle chooses a random question from a higher level. If she answers incorrectly, moodle randomly chooses one from the same or lower level. Moodle would stop after a certain number of correct/wrong answers.
I am not a programmer, but it seems like this could be part of the quiz module, OR it could be like a lesson, but make use of the quiz questions. That is, a different engine takes advantage of quiz content.
Just an idea. I hope I expressed it clearly.
I would add one item to the development roadmap: question difficulty/level.
As a language teacher I am constantly seeking to assess ability generally (i.e., not the knowledge of course content). It would be a great tool if we could rate question difficulty and have moodle keep track of how high a student could go. If a student answers a question correctly, then moodle chooses a random question from a higher level. If she answers incorrectly, moodle randomly chooses one from the same or lower level. Moodle would stop after a certain number of correct/wrong answers.
I am not a programmer, but it seems like this could be part of the quiz module, OR it could be like a lesson, but make use of the quiz questions. That is, a different engine takes advantage of quiz content.
Just an idea. I hope I expressed it clearly.
What you are talking about is what I refer to as computerised adaptive testing under "More activities using the question bank and engine".
Difficulty of questions could be implemented by tagging questions, which is already possible in Moodle 2.0.
Difficulty of questions could be implemented by tagging questions, which is already possible in Moodle 2.0.
Hi Tim,
We would like to share some experiences in order to help improve the quiz module
We are starting to design a maths course (12-14 years old) making an intensive use of QUIZES. In fact we use a mixing of a quiz with a format lesson, the reason is to use an activity with a heavily formative purpose.
.
The course (work in progress) can be visited at:
http://www.itemspro.net/moodle/
GEOMETRY: Lines, Angles and Circles
Access to the course as a Teacher using
username=euro
password=euro
We would appreciate to get feedback and to improve the course design. After testing it in different schools and in different countries (available in English, later in Spanish, Catalan, German and Slovenian), the course will be freely released.
Thanks in advance for your help and advice
Bernat
We would like to share some experiences in order to help improve the quiz module
We are starting to design a maths course (12-14 years old) making an intensive use of QUIZES. In fact we use a mixing of a quiz with a format lesson, the reason is to use an activity with a heavily formative purpose.
.
The course (work in progress) can be visited at:
http://www.itemspro.net/moodle/
GEOMETRY: Lines, Angles and Circles
Access to the course as a Teacher using
username=euro
password=euro
We would appreciate to get feedback and to improve the course design. After testing it in different schools and in different countries (available in English, later in Spanish, Catalan, German and Slovenian), the course will be freely released.
Thanks in advance for your help and advice
Bernat
In reply to Bernat Martinez
Re: I just tried to write a roadmap for the Moodle quiz
by Jeff Forssell -
Though I really should be doing some other paperwork, I just had to take a look when you are using the quiz formatively for math.
Just made a quick look and it looks interesting and ambitious!
(You wouldn't by any chance want more working EU participants with Math/physics and Swedish background? 
)
Since you asked for feedback, just a detail I noticed:
in a multianswer question with 2 out of 4 correct, I tested what happened if I answered with 3 or all 4. I was still given "correct" for the question (though the incorrect ones had red crossses) and there was no additional feedback. One of the things I LOVE about Moodle is the abillity to give feedback, especially formative feedback for wrong answers.
The way of dealing with 2 of 4 type of question is usually to set negative points on the wrong alternatives.
(The particular question I tried was
Just made a quick look and it looks interesting and ambitious!
)Since you asked for feedback, just a detail I noticed:
in a multianswer question with 2 out of 4 correct, I tested what happened if I answered with 3 or all 4. I was still given "correct" for the question (though the incorrect ones had red crossses) and there was no additional feedback. One of the things I LOVE about Moodle is the abillity to give feedback, especially formative feedback for wrong answers.
The way of dealing with 2 of 4 type of question is usually to set negative points on the wrong alternatives.
(The particular question I tried was
UNDERSTANDING.2: Parallel and Perpendicular Lines - Attempt 1
Question 2
Drawing a perpendicular line
In reply to Jeff Forssell
Intensive use of quizes as a formative tool in a MATHS course
by Bernat Martinez -
Thanks Jeff for your useful feedback
You are right, and we should set negative points on the wrong alternatives. We will correct this.
In fact the course is work on progress and we are using it to test many different things (maybe too ambitious, as you said), especially the intensive use of quizes as a formative tool.
Now we are going to present the material via a EU online course for Maths teachers. Later, when we are pretty sure the content of the questions ( and answers too) is ok, we will start to include the formative feedback you mentioned for each alternative.
Referring your suggestion about more EU participants, I'm sorry but according our contract we got funds for translating the course only to the project partners national languages. Anyway, if you think there are any Swedish educational institution interested in translating it, we would be very happy to collaborate.
Thanks again
Bernat
You are right, and we should set negative points on the wrong alternatives. We will correct this.
In fact the course is work on progress and we are using it to test many different things (maybe too ambitious, as you said), especially the intensive use of quizes as a formative tool.
Now we are going to present the material via a EU online course for Maths teachers. Later, when we are pretty sure the content of the questions ( and answers too) is ok, we will start to include the formative feedback you mentioned for each alternative.
Referring your suggestion about more EU participants, I'm sorry but according our contract we got funds for translating the course only to the project partners national languages. Anyway, if you think there are any Swedish educational institution interested in translating it, we would be very happy to collaborate.
Thanks again
Bernat
In reply to Bernat Martinez
Re: Intensive use of quizes as a formative tool in a MATHS course
by Jeff Forssell -
I never said TOO ambitious. It looks interesting!
In reply to Jeff Forssell
Re: Intensive use of quizes as a formative tool in a MATHS course
by Bernat Martinez -
Sorry by the TOO, I didn't express myself correctly
Hi Tim!
So exciting you are getting back to working on Quiz! Good that you already have 'Better user interface for creating and editing questions' there, though I wonder if that should be solved along with other humongous forms in Moodle. With my current amount of resources (none) I'll just point you to some old research results and hope you will consider them.
I've been a bit worried about some aspects of the editing having gotten broken for quite some time, as well as the design (there are reasons for the corners only having been rounded on the right side - a detail, yes, but the visual flow is important since the screen is so cluttered). Which reminds me of Julian having said he would like to work on the graphical design of it, hm wonder what is up with that...
I would love to work with you further on all of this, but at earliest that might be possible would be in the spring (and of course I still need to find someone to fund me in case that would otherwise be welcomed). And I agree with Martin the add random question dialog needs to have something done to it. Perhaps a poll here in the forums about what people think about it would be appropriate, at some point.
So exciting you are getting back to working on Quiz! Good that you already have 'Better user interface for creating and editing questions' there, though I wonder if that should be solved along with other humongous forms in Moodle. With my current amount of resources (none) I'll just point you to some old research results and hope you will consider them.
- Feature requests from teacher interviews: http://docs.moodle.org/en/Development:Quiz_UI_redesign_scenarios_-_conclusions#Considerations_for_future_development
- Unsolved issues found in usability testing: http://docs.moodle.org/en/Development:Usability_testing_in_August_2008/Summary_of_solutions
- Overall navigation model: http://docs.moodle.org/en/Development:Quiz_Usability_portal/Where_to_go_next_with_Quiz_UI%3F_Autumn_2008 (somewhat outdated and I already know your opinion on Quick Switch, but many of the points still hold)
I've been a bit worried about some aspects of the editing having gotten broken for quite some time, as well as the design (there are reasons for the corners only having been rounded on the right side - a detail, yes, but the visual flow is important since the screen is so cluttered). Which reminds me of Julian having said he would like to work on the graphical design of it, hm wonder what is up with that...
I would love to work with you further on all of this, but at earliest that might be possible would be in the spring (and of course I still need to find someone to fund me in case that would otherwise be welcomed). And I agree with Martin the add random question dialog needs to have something done to it. Perhaps a poll here in the forums about what people think about it would be appropriate, at some point.
Well, the interface probably broke during the $OUTPUT changes, and I have not been able to give it any love since then. I may not be able to before the beta release happens. I really want to get the question engine rewrite done.
In reply to Olli Savolainen
Re: I just tried to write a roadmap for the Moodle quiz
by Pierre Pichet -
"I've been a bit worried about some aspects of the editing having gotten broken for quite some time,"
I just fix the paging problem in showbank.
Is there other problems or suggestions that I could try to solve so Tim can continue on the main quiz code?
Pierre
I just fix the paging problem in showbank.
Is there other problems or suggestions that I could try to solve so Tim can continue on the main quiz code?
Pierre
In reply to Pierre Pichet
Re: I just tried to write a roadmap for the Moodle quiz
by Olli Savolainen -
Hi Pierre,
Great that you can offer a helping hand.
What was the issue with paging in showbank?
The things that looked broken to me the last time I checked (with the last cvs update I got a "Unknown driver adodb/mysqli" error so could not check now)
- the question bank opening/closing link was not a link so you can't open/close the question bank
- lots (all?) of icons were missing (edit question icon, preview etc) rendering the entire UI unusable
Olli
Great that you can offer a helping hand
What was the issue with paging in showbank?
The things that looked broken to me the last time I checked (with the last cvs update I got a "Unknown driver adodb/mysqli" error so could not check now)
- the question bank opening/closing link was not a link so you can't open/close the question bank
- lots (all?) of icons were missing (edit question icon, preview etc) rendering the entire UI unusable
Olli
In reply to Olli Savolainen
Re: I just tried to write a roadmap for the Moodle quiz
by Pierre Pichet -
First thing done, moving qtype help files so that they can be usefull when editing a question (MDL-20597)
Pierre
Pierre
A very good summary of the things to do...and this insure that we will not be "unemployed" in a near future
.
Among other things, I agree that versioning should be reinstalled ( it was broken in 1.5 or 1.6, I think) as this is an almost necessary feature to ensure that the quiz results will remain valid if the question is modified by another user.
Pierre
.Among other things, I agree that versioning should be reinstalled ( it was broken in 1.5 or 1.6, I think) as this is an almost necessary feature to ensure that the quiz results will remain valid if the question is modified by another user.
Pierre
As far as I am aware, versioning has never worked. There was an abortive attempt to add it around the Moodle 1.5 or 6 time (before I started working on the quiz) but they just added some columns to the database and wrote some code, but it was never finished.
Seeing bugs like MDL-20570 makes we think we will have to do something about this sooner rather than later, but I think it will require some serious thinking to design a good solution before we can write any code.
Seeing bugs like MDL-20570 makes we think we will have to do something about this sooner rather than later, but I think it will require some serious thinking to design a good solution before we can write any code.
The other way is to restrict edition of a question that have attempts stored.
So we need to improve the edition process.
We could also strongly suggest to create a new question .
This could be easier to implement.
Pierre
So we need to improve the edition process.
- attempts detected on loading the edition_form and answers displayed without the add answers button.
- the validation process improved so that the user cannot modify the number of answers (or multianswers in the Cloze questions) by setting them to empty ones(nullified) .
We could also strongly suggest to create a new question .
This could be easier to implement.
Pierre
I have created a new issue in the tracker
MDL-20584 Warning user of changes when editing a question used in a quiz
And will work first on multianswer (Cloze) as this is the more critical one: not only answers can be lost but also subquestions.
Pierre
P.S. I will first do the warnings ( validation process) then the control of save and save as new options.
In reply to Pierre Pichet
Re: I just tried to write a roadmap for the Moodle quiz
by Pierre Pichet -
So the edit_question_forms should have
To be fully useable, the original value (i.e. the value stored in the database) should be displayed in a convenient mean (i.e.alongside) the new one.
Pierre
- a warning state displaying what has been changed in editing a question i.e. in a yellow color, with a confirm button so that the user can save or save as new the question.
- a warning state displaying what has been changed in editing a question I.e. in an orange color), with a confirm button so that the user can only save as new the question.
- an error state i.e. the actual validate process (in red color), that proscribe either save or save as new.
To be fully useable, the original value (i.e. the value stored in the database) should be displayed in a convenient mean (i.e.alongside) the new one.
Pierre
Using PHP Xref I just find
/lib/pear/ -> PEAR.php (source)
/lib/pear/ -> PEAR.php (source)
(one of 4 // {{{ raiseError() 474 475 /** 476 * This method is a wrapper that returns an instance of the 477 * configured error class with this object's default error 478 * handling applied. If the $mode and $options parameters are not 479 * specified, the object's defaults are used. 480 * 481 * @param mixed $message a text error message or a PEAR error object 482 * 483 * @param int $code a numeric error code (it is up to your class 484 * to define these if you want to use codes) 485 * 486 * @param int $mode One of PEAR_ERROR_RETURN, PEAR_ERROR_PRINT, 487 * PEAR_ERROR_TRIGGER, PEAR_ERROR_DIE, 488 * PEAR_ERROR_CALLBACK, PEAR_ERROR_EXCEPTION. 489 * 490 * @param mixed $options If $mode is PEAR_ERROR_TRIGGER, this parameter 491 * specifies the PHP-internal error level (one of 492 * E_USER_NOTICE, E_USER_WARNING or E_USER_ERROR). 493 * If $mode is PEAR_ERROR_CALLBACK, this 494 * parameter specifies the callback function or 495 * method. In other error modes this parameter 496 * is ignored. 497 * So just using correctly PEAR library could be the solution Pierre
Sorry for the display
.
One of the mysteries of HTML editor filtering.
.One of the mysteries of HTML editor filtering.
This method is a wrapper that returns an instance of the 477 * configured error class with this object's default error 478 * handling applied. If the $mode and $options parameters are not 479 * specified, the object's defaults are used. 480 * 481 * @param mixed $message a text error message or a PEAR error object 482 * 483 * @param int $code a numeric error code (it is up to your class 484 * to define these if you want to use codes) 485 * 486 * @param int $mode One of PEAR_ERROR_RETURN, PEAR_ERROR_PRINT, 487 * PEAR_ERROR_TRIGGER, PEAR_ERROR_DIE, 488 * PEAR_ERROR_CALLBACK, PEAR_ERROR_EXCEPTION. 489 * 490 * @param mixed $options If $mode is PEAR_ERROR_TRIGGER, this parameter 491 * specifies the PHP-internal error level (one of 492 * E_USER_NOTICE, E_USER_WARNING or E_USER_ERROR). 493 * If $mode is PEAR_ERROR_CALLBACK, this 494 * parameter specifies the callback function or 495 * method. In other error modes this parameter 496 * is ignored. So just using correctly PEAR library could be the solution Pierre
Pierre, I don't understand at all what you have against "save as new". It creates a new question that certainly wouldn't have any attempts belonging to it.
But it is better to implement versioning. A simple one for beginning, not for sharing, just for applying an older versions to the old attempts while having a new ones for new attempts. It's not hard actually and it is far, far better way to handle changes.
I expect that number of users that will be angered by restrictions on question editing will be much more than number of users that are in pain with edited questions now.
But it is better to implement versioning. A simple one for beginning, not for sharing, just for applying an older versions to the old attempts while having a new ones for new attempts. It's not hard actually and it is far, far better way to handle changes.
I expect that number of users that will be angered by restrictions on question editing will be much more than number of users that are in pain with edited questions now.
Sorry I was not clear enough.
How often the preview step reveal that you forgot to do something when editing?
Pierre
- Notice state if the user did some modification to a question and you need to confirm before saving the same way your text editor ask you before saving your text with the same file name.
I think that this should be the way edit_question_form should work in any case when editing a question.
In multianswer with a slight modifcation in the text you can destroy one or many subquestions without any error in the remaining question.
And, Alas, the subquestion is gone by just clicking on save... - Warning state When editing a question that is in a quiz that you cannot modify because there are valid attempts, the parameters you can modify are limited.
But you can save the modified question as a new question - Error state which is the actual code. The user cannot save or save as new without correcting the errors.
How often the preview step reveal that you forgot to do something when editing?
Pierre
Give user a freedom instead of control on every step - and they will learn to become responsible. Adn if you do it along with abilities to undo their work - all will be happy.
1. I have nothing against notice while it isn't too importunate. At least it could work when the number of subquestions become lower. Thought having versions and undo command will be definitely better.
"And, Alas, the subquestion is gone by just clicking on save..." - the most often "Alas" I had editing questions is when after thorought editing I automatically pressed "save" instead of "save as new" (when trying to save a work creating a new question on the base of existing one), but there are not many things you can do there.
2. "Warning state When editing a question that is in a quiz that you cannot modify because there are valid attempts, the parameters you can modify are limited.
But you can save the modified question as a new question" - a big problem there.
Imagine a typical workflow of editing an existing question in a large bank. There are some categories from which random question chooses (since it large bank). You are spotted an error in a question and want to correct it. What good will do to you been able only as "save as new"? You'll end up with two questions in that category, so the old, erroneous one will appear in new attempts too and this question will appear in attempts more often.
At the very least you should use an easy versioning that is already implemented. When a question that used in attempt is deleted, it is marked as old and don't show on the editing / new attempts. I.e. when saving a question in a warning state you should transparently for the user save a question as new and mark old version of question as old. The downside is in fact there is no automatic delete of old question when all attempts that use them are deleted (which you can improve too).
1. I have nothing against notice while it isn't too importunate. At least it could work when the number of subquestions become lower. Thought having versions and undo command will be definitely better.
"And, Alas, the subquestion is gone by just clicking on save..." - the most often "Alas" I had editing questions is when after thorought editing I automatically pressed "save" instead of "save as new" (when trying to save a work creating a new question on the base of existing one), but there are not many things you can do there.
2. "Warning state When editing a question that is in a quiz that you cannot modify because there are valid attempts, the parameters you can modify are limited.
But you can save the modified question as a new question" - a big problem there.
Imagine a typical workflow of editing an existing question in a large bank. There are some categories from which random question chooses (since it large bank). You are spotted an error in a question and want to correct it. What good will do to you been able only as "save as new"? You'll end up with two questions in that category, so the old, erroneous one will appear in new attempts too and this question will appear in attempts more often.
At the very least you should use an easy versioning that is already implemented. When a question that used in attempt is deleted, it is marked as old and don't show on the editing / new attempts. I.e. when saving a question in a warning state you should transparently for the user save a question as new and mark old version of question as old. The downside is in fact there is no automatic delete of old question when all attempts that use them are deleted (which you can improve too).
In reply to Pierre Pichet
Re: I just tried to write a roadmap for the Moodle quiz
by Pierre Pichet -
Globally the attempt storing should enable to replay the quiz as the student answered it.
This is why the student response is stored and in question like multichoice with the set shuffle set-on, the order of the multiple choice response is also stored.
In questiontype like short answer, numerical or calculated the student response is stored as written by the student.
In multichoice or other similar questiontype the student response is stored as a number which is the id of the multiple question answer.
If after the quiz attempt is stored the multiple choice answers are modified, deleted or new ones added, you cannot replay the quiz and the stored student response could refere to a deleted answer.
This should not happen.
On multiple answers (cloze) the problem can be worsed because you can delete a subquestion (short answer, numerical or multichoise) or even change its type so you cannot rebuild the quiz from the actual attempt stored.
This is why I think that the editing possiblity of a question already in a quiz should be limited until question versioning is established.
My "limited" understanding of question versioning is that when you edit a question you create a new version without destroying the old one or its main components like answers.
A version number allow the quiz to locate which version was used in a specific attempt.
Pierre
P.S. from type/multichoice/questiontype.php
function restore_session_and_responses(&$question, &$state) {
// The serialized format for multiple choice questions
// is an optional comma separated list of answer ids (the order of the
// answers) followed by a colon, followed by another comma separated
// list of answer ids, which are the radio/checkboxes that were
// ticked.
// E.g. 1,3,2,4:2,4 means that the answers were shown in the order
// 1, 3, 2 and then 4 and the answers 2 and 4 were checked.
This is why the student response is stored and in question like multichoice with the set shuffle set-on, the order of the multiple choice response is also stored.
In questiontype like short answer, numerical or calculated the student response is stored as written by the student.
In multichoice or other similar questiontype the student response is stored as a number which is the id of the multiple question answer.
If after the quiz attempt is stored the multiple choice answers are modified, deleted or new ones added, you cannot replay the quiz and the stored student response could refere to a deleted answer.
This should not happen.
On multiple answers (cloze) the problem can be worsed because you can delete a subquestion (short answer, numerical or multichoise) or even change its type so you cannot rebuild the quiz from the actual attempt stored.
This is why I think that the editing possiblity of a question already in a quiz should be limited until question versioning is established.
My "limited" understanding of question versioning is that when you edit a question you create a new version without destroying the old one or its main components like answers.
A version number allow the quiz to locate which version was used in a specific attempt.
Pierre
P.S. from type/multichoice/questiontype.php
function restore_session_and_responses(&$question, &$state) {
// The serialized format for multiple choice questions
// is an optional comma separated list of answer ids (the order of the
// answers) followed by a colon, followed by another comma separated
// list of answer ids, which are the radio/checkboxes that were
// ticked.
// E.g. 1,3,2,4:2,4 means that the answers were shown in the order
// 1, 3, 2 and then 4 and the answers 2 and 4 were checked.
Pierre wrote:
"If after the quiz attempt is stored the multiple choice answers are modified, deleted or new ones added, you cannot replay the quiz and the stored student response could refere to a deleted answer.
This should not happen." Fiat justitia ruat caelum. Or, as Cicero said, "summum jus summa injuria".
I am tired of saying you that you may create a greater evil exorcising this situation. You are ignoring all use cases I described and deafly persists on having this done regardless of consequences. I do not know what to say more to be heard. Do perform cost/benefit analyze first. Create a thread and ask a users, what they want more - ability to edit questions freely or extreme integrity of old attempts.
"If after the quiz attempt is stored the multiple choice answers are modified, deleted or new ones added, you cannot replay the quiz and the stored student response could refere to a deleted answer.
This should not happen." Fiat justitia ruat caelum. Or, as Cicero said, "summum jus summa injuria".
I am tired of saying you that you may create a greater evil exorcising this situation. You are ignoring all use cases I described and deafly persists on having this done regardless of consequences. I do not know what to say more to be heard. Do perform cost/benefit analyze first. Create a thread and ask a users, what they want more - ability to edit questions freely or extreme integrity of old attempts.
I think it is remarkable how often you can edit quiz questions, and it does not break the quiz - even though, theoretically, the system gives you no guarantees of correctness.
And this aspect of the Moodle quiz has not changed in many versions (since before I started working on the quiz) so my preference is to let sleeping dogs lie until we work out the wonderful new versioning scheme that will solve all these problems forever.
However, if multi-answer questions are particularly likely to break (and I cannot say that I have heard many people complaining about it here) the a purely informative warning at the top of the editing form for just that one question type would be an OK user-interface experiment. We could see if that helps more people than it confuses.
By the way, before we can even start to design the new versioning scheme, we need to know what the requirements are, so I stared a new blank wiki page where we can start to list all the things we want the question bank to do in future. Feel free to add stuff to Development:Future question bank/sharing/versioning requirements.
And this aspect of the Moodle quiz has not changed in many versions (since before I started working on the quiz) so my preference is to let sleeping dogs lie until we work out the wonderful new versioning scheme that will solve all these problems forever.
However, if multi-answer questions are particularly likely to break (and I cannot say that I have heard many people complaining about it here) the a purely informative warning at the top of the editing form for just that one question type would be an OK user-interface experiment. We could see if that helps more people than it confuses.
By the way, before we can even start to design the new versioning scheme, we need to know what the requirements are, so I stared a new blank wiki page where we can start to list all the things we want the question bank to do in future. Feel free to add stuff to Development:Future question bank/sharing/versioning requirements.
From the comments on MDL-20584
The actual design of cloze code use the already existing subquestions to store the edited question.
The main cloze question contains say
question->id =1000 ;
question-> name ="example for Oleg";
question->text = "<p>This question consists of a first multichoice question {#1} and right after a second one short answer {#2} and finally we have a floating point number {#3}.</p>"
They are stored as say
question->id =1001 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:MULTICHOICE:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = multichoice
question other parameters = defined from this question text
question->id =1002 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:SHORTANSWER:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = shortanswer
question other parameters = defined from this question text
question->id =1003 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{2:NUMERICAL:=23.8:0.1#Feedback for correct answer 23.8~%50%23.8:2#Feedback for half credit answer in the nearby region of the correct answer}"
question->qtype = numerical
question other parameters = defined from this question text
So if you change change the order say 3, 1,2
The main cloze question contains say
question->id =1000 ;
question-> name ="example for Oleg";
question->text = "<p>This question consists first of a floating point number question {#1} and right after a second one multichoice question short answer {#2} and finally we have a short answer {#3}.</p>"
question->id =1001 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{2:NUMERICAL:=23.8:0.1#Feedback for correct answer 23.8~%50%23.8:2#Feedback for half credit answer in the nearby region of the correct answer}"
question->qtype = numerical
question other parameters = defined from this question text
question->id =1002 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:MULTICHOICE:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = multichoice
question other parameters = defined from this question text
question->id =1003 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:SHORTANSWER:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = shortanswer
question other parameters = defined from this question text
so the question->id =1001 ; question->id =1002 ; question->id =1003 ; referred to a different question than in the first version.
The attempt data is corrupted .
The student response is stored differently for multichoice than for short or numerical.
I think that this should not be allowed.
My last comments was
Adding notice or warning remain a good thing to do.
About restricting saving to an existing quiz with attempt conditions, we could create a new moodle:question capabitlity that control this.
This should remove most of concerns about liberty...
I think.
Pierre
P.S. I remember complaints about this problem but cannot locate them easily.
P.S. It is quite easy to at least detect the number of subquestions or their qtype in cloze and generate a visible warning and a check confim saving button.
In any cases small typos could eliminate subquestions, so a confirm is not a bad idea. In other question types you cannot delete an existing question on the edit_question_form.
The actual design of cloze code use the already existing subquestions to store the edited question.
The main cloze question contains say
question->id =1000 ;
question-> name ="example for Oleg";
question->text = "<p>This question consists of a first multichoice question {#1} and right after a second one short answer {#2} and finally we have a floating point number {#3}.</p>"
They are stored as say
question->id =1001 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:MULTICHOICE:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = multichoice
question other parameters = defined from this question text
question->id =1002 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:SHORTANSWER:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = shortanswer
question other parameters = defined from this question text
question->id =1003 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{2:NUMERICAL:=23.8:0.1#Feedback for correct answer 23.8~%50%23.8:2#Feedback for half credit answer in the nearby region of the correct answer}"
question->qtype = numerical
question other parameters = defined from this question text
So if you change change the order say 3, 1,2
The main cloze question contains say
question->id =1000 ;
question-> name ="example for Oleg";
question->text = "<p>This question consists first of a floating point number question {#1} and right after a second one multichoice question short answer {#2} and finally we have a short answer {#3}.</p>"
question->id =1001 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{2:NUMERICAL:=23.8:0.1#Feedback for correct answer 23.8~%50%23.8:2#Feedback for half credit answer in the nearby region of the correct answer}"
question->qtype = numerical
question other parameters = defined from this question text
question->id =1002 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:MULTICHOICE:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = multichoice
question other parameters = defined from this question text
question->id =1003 ;
question->parent =1000 ;
question-> name='example for Oleg';
question->text = "{1:SHORTANSWER:Wrong answer#Feedback for this wrong answer~=Correct answer#Feedback for correct answer}"
question->qtype = shortanswer
question other parameters = defined from this question text
so the question->id =1001 ; question->id =1002 ; question->id =1003 ; referred to a different question than in the first version.
The attempt data is corrupted .
The student response is stored differently for multichoice than for short or numerical.
I think that this should not be allowed.
My last comments was
Adding notice or warning remain a good thing to do.
About restricting saving to an existing quiz with attempt conditions, we could create a new moodle:question capabitlity that control this.
This should remove most of concerns about liberty...
I think.
Pierre
P.S. I remember complaints about this problem but cannot locate them easily.
P.S. It is quite easy to at least detect the number of subquestions or their qtype in cloze and generate a visible warning and a check confim saving button.
In any cases small typos could eliminate subquestions, so a confirm is not a bad idea. In other question types you cannot delete an existing question on the edit_question_form.
An example to show the detection that the number of questions (or subquestions) is different than the number of questions stored in the database.
Less questions means that some were deleted either voluntary or accidentally.
The other problem is that the user could have change the question type which is the more important parameter that could scramble the attempt data.
We need just to agree on how we apply these tests.
My proposal is to add a confirm checkbox to appear when the number of question or their type has changed when editing a question that is used in a quiz, is edited.
This should suffice as a first step.
Pierre
Less questions means that some were deleted either voluntary or accidentally.
The other problem is that the user could have change the question type which is the more important parameter that could scramble the attempt data.
We need just to agree on how we apply these tests.
My proposal is to add a confirm checkbox to appear when the number of question or their type has changed when editing a question that is used in a quiz, is edited.
This should suffice as a first step.
Pierre
I like this. Changing the number of subquestions, or the type of a subquestion are the two things that could really screw up existing attempts. Therefore, it is good to warn about these, but still give teachers a chance to override the warning if they know what they are doing.
I suggest trying to make the warnings look more like warnings, somehow.
I suggest trying to make the warnings look more like warnings, somehow.
"I suggest trying to make the warnings look more like warnings, somehow."
This was a first draw.
Just looking deep in the moodle form code, I realize that actually there is only one error state namely "error" defined.
I will experiment by adding new states like "notify" and "warning" by locally modifying edit_multianswer_form class.
After testing this could be added to moodle form.
To display states like "notify" and "warning" correctly the themes should be modified also.
I have a plan to modify the saving (and restore session) of subquestions so that they are not deleted but somehow hidden.
But first lets improve the user interface.
Pierre
This was a first draw
.Just looking deep in the moodle form code, I realize that actually there is only one error state namely "error" defined.
I will experiment by adding new states like "notify" and "warning" by locally modifying edit_multianswer_form class.
After testing this could be added to moodle form.
To display states like "notify" and "warning" correctly the themes should be modified also.
I have a plan to modify the saving (and restore session) of subquestions so that they are not deleted but somehow hidden.
But first lets improve the user interface.
Pierre
One advice on user interface: you may react on not checked confirmation check boxes as an error during form validation, so the user will see on second try familiar red rectangle with a message that he should confirm his actions. However you should be sure that confirmation is still needed (i.e. user don't edit a question after seeing you warnings in a way that invalidates them.
Thanks,
The checkbox only appears if the user has either delete a response or change the qtype (which is the only simple test that a question has been move or change).
There is no question number in the cloze syntax.
Pierre
The checkbox only appears if the user has either delete a response or change the qtype (which is the only simple test that a question has been move or change).
There is no question number in the cloze syntax.
Pierre
I just CVS a first "beta" version, see MDL-20585 for screenshots.
Put in the tracker issue testing results, comments and text corrections unless you think they should be discussed here in the forum.
Thanks
Pierre
P.S. I use the class="error" color as this is the only one clearly defined in themes for questions.
Put in the tracker issue testing results, comments and text corrections
unless you think they should be discussed here in the forum.Thanks
Pierre
P.S. I use the class="error" color as this is the only one clearly defined in themes for questions.
Pierre, is there a public testing site where we could test you beta version?
It is already on Moodle HEAD.
But if you don't have a Moodle HEAD installation, you can try it on
http://132.208.141.198/moodle_head_new/
log as: moodle
pw:moodle
Waiting for your comments
.
Pierre
P.S. This is my office windows computer but as I work for an university that was there at the internet beginning, we can quite easily have permanent IP addresses for special projects....
But if you don't have a Moodle HEAD installation, you can try it on
http://132.208.141.198/moodle_head_new/
log as: moodle
pw:moodle
Waiting for your comments
.Pierre
P.S. This is my office windows computer but as I work for an university that was there at the internet beginning, we can quite easily have permanent IP addresses
for special projects....
I'll try to see in the near future. In the meantime I added you to MDL-20629, that is aimed to resolve a problem of breaks upon question editing too. Have a look on it.
I placed my thoughts after testing on relevant issue.
I also noticed a good site name of http://132.208.141.198/moodle_head_new/ ;) Moodle 1.6 forever
I also noticed a good site name of http://132.208.141.198/moodle_head_new/ ;) Moodle 1.6 forever
In reply to Oleg Sychev
Re: I just tried to write a roadmap for the Moodle quiz
by Olli Savolainen -
Just to try to understand the situation:
If we can not do complete versioning right now, would not an UI for making copies of quizzes solve the issue for now? That is, on behalf of a user 1. make a backup of a quiz and 2. restore it without the attempts, making it editable again, while keeping the integrity of the attempts of the old copy?
Or am I talking about a separate issue somehow?
If we can not do complete versioning right now, would not an UI for making copies of quizzes solve the issue for now? That is, on behalf of a user 1. make a backup of a quiz and 2. restore it without the attempts, making it editable again, while keeping the integrity of the attempts of the old copy?
Or am I talking about a separate issue somehow?
Making a UI like that actually work would be a nightmare that I do not even want to contemplate.
Also, it would do things like create a new column in the gradebook, would the teacher really want that?
Also, it would do things like create a new column in the gradebook, would the teacher really want that?
For me, it is the opposite - embedding versioning into the UI sounds like such a large scale rework in terms of the UI design (lots of added modality) that the risk of making it a LOT more confusing, especially if you try to just 'drop it in'.
On the other hand, making automatic copies of modules is something I have heard users ask for so apparently they can easily understand the conceptual mechanics of that. And I have already designed a prototype last year, which seems to fit into at least the quiz interaction model relatively easily. http://docs.moodle.org/en/Development:Quiz_Usability_portal/Attempted_quiz_enabling (It would probably be beneficial to make it possible to do some of this from the course front page too, though)
So the UI really is not the problem, but the underlying software mechanics? Sorry to ask you to contemplate on it, but what is it that you find nightmarish?
Related to gradebook, that depends largely on the use it seems. If the teacher runs the same quiz with the same group again, then I guess not. If the same course is run with different groups, then the teacher would likely want the old quiz to disappear from the gradebook. But my guess is probably not as good as yours here.
On the other hand, making automatic copies of modules is something I have heard users ask for so apparently they can easily understand the conceptual mechanics of that. And I have already designed a prototype last year, which seems to fit into at least the quiz interaction model relatively easily. http://docs.moodle.org/en/Development:Quiz_Usability_portal/Attempted_quiz_enabling (It would probably be beneficial to make it possible to do some of this from the course front page too, though)
So the UI really is not the problem, but the underlying software mechanics? Sorry to ask you to contemplate on it, but what is it that you find nightmarish?
Related to gradebook, that depends largely on the use it seems. If the teacher runs the same quiz with the same group again, then I guess not. If the same course is run with different groups, then the teacher would likely want the old quiz to disappear from the gradebook. But my guess is probably not as good as yours here.
When I talk about versioning, please do not imaging something like CVS of git.
Imagine MediaWiki instead. In MediaWiki, there are pages, and you can edit them. Now, it just happens that whenever you edit a page, MediaWiki keeps a copy of the old version of the page in its history. However, it does this without bothering the user. However, if the user really wants (perhaps they need to know what someone else saw last week, or when the page was last updated, or undo the last change) all this is possible.
In terms of editing quizzes. The teacher has a quiz in their course. It is quite natural that they might want to edit it - even after some students have started it. And, if they do, then clearly students who did the quiz before the edit saw something different from students who did it after the edit. To the teacher's mind, that is just how it should work. As software developers we realise that since this involves a lot of interrelated data in a database, actually making that happen is really hard. I cannot understand why you might think that the teacher would wish to make a copy of their quiz. The teacher wants to edit. (Yes, there are separate use-cases where the teacher wants to copy a quiz, but they are separate.)
The thing that is nightmarish to contemplate is the software mechanics of backup and restore.
Imagine MediaWiki instead. In MediaWiki, there are pages, and you can edit them. Now, it just happens that whenever you edit a page, MediaWiki keeps a copy of the old version of the page in its history. However, it does this without bothering the user. However, if the user really wants (perhaps they need to know what someone else saw last week, or when the page was last updated, or undo the last change) all this is possible.
In terms of editing quizzes. The teacher has a quiz in their course. It is quite natural that they might want to edit it - even after some students have started it. And, if they do, then clearly students who did the quiz before the edit saw something different from students who did it after the edit. To the teacher's mind, that is just how it should work. As software developers we realise that since this involves a lot of interrelated data in a database, actually making that happen is really hard. I cannot understand why you might think that the teacher would wish to make a copy of their quiz. The teacher wants to edit. (Yes, there are separate use-cases where the teacher wants to copy a quiz, but they are separate.)
The thing that is nightmarish to contemplate is the software mechanics of backup and restore.
This is way late but thought I would respond anyway.
Yes, you are right: being capable of just editing the quiz is important, and all is fine as long as the versioning guts of the system do not show.
However, the instant you add an UI for the user to actually handle the versions, you are adding modality, with things like making an old version the current one and making sure users know if they are viewing an old version, and not the current one, and seeing the differences between different versions. These are challenging do design into the UI, and more so to the already crowded Quiz UI - and certainly not something I would ever do without some considerable usability testing, to say the least.
Mediawiki and even Wordpress do this pretty horribly, in my subjective opinion. Google Docs' implementation is more limited and in that also adds restrictions that seem to make it clearer (again, in my very subjective view). Though you can cope with it after practice, it seems to me pretty easy to get lost. Of course the upside is that even if you do get horribly lost, the whole system is built so that you cannot do anything that someone who does understand the system could not undo.
Yes, you are right: being capable of just editing the quiz is important, and all is fine as long as the versioning guts of the system do not show.
However, the instant you add an UI for the user to actually handle the versions, you are adding modality, with things like making an old version the current one and making sure users know if they are viewing an old version, and not the current one, and seeing the differences between different versions. These are challenging do design into the UI, and more so to the already crowded Quiz UI - and certainly not something I would ever do without some considerable usability testing, to say the least.
Mediawiki and even Wordpress do this pretty horribly, in my subjective opinion. Google Docs' implementation is more limited and in that also adds restrictions that seem to make it clearer (again, in my very subjective view). Though you can cope with it after practice, it seems to me pretty easy to get lost. Of course the upside is that even if you do get horribly lost, the whole system is built so that you cannot do anything that someone who does understand the system could not undo.
I'm not sure I fully understand what is at stake with this current discussion about question editing and Pierre's vs Oleg's points of view, but I thought I'd jump in with my own case.
I only use Moodle's quiz activity for learning/training purposes, never for testing. In that context, of course I always have Adaptive mode ON. I absolutely need to retain the possibility to edit questions which are used by quizzes currently open to the students, since I often need to correct minor errors, or, more importantly, add to the list of accepted correct responses, especially when using the short answer question type or my own regexp question type, either on their own or as sub-questions within a cloze question.
I do appreciate that one must be very careful when editing such questions that belong to a quiz currently open to the students, but that is the teacher's responsibility. Any new feature which would restrain my liberty to edit the questions on the grounds of "safety" would meet with my disapproval.
Joseph
Yes Joseph, that is one important use-case that is currently possible, and which we must not break.
Actually, there are edge cases where it is not clear what the right behaviour is. Here are some examples
Simple case A:
1. Teacher makes version 1 of question.
2. Teacher notices mistake
3. Teacher makes version 2 of question.
4. Student starts attempt - they see version 2 of the question.
Simple case B:
1. Teacher makes version 1 of question.
2. Student starts attempt - they see version 1 of the question.
3. Student does submit all and finish.
4. Teacher notices mistake
5. Teacher makes version 2 of question.
6. Teacher or student reviews the students attempt - they should be able to see that the student got version 1 of the Q.
7. Teacher goes to the regrade UI. There should be an option to regrade the attempt switching to version 2.
8. Teacher or student reviews the students attempt - they should be able to see version 2 of the Q, with a note that this was changed during regrading.
Tricky case C:
1. Teacher makes version 1 of question.
2. Student starts attempt - they see version 1 of the question.
3. Student enters answer but does not submit, only saves.
4. Teacher notices mistake.
5. Teacher makes version 2 of question.
6. Student goes back to the quiz.
What should the student see now?
Actually, there are edge cases where it is not clear what the right behaviour is. Here are some examples
Simple case A:
1. Teacher makes version 1 of question.
2. Teacher notices mistake
3. Teacher makes version 2 of question.
4. Student starts attempt - they see version 2 of the question.
Simple case B:
1. Teacher makes version 1 of question.
2. Student starts attempt - they see version 1 of the question.
3. Student does submit all and finish.
4. Teacher notices mistake
5. Teacher makes version 2 of question.
6. Teacher or student reviews the students attempt - they should be able to see that the student got version 1 of the Q.
7. Teacher goes to the regrade UI. There should be an option to regrade the attempt switching to version 2.
8. Teacher or student reviews the students attempt - they should be able to see version 2 of the Q, with a note that this was changed during regrading.
Tricky case C:
1. Teacher makes version 1 of question.
2. Student starts attempt - they see version 1 of the question.
3. Student enters answer but does not submit, only saves.
4. Teacher notices mistake.
5. Teacher makes version 2 of question.
6. Student goes back to the quiz.
What should the student see now?
Thanks Tim, interesting cases.
B7 may require a complex function from a question type trying to do switching correctly, or, at least, the check Pierre mentioned - one place where it is really useful, question can detect that correct switching is impossible and refuse to switch (instead of corrupting data).
In case C. With versioning student should definitely (IMHO) see a version that he sees while starting the quiz. Later a teacher can switch on regrading (it is useful to have an offer of regrading while editing question that is used in attempts, which will go to a page where all instances of this questions will be displayed, so the teacher could easily see potential problems). Any changes, if they are happened during attempts should be either postponed to new attempts or seriously highlighted, otherwise student can just don't notice them.
I think we should eventually implement a solution preventing editing while quiz (question) is in use and vice versa (but then again, for quizzes with no timeout we should have some ability to force quiz to end all attempts with logged out users, or they could block editing at all). This can produce all sort of strange effects. For instance (if quiz versioning will be implemented too, it's actually easier then question versioning and could be used as pilot project):
Case D
1. Teacher notice mistake in quiz
2. Teacher deletes wrongly placed question (maybe a random from wrong category)
3. Student starts attempt
4. Teacher adds a new, correct question.
We could start doing this by just placing a notice on a page with quiz/question editing that it is in use right now.
I believe in gradual development. We may start at simple form of versioning needed just to insure correctness of a quiz. That already places away many problems. Then add an ability to undo question editing, than resolve question sharing problems (I wonder if versioning is a right tool there. Maybe problems with question sharing should be resolved by carefully designed/assigned capabilities? Or even by making a question category a kind of context...)
B7 may require a complex function from a question type trying to do switching correctly, or, at least, the check Pierre mentioned - one place where it is really useful, question can detect that correct switching is impossible and refuse to switch (instead of corrupting data).
In case C. With versioning student should definitely (IMHO) see a version that he sees while starting the quiz. Later a teacher can switch on regrading (it is useful to have an offer of regrading while editing question that is used in attempts, which will go to a page where all instances of this questions will be displayed, so the teacher could easily see potential problems). Any changes, if they are happened during attempts should be either postponed to new attempts or seriously highlighted, otherwise student can just don't notice them.
I think we should eventually implement a solution preventing editing while quiz (question) is in use and vice versa (but then again, for quizzes with no timeout we should have some ability to force quiz to end all attempts with logged out users, or they could block editing at all). This can produce all sort of strange effects. For instance (if quiz versioning will be implemented too, it's actually easier then question versioning and could be used as pilot project):
Case D
1. Teacher notice mistake in quiz
2. Teacher deletes wrongly placed question (maybe a random from wrong category)
3. Student starts attempt
4. Teacher adds a new, correct question.
We could start doing this by just placing a notice on a page with quiz/question editing that it is in use right now.
I believe in gradual development. We may start at simple form of versioning needed just to insure correctness of a quiz. That already places away many problems. Then add an ability to undo question editing, than resolve question sharing problems (I wonder if versioning is a right tool there. Maybe problems with question sharing should be resolved by carefully designed/assigned capabilities? Or even by making a question category a kind of context...)
Oleg, until yesterday I would have agreed with you about C. It is clearly the simplest option to implement, and is simple and logical to explain.
However, Joseph's scenario was actually this:
Teacher has set the quiz as a classroom activity, and is standing at the front of the classroom as students attempt it. Then the problem is found and the teacher want the fixed question to immediately be part of all students' attempts because the teacher can warn the students about what is going on.
Now that is a much harder use-case. Particularly if you are in adaptive mode, and the students have already tried several answers to that question. It may even be a use-case that we choose not to implement because it is not a common requirement, and it is too difficult. But we have to acknowledge that it is a valid requirement.
Your case D is another yucky case to deal with. It is much, much easier if a quiz is a fixed list of questions. The main reason for that is the quiz reports. It is so much easier for everyone if all quiz attempts contain the same questions, so the quiz reports are simply tables - that is, the row for each student contains the same columns. Hence, it is really good if we can keep the rule that you cannot change the list of questions after an attempt has been started.
The simpler solution to your requirement is for the teacher to make sure that students are not allowed in to start attempting the quiz until they have finished editing. Therefore, you should add a step 1.5: Teacher temporarily makes the quiz invisible to students.
However, Joseph's scenario was actually this:
Teacher has set the quiz as a classroom activity, and is standing at the front of the classroom as students attempt it. Then the problem is found and the teacher want the fixed question to immediately be part of all students' attempts because the teacher can warn the students about what is going on.
Now that is a much harder use-case. Particularly if you are in adaptive mode, and the students have already tried several answers to that question. It may even be a use-case that we choose not to implement because it is not a common requirement, and it is too difficult. But we have to acknowledge that it is a valid requirement.
Your case D is another yucky case to deal with. It is much, much easier if a quiz is a fixed list of questions. The main reason for that is the quiz reports. It is so much easier for everyone if all quiz attempts contain the same questions, so the quiz reports are simply tables - that is, the row for each student contains the same columns. Hence, it is really good if we can keep the rule that you cannot change the list of questions after an attempt has been started.
The simpler solution to your requirement is for the teacher to make sure that students are not allowed in to start attempting the quiz until they have finished editing. Therefore, you should add a step 1.5: Teacher temporarily makes the quiz invisible to students.
On case C and Joseph's example. I guess we can (not in the first place, but eventually) create a clearly visible icon that notifies student about qustion change, and the actual change for the attempt may be done only if student will press it. A question type could also have a hand in the matter and tell it couldn't really upgrade a question with such editing.
BTW, I guess we could start all this with a very simple change for question engine. A question type should be given a possibility to (note I don't suppose that all core qtypes should start using them with full strength right now):
1) receiving $question objects for two old and new question tell, could attempts be upgraded from one version to another. Default implementation will return true retaining current behavor.
2) receving these two $questions and an array of student's responses for that question do upgrade of these responses to the new version of question.
"Hence, it is really good if we can keep the rule that you cannot change the list of questions after an attempt has been started." Well, actually what I really want to have a possibility to edit is not a list of questions of a quiz, but a random questions that are added to the quiz (i.e. to change category from which questions will choosen). I guess nothing really prevent this except UI. If we improve that somehow, and any question (including random) in a quiz could be edited I could live with inability to edit quiz (i.e. change number of questions, anything else depends on a questions). What do you say on that? (I would like also Olli's opinion as an author of new quiz editing interface).
Adding step 1.5 will resolve case D, but not other cases with quiz editing. I always can make somthing like case D1:
1. Student start attempt
2. Teacher delete a question from the quiz
3. Student save attempt
4. Teacher add a new question to the quiz
5. Student finish attempt
Actually we will have a classical synchronisation issue of readers and writers with question versioning anyway.
BTW, I guess we could start all this with a very simple change for question engine. A question type should be given a possibility to (note I don't suppose that all core qtypes should start using them with full strength right now):
1) receiving $question objects for two old and new question tell, could attempts be upgraded from one version to another. Default implementation will return true retaining current behavor.
2) receving these two $questions and an array of student's responses for that question do upgrade of these responses to the new version of question.
"Hence, it is really good if we can keep the rule that you cannot change the list of questions after an attempt has been started." Well, actually what I really want to have a possibility to edit is not a list of questions of a quiz, but a random questions that are added to the quiz (i.e. to change category from which questions will choosen). I guess nothing really prevent this except UI. If we improve that somehow, and any question (including random) in a quiz could be edited I could live with inability to edit quiz (i.e. change number of questions, anything else depends on a questions). What do you say on that? (I would like also Olli's opinion as an author of new quiz editing interface).
Adding step 1.5 will resolve case D, but not other cases with quiz editing. I always can make somthing like case D1:
1. Student start attempt
2. Teacher delete a question from the quiz
3. Student save attempt
4. Teacher add a new question to the quiz
5. Student finish attempt
Actually we will have a classical synchronisation issue of readers and writers with question versioning anyway.
I forgot to respond to Oleg's comment about gradual development, which is an important point.
In general I am highly in favour of gradual incremental development, it is easier for all concerned. However, it is not always possible.
If you are half way up a hill and want to get higher, it is only natural to walk up hill.
If you are at the top of one hill, but think that that hill over there is much higher, and it is worth moving there, then you have to go down and tramp through all the nasty smelly bog at the bottom of the valley before you can go up again.
That is why in Moodle we have major releases, with periods of development between them. We don't want to bother users with the going through the smelly bog bit. We want to wait until we have got to the top of the next hill, and that waive and say look at us, after we have brushed the bug off our boots.
To add the the difficulty, the hills we walk through when doing software development are not normally clear and sunny. It is much more likely to be misty. You can see the land around where you are now - which is why if one direction is clearly up hill, the first thing you should try is moving that way - but when it comes to moving to another hill, you cannot really be sure it is there, or that it is really higher than you are now. Maybe someone walking by told you about some wonderful hill over there that you should go and check out, but can you really trust them?
In general I am highly in favour of gradual incremental development, it is easier for all concerned. However, it is not always possible.
If you are half way up a hill and want to get higher, it is only natural to walk up hill.
If you are at the top of one hill, but think that that hill over there is much higher, and it is worth moving there, then you have to go down and tramp through all the nasty smelly bog at the bottom of the valley before you can go up again.
That is why in Moodle we have major releases, with periods of development between them. We don't want to bother users with the going through the smelly bog bit. We want to wait until we have got to the top of the next hill, and that waive and say look at us, after we have brushed the bug off our boots.
To add the the difficulty, the hills we walk through when doing software development are not normally clear and sunny. It is much more likely to be misty. You can see the land around where you are now - which is why if one direction is clearly up hill, the first thing you should try is moving that way - but when it comes to moving to another hill, you cannot really be sure it is there, or that it is really higher than you are now. Maybe someone walking by told you about some wonderful hill over there that you should go and check out, but can you really trust them?
I've already told you about question bank developing proposal.
Adding a tags to question categories and using those tags as a filter instead (or in addition) to the current hierarchical system (for diplaying, editing or random question selection). Hierarhy isn't sufficient to keep up complex question banks: for example one hierarchy (main in our case) is the topics questions are belong, but then there is a demo, training and control categories in each topic. And having a task of exporting to another course 7 training categories out of about 100 categories I find it not easy...
You could choose to allow tags to specify parent tag (but not mandatory), then current tree can be mapped into one entity, tags (if you like Occams razor).
That's not hard, but add another dimension to question bank organisation.
Adding a tags to question categories and using those tags as a filter instead (or in addition) to the current hierarchical system (for diplaying, editing or random question selection). Hierarhy isn't sufficient to keep up complex question banks: for example one hierarchy (main in our case) is the topics questions are belong, but then there is a demo, training and control categories in each topic. And having a task of exporting to another course 7 training categories out of about 100 categories I find it not easy...
You could choose to allow tags to specify parent tag (but not mandatory), then current tree can be mapped into one entity, tags (if you like Occams razor).
That's not hard, but add another dimension to question bank organisation.
Tagging questions is already mostly implemented in Moodle 2.0. (I must finish implementing backup/restore and import/export before 2.0 is released). However, for now you cannot do very much with the tags, they are just listed in the question bank. Further features can come later.
I just want to state my proposal more clearly: I propose a tagging of categories too and using (a combination) of such tags in any place where you use category name for now.