Hello. This is question to Tim Hunt firstly, and to any from core developers.
The ordering question is missing in moodle core, but as far as i know it is popular question type, that is used in almost any quiz system. But moodle does not have builtin ordering qtype - strange.
I found some time ago, that much more specific question types like drag-to-image are included into core, but ordering is still missing.
I have seen two question types - order and ordering, both of them are currently not ready for adding to core. I can try to make changes nedded to conform moodle guidelines. But i do not want to do in vain, it there is no need of such work. But before i even begin thinking about this, i want to hear, maybe there is some story, why this qtype was not developed, when moodle quiz module was born.
There is the same question about ddmatch question. We have multiple draggble question types. Why not to add ddmatch to core? Or why do not we add draggable function to base matching question type? Should we do so or should not?
I agree that ordering is one of the most obvious missing question types from Moodle core.
The reason to be extremely hesitant to add any new plugin to core is that once you have done that, you have to maintain it forever afterwards (to a first approximation).
Meanwhile, with each Moodle release, it becomes easier and easier to install plugins for things you want that are not in Moodle core.
It took several years to get around to adding the drag-drop question types. (Mainly because I always seemed to by busy with other things.) It need not take that long. The main criterion is that lots of Moodle users want it.
I would say that the bare minimum in terms of code quality for adding a new question type to core is:
- The way it works, both for students and teachers, should be very similar to other core question types. (Which, in this case, means it should look and work like the other core drag-drop question types.)
- It should follow the Moodle coding style (no code-checker issues, JavaScript packaged as AMD or YUI modules.
- Good unit test coverage.
- Good Behat test coverage.
Although clearly popular as an add-on (https://moodle.org/plugins/stats.php?plugin_category=29) qtype_ordering currently 'fails' on all those points. The first point is clearly important. The other three are important because of the "have to maintain it forever afterwards" requirement.
Thanks, Tim, for outlining very clearly what it takes to get a question type into Moodle core. And I agree, too, that qtype_ordering is needed there. It is essential for language learning to have story plot ordering (vertical ordering) conversation ordering (vertical ordering) and word ordering in sentences (horizontal ordering)--and very engaging/fun for learners. Recently, I funded Gordon Bateson to add horizontal ordering to his qtype_ordering question type, so now we no longer need the legacy drag-and-drop ordering (qtype_order)--thanks to Boyd for many years of use of that qtype.
So I was disappointed to hear that qtype_ordering was rejected on three software tests, but at the same time, I was happy to see a clear path (but not necessarily an easy path) to core integration:
- Follow Moodle coding style (no code-checker issues, JavaScript packaged as AMD or YUI modules.
- Good unit test coverage.
- Good Behat test coverage.
As for the interface design following other drag-and-drop types, could you say, Tim, what were the problems you noticed? Or are these problems noted on the Tracker or somewhere?
Over the next year, I want to help Ordering, Drag-and-drop Matching, and Gapfill, become core or closer to core, because they are all in the most popular question types used by language teachers world-wide. The current core ddimage built by OU-UK is great! But the Select Missing Words has a horrible design interface that all of our teachers refuse to use--they love Gapfill for its ease of authoring. Also we have replaced more than 50% of our Multiple Choice items with DDMatching because is more fun for students and more efficient for testing (we can combine 5 m/c questions into one ddmatch question for example).
And just a reminder, the main reason we need these popular question types into core, is because Moodle Partners continue to charge very large sums of money for adding non-core plugins. In some cases, they charge US$800/year per plugin. Our school uses 25 custom plugins, so it is not an option for us to use Moodle Partners due to cost concerns. I wish Moodle Partners would re-invest some of those funds into improving the plugins so they meet core standards. As far as I know, that is not happening. Instead it has become a structural dis-incentive to use contributed Moodle plug-ins.
Hi Don, would my suggestion here
https://moodle.org/mod/forum/discuss.php?d=332154
Make the select missing words question type more acceptable (given better wording)?
As Tim says, once code is in core it needs to be supported approximatly forever, which with Moodle is going require a Mayan calender. I suspect that ordering has not been specifically rejected in the past, more that previously it had not been considered and even if it hit all those criteria it would not necessarily be in core, but that's for Tim to confirm.
It is worth noting that some people might feel the inverse of Tims point 1, particularly for teachers (grins). Some 3rd party developers may not want their code in core, for various reasons (e.g. loss of control or flexibility). I have always assumed that my Gapfill question type would never be in core for various reasons (e.g. duplication of functionality in other types) but if that ever happened I would truly enjoy the boasting rights.
My Gapfill question type has the Unit tests but not the behat tests, so will look into that for my next release. I am still working on polishing various interface/usability items.
Gapfill is a good illustration why it is good to be prudent in what goes in to core. If the embedded(cloze) question type were placed into plugin database instead of the core, there might be a better argument for including gapfill in the core now. Now cloze must be supported, and it duplicates some of the functionality of gapfill.
There is a good argument for making question types that are rarely used default to disabled on new Moodle installs. An example of this is that instead of having a hard to use calculated question type as well as a really, really hard to use calculated question type, by default Moodle could have just the hard to use calculated question type.
Do not take my remarks as an in-depth review of qtype_ordering. I just went and had a 5-minute look in the repository, to get a feel for what was there.
Also, the conditions I gave are not sufficient conditions to get a plugin into Moodle. They are just the most basic necessary conditions off the top of my head.
For me, the most instesting question about an ordering question type is how you award credit for partially correct responses.
How to grade an Ordering question is an interesting problem indeed.
Currently the Ordering question offers 8 grading methods. (full list below) .
Each has pros and cons, and was developed as the result of requirements for a particular situation.
The users of the Reader module are currently discussing which method to use as the default grading method on Ordering questions in quizzes used by the Reader module, in which events from a story have to be correctly ordered. In such questions, checking students understanding of the RELATIVE order of events is more important than the ABSOLUTE order. Several people have suggested that the "Longest contiguous subset" grading method maybe the most suitable, but the discussion continues.
- All or nothing
- If all items are in the correct position, then full marks are awarded. Otherwise, the score is zero.
- Absolute position
- An item is considered correct if it is in the same position as in the correct answer. The highest possible score for the question is the same as the number of items displayed to the student.
- Relative to the next item (excluding last)
- An item is considered correct if it is followed by the same item as it is in the correct answer. The item in the last position is not checked. Thus, the highest possible score for the question is one less than the number of items displayed to the student.
- Relative to the next item (including last)
- An item is considered correct if it is followed by the same item as it is in the correct answer. This includes the last item which must have no item following it. Thus, the highest possible score for the question is the same as the number of items displayed to the student.
- Relative to both the previous and next items
- An item is considered correct if both the previous and next items are the same as they are in the correct answer. The first item should have no previous item, and the last item should have no next item. Thus, there are two possible points for each item, and the highest possible score for the question is twice the number of items displayed to the student.
- Relative to ALL previous and next items
- An item is considered correct if it is preceded by all the same items as it is in the correct answer, and it is followed by all the same items as it is in the correct answer. The order of the previous items does not matter, and nor does the order of the following items. Thus, if n items are displayed to the student, the number of marks available for each item is (n - 1), and the highest mark availabe for the question is n x (n - 1), which is the same as (n² - n).
- Longest ordered subset
- The grade is the number of items in the longest ordered subset of items. The highest possible grade is the same as the number of items displayed. A subset must have at least two items. Subsets do not need to start at the first item (but they can) and they do not need to be contiguous (but they can be). Where there are multiple subsets of equal length, items in the subset that is found first, when searching from left to right, will be displayed as correct. Other items will be marked as incorrect.
- Longest contiguous subset
- The grade is the number of items in the longest contiguous subset of items. The highest possible grade is the same as the number of items displayed. A subset must have at least two items. Subsets do not need to start at the first item (but they can) and they MUST BE CONTIGUOUS. Where there are multiple subsets of equal length, items in the subset that is found first, when searching from left to right, will be displayed as correct. Other items will be marked as incorrect.
Most of those suggested grading methods are useful. However, you have missed out my favourite:
- For each pair of responses, are they in the right relative order (R)? Count the number of pairs that our out of the total number of pairs (T = 1/2*n*(n+1)).
The only down-side with this is that the average score for a random guess is 50%. Therefore, I actually favour this rather arbitrary modification of that.
- Count the number of pairs that are the wrong way around (W = T - R). Double it, and use that to compute the score. (T - 2*W)/T = (2*R - T)/T, but don't let the score go below 0.
If you work that out for some sample answers, I think you will find it is quite a close match for your intuitive judgement of how close to right a particular wrong answer is.
Thanks for these ideas, Tim.
I'm not quite sure how to count the total number of pairs, T. For example if there are 4 items, 1-2-3-4, I can only see the following 6 pairs:
- 1-2
- 1-3
- 1-4
- 2-3
- 2-4
- 3-4
But the formula for T, namely T = 1/2 * n * (n+1), says there should be 10 pairs (= 1/2 * 4 * 5).
Perhaps that formula should be T = 1/2 * n * (n - 1) ? In which case, this grading method would become essentially the same as "Relative to ALL previous and next items" described above, except the grades calculated for this one would be exactly half of those calculated for "Relative to ALL previous and next items".
I'm intrigued by the formula for determining the grade based on wrong pairs. Thanks for that
Sorry, yes: T = 1/2 * n * (n - 1)
I know, that qtype_ordering breacks some coding standart. But as for this, i clearly understand what to do with it - i've got about 10 contributed plugins, most of them currently give 0 errors/0 warnings. I can replace jquery in qtype_ordering to yui, taking qtype_order as example. I don't clearly understand how to write unittests for question type, because the main problem with this question is how it looks and behave in browsers, and not how grade is calculated. Maybe behat is for such situations. But i can start such work only if i know that at finish this will become part of core. If there's no chance, then current state of qtype_ordering is enough. I can do much work once, but i cannot support this plugin for all my life, i wish Moodle HQ to support it.
There are some problems, that i do not clearly understand how to do right and want some opinions:
ddmatch and order question have no-javascript fallback. If JS does not load, they became simple matching questions. Should core-candidate plugin have such fallback, or currently we support only situation, when JS is enabled? Should plugin show some error message, when JS is required, but disabled, or we do not need to think about it?
ordering qtype is currently very compex - it supports 8 types of grading, selecting part of list for question and so on. I suppose this is too hard for regular teacher to understand how it works. Maybe order (with only vertical/horisontal switch) type is enough? order qtype have "i don't know" answer. Is it needed?
Should ddmatch and match questions exists together if both in core? Or match should be extended for drag&drop? Or we need add ddmatch and add converter tool, like with assign and assignment in 2.2? Or we should add ddmatch, remove match, convert questions on update and move match to plugins repository like mod_journal and theme_standard?
ordering allows all-or-nothing grading. I've found deffered all-or-nothing question behaviour in plugn repository. I think this plugin is first candidate on including into moodle core. Is there any chance?
"I can replace jquery in qtype_ordering to yui, taking qtype_order as example. "
Nooooooo there has been a decision that JQuery is a big part of Moodles future.
There are lots of examples of Unit tests for question types (see my gapfill code), Yes, behat is the way to go for what you are talking about. Have I mentioned lately how awesome Gordon and his code is?
Marcus is right - there's no need to replace jQuery code with YUI code. However, there are a couple of blocks of JS that should be moved out to a separate file.
Also, there some if-then-else blocks that sniff out and handle different versions of Moodle. These would need to be split out into different branches in the GIT repository on github.com for smooth integration into Moodle core.
If Vadim is willing and able to take on the work to get qtype_ordering into shape for integration into Moodle core that would be wonderful !!
P.S. thanks for the appreciative comments Marcus
And what do you think about all other questions?
I believe that jQuery code needs to be loaded via AMD for page loading performance. The version sniffing would be removed since new plugins are only integrated into the development branch. All current versions would still need to use the code in the database.
A reasonably good ordering type could be added to the core simply by adding some of the scoring mechanism to the existing drag and drop types. I have done this before with my all or nothing behaviour plugin and the drag and drop to text plugin. This is really the reason I made that plugin. An option could be added to allow the above scoring schemes