For those of you with large sets of GIFT questions (or who have the time to export your questions to GIFT format) would you please check to see how frequently the following characters appear in the {answer specification} portion of your questions: @, !, $, *, ;, and _. Please post your totals at http://microformats.org/wiki/gift-syntax#Possible_choices or here.
If you are unsure how to do extract just the answer segments to perform this count, please just use 'grep -c' on the entire file(s) and let me know the totals.
Thank you very much for your help. This is for a learner adaptation system capable of selecting appropriate questions in free-form self study from a volunteer commons of assessment content as described at http://strategy.wikimedia.org/wiki/Proposal:Assessment_content
I can't work out what you are really trying to do. The links you give go to pages full of jargon that I cannot understand.
Reading between the lines, and guessing a bit, I think what you want to do is is make a big public question bank under a license like creative commons. And you want to include lots of metadata with the questions.
I can't work out why this leads you to want to modify the GIFT format.
It would be like Wikipedia saying 'if you want to upload in image', you have to use JPEG format, and you have to use our custom, non-standard version of GIFT which includes extra metatdata.
That is not how Wikipedia works. Instead they say you can add any sort of image, JPEG, PNG, GIF, and we will keep the metadata beside the image, like
.
Would not that model be better for what you are trying to do?
Reading between the lines, and guessing a bit, I think what you want to do is is make a big public question bank under a license like creative commons. And you want to include lots of metadata with the questions.
I can't work out why this leads you to want to modify the GIFT format.
It would be like Wikipedia saying 'if you want to upload in image', you have to use JPEG format, and you have to use our custom, non-standard version of GIFT which includes extra metatdata.
That is not how Wikipedia works. Instead they say you can add any sort of image, JPEG, PNG, GIF, and we will keep the metadata beside the image, like
.Would not that model be better for what you are trying to do?
Right, a big public question bank under a free open source license is exactly the goal. And we want to add specific kinds of metadata in particular, to: (1) enable learner adaptivity for low-stakes, free-form study, so that a learner can start with a medium-difficulty question and be served subsequent questions based on an estimation of their ability to answer them correctly. So the system could present questions that it thinks the learner has, for instance, a 50% chance of answering correctly based on their session track record; (2) enable feedback, which GIFT already has with the { ... ~wrong answer #feedback ... } construct; (3) enable general and detailed help on questions, which can be used to estimate learner independence, further tailoring low-stakes instructional feedback and question selection; and (4) enable public, anonymous, crowdsourced review of question and answer accuracy by requiring questions to be vetted by a third party or parties before going in to production, challenged by those who disagree with the scoring, and reviewed again if and when they are challenged.
The wikitext format is more complicated than GIFT, but has been tremendously successful at developing a large collection of free encyclopedia articles, dictionary definitions, and technical documents; much more than any xml-based format, because it is simpler to type than xml-derived formats. Both are valuable because they are easily edited by humans: "]]" is a lot easier to type than "</wikilink>" or the like, just as GIFT files have a lot fewer characters in them than Moodle's xml format. We looked at IMS QTI xml for a long time, but it's far too complicated to even attempt to subset instead of starting with a well-supported format like GIFT.
As for uploading non-textual files like images on Mediawiki, there are some new plans in the works to make that easier, too.
The wikitext format is more complicated than GIFT, but has been tremendously successful at developing a large collection of free encyclopedia articles, dictionary definitions, and technical documents; much more than any xml-based format, because it is simpler to type than xml-derived formats. Both are valuable because they are easily edited by humans: "]]" is a lot easier to type than "</wikilink>" or the like, just as GIFT files have a lot fewer characters in them than Moodle's xml format. We looked at IMS QTI xml for a long time, but it's far too complicated to even attempt to subset instead of starting with a well-supported format like GIFT.
As for uploading non-textual files like images on Mediawiki, there are some new plans in the works to make that easier, too.
Thanks for explaining.
So the choice of GIFT is that it is the best trade-off between rich features and ability to be hand-edited. That seems reasonable.
In terms of the metadata you want, let me tell you about what Moodle provides already:
There are, of course, the question categories. You may wish to think about using a defined category structure for questions in your bank. That lets you encode some of the metadata.
Moodle 2.0 is adding tagging for questions. I would have thought that is likely to be the best way to encode the difficulty information you want (1), by creating a defined tag vocabulary. It would probably also work well for review information (4).
Feedback (2) is part of the question definition, to my mind. Not metadata.
(3) might map naturally to what Moodle calls 'general feedback', also part of the question, not the metadata.
Note that, at the moment, GIFT does not contain the tags or general feedback, but I would not be averse to extending the format to include them, especially if it could be done in a way that is practically backwards compatible.
If you have not already done so, I strongly encourage you to install Moodle, and try importing various files. That is the best way to work out exactly what the GIFT format means.
So the choice of GIFT is that it is the best trade-off between rich features and ability to be hand-edited. That seems reasonable.
In terms of the metadata you want, let me tell you about what Moodle provides already:
There are, of course, the question categories. You may wish to think about using a defined category structure for questions in your bank. That lets you encode some of the metadata.
Moodle 2.0 is adding tagging for questions. I would have thought that is likely to be the best way to encode the difficulty information you want (1), by creating a defined tag vocabulary. It would probably also work well for review information (4).
Feedback (2) is part of the question definition, to my mind. Not metadata.
(3) might map naturally to what Moodle calls 'general feedback', also part of the question, not the metadata.
Note that, at the moment, GIFT does not contain the tags or general feedback, but I would not be averse to extending the format to include them, especially if it could be done in a way that is practically backwards compatible.
If you have not already done so, I strongly encourage you to install Moodle, and try importing various files. That is the best way to work out exactly what the GIFT format means.
Excellent! I hope you will please propose your preferred way to extend GIFT to handle these kinds of tags and fields:
As you know, I am ambivalent about which character prefix to use. And since reading your message above, I now wonder whether this information would be best placed inside the answer specification or after the question. Putting it inside the {answer specification} would likely make short answer questions confusingly long in the middle. Is there a way we can add this information to the end of the question in GIFT in a backwards-compatible way without having to collect statistics on character usages?
I have used Moodle but haven't installed it yet. I plan to do that in the next two or three days. I have looked at the xml spec, but only saw the generalfeedback tag just now. I will continue to study it and eagerly await your reply. Thanks for thinking about this extension.
helps-answer: title[, title...] helped-by: title[,...] easier-than: title[%n%][,...] harder-than: title[%n%][,...] general-help: text // already generalfeedback? specific-help: text review-state: text
As you know, I am ambivalent about which character prefix to use. And since reading your message above, I now wonder whether this information would be best placed inside the answer specification or after the question. Putting it inside the {answer specification} would likely make short answer questions confusingly long in the middle. Is there a way we can add this information to the end of the question in GIFT in a backwards-compatible way without having to collect statistics on character usages?
I have used Moodle but haven't installed it yet. I plan to do that in the next two or three days. I have looked at the xml spec, but only saw the generalfeedback tag just now. I will continue to study it and eagerly await your reply. Thanks for thinking about this extension.
I don't yet understand what exactly general-help and specific-help are. Could you explain, or give an example please.
For the others, if we put them before or after the question, I am wondering whether comments wouldn't be the best way:
// helps-answer: biology102, biology103
// helped-by: biology100
// review-state: proposed
::biology101:: Which animal eats which food? { =cat -> cat food =dog -> dog food }
I think that keeps a nice clear separation between the metadata and the question definition. You could, of course, add more punctuation to make the keywords easier for a computer to find, but I don't think that is necessary.
However, I also have some reservations about using comments for significant information. Anyway, that is the best suggestion I have right now.
If you are on Windows or Mac, then the all-in-one download packages http://download.moodle.org/macosx/ or http://download.moodle.org/windows/ will get you a working Moodle install with just a double-click.
On Linux, I would usually, use the package manager for all the prerequisites (php, apache, mysql) but then download a zip of the source, and then let Moodle self-install.
Anyway, I'm sure you've already found the install instructions Administrator_documentation#System-specific_Instructions_.26_Packages.
The first 2:30 of this video
will show you very rapidly where to look in the Moodle interface to create questions once you have it installed. You can see the import/export tabs there too although they are not explained.
(I know, you did not ask for install instructions, I was just shocked at the thought it might take a few days to install
As a Moodle developer I often throw up a new Moodle install to test, and it takes me less than 10 minutes, but then I have done it quite often.)
For the others, if we put them before or after the question, I am wondering whether comments wouldn't be the best way:
// helps-answer: biology102, biology103
// helped-by: biology100
// review-state: proposed
::biology101:: Which animal eats which food? { =cat -> cat food =dog -> dog food }
I think that keeps a nice clear separation between the metadata and the question definition. You could, of course, add more punctuation to make the keywords easier for a computer to find, but I don't think that is necessary.
However, I also have some reservations about using comments for significant information. Anyway, that is the best suggestion I have right now.
If you are on Windows or Mac, then the all-in-one download packages http://download.moodle.org/macosx/ or http://download.moodle.org/windows/ will get you a working Moodle install with just a double-click.
On Linux, I would usually, use the package manager for all the prerequisites (php, apache, mysql) but then download a zip of the source, and then let Moodle self-install.
Anyway, I'm sure you've already found the install instructions Administrator_documentation#System-specific_Instructions_.26_Packages.
The first 2:30 of this video
will show you very rapidly where to look in the Moodle interface to create questions once you have it installed. You can see the import/export tabs there too although they are not explained.
(I know, you did not ask for install instructions, I was just shocked at the thought it might take a few days to install
Tim, thank you. I just saw some comments on the assessment content wikimedia strategy proposal that I wanted to make sure I understood. Apparently the goal of those comments were to suggest a specific kind of adaptation compatible with the spacing effect and the sub-category structure of Bloom's cognitive taxonomy ( http://en.wikipedia.org/wiki/Bloom%27s_Taxonomy#Cognitive .) I agree.
I guess I can only link to three or four things at a time here. I've got most of the install done and think your ideas for using comments to store question metadata and state are excellent. I have updated the microformats.org wiki with your suggestions.
I want to answer your question about the difference between general and specific help but I need to review rules N8 and N13 in http://web.archive.org/web/20050220153210/www.bovik.org/idas/dsd-tutor-rules.txt first from the 1995 del Soldato and du Boulay article first.
I guess I can only link to three or four things at a time here. I've got most of the install done and think your ideas for using comments to store question metadata and state are excellent. I have updated the microformats.org wiki with your suggestions.
I want to answer your question about the difference between general and specific help but I need to review rules N8 and N13 in http://web.archive.org/web/20050220153210/www.bovik.org/idas/dsd-tutor-rules.txt first from the 1995 del Soldato and du Boulay article first.
Tim, I read elsewhere that you are working on recording confidence measures. Self reported confidence is great for this low-stakes application (e.g., for a true/false question you could have certainly/maybe/uncertain/probably not/certainly not; for multiple choice and short answer, a separate set of radio buttons or a pull-down menu.) Also, in addition to keeping those confidence statistics and difficulty/score statistics, independence (which is basically defined as how often help is requested by the del Soldato/du Boulay rules) is recorded too, and a measure of effort. I am not sure whether the effort statistic is used in a way specifically to encourage occasional topic changes for the spacing effect, but it may be.
My question engine changes, which include the Certainty Base Marking, is work-in-progres, and can be downloaded from http://github.com/timhunt/Moodle-Question-Engine-2. The CBM part is done.
Tim, from what I understand that looks fantastic.
Should we collaborate on accuracy review systems? Those seem much simpler and I wonder whether making a PHP module which would work under both Moodle and Mediawiki is possible or advisable. It seems there is a natural synergy along the high-stakes/low-stakes axis so I see no reason that the Wikimedia Foundation would not be able to cooperate with you and Moodle both. Do you think Moodle users would benefit from adaptive testing and challenge/review systems?
The five confidence measures for true/false questions seem related to Likert scales.
Should we collaborate on accuracy review systems? Those seem much simpler and I wonder whether making a PHP module which would work under both Moodle and Mediawiki is possible or advisable. It seems there is a natural synergy along the high-stakes/low-stakes axis so I see no reason that the Wikimedia Foundation would not be able to cooperate with you and Moodle both. Do you think Moodle users would benefit from adaptive testing and challenge/review systems?
The five confidence measures for true/false questions seem related to Likert scales.
To be honest, right now, I don't really want to collaborate with anyone I just want to keep my head down, and finish of this rather large chunk of work I have embarked on.
However, generally I like to see people trying innovative things around the Moodle quiz, and I like to try to be helpful. I would definitely like to see adaptive testing in Moodle. So for now I would just like to watch to see how what you are working on develops. Hopefully in a few months I will have a bit more time. Sorry not be be more helpful right now.
However, generally I like to see people trying innovative things around the Moodle quiz, and I like to try to be helpful. I would definitely like to see adaptive testing in Moodle. So for now I would just like to watch to see how what you are working on develops. Hopefully in a few months I will have a bit more time. Sorry not be be more helpful right now.
No problem, I found someone else to help. I'll ask him to try to write the adaptive testing and accuracy review algorithms as simple PHP modules or pseudocode first, and put them somewhere you can find them when you want to get to them. There may not be a whole lot of database extension overlap between MediaWiki and Moodle, but I don't know either schema very well, so I hope I'm mistaken. You should be able to see that discussion here.
Moodle uses its own DB libraries to talk to the various databases it supports. I think the sensible thing would be to design the code so that all the DB access is in one class, where it can easily be changed, and have the rest of the code make no reference to the database, other than calling methods on the DB access class.
Tim I hope you've been well, and thanks again for your help last month.
For the low stakes only Mediawiki approach, we've decided to simply store the GIFT files in ordinary wiki pages, avoiding database schema changes and obtaining the benefit of undo, diffs, etc. There will be a special namespace for them, and they will display as raw text or rendered quiz questions instead of being wikitext-transformed. I expect it to work out somewhat easier that way.
Google Summer of Code approved the project as part of Wikimedia Foundation's submission, and the Foundation approved my mentoring it, so I'll be working with a volunteer college student named Meadowlark Bradsher this Summer. He agreed to start off with multiple choice questions and has already started in on a format converter between GIFT and the Mediawiki Quiz extension formats to get familiar with php.
I am convinced that there needs to be some sort of an accuracy review system prior to this being a popular Wikiversity sub-project. Do you know of any existing tools or systems to help reviewers vet or proofread quiz content?
For the low stakes only Mediawiki approach, we've decided to simply store the GIFT files in ordinary wiki pages, avoiding database schema changes and obtaining the benefit of undo, diffs, etc. There will be a special namespace for them, and they will display as raw text or rendered quiz questions instead of being wikitext-transformed. I expect it to work out somewhat easier that way.
Google Summer of Code approved the project as part of Wikimedia Foundation's submission, and the Foundation approved my mentoring it, so I'll be working with a volunteer college student named Meadowlark Bradsher this Summer. He agreed to start off with multiple choice questions and has already started in on a format converter between GIFT and the Mediawiki Quiz extension formats to get familiar with php.
I am convinced that there needs to be some sort of an accuracy review system prior to this being a popular Wikiversity sub-project. Do you know of any existing tools or systems to help reviewers vet or proofread quiz content?
GIFT content in ordinary wiki pages sounds like a really simple solution that will work nicely.
I am glad you have a GSoC student. Moodle decided not to participate this year, for reasons I understand, but it still makes me sad.
I don't know of specific tools to help with proof-reading questions. Two thoughts:
1. it is not an efficient way to review a lot of questions in bulk, but something like Moodle's question preview pop-up is essential, I think. The situation I am envisaging is someone in the middle of reviewing a lot of questions off-line, and the come to something they don't quite understand, and want to see exactly how it looks to the student at runtime. They need to be able to fire up a preview.
2. They just rewrote the entry-level maths degree course here at the OU, and as part of that they created 2000 questions. They built a custom system to do that as a python script. The script would load a template then fire up Mathematica to generate a lot of random variants. Then it would write the questions out as Moodle XML format, including embedded LaTeX for the equations.
Anyway, the point is that to facilitate reviewing the questions, there script also output a TeX document (well, actually, it automatically rendered it to PDF) that showed all the options for all the questions for reviewing.
I am glad you have a GSoC student. Moodle decided not to participate this year, for reasons I understand, but it still makes me sad.
I don't know of specific tools to help with proof-reading questions. Two thoughts:
1. it is not an efficient way to review a lot of questions in bulk, but something like Moodle's question preview pop-up is essential, I think. The situation I am envisaging is someone in the middle of reviewing a lot of questions off-line, and the come to something they don't quite understand, and want to see exactly how it looks to the student at runtime. They need to be able to fire up a preview.
2. They just rewrote the entry-level maths degree course here at the OU, and as part of that they created 2000 questions. They built a custom system to do that as a python script. The script would load a template then fire up Mathematica to generate a lot of random variants. Then it would write the questions out as Moodle XML format, including embedded LaTeX for the equations.
Anyway, the point is that to facilitate reviewing the questions, there script also output a TeX document (well, actually, it automatically rendered it to PDF) that showed all the options for all the questions for reviewing.