Since we have at least around hundred teachers we need to have learn to create exams, we cannot possibly teach the quirks of the current UI. We will be very interested in cooperating the moodle community in working on it during this Spring and Summer.
For now I would be very interested in knowing what are the current plans to develop quiz for 1.8 and 1.9? Also, where and in your development cycle I could possibly plug in to do usability testing and planning? I'd be interested in any related documentation and discussion, though I have skimmed http://docs.moodle.org/en/mod/quiz/grading and wandered around in http://docs.moodle.org/en/Developer_documentation. We already have a rough list of what is the most hard to teachers - we are still working on it (through user testing) though, but I'll post it also here as soon as we get it just a bit more ready.
I am a professional PHP guy myself and also a student here in HCI/usability and have been integrating our e-exam app to moodle since last June.
Tim Hunt of Open University UK will be the key responsible one to answer your questions. I would like to say we at Sapporo Gakuin University will be glad to help with comments or field testing if you need that. We were involved early on to use Moodle for large scale testing with simultaneous classes of students. From four years ago, we helped fund development of the flash mp3 player and recently other enhancements, including a split screen mode for quiz media. Any UI efforts are greatly appreciated--especially ways to simplify for new students and new teachers. Looking forward to seeing your "rough list" when it is ready.
Don: as you can see from what's below, it will still need to be verified what's really critical, and of course if you already know about something that's problematic with the quiz UI (and this possibly verified through testing), that would be great. Once we get some prototypes, we'll be grateful for any sort of testing done on those, for example paper prototyping. What do you think?
Tim: thank you for letting me know the status of your work. Hope we'll be able to help you with your workload .
The main problems reported with quiz and new teachers by our support team have been as follows. However, we will still need further testing during the spring to actually know what are the most critical problems. Just to try to keep you up to date about what's going on here, I'll give you some of the thoughts we have so far. Hope I'm not being too aggressive with the following!
Question categoriesThe concept "question categories" versus the questions which are actually already in the quiz. Since we here are developing an exam system which is supposed to partially replace conventional paper exams, the natural mindset for teachers is: to create an exam you design questions and write them into the exam.
In contrast, moodle's current model requires the teacher to first create the questions into a category and then add them to the exam, which has been reported to be very counterintuitive. They just want to put the questions directly into the exam (Nielsen's heuristics: Flexibility and efficiency of use); the fact that they are at the same time creating them in a default category could parhaps be hidden in an "advanced" screen. The system of concepts that needs to be understood isn't really communicated to users who don't already know how it works.
Even if they could grasp the idea, there's simply too much going on in the quiz building screen (including the subtabs). Most teachers don't want to bother themselves with sharing questions; however, many of the exams we will have will have multiple essay questions randomized from categories (which is experimental functionality it seems). This is something most teachers will want, but with the current UI, have no idea of how to create. We have way too many (new) teachers to teach all of them to use the current UI for that.
I'm thinking about initially supporting only the simplest of operations (see progressive disclosure), and then having much of the current things behind an "advanced" button. Anyway, this is just the first thought, completely untested. Since managing questions and categories already has a separate screen of its own, it would seem simpler to keep those tasks away from the quiz UI, but this is problematic of course since the tasks really are related.
TabsThe current hierarchical tabs in quiz management are rather confusing. If you have no questions you will be thrown to the edit tab no matter what, but there's no feedback about why that happens even though all the other tabs are active (links), too.
The active tab isn't really highlighted so it's hard for users to even know where they are; furthermore, the whole idea of "hierarchical tabs" seems problematic, and would probably be best replaced with a hierarchical menu on the left side of the screen (see for example winamp's or eclipse's config windows). Or perhaps the fact that there is a hierarchy could be made more visible (see, for example, firefox 1.5's - not sure about ff2 - tools -> options window). I'm not even sure that a hierarchical menu structure is really required at all for quiz, but naturally redesigning the entire UI structure is something that will take more time than just the half hour I have right now to try to continue the discussion.
That is my 5 cents for now. Looking forward to hearing your thoughts about all this.
Consider a PC file system. Should new users be encouraged to keep all their files in the one big directory (by hiding the hierarchy), or should they be encouraged to think about organizing their files hierarchically from the beginning, to avoid headaches later?
Given our lack ability to search questions in Moodle, hierarchical organization is even more important for questions than for files. Perhaps the problem of teachers who do not understand categories could be solved, not by hiding the categories, but by making them more obvious through improved user interface design, better documentation, etc. (Personally, I like the current design!).
Anyway, this is a "philosophical" choice: in order to market our electronic exam system here (in the University of Tampere), we need to ensure that teachers don't have to make things more complicated than is necessary at any given moment, since there are a lot of them: any potential complexity is a risk for us that the service might not be taken into use as widely as we'd like. Teaching them is not always a choice we can make.
It is important to allow teachers, who think they simply will not have more than one exam, to keep it simple. The UI should definitely allow them to start organizing things from a default category, in case they later realize that they need to organize questions.
And it is really helpful to have your observations about what currently is causing problems for your teachers.
In Moodle 1.8, the design of the tabs has been changed, which makes things much clearer.
Perhaps a minimal change to make the current interface clearer is just to add "This quiz" and "Question bank" headings to the left and right of the screen. What do you think?
Indeed headings titles probably are already an improvement. However, the box with the category selection would also seem problematic to me; as a minimal solution still, it might be a good idea to clearly make the questions box look subordinate to the category, since I guess that's what they are. Correct me if I'm wrong. Below, a quick GIMP sketch about how to perhaps do it.
Good to hear that the tabs will be fixed. However this is problematic for us, since we've made quite many changes to 1.6 so upgrading will probably be painful. Of course there's not much you can do to help this. In an "ideal world", our electronic exam functionality could perhaps be made an alternate "mode" or something of moodle core, but I guess there are lots of barricades to comfront before really imagining this to be possible.
I have now started to design a new UI for quiz from the ground up, which focuses on the current exam being made and thus simplifies the process a lot for teachers who are beginners with moodle (or a keyboard and a mouse ;) ). I do see the new UI also a potential replacement for the current quiz UI, or at least an optional add-on of some kind: it can support all of the features of the current quiz and it is still an option to have the questions in a category, just not emphasized so strongly. I have paper prototype tested the UI with two elder female teachers from our university so far (with stunning success ), and more to come. After finishing with the testing I will be posting the proposal linked to http://docs.moodle.org/en/Developer_notes as Tim suggested above.
Now, what I would need is any documentation of the quiz module there is. What I'm looking for most is a requirements document of some kind: what are the things that quiz will enable users to do/get. This is not exactly the same as a feature listing; I also would need use cases or usage scenarios, on the base of which one could design to all the audiences that Moodle has.
Also, any technical documentation will come in handy. I have to design this primarily for our electronic exam service, but it is an important goal to make it work with moodle core or to be part of core in the future. I will search the development wiki, but will appreciate any redundant links as well. I'm still at Moodle 1.6 but as I've seen the new features I'm thinking about upgrading to 1.8 - the new UI will, right away, have to be compatible with future versions
of Moodle, anyway. If the interfaces have not changed that much and you think we could do the development work on Moodle 1.6, I'd appreciate if there were any notes on what to take into account. I understand that formslib/QuickForm will have to be used - is it possible to use this, for instance, also from 1.6?
My deadline for finishing at least a simple version of the UI (which will at least enable easy making of a quiz) is at the beginning of August 2007.
LOL, we should be so lucky.
The developer documentation that exists is here: http://docs.moodle.org/en/Quiz_developer_docs
Anything not clear from there, the only description of how it should work is to look at the code to find out how it currently works.
And if you you ever try to remove a feature in the name of simplicity, people here will tell you why you cannot possibly do that
However, there are special cases. At Université du Québec à Montréal we are moving from WebCT to Moodle 1.6.3 and will stay to 1.6.3 for at least a year .
Converting 40,000 students and > 2000 teachers to a new interface is a large operation. We will not change the interface (and 1.8 is quite different to see and to manage) during the first year of Moodle .
This is why for my part , I am improving the calculated questions first to Moodle_HEAD then merge mostly all the new features down to 1.6 as I want to use them in my courses.
P.S. Do you plan to use calculated questions in your quiz?
If so, have you done or do you plan to improve them?
I am interested to hear about any upgrades to moodle 1.6, as we are still using it. Also I'd like to know about any challenges downgrading 1.8 code to 1.6 (and perhaps solutions) since I will probably face them myself, too.
P.S. The upper table is updated regularly but not the other parts. As always docs do not synshronize well with updates but this should change for september as our teachers will need them.
Organizing questions seems nearly obligatory in the current quiz creating screen, which gives functionality for (at least) two seemingly separate tasks: creating the quiz and organizing the questions. The screen, being crammed so full of functionality, is something that confuses users with minimal experience, and for our use it is to a disastrous degree (refusing to use the whole thing at all).
That is: I am not planning to de-emphasize categories, but as you also seem to imply, creating and organizing questions is a separate task from actually creating the quiz. I plan to enable users to create quizes without having to explicitly understand too much about categories, but helping them along the way to start understanding them.
Your feature requests indeed seem very relevant and I'm considering incorporating the first two of them to my UI prototype. The third one requires data structure changes so that's something that would perhaps require further discussion. Could you further describe what you mean by "category preview", please?
In my opinion, the Editing quiz page layout in 1.8 is nearly perfect. There are two columns clearly labeled Question bank and Questions in this quiz. This is a huge improvement over previous versions. My only change would be to swap the left and right and halves of the page. People think from left to right (at least it is so in the Northern Hemisphere ), so this would more closely model the two-step process (I would make the same suggestion for the Group and role assignment pages).
Organizing questions seems nearly obligatory
By default, all questions go into Default, so a user who doesn't care if all questions go into one big category can ignore the category menu. Hiding the category-related features under "Show Advanced" would cater to users with minimal experience, but I personally find any such "dumbing down" of the UI to be distasteful.
Thanks to Tim Hunt, "Category preview" is already a feature in 1.8. It is controlled by the checkbox "Show question text in the question list" (see screenshot).
I don't find the current Editing quiz UI confusing. When I teach the quiz module, I explain that the screen has two halves.
That's basically the problem. People are masters at adapting/adjusting; once you learn it by heart you can do it. But it's the technology that needs to adopt, not the users. Creating an exam/a quiz is in itself such a simple task that it shouldn't need to be taught to anyone, especially no one who has already gone through pedagogical education. The current quiz module does indeed do its job - but it doesn't sufficiently communicate to the user how the job, which the user wants done, is done. As we don't have and don't want to spend resources to educate teachers in using a pen and a paper, I don't think we should have to teach professionals how to create exams.
Granted, some of the concepts Quiz uses (such as a random question) are computer-aided exam -specific, and thus need to be taught. But things such as "how do I add a question to an exam" should be expressed to the user far more obviously than what the current UI does. This is a big challenge, and I won't solve everything, but I do believe that making it much clearer is possible. It ain't dumbing down, it's smartening up when things are communicated consistently .
I'm sorry to keep talking about the new UI and not showing it yet. Just want to make sure (as sure as my resources grant me to be) it actually is usable first.
Also we are heavy believers in student-generated content and student-created quizzes. So the interface needs to be *really* intuitive!
Many teachers who are new to Moodle have prior experience using computer resources provided by textbook publishers, including tools for making tests and quizzes like ExamView and MyMathLab. These tools are at least as complex as the Moodle quiz module, and they DO have a learning curve, even for professional pedagogues. But some good documentation and a few short videos are all that's needed to get teachers using these tools. Why should it be any different with Moodle?
The main people doing development on the quiz at the moment are here at the OU. We have two of us working on it pretty full time (me and Mahmoud).
I have been working on integrating Moodle with another OU system, in a way that is basically a replacement for RQP. Mahmoud has been working on improving the quiz navigation - screenshot below. This is now live to our students. However, while it works for our quizzes, it does not work with all possible quiz settings. Also the code is not yet cleaned up enough to commit to core. We hope to get this into Moodle 1.9.
We are in the middle of deciding what are our priorities for the next 6 months of development. We have a huge list of features we want to do, that by my estimates is several person-years of work, so we can't do everything. The things that are likely to happen are:
1. Make the quiz navigation work with a wider range of quiz settings and get it in to core. Hopefully for Moodle 1.9.
2. Make Adaptive mode quizzes work much better.
3. A couple of new question types. (Multiple choice with the scoring rules we like, and some variants on drag and drop.)
4. A workflow system, for when you want a formal process for creating and approving a quiz. This won't go into core. It will probably be a 3rd party module in contrib.
5. We don't have time to implement it ourselves, but we will probably foot the bill for the gradebook development work that Martin Dougiamas want to do. In particular, we want the import/export API.
6. I also have to spend some time developing another OU system related to eAssessment (to talk to the new grade import/export API).
7. Start building up a test suite, to try to make the quiz more reliable.
So, as you can see, the making the quiz editing page much nicer to use does not make it into the next 6 months on our priority list, so it would be really great of someone else could work on it.
Do I understand well that 1.8 will be mostly what is already on the actual Moodle_Head code.
Any other improvements will be for 1.9.
For example, the control of question category sharing (publishing)
involve the category and question display.
As you wrote:
"Unfortunately, the page where you add questions to a quiz has not been touched".
So any improvement should be worked using the new formslib.
Should I wait until the first new formslib version of this page is available or can I submit a proposal for a control of publishing down to the course level by the owner (teacher) of the category using the actual roles/assign.php and roles/assign.html as an example of how to code it.
here an example
I am in the process of adapting my REGEXP question type for Moodle 1.8 and have just discovered the new formslib PHP library and am building from the question_edit_form class. This is a real improvement, although it took me some time to figure out which elements did what and what parameters could or should be passed.
I especially liked the validation($data) function, which performs all the necessary validation before the question being authored/edited is actually saved, refusing to save it if not validated, and precisely displaying the errors exactly where they occur. I could even manage to test that parentheses and square brackets are correctly matched in the regular expressions used in the Answer fields before saving the question, rather than afterwards as I had to do before.
Thanks to all who made this possible. And I'm looking forward to a more user-friendly, less cluttered Quiz interface later on, as described in your post.
PS.- Could you please have a look at bug MDL-8576 which is really blocking me at the moment? (after you have recovered from the jet lag...)
Thanks in advance.