Community effort to learn about teacher quiz/exam usage habits

Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Število odgovorov: 17
Hi,

I am Olli Savolainen and I am working on the Quiz user interface this Summer. I have been studying the usability aspects during this spring already, and what we have learned so far about the users is here: Quiz UI redesign scenarios index

It is a set of six fictional characters (personas) who are described doing different tasks (scenarios) with exams.

Now you can help. It does not matter if you have not used Moodle Quiz or other electronic exams before. Please reply to this message, telling us how, and for what purpose, YOU are creating exams - electronic or not. Instructions below.

(duration 4min45s) (I tried to embed the video but it did not work; the editor cleaned up the html too much - can an admin fix this please, the embedding HTML can be found on the youtube page?)
(The video contains roughly the same information as what is below.)

The way you can help is as follows:
  1. In the personas page, find a persona that matches your way of making exams (electronic or not).
  2. Read about how that persona is doing exams on the two other pages about making exams: Quiz UI redesign scenarios index
  3. Scan for differences between your way of working with exams, and the way your persona does it
  4. Tell us, in this thread, about what you found: How do you do it?
Listening to you will enable us to *know* what the users actually need the Quiz UI to support, AND there will be documentation to help us in future development work.

The goal is to get a new user interface (UI) in Moodle 2.0. Please also post this message whereever you think we could find teachers interested in exams, who might like to help. (However, please do bear the netiquette in mind.) Thanks.

(I would crosspost this to the following forums, too, since the users of the current quiz probably are best represented in the existing UI already - the most important is to hear the users who, for a reason or another, are not using quiz yet. I will wait if the video above to get properly embedded, though.
http://moodle.org/mod/forum/view.php?f=485
http://moodle.org/mod/forum/view.php?f=24
http://moodle.org/mod/forum/view.php?f=33
http://moodle.org/mod/forum/view.php?f=820)
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Gary Anderson -
Olli:

Thanks for working on this. I am probably closet to Ida, although my use of formal assessments differs in significant ways. Here I give an edited version of her's to describe how I use the Quiz module and and formal test type assessments in general.

Gary (Math Teacher)

Gary teaches several math courses in high school including Advanced Algebra, Calculus II, Statistics, Math Modeling, and an elective course in Software Development. He also works with his faculty and students in customizing Moodle to work well in the secondary school environment. He is quite busy, as are most high school teachers, and needs to economize on his schedule so that he can spend more time collaborating with his colleagues and students.

Quizzes, tests and exams: why and what

Such assessments are about:

  • Evaluating learning of basic factual knowledge of the field
  • Multiple choice question, short answers, and essay questions are used.
  • Paper quizzes are also given when step-by-step processes must be evaluated.
  • Automatically graded questions are especially beneficial when students immediately get feedback on their answers and have a chance to give it a second or third try. It also significantly reduces the teacher load while giving better feedback on the progress of individuals and the whole class on topics covered.
  • Questions banks are generated from Examview software that is aligned to our textbooks and imported when possible to facilitate construction of appropriate questions. This also allows easy generation of sample tests for the students to study from.

Gary sees such assessments as beneficial since they provide measurable grounds on which to compare and grade learning of certain elements of instruction and for quick feedback to students. He wants to have his exam as automated as possible except when hand grading gives him better insight on student progress (seldom does he find that hand graded work does this when dealing with many students). He also believes that there are other valid assessments of student progress that must be included in evaluations including formative responses, measurements of process (rather than just outcomes), projects, and contributions to the development of the whole group of learners.



Povprečje ocenitev: -
V odgovor na Gary Anderson

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Gary,

Wow, you were quick! Thank you, this is valuable insight. I am still interested though, about how Gary goes on to actually creating the exam contents. No pressure, though.
Preparing questions
Organizing questions and adding them into an exam
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Gary Anderson -
Olli:

For Preparing questions, I have them generated by Examview whenever possible. I will then sometimes add an essay question or two that I decide on manually to check for understanding of something emphasized in class and that is generally more open-ended hence giving students a chance to be more creative.

The organization is therefore added by the external program which allows me to select from items taught in the textbook.

I also agree with Joseph's comment that the quiz module should go beyond just assessment and allow for quick feedback for students in checking their understanding and mastery of the material.

--Gary
Povprečje ocenitev: -
V odgovor na Gary Anderson

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Hi, Gary.

Sorry for the delay. I already took a bit of your text into Ida's persona nasmeh. To clarify a bit more, I have a couple more short questions for you.

I understand that you use Examview for storing and organizing your questions, but Moodle for actually implementing them into an actual test/quiz/exam? Or is Examview doing that, too?

I downloaded Examview but since I mostly work on Ubuntu myself, I have not yet tried it out. What does it mean, that you generate questions in Examview? What do you get from Examview, entire question banks for importing to Moodle, or single question categories, or...?

"He wants to have his exam as automated as possible except when hand grading gives him better insight on student progress (seldom does he find that hand graded work does this when dealing with many students)."

Just to make sure: hand grading here equals essay questions, and automatic grading short answers and multiple choices?

"Paper quizzes are also given when step-by-step processes must be evaluated."

Why are paper quizzes best for this purpose?
Could you further elaborate on what you mean by evaluating step-by-step processes, please?

Thanks again nasmeh
Olli
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Joseph Rézeau -
Slika Core developers Slika Particularly helpful Moodlers Slika Plugin developers Slika Testers Slika Translators
Hi Olli,
Thanks for your efforts at re-designing the Moodle Quiz activity. I cannot answer your "persona" questions because I never use Quiz for testing, only for learning. Which leads me to the remark that - from my point of view - your approach is flawed by the fact that the only use you consider for Quiz is "exam". Could you please re-consider your approach to include uses of Quiz for learning or self-testing, something quite different from the "testing/exam" approach?
Thanks
Joseph
Povprečje ocenitev: -
V odgovor na Joseph Rézeau

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
This is indeed a good point. It is brilliant that you commented on this, since especially the new, previously unaccounted for scenarios need to be discovered. Using quizzes as pedagogical tools instead of testing is a central scenario indeed.

There is a question list template that I used to gather the questions, and I will publish them shortly. If you wish, after I publish those more open-ended questions, you can answer them instead, they do not make such assumptions as we have in the current personas. Do you think you might have time to answer such questions?

But if you do not wish to wait for that, I will also appreciate a similar kind of a "descriptive story" as the current scenarios, only with how and from what reasons you do it. This will most likely need to be an entirely new persona, since your presumptions are so different.

Or, if you prefer, we can try a Skype call and I will interview you with my moderate English and my horrible French ;).

Thank you! nasmeh
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Added Paul Pedagogue to the list. Perhaps we can build on that?
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Jeff Forssell -
I've tried to "fill out" http://docs.moodle.org/en/Development:Quiz_UI_redesign_scenarios_-_Background#Paul_Pedagogue. I think that Jacques would probably agree with most of what I've written. So I haven't removed his name there.

Hopefully he'll check it.

Povprečje ocenitev: -
V odgovor na Jeff Forssell

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Tack sååå mycket, Jeff!

I thought your description was indeed useful and adds depth to our understanding to this persona that might be the most important one in the long run. If you wish to make it more easily readable, below are my thoughts.

What I liked about the original text was that it was brief. With the other descriptions it is relatively easy (though this could be further improved) to see an overview of what kinds of different users are involved. So some of these might be useful:
  • Add a brief summary in addition to the body text, which lists the main points
  • Add bold to key concepts so that your writing is more easily scannable
  • If you see appropriate, divide your description, moving some of your text into the two other pages with more practical, action oriented emphasis, to help programmers and UI designers find what is relevant to them at a given moment.
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Jeff Forssell -
What do you mean by " the two other pages"?
Povprečje ocenitev: -
V odgovor na Jeff Forssell

Re: Community effort to learn about teacher quiz/exam usage habits

od Jeff Forssell -
Now I see what you meant by the other pages. I thought those were "in page anchor links" blush

Ugh , back to the drawing board.
Povprečje ocenitev: -
V odgovor na Jeff Forssell

Re: Community effort to learn about teacher quiz/exam usage habits

od Jeff Forssell -
I didn't have this on my to-do list, but now I've written quite a bit on the 3 pages. Perhaps some of the stuff on the persona page would be better moved to the others, but I'm not going to do that now.

(I'm also wondering if the division onto the different pages could be made clearer, but maybe i was just being "dense" sleepy )

Feel free to move anything.
Povprečje ocenitev: -
V odgovor na Jeff Forssell

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Thanks. The UI indeed is not ideal, but it seemed the best I could do with the wiki. It would be great to have a dedicated software to document usability data, but I haven't really seen any.
Povprečje ocenitev: -
V odgovor na Joseph Rézeau

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Hi Joseph,

I have now published the interview questions. If you can, please answer to them in this thread, and please also tell us about anything that those questions may miss about your way of working, so we can get your persona included in the work. Also the background information form's questions are available, you might want to answer them, too. Thank you so much already!

- Olli
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Olli Savolainen -
Admins: Never mind embedding the video after all, since the annotations that are crucial to this video do not show in the embedded version due to youtube restrictions, which sucks.
Povprečje ocenitev: -
V odgovor na Olli Savolainen

Re: Community effort to learn about teacher quiz/exam usage habits

od Shannon Parrish -

Good Morning Olli,

I am just now trying to learn how to create assessments that the computer can score for me. I think the Ida persona is closest to what I am trying to do. I teach Chemistry, Biology and Earth Science in a charter school( Blended Learning Community). I see my high school students in a college -like schedule, so I desire to have activities for them online. I want Quizzes that can show them and me that they have understood and mastered the homework assigned from the text.

I have just discovered SCORM and Hot Potatoes. I attended CUE this spring. Our Moodle Administrator last year did not include SCORM. I am asking for the module to be included in the update.With 125 students, I am overloaded trying to grade traditional homework papers and tests. I have tried to use the Quiz creator within Moodle 1.8 once and got frustrated by the poor word processing within it. Mistakes I deleted persisted; one question said it was right and wrong at the same time...then I was out of learning time.

What I want is a bank of premade Quizzes and tests I can select from that the computer will grade and just give me results. I don't have time to write all these assessments for three different courses( Earth Science, Biology and Chemistry). I am using HOLT Chemistry and Prentice-Hall Earth Science - both have question banks on Examview. Prentice-Hall Biology( Miller and Levine) does not. The authors' website has nice Chapter Self-tests, but no way to collect data on how the student performs out there.

I hope my frustration and inexperience benefits this project! Thank you for working on this stuff! Shannon Parrish, Horizon Charter School, Roseville, California

Povprečje ocenitev: -