This is a report about the usability study that was held at the University of Minnesota on Aug 3, 2015.
Thanks to the people who contributed to the design of this study by providing tasks.
The setup in the usability lab at UofM was really impressive and the staff who ran the study were true professionals. We would not have been able to achieve the insights gained without them. The lab was divided into two rooms, one for a participant and one for the observers. There was a one way mirror separating the two rooms so observers could see into the room.
The study involved six
participants. The participants were screened and were expected to have
little to no experience using Moodle. Participants were
each asked complete a set of five scenarios (linked here earlier).
Briefly, the scenarios covered:
- editing the user's profile and sending messages to students and a fellow staff member,
- creating a page resource and embedding a video,
- creating a quiz including a number of question types,
- creating an assignment and
- creating a choice activity.
data for things like profile fields, questions, descriptions, options,
etc, were provided to the participant in a text file that they were
directed to use. Other files, such as a profile image were also
was still a range in participants' abilities and success. In the 20min
allowed, some participants managed about two scenarios, others more and
one participant was able to complete all six.
Participants used a computer that was equipped with eye tracking to show observers where they were looking. The screen itself, including the user's action, and the view of the participant was also shared with observers. Perhaps the most insightful information was the verbal feedback from participants who were asked to "think aloud". All of these streams were recorded.
The scenarios were deliberately vague, allowing for multiple possible solutions and avoiding leading participants. When participants got stuck (and they all did), they were able to access a "helpdesk", which was a phone line into the observers' room. (Imagine being able to ask Martin Dougiamas for support whenever you got stuck with Moodle.)
Observers noted issues during the sessions. At the end of each session, issues were discussed and collected centrally. At the end of the day, a review of issues was conducted. Some issues were consolidated while others had recommendations recorded against them. I have attached the issues identified to this post as a PDF.
The issues will be reported in the Moodle Tracker so they can be worked on. You will be able to see them together in the epic MDL-51211 and you can follow the progress of these issue from there.
Would you like to be involved further?
- UX Researchers are welcome to view the complete recording to see if they can find more issues. Before you can view the recording, you will need to agree to a code of conduct. A link to the recording can be requested by email to me at email@example.com.
- Developers are welcome to be involved in resolving the issues in MDL-51211.
- Users are welcome to watch issues within MDL-51211 and test out the functional changes to see if they make Moodle more usable.