Moodle research

What Moodle teacher tasks should be included in usability study?

 
My mug
What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, all.

Equipment at the usability labWe have an opportunity at the University of Minnesota to access their superb usability lab to conduct usability testing.

The target of the usability study will be teachers as new Moodle users.

I thought it would be good to crowd-source the tasks we ask participants to undertake. So here is a question for you.

What basic tasks should teachers be able to complete when they first use Moodle? (let's say 5 suggestions max)

Tasks could be creation of activities, resources or other course related activities. Assume the teacher has a course ready to play with.

The overarching goals of the study are as follows.

  • To gain insight into Moodle features.
  • To pinpoint areas for improving documentation, if needed.
  • To identify areas that lead users to experience confusion, frustration or inefficiency.
  • To ensure users can accomplish their own goals without outside help.
If you're interested in seeing results and collaborating after testing takes place, subscribe to this discussion.

 
Average of ratings: -
Picture of Toni Soto
Re: What Moodle teacher tasks should be included in usability study?
 

Hi,

I think that all participants in moodlemootUS15 already know how to start adding resources or activities to a Moodle course. but I wonder how long does I take to a novice to find the way to start adding something to a course without help.

Moodlecloud recently presented by Martin, is a great idea but how easy is for a new user to perform without support the most important task for a teacher when taking the control of a virtual course?

My apologies if my post is not related to what you were looking for.

Best regards from Spain. I'd like to go to moodlemootUS15 this year but its a a bit expensive from where I live. Enjoy your moot and share as much as possible through twitter.

Toni Soto


 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, Toni.

It's new users who would be asked to undertake the tasks of the usability study. I might amend my post to make that clearer.

What tasks would you like to see a new teacher using Moodle to complete?

 
Average of ratings: -
Picture of Marcus Green
Re: What Moodle teacher tasks should be included in usability study?
Core developersParticularly helpful MoodlersPlugin developersTesters

Take a non Moodle user who is comfortable using web sites and ask them to create a quiz with one instance of each Core question type in it. See which question types they click on the help icons/ do google searches/ask for help with.

Time how long it takes to create each question.

Count how many "errors" they make (you decide what an error is).

A day later get them to do the same exercise without access to any help.

All the question types they cannot figure out get them in a new "advanced" category (or perhaps a disabled and invisible category) of question type......

 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Thanks, Marcus.

Yes, Quiz and a few questions would definitely need to be one task.

I'm don't think we'll have the chance to re-test participants.

What other activities/resources/course setup should we test?

 
Average of ratings: -
Picture of Mark McKay
Re: What Moodle teacher tasks should be included in usability study?
 

Keep in mind that each participant will only be working for about 30 minutes.

 
Average of ratings: -
Picture of Emma Richardson
Re: What Moodle teacher tasks should be included in usability study?
Documentation writersParticularly helpful Moodlers

Perhaps they can pick certain tasks or be randomly assigned certain tasks to complete from a list of the most important ones.

 
Average of ratings: -
Picture of Emma Richardson
Re: What Moodle teacher tasks should be included in usability study?
Documentation writersParticularly helpful Moodlers

Edit their profile and add a photo of themselves.

Create activities. 

Add resources.

Set up a gradebook and enter grades. 

Look at course participants and send a message out to one or all of them.


 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

That's a good example of the sorts of tasks we need.

Thanks, Emma.

 
Average of ratings: -
Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

Clearing inline style tags out of atto to try make your content usable or professional looking - always fun time smile

 
Average of ratings: -
Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

Organising content in a large course with drag n drop.

Also lots of fun on touch screen devices.

 
Average of ratings: -
Picture of Marcus Green
Re: What Moodle teacher tasks should be included in usability study?
Core developersParticularly helpful MoodlersPlugin developersTesters

Organising content in a large course with drag n drop.

Could you expand on that Stuart

 
Average of ratings: -
Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

Trying to delete the course news forum they didn't request or add.

 
Average of ratings: Useful (1)
Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

Sending a message to a student, group of students or the whole course.

 
Average of ratings: -
Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

Figuring out why some items in the breadcrumb are clickable - normally the page you are already on - and other things in the breadcrumb are not clickable.

 
Average of ratings: -
Mary Cooch
Re: What Moodle teacher tasks should be included in usability study?
Documentation writersMoodle Course Creator Certificate holdersMoodle HQParticularly helpful MoodlersTestersTranslators

  • uploading files by drag and drop
  • setting an assignment (ie a task for their students to submit)
  • grading on Moodle
  • viewing the course participants and messaging them
  • using  Moodle with their mobile or iPad/tablet

 
Average of ratings: -
Picture of merrin fulton
Re: What Moodle teacher tasks should be included in usability study?
Moodle Course Creator Certificate holders
  • Set up their profile
  • Choice activity
  • Locate the Moodle video channel on You Tube (so they can get access to further help)
  • add in the online user block
  • check what sort of enrollment is set

Each of these would help them to understand our community ideas of

  • learners expect to know who is teaching them and something of their background
  • learners are encouraged to be interactive not passive
  • teaching them to be even a little bit self-sufficient is always a good idea (our guys find moodle docs straight off a little daunting)
  • teaching them how to tell who can enrol in their course.



 
Average of ratings: -
Picture of Jessica Wicks
Re: What Moodle teacher tasks should be included in usability study?
 

Thanks for asking.

From a user support perspective, here are the things I get asked about a lot:

  • Gradebook configurations:
    • Setting up a points-based vs. weighted gradebook with categories
    • Hiding/displaying/editing what students see in relation to the gradebook (e.g., running totals)
  • User activity:
    • Locating details such as last date of participation, activities accessed, etc. for a single user.
    • Viewing a single user's forum posts together for scoring.
 
Average of ratings: -
Ryan Hazen
Re: What Moodle teacher tasks should be included in usability study?
 

Change the number of sections in a course

Show and hide resources

message users who have / have not completed a certain task

embed a video from youtube

embed a video from elsewhere (TED, Vimeo, etc)

view students' logs

set up conditional access activities

set up any sort of form (feedback, questionnaire, etc)

 
Average of ratings: -
Mary Cooch
Re: What Moodle teacher tasks should be included in usability study?
Documentation writersMoodle Course Creator Certificate holdersMoodle HQParticularly helpful MoodlersTestersTranslators

(actually yes- I want to add Ryan's "embed a YouTube video" to my list too -very important for newbies.)

 
Average of ratings: -
Picture of Dan Froelich
Re: What Moodle teacher tasks should be included in usability study?
 

  • Time to construct a gradebook that reflects the proper weighting and organization of a course.
  • Using a sample course with data to learn how to grade/ provide feedback on assignments, forums, and other activities.
  • Building a branched lesson.
  • Create an instructional design model for delivering content that is practical to duplicate.
  • How long it take for them to see how awesome Moodle is.
 
Average of ratings: -
ME
Re: What Moodle teacher tasks should be included in usability study?
Core developers

Tasks I would ask to check for a student are:

  1. Add or update a profile picture.
  2. Navigate to a different course when they are in an activity
  3. Determine which activities are due
  4. Submit an assignment
  5. Message another student or teacher
 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Thanks for all the excellent suggestions, guys.

I've tried to collect your ideas together and group them. I've put them in an implied order.  Within our time limits, we would probably only be able to achieve one from each list and probably only down to organising the course.

Would you agree that this represents a priority/order for basic teacher tasks?


User profile

  • Edit their profile and add a photo of themselves.
  • Set up their profile
  • Add or update a profile picture.

Viewing participants and messaging them

  • viewing the course participants and messaging them
  • Look at course participants and send a message out to one or all of them.
  • Sending a message to a student, group of students or the whole course.
  • message users who have / have not completed a certain task
  • Message another student or teacher

Adding a resource

  • uploading files by drag and drop
  • embed a video from youtube
  • embed a video from elsewhere (TED, Vimeo, etc)
  • Locate the Moodle video channel on You Tube (so they can get access to further help)
  • Add resources.

Creating an activity

  • create a quiz with one instance of each Core question type in it
  • setting an assignment (ie a task for their students to submit)
  • Choice activity
  • set up any sort of form (feedback, questionnaire, etc)
  • Using a sample course with data to learn how to grade/ provide feedback on assignments, forums, and other activities.
  • Building a branched lesson.
  • Create activities.

Organising the course

  • Organising content in a large course with drag n drop.
  • Change the number of sections in a course
  • Show and hide resources

Working with the Gradebook

  • Set up a gradebook and enter grades.
  • grading on Moodle
  • Setting up a points-based vs. weighted gradebook with categories
  • Hiding/displaying/editing what students see in relation to the gradebook (e.g., running totals)
  • Time to construct a gradebook that reflects the proper weighting and organization of a course.

Viewing student information

  • Locating details such as last date of participation, activities accessed, etc. for a single user.
  • Viewing a single user's forum posts together for scoring.
  • view students' logs
  • add in the online user block

Tasks we're not likely to be able to achieve in the testing environment

  • set up conditional access activities
  • check what sort of enrollment is set
  • Determine which activities are due
  • Submit an assignment
  • Navigate to a different course when they are in an activity
  • using  Moodle with their mobile or iPad/tablet
  • Create an instructional design model for delivering content that is practical to duplicate.
 
Average of ratings: -
Picture of Mark Glynn
Re: What Moodle teacher tasks should be included in usability study?
 

Thanks for collating all of the suggestions. Can I add one more topic and one more navigation task please:  

  1. the creation of groups within Moodle. 
  2. getting back to the home page or course page

But otherwise you have enough to start with in my opinion.


I look forward to seeing the results


Kind regards


Mark

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: What Moodle teacher tasks should be included in usability study?
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

I would not include every core question type. Probably just multiple-choice, given the time available.

 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Thanks to everyone who has been involved in the preparation for this usability study so far.

The study will take place at the University of Minnesota usability lab on Monday 3 August between 8am and 3:30pm (see this in your timezone) with some reflection afterwards.

Would you like to be an observer, viewing user actions through live video and eye tracking and helping to comment on a collaboratively created document during the usability study? You can can be part of this online and you don't have to be available for the entire day. If so, please email me on michaeld@moodle.com so I can arrange for you to be added to the live session.

For those interested in the plans at this stage, see the attached documents including the scenarios developed from your suggestions.

For those unable to make it in person or participate online, we will hopefully be able to share videos and files here afterwards.

Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

I'm really pleased to see hq engaging in real user testing.

https://moodle.org/mod/forum/discuss.php?d=200924

The tracker issues seems to have been closed at the time, so good to see you guys are now doing this and look forward to it being reopened.

https://tracker.moodle.org/browse/MDL-32544

Once you get an idea of the terrible usability of moodle and  benefits of user testing it would perhaps be beneficial to reach out and see how other open source projects overcame this?

https://moodle.org/mod/forum/discuss.php?d=176557

 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

The usability testing was a revealing exercise and showed how a close look at usage of a small number of users can be a worthwhile exercise.

I'm writing up a report and I'll share it here next week. We're happy to share the results so others can see and contribute to the follow-up work.

 
Average of ratings: Useful (3)
Picture of Stuart Lamour
Re: What Moodle teacher tasks should be included in usability study?
Plugin developers

Hi Michael, any updates on this? Will the videos be available?


 
Average of ratings: -
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

This is a report about the usability study that was held at the University of Minnesota on Aug 3, 2015.

Thanks to the people who contributed to the design of this study by providing tasks.

The setup in the usability lab at UofM was really impressive and the staff who ran the study were true professionals. We would not have been able to achieve the insights gained without them. The lab was divided into two rooms, one for a participant and one for the observers. There was a one way mirror separating the two rooms so observers could see into the room.

Usability lab rooms


The study involved six participants. The participants were screened and were expected to have little to no experience using Moodle. Participants were each asked complete a set of five scenarios (linked here earlier). Briefly, the scenarios covered:

  • editing the user's profile and sending messages to students and a fellow staff member,
  • creating a page resource and embedding a video,
  • creating a quiz including a number of question types,
  • creating an assignment and
  • creating a choice activity.

Textual data for things like profile fields, questions, descriptions, options, etc, were provided to the participant in a text file that they were directed to use. Other files, such as a profile image were also provided.

There was still a range in participants' abilities and success. In the 20min allowed, some participants managed about two scenarios, others more and one participant was able to complete all six.

Participants used a computer that was equipped with eye tracking to show observers where they were looking. The screen itself, including the user's action, and the view of the participant was also shared with observers. Perhaps the most insightful information was the verbal feedback from participants who were asked to "think aloud". All of these streams were recorded.


Streams of information


The scenarios were deliberately vague, allowing for multiple possible solutions and avoiding leading participants. When participants got stuck (and they all did), they were able to access a "helpdesk", which was a phone line into the observers' room. (Imagine being able to ask Martin Dougiamas for support whenever you got stuck with Moodle.)


The helpdesk


Observers noted issues during the sessions. At the end of each session, issues were discussed and collected centrally. At the end of the day, a review of issues was conducted. Some issues were consolidated while others had recommendations recorded against them. I have attached the issues identified to this post as a PDF.

The issues will be reported in the Moodle Tracker so they can be worked on. You will be able to see them together in the epic MDL-51211 and you can follow the progress of these issue from there.

Would you like to be involved further?

  • UX Researchers are welcome to view the complete recording to see if they can find more issues. Before you can view the recording, you will need to agree to a code of conduct. A link to the recording can be requested by email to me at michaeld@moodle.com.
  • Developers are welcome to be involved in resolving the issues in MDL-51211.
  • Users are welcome to watch issues within MDL-51211 and test out the functional changes to see if they make Moodle more usable.
 
My mug
Re: What Moodle teacher tasks should be included in usability study?
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

An update to say that work has started on a number of issues that were raised in the Usability study. See MDL-51211 for progress there.

Thanks again to the people involved in preparing for and being part of the usability study.

 
Average of ratings: -