have put this into the tracker, as suggested by moodle helen & martin at http://moodlemoot.ie
Please vote if you think its important, comment and get involved!
In reply to Stuart Lamour
Re: Introducing user testing into moodle development process
by Dan Marsden -
I guess we do this already as a Moodle Partner when submitting (some) of our patches to the tracker - usually the patch is a response to a client reporting an issue - we fix the issue on their site (they test and let us know they're happy) we then submit that patch upstream for inclusion in Core.
We also have an existing testing process that happens after the integration reviewer has looked at the code which you miss in the steps lisetd on the tracker.
http://docs.moodle.org/dev/Process#Integration_workflow_in_the_tracker
so by the time it's hit the core build it's had at least 4 different peoples input.
- developer writes patch and submits for peer review
- 2nd dev peer reviews code
- Integrator (senior HQ Moodle dev) review/test code
- Tester tests interface to make sure operates as expected.
In many cases there are a lot more people involved in a single patch.
I'm unclear "who" would be responsible for this extra "user testing" - part of the issue is that we need the patch to be installed on a site for the "users" to test - so this requires a test site for them to use and a developer/admin to set that up for them. - the testers in the existing integration process use qa.moodle.net which is built with the integration code before it is released in the main build.
Of course the more Testing we do the better - my only concern is the time-frame around this - the new development process stretches out the time it takes to get a patch included in core - I've had a couple of my patches sitting in peer-review state for over 4 weeks. - we don't really want to add another "task" that could delay that even further? - what happens if none of the "users" are interested in testing? - this makes it really difficult and time-consuming to keep a "Current" patch that applies cleanly.
I wonder if instead of adding a step before integration - we could add more user testing to the existing testing process that happens? - that way the user testing can happen on a pre-built/ready testing site?
In reply to Stuart Lamour
Re: Introducing user testing into moodle development process
by Dan Poltawski -
I completely agree with what Martin said in closing that issue. To be honest I found it slightly condescending that you are suggesting we are ignoring user testing in our current development process. So I wanted to comment on a few things:
We are an open source project working with a worldwide community of users and developers. We are not a single organisation, we have a disparate set of users rather than a single prescribed set of users. The way that you suggest we incorporate user testing suggests you have not considered this.
As we are an open source project and the code is available to all, user testing happens a lot more frequently than you suggest, for example:
Many things going into core are tested by contributors with their own users on their own sites (moodle partners, universities etc) before getting near moodle core.
During the 'integration into core' process there is an explicit testing stage
Once its landed into the development releases many community members test these out weekly
Before creating a new Moodle release there is a QA process which also involved real users testing the software out
I guess what you were trying to suggest is that regarding 2) we should have 'a set of users' who do this job? Unlike a single organisation which might have some organisational influence, we cannot force people to test. Moodle HQ can (and does) pay people to test, but by the very nature of the fact that testers need to spend a working day per week testing the issues coming through Moodle, they are not 'normal users on the coalface'.
The process is open. Testers working at Moodle HQ do not need to be the exclusive testers of issues and we would welcome much more testers! If you can find a way a way to get more people to be involved in this process please go ahead and we will support you. The volume of work is quite extensive. Each week about 40-50 issues are fixed in the Moodle releases. Wednesday is testing day where development stops for testing and each test has to pass before we incorporate it into the Moodle release.
Finally, I speculate that anyone motivated to be part of this user testing of Moodle wouldn't be ideal anyway, as they
clearly have a motivation to do this work for Moodle which, frankly, doesn't make them a typical user.
In reply to Dan Poltawski
Re: Introducing user testing into moodle development process
by Stuart Lamour -
Hi Dan,
We also agree with Martin in what he says closing the suggestion :
“I'd like to see more of this taking place during the prototyping/design phase of each new sub-project, and will always recommend it to developers involved.”
and it will be great to see this in the proposals for 2.3/4 http://docs.moodle.org/dev/Roadmap - so apologies if you read our suggestion as condescending in any way.
As an institution external to Moodle HQ we find it easy to see how bug fixes/patches etc are efficiently dealt with in moodle, but difficult to see where ux and end users needs are included in this.
We have snipits on the the dev process from moodle docs, roadmaps and most recently http://salvetore.wordpress.com/2012/03/16/how-moodle-gets-from-hq-to-you-and-how-you-can-help-too/ which obviously do not include any of the steps of user testing for usability, so hopefully this clarifies one reason we feel the need to highlight it.
Both Helen & Martin spoke about community engagement at the recent Moot in Ireland but the number of e-learning managers there or end users (students or teachers) who actually use the tracker was sadly disappointing, so this is another reason we added this - usability testing is something everyone we met felt strongly about but felt that had no control, input or influence over in moodle’s development process. No one did it themselves, or were aware if it actually happened.
What do we mean by user testing for usability?
For us user testing has nothing to do with a developer testing code works, its all about if the end users (mainly students & teachers in moodle) find something easy to do. Personally i find this article, from 2006, still relevant and useful when explaining how to do quick effective user testing - http://24ways.org/2006/fast-and-simple-usability-testing - traditionaly user testing is never done by a coder/developer.
From my own institutions perspective as users of moodle (and i’d hazard a guess most of the moodle community) we are not aware of any of this process you are describing Dan - where moodle hq are testing usability with end users and potential users for any developments that will significantly impact on the user’s workflow - but there could be a solution to this.
Is this type of user testing Moodle HQ's responsibility?
As i’m sure you are aware with products where the end user is directly the person who makes the choice whether or not to use a product, it lives and dies by its usability, and people go to a great deal of care with this. Also i’m sure you are aware that with enterprise software (such as how moodle is often deployed) this direct connection is lost, which is why enterprise software so frequently suffers from poor usability for the end users. But Moodle can be better than most other enterprise software...
Moodle is a caring community, with users needs at its heart, so we openly accept that as curators of the system we need to ensure the end users (students & teachers) needs are represented. As the main curator Moodle HQ is the cornerstone of this, so just as HQ holds the keys to standards of code and developments for moodle admins, when it comes to changes to the end users (students & teachers) workflow HQ could share best practice on the user testing you yourselves do, and lead by example.
Improving transparency
Taking for example navigation or file management in m2 (two of the current issues that cause usability problems for our users) - it would be great if there were some notes, personas, facilitation guides and outcomes from the user testing done by hq that led to their development and current state. It will also be great to see the same for the improvements your working on in files http://docs.moodle.org/dev/Files_usability_2.3
So again, please accept our apologies if you found the ticket condescending but hopefully this reply clarifies the reason we entered the ticket, and helps you understand it in the spirit in which it is intended - as constructive feedback and engagement on behalf of the moodle community, and the end users.
We are sure any kind or increased openness and transparency that yourself and others at HQ are doing usability testing early on with end users/potential user of moodle as part of the development and release cycle would be great for the community to see, and provide a rock solid standard for others submitting significant changes that directly impact on these end user’s workflow.
Cheers
Stuart Lamour
E-learning team @ sussex
In reply to Stuart Lamour
Re: Introducing user testing into moodle development process
by Tim Hunt -
One of my abiding impressions of working at Moodle HQ for a year was how little contact with users there was. (That may well be the only negative thing.) So, actually, HQ is probably not the place to try to do user testing.
A key question for any open source project is how to coordinate what everyone is doing, and how to maximise the number of people who can contribute simultaneously.
Therefore, it seems to me that usability testing is best done by people like us: universities (and Moodle partners) who have access to both usability people and end users. They can do the user testing of the current version, and propose specific changes to improve certain parts of the UI as tracker issues. It could then be either the university, or Moodle HQ that codes the fix.
I have seen this model work well for accessibility testing. There have been several people who have done excellent accessibility reports on Moodle. Their recommendations were sliced up into tracker issues that were then implemented. I have not yet seen this work for usability tesing, but I live in hope.
In reply to Tim Hunt
Re: Introducing user testing into moodle development process
by Stuart Lamour -
tim -as i'm sure you understand accessibility issues could be addressed by ticking some boxes.
Unfortunately, by ticking the boxes it does not necessarily make a site accessible, it just makes it pass some code based automated check tools.
Real accessibility testing comes from observing if a user with accessibility tools can use your website, not from ticking those boxes.
And as we all know, there is no way of automating such a test, you just learn - from observing, then using the tools yourself - how to write good html.
With usability there is no automated report anyone can generate that tells you how to improve your code - as with real usability/accessibility testing, no mater how many boxes you tick it does not make your site/interface more usable/accessible.
If your page takes more than 2 seconds to load - that is bad.
If you don't have a meaningful transparent navigation - that is bad.
If it's complex/difficult for people to make content - that is bad.
these things are accessibility, but also usability.
I have no idea how to auto-generate a report of moodle code about accessibility, except the really basic stuff it fails on.
Do You?
I can only evidence real usability and accessibility issues, not those an automated service makes.
So, i'm sorry there is no ux/usability report you can run against your code. Maybe when we pass the turing test?
In reply to Tim Hunt
Re: Introducing user testing into moodle development process
by Stuart Lamour -
p.s. never met anyone uses jira for ux. It's great for 'bugs' in the trad code meaning, but rubbish for ux issues. It's a great tool, but not the right one for usability or ux stuff.
In reply to Stuart Lamour
Re: Introducing user testing into moodle development process
by Paolo Oprandi -
Stuart,
I like your point about enterprise solutions. There appears to be a pattern of institutions acquiring software solutions apparently with all the desired functionality but poor usabilty and then wonder why the users have such difficulty using it.
I also thought the 24 ways article you have pointed to useful - especially the "what to look for" section. Testers (aka hedgehogs) very often say what they think will please me (especially if they know I am a developer) and as a result the issues go undetected.
Cheers Paolo
E-learning team @ sussex