Concerns about testing and release process for the Workshop Module

Concerns about testing and release process for the Workshop Module

by Nic McPhee -
Number of replies: 5
First, let me be real clear:  I really like Moodle and greatly appreciate the work that everyone does to make this tool available to folks like me.  I also really like at least the idea of the Workshop module, but the fact of it has been a much greater struggle than any other Moodle component that I've worked with.

To be blunt, the implementation is incredibly fluid and seems pretty flakey at times.  This may be swell if you're able to grab the latest from CVS at a whim, and roll with it when things act up, but it's a bit rough when I've got to go through several layers of admin to get a simple three line edit performed, and am trying to sell this to both students and faculty that are fairly familiar with WebCT and often suspicious of this "new thing".

I certainly don't want to put a damper on the enthusiastic work going on for this nifty module, but I would like to be able to download a stable version and have some reason to believe that it will work as advertised (and that someone actually knows what "as advertised" means and has written it down).  We did our initial install of Moodle in late August, and the only thing that ever needed a patch was the Workshop Module (which, BTW, was delivered by Ray with amazing speed and was greatly appreciated).  We just did an upgrade to 1.4.3, and everything appears to be working fine except for the Workshop, which has a variety of issues ranging from silly cosmetic problems to potentially significant algorithmic errors.

Can I be bold and suggest two things that might help?
  • Use branches in CVS to manage the process in a way that will allow for backpatches to fix older versions?  My sense from the discussion and looking at the CVS logs is that work is currently barrelling forth on the main branch and that concrete changes/fixes aren't being committed/managed in a way that would allow them to be applied to older versions.  Better use of branches and tags would allow the Workshop module in version 1.4.4 of Moodle to actually be 1.4.3 + plus some basic bug fixes and tested, stable enhancements instead of whatever happens to be in the main development branch when it is released.  This would create a meaningful notion of a "stable" version of the Workshop Module, and allow things like bugfixes on the development branch to be patched back onto the stable branch in a controlled way.  (I apologize if I'm reading this wrong and there's more process here than I've been able to discern.)
  • Come up with some notion of functional and/or unit tests that can be run to help define the desired/correct behavior of the Workshop Module, and to help verify that the "stable" version of it is correct before a release.  This module is unusual in that it includes a lot of non-trivial number crunching and very complex interactions.  This makes it very difficult to test (as has been pointed out by several developers in this forum).  The problem, then, is if you don't use it exactly how the main developers do, you're essentially stepping into uncharted waters, but you don't have any way of knowing that, because there's no documentation of what's tested and what isn't.  Note also that there would probably need to be at least slightly different specs for different versions so, for example, you can verify that a patch applied to an old version doesn't break the spec for the behavior of that version.

Unfortunately I just don't currently have time to contribute to the coding side of things in a meaningful way, and am not likely to until this summer sad.  Thus I'm being the bad guy that throws stones and runs away, and for that I apologize.  I would certainly be willing to help, for example, sketch out the specs for the module and suggest some tests that might help operationalize those specs.

Again, Moodle rocks, and the Workshop Module has enormous potential.  I just think it would benefit everyone if a little structure was brought into its development process.
Average of ratings: -
In reply to Nic McPhee

Re: Concerns about testing and release process for the Workshop Module

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Hi, Nic.

Just a correction on your perception here ... the STABLE branches (for all modules in Moodle) are stable, and only receive bug fixes which are then ported to the development trunk (rarely the other way around).

You can see the ages of the workshop files in 1.4.3 - there's almost no new stuff in there.

Do you perhaps have a copy of the workshop from 1.5dev by accident?

I do agree in general that some sort of more formalised testing regime would be useful, it's just so damn difficult with this fluid web stuff!

In reply to Martin Dougiamas

Re: Concerns about testing and release process for the Workshop Module

by Nic McPhee -
Thanks for the correction and my apologies for having read that wrong.  I'd never seen any indication of people releasing patches relative to the stable branch (e.g., "If you want to add this feature to 1.4.2 apply this patch") so wasn't clear to me that things happening in the development branch were being made available as patches to stable branches.

I'm pretty sure I don't have a 1.5dev copy of the workshop (it's version 2004060401).  It certainly doesn't have the new UI that Gustav has posted about and which sounds like a significant improvement smile.  The fact that you ask makes me increasingly worried that the workshop part of my DB is corrupted in some fashion sad.

If I'm right and I'm running 1.4.3, then so far I'm 0 for 2 on having implementations of the workshop module from the stable branches that worked in what I would consider a stable way.  And I've found no bugs in any other module.

I realize that testing the web stuff is very hard, and particularly so with a module like Workshop that's being constantly modified.  I wouldn't, for example, suggest testing for all the UI elements, because they're just too unstable and ultimately not crucial to things like grades.  What concerns me a lot is the possibility (probability?) that mistakes are going to creep into the complex calculations that determine students grades and can thus profoundly affect their lives.  I suspect that there is an error that could (depending on the way a particular workshop is configured) cause a student to receive a substantially lower grade than the others even if they did all the work that was assigned them, and did it well.  If I'm right, then that's a major problem.  It's unethical at the very least, and potentially the stuff of lawsuits in extreme cases.

As far as I can tell, the workshop module has unique risks in this regard.  Other modules do grade math, but it's typically public and transparent so that students and instructors can double check it if they are feeling nervous.  If I'm concerned about the forum module's averaging of peer ratings, it's pretty easy for me to look at a set of ratings, do the average in my head, and then check that the module did it correctly.  The workshop module is an entirely different beast, however, full of black magic and mystery.  There are no specs (that I'm aware of) on things like the allocation algorithm or the algorithm that decides which evaluations to leave out.  Even after running quite a few workshops I'm still not clear on how exactly the grade on how well a student did their assessments is computed, or how exactly it's factored into the overall workshop grade.

My point, though, is not to have someone post these here to make me happy.  My point is that the algorithmic backend of the workshop module is
  • Sufficiently important,
  • Sufficiently complex, and
  • Sufficiently opaque
that it really needs to be documented carefully and tested.  I'm not suggesting tests for all of Moodle (although that would be wonderful smile), or even for the UI parts of the workshop module.  Just the backend where there are these complex, crucial calculations and mistakes aren't easy to detect.
In reply to Nic McPhee

Re: Concerns about testing and release process for the Workshop Module

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
"If you want to add this feature to 1.4.2 apply this patch" ... this never happens because stable branches receive bug fixes only. You can download complete recent versions from the modules download page.

To make things more stable in the stable branch we need more people reporting bugs using the bug tracker, and more people taking on the task of fixing bugs.

I totally agree with you about the Workshop calculations needing to be more transparent. approve I've never had occasion to use that module much so the details are a mystery to me too! surprise
In reply to Martin Dougiamas

Re: Concerns about testing and release process for the Workshop Module

by Nic McPhee -
"feature" was a poor choice of word there.  Something like "If you want to fix this bug in 1.4.2 apply this patch" might have been more accurate.  And maybe the small releases (like 1.4.3) may be frequent enough that bug fixes there work fine.

Looking through the CVS lots for this module, though, there's very little evidence that work in the main branch is ever brought into the stable branch.  In a fair bit of looking I only found two or three examples of that.  This suggests that the workshop for Moodle versions like 1.4.2 and 1.4.3 are essentially whatever was released in 1.4.0 because bug fixes aren't being patched back in to the stable branch.  Given my perception of its stability, that surprises me.  I am, however, guilty of not always posting/documenting bugs (trying to be better), and certainly haven't taken the time to learn the code and contribute sleepy.

Transparency = Good! big grin

Thanks again to all.
In reply to Martin Dougiamas

Re: Concerns about testing and release process for the Workshop Module

by Michael Schillo -
As I keep having trouble with the workshop module and ray is not answering email I am more than happy to find a new way of directing attention to the flaws in this module. I am happy to use the bug tracker to report bugs. So far I thought the forums were the place to report bugs.

I finally gathered that the bug tracker is under "developer tools". I am not a developer, so I never thought of looking at this section. You could increase the flow back on bug reports if you moved that section to a more obvious place.

Also it's another obstacle for users that they need to set up a second account to be able to post a bug report.

I disabled the unusable workshop module in an otherwise great, stable and beautiful tool.

Cheerio,

Michael.