Here's another case of a UI that doesn't map to a decent mental model: Workshop.
Teachers can't figure out how to read the grades and therefore can't explain the grades to their students. No developer to maintain workshop. Bugs in grading. Net result: Workshop fades into the background and Moodle moves one step closer to becoming a commodity LMS.
Incidentally, the typical user documentation for Workshop gives a blow-by-blow description of the UI and regurgitates the help files. This kind of documentation is useless to readers, and I doubt that its authors understand grading any better than their readers.
Workshop needs is a UI that is closer to the user's mental model of how grades are determined. Or, if the user doesn't have a mental model, the UI must help the user form a mental model. Then we need documentation that references the model.
Your grade in a Workshop has two components: a grade for your submission, and a grade for your assessments. My mental model is an N x M matrix, where N is the number of students and M is the number of submissions.
N and M are usually about equal, since each student makes a submission. Therefore we have a roughly square table. For a decent size course, the table is big, maybe 30 x 30. If the student does only self-assessment (as in an Exercise) we get a diagonal matrix. If there are peer assessments, there are off-diagonal elements. But in any case, the table tends to be sparse.
To conserve screen real estate, the current Workshop UI compresses the big square matrix into a bunch of compact tables. And by so doing so, it destroys the connection to the model.
What if Workshop actually presented a big square matrix? Would users have a better understanding of how grading works? Even if they had to scroll, which would be annoying, I would say the answer is Yes. I say this based on my own experience remapping the output of several workshops. Once you do this, the grades virtually jump out at you. Documentation keyed to this matrix, rather than to the current UI, would be able to explain the subtleties of workshop grading very easily. See screenshot.
Once you see the grades displayed in a natural way, bugs in the grading algorithm also jump out at you. If the bugs jump out, maybe some developer would be willing to jump in and fix them. With the current UI, a single bug has a ripple effect causing it to show up in several tables.
Can we please focus on Workshop and not bring "peer lite" into this discussion?
workshop.png
I coded the matrix UI in Workshop on my test system. The screenshot below shows the matrix for an actual Wokshop that I set up and ran as follows:
The Submission grades (bottom row) are calculated by averaging the raw grades in each column. For example, submission w-s4 gets a submission grade of (100 + 100 + 90)/3 = 97%.
The Assessment grades are gotten by averinging the cooked grades in each row. For example, student s4 gets an assessment grade of (100 + 50)/2 = 75%.
Student s4's final grade for the Workshop will be
Does this matrix help you understand how grading works? Would you have any difficulty explaining the grades to a student? IMO the only difficulty I see is explaining the Comparison of assessments (how the cooked score is computed), but that's not a user interface issue.
- ten students: s1, s2, ... s10
- teacher John
- no examples provided by teacher
- weight for teacher assessments = 1
- student s1 submitted work w-s1, s2 submitted work w-s2, and so-on
- Very strict comparison of assessments
- each student assesses his own work and work of one other student
The Submission grades (bottom row) are calculated by averaging the raw grades in each column. For example, submission w-s4 gets a submission grade of (100 + 100 + 90)/3 = 97%.
The Assessment grades are gotten by averinging the cooked grades in each row. For example, student s4 gets an assessment grade of (100 + 50)/2 = 75%.
Student s4's final grade for the Workshop will be
submission grade x points for submission + assessment grade x points for assessment
Does this matrix help you understand how grading works? Would you have any difficulty explaining the grades to a student? IMO the only difficulty I see is explaining the Comparison of assessments (how the cooked score is computed), but that's not a user interface issue.
workshop_grading.png
John, you're a legend!
The UI weirdness has stopped me from using the workshop module at all due to it being so hard to understand and explain.
This is infinitely better. Could I suggest a small bit of text above the table showing the how the calculation for cooked grades is set in the workshop, and rollover javascript bits for the grades to show the origin and calculation for each of the numbers?
What will the student see, presumably just their own column?
Matt
The UI weirdness has stopped me from using the workshop module at all due to it being so hard to understand and explain.
This is infinitely better. Could I suggest a small bit of text above the table showing the how the calculation for cooked grades is set in the workshop, and rollover javascript bits for the grades to show the origin and calculation for each of the numbers?
What will the student see, presumably just their own column?
Matt
Hi Matt,
I had originally posted the above to another discussion where I offered it as an example of how user interfaces should try reflect the user's conceptual model as directly as possible. Then a moderator split the discussion and my post ended up here. So you really need to understand the above post in the context of that other discussion, which was mainly about the quiz UI.
So although I was not actually proposing 'The Matrix' for the actual Workshop UI, I would obviously like to see something like it implemented.
For a more compact view, you could take the user's row (the assessments he has done) and the user's column (assessments by others of his work) and join them, squeezing out the empty cells. The result would have a "tee" shape like the screenshot below, which is just a mock-up for user s4 (who submitted w-s4). I guess a student would see the same view without the names of other users.
As you know, Workshop is in a strange state at the moment. It's a core activity without a maintaner, on its way to a silent death unless someone steps forward to save it. I know you're aware of Peter Sereinigg's project to replace Workshop with a new "lite" activity because I've seen your posts in those discussions, but IMO that effort is still in the requirements phase. Meanwhile we have an activity that already works pretty well except for some minor usability issues (the grade table, those ugly split screens!) and some fairly sigificant issues (e.g., need to integrate with the new gradebook). I would to help in any way possible to save this beautiful activity.
Workshop....
I had originally posted the above to another discussion where I offered it as an example of how user interfaces should try reflect the user's conceptual model as directly as possible. Then a moderator split the discussion and my post ended up here. So you really need to understand the above post in the context of that other discussion, which was mainly about the quiz UI.
So although I was not actually proposing 'The Matrix' for the actual Workshop UI, I would obviously like to see something like it implemented.
For a more compact view, you could take the user's row (the assessments he has done) and the user's column (assessments by others of his work) and join them, squeezing out the empty cells. The result would have a "tee" shape like the screenshot below, which is just a mock-up for user s4 (who submitted w-s4). I guess a student would see the same view without the names of other users.
As you know, Workshop is in a strange state at the moment. It's a core activity without a maintaner, on its way to a silent death unless someone steps forward to save it. I know you're aware of Peter Sereinigg's project to replace Workshop with a new "lite" activity because I've seen your posts in those discussions, but IMO that effort is still in the requirements phase. Meanwhile we have an activity that already works pretty well except for some minor usability issues (the grade table, those ugly split screens!) and some fairly sigificant issues (e.g., need to integrate with the new gradebook). I would to help in any way possible to save this beautiful activity.
Workshop....
"One of the most modern and beautiful modules in Moodle" -- Ger Tielmans
"An extremely innovative part of Moodle with a lot of pedagogic potential which needs to be developed and promoted more" -- Gustav Delius
"One of the most useful modules for us" -- Rob Brenstein (and many others)
"An extremely innovative part of Moodle with a lot of pedagogic potential which needs to be developed and promoted more" -- Gustav Delius
"One of the most useful modules for us" -- Rob Brenstein (and many others)
compaction_of_workshop_grade_table.jpg
Matt,
Re difficulty in explaing Workshop: I have a short power point "animation" that I use to explain Workshop to teachers. If you e-mail me (my e-mail address is in my profile), I'll send you a copy. It's too big to attach.
Re difficulty in explaing Workshop: I have a short power point "animation" that I use to explain Workshop to teachers. If you e-mail me (my e-mail address is in my profile), I'll send you a copy. It's too big to attach.