Workshop

Hi David!

It's awesome that you're revitalising this module!

I just have one comment - at some point I worked on a project using this where the client wanted to use something very similar to the 80/20 submission/assessment grade method, except that the 80% submission grade should come only from the teacher, ignoring the peer assessment.

In fact, the peer assessment was formative only, no grade and the 20% came from the quality of the formative assessment. The teacher was 100% responsible for grading the submission itself.

I don't think that is possible in the current system, and I tried to look through the code and recoiled in horror and ended up using a combination of the assessment module and rated q/a forums instead. Which actually worked very well!

Hi David!

Excellent! We actually switched to using forums or the glossary for doing peer review and grading, but one feature the workshop module used to handle was the random assignment of peer work for review to students that has not been replicated by any other tool yet.

It would be great if the user-interface for setting up the workshop were a little friendlier, too, because the old interface (we last used in version 1.6) was a little overwhelming to figure out without assistance (group huddle).

I haven't looked at the specs yet - but will try to. Just wanted to add my encouragement that you're reviving this feature.

Thanks again,

Mary

The one thing that stopped me from using workshops with 11 and 12 year olds was that allocation of work to peers for assessment. In a class of 30 it was almost guaranteed that at least one pupil would not hand work in on time. They were then left out of the peer assessments. I ended up having to run the different stages on different days to give me the chance to check that work had been handed in. The option to do the allocation of peer feedback based on the groups, would have solved this. Pupils might initially be assigned a blank assessment but it would arrive later once work was submitted.

The option for the teacher to manually end one stage and start the next instead of working to set time frames would be very useful as working with different classes they can't all use the same time settings and changing them between classes then changing them again was a big problem.


I have moved to using forums for peer feedback but pupils tend to ignore instructions on who to assess and assess their "friends" I spend more time getting pupils to follow the instructions and stay on track than checking the peer assessments and constructive feedback this way.

I am very pleased to see that the workshop module will be rewritten. I tried it out for a college level science class a couple years back. This was with Moodle version1.6 or 1.7. I ended up having to pull out the different grade components for each student and calculate the grade via a spreadsheet. This was because the module was not calculating the grade as I had expected (and advertised to my students).

I have also tried the Calibrated Peer Review web application (http://cpr.molsci.ucla.edu/) and was surprised when it simply did not work for me. Even though the Moodle workshop doesn't calculate grades correctly and has other flaws it is clear to me that it was created with more built in flexibility than CPR.

I hope that the new Workshop module will allow images, audio files, and multimedia files to be submitted. The workshop tool, although usually used for writing projects, should be able to accommodate multimedia projects, submitted in common file formats. This would be of great benefit to me and my students.

I look forward to finding a way to contribute to this project.

Daniel

Hi David,

very exciting to see this finally being refactored! I've got two requests:

• A proper matrix showing who has marked what, like John Isner proposed here
• Rubric based aseessment marking, with the potential for more than one rubric to be used at once

Can't wait to see how it turns out.

David, bless you I use workshop extensively in graduate school. Most of the problems you are addressing. But one that has not been touched on is having flexibility as to when the teacher's examples will be available for students to assess. The current program has the student's evaluating the examples before they do their submission. I would like the examples to be unavailable until after their submissions. I would even like to require them to assess teacher's example before they assess student submissions. I have numerous example that I use with law students. Thanks.

Hi David,

just a little note to say how much I appreciate all your work on workshop 2.0! I don't understand everything on the specification page; everything I do understand though seems absolutely fine for my needs and I cannot wait to test the module with my students.

I wish you strength, patience and fun while coding the module.

Thanks again,

Stephan

David here is a screenshot of the unmodified specimen assessment form used in a work on in law school.

Here is a screenshot of a speciman form that has been modified to include detail information. This information can't be given ahead of time because it essentially gives the student the answer to the problem. On the other hand, They need the details  for assessment in order to assess submissions accurately. I would like to have some solution that allows me to hide the specimen assessment form if I chose. I can provide the general assessment form as a link in the body of the course. Right now the work around solution is time consuming. Because I have to go back and modify every specimen assessment form after submissions but before assessments.

Hi David

Good to see that this module is being revised. I'm new to Moodle, but have spent a lot of time recently shifting one of my courses onto it, and thus far I've only got one request - I hope a simple one!

I teach AQA A level Sociology, and whilst the marking scheme for essays is complex for students, its been broken it down into something that should be easier for students to both understand and grade. They end up with a different total of marks from the AQA scheme, but it does get them to assess on the criteria of knowledge, analysis, evaluation etc. Each of these is unpacked to several sub-sections and a mark given to each - this means that I would need 25 Dimensions/Criteria etc, rather than the 20 at present.

Is this possible?

Further, would it be possible to alter the current workshop module in-house (meaning our college - given a sufficient knowledge of PHP and SQL) to accomodate this change?

best

Bob

Hi David (et all),

I've been reading the specs last week and here there are some thoughts, numbered for easier reference:

E1 - BIG Thanks for your effort! Workshop is one of those great tools that MUST be there.

E2 - Also, great specs! They really help to "peers" to review, comment and discuss about the whole thing.

E3 - Along the specs I've seen than then submission/assessment grades are weighted 80%/20% to get the final grade. Are those weights immutable? Is there any magic in that number? If not, couldn't have sense to allow configuration of those weights on each Workshop? I think it's possible to plan activities where one teacher want to add more weight to the assessment or the opposite.

E4- All the assessment grading (automatic) is performed based in one entity called the "best" assessment (1 per criteria I guess), and divergence to that "best" is used to perform grading, clever and cool! And that best (or bests if multiple) assessment is calculated as the assessment nearer to the mean. Is that rule to calculate the best assessment there because it's the way old workshop used to work? Or is also some sort of "magic" best calculation? Similarly to my previous point I think it could interesting to have some alternatives in order to get calculated those best assessments. For example, on of the more obvious, for me, is that, in activities where the teacher is also performing assessments, it could be possible to define the best one (for ulterior calculations) as exactly the one introduced by the teacher. Just an example, but could have sense to have that feature. Or perhaps the "best" could be the "worst" and apply inverse distance measures, surely causing different - but also correct - grades. In any case, think about the "best" detection as something that could be calculated in different ways (I just have exposed 2).

E5- In any case, that "best" assessment... should be marked in the corresponding assessment/grades tables IMO, just for reference to be showed in reports or wherever necessary. Also that "mark" could help speeding-up recalculations and so on.

E6- I guess that grade calculations only happen at the end of the activity, when all the submissions and own/peer/teacher assessments have finished, correct? Then, one process performs all the calculations and generate submission/assessment grades and the "official" total to go to the gradebook. And, in that phase, is when teacher can override the final submission grade or particular assessment grades, correct?

E7 - If prev exposition was correct, then I also guess that the "best" assessment won't be influenced by any override, and will be calculated only using "original" assessment grades. Correct?

E8 - Another thing I think we must be really carefully is about settings and their changes. The beginning of each phase implies that some setting won't be modified any more and we must be hard (rigid) here. I remember the old workshop exhibiting problems with the change of settings/configuration once the assessment phase (examples or real) had started. Just a call to be careful with that.

E9 - I've found some setting in the workshop table that, perhaps, should be considered permissions instead of settings, things like "anonymity" or "hidegrades" have sounded me as capabilities, not settings.

E10 - Where in the DB structure is the final grade stored? In the submissions table there is one place for the grade (submission grade) and the gradinggrade (assessment grade) but I don't see the weighted final to be there. Not 100% necessary as you can push it straight to the gradebook... but personally I love to have those grades on each module.

E11 - In the assessments table, I've seen a bunch of fields (grade, gradinggrade...) defined as number 10... are they missing their decimals part? Or are they really integers? Same aparently in rubric_levels.

E12 - I think we should "name things in a official way" since day 0 (and everywhere: here when discussing, in docs, in php code and in DB) . I must confess that the "dimensions" concept caused me REAL troubles when trying to understand all the grading strategies, just to discover, laaaaater, that they are, simply, criteria, or grading elements. I must say that I thought that you had became crazy, talking about dimensions and spaces, and galaxies and all sort of math things in the specs! Just to discover they are simple criteria, grrr.

E13 - Also about "naming" there is another (internal) thing that always makes me to think twice, the "grade" word (used for submission grades) and the "gradinggrade" word (used for assessment grades), couldn't we name them that way (submission/assess grade) ? Specially the gradinggrade one causes my mind to round and round and round!

E14 - Finally, one "silly" thing... is there any reason to be so "upload-based" in the new workshop? Couldn't we add support for "online-based" or "url-based" workshops? i.e. it only means that the upload won't be necessary/but fulfilling the corresponding text field will. I think it isn't technically complex and can open the cool peer-grading to different types of assignments. Just a quick thought, surely teachers will know if that is an useful feature or no.

And basically that's all I can comment about the specs. I think they sound HIGHLY PROMISING and cannot say anything but: go, go, go!!!

Great work, David, sincere kudos4you, thanks! (muscle)

Ciao

PS: Sorry for the long post. I hope it helps and don't cause too much headaches to all you, dear moodle-peers.

Hi everyone!

About the change proposed to merge the current Criterion and Rubrics grading strategies (GS). Why do we need GS anyway? What about merge all GS (i.e., eliminate the concept of GS)?
IMHO, GS are a limitation, add more unnecessary concepts to be defined in the documentation, and it role can be replaced by adding scales.

For example, without GS we can define a single formula in order to calculate grades:

$$ \dfrac{\mu_{1}w_{1}g_{1}+...+\mu_{N}w_{N}g_{N}}{w_{1}+\ldots+w_{N}}$$

For the formula above:
Let $$w_{i} \in \mathbb{N}$$ be weight of ith, and g_i the evaluation given to the ith dimension (in terms of the respective scale; for example, if the first scale is Yes/No then $$g_1 \in \{0,1\}$$, if the second scale is Score out of 100 then $$g_2 \in \{0,1,2,...,99,100\}$$, etc.). Let $$\mu_{i}\in \{.8,.81,.82,...,1,...,1.18,1.19,1.2\}$$ represent a factor used by the assessor to adjust the suggested grades by up to 20% either way.

Also, it will be possible to have both subjective and objective dimensions in the same assessment form (i.e., some dimensions having quantitative grades and some other dimensions only with qualitative grades).

First of all David, thanks for taking on this project. The workshop has so much potential.

I have used the module many times but have had some problems in the past. If I were composing a wish list I would want the following:

1. Flexible and changeable due dates / peer editing dates. I have made the mistake more than once of using the wrong due date and tried to change it only to have the Workshop become tangled up. Setting dates in stone is a recipe for disaster. It would be great to change a due date as you see how long students actually need or to accommodate a student that has been absent.

2. Open editing: I would like students to have the chance to edit work up until I mark it. Half and hour of editing time is just not enough. Often students realize a mistake the next day and want to change it. If I have not marked the assignment I do not see why students cannot go in and edit it. It would be great if they could edit without having me delete their work manually for them. This is a bottleneck. I like to have students start the assignment at school, and then work on it at home. A button that indicated they were ready to have it marked would be great.

3. I would like to choose from rubric that I have already made. Re-typing and or tweaking rubrics each time is time consuming.

4. I would like to be able to save comments that I use often with the rubric so that I do not have to retype them for each class.

5. A major issue is where I can comment on a piece of writing. I really like the 'Marginalia' module (is it even a module?) which allows comments to be made at any given point in a piece of writing. Anyway, it is really important to be able to comment right where a mistake is made rather than copying and pasting.

6. The formatting (CSS) of a teacher created rubric needs to be redone in the new workshop module. Some better thought out CSS would go a long way. I would love to see a tabbed rubric which would help avoid laborious scrolling. Basically the rubric needs to be more compact.

7. The comment boxes need to be bigger or at least the last 'general comment' box needs to be bigger.

8. Bold, font, italics, and color need to be made available when making comments. The option to enlarge editing boxes like you can do elsewhere in Moodle would be great.

My apologies if any of these has already been covered. I do not have time right now to read through all the comments.

Best of luck David.

Steven.

Hi David,

I don't know if my following question is already answered elsewhere (I can't find it, sorry):

I want to leave a large group of students working alone in a workshop, and I want to participate (as teacher) only when peer assessments present a large dispersion, more specifically, when some student is having very different evaluations on his/her work.

Supposing that the situation is clear... Is it going to be any automatic mechanism to inform about this lack of consensus (so teacher can go and check on students)?

Thanks!
A.

David Thanks for your effort. I think all I was using in the "old" workshop module is included in your drfts for the 2.00 vers. Great!!!!

Thanks for your work. I'm using WORKSHOP until 3 years ago. I think that it's the most complet and complex tool of Moodle, but the worst is the Comparison of Assessments system: I think doesn't work. I've tried to design some diferent experiments, trying to change the assesments and the response of the system is impredible. I can't be sure of the final result.
In my institution we're working over the 1.94 version (I think...). Can I supose that with your new Workshop, this kind of problems will be finished?

Thanks and sorry for my bad English ()

I'm very new to Moodle, so apologies for any newbie ignorance I display - and also if my question has previously be addressed in this forum. I was very very pleased to discover the workshop module (in its current form) within moodle. It potentially meets a real need I have with my journalism students - enabling a tutorial group (typically around 25 students) to submit their attempt at writing a news report to a central repository where each student's work can be viewed and commented on by all the other students in that tutorial group. That is to say - the workshop repository would contain 25 news writing exercises - one from each student - and then each of these attempts at a news report would be available for all the other 24 students to comment on (so potentially 25 texts available for review/assessment by all the students in the tutorial group). However, it seems that the current version of Workshop doesn't actually permit this - as I understand the documentation, all I can do at the peer-review/assessment stage is arrange for each student to be "randomly" supplied with other studens' work for comment/assessment (with an upper limit of 20 for the number of pieces of work the student can randonly be supplied with for peer assessment). It seems that this same limitation is being planned for the new Workshop 2.0 - thus the Workshop 2.0 specification states:
"

1. Peer-assessment (optional)
   1. the module randomly selects a given number of submissions to be reviewed/commented/evaluated/assessed by student.

So I'm wondering if there's any chance that it be made possible in Workshop 2.0 for this "random selectivity" to be over-ridden. Thas is to say, would it be possible for there be a setting by which it becomes possible for all students in a given group to access and comment on/assess the work submitted by all the other members of the group. Thus once all work had been submitted to the Workshop, student 1 would have access to and be able to record comments on the the work of students 2 through 25, student 2 would have access and be able to record comments on the work of student 1 and students 3 through 25 - and so on.
Again, my apologies if I have misunderstood the documentation - it's just the Workshop comes so tantilizingly close to providing the functionality I need - but is potentially not nearly so useful if it "randomly selects" work for peer review.
best
Pedro

I have a few feature suggestions:

(1) I would like -- from the student's perspective -- for the instructor-provided examples to be indistingishable from the student examples. My concern is that students will put more effort into spoof assessments than peer assessments. But, if they cannot tell the difference, this problem diminishes.

(2) I currently use Forum for peer assessment. It's set up as the "Lakeview Journal of Sociology". Students not only gain from reviewing their peer's research reports, but they are also being socialized into professional practices. I'm looking forward to the added rigor and automation that Workshop 2.0 promises, but...I don't want to lose the public aspect of student-published work.

Will it be possible, in Workshop 2.0, for a selection of student work to be publicly visible to the entire class? Ideally, Workshop would select the top five scoring submissions to that particular assignment (or whatever number the instructor specifies) to publish as exemplars of first-rate work.

This gives students a social motivation to excel, which I find more productive than a points-based motivation. And my Moodle journal will function more as a real scientific journal, in which researchers compete to get published.

Hi David

Would it be possible to include customised grade schemes in the workshop, i.e. not 1-100 but maybe A - E or Pass, Merit, Distinction?

Thanks in advance

Doug