This topic is the place to discuss the ongoing development of Ray's new module, provisionally called the "workshop" module.
This follows on from the old discussion in another forum.
Well after a bit of a delay here's the Workshop module or it give it it's full title the Workshop Assignment Module. It basically the assignment module with extras really.
The attachment is the beta test version. If you want to try it here's the drill - move the zip file to the Moodle directory (you need a reasonally recent version of Moodle something like 1.0.8 or greater), unzip will put files into the correct directories, login as the Moodle Admin and go to the Admin page (on the way the module will set up its tables in the database and do other magical things).
When creating a workshop assignment have a look at the help files behind the yellow help buttons. The module's got lots of options which give very different flavours of assignments! I very much doubt whether I've tested all the combinations
Any suggestions, comments for improvements would be much appreciated.
The attachment is the beta test version. If you want to try it here's the drill - move the zip file to the Moodle directory (you need a reasonally recent version of Moodle something like 1.0.8 or greater), unzip will put files into the correct directories, login as the Moodle Admin and go to the Admin page (on the way the module will set up its tables in the database and do other magical things).
When creating a workshop assignment have a look at the help files behind the yellow help buttons. The module's got lots of options which give very different flavours of assignments! I very much doubt whether I've tested all the combinations
Any suggestions, comments for improvements would be much appreciated.
Thanks Ray!
I'm looking forward to trying this out.
Sean K
I'm looking forward to trying this out.
Sean K
Wow, Ray, looks great, but it's a mindbender! I'm having a play with it now. (People trying it should make sure they have site "debug" turned off, by the way).
The main question in my mind is: does this more or less subsume the peer-graded assignment? It looks like it does but I've not used either in a real situation.
I'd rather not include 'pgassignment' within the 1.0.9 download if 'workshop' can replace it (or will eventually), just to avoid upgrade-path confusion for teachers. It might make more sense to offer pga as an optional download for those who really want it. (A side benefit is my "add" menu would look neat again. 
)
The main question in my mind is: does this more or less subsume the peer-graded assignment? It looks like it does but I've not used either in a real situation.
I'd rather not include 'pgassignment' within the 1.0.9 download if 'workshop' can replace it (or will eventually), just to avoid upgrade-path confusion for teachers. It might make more sense to offer pga as an optional download for those who really want it. (A side benefit is my "add" menu would look neat again.
I'm going to set up an example "workshop" here in this course so we can run through it together as students to get a feel for it.
Yes I agree it's better to drop the peer-graded assignment module at 1.0.9. Having got the workshop module out I can see that it needs two extra options, a couple of extra fields in two tables, a bit more code (whoopee!) and then the peer-graded module is dust.
The options are "Are student assessments to be agreed" and "Show grade before agreement". With the first turned on only assessments which are accepted by the originator of the assignment are "counted" and students are allowed to loop while doing an assessment. And you think the current module is mind-blowing
It's odd why I didn't add that before pushing it out but there you go
The mark 2 Workshop module should be about a week.
The options are "Are student assessments to be agreed" and "Show grade before agreement". With the first turned on only assessments which are accepted by the originator of the assignment are "counted" and students are allowed to loop while doing an assessment. And you think the current module is mind-blowing
It's odd why I didn't add that before pushing it out but there you go
The mark 2 Workshop module should be about a week.
Fantastic!
Let me know if/when you want to start working via CVS so the rest of us can help with tweaks and initialising variables etc.
Unrelated, but workshop has a nice sound about it ... workshop workshop workshop workshop workshop ...
Let me know if/when you want to start working via CVS so the rest of us can help with tweaks and initialising variables etc.
Unrelated, but workshop has a nice sound about it ... workshop workshop workshop workshop workshop ...
ha ha .... I like it as well Workshop Workshop Workshop Workshop ... having play around with it today and got the latest news from Ray I told myself ....Lucky me I only translated ten lines in PG Assignment (lol)...
Another handful job for translation, before doing so I probably will have to read the overview of this module once again of what it can do. Too many options to explore in a few hrs..
but excellent job!!
Yes, you are very lucky!, I on the other hand have translated the whole " pgassigment " documentation 
.
So I will wait to see if finally it is integrated in the Workshop, before beginning to translate everything again 
Regards,
Emmanuelle
The Workshop Module has been enhanced so that during the peer assessment stage the two students involved, the one who submitted the work and the one doing the assessment, can enter into a (semi-private) dialogue and hopefully reach agreement on the assessment. The assessor is also given the opportunity to change their mind and re-assess the work. (And the teacher can chip in when things get out of hand!) This is termed reasonably enough "Assessments must be agreed" and it's an option. The option is OFF by default which means the students just do the (peer) assessments and that's it (no loop )
There's another twist (i.e. option) which is in the Peer graded assessment module which has also been added to the Workshop. It means, I think, the peer graded module has been subsumed by this new module. (Sounds painful
)
This version is in cvs and there are db changes so if you do update you will need to bounce into the admin page as the admin user to get the old tables amended and a new table added. I've also fixed a couple of bugs in the first version and created many more... only kidding
There's another twist (i.e. option) which is in the Peer graded assessment module which has also been added to the Workshop. It means, I think, the peer graded module has been subsumed by this new module. (Sounds painful
This version is in cvs and there are db changes so if you do update you will need to bounce into the admin page as the admin user to get the old tables amended and a new table added. I've also fixed a couple of bugs in the first version and created many more... only kidding
Oh, I meant to say that the module is now quite happy to run with debug turned ON. Also there was a bug in the mysql.sql file (Sorry Eloy) but that's been now fixed in CVS.
What's needed now is more flexibility in the management of a workshop assignment. I'm adding a "admin" section so that the teacher can recover from the "unexpected"
The teacher needs to be able to delete submissions and assessments, re-assess and re-grade and generally feel "in control". As the module stands it's too fixed in time from the teacher's point of view. Watch the sky for the next installment
What's needed now is more flexibility in the management of a workshop assignment. I'm adding a "admin" section so that the teacher can recover from the "unexpected"
One of my phantasies about Moodle is to have inside a week/topic-activity-box, a horizontal row with 4 or 5 steps, a kind of mini-workflow. (With checkmarks under the icons of each step for "passed this step")
- I was wondering how much this workshop module could cover that phantasy..
(I only miss the visualisation, the rest is already there..) - I also wonder -because this will cover more then one week - how I can repeat the announcement of this activity in another week-box
- During the peerreview you can open the submission in a pop-up window. Nice, but if I go back to the assessment questions, it disappears under the carpet, clicking again does not bring it back on top. (Students/pupils need here a little bit of ICT-skills: or can you create a version with split windows? one for the submission, one for the questions?)
Another wish: having a RUBRICS criteria-list to give the students for peer-reviewing must also be possible inside this modul... Reading the documentation more close... IT IS ALREADY THERE!!
- I can bring in a Rubric line, only wondering about the layout in the descriptionfiled of each element in comparison to the final scale
- I wonder how to bring-in a complete CSV formatted Rubrics (next release?)
So a MORE THEN GREAT job Ray, this makes Moodle even more the head of the herd!
1. The teacher's view of the workshop module did have a "passed this step" approach with neat right and left pointing arrows showing where the workshop had got to. However, I dropped that for the simpler "just show the people what they need" approach. There just seemed to be too much on the screen. The student view is more "in your face" with the four steps - assess some examples, submit something, do some peer assessments and here's some of your work accessed - not quite phasing into the picture smoothly. That's mainly because after the first stage and the student's submitted something, the resubmission, peer assessment and "agree to assessments" actions just happen, they're not sequential. That said the student view could be made a bit more "logical" possibly.
2. Yes, in a course structured by weeks and the assignment running over several weeks you almost want symbolic links in the subsequent weeks. The links all point to the same assignment but could be labeled (slightly) differently. If that is wanted it could be implemented through a new module called something like "link" which just creates an entry which points to an existing activity. Has that been requested/done?
3. Ah, the power of windows. Under Windows there's the "Tile Windows" option which imho makes all the windows equally unusable in one simple stroke The problem here is that the submissions could be anything. If you look at the Jokes even those are different "shapes", there are some which are horizontal, some are vertical. I'm afraid it's back to the Task bar
Finally that's an excellent point about rubics. They do need to be "re-usable" and probably a import/export feature (via xml) is the way to go.
Thanks for this
2. Yes, in a course structured by weeks and the assignment running over several weeks you almost want symbolic links in the subsequent weeks. The links all point to the same assignment but could be labeled (slightly) differently. If that is wanted it could be implemented through a new module called something like "link" which just creates an entry which points to an existing activity. Has that been requested/done?
3. Ah, the power of windows. Under Windows there's the "Tile Windows" option which imho makes all the windows equally unusable in one simple stroke
Finally that's an excellent point about rubics. They do need to be "re-usable" and probably a import/export feature (via xml) is the way to go.
Thanks for this
I've added the teacher admin features now. They are under the teacher's link "List all submissions". Teachers can now delete individual items and re-assess submissions.
This feature breaks the anonymity of the module as the admin lists contain real student names. I thought that is necessary especially when assessments are being deleted. In normal operation the module is intrinsically anonymous by design (apart from the teacher's listing of the final grades). In practise it can be run non-anonymously by getting the students to label their submissions (adding their name to the titles) and to "sign" their assessments in the general comment box. I'm NOT planning to implement an anonymous option which switches names ON and OFF. (Unless there's overwelming public demand of course
)
The new version is in cvs, I've also added some of the missing strings
This feature breaks the anonymity of the module as the admin lists contain real student names. I thought that is necessary especially when assessments are being deleted. In normal operation the module is intrinsically anonymous by design (apart from the teacher's listing of the final grades). In practise it can be run non-anonymously by getting the students to label their submissions (adding their name to the titles) and to "sign" their assessments in the general comment box. I'm NOT planning to implement an anonymous option which switches names ON and OFF. (Unless there's overwelming public demand of course
The new version is in cvs, I've also added some of the missing strings
Looking good - I did some grading of the jokes this afternoon and was starting to enjoy the flow of the thing. I've got a few requests, though. I know I could do them myself but I'm enjoying being a consumer for a change :
a) I'd like that option to make it non-anonymous throughout. At least for the teacher. I can't see a reason for the teacher not to know who is doing what, but I do feel it can be important to learn further about each student.
b) n2 comments is a lot of clicking for a teacher for any reasonable n. One place to remove a click is after a submit (you could use an auto-redirect like the forum postings rather than a continue link).
c) When I'm grading student comments, it appears that I (as teacher) can also update their comments (as well as writing my own). Is that intentional?
a) I'd like that option to make it non-anonymous throughout.
b) n2 comments is a lot of clicking for a teacher for any reasonable n. One place to remove a click is after a submit (you could use an auto-redirect like the forum postings rather than a continue link).
c) When I'm grading student comments, it appears that I (as teacher) can also update their comments (as well as writing my own). Is that intentional?
Ah yes, the foot's on the other boot now eh!
a) To Anonymouse or Not To Anonymouse that's the question. Yes, certainly the new teacher admin stuff is non-anonymous (there must be a better word
) So to be consistent everything the TEACHER sees should be that-word-again(!)
b) Yes, you deserve a medal doing all that grading
Normally I expect the teacher NOT to grade student assessments when there are even small classes. Provided the students do three or more peer assessments the module tries to "grade" the students by looking across the assessments. (Actually to get sensible results it probably needs more than three assessments from each student.) It does need a bit of field testing to see how it performs and if it does indeed produce sensible figures. I will, however, think about how the grading page could be lengthened to reduce the click factor.
c) That sounds like a bug. A student can update their own assessment provided it hasn't been agreed. A teacher can add an additional comment to an assessment which has not been agreed but they shouldn't be able to change a student's actual assessment. I'll look at that
Now I've taken part as a student, I would like to know how my submission faired against the other submissions and in particular which one "won the prize". Where did mine come and where did the ones I rated come? As it stands the module does not tell the students that and I don't see any reason why it shouldn't. The teacher is shown the final grades from a student point of view, the students could be shown the final grades from a submission view of point, a submission league table.
The difference between the two is that the former includes the students' grading grades while the submission table ignores all that nonsense and tells us what we want to know - the best joke - sorry Martin and all those clicks too
a) To Anonymouse or Not To Anonymouse that's the question. Yes, certainly the new teacher admin stuff is non-anonymous (there must be a better word
b) Yes, you deserve a medal doing all that grading
c) That sounds like a bug. A student can update their own assessment provided it hasn't been agreed. A teacher can add an additional comment to an assessment which has not been agreed but they shouldn't be able to change a student's actual assessment. I'll look at that
Now I've taken part as a student, I would like to know how my submission faired against the other submissions and in particular which one "won the prize". Where did mine come and where did the ones I rated come? As it stands the module does not tell the students that and I don't see any reason why it shouldn't. The teacher is shown the final grades from a student point of view, the students could be shown the final grades from a submission view of point, a submission league table.
The difference between the two is that the former includes the students' grading grades while the submission table ignores all that nonsense and tells us what we want to know - the best joke - sorry Martin and all those clicks too
Is there a way to hide grades from other students in this module? I like the submission league table at the end, but I have had many requests from my students to please not publish their grades, and I would like to comply...? any ideas?
Thankyou
David Hoare
Thankyou
David Hoare
I'm adding anonyminity (if that's the word ) from the student viewpoint. That will use the flag that's already there. It may eventually be three levels (show names, hide names from students, hide from names students and teachers) but I'll do the first two for now.
Again there is a "league table" flag and I'll use that for number of entries in the table (from 0 to 20, and all submissions). There's been a request not to show the weakest submissions.
Ray
Again there is a "league table" flag and I'll use that for number of entries in the table (from 0 to 20, and all submissions). There's been a request not to show the weakest submissions.
Ray
I've had a look at (c) but I'm not seeing the problem 
When I, as teacher, grade an assessment I do see the grades within the assessment as drop down boxes but any changes to these are not saved. All I see is the Teacher's comment box and the teacher drop down grade both of which are saved. The only button I see is the one at the bottom labelled "Save my Grading". Do you see any other submit buttons in your "Grade Assessment" screen?
Sorry, Ray ... I must have gotten a little mixed up there between the grading form (with form widgets) and the comment form. I can't find again the screen I thought I'd been seeing. All's well - forget about c). 
I love the idea of a ranked list of results to add a little competition into the mix. It would have to be (another) option since I imagine it could be counter-productive (or at least counter-cultural) in some situations. Maybe this is becoming the Delphi module.
We gotta get these rubrics into Moodle-core so they work in all of the modules.
Non-anonymous --> named? labelled? tracked? transparent? non-private? orwellian?
I love the idea of a ranked list of results to add a little competition into the mix. It would have to be (another) option since I imagine it could be counter-productive (or at least counter-cultural) in some situations. Maybe this is becoming the Delphi module.
We gotta get these rubrics into Moodle-core so they work in all of the modules.
Non-anonymous --> named? labelled? tracked? transparent? non-private? orwellian?
"forget about c)" - yes I know the feeling, this is not the easiest thing in the world to test. But in real life things are complicated...
Yes, the ranked list of results should be an option but it need not be one on the already crowded creation screen. That option is best handled when the teacher does the calculation of the final grades. There's a preliminary screen at that stage which could hold this option.
Rubrics? what rubrics? Not that funny little cube I was absolutely hopeless at. Oh no, it's not coming back is it?....
I thought "named" for non-anonymous. Ah, great minds and all that...
Yes, the ranked list of results should be an option but it need not be one on the already crowded creation screen. That option is best handled when the teacher does the calculation of the final grades. There's a preliminary screen at that stage which could hold this option.
Rubrics? what rubrics? Not that funny little cube I was absolutely hopeless at. Oh no, it's not coming back is it?....
I thought "named" for non-anonymous. Ah, great minds and all that...
O.K. Suppose, you as a teacher hate m.c-tests and switch to open assignments...
How will you grade these? In the weekend, 10 o'clock in the evening your visiting family (suprise, suprise!) is finally gone and you can start to mark 20 open assignments, all different in sizes and choosen direction.... Yes open assignment, you harvest your own decision.
- How do most of us at that time grade these 20 products? You start with the first one, skip, skip, three bright ideas, one red cross, gets a high mark...
................. At last number 20: you have seen already all great ideas (AND BETTER) this kid delivers in the other 19, this kid gets just a pass, while his work could even be better then the first one - The better teachers under us make first a criterialist and then start the grading, this is in favour of number 20
- People In Cananda, Alaska, Kansas, etc... make an art of this criteria-list-approach and call this RUBRICS
Just do this mental exercise:
- Think of the last open assignment you gave to your students
- Think of the resources and exercises you gave them
- Then describe the qualities you would expect in a open assignment (piece of work) in this special situation if you would mark it with good
- No, you are not ready, do it again for an excelent piece of work (same qualities, higer level..
- Yes, as you expected now for a fail
- These Rubrics people go to five levels
- Given the students also these criterialist will prevent that they go in all directions. (Unless you just want that...)
If you plan to use Moodle I predict that you will feel the need for a Rubrics.
The good news is that there are so many already on the internet, HOW CAN WE READ THEM IN A WORKSHOP-CRITERIALIST?
Ah, I'm beginning to see what Rubrics are. (In UK Universities we use the term for the do and don'ts that go on the front of exam papers ) They are definitely two dimensional. Doing a quick look around a couple of sites, some rubrics just go for levels (leaving the translation of level to grade up to the teacher?). I assume these attempt to give the item being assessed a single overall level. HOWEVER other "rubrics" seem more flexible and define a table where an accumlative score pops out the bottom. These do still have a "level" orientation but allow for differing levels for different aspects of the assessment. (Perhaps all rubrics allow this?)
My initial thoughts are that rubrics a) contain a lot of text, b) are two dimensional and c) the second type of rubric doesn't fit either the accumlative or criteria grading strategies as currently implemented in the workshop module.
You ask how do you read them into the workshop module, does this imply that they are "out there" in machine readable format? My quick browse only found html or pdf rubrics which means typing
The module will also need a "rubric grading strategy" to accommodate the structure of the second type of rubric.
My initial thoughts are that rubrics a) contain a lot of text, b) are two dimensional and c) the second type of rubric doesn't fit either the accumlative or criteria grading strategies as currently implemented in the workshop module.
You ask how do you read them into the workshop module, does this imply that they are "out there" in machine readable format? My quick browse only found html or pdf rubrics which means typing
The module will also need a "rubric grading strategy" to accommodate the structure of the second type of rubric.
I've not read much nor do I know much about rubrics, but when I mentioned it above I actually just meant what you already have in Workshop, ie "Comments, Assessment Elements, Grade Bands or Criteria Statements"
I think that your last remark is important: the module needs also a "rubric grading strategy"
A Rubrics-scheme looks like a two dimensional table, but that is not true.
- You are talking about qualities, so its a set of ordinal scales.
- Inside each scale you describe relativ positions, so in the middle in one scale says hardly anything about in the middle in another:
- So summarizing line-positions to an end-total is not correct
I see it more as an attempt to make interobserver realibilty less insecure, when working with open assignments or observable behavior. Looking from different perspectives, building a more complete and more fair impression.
Your other observation is also correct: There are very much different rubrics-table formats
I don't remember if I gave you a set of examples, so maybe again: https://elo1.hetstedelijk.org/rubricsColumbia.html
The Rubrics with 5-columns are a little overdone?
- I like for simplicity this table-look-a-like, in your design your set of peerreview-scales.
- I prefer constructing a 1-2-3 scale for teachers
- then construct a 1-column-student-advice-sheet from those 3)
The product Rubistar organizes this kind of tables on a server (php, mySQL?)
They don't know if they should make their product commercial or Open Source.
I beg them to make it Open Source, and let them earn money by selling courses: training people how to make GOOD rubrics..
How could we use Rubistar: http://rubistar.4teachers.org/
- look for a subject/topic in the store (sometimes very specific)
- look at the scales that others did create
- improve the one that you like the most
- put that back on the server as a new template for others (I like that approach)
- then IMPORT THE SET OF SCALES (mySQL-format?) into the workshop in Moodle
I know of two situations at University Twente were professors use ranked lists:
- One professor places a winnerscup behind the best result, that does boost
- In another project they use it more complex:: To get students better prepared at the entrance of a project, they get BEFORE the project a project related assignment, only the persons that pass the pre-assignment are allowed to see the highest ranked pre-assignment. That also boosts (we were supprised..)
>>We gotta get these rubrics into Moodle-core so they work in all of the modules>>
Any plans for this? I dodn't know whether to post here or in the 'Assignment' forum - but I am dying to use the flexibility of the workshop module's marking strategy with non-uploaded files or simple assignments.
Anyone know of a way to do this? or is there a way to run the workshop module without a submitted file - ie. just for teacher evaluation of student work using the flexible scales, etc... (I recognize this defeats much of the purpose... but)
Any plans for this? I dodn't know whether to post here or in the 'Assignment' forum - but I am dying to use the flexibility of the workshop module's marking strategy with non-uploaded files or simple assignments.
Anyone know of a way to do this? or is there a way to run the workshop module without a submitted file - ie. just for teacher evaluation of student work using the flexible scales, etc... (I recognize this defeats much of the purpose... but
Hi,
I just downloaded and installed Moodle as well as the Workshop mod.
The chat mod installed fine but I can't get Workshop to works. I've upload the most recent version of it to the mod folder, I uploaded the worhshop.php file and its help files to the lang folder but when I try to add a workshop as an activity of a week inside a course, I can't see its link.
Did I miss something?
Thanks in advance,
Will
Well.. I solve it the hard way: I insert a record into the modules table, ran the mysql.sql of workshop (changing the prefix accordanlly.).
If there is anything I should do and I ddidn't, please, let me know.
Thanks!
Will
Possibly, after setting up the files for the module (by unzipping them) you should login as the Admin user and view the Admin page.
This gets Moodle to create the tables and other table entries used by the Workshop module. As the Workshop is an "add-on" these tables are NOT set up when you first install Moodle.
This gets Moodle to create the tables and other table entries used by the Workshop module. As the Workshop is an "add-on" these tables are NOT set up when you first install Moodle.
Thanks for your reply...
Yes.. Perhaps I didn't enter to the admin zone... I though the changes were commited from wherever place Moodle finds new mods.
By the way... This program... bah, this platform is TERRIFIC! Congratulations to all those involved in the developing process...
Kindest regards,
Will
Hello Ray,
I would like to congratulate you on your module. It seems to fill a real need in large courses where reading and grading assignments can become quite a task. Your module looks like a similar system to Calibrated Peer Review (http://cpr.molsci.ucla.edu/). Is this somewhat correct? In the UCLA system students first of all, make a submission of their own work. Then they are required to read and evaluate a criterion piece which has been pre-evaluated by the instructor. Once their scoring is within tolerable limits they are allowed to evaluate a subset of their peers. A different subset evaluate their work. The end result in an assignment like a lab report, is that everyone a) sees what a good report should look like, b) learns to identify a good report, c) evaluates their peers (e.g. 3 other people) and d) is evaluated by three other people. Their lab mark would be some combination of their own accuracy in rating the criterion piece and their average mark from the three students who evaluated them.
The rubrics discussion was very interesting and the links to other sites were very informational. You have incorporated the rubric concept well into your module. I guess my question is whether your module as it stands could be set up to do something like Calibrated Peer Review.
Rick Danielson
No, I haven't heard of the UCLA sytem, the peer assessment model I was working to is based on an internal system here which has been used successfully for a couple of years. (It's main problem was its "stand-alone" nature.)
The Workshop module can be used in a similar way. Students submit work, it gets peer assessed, the teacher grades and comments on the assessments, the students possibly(?) resubmit. But it does lack the "test" assessment before the students do the peer assessments.
Personally I find this business of submitting work and then assessing like putting the cart before the horse. In the Workshop module an assignment can be set up where the students are required to ACCESS some examples of work before SUBMITTING their own work. I would have thought it's better to get the students into the assignment (and how it's graded) by doing the assessments first.
It works in this way. The teacher submits a number of example/model pieces of works before the assignment is opened up to the students. In fact, in an early version of the module I did build in the constraint that these (pre-submission) assessments had to "pass". That is, the teacher had to go in, look at the assessments and give them a grade, if that grade was less then a threshold level the student was required to re-assess the example. In that version the students were required to "Pass" all their assessments before being allowed to submit their own work.
That all sounds great but in practice I found three problems - (a) the teacher had to do a lot of assessing and is a brake on the students submitting work; (b) the students were confused by the teacher's grade, thinking it was a grade on the example rather than a "grading grade" - a bit too subtle for my students I'm afraid; and (c) where do you set the threshold level which said the assessment was a pass, to make it "flexible" it would have to be yet another parameter to the Workshop module.
So I removed the requirement to "pass" the pre-submission assessments. The teacher can still assess them and they are graded. Now, however, all the students see is the teacher's comments on their assessment, the (confusing) grade is hidden. (although those grades can, if the teacher wants, contribute something to the students' final grades.) Further, the "brake" has gone, once the students have done the assessments they can submit their own work, the teacher is NOT required to go in and grade these assessments.
I think in a way that answers your question. I suppose doing it the UCLA way round the students are more clued into the assignment once they have submitted their own work and can then make better "judges". But what I am trying to do is to make the SUBMISSIONS better. By getting to students to look at some examples, evaluate them and build on those examples their work will hopefully be of a higher standard.In the UCLA model the students may well recognise their own initial submissions are poor but do they go on to improve them? (The Workshop module allows resubmissions if that's what the teacher wants.) Also if all the students see is other STUDENTS' work it's a bit like the blind leading the blind... In most of my assignment I'm more interested in the quality of the work rather than the student's ability to see quality.
The UCLA model is submit -> assess an example -> teacher grades assessment -> peer assess (I think). Does the student get an opportunity to resubmit? Where do the final grades come from?
What's really needed is more experience of running these sort of novel assignments, that's for the future...
Ray
The Workshop module can be used in a similar way. Students submit work, it gets peer assessed, the teacher grades and comments on the assessments, the students possibly(?) resubmit. But it does lack the "test" assessment before the students do the peer assessments.
Personally I find this business of submitting work and then assessing like putting the cart before the horse. In the Workshop module an assignment can be set up where the students are required to ACCESS some examples of work before SUBMITTING their own work. I would have thought it's better to get the students into the assignment (and how it's graded) by doing the assessments first.
It works in this way. The teacher submits a number of example/model pieces of works before the assignment is opened up to the students. In fact, in an early version of the module I did build in the constraint that these (pre-submission) assessments had to "pass". That is, the teacher had to go in, look at the assessments and give them a grade, if that grade was less then a threshold level the student was required to re-assess the example. In that version the students were required to "Pass" all their assessments before being allowed to submit their own work.
That all sounds great but in practice I found three problems - (a) the teacher had to do a lot of assessing and is a brake on the students submitting work; (b) the students were confused by the teacher's grade, thinking it was a grade on the example rather than a "grading grade" - a bit too subtle for my students I'm afraid; and (c) where do you set the threshold level which said the assessment was a pass, to make it "flexible" it would have to be yet another parameter to the Workshop module.
So I removed the requirement to "pass" the pre-submission assessments. The teacher can still assess them and they are graded. Now, however, all the students see is the teacher's comments on their assessment, the (confusing) grade is hidden. (although those grades can, if the teacher wants, contribute something to the students' final grades.) Further, the "brake" has gone, once the students have done the assessments they can submit their own work, the teacher is NOT required to go in and grade these assessments.
I think in a way that answers your question. I suppose doing it the UCLA way round the students are more clued into the assignment once they have submitted their own work and can then make better "judges". But what I am trying to do is to make the SUBMISSIONS better. By getting to students to look at some examples, evaluate them and build on those examples their work will hopefully be of a higher standard.In the UCLA model the students may well recognise their own initial submissions are poor but do they go on to improve them? (The Workshop module allows resubmissions if that's what the teacher wants.) Also if all the students see is other STUDENTS' work it's a bit like the blind leading the blind... In most of my assignment I'm more interested in the quality of the work rather than the student's ability to see quality.
The UCLA model is submit -> assess an example -> teacher grades assessment -> peer assess (I think). Does the student get an opportunity to resubmit? Where do the final grades come from?
What's really needed is more experience of running these sort of novel assignments, that's for the future...
Ray
Thanks for that detailed answer. The reason I asked the question was that the Workshop module looks to be a good way to do lab reports. I teach a course in motor learning and it uses a problem based learning methodology. The "labs" are done by providing textual answers to a number of questions. The authors of my course textbook give some model examples for the questions, and students are invited to go over these submissions before submitting their own work. It looks fairly similar to the way you have set up this model.
Last year I had a TA do the marking for the labs and it was fairly successful. However I think the students may benefit more from doing some assessments of other students' submissions. That is why I was thinking of using the calibrated peer review (CPR) software. I haven't actually used it, so can't offer any experienced feedback as to how well it works. However, since the Workshop module is so close to CPR system, and since it runs on one's own server (as compared to CPR which is housed on the UCLA server) I think the Workshop module is the way to go.
Personally I find this business of submitting work and then assessing like putting the cart before the horse. Good point. As noted above, I really don't know whether the trial assessment results in a better attempt the next time. Your explanation of how seeing exemplary work can help students to do better work themselves is quite logical.
Does the student get an opportunity to resubmit? Where do the final grades come from? I don't think they get an opportunity to resubmit their assignment. The final grades come from an average of evaluations done by their student peers. The number of evaluations seems to be under the control of the professor. My understanding of the CPR system is that everyone is randomly assigned to evaluating a certain number of other students and their work is in turn, evaluated by the same number of randomly selected peers. The main point of the CPR system seems to be that students are not allowed to grade others until they have demonstrated that their own evaluations are reliable and valid as determined by how close they come to the criterion evaluation grades of the exemplar material previously submitted by the professor. Once they get to a certain standard, using the rubrics of the evaluation protocol, they are assigned to evaluate other students. At the end of the evaluation period, their grades are calculated on the basis of how well they graded the exemplary material and how their peers graded their own submitted material.
In September I will have 80 students submitting lab reports once per week. It would be really helpful to have student participation in this grading process. Theoretically, it should be possible to get objective, reliable and valid grading from students grading each other. In addition, they will learn from reading other submissions. That's what I am attempting to achieve using the Workshop module. One tricky phase of this process is to get the students to buy into the accuracy of other students' assessment of their work. The CPR approach seemed to be one way to do that. From your explanation, it seems that your students did not have a problem with this scenario. If that's the case, maybe mine won't either. However, they have never agreed with me in the past.... 
Rick
Just an idea I've had rattling around that I'm sort of filing here before it falls out my ear, is that I think the main confusion with Workshop lies in the sequence of stages and understanding where one is a any point of time.
The old Peer Assignment had a more graphical approach which was useful, but the best interface I can think of for Workshop is a tabbed interface, with tabs along the top representing the stages.
The current tab would be highlighted, and under it would be a box containing the currently available optons (and perhaps even a short text explaining what was going on).
The old Peer Assignment had a more graphical approach which was useful, but the best interface I can think of for Workshop is a tabbed interface, with tabs along the top representing the stages.
The current tab would be highlighted, and under it would be a box containing the currently available optons (and perhaps even a short text explaining what was going on).
I hear what you say and I want to revise the teacher's interface to the management of the module. Within each stage there's also a need to make a clearer distinction between the Actions and the Admin functions (which are growing).
Any pointers to a neat tab toolkit would be welcome. (I had a quick look in weblib but nothing jumped out at me.)
Any pointers to a neat tab toolkit would be welcome. (I had a quick look in weblib but nothing jumped out at me.)
The teacher's screen is now tabbed. It's not as pretty as I would have liked but it had to work across themes and it does NOT add new images... to the existing themes. The routine workshop_print_tabbed_heading($tabs) which contains the code is in workshop/lib.php.
I'm also added more information for the teacher about who did what. This mainly comes from the new "Administration" screen but other screens are more informative. I'm planning to take this through to the student screens and add an "Anonymous" option so that student names can be turned "off".
I'm also added more information for the teacher about who did what. This mainly comes from the new "Administration" screen but other screens are more informative. I'm planning to take this through to the student screens and add an "Anonymous" option so that student names can be turned "off".
Absolutely excellent! Looking in good shape for 1.1!
I've added Rubrics (what a funny word!) to the Workshop module now. Somewhere in this thread I saw the number five associated with rubrics so that's the spec I worked to
I don't know the proper rubric terminology, I'm afraid but here's what rubrics mean in the workshop module. In the assignment you can specify a given number of "categories" (probably a better term is topics?), within each category there is up to five descriptions. So when an assessment is made, within each category you choose the description which best fits the piece of work. The descriptions have the "scores" 0, 1, 2, 3 and 4 (if there are five of them). The categories can have weights (even negative weights
) and the grade is simply a weighted sum of the scores. The categories do NOT have to be balanced in the sense that they do not have to all have the same number of descriptions. The maximum number is five (should it be more?) but some categories can have two, some three...
I think the rubrics are fairly straigtforward to set up (if the category has four descriptions you simply leave the last box empty) and pretty easy to use in assessment terms. The assessor can add both a general comment and a comment against each category.
If someone could advise on the proper terminology I'd be most grateful.
I've also got round to adding a "league table". This appears at the end of assignment and lists the student submissions in grade order, the winner at the top. The list also gives access to the teacher's and the students' assessments, not sure that's too wise but it's there at the moment!
ps to martin, I think I now stopped stepping on your foot in cvs (workshop/asssessment.phppp)
I don't know the proper rubric terminology, I'm afraid but here's what rubrics mean in the workshop module. In the assignment you can specify a given number of "categories" (probably a better term is topics?), within each category there is up to five descriptions. So when an assessment is made, within each category you choose the description which best fits the piece of work. The descriptions have the "scores" 0, 1, 2, 3 and 4 (if there are five of them). The categories can have weights (even negative weights
I think the rubrics are fairly straigtforward to set up (if the category has four descriptions you simply leave the last box empty) and pretty easy to use in assessment terms. The assessor can add both a general comment and a comment against each category.
If someone could advise on the proper terminology I'd be most grateful.
I've also got round to adding a "league table". This appears at the end of assignment and lists the student submissions in grade order, the winner at the top. The list also gives access to the teacher's and the students' assessments, not sure that's too wise but it's there at the moment!
ps to martin, I think I now stopped stepping on your foot in cvs (workshop/asssessment.phppp)
Hi,
I'm trying to create a course to teach moodle to students (based on Sean Keog's) and now is the turn for the workshop module..
Well.. I think I create it correctly... I uploaded (as a teacher), three sample assigments and I've grade them...
Then I got out of the system and loged again as a student... Tried to access the workshop and I still get the "not available" message...
What am I missing?
Thanks in advance,
Will
PS: if you want access to my course, just let me know.
Forget it... I set the workshop to phase 2 and everything is ok now (hopefully!).
However, I think this step should be more explicit in the help files (or it is and I miss it?)...