I am starting this thread to gather your feedback, comments, questions and ideas regarding the new project we are working on for Moodle 2.2. We would like to support Rubrics and other advanced grading methods consistently across all activity modules, not just for the Workshop as it is now. At the moment, the project is at the end of the planning phase and there is a proposed specification of the framework available at Advanced grading methods docs page.
We will appreciate if you find time to look at the spec to see if the suggested implementation works well for your needs. Of course I can't promise it will 100% work for 100% of the community. We would just like to make sure that we work on something actually usable for real teachers in real courses. The spec is full of technical details that you might want to simply ignore and focus on the usability, UI and overall logic of the specification. Such a feedback is valuable for us, too.
Unfortunately (for the project, not for me ) I'm leaving for annual holiday tomorrow and will not get back to this thread before September 19th. Sorry for the delay with my eventual responses. I hope this time period would give the community enough time to eventually discover all weak parts of the proposal. Thanks in advance for your time and replies.
I am interested in this from the UI side.
One of my sites is relatively simple, using just Quiz, Lesson and Questionnaire activities. I am starting to document Advanced file loading of assignments with the goal of eliminating paper versions. We should be using ruberics with these assignments but do not. This also got me thinking about other places to apply ruberics.
Send me a message if you need testing. I am placing the DEV page on my watch list.
David - this is very good!
I work with grading student papers and today we work with a rubric-like grading in a "manual" way. This is really a hot topic for me right now.
I can not give you any help on the programming but would love to test this with some of the students and give you feedback!
Like the others here, we are really excited about the prospect of rubrics being integrated throughout all gradeable activities.
Would be more than happy to do some beta testing when required.
Thanks for your hard work in this area - if it's as good as your re-designed Workshop module, it should be awesome.
Thanks everybody for your early feedback. Me and Marina are going to start with the development of this new feature.
It would be very helpful for us if we have some real usage examples from you now. Max Stenlund mentioned they are work with rubric in manual way. Max, can you please describe how your typical rubric looks like (number of columns and what values are assigned to them), how you calculate the total score and how you convert it to a single grade to be stored in Moodle? Note there is a section in the spec discussing this at http://docs.moodle.org/dev/Advanced_grading_methods#Rubric Are you OK with the proposed behaviour? What about the columns with the zero value? And is it clear that a minimal possible rubric score maps to grade "0" in Moodle?
Thanks in advance.
The number of columns varies depending on assigment but its usually 5-7. One column is sometimes used for the weight , the factor which the value should be multiplied with. If there is no weight factor the values varies like 1,2,34 or 2,4,8,12. If something is left out the score is 0. Besides the weight factor we sometimes use the specs are fine. Just remember that some cells might dont have a description, just beeing "between"-scores. For instance could a row have five cells but only the top,middle and lowest have a description. If the teacher really thinks its somewhere between then the option is there to do so.
All the values are then added together. The total score is never 0 in our case but its make sense its 0 in moodle.
The real treat would be if teachers (and students in some cases) could grade the same made assignment to cross validate the rubric. I could explore this furhter if you want...
If something is left out the score is 0
This is essential information for me as it is a bit different from what the spec claims at the moment. So in your case, the student would get the grade 0 if and only if the teacher left the rubric completely empty, right?
Thanks for the tip on using empty descriptions for the "between" scores.
How would you deal with the situation when you have a rubric that gives you score from a range like 10 - 63 and you were asked to convert it to a grade out of, say, 10?
The support for rubric cross-validation or even peer assessment is out of scope at the moment. However the infrastructure should be able to face such requirement in the future.
Thanks for your contribution Max.
To leave out a grade of a criteria may be considered that the total grade is not complete. It would make sense for my anyway.
Its a little bit tricky because some criteria in the rubric could crucial for the total grade, failing to get a grade on some criteria could be considered failing the whole assignment?
But it depends also how you make the transition to the total grade which you also ask about.
We (Consider we are doing this manually today) use the rubric like this: The student makes a written assignment and we grade it using a Rubric. A total score is calculated.
We put min-max score for each grade. This scores could be different of course depending on the nature of assignment. We use Bologna system which is grade A,B,C,D,E,F,Fx= 7-grade scale. A top grade like A means a full score with maybe a lesser score on some items with lower weight.
The "min-max" value is depending on different factors and the only way for us is to have table where this values are put in. It could be calculated of course but the range for different grades varies depending on which criterias on how they are weighted.
So, two ways are possible(?): a calculated total transfered to grades by some sort of rule or a table in which max-min values are stipulated for different grades. In the later case the final grade could be ordinal scale. (Im not sure if this is what you mean in the spec?)
My suggestion for some kind of cross validation origins from the fact that using rubrics scale with subjective judgements always leads to discussion about how the rubric are built up and used. If two or more teachers can compare their assessement before the grade is finalized it would be a valuable tool that not only would do grading more transparent and more fair but also help make grading a collaborative task between teachers which enhances teachers skills.
I will give all feedback you need from my perspective. I have possibilty to test things on experimental moodle site if you you want some feedback on how it works.
I've been experimenting with rubrics quite a bit and am really excited to see that there are plans to incorporate them in moodle.
Here are some suggestions:
I like working with Rubistar: it gives you enough space to give students comment nog only on what's wrong, but also on what they can do about it and what they keep in mind.
Very often I use rubrics when students do a presentation and very often they do presentations in groups. So what I would really be interested in is a rubric in which you have an individual part and a group part (content is usually a group grade). And more importantly, it would be really great if you could switch between rubrics (eg. student A is talking, then student B, then student A again...) without losing data.
And a final suggestion: it would be nice if the rubrics when finished could be saved in a sort of portfolio (maybe Mahara?) so that students can take these forms with them to the next year.
I have no idea whether these suggestions are technically possible, but I hope they will be helpfull. I would also be willing to test out a beta-version once it is available... Just let me know!
'Very often I use rubrics when students do a presentation and very often they do presentations in groups. So what I would really be interested in is a rubric in which you have an individual part and a group part (content is usually a group grade). And more importantly, it would be really great if you could switch between rubrics (eg. student A is talking, then student B, then student A again...) without losing data.'
This would be really great.
Also, the option to fill in a comment for the student with each rubric, would be great.
But, how is the work going?
More feedback wanted
Firstly, thanks everybody for your valuable comments and suggestions. Our development team just started with the implementation of the scaffold for the framework and we are still discussing some design aspect in parallel. For some questions we did not find clear answers yet and we would like to hear your opinion, should you find a minute and think about it.
A1. Half of the development team (read: David ) thinks that the rubric can't be considered as filled unless the rater chooses a cell in each row. In other words, the teacher must explicitly select one cell in each rubric criteria/row. If the rubric designer is used to use empty rows as giving the score "0" they will have to define such cell explicitly. The other point of view suggested by Marina is that we could allow some rubric rows being unanswered (and counted as "0" implicitly) once the teacher submits the whole grading form.
A2. Should it be possible to alter rubric texts after the rubric was released (ie enabled to use) and some assessments were actually filled? One use case is that the teacher can spot a typo or misleading wording in the rubric and wants to fix it. On the other side, as a matter of principle, no assessment form should be altered once there were some assessment using it done. Even when the change may look as a cosmetic re-wording for the rubric editor, the meaning of the new description can be different for another rater who has already used the rubric for assessment. So the chance is that the rater who chose the previous description would not choose the new one. Note that this applies to a rubric within a single activity (like one Assignment). If the same rubric is re-used later for another activity, it starts as a editable copy by design (on contrary to how scales behave now).
My opinion on this is:
A1) All the rubric criteria should be explicitly required. If anyone wants to allow a "0" then they can explicitly include a "0" value in the scale for that criteria. If they don't, then don't. Moodle won't need to care or do any guessing.
A2) Yes, I think it's going to be common that faults in rubrics will be discovered after grading is started, and teachers must be allowed to fix the rubric. We can warn the teacher about the effects and use a "dirty" flag to indicate which graded items need to be checked afterwards.
I agree with Martin.
I also want to ask about how rubrics can be stored and re-used - I guess its like the other grades?
Is it possible to see some of repository where rubrics can be exchanged? That would be really neat. That I could look into if there is no plan for that....
I'm very excited and encouraged to see the progress that has been made with rubric development
The Lightwork application currently creates 2 types of rubrics, holistic rubrics (which have criteria, levels and weightings as you describe) and analytic rubrics (which provide a more traditional mark/criteria rubric). These rubric definitions and marking sheets(what you call grading instances) are stored as xml and we use xslt to create PDF documents of the rubric and marking sheets for the students.
Our aim is to include Lightwork in the Moodle 2.2 core code and since rubrics are fundamental to Lightwork it would be great if we could work together so that our rubrics are compatible. In this way teachers could choose to work with rubrics created either in Lightwork or using your interface. We could agree to a common XML schema and use the xslt stylesheets that we have already developed.
Lightwork has a different way of storing rubrics than your proposed model. We have tables, MDL_LW_RUBRIC and MDL_LW_MARKING. These tables use an xmltext field for the rubric with additional fields that link to the activity and marker and define statuses such as "ready for release". This allows us great flexibility in designing rubrics since we are not constrained by the database model.
We have found it to be very common for teachers to want to make changes after "releasing" the rubric so:
A2) Yes, I think it is important to allow this. In fact, we have even had teachers who want to create the rubric after starting marking!
A1) Yes, All criteria should be filled. If they wish to put a score of 0 then they should do it explicitly.
Sorry there is no user documentation yet but that is on our todo list.
I tried this out today in school where we have IE8 and when I went in as a teacher to mark an assignment I couldn't click into the levels at all. I tried to switch to compatablity mode (that is, IE7 I guess) and for the first criterion I had a radio button which I could click - but then it just vanished; the page refreshed and it vanished as did any other radio buttons for my other criteria -so I couldn't use it. At home in Firefox it worked fine. (But my school, whenever we get Moodle2 - we use IE!)
Can you not get the School's IT people to add FireFox?
Also when you are in IE8 - to check that you are in the correct "compatibility mode" try clicking Ctrl+F5 (Control Key plus Function (F5) Key) this splits the screen and allows you to view the source code & CSS for the page you are on, a little like FireBug, it also lists the compatibility view either IE7 or IE8 etc.,
No they won't add FF because (apparently loads of our stuff works best on IE) But I have got FF on a USB pen portable apps so can use that if needs be. I just clicked the icon (what looks like a broken icon) to get to compatiblity mode. I didn't know about that split screen thing - I will check tomorrow