Hey Tim!  

Thanks for reviewing.  I appreciate the comments, and I'll try to respond to some of them.  First of all, I just posted this spec over the last couple of weeks, and I still have some cleanup, and some additional screens that I need to write requirements around.  Secondly, this is just the functional spec.  At Moodlerooms, we generally write full functional specifications, review them with our clients and end users, and then move on to technical specifications.  I'll be doing a formal review of the functional spec with ten or twelve end users, probably in about two weeks.  I expect to have a few changes, but I've already been socializing this with many of them, and the feedback has been positive.  Kris Stokking and Mark Nielsen will be doing the tech specificaiton.  I believe that they've started thinking about it, but I don't think that they have posted it.  I'll follow up with them, and we'll find a place to host it.  

Okay, so, let me hit your questions or comments in order: 

#1,#2 - Per my note, this is still pre-tech design.  I'll leave that to people who are smarter than I am.  (I'm a developer, but I haven't been developing for several years.) 

#3 - I have some research on this, which I'll dig up tomorrow and include in the spec.  However, I have a bit more work to be done here.  A couple of broad concepts that I am working under.  1) Plugin System - By making this a pluggable system, we allow people to have different methods of completion for different types of programs.   2) We will probably only build the simplest ones by default.  I'm thinking of just one based on % correct and maybe one based on streaks.  3) All of these are suggestive, rather than conclusive.  One screen which I am currently mocking up is the Instructor Marking screen.  They can choose one of the completion metrics, which will suggest which Outcomes have been achieved, but they can manually mark them or unmark them as well.  

#4 - There's still a bit of work here (see #3).  On the quiz stuff, we should clearly chat sometime in the next couple of weeks. 

#5 - In answer to your specific question, I'd personally like to see the outcome as a property of the question itself, and independent of the quiz on which it is attempted.  Mapping questions is a time consuming process, and including it on the question will encourage re-use of the question.  In my research on how outcomes/standards are used in most professional schools and primary/secondary institutions, a well-written question is typically very granular, and maps against one outcome, and likely couldn't be mapped against a second outcome.  However, we won't do anything to prohibit mapping against multiple outcomes. 

#6 - Martin and I have talked about this at a high level, and I agree, this needs to tie into badges.  I will defer technical implementation details to the tech team and MD.  I think we all agree on the goals, however. 

#7 - Yes.  If that is not clear, I will update on my next revision. 

#8 - No.  Haven't thought about it, but I will think about it this week, and update, if necessary. 

#9 - Gradebook - More work needed here for sure.  I can share that very few clients are using outcomes as they are implemented today.  We did some data mining on client usage.  Of our approximately 900 clients, only about 25 are actually using outcomes.  So, while I want data to migrate for those clients, the current implementation is not meeting the needs.  In my spec review focus group in a couple of weeks, that seems like a good topic to probe with those few that are using the functionality. 

#10 - Scales - So far, of the 20 or 30 people who have reviewed this specification (including our training and implementation team, who have been training people on how to use gradebook, scales, and outcomes), not one has asked that we maintain the dependency on specs.  I'll wait for more people to chime in, but I don't know how we can leave the dependency on scales, unless we were to make "Scales" one of the achievement plugins (mentioned in spec and above).  

Thanks again, and thanks for starting the discussion.  I'll keep working, and we'll hope that other people start getting involved as well! 

Phill

Tim, thanks for all of your feedback on the proposed Outcomes system!  As Phill has stated, it is clearly a work that is still in progress.  We have lots of ideas on how we could implement certain requirements, but nothing has been fully defined enough to share yet.  We won't be breaking ground on development before the technical architecture has been drafted and generally agreed upon.  

Our primary objective is to create a system that can be powerful enough to meet the demands of the most requested features, while remaining flexible for developers to extend and implement in their own creative ways.  It is not our intention to define a white-list of plugins that can support Outcomes, and the way(s) in which they do so.  While the specification may identify a specific use case for associating outcomes to quizzes and rubrics, our technical team will need to abstract that as a way to allow outcome association at a more granular level per module and advanced grading method, using quizzes and rubrics as a specific implementation example.

Let me also try to address the questions you've raised:

1.  The DB schema will be part of the technical documentation.  It makes more sense to me to hold off on it until we've addressed the requirements further, but we can craft something if that would be helpful for the conversation.

2.  There's a balance that must be struck between creating new plugin types vs. the overhead of maintaining and configuring them. The 2 primary areas for plugins would be the reports (piggybacking off of the existing system) and the algorithm for marking whether a student has met the outcome (aka Streaks).  There may be other areas that would be useful, and they will likely come out in discussion.

3.  I would agree that Streaks should be a pluggable system - for those who attended, this would be the same system that we discussed during the Outcomes session at Hackfest.  We need a little more definition around Risk Metrics to determine whether it should be pluggable system or can be addressed through configuration.

5.  Obviously we'll defer to your judgement here, but couldn't we technically have both?  From the feedback we have received, it's pretty critical that the outcome is associated with a question in isolation, but that could simply be used to facilitate the outcomes associated with the question in a particular quiz.  I could see it being useful to allow the instructor to override that in some cases, preferably based on a capability to do so.

9.  The new Outcomes system would operate independently from the Gradebook.  I personally don't think it makes sense to mix Outcomes and Grades on the same report, especially since the Outcomes could be organized very differently in the new system.  It's possible that the Outcome Marking plugin could create a bridge in some cases, but that still needs to be defined.  

I freely admit that much of the Moodle-speak in what I have looked at here goes straight over my head and that might be why others tend not to comment.  However here are some comments from a purely educational practitioner viewpoint:

1. it would be good to see a less USA-centric description toencourage development to be flexible.  I realise it is easier to speak of what you know but please do not neglect the non-US market as there are many of us!
2. a big part of my concerns regarding outcomes (as they are now) is the viewing of those and their grading and a better gradebook view of outcomes would help a lot (but see also 3)
3. I use outcomes all the time and my use of them may refelect that of others.  I do not care whether students pass an assignment/quiz/activity at all!  What matters is if they hit the outcomes.  Admittedly the one normally implies the other but it is a very different emphasis.  It makes no difference to students (in terms of qualification) how good their work is just whether they hit the outcomes.  This viewpoint should be reflected in an ability to track outcomes rather than grades.  Activities might be just marked complete/not complete (or percentage) as currently but the outcomes can be seen clearly as met or not met.
4. results should be clearly visible at least course-wide so outcomes in the whole unit/moduel can be seen in one place.  Ideally it would also be possible for a student to see outcomes across Moodle courses as well (for those of us who use a separate course for each unit/module ina  qualification).  I am also aware of some units/modules which are taught by more than one person and I guess some might use separate Moodle courses and yet the outcomes are only relevant as a whole.  I suspect you would not want to include this in this particular project as it would presumably be best done as a separate plug-in.
5. Have you considered what will happen if one outcome is met across multiple Moodle activities?  Often an outcome might say something like "Show examples of dealing with clients in ..." or it might say "Design, create and test ..." and again that naturally exists as three activities or more.  This is not hypothetical as I have already had to split outcomes including one into 8 separate outcomes (even though it is one official criterion).   For me the answer is simple - each outcome should have a configurable number of "mets" and for it to be completed it would have to be met in that many activities.  Students complete an activity and the outcome can be shown as 1/3 met or a percentage.  This simplistic approach may well not work for others so a one-to-many use of sub-outcomes might be unavoidable

Overall this seems to me to be a potential monster so well done for tackling it!

I am aware of PHP (but not OO PHP) and databases so if I can be of any use in opinion or testing please let me know.  The way Moodle works behind the scenes is something I have not looked at though so I may not be much use!  Hope this helps.

David, 

Great comment.  I'll update the spec to make it more general.  In our research, the rubric is by far the most commonly used Adavanced Grading Method, although we also have Checklist, etc.  I'll see about adding these into the spec.  I suppose the best approach is to build outcome support into the Advanced Grading framework, and then builders of Advanced Grading methods could use outcomes in their methods.  Does that sound right? 

Phill

As a Moodle Admin and ComSci teacher at an international school that is exploring a move to Standards Based Instruction and Reporting (I'm using the US parlance because I am most familiar with it - apologies to those who must translate), I am delighted beyond words that all you good people are taking on this project.  I had been considering taking a stab at it myself, but I can see from the above and the specs that it is in far better hands.

I do have a couple of thoughts/questions, and I apologize ahead of time if I missed these being addressed already in the spec or above.  These are primarily reporting (module/plug-in) specific, but I thought I would mention them because they may have some implications for data capture and storage design.

I saw in the specs that the intent is to provide for hierarchical Outcome structures.  If I understand you correctly, this would be wonderful.  One of the major limitations of the current Outcomes schema is that it is difficult to know which "level" of the standards hierarchy to track.  Using Outcomes to track the 4-6 "Strands" (in my terminology a Strand aggregates several Standards which themselves aggregate many Benchmarks) that would go on a report card.  From a feedback perspective (assessment for learning and not simply assessment of learning) using Outcomes to track Strands or Standards provides essentially no helpful data.  To be helpful for learning, the Outcomes need to track Benchmarks (specific learning/performance targets), and at the high school level in some courses (e.g. ComSci) these are legion.  In the specs it is clear that you intend to have Outcomes tied to Benchmarks, but will tying an Outcome to a Benchmark automatically tie it to the Standard and Strand of which it is a component?  I am assuming so, but do want to raise the issue.  Doing so would certainly help teachers in schools that continue to give grades (whether simple A, B, C..., or 1, 2, 3...with descriptors, or some other system) help in deciding to what degree a student has met the overall Strand/Standard requirements of the course.

In the same vein, will there be a way to assign a "weight" to each instance where a question, rubric evaluation, etc. is tied to a Benchmark, and each Benchmark is tied to a Standard, and each Standard is ties to an ultimate report card Strand?  If a student has met a Benchmark 10 times over the course of a semester, will each success be weighted equally even though they arise from different assessment modalities (e.g. multiple choice questions, essay questions, performance assessment).  Will there be a way to weight the reporting data in terms of where an assessment item falls in the learning process (most recent performance vs early performance)?  Will it be possible to aggregate performance data for Benchmarks by mean, mode, median?

Again, I realize these are primarily report plug-in related questions.  But, if the right data is captured, I can envision some wonderful graphical reporting tools that would make Standards Based Instruction and Reporting both productive and a pleasure.

Finally, any remote sense of what your target for release would be on this?

Cheering you all the way,

Doug

Folks, 

As you can see in the thread, our technical guys have made huge progress on the tech spec side, and we are getting closer to really starting the coding.  During the tech review, we uncovered some possible issues, and so we have refined the functional specification with a number of changes.  Some of these were do to technical constraints (reporting against Completion, rather than against activity, so as not to have performance problems on the logs), and others were do to actually working through some use cases technically (after a review of a lot of the various state standards, their XML, and the nesting requirements, we made some changes to the format of an outcome set and how they are nested).  I still have a few more changes, which I'm hoping to get in before the Developer meeting tonight, but I thought I would post now, so that anyone who is going to the developer meeting tonight could have time to review before then, if they are interested.  

http://docs.moodle.org/dev/Outcomes_Specification_Change_Log

Thanks! 

Phill

Hi Kris,

Good to see you folks moving ahead with speed, but I must also say I am concerned about two thing I see in the Outcomes Technical Specification: 1) Outcomes no longer appearing in the gradebook, and 2) Outcomes no longer supporting scales.  Perhaps I don't understand where this is going, but it seems likely to me that these two changes will make the Outcomes module virtually unusable for K-12 schools in the U.S. 

The move is on in a very, very big way toward Standards Based Reporting and Assessment in the U.S. but the ususal approach is to assess and report student achievement on a scale - typically something like: 1 Not Present; 2 Area of Concern; 3 Progressing; 4 Mastered.  I don't see how a pass/fail approach to data collection can support this model, nor how pass/fail can be used to meaningfully report student performance on essays or other performance based assessments.  Also, in the end the teacher's assessment of a student's  performance on such a scale (perhaps by looking at the mean, mode, median, most recent or a combination of these) IS the grade in the gradebook.  I'm not seeing how outcome tracking can be separated from the gradebook.

I am attaching a short article by Thomas Guskey (one of the leaders in SBG in the U.S.) which provides a basic example of how Standards Based Reporting and Assessment is being implemented in U.S. schools.

I realize that standards or criteria based grading is being implemented differently around the world, but would hope the new Outcomes module will not exclude K-12 teachers and schools in the U.S.

All the best,

Doug

We've made a couple of notable improvements to the technical design which bear mention.  

1.  Instead of mapping the outcome/content association to a course, we map it directly to an activity.  We realized we were making it harder on ourselves by giving the system the flexibility to map to anything in the course.  While that might be nice in theory, in actuality a user's performance will be tied to Activity Completion and Grades, concepts that are fundamental to activities but not other plugins/objects.  This works out quite nicely since questions and advanced grading criteria both relate to an activity (when implemented), and we get a major bonus of identifying the content item's relevant context within a course - it's no longer Question A in Course B.  It's Question A, part of the Final Exam (or Practice Exam) in Course B.  The Functional Specs are currently being updated to reflect this.

2.  Outcome attempts now capture additional data points for mingrade, maxgrade, and rawgrade.  This isn't a major change, we are simply updating the spec to capture what we said we would to make it more useful for instructors and recommendation plugins.

You can see the specific changes made by clicking on the following link: http://docs.moodle.org/dev/index.php?title=Outcomes_Technical_Specification&action=historysubmit&diff=37983&oldid=37930

Howdy folks,

The technical documentation has been updated with some clarifications and refinements.  The main highlights though are that the tables have been renamed for clarity and an ERD like diagram has been added to show the relations between the new outcome tables and core tables.

Cheers, Mark

Wow, lots of great questions in here.  First, let me say that I appreciate the feedback.  Development is progressing quite nicely, but we still have a ways to go.  Let me hit some of the points that you make, and try to answer questions.  Some of these questions are already addressed in the spec, so if you haven't given it a read yet, please check it out.  

1.  Outcomes are new, as are badges, so we haven't integrated them yet, but there is a clear connection here.  On the grade side, this is actually possible with the design that we are pursuing, depending on how you grade.  For example, if you have an assignment with a rubric tied to it, that rubric could be tied to outcomes (as criteria).  So, when you grade it via the rubric, both outcome achievement and grades themselves are actually updated.  Right now, there is no design for how the aggregate of outcomes could translate directly to a grade without going through an activity, although this is an interesting idea. 

2.  So, multiple activities can be tied to the same Outcome, and the results are aggregated into reports, both at the course level, and (eventually) at the Administrator or Assessment Coordinator level.  

3. No plans right now for multiple graders, although we are looking at the grading workflow with some of the core developers right now.  Stay tuned. 

4. Interesting idea.  We haven't gone down this path.  I agree with your comment about University criteria being less granular.  However, the Outcome Sets are a heirarchy, which means that there could be high level objectives, and then you could create more granular objectives that lead up to the higher level objectives.  

5.  Rubrics can include Outcomes, and the measurement rolls up into the reports. 

6. Great idea.  We don't have this now, but we are looking at how you grade forums with Core, and there might be progress here soon.  

Hope that this all makes sense!  

Phill

I, too, am disappointed that there's not more to work with regarding Outcomes, Competencies, Badges, or whatever we end up calling the system of how we keep track of who's doing what.  What can I do to help.

Hi Kimberly,

Damyon Wiese from Moodle HQ gave me the following answer yesterday by email about whether or not this feature will be in Moodle 2.9 :

Hi Jean-Philippe, 

The short (and not very helpful) answer is that we have not yet decided exactly what we will do with outcomes for 2.9. As soon as we do we will post about it in the dev forums and the future major features forums, so I would just watch one of those places for updates.

Regards, Damyon

So no progress right now but if it goes on, you're at the right place to know

Look at

Competency Based Education

If you need a tool right now in Moodle 2.7, 2.8 or 2.9 look at Skills repository plugin here:

https://github.com/jfruitet/moodle_referentiel_m27

Skills repository ("referentiel") is a Moodle module for skill certification.

You can:

• Teachers specify a Competency Framework (skills repository) or import it
• Teachers create Skills repository activities in courses (it's a Moodle module)
  
• Students declare activities linked with competencies
  
• Teachers follow students declarations, comment, evaluate competencies with a binary (default) or a multivalued scale
  
• Teachers propose tasks (i.e: a mission, a dead line, a list of competencies to prove, a documentation, etc.)
• Teachers export and print certificates
  
• If your site enables Outcomes (also known as Competencies, Goals, Standards or Criteria), you can now export a list of Outcomes from referentiel module then grade things using that scale (forum, database, assigments, etc.) throughout the site.
  Then these evaluations will be integrated in the Referentiel instance of the course.