Competencies

Some quick thoughts.

I think that removing outcomes from gradebook is good idea, so only grades are there.So an assignment / quiz whatever, can issue a a grade and have competencies attached - but they are reported on differently but with the course.

I think we need to allow competency frameworks at category and course level, not just site level. Current use could have the Variations of the same framework (core and industry specific) used in 2 different courses - you would end up with a wide variety at site level which could be confusing.

Will the import handle the standard 3 level frameworks of Domain, Competency, Behaviour ? 

Are you sticking with the language of competency, sub competency - nearly all the frameworks I have seen use Domain/ Category, Competency, Behaviour wording which may make more sense to those used to using frameworks already.

Will you be adding a visualisation for the framework - for example in a competency self assessment tool ->

Damyon,

CBE is my "ultimate goal" for what I teach. This has been my goal for years, and I am now at the point of being ready to implement it next school year. In fact, your post is extremely timely. I look forward to working with you on this project and am offering my help if you would like.

Reading through your current docs, I found one thing that I think should be philosophically re-examined . It says"

"A competency is a unit of learning. All competencies belong to a competency framework, and it is likely that a competency framework will be defined by an external body such as an academic standards board."

Please don't assume this as a default position and create coding within moodle that would require an external board or give an external board supreme authority to govern what the "standards" ought to be. 

As I see it, the reason CBE works in the first place is because each teacher is able to define what the standards ought to be for his or her own classroom. I am not opposed to standards, but I am afraid of standards being created which cannot be overridden by the local teacher. Currently, from what I see in your docs, the teacher can still create the competencies. All I am saying is that I would like the teacher to always be able to override and/or create new competencies, in addition to competencies that might be "pre-made" by some academic board. 

Next week I am starting to remake my class to include CBE, so your posting is right on time. In fact, I would like to work with you on testing it out. Is there any chance we could correspond and work together on it?

Thanks.

More thoughts.

I think that competencies being a unit of learning is too defined.

If the system is going to hold professional frameworks, has to be multi-layer, but not necessarily a competency equating to a unit of learning but to include performance.

Take example of a framework where:

One Domain/Category  is Professional Practice

One Competency in that is Practises patient centred care

The behaviours belonging to the competency could be "Acts as a patient advocate to ensure that patient care is not jeopardised"

So when someone is being assessed on a submission, or whatever, they would be rated on the behaviour which add up to a competency as such. 

(example from PSI Core Competency Framework for Pharmacists http://www.thepsi.ie/gns/pharmacy-practice/core-competency-framework.aspx )

However you would find similar aspects in many other professional frameworks.

Also - the word proficient - just as the scale for different levels can be simple met/not met, complete/not complete, they can be complex with 5 levels. equally what the framework refers to as being proficient should be customisable per framework such as competent instead, for example.

Regarding the structure, Others can use  Domain, Standard, Criteria  

Perhaps to avoid confusion the labelling should also be configurable per framework instead of just competency, sub-competency as an institution dealing with 2 different frameworks with different wording describing each layer could have confusion.

Hello everyone,
This is just to inform you that in addition to this forum, there is a survey available in English and in French.
English version: https://docs.google.com/forms/d/1T4ScNUsWSqKkA4-cVdXpO4R5dneaQ1vgcTtqnH-bCJM/viewform?usp=send_form
French version: https://docs.google.com/forms/d/1uJWklifC5xDO1FSk-j3bp9Dsezyfpgi4kWFAWM9p_I0/viewform?usp=send_form
Your answers will be much appreciated.
Thanks,
Allison.

Damyon, 

Sorry to be late to this, but it has been a busy time.  I must admit that, as the guy who wrote the specification for the original re-do of Outcomes (https://docs.moodle.org/dev/Outcomes_Specification), I'm very confused as to why we are going to re-write it again.  I think that some of the stuff that you have added (Learning Plans for example) are brilliant, and necessary, but most of the framework stuff is actually already done in the Outcomes 2 work that we did a couple of years ago.  The code is already here (https://tracker.moodle.org/browse/MDL-40230?jql=text%20~%20%22outcomes%202%22).  

We've got dozens of clients using it, and for K12 institutions in the United States, it already has all 50 states of standards including the common core available for it.  We've gotten great feedback on how it has been going.  It seems like if you use this as a starting point, you would be able to get to the "further actions required" more quickly, and make it a much better tool.  It has already been peer reviewed, we have test plans, it has gone through QA and everything.  This is open source at its best!!! 

Okay, on to the feedback on this specification in particular.  I have a unique perspective since we hosted a bunch of focus groups and have done a lot of user research out Outcomes and competency based education.  First bit: we should not use scales, or at least should make them optional.  Very, very few institutions really use scales, and the use of them is not standardized, which makes reporting on them much more difficult, and in the end, the reporting is one of the most important bits.  

You have to be able to to link competencies to quiz questions, or it is essentially a no-go for courses that have any type of scale at all.  There has to be a way to do auto grading, and you have to be able to say that someone got it right or wrong without doing each one individually.  Getting down to the question level (or the rubric row level) is a MUST.  We've done that in the Outcomes 2 code, and we have gotten really good feedback.  

Finally, I think that the way that you set up permissions and default roles will be critical, especially around Learning Plans.  This is one of those areas that will likely be managed by someone who is not a typical Moodler, and may not even regularly use Moodle.  Perhaps the best way would be to integrate with external systems, where these competencies are already defined.  In Outcomes 2 (or as I may start calling it, Competency Based Education 1), we did this by supporting imports, but the permissions should really be extended to be more flexible, especially for people that don't typically use Moodle (Program Designers, etc.).  

As a side note, we here at Moodlerooms would really appreciate you building a migration from Outcomes 2 to whatever this becomes, if you decide to NOT use the Outcomes 2 code.  As I mentioned, we have dozens of clients using this already, and we also have gotten feedback from others in the community that they are using Outcomes 2.  It would be a real shame to make them go back on what they already did.  We also need to make sure that you get to feature parity with Outcomes 2, so that they don't lose functionality  (quiz questions being the most important).  

Phill

Hello,

It's very encouraging to read about the new developments with the competency framework - I think this work is reflective of the shifts in mindset towards assessment practices in education.

1.1 Summary

• “The goal is to implement a system that supports competency based education in a flexible way so that it can be used effectively by the many different target audiences.”

• suggest adding:

  • “The Framework for Competency-Based Education builds on the following fundamental beliefs that each course within the framework:

    • includes specific competencies which are course requirements;

    • provides learning opportunities to develop attributes, skills, and understanding; and

    • incorporates activities which are evaluated to determine achievement.”

1.1 Learning Plans

• suggest:

  • “Initially, they will all be marked “incomplete.”

    • add section from 1.4 Learning Plans: “There can also be multiple examples of “Evidence of prior learning” attached to each competency.”

• reasoning:

  • allows for the opportunity to represent the competencies in one course within a continuum of competencies in other courses

  • philosophically, mentioning this acknowledges that students in the course come with attributes, skills, and understandings and that the course will develop their competencies within the content and context of the course goals and rationale

1.2 Competencies

• “Note that the import operation can work like a merge, so if you are importing a newer version of a framework into a site that has an older version of the framework, the import will update existing competencies with the same ID number, instead of creating new competencies.”

  • question:

    • If someone is reviewing evidence of completed competencies, is there a way where they can see which version of the competency framework was being used at the time of evaluation?

1.4 User Learning Plans

• “A user learning plan is a collection of all the competencies they should achieve, optionally, before a certain date.”

  • suggest: “A user learning plans articulates the plan, the progress, and the evidence of achievement of the required course competencies.”

    • “The Plan: Draft stage

      • The draft stage represents the proposed plan a student will follow to develop the necessary attributes, skills, and understandings necessary to demonstrate achievement in the required competencies.”

    • “The Progress: Active stage

      • In the active stage, there is a listing of the scope and sequence of the selected learning activities which the student will progress through to develop the required competencies.”

    • “The Evidence: Completed stage

      • The completed tab includes the learning activities and evidence of achievement which the student has completed for the required course competencies.”

• “Competencies that exist in the template cannot be removed from the user’s learning plan.”

  • opinion:

    • important feature for credibility of course content and credentials/badges which acknowledge successful course completion

• “Competencies can be added to a user’s learning plan.”

  • opinion: this feature,

    • allows for students with accommodations to be able to outline a learning plan which aligns with course competencies

    • provides the opportunity for the articulation of additional competencies, such as behaviour competencies, to be included alongside curricular competencies without affecting course requirements

    • creates a space for students who are working towards “credit recovery” from a pre-requisite course to be able to include additional competencies

• “There can also be multiple examples of “Evidence of prior learning” attached to each competency.”

  • opinion:

    • important feature for apprenticeship/credit recovery programs

    • acknowledges diversity and supports inclusive learning environments

• “Whenever a comment is made against a learning plan approval request is accepted or rejected, the student will receive a message about the change.”

  • question:

    • is there a way multiple reviewers can make comments which are not visible to the student?

1.5 Course Competencies

• “When manually marking a competency, it is recorded as evidence… “Teacher X manually specified this value in this course…”

  • need to consider implications for credibility of course content and credentials/badges which acknowledge successful course completion

Outstanding Work

• I’ve developed a competency framework and learning plan for my research and I will develop an exemplar of what it would look like in a Moodle competency framework if this would be useful

The following spec was added :

"The taxonomy refers to the naming for each level in the framework. Up to 4 levels are supported, and the name for each level can be set from the following list: Domain, Competency, Behaviour, Indicator, Outcome, Level, Concept, Value, Practice, Skill, Proficiency"

• Would it not be more flexible and appropriate to just let the user customize his own competency level names?
  
• Why is naming limited to 4 levels? I don't think right now we have people using more than 4 levels naming but again, would it not make it a more flexible Framework to have "unlimited" level terms?

Thanks!

Hello Damyon

First congratulation for those specs which are a real work outcome. I'm exchanging for years with Jean around his Refrentiel module, and get somewhat really happy i can recognize some nice ideas and topics we were dicussing in France around Referentiel reshaping.

I guess the future outcoming implementation will be supporting quite widely th academic area of assement with a competency approach.

I am also working for a long time around Professional Development area in which we use very similar models. In those models appear IMO a new concept related to scaling the competencies. Jean initiated this approach using a "footprint" concept of the competency, that is the number of times it is verified (quantitative evaluation of success regularity) to really be marked as proficient. In the same trend f.e. the European Asociation of Cardiologists require from his members 100 electrocardiogram successful analysis to approve the "cardiogram diagnostic skill".

I read in some final posts of the thread that larger scales than "met, not met" might be used. Givin provision to use real scales could serve such situation.

In the same programs, we also face a regular model that needs three evaluation tracks to be managed :

- The first track asks the student to self evaluate the competency he thinks he is capable in (probably with more or less lucidity)

- The second track is the "expected level", so that decided by the Framework Administrator or the Learning Plan Administrator. Just tells the skill "level" or "status" required to get proficient in it.

- The last track is the "really assesed" competency evaluation, usually be teacher. This track would be prone to evolve from a first evaluation to a final state, tracking a time period in which students work for improving their stength. 

The purpose of introducing the self evaluation is of course to reinforce self-examination, self-diagnostic and motivation reinforcement, by clarifying the distance between reality and fantasmatic egocentric positionning,

Of course activating self evaluation track should be optional.

Do you think there is some room in your design for such a way to score competencies ?

Again, thanks for your spec freshness !

Hi everyone,

We have an important question about the competency framework model and we would like to have some feedbacks from the community on that.

Question : In your knowledge of competency-based education, is it possible that a framework has more than one unique scale? In other words, would it be necessary to be able to override the scale for a specific competency in a framework?

Your help would be much appreciated.

Thanks!

J-P

Incredible! Exactly what we in the k-12 field who are transitioning to a competency-based model need. Any idea on when it might become available?  

My next question is will 3rd parties or graders other than the teacher be able to mark competencies?

This would be an important feature to enable competencies to be used as site wide outcomes for departmental or institutional review and accreditation reporting. A new report, Trends in Learning Outcomes Assessment, was just released which indicates that very few higher ed institutions in the U.S. are using student learning outcomes collected via a learning management system even though 88% of baccalaureate have a common set of learning outcomes that apply to all students.

The CBE feature set should allow institutions to begin to assess student learning on a wider scale and in a much more timely manner, before it's too late for those that drop out. The report notes that it does not include very many community colleges which is where the innovation in CBE is becoming much more common.

We are very excited about the progress here, and would like to ask about a specific use case that I hope is met within the current spec.

Let us suppose we have a set of competencies pre-defined by a standards body, as described in your spec.

Let us then suppose that we have a course in which some students are enrolled with at least one instructor. Throughout the course, the students will be identifying competencies they propose to pursue within the defined subset available within that course.

1. Example: I'm a student, and in the third week of the course I'm supposed to hand in a proposal for a project in which I will address several sub-competencies selected from a couple of key competency areas. I go to my learning plan (competency framework) and indicate several of the subcompetencies I want to provide supporting evidence for. I hand in an assignment describing my plan as usual in Moodle, and the instructor approves the plan.
2. Important: a rubric needs to be defined at this point, on which my project will be assessed. Ideally, this rubric is automatically constructed from criteria attached to the subcompetencies I, the student, selected. If manually constructed, it will need to be tailored to each individual student, not a generic rubric attached to an assignment that is the same for all students.
   
3. I start my project, e.g. I convene a team in my workplace to solve a certain kind of problem and demonstrate team leadership skills.
   
4. I begin attaching evidence, e.g. minutes or progress from my team project, a survey my team conducts and its results, a video of me facilitating a meeting, a reflection journal, etc. (Question: how can this evidence be grouped together? In the spec, I see evidence being attached to subcompetencies, but what if I am simultaneously addressing several at once in my project?)
   
5. Eventually, I think I've met the requirements for my identified subcompetencies. I fill out a self-assessment against the rubric defined earlier.
   
6. I request an evaluation. This may be conducted by the instructor of my course, or by other faculty who specialize in assessment.
7. The faculty member uses the same rubric, created from the subcompetency criteria list, and provides a numerical score and individual feedback and suggestions. The faculty member may consider me "proficient" on some subcompetencies, but "emergent" or "incomplete" on others. My learning plan is updated accordingly.
   
8. As I progress through my program, I and my instructors can review the learning plan of my program and see which subcompetencies I have addressed and when, and see if I still need to address some.

When I viewed a Test Student profile as Admin user I can see Learning plans link. But I didn't get that link when I viewed as Teacher account. I already gave permission to moodle/competency:planview for Teacher role.

Could you please advise is there any permission I need to enable?

From the dev description:

Open questions for further feedback

• Would you expect a teacher to be able to create their own competencies (limited to their own course)? Would competencies that are visible only to a single course (and potentially conflicting with competencies from another course) be really useful?
  • Our institution would not normally find this helpful. Standardized competencies should be attached to courses and/or individual students (or possibly cohorts), but non-standard competencies are hard to interpret. Perhaps instructors could add a detail description to a standard competency within a course or individual learning plan? I think the idea here is to support IEPs (Individualized Education Plans) e.g. for Special Education in the US. It is common for these competencies to come in general from an existing list, but details (e.g. a particular level of success) may be specific to the student. If someone at another institution has a use case for this that is more clear, it would be good to provide a detailed description.
    
• Do we need to allow competencies to be linked to individual questions?
  • We don't expect to use quizzes to evaluate competencies in most cases at our institution, but if we did, we would need this feature. Imagine a student taking a placement exam. The exam should specify which competencies the student has already attained. Ideally the exam would draw randomly from a pool of questions intended to test each specific competency.
    
• Do we need to allow competencies to be linked to individual items on a rubric/marking guide?
  • This is critical to our use-case. A given competency should have, in addition to a scale, specific criteria to meet each level of the scale. These criteria vary per competency item.  A rubric needs to be assembled for an assignment (or learning plan) based on the competencies that assignment (or learning plan) purports to evaluate.
    
• How should self-evaluation of competencies work?
  • This is related to the question of "When to update the final rating for a competency." It should be possible for multiple evaluations, or ratings, to exist per student per competency, and how these are weighted should be dictated based on rules that take roles into account. Self, peer, and expert ratings need to be combined or displayed separately in some way. Rating aggregation functionality could be used to pick a median, maximum, or minimum rating from all ratings by a specific set of roles. Note that an "average" or mean is not a relevant aggregation method for ordinal data like scales. A decision of how to break ties needs to be made, as well (e.g. two "emerging" and two "proficient" ratings = what?)
    

Great work on the functionality! I've been trying out the frameworks since about a month ago before the repo was integrated into master and I'm seeing a lot of potential for our clients and even our own internal education efforts.

I have seen many definitions of what a competency within a framework entails and how they are measured; and all of them have a structure that is more or less "nonhierarchic". In a lot of Dutch HRD literature a competency is defined to encompass a body of behaviours, skills, knowledge and attitudes within a set context (workplace).

In this definition the behaviours, skills, knowledge and attitudes define what you'd select in the taxonomy, when creating the framework. They, however, exist laterally to eachother and that is not what the functionality is currently able to do. Each entity in the taxonomy is a 'child' of the thing above and this invites to use definitions within the competency framework that are hierarchical. How hard would it be for us to revisit this  functionality and enable learning developers to create either hierarchical or parallel taxonomies?

Does anyone know when Moodle Cloud will upgrade to 3.1? We expect to upgrade in the first week of September, but we may have a request to run a pilot class before then.

Thanks,

Elizabeth

Hi Damyon,

I've just installed the 3.1 dev and built a learning plan with a competency framework. This works well as I work through different scenarios and set ups for the learner and reviewer. 

The only point that could be added is a CBE enrolment plugin. If a learner has a LP with courses required for completion they need the option to enrol. I read the point about cohort specific self enrolment but will there be a specific enrolment plugin to make it easier?

Cheers

Pete

Dear subscribers to this thread,

Just to let you know, a new forum has been set up which you may wish to subscribe to - Competencies. This discussion thread will be moved there shortly.