Analytics and reporting

Outcomes, stage 2

 
 
Martin Dougiamas
Outcomes, stage 2
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers

The Outcomes we have in Moodle at the moment are not complete.  You can choose a list of outcomes for the site, and you can choose a subset of those for the course, and you can assign outcomes to various activities, and even set grades (via scale) for them in assignments.  This helps somewhat for course design and some grading, but there is no transference into competencies, and there is no concept of progress tracking for students based on these.

Others have built systems in Moodle that support these things (such as ELIS and Totara), but we need something generic and simple in core.

Some plans for a stage 2 development of Outcomes in core are being developed.  I'd appreciate your thoughts on this direction.  It has the potential to go deep into every activity of Moodle that supports any kind of grading.

 
Average of ratings:Useful (1)
Picture of Gavin Henrick
Re: Outcomes, stage 2
Group Moodle Course Creator Certificate holdersGroup Particularly helpful MoodlersGroup Testers

I think it would be good to have outcomes to be natively structured in a hierarchy both for visual impact, management and to provide inheritance.

At present when people have a long list of outcomes, it is not so easy to add them quickly, so a tree will be easier.

So a simple father/child aspect added to outcomes would be good.


Of course people don't have to use a hierarchy, but most of the standard outcomes I have seen specifications for are hierarchy based where a number of outcomes achieved, achieve the outcome above them and so on.

I love the idea of a rubric being created automatically from a chosen outcome (and its children outcomes)

 
Average of ratings: -
Picture of Amanda Fickes
Re: Outcomes, stage 2
 

I like the simple tree-like hierarchy structure initially.  

In addition to that,  being able to "tag" outcomes by grade level would be useful.  

The grade level isn't always part of the tree. For instance, US career and technical standards are not grade level specific - they are for all high school students, so they could all be tagged with 9, 10, 11, and 12.   Whereas national science standards, even covering similar topics, have separate standards written for each grade level.  One might want to sort by a specific topic, and ignore level - for instance, a teacher teaching many grade levels.  I am an agriculture teacher, and teach students grades 9-12, but I might be interested in viewing lower-level math standards by their topic areas.  Even though measurement conversions may be learned in elementary and middle school, I see value in reinforcing and practicing that at the high school level, so I might want to ignore grade-specific aspects of a tree structure when searching a standard set in Math, for instance.  But some teachers who only teach one grade may want to sort out all standards so that they're seeing other subject areas beyond their own even, but only for that grade, so they have some idea of what their students are learning in other classes.

Having the grade level be part of the tree structure would require making a "duplicate" of those standards in each of the grade levels it is part of - and that might be 13 duplicates that aren't actually connected, in some subject areas. 

It would probably be good if institutions could add their own grade level options, too.  For instance, a middle school may only want to include grades 6-8.  Or, the entire school district may have a Moodle and want pre-K through 12.  Or maybe, a school has an articulation agreement with a local college and offers some courses for college credit, and/or high school credit.  So some "odd" grade level might be required, such as Grade 13 at any university.  Or Grade 13 - ABC Institution only.  Etc...   So the grade level options should be editable.

This would also help teachers to facilitate looking at how "well" a particular student reached a particular outcome in a grade-specific course, and they might know that that student may need extra practice in a certain area in their grade level if outcomes are also tied to specific users.   Teachers of different subject areas would benefit too.  For instance, sometimes as an agriculture teacher, I'm not familiar with how much all my students know about graphing - parabolas, sin waves, etc, and I need to talk about those types of graphs in a wildlife ecology class, but it might be useful to know which students have that subject down, and which never elected to enroll in Algebra.  Or, being I have a wide age-range, sometimes geometry concepts are familiar to some of my students, and completely new to others.  

So also that idea of having outcomes being linked to a user - and somehow even when the same outcome may be used in different subject area by different teachers - would be a neat tool if it could be implement.  

 
Average of ratings: -
Floyd Saner - Tail of the Dragon, U.S. 129, NC
Re: Outcomes, stage 2
Group Particularly helpful Moodlers

Martin, I fully support enhancing this area of Moodle.   Providing strong support for outcomes, standards, analytics and reporting will meet the needs of many users and will further distinguish Moodle as a (the?) premier LMS.

However, it is a large task.  To be useful the system will have to be:

a) easy to use (from configuration, data input and reporting perspectives).

b) flexible/adaptable, so it can be used with many different standards and analytics requirements.

Here in the U.S., the Common Core standards are rapidly being adopted by many states.  Any development to make it easy to incorporate Common Core with Moodle will be welcomed.

-Floyd

 
Average of ratings:Useful (1)
Picture of Royce Robertson
Re: Outcomes, stage 2
 

Hello,

In order to make the most out of any Outcomes functionality, I'd like to see:

- An Outcome library that can be organized by program, etc.

- Ability to associate Outcomes with Activities, Grades, Rubrics, etc.

- Outcome Report that can be exported to CSV, etc. for further manipulation

As an aside, is it possible for students and faculty to see what outcomes are associated with which Activities, Grades, etc. in their native view without having to recieve a report?

Thanks,
R

 
Average of ratings: -
Picture of Calvin Ngotho
Re: Outcomes, stage 2
 

From documentation I read from UK Quality Assurance Organization QAA . I got to know that credits of a course is awarded as a result of a student showing evidence of outcome achievement (qualitative judgment) not necessarily by grading an assessment (quantitave judgement).

In designing a course, the course objectives/outcomes act as a guide to what the syllabus, assessment plan, the credit level and a whole other factors will be.

I am developing (still on going) a course management for A1Academia and here are a few ideas I would like to share.


Outcomes can be categorized in these categories:

  • Qualification level(SIS)
  • Program Level(SIS)
  • Course Level(SIS/LMS)
  • Assessment Level(SIS/LMS)
  • Question level (LMS)

The brackets beside the categories indicate where they are best defined and managed.

Qualification level outcomes(QLO) focus on a student's qualification achievement e.g has minimum credit for the qualification been achieved?

Program level outcomes (PLO), more like a degree audit evaluation, deals with an assessment if the student has met the requisite requirements, has registered min required class attendance, has met the overall academic program outcomes linked to course level outcomes e.g if a PLO is "a student should employ scientific analysis and problem solving in the field of study".

Course Level Outcomes (CLO) are more subject specific and extend some of the PLOs. CLOs determine if a student gets credit for the course or not. These outcomes dictate on the assessment plan to be employed, on the syllabus etc.

Assessment Level Outcomes (ALO) narrow down to specific areas within the course. They extend CLO in a stricter fashion in that achievement of all mandatory assessment outcome leads to an automatic achievement of parent CLOs . A student may excel in an assignment but doesn't convince the instructor interms of outcomes.

Assessments have questions. Questions could have outcomes also. An example is an essay kind of question that tests a student on a particular concept can have one or more outcomes that contribute to ALOs.

An outcome in my view is a scoring rubric where it defines an attribute to be measured, and a list of possible outcomes,
some +ve and others -ve. In a Programmer's language, those with +ve return 1 and the -ve a 0.

For moodle(LMS) the cycle revolves around the cycle of the course. For an SIS the cycle is the student's academic career. Which means moodle will be producing a students outcomes profile for each course offered so that an SIS can determine whether to award credit for the course and later the PLOs and so on.

 
Average of ratings:Useful (3)
Picture of Nancy K Hoke
Re: Outcomes, stage 2
 

Greetings - Just wish to add my support for the development of the Outcomes.  I am just starting to look at adding the ABET outcomes for our Engineering programs and if I could add the unique outcomes for a program that would be excellent. 

Would it possible to create a unique set of outcomes and then be able to assign the set to a course?  Perhaps the use of categories for Outcomes as it works in the Gradebook.

I would also like to way to add multiple outcomes to an activity. This would allow for multiples outcomes are assessed in an activity but not graded individually.

 

 
Average of ratings: -
Picture of Enrico Canale
Re: Outcomes, stage 2
 

I would also support the notion of a hierarchy or at least the ability to work with outcomes across contexts (or containers?).

At our university we have program-level Graduate Capabilities, which are assessed within cornerstone, mid-point and capstone courses and at different year levels. They are defined as site-wide outcomes in Moodle with a standard scale. It would be a great help if evidence of individual student achievement of the Grad Caps could be drawn from across multiple courses.

In this sense the outcomes belong to the student, not the course, though the evidence is found at the course level or, using Calvin's hierarchy, at the level of the assessment item.

 
Average of ratings: -
Jim Judges
Re: Outcomes, stage 2
 

This is a key area for development and would make a big difference to the great number of Moodle users who teach on competency based programmes.

Regarding Overview item 3 "Develop reports with graphs for teachers" I would suggest using the completion tracking options in Moodle 2 as a model as this produces an excellent and very teacher-friendly view of progress in the "Activity completion" report. We need something just as clear for outcomes.

Building on Gavin's suggestion of hierarchy I think that a similar system to that used by completion tracking and conditional access settings could be used or built on e.g. check if the three child outcomes x.1, x.2, x.3 are complete then indicate that the parent competency x is complete. In other words instead of allowing access to the next activity, satisfactory completion (marked manually or automatically) of any selected outcomes will provide a further group or parent "tick" that indicates completion of this group of outcomes. 

Look forward to following suggestions and developments on this important proposal.

 
Average of ratings: -
Picture of Sarah Phillips
Re: Outcomes, stage 2
 

Hi Martin, 

We use the outcomes to map to the criteria for each unit against the activity they relate to. 

We have noticed when using outcomes that when you apply them to more than 1 activity in a course, (Which of course is common in this capacity) they present themselves in the gradebook next to each of the activities, but it would be nice for a final collective outcome result when all the activities using the particular outcome are satisfied. This would include something that recognises the number of activities the outcome is aligned to and therefor provides a percentage of completion for that outcome each time another activity is completed.

We cannot seem to find where to do this unless it is not possible, and if that is the case, it would be a great feature to improve within the outcomes area. 

Thanks, i really want to use this area to its fullest potential.

 
Average of ratings: -
Picture of Phill Miller
Re: Outcomes, stage 2
 

Martin, 

As we have discussed, the folks here at Moodlerooms are really keen to take on this project to help out our clients.  In that spirit, I've posted a pretty comprehensive specification of how we believe that this should work, after talking about it with our clients.  We are hoping that other members of the community will comment on this and help us finalize the specification.  I'll also be doing a client working group with our clients to review the spec and make any changes.  

 

The Moodlerooms development team will probably start developing against this specification in the next few weeks.  

Should be a fun project! 

Phill Miller
VP of Products, Moodlerooms

 
Average of ratings: -
Picture of Phill Miller
Re: Outcomes, stage 2
 

I forgot to post a link to the spec:

http://docs.moodle.org/dev/Outcomes_Specification

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Outcomes, stage 2
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

I am going to post my thoughts in the Major new features forum since the Moodle development process says to discuss major new features there (I think this qualfies) and I believe that forum gets more attention.

See https://moodle.org/mod/forum/discuss.php?d=220143.

 
Average of ratings: -
Picture of Richard Baker
Re: Outcomes, stage 2
 

Hi Phil,

Do we know of these features will be available for Moodle 2.6 in Nov?

 
Average of ratings: -
Picture of Phill Miller
Re: Outcomes, stage 2
 

I believe that they are targeted for November.  Moodlerooms is rolling out to its clients in July.  There is a chance that some specific functionality may not make it in the release, but we will document that where possible.  

 
Average of ratings: -
Picture of Elizabeth Dalton
Re: Outcomes, stage 2
Group Particularly helpful Moodlers

Since Outcomes will be tied to Competencies, how will they relate to Badges as included in 2.5?

 
Average of ratings: -