Development roadmap and Stanford MOOC

Development roadmap and Stanford MOOC

by Brian Mulligan -
Number of replies: 30

I have just signed up for a MOOC from Stanford on Designing a New Learning Environment.  If I can be honest, folks, recently I've been a little disappointed in Moodle on 2 points.  

Firstly, we've moved to 2.2 and I, and a lot of lectruers in my institution, find the user interface to be "difficult", and I am wondering if the design is overly influenced by techy types as opposed to regular users (educators).

Secondly, i've been trying to see if there are any developments in the pipeline for massive peer-assessment and taking part in a discussion in the Loung on this (http://moodle.org/mod/forum/discuss.php?d=204799).  I have to say that I'm a little disappointed in that there does not seem to be much going on around this topic.  Perhaps, i am looking in the wrong place.

So I am posting here to see if anyone knows if MOOC facilities are figuring in a roadmap for Moodle.  I selected Moodle for my institution in 2004 and have been very happy with it.  Only recently when I spoke to a friend in Penn State and found that they are considering a new LMS instead of Angel and Moodle is not one of the top 3 being considered that I started questioning my own assumptions abaout Moodle (Canvas by Instructure is one of the 3).  

It is nearly 20 years since I last worked in software development and my responsibilities now include the selection fo systems for our organisation, so I am now thinking I now need to seriously review our commitment to Moodle.  To improve my own understanding of these issues I have signed up for this Stanford MOOC.  In this MOOC you can select the group you wish to work with and I'm wondering if it might be worthwhile for a group of Moodle developers to be part of one.  For that reason, I'm posting this message.  I am very attached to Moodle, and if other systems are popping up with desirable features that Moodle does not have, I would prefer if these were on the Moodle Roadmap, rather that have to make a change.

Below is some text from the MOOC.

Regards,

Brian

Welcome edu-preneurs!

I am very excited to connect with you and work together to design new, effective learning environments!

The course officially starts on October 15 and will run for 10 weeks. During the course period, you will watch a series of short lecture videos, work on individual and team assignments, evaluate your peers’ individual and team assignments, and ultimately design and pitch a new learning environment as your final team project. Examples include learning management systems that do more or are better than existing tools, mobile learning models or e learning pedagogies, a technology that supports learning and assessment, new school system models, or a blended learning program. The strongest designs will trigger and facilitate active, constructive, real-world problem solving and collaborative learning, and will leverage the 21st century communication media.

During the first two weeks, you will focus on individual assignments and learning about and connecting with your classmates. There are already 6,000 students enrolled, and I expect more over the next weeks leading up to the course, so make your presence known in the discussion forum! It is very important to share about your background, skills, interests, philosophies, and experiences during the early course period because by the 3rd week, you will use these connections to form project teams. You will form your own teams—they are not assigned—so make sure people in the course know your project ideas and work style. You may only belong to one team at a time and you are encouraged to create or join a team that consists of classmates who share a similar interests or beliefs about education, but with a broad range of experience and skills (particularly any skills or experience you don’t possess that would help your work).

There are no quizzes multiple choice questions in this course. Learning here is not memorizing— it is creating, designing, negotiating, presenting, evaluating, and building the working relationships and being the active contributor that it takes to do all of this on a team. Instead of exams, you will be evaluated by your classmates on the level of quality and originality of your work, as well as on your level of active participation and contribution in the class. Your course “rank” is on a 0-5 scale. Completing only your individual assignment will not earn you enough points to complete this course. You need to earn a ranking of at least 3 out of 5 to have successfully completed this course. Your active engagement throughout the course will be crucial. I estimate that you will need to spend about 4 hours each week, though actual time needed may vary depending on individual experiences and team effectiveness.

Those highly ranked teams presenting excellent solutions will be featured on a Stanford website so the world beyond this course may learn about your work and perhaps help your team make its design a reality.

I am glad to see so many people from all over the world interested in this course, and in our shared vision of helping every student reach his or her full potential regardless of age, color, or nationality. We are here to cause a paradigm shift in education settings around the world.

Once again, welcome my fellow innovators! Let’s design a new learning environment. If we imagine it together, we can make it happen.

Thanks for your passion.

Sincerely,

Paul Kim 
Chief Technology Officer and Assistant Dean 
Graduate School of Education 
Stanford University

http://venture-lab.stanford.edu/education

P.S. If you know of someone who might be a good teammate for your project idea, don’t be shy to have her or him join!

Average of ratings: -
In reply to Brian Mulligan

Re: Development roadmap and Stanford MOOC

by Colin Fraser -
Picture of Documentation writers Picture of Testers

Brian, I tend to agree with your comments about the interface, but I would suggest it is downright ugly. Yes, I suspect that programmers designed it, or it  might have been designed by committee, but it does not matter, you will get used to it. I am sure you already know there are much worse than Moodle out there without the functionality. You can expect to see the essential interface designs changing as well. As themes change, so will layouts and eventually real improvements to the interface will occur, without compromising stability or usefulness of the product. Lets face it, v1.9 was a very hard act to follow - didn't matter what was done, it was never going to measure up.  

As far as peer-to-peer assessments are concerned, this was mentioned at last year's Australian Moodlemoot, Martin made a reference to it as a future development, so it is on the burner. However, the lead times can sometimes be dictated by other "more important" issues, e.g. Moodle Partner demands, or internal debates or other important issues. It can be done now in a limited sense, ratings of Forum Posts, or Permissions overrides will allow you to use peer assessments for assignments. You may you want students to use the Moodle Gradebook, or if you want students to be able to access uploaded assignments, look at the permissions. Otherwise, you might consider doing what I do, ask for students to publicly present their assignments for peer assessment. I also provide a rubric as an assessment guide, and a grading sheet. Teachers/lecturers/tutors can still use the Moodle gradebook to record results - I do.

Also, we are still in the first two years of the largest shakeup of an outstanding product - there is bound to be some disappointments. It is funny you should mention Canvas, I have had occasion to look at it recently, and think it might be a future rival for existing systems, but it is still very immature. Be a while yet. Other systems have their flaws but I suspect if people are having issues with their Moodles then it is possible their practices have not kept up to the demands of the product. That is certainly an issue I have seen in some places I have been at and some reskilling/retraining has helped considerably. Once you get used to something, and it changes, it is too easy to complain than to look at how the changes require commensurate changes in practice. I am guilty of that myself.

 What do you mean by "MOOC"? Not a term I am familiar with, but then I am a long, long way from Stanford.

Average of ratings: -
In reply to Colin Fraser

Re: Development roadmap and Stanford MOOC

by Brian Mulligan -

Thanks, Matt and Colin for the feedback.

Yes, I think that part of the problem is that it is 2.2 is new to me, but I also suspect that there are a few design faults - which I'm sure can be easily sorted out, so I won't throw out Moodle for that.  (eg.  I have had 20 cases of lecturers locking themselves out of courses by clciking on the eye beside a manual enrolment instance - I''m thinking of starting a "Beware the Eye" campaign).

It is great to hear that Martin has made reference to peer assessment.  MOOCs are these new free Massive Open Online Course from the likes of Stanford.  They have cracked the low unit delivery cost by delivering excellent content to very large numbers of people.  In technical areas they are doing well by using multiple choice questions and can develop very sophisticated questions by automatically analysing responses by large numbers of users.  However, feedback from humanities MOOCs is poor on the assessment end.  Siemens and Downes have been somewhat critical of the behaviourist approach of Stanford/Coursera but even within their MOOC they don't have a solution for low cost formative assessment.  So far the only solutions being floated for low-cost assessment in MOOCs is peer assessment.  Coursera/Stanford are not there yet, but I think they are working on it and are likely to crack it soon.  And that is when I'll be tempted to leave Moodle.

By the way, Matt, I'm looking for more than a social networking platform in the VLE.  The grading of participant contributions needs to be reliable and accurate.  I think the way this is going to be cracked is by what I call "calibrated peer assessment".  If we have a system where each participant grades 4 others and they grade 4 others, but not the same 4, the instructor should be able to sample certain gradings and calibrate them upwards or downwards.  It should then be possible to have an algorithm that allow this calibration to then circulate through the system and calibrate other graders.  Graders could also have their grades graded.  With an iterative system that was able to indicate the reliability of certain graders, those with higher reliability would have heavier weighting in future grading.  If graders were rewarded for accurate grading (which would be an indication of theri grasp of the course material), this would be an incentive to grade accurately.  It might turn out that an instructor might be able to calibrate a class of 1000 by looking at a relatively small number of assignments.

Thanks for the opportunity to express this idea as i need to refine it for contribution to the course I just signed up for.

Thanks for the links to Krebs and Cross.  I think they are looking more at networs than measurement, but some of their ideas might be required to develop an algorithm that will reliably work.

I do think dating agency software would be best for dividing into workgroups, but I also feel that for unit cost reduction it is the peer-assessment rather than the peer-learning that is going to make the difference.

Brian

Average of ratings: Useful (1)
In reply to Brian Mulligan

Re: Development roadmap and Stanford MOOC

by Matt Bury -
Picture of Plugin developers

Hi Brian,

I think effective assessment is necessarily labour intensive since computers only deal in semantic rather than pragmatic meaning, i.e. They aren't us, they have no concept of "We", and simply can't interpret what we mean to say. Like you say, especially true of the humanities but also true to a great extent in the parts of the sciences that really matter (science is deeply philosophical). So I think leveraging learners pragmatic cognitive skills for assessment is the way to go.

I've done a fair amount of peer assessment, and like you say, it's an excellent way for learners to demonstrate through analytical and critical thinking, that they have procedural (not just declarative) knowledge of the subject matter. In some ways, I think it's a more meaningful and reflective of the things that really matter in higher education (and should matter in K-12 too), although I think we do have to assess lower order thinking skills to give a bigger picture of how and why learners are performing well and/or poorly, i.e. Are they doing well or badly because of their information (lower order thinking skills) or how they use it (higher order thinking skills)?

Moodle has peer assessment tools, I prefer the peer assessment Assignment type plugin to the Workshop activity module. I don't know if either are available in Moodle 2.x but, if Moodle stays true to it's constructivist roots, I should think it'd be a priority. I'm not sure how meaningful the rating system is on the forum and glossary module. If anything, I think it may only reflect the status quo and what's popular rather than how much learning is going on; Could you use cute cat videos to boost your peer ratings?

The challenge for MOOCs is for them not to become just another form of self access learning and being able to cultivate the kinds of interpersonal relationships between cohorts of learners, as well as intervention by teachers where necessary, that lead to cognitive engagment and hopefully collaboration (co-construction of knowledge). There's more than 10 years' worth of research in this area here: http://communitiesofinquiry.com/ I suspect that in addressing issues of learner engagement and high rates of attrition, MOOCs may morph into something quite different to what they are now. It's exciting stuff!

BTW, here's a cute cat video:

Do you think it'll boost my Moodle rating? ;)

Average of ratings: Useful (2)
In reply to Matt Bury

Re: Development roadmap and Stanford MOOC

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

The Workshop module is, of course, still present in Moodle 2.0+, and better than ever.

Average of ratings: -
In reply to Matt Bury

Re: Development roadmap and Stanford MOOC

by Brian Mulligan -

Hi Matt and Tim

the cute cat video comment does illustrate the point I'm making.  Peer assessment can't be trusted in itself as it is subject to many potential distortions (including popularity).  That's why I am pushing the idea of calibration.  If the calibration can ripple through the system, and there are incentives to grade peers accurately, small sample by instructors may be all that is required.

I'm not familiar with the peer assessment plug-in for Moodle.  I should look at it and see what would need to be added.  In the other discussion I referred to  (http://moodle.org/mod/forum/discuss.php?d=204799) it was also pointed out that to do this for free (or very low cost) at large scale a learning management system would also need to be able to automatically assign people to groups (speed dating technique?) and assign peer assessments in a chain like manner to allow calibration to ripple through the system as well as actually use the raw peer grades and instructor s small number of grades to carry out the actual modifications.  So even though the Moodle plug-ins might be a start, my guess is that there may be some work yet on defining the algortihms and developing the code before it would be ready for really massive courses.

Tim, I will challenge the statement on multiple choice quizzes.  The insinuation seeems to go against the other Stanford MOOCs that make heavy use of objective testing.  I agree that they can be quite effective at  assessing learning at many levels (I got into this business using QuestionMArk in the nineties).  Not only does a VLE need good objective testing but also the tools to identify which questions are good and which are not.  Question Mark used to have those tools, but I have not noticed them in Moodle - do they exist?  (They correlate performance on individual questions with complete tests - those with the best scores are deemed to be best at discriminating between candidates).  Khan academy seems to be devloping very sophisticated tools in this area and I think that Coursera are developing these not only to analyse the questions but also the performance of learning items in their courses.

Brian

 

Average of ratings: -
In reply to Brian Mulligan

Re: Development roadmap and Stanford MOOC

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Moodle does have good test statistics: http://docs.moodle.org/22/en/Quiz_statistics_report

Moodle Workshop is maintained by David Mudrak, who is interested in discussions about how in can be made better. There is a whole forum for Workshop-specific dicussions, so probabaly best to start a thread there to discuss it. http://moodle.org/mod/forum/view.php?id=740

Average of ratings: -
In reply to Tim Hunt

Re: Development roadmap and Stanford MOOC

by Colin Fraser -
Picture of Documentation writers Picture of Testers

Tim, I suggest a clear set of published rules about how to write your own queries might help. This assumes though that the users have a clear idea of what fields and tables thay can query and an understanding of what data is being recorded. The more difficult task has always been what data to record, rather than how to access it.

Brian, in a previous life, I have had managers ask me if it was possible to create a database that included a lot of really trivial information, or data that would seriously compromise many legal, ethical and even moral standards. I have had to cut people short here and explain basic ethics to them, not making myself popular in the process. I am sure any database developer could tell you the same...mixed

Average of ratings: -
In reply to Colin Fraser

Re: Development roadmap and Stanford MOOC

by Eamon Costello -

Here's the best overview of MOOCs that I have read: http://www.tonybates.ca/wp-content/uploads/Making-Sense-of-MOOCs.pdf . It is a fascinating read. As well as the potential it also points out some problems such as teaching quality, dropout rates, lack of business models and gives historical precedents of similar ventures that failed.

The original MOOCs from Athabasca University use Moodle successfully . I think I enrolled enthusiastically but never quite managed to take the course.

 

Look forward to hearing about your experiences taking the Stanford course Brian.

Average of ratings: -
In reply to Eamon Costello

Re: Development roadmap and Stanford MOOC

by Colin Fraser -
Picture of Documentation writers Picture of Testers

This is an interesting perception Eamon, Bates' assertions that completion rates are going to be a major factor in determining the value of a MOOC are pretty obvious though. I know that different learning styles impact seriously on how successful people are in their academic achievements. I am very much a social learner, and the idea of doing a course entirely online is just not appealing. I prefer my interactions with peers, lecturers and tutors, face to face. It is from them I get the motivation to continue when it gets hard, to ideas from discussions, the various perspectives needed to complete assignments. Wikis and forums become really important, general discussions about themes and concepts critical for online courses, for people like me. I am pretty sure now the only reason I completed university was the intellectual stimulation I got from the people around me. Can a MOOC provide the same atmosphere? I doubt it, so I suspect there will be an alarmingly high dropout rate.

Average of ratings: Useful (1)
In reply to Colin Fraser

Re: Development roadmap and Stanford MOOC

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Colin " What do you mean by "MOOC"? Not a term I am familiar with, but then I am a long, long way from Stanford."

Neither was I. Interestingly, the Wikipedia article on MOOC has a warning:

This article appears to be written like an advertisement. Please help improve it by rewriting promotional content from a neutral point of view and removing any inappropriate external links. (August 2012).

Joseph

PS.- the OP (original poster) in this thread (Brian) should have explained the MOOC acronym in the first place.wink

Average of ratings: -
In reply to Joseph Rézeau

Re: Development roadmap and Stanford MOOC

by Brian Mulligan -

Joseph,

you are correct.  I should have defined MOOC, and given a reference.  However, this is a fault that most of us have.  If you scan back on this discussion you will see the terms LMS, VLE and MCQ used without explanation.  My apologies.

Brian

Average of ratings: -
In reply to Brian Mulligan

Re: Development roadmap and Stanford MOOC

by Matt Bury -
Picture of Plugin developers

Hi Brian,

I've participated in Dave Cormier, George Siemens and Stephen Downes' and Change MOOC: http://change.mooc.ca/about.htm It was an interesting experience and I learned a lot about MOOCs from a learners' perspective.

On the software front, I think what you're looking for is a cross between an LMS and a social networking platform, or more likely, primarily a social networking platform that has LMS capabilities. I would tend to agree that Moodle, and indeed most existing LMS' don't fit the bill too well. Tracking that many learners and the relationships between them, especially if they self organise, is the kind of challenge most LMS users tend not to prioritise.

I've been looking at online platforms for extensive reading programmes and am trying out http://elgg.org It's a bit like Facebook but especially designed for learning. There are some plugins to automatically award points and badges for participation but there isn't an overall grade book. Learners are free, however, to self organise and form their own groups and relationships, a la Facebook. Another option is http://buddypress.org, an offshoot of Wordpress, which has similar social functions but is less geared towards education. There's also a decentralised option with Diaspora* https://joindiaspora.com/ but that's still a very young project.


Even with SN software, I think tracking large numbers of personal relationships and interactions is a challenge. Two of the leading researchers in this field are Valdis Krebs: http://networkweaving.com/valdis.html and Rob Cross: http://robcross.org/

BTW, in organising learners into groups, Dave Snowden (http://cognitive-edge.com) says he uses dating agency software rather than letting learners self organise.

I hope this helps! smile

Matt

Average of ratings: -
In reply to Brian Mulligan

Re: Development roadmap and Stanford MOOC

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

As has been said before, this course here, Using Moodle, is and has been a MOOC about "work[ing] together to design new, effective learning environments".

If someone wants in interesting topic to raise (it is not a convenient time for me to participate myself) then they should challenge one of the statements implied in the course introduction.

"There are no quizzes multiple choice questions in this course. Learning here is not memorizing"

Actually, multiple choice questiosn can assess knowledge at all levels of Bloom's Taxonomy. See http://www.biomedcentral.com/1472-6920/7/49/. (Of course, I am not trying to claim that MCQs are the best way to assess any given topic, just that the course introduction seems to be betrying the author's predjudice, which is not a good start for an academic course.)

Average of ratings: -
In reply to Tim Hunt

Re: Development roadmap and Stanford MOOC

by Colin Fraser -
Picture of Documentation writers Picture of Testers

Ahh, OK, now I can see there is a growing need for something I have been touting for years. I have been at my ITC masters for years about integrating Moodle with a wiki app, and a blogging app, and a social networking app. A Mahoodle, for me, does not cut it, although it does go a way towards it. I would seriously suggest to Brian, and to any one else interested, that a server with all these different programs on it, either fuly integrated into one app or not, as long as they have a single login, is a tool of the future.

One of the things we often get told is that we do not really know where the technology is going to take us, but there will be more and more convergence in it, we already see it. Why should we consider software any differently? We had someone ask if there was an integrated School Management System (note school, not just student) and an LMS recently. Sounds like a silly question, but not really, I would think a seriously innovative idea. Create a one stop shop? The PHP apps are ideally suited for this purpose, all it takes is for someone to strip out their functionality, reintegrate it and make sure that it works with a single security framework. A very big all, and an incredibly complex task, for sure, but not one that is beyond a group of seriously dedicated PHP programmers, I suggest. Oh my... could we be talking about a bunch of people like Moodle Devs?

Average of ratings: -
In reply to Tim Hunt

Re: Development roadmap and Stanford MOOC

by Matt Bury -
Picture of Plugin developers

Re: MCQs, I've seen this argument come round and around. It would be so convenient if MCQs could reliably and effectively assess higher order thinking skills. Unfortunately, I haven't heard anyone come up with an example of MCQs that can effectively assess procedural knowledge, tacit knowledge, implicit knowledge, etc. (from my perspective, procedural knowledge is not the conscious application of a formula).

Can there be uncontrived, authentic, real world situations (I mean whole situations, not subsections) where there is only one or a few possible answers (a pre-requisite for MCQs)? How many really interesting questions have only one or a few answers? If we chop the question up into prescribed parts, we are in fact, telling the candidate what the acceptable answer is rather than letting him/her answer and then assessing it on its own merits. Sometimes valid answers come from unexpected and unforeseen perspectives. It's similar to how debates can be manipulated towards a desired outcome by framing the questions. No matter how hard the candidates try to argue their case, the question procludes any answer but the desired one. In this case, what are we teaching; analytical and critical thinking, or conformity and compliance?

Who do you want running the world, looking after you and rescuing the environment when you've retired? Someone who's good at MCQs?

Average of ratings: -
In reply to Matt Bury

Re: Development roadmap and Stanford MOOC

by Brian Mulligan -

"Who do you want running the world, looking after you and rescuing the environment when you've retired? Someone who's good at MCQs?"

Yes, if the MCQs are good enough.

And it is probably possible for most things.  More importantly it can be verified.

Certainly real world scenarios can be presented and options given that should be able to discriminate between those who understand more and less.  Although there may be a debate over which answer is best (particularly if the wrong answers are plausible), over a large number of questions this can be tested against other methods to see how reliable it is.

By the way, it's not as if the other assessment systems are that great - try correlating a bunch of students examined by 2 different examiners and have a look at the scatter plot.  And what percentage of course use these very sophisticated methods of assessment you are referring to.

It is a valid objective in education to do things almost as well as we are doing them already, if we can do them at much lower cost.  The biggest barrier to higher education is cost.  It is as much our duty to decrease cost as to increase quality.

Having said all that, it is very difficult to develop high quality MCQs for certain topics and that is why I am interested in calibrated peer assessment.

Brian

Average of ratings: -
In reply to Brian Mulligan

Re: Development roadmap and Stanford MOOC

by Matt Bury -
Picture of Plugin developers

Hi Brian,

I think we're in agreement that MCQs are often a necessary part of overall assessment (I think as much for evaluating course curricula as assessing learners). Yes, MCQs are more efficient, they're cheaper and quicker to administer, you can roll them out on a massive, global scale and still keep them consistent, and if you're successful, have them recognised everywhere. There's a lot of profit to be made from standardised testing and MCQs are the way forward.

I would argue that, because of their convenience, they tend to get overused. In the "old days" of transmission and behaviourism based education (are we still in the old days?), they perceived fewer problems with testing learners' ability to Remember, Understand and Apply declarative knowledge. Procedural knowledge is notoriously difficult to get at for assessment purposes, i.e. how can we gather evidence of what's going on inside their heads? This led to a great number of graduates who had marvelous certificates, diplomas and degrees but couldn't think their way out of a paper bag and performed poorly on more complex, cognitively demanding tasks.

However, how many times have we heard university staff bemoan that each years' intake of undergraduates are insufficiently prepared for university study? That they don't have the cognitive, metacognitive and social skills to be successful learners? And what do they usually point to as the main culprit?

Examples of qualifications that require assessments of procedural knowledge would be:

  • Medical practitioners: doctors, nurses, dentists, radiologists, surgeons, etc.
  • Engineers: civil engineers, architects, aviation, etc.
  • Teachers
  • Software developers
  • Trades people: electricians, plumbers, machine operators, musicians, etc.

These are professions that require apprenticeships and assessment of procedural knowledge to be considered fully qualified. There are also professions that happily ignore assessments of declarative knowledge and it usually doesn't matter if you have certificates, diplomas or degrees. Potential employers are more interested in auditions and portfolios:

  • Musicians*
  • Artists
  • Designers
  • Illustrators
  • Actors
  • Sales people

*Most of the full time professional musicians I met in my career didn't finish music college and many didn't go at all, yet they were all highly knowledgeable. I also met a lot of graduates of music college (from exemplary music colleges too) who didn't really understand how music works and couldn't really play or write music very well.

Average of ratings: -
In reply to Matt Bury

Re: Development roadmap and Stanford MOOC

by Brian Mulligan -

Hi Matt.

I think we are wandering off into another big discussion area - separate from peer assessment.  I don't disagree with what you say but I would like to make one final comment.  Perhaps higher Education is too ambitious or arrogant to think it can give people all the skills they need to succeed in life.  A lot of the skills you refer to are not imparted at all in most higher education course, much less tested for.  Perhaps, that is how it should be, because it may be a very inefficient way to do it.

Brian

Average of ratings: -
In reply to Matt Bury

Re: Development roadmap and Stanford MOOC

by Elizabeth Dalton -

The tradeoff with MCQs and constructed responses is well researched-- MCQs are harder to write well, especially at higher levels of Bloom's taxonomy (or your model of choice), but easier to score (and can have automated scoring), whereas essay questions are easier to write but much harder to score reliably. The problem is, very few faculty have any formal background in assessment design. The quality level of faculty-generated multiple choice questions tends to be very low, and no analysis is done to eliminate bad questions. Short essay questions tend to generate more reliable results with faculty who are experts in their subjects, but not in assessment design. Yes, ensuring reliable grading can still be a problem. Rubrics can help... but again, the art of constructing a good rubric depends on the kind of educational experience few faculty have. At least faculty do have the advantage of years of practice in writing and grading essays... but it would be a real advantage to have stronger tools in place to support essay scoring.

I did a stint of work at a professional testing agency as an essay scorer. For each question, the team of about a dozen graders had to be trained, and tested on 10 samples of the question. During the grading process, differences between two graders would be forwarded to a lead grader who would make the judgment call. Too many such differences could result in the grader who was further from the norm having to recalibrate or even being dropped from the grading project.

I found this thread because I was specifically searching on "moocs" at moodle.org. We use Moodle at our institution, and while we aren't necessarily going to offer free unlimited moocs, we are very interested in some of the methods used, especially peer assessment.

The Workshop activity, with its inclusion of a grade component for the peer assessment activity itself, has the right idea, but wouldn't work in something like a mooc, because the process stages are too lock-step. A common pattern in a mooc is that a student is only able to rate the work of other students after having successfully been rated themselves... the Workshop activity assigns everyone at once, and has no means of differentiating based on prior success at the activity. Workshops also assume everyone will finish an activity and proceed to the grading phase at approximately the same time. In a "rolling" MOOC, this wouldn't be the case.

I am still hopeful that more sophisticated tools for peer assessment will become available within Moodle.

Meanwhile... what permission would one need to override to allow students to grade one another's work? I looked through the options and couldn't find anything that looked right.

Average of ratings: Useful (1)
In reply to Elizabeth Dalton

Re: Development roadmap and Stanford MOOC

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

II am very interested in the idea of Peer Assessment and I was impressed with  Michael de Raadt's simplified Peer Assessment Assignment tool for Moodle 1.9

However that has not been updated for Moodle 2.0.  Another wonderfully simple example of Peer marking is allowing students to grade other students posts on discussion forums. This can be according to cusom scales, i.e. Awful, Ok, Great, Awesome or whatever.

Elizabeth, are you aware of any other tools (MOOC or whatever) that have peer assessment activities that could inspire Mooldle and its developers.

 

Average of ratings: -
In reply to Marcus Green

Re: Development roadmap and Stanford MOOC

by Elizabeth Dalton -

Sorry for the delay in response-- since I can't subscribe to just this thread without getting the whole "general developer forum," I don't see posts unless I remember to check the thread manually.

BadgeStack is built around the idea of peer assessment, as an example. However, BadgeStack doesn't have rubric tools as nice as those in Moodle.

I think the advanced grading features in Moodle 2.4+ are what we need, but we need more granular permissions over who can grade and what else they can do. In particular, we need to be able to make someone's ability to grade dependent on completing a prior activity (just as we can currently limit access to activities based on completion), and just because a student is allowed to submit a grade, doesn't mean they should be allowed to see the rest of another student's grades or fiddle with the gradebook.

I also like some of the features of Workshop, but again, Workshop assumes everyone is going to go through the same activity at the same time and pace, beginning and ending approximately in unison. I don't think this recognizes that different learners need different amounts of time to master concepts.

What I really hope is that this feature request will gain some traction: MDL-27238 There is also more discussion here:

https://moodle.org/mod/forum/discuss.php?d=225896

Average of ratings: -
In reply to Tim Hunt

Re: Development roadmap and Stanford MOOC

by Elizabeth Dalton -

On reflection, I disagree that "Using Moodle" is a MOOC, or a course, for that matter. It is a set of discussion areas, loosely organized around areas of Moodle functionality. It offers no guided practice, feedback, or even learning objectives, and I am not speaking here of behaviorist "memorization" objectives, but even of sociocultural objectives like "become a fully participating member of the community of practice of Moodle administrators."

I don't mean to say that this discussion environment is useless. It is very valuable as a resource. But it is not a course, by any definition of the word "course" that I can imagine accepting.

Average of ratings: -
In reply to Elizabeth Dalton

Re: Development roadmap and Stanford MOOC

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

In a Massive course, is it reasonable to suppose it is possible to have learning outcomes that would apply to all, or even most of the learners?

Average of ratings: Useful (2)
In reply to Tim Hunt

Re: Development roadmap and Stanford MOOC

by Elizabeth Dalton -

Yes, I think it is reasonable. That's really what makes it a "course," rather than a social network or a collection of resources.

Edited to add: and again, I mean no criticism of "Using Moodle" as a resource by saying this. It's a very valuable resource. It's just not a course.

Average of ratings: -
In reply to Elizabeth Dalton

Re: Development roadmap and Stanford MOOC

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I suppose it all comes down to how you define a course.

I think Learning Outcomes are a good thing. The OU uses them as a matter of course. They are helpful for the teacher in preparing the course, and they are helpful to the student studying the course - at least when the course is teacher-driven.

But, are all courses teacher-driven? If a course has learning outcomes, then are those the only possible outcomes for a student on the course? (Almost certainly not.)

(Right now, I am feeling that this is a potentially fascinating philosophical discussion, but I am not sure I have the energy to engage in it at the moment.)

Average of ratings: -
In reply to Tim Hunt

Re: Development roadmap and Stanford MOOC

by Elizabeth Dalton -

I think a "course" could be based on a collection of learner-defined or learner-selected objectives, rather than teacher-defined objectives... but the idea of "course" really does seem to me to entail objectives, a plan, a path... it's sort of embedded in the etymology of the word.

Apart from that, if I want to learn something, one of the things I might very reasonably do is go to a prospective mentor who already knows how to do the thing I want to do, and ask them for some pointers of what I might do to learn. Otherwise I'm not likely to know where to start. Note that reading a book about the subject amounts to the same thing-- books are organized by the author(s) to select and sequence topics.

Actually, a feature that I would very much like to see in Moodle is the ability for learners to select from a universe of learning outcomes, set their own priorities and timelines for completion, be presented with resources, activities, assessments, etc. that others have found useful in reaching those objectives, and use Moodle as a tool to manage their own learning... and I think this would be enabling learners to construct their own courses. smile

I'd also like built-in tools to include a questionnaire about what the learner's own goals are at the beginning of every institutional "course," and I'd like to automatically trigger this questionnaire again at times the learner specified as when they hoped to reach their goals, to find out how well the learning environment (and institution) is serving the learner's needs.

Again, I'll stress that I think lots of learning occurs outside of "courses," whatever they are. And lots of very valuable resources exist other than courses, or external to courses. My interest at the moment is in trying to develop tools to support intentional learning, where someone in the process (preferably at least the learner) has some objectives or intended outcomes of some kind. I'd like to be able to use Moodle as a foundation tool for that kind of environment. I think it's rather limited that way at the moment-- it really lacks integrated support for learning outcomes, being based more around collections of resources authored by one or more specified persons. Not all courses are teacher-driven, but Moodle rather assumes they are in its very design, from the assumptions about roles and permissions to the forms of feedback allowed. (Consider: only one tool allows self-evaluation as a grading component, and no tools are designed to allow students to contribute rubrics.)

Average of ratings: -
In reply to Elizabeth Dalton

Re: Development roadmap and Stanford MOOC

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I would say that most people come here with very specific learning outcomes. They have some specific thing they want to learn about Moodle, and so come here to ask it.

Ironically (given the observations you make about the very teacher-centric way Moodle is commonly used) Moodle started out with a much more student-centric vision: http://docs.moodle.org/24/en/Pedagogy

This course (if it is a course) is one of the few places where that Pedagogy is really used. Every learner who appears with their specific question and asks it, and gets an answer, is increasing the collective knowledge of everyone who reads that thread.

As well as some learners coming here to learn about how to use Moodle in their teaching (or how to adminster Moodle) sometimes developers need to learn what users want from Moodle in the future. Two simple examples: https://moodle.org/mod/forum/discuss.php?d=227286, https://moodle.org/mod/forum/discuss.php?d=224916. However, as you say, the fact that learning is desired, and takes place, does not necessarily make this a course.

Average of ratings: Useful (1)
In reply to Elizabeth Dalton

Re: Development roadmap and Stanford MOOC

by Aaron Barnes -
"Actually, a feature that I would very much like to see in Moodle is the ability for learners to select from a universe of learning outcomes, set their own priorities and timelines for completion, be presented with resources, activities, assessments, etc. that others have found useful in reaching those objectives, and use Moodle as a tool to manage their own learning... and I think this would be enabling learners to construct their own courses."

Hi Elizabeth,

This has actually already been implemented as part of Totara LMS, they call it a "Learning Plan". A plan of courses and outcomes with priorities and duedates that a learner can create and update themselves (or their Manager can, in the corporate context).

I'm sure Totara would love to upstream Plan's to Moodle if it was something HQ wanted.

Cheers,
Aaron
Average of ratings: -
In reply to Aaron Barnes

Re: Development roadmap and Stanford MOOC

by Elizabeth Dalton -

This is very interesting. I looked at Totara a bit today, but didn't find detailed information about this feature yet. It's the ability for the learner to select outcomes and due dates that I'm really looking for... though some completion management features (like the ticklers I mentioned above) would also be great... do you know if Totara LMS has anything like that?

Average of ratings: -