Objective Comparison Learning Management Systems

Objective Comparison Learning Management Systems

by Mark Pearson -
Number of replies: 9

So, I'd be interested in community feedback (Derek Chirnside's views in particular) of the following:

We need some means of comparing our current LMS, Moodle, (and remember that we're moving to 2.8 over the summer which will introduce new feature and streamline others) with 'hosted' LMS (that is an offsite system, maintained by a 3rd party -- ie Canvas) in objective and replicable ways which compare apples to apples as far as possible.

Firstly I think that feature lists are irrelevant. Every system that we look at will claim to offer the same or equivalent features. Comparing lists of features doesn't get us very far.

Here's my suggestion:

Compare ease of completing common tasks

Make a list of reasonably frequent tasks that different stakeholders execute, and compare the execution in terms of # mouse clicks, cognitive overhead ('easy' -> 'difficult'), flexibility (options to configure to personal taste),

1. Stakeholders

  • Students
  • Teaching Faculty
  • Administrative Assistants
  • System administrators

2. Experimental setup

a) Design tasks

  • Each task should have a single clearly prescribed goal which mimics a commonly executed task performed by a stakeholder in the real world.
  • The description should give sufficient details that are normally needed to complete the task. The subject should not have to spend time making up dates, times, and other inputs required.
  • The tasks should be sufficiently generic but useful and not focussed on any particular characteristic of an LMS.
  • Different tasks for different stakeholders (total # of tasks ?)

b) Equipment Setup

  • Mac & PC with latest browsers (Firefox & Chrome), and screen recording software.
  • Access to all LMSes being compared

c) Methodology

  • Screen recording s/w switched on and subject logged in to LMS.
  • Subject reads the task (online & paper copy available) and attempts to complete it within a certain time.
  • Online help documentation available open in a separate browser window
  • Repeat same task with other LMSes under comparison.

3. Comparison metrics

  • Level of Expertise (self assessed). Subject self assesses their level of expertise with the LMS at hand on a Likert scale (expert, confident, need help, utter novice / Not used LMS before)
  • Task completion. Self assessed - were you able to complete the task to your satisfaction ? And assessed by experimenter - % of task goals actually completed.
  • Cognitive overhead. Ease of accomplishing task rated by subject - Very easy, Easy, OK, difficult, very difficult. Compare to experimenter's rating.
  • Flexibility - number & usefulness of options to configure to personal taste. Rate by subject & experimenter.
  • # clicks / drags required to complete task (assessed by experimenter from recording)
  • Total time taken to complete task (assessed by experimenter from recording)

4. Statistics & Outcomes

  • Test each task with a number of subjects.
  • Aggregate each metric to mean +/- Standard deviation.
  • Assign a relative weight to each metric and compile an overall Usefulness value for the Task per LMS.
  • Compare Usefulness values between LMSes and determine level of difference required to be significant.
Average of ratings: -
In reply to Mark Pearson

Re: Objective Comparison Learning Management Systems

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Have you considered Hosted Moodle? 

Average of ratings: -
In reply to Marcus Green

Re: Objective Comparison Learning Management Systems

by Mark Pearson -

No. We're hosting Moodle on site right now. There is no way I could handle a hosted Moodle solution. I regularly poke around in a copy of the database, I install a selection of plugins which I can pre-test, etc etc. I'm not altogether very happy with the prospect of losing this kind of access & control.

Average of ratings: -
In reply to Mark Pearson

Re: Objective Comparison Learning Management Systems

by Michael Penney -

I regularly poke around in a copy of the database, I install a selection of plugins which I can pre-test, etc etc. I'm not altogether very happy with the prospect of losing this kind of access & control.


Seems like full read/write db access and plug-in installation/removal at will should be on your list of required LMS features then? 

Average of ratings: -
In reply to Mark Pearson

Re: Objective Comparison Learning Management Systems

by Derek Chirnside -

I'm not sure why you are mentioning me in particular Mark.

I agree with your basic approach.  The concept of workflow: execution of basic tasks.  To do this with depth is not a trivial task.

Not all features are the same.  eg

  1. Drag and drop images: can you do this to create content, make a post
  2. Quick grading.
  3. Printing of quizzes, content like books.
  4. Managing video deployment
  5. Roles
  6. Ease of One<>one communication
  7. Reporting
  8. Edit in place

These can translate to a few simple tasks.

But, as we have said before, one persons Critical is another person's Trivial.  smile  https://moodle.org/mod/forum/discuss.php?d=180865

So just to clarify Mark: what are you looking for?  Help with compiling the indicator task list?

-Derek


Average of ratings: -
In reply to Mark Pearson

Re: Objective Comparison Learning Management Systems

by ryan sanders -

hardware and it staff....

my initial first reaction. is i just "cringed in fear"  when i saw "offsite"  and in that is dealing with pure internet connection and amount of data you can send and receive (monthly cost in bills) both at server and at school itself.  and dealing with sudden loss of internet due to someone cut the internet some place working along the road. 

if you have any major video / sound, or other large files, i would be uneasy putting things "offsite" 

if you can run your own say flash or red5 server on site. and if need be deal with different plugins that rely on these type of server software to work. your internet bill will go down most likely, along with much faster access, and not tied to your internet connection.  the issue is. all the extra work maintaining servers on site, and needing personal onsite that can deal with issues.   larger brick / mortar schools more likely have enough students and over all cash flow to support onsite setups, more so collage / universities.    small schools and/or were majorty of students are all over the place around the world, more likely better off with something "off site"  on a hosting provider setup.  were the servers themselves are better situated right up on the main high speed internet lines and multi backup internet connections. 

if your IT staff / hardware admin is worth their salt, they should have some local warehouse connections for computer parts, perhaps setup with a local IT company for spare computer parts.  granted now a days with 1 day shipping at a cost. and internet of what it is today....  

but there are a few companies dedicated to "wholesale" and for brick and mortar schools, needing a variety of equipment, can be rather nice for server racks, routers, switches, load balancers, storage raid arrays, etc... vs trying to go to say apple, dell, etc,.. directly, and attempting to get a "pre installed os, and reinstall software of your choice installed"  for just student laptops

==============

for the rest of your comparison. i do not want to touch!  it can be teeth pulling at times with some companies (no idea on actual LMS companies) but other companies i have dealt with for other things, and getting a demo i can actually install and try for myself on my own test computer. with full rights to do anything i want.  can be a challenge. ((marketing at its finest))  and companies trying to keep more of a tight fist on their software.  so as to prevent their software getting on torrent websites and like and having folks use it illegally.  as a result have seen many youtube videos, or talked over phone with someone, but reality never really got a good grasp of companies software. because it was too much of a hassle to get an actual demo i could install and try it out for myself. 

many things you simple do not understand, until you actually do it yourself. and finding out how much of a hassle it is to do this or that. and go through the actual program. 

Average of ratings: -
In reply to ryan sanders

Re: Objective Comparison Learning Management Systems

by Derek Chirnside -

Ryan, I'm not sure whether your second comment is correct. It (maybe) misses the point.  In this case the comparison Mark is interested in is between Moodle and Canvas.  Both of which are open, and you can try to the nth degree.

Actually: I'm not sure whether there is an 'objective' way to do this.  I could find people I respect who would study the question and come up with different answers.

My my question to Mark remains.  Do you want help with use cases? Or??

My conspiracy hat comes on here.  I can just imagine a keen person in your orbit seeing 97 seconds of another LMS (in this case Canvas) and they say Wow, it can do that? (like maybe It can upload a video in one click? or Look, students can sign up for their own groups! or Hey, cool, I can insert cross site scripting whenever I want!!) and then "Lets change to this new LMS!!"

-Derek


.

Average of ratings: Useful (1)
In reply to Derek Chirnside

Re: Objective Comparison Learning Management Systems

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

I would agree that it is not possible to have an objective comparison, it really depends on what is important to you. 

Average of ratings: -
In reply to Derek Chirnside

Re: Objective Comparison Learning Management Systems

by Mark Pearson -

Yes, I am interested in a Task List. And I'm rethinking how to go about the process of comparing. Here's a start.

Background assumptions

LMS used at different levels of sophistication and depth for support of face to face teaching at a Liberal Arts College. We may get into Online courses & Hybrid courses, but F2F is where we are.

We assume class meetings three day per week - Monday/Wednesday/Friday - and a 2 day mid-term break and 1 week Spring break/]

Assessment

For the gradebook :

  • Written Assignments : 20%
  • Weekly reflective Journal : 10%
  • Discussion prompts & commentary (number & quality) : 10%
  • Quizzes : 10%
  • Class Attendance & Participation : 10%
  • Group Project : 20%
  • Exams (Midterm & Final) : 20%

Total : 100%

Task List

  1. Set up a course with 15 weekly sections/modules labelled with dates
  2. Configure the Gradebook with the above categories
  3. Add a number of readings (say 3 pdf files) to different weeks
  4. Add a folder of readings (different set of related pdfs) with a description that is displayed.
  5. Add 3 tracks of music to play inline (ie not downloadable).
  6. Add Youtube & Vimeo video to play inside LMS (ie not just a link).
  7. Scaffold the context of the reading by describing the topic context for the associated class. So we'd want the topic description (given) and a link to the reading.
  8. Track student progress of doing those readings (we do have Progress Bar installed)
  9. Attendance. Configure attendance for all class meetings so that 3 absences lose a grade point (ie A -> B, B -> C etc). This should go straight into the gradebook.
  10. Class participation. Record class participation on a 3 point scale - fully engaged, moderately engaged, unengaged/asleep with appropriate penalties.
  11. Students will need to do a reflective journal entry every week. Set this up
  12. Create four written assignments to be due at equal intervals.
  13. After each class a nominated student must make a controversial posting in a Forum to which everyone else replies.
  14. There are weekly pop quizzes of 5 questions at random point during the week to assess whether students have been doing the readings.
  15. Two in class exams, one at mid term, the other a final.
  16. Multimedia group project. Project content & presentation is peer assessed. Groups are assigned by the Teacher.
Average of ratings: -
In reply to Derek Chirnside

Re: Objective Comparison Learning Management Systems

by Mark Pearson -

" I can just imagine a keen person in your orbit seeing 97 seconds of another LMS (in this case Canvas) and they say Wow, it can do that? (like maybe It can upload a video in one click? or Look, students can sign up for their own groups! or Hey, cool, I can insert cross site scripting whenever I want!!) and then "Lets change to this new LMS!!" "

That's exactly it! Quote : "Moodle is clunky" "I hate Moodle" 

Well, some folks are going to try out Canvas over our Spring Break. I think they'll experience a cold shower. Adding a resource in Canvas is like it used to be in Moodle 1.9, the only upside being that you can drag & drop into the Files. But not into the main course display. And I'm finding lots of things that folks like to do -- such as customizing their course with a different  course theme (complaints about the paucity of themes since I pulled a whole bunch as being incompatible), adding pictures to the course outline, changing colours of assignments etc -- none of which are possible in Canvas.

As far as the task list goes, in order to level the playing field I was thinking of writing step by step documentation for each task item (looks like quite a big job). Using something like Screenflow (Mac) or Camtasia (PC) I'd record the subject doing each task from the documentation with supplied files. Then we can measure how long it takes them and the number of mouse clicks. I think that part is objective.

Thanks for the helpful feedback Derek. I always enjoy reading your thoughtful posts.

Mark

Average of ratings: -