Comparisons and advocacy

Blackboard vs. Moodle study

 
Picture of Michael Penney
Blackboard vs. Moodle study
Group Core developers

A Humboldt course designer and faculty member recently concluded a study where they taught the same class in Blackboard 6 and Moodle 1.3.

http://www.humboldt.edu/~jdv1/moodle/all.htm

The students were randomly divided into two groups on the first day, with half in Moodle, half in Blackboard.

Interesting data including grades, participation, and student satisfaction.

Joan will be presenting this at the CATS (California State University Academic Technology Staff) 2005 conference in Sacramento.

http://cats.cdl.edu/catsconf2005

 
Average of ratings:Useful (1)
Picture of Don Hinkelman
Re: Blackboard vs. Moodle study
Group Particularly helpful MoodlersGroup Plugin developers
Very useful report, Michael!   Will there be a text version published somewhere?   I think it was a fair comparison but at the beginning it appeared a little Moodle-biased, where Moodle was described as "free!".  No mention was made of the clever, humourous, and good-looking community of supporters.  smile
 
Average of ratings:Useful (1)
Not a PHM :-)
Re: Blackboard vs. Moodle study
 

I "feel" there may be some bias here as well (please notice I say "feel" since I haven't seen the data to develop a more informed opinion). I don't think the delivery system can make that much difference -- especially between systems like Blackboard and Moodle. Now, if we were comparing online learning though email and Moodle, maybe smile

The difference comes from organization, pedagogy, facilitation. etc...in short, the way the system is used, not whether one system is free and one is not.

I like Moodle and I use Moodle, but I'm not ready to conclude that there is enough difference in the technology between Moodle and Blackboard for one to cause a superior online learning experience.

Just my opinion smile

Steve  

 
Average of ratings:Useful (1)
Picture of Don Hinkelman
Re: Blackboard vs. Moodle study
Group Particularly helpful MoodlersGroup Plugin developers
Hi Steve,

You are right to point out that this study cannot judge whether Moodle or Blackboard is superior.  But look at the research question.  It is about whether a free software can compare with a commercial one.  I think it was an excellent question that can be tested.  The result was conclusive.  Yes, a free system compares favorably with commercial one in the opinions of students and teachers who have never tried either.

As for the pedagogy, the research design is very interesting.  The teacher had long experience teaching the subject, but had not taught the class online, and was unfamiliar with both systems.  I think I would love to read her/his diary of experiences using both for the first time.  An excellent and rare inside view to a new teacher testing a new platform!
 
Average of ratings:Useful (1)
Picture of Ger Tielemans
Re: Blackboard vs. Moodle study
 

Yes,

even if you see this as a "draw" then your conclusion should be:

"What a pitty, all that money spend for the same result..."

 
Average of ratings: -
Art Lader
Re: Blackboard vs. Moodle study
Group Documentation writers

That is exactly how I see it, Ger... Moodle holds its own and costs zero dollars!

-- Art

 
Average of ratings: -
Picture of Cris Fuhrman
Re: Blackboard vs. Moodle study
 
I just want to chime in on the "zero dollars" remark. I'm sure that Martin doesn't see it as costing "zero dollars" in terms of his time. wink (No offense intended to Art)

I think that the results of the comparison mentioned are yet another proof that Open Source collaborative projects are capable of competing with high-dollar commercial ventures.
 
Average of ratings: -
Picture of Ger Tielemans
Re: Blackboard vs. Moodle study
 

No, it costs the same as commercial products: you buy/hire a server, you buy/hire a backup system, you hire educational support staff (you save on training at basic level because Moodle is so easy, but "loose" this money on training moodle on advanced level: 30+ modules, 10+ blocks, all these extra's smile)

But you save on license costs, EVERY YEAR AGAIN: So if I compare Moodle with the cheapest alternative, I can spend 10.000 euro EVERY YEAR, to hire Martin (i tried it once but this man is so busy!) or hire other companies to make local adapations that fit our local educational wishes!

 
Average of ratings: -
Not a PHM :-)
Re: Blackboard vs. Moodle study
 

I agree completely.

Steve

 
Average of ratings: -
Not a PHM :-)
Re: Blackboard vs. Moodle study
 

I agree completely Don...that is an excellent question that can be tested. But on the surface, when you look at the specific questions that were asked of students you see things like:

Did Blackboard/Moodle enhance instruction?

Technology-based activities developed problem-solving skills?

Communication tools: Interaction with classmates?

Communication tools: Interaction with instructor?

Etc.....

And on everyone of those questions (and nearly all of the questions), Moodle outperformed Blackboard. The results from the questions, seem to indicate that Moodle not only holds its own with Blackboard (which I believe it does), but that Moodle is better at allowing for the "development of problem-solving skills", "Interaction with classmates", "interaction with instructor", etc.

I just don't think that the differences in the "platforms" (the technology) can account for that. It seems to me the research question is more of a "technical question", and the results seem to go beyond answering that technical question.

Again, I teach on a Blackboard campus, but I have started using Moodle because I like it, I like the interface, I like the fact that I can do everything (or nearly everything) that I can do in Blackboard with an open source system. I will continue to use Moodle and am teaching public schools personnel how to incorporate it in their schools. I would like to see our campus move away from Blackboard and to Moodle, but that is a battle I'm not willing to fight at the moment...and I don't think I could fight that battle on the grounds that it would allow faculty to create a superior learning environment.

I can't say that Moodle allows me to create a "superior" online learning environment. I can teach online courses just as effectively in Blackboard as Moodle, but I can do it in Moodle for free and if I want to customize something, with a little knowledge I can do that.

I think the study is very interesting...I'm just a little suspecious of the results when I see Moodle being judged as superior in things like "developing problem solving skills"....whatever the teacher is doing in Moodle to develop those skills, can be done in Blackboard...in my opinion.

Steve

 
Average of ratings: -
Picture of Don Hinkelman
Re: Blackboard vs. Moodle study
Group Particularly helpful MoodlersGroup Plugin developers

Steve wrote:

...when you look at the specific questions that were asked of students you see things like:

  1. Did Blackboard/Moodle enhance instruction?
  2. Technology-based activities developed problem-solving skills?
  3. Communication tools: Interaction with classmates?
  4. Communication tools: Interaction with instructor?

The results from the questions, seem to indicate that Moodle not only holds its own with Blackboard (which I believe it does), but that Moodle is better at allowing for the "development of problem-solving skills", "Interaction with classmates", "interaction with instructor", etc.

Yes, you are exactly right.  I think the first two above questions are particularly inappropriate questions to be asking on a survey of student impressions.  To answer those questions a different research design would be necessary.  So all of us should beware of making too many conclusions from the study. 

At least we can say, "A teacher and her 35 students, all new to Moodle and Blackboard, found that a commercial LMS (BB) and an open source LMS (M) were both able to deliver similar functionality with slightly higher levels of satisfaction amongst the Moodle-using students".   Would you agree with that conclusion?

 
Average of ratings: -
Picture of Michael Penney
Re: Blackboard vs. Moodle study
Group Core developers
Hi Don, I don't see how those questions are innappropriate measures of college level student satisfaction. If you just asked them whether they liked a course delivered via and LMS better than face to face, you might not have gotten very deep opinions.

Instead, they were asked in various ways, to evaluate whether the online experience produced equal or superior learning vs. face to face, which I think is actually quite an elegant way of getting at how they really thought about it.

Of course these questions are all clearly measures of what the students felt and thought about the particular learning experience, not direct measures of actual learning.






 
Average of ratings:Useful (1)
Picture of Don Hinkelman
Re: Blackboard vs. Moodle study
Group Particularly helpful MoodlersGroup Plugin developers
Hi Michael,

This is a wonderful study.  I am wondering if there will be a text version published in proceedings or some journal.  Do you know?

If so, I would ask Kathy and Joan to consider a few very minor comments.  First, we should imagine that the audience will be open source skeptics--so let's remove any Moodle-evangelistic style (ie:  "free!").  Present it all cold and undramatically.  Second, as Steven warned, it is easy to jump to superiority conclusions.  Soften the conclusions, perhaps as I wrote.  Third, the questions rightly focus on "satisfaction", except one or two such as "Technology-based activities developed problem-solving skills?"  which was confusing to me and requires students to judge the activities on something other than satisfaction.  Finally, I think you will be questioned on using quantitative statistics on the graded results.  Corelations on a 35 person sample?  Useless in my view, but on the other hand, the graphs look beautiful (maybe some skeptics will like that).  Personally, I would omit them and expand the qualitative analysis--maybe add richer descriptions of compromises, problems, or revelations as the developer/teacher moved into online tools, and compare implementation differences of BB and M.

Most of all, congratulations to that team for taking the effort to do this. smile Please tell us how the presentation goes and pass on any further documents.  And a big thanks to all the Humboldt State folks for all the new modules, functions and upgrades we are getting in version 1.5!   big grin
 
Average of ratings: -
Not a PHM :-)
Re: Blackboard vs. Moodle study
 

Ditto...I think it is a good study as well....just need to tighten-up any conclusions that may jump to show the "technology" results in better problem-solving skills....like Don pointed out, the results should focus on "satisfaction". Just my opinion.

Steve

 
Average of ratings: -
Not a PHM :-)
Re: Blackboard vs. Moodle study
 

Don,

Yes, I think you summarized this very well...that conclusion could easily be supported by the respones. "Higher levels of satisfaction" seems to be clearly evident.

Steve

 
Average of ratings: -
Picture of Mark Stevens
Re: Blackboard vs. Moodle study
Group Moodle Course Creator Certificate holders
Michael,

Thank you so much for sharing this information, and thank Kathy and Joan for all of their hard work.  If you have any other information regarding this study, I think a lot of people would love to see it smile

Does anyone else know of any research regarding head-to-head studies?
 
Average of ratings:Useful (1)
Picture of Dave Bethany
Re: Blackboard vs. Moodle study
 

We are sort of doing it now, but not a same-semester comparison.

I am producing a very involved, upper-level capstone course. I run it on my home computers because the university wouldn't allow us to install Moodle on their servers. 

Over the past 6 years this course has resided on WebCT, Blackboard and now Moodle. This course has always been used as an ongoing research project into distance learning and autonomous learning. It must be understood that this course is never advertised as a "distance learning course" in the university catalog. The students learn of this on the first day when we meet in a physical classroom.

In the past, we have had a 50% student drop rate, each semester. This semester our drop rate was under 25%. All content is the same, what we changed was the delivery system (using Moodle now) and we added in the option of attending the 5 live classes either online with our N*E*W system or in a physical classroom. The online classroom attendance has been very low.

The students who have previous experience with WebCT or Blackboard have almost universally accepted the Moodle system as a better way of doing things. The students without prior DL experience comment on how easy and intuitive things are. In the next 3 weeks we will be doing a true survey with the students addressing all of these issues.

Is Moodle perfect? Not at all. I find it frustrating and clumsy the way things are displayed at times. But considering that the course has been running 24/7 for 4 months, without a single software related interruption is amazing.

Dave

 
Average of ratings: -
Picture of Richard Treves
Re: Blackboard vs. Moodle study
 
Hi Dave,

This is very interesting, FYI dropout on 1st level Open University UK (distance learning) courses is about 50%.

I wonder how much an effect BB vs Moodle was compared to the effect of adding N*E*W (aside: I do wish they'd change that acronym - "hey, we have a new system for synchronous learning its NEW..."
"Yes, you said that already..."). 

Although I hate BB with a vengance c.w. moodle I have to say I think adding synchronous distance learning adds so much that this could account for a whole lot of your improved retention.  Did you ask any students about this?

Cheers

Richard
 
Average of ratings: -
Martin Dougiamas
Re: Blackboard vs. Moodle study
Group Core developersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Plugin developersGroup Testers
Very interesting, thanks!  I love seeing studies like this.  I've put this on the Moodle Buzz page.
 
Average of ratings: -
Picture of Cheri .
Re: Blackboard vs. Moodle study
 
This is pretty cool, Thanks for posting it. I was wondering about the differences!
big grin
 
Average of ratings: -
Claude Whitmyer
Re: Blackboard vs. Moodle study
 

Ok. I get that the concensus is that Moodle holds its own or perhaps even edges Blackboard out a little. But this begs the business questions that any Administrator is going to ask:

1) What is the true cost of installing, maintaining the software and supporting the users?

2) If we decide to switch to Moodle, what will it cost to convert all of our existing courses?

We need clear answers to these questions to make the case for switching (I think. Am I wrong?).

claude

 
Average of ratings: -
Picture of Richard Wyles
Re: Blackboard vs. Moodle study
 

From my perspective the answer to question 1) is more or less moot. There may be some variation in the costs of maintenance dependent on the system interfaces and level of customisation but overall I tend to regard maintenance costs as comparable across comparable application software.

The 2nd question largely depends on the format of your course materials. Late last year we migrated from a legacy system to Moodle for 6000+ courses supporting 35,000 students.

We didn't migrate any forum postings - allowing access into the old online campus for a set period. Unfortunately we didn't have our courses as SCORM compliant but still the migration exercise still went relatively smoothly.  

All the best,

Richard   

 
Average of ratings: -
Claude Whitmyer
Re: Blackboard vs. Moodle
 

Richard,

Good points.

I'm ok with assuming that the costs of supporting any CMS are going to be about the same (unless your talking about one that is technically challenging). However, the waters are somewhat muddied by the fact that while Humboldt is paying $8.600/year for it's Blackboard license, there IS some maintenance and technical support included in that. So the full compliment of maintenance and technical support that Humboldt receives = Bb M&S plus internal M&S.

So the question remains, is it going to cost Humboldt $8,600/year to maintain and support Moodle? Or will there be SOME savings over the Blackboard costs to account for the pure cost of the License (the cost that is not maintenance or support)?

Also, as far as #2 goes, you say the migration went relatively smoothly, but how much did it cost?

I find that if we assign internal people to a project, such as a Moodle migration, and don't calculate the proportion of their salaries that should be attributed to that project, then we are adding yet another source of mud, making it difficult to create accurate estimates of true costs. We need true costs to make our business case to administration. It's too easy for administration to shoot down our argument if we just say "It went smoothly during the pilot."

When you migrated a typical course how many people were involved? How many hours did it take? What would that proportion of their salary be? These are the type of questions that will help make the business case for Moodle (I think...)

claude

 
Average of ratings: -
Picture of Richard Wyles
Re: Blackboard vs. Moodle
 

Hi Claude,

In answer to Qu1), the cost gap would theoretically be the license fee - assuming all other things are equal. Bear in mind the $8,600pa for BB only gives you the basic version. 

We haven't stopped there though - the NZOSVLE project now has six higher education organisations supported by a high availability cluster and technical team - so the cost of ownership drops for hardware investment, technical support etc. We've been able to drop the support costs for an institution of 2000 learners to around $US300 per month for 24*7 technical support, hardware, bandwidth etc.

For Qu2), I'd estimate approximately 3 person-months - we could accurately define this from time-sheet reports but it would vary from organisation to organisation.

I believe the business case needs to take into account a strategic shift in using FOSS - the freedom to pursue different online pedagogies. The ROI won't be just the $ but quality improvements by not being contrained by the commercial imperatives of a vendor. As an example we were motivated by being able to supoort local and Pacific languages like Maori, Tongan, and Samoan - we'd be a long time waiting for this to be supported in a commercial application. A second example is that we've customised the Assessment module to support the particular workflows of the Open Polytechnic - being a large distance learning provider.

I hope this helps - business cases can be tricky beasts - not everything is black and white.

 
Average of ratings: -
Picture of Michael Penney
Re: Blackboard vs. Moodle
Group Core developers
We had to pay an extra $4000/year to 'enable' SSL this year.

It costs us alot more than 12,600 to maintain and support BB, and due to the inability to automate repetitive tasks such as enrollment and (esp.) course set up, it costs us much more to maintain and support BB than Moodle.

With Moodle we get BB Enterprise level features such as LDAP integration for enrollment and course set-up (thanks to NZVLE!smile), quiz import formats, modifiability, etc.

So Moodle gives us Enterprise features without the (fixed->rising) yearly Enterprise license cost.

Moodle's free support is at least as useful as BB's (the best BB support is user-to-user forums--BB trouble tickets tend to take a while to get answered and longer to get patched). You can pay for Moodle support from a partner and if you don't like them, switch partners without switching LMS.

When you have all of your courses in Blackboard Enterprise and have it integrated with your SIS and portal, you have to pay whatever the license fee rises to, or lose your entire business model (or be forced to change rapidly and in an uncontrolled fashion).
 
Average of ratings: -
Picture of John Curran
Re: Blackboard vs. Moodle
 
Michael makes some excellent points concerning the open source approach.  The ability to switch support between different partners is particularly valuable and gives you to ability to source a partner that 'understands' what you are trying to achieve - a much more important attribute than simply having the technical know-how.
 
Average of ratings: -