Dear Moodle Community,
my Master's dissertation in Information Management will be partly a Usability Study regarding Moodle. The current working title is: "Make the most of your studies: A comparative usability study on the collaborative virtual learning environment Moodle with postgraduate students from Oil & Gas related courses".
The overall aim of this study is to find out how the collaborative virtual learning environment Moodle is integrated and embedded in the studies of postgraduate Oil & Gas related courses at The Robert Gordon University (Aberdeen, Scotland).
From the literature review I undertook during the proposal stage, I noticed, not many usability studies regarding virtual learning environment (and in particular Moodle) were undertaken.
So, my question is: Is there anyone out there, who does have experience? I would also appreciate, if any links or references to this question could be posted.
Many thanks.
Syndia
Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Syndia Lengyel -
Number of replies: 15
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Tim Hunt -
Two thinks I know of:
See http://www.pilpi.net/software/moodle_quiz_ui/
Also, there was a talk about some Moodle usability studies at the Brisbane Moodle Moot last October.
See http://www.pilpi.net/software/moodle_quiz_ui/
Also, there was a talk about some Moodle usability studies at the Brisbane Moodle Moot last October.
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Eoin Campbell -
I recently did a position paper on LMS usability, and found the references below.
Martin 2008 was the most interesting, and highlighted efficiency of use as a big problem area in Moodle, Sakai and dotLRN.
Contact me directly if you want a copy of my paper.
Bremer, D., & Bryant, R. (2005). A Comparison of Two Learning management Systems: Moodle vs Blackboard. Paper presented at the 18th Annual Conference of the National Advisory Committee on Computing Qualifications, Tauranga, New Zealand.
Graf, S., & List, B. (2005). An Evaluation of Open Source E-Learning Platforms Stressing Adaptation Issues. Paper presented at the Fifth IEEE International Conference on Advanced Learning Technologies (ICALT), Kaohsiung, Taiwan.
Krantz, P. (2007). Evaluation of WYSIWYG editors (2007). Retrieved 16/11/2008, 2008, from http://www.standards-schmandards.com/2007/wysiwyg-editor-test-2/
Machado, M., & Tao, E. (2007, October 10 – 13, 2007). Blackboard vs. Moodle: Comparing User Experience of Learning Management Systems. Paper presented at the 37th ASEE/IEEE Frontiers in Education Conference, Milwaukee, WI.
Martin, L., Martínez, D. R., Revilla, O., Aguilar, M. J., Santos, O. C., & Boticario, J. G. (2008, 6-7 November). Usability in e-Learning Platforms: heuristics comparison between Moodle, Sakai and dotLRN. Paper presented at the 7th European Conference on e-Learning (ECEL), Agia Napa, Cyprus. (https://adenu.ia.uned.es/web/en/system/files/openACS08_lmdrmormmjaocsjgb.pdf)
Pacurar, E. G., Trigano, P., & Alupoaie, S. (2005). A QTI editor integrated into the netUniversité web portal using IMS LD. Journal of Interactive Media in Education.
Pierotti, D. (1995). Heuristic Evaluation -A System Checklist. from http://www.stcsig.org/usability/topics/articles/he-checklist.html
Rusama, M. (2006). Microsoft Word template for making Moodle questionnaires in GIFT format. from http://www.soberit.hut.fi/sprg/resources/moodle/GiftConverter.html
Storey, M.-A., Phillips, B., Maczewski, M., & Wang, M. (2002). Evaluating the usability of Web-based Learning Management Systems. Educational Technology & Society, 5(3), 91-100.
Yatskovsky, V. (2006). Microsoft Word template for making Moodle quizes. from http://www.finemetronome.com/moodle/
Martin 2008 was the most interesting, and highlighted efficiency of use as a big problem area in Moodle, Sakai and dotLRN.
Contact me directly if you want a copy of my paper.
Bremer, D., & Bryant, R. (2005). A Comparison of Two Learning management Systems: Moodle vs Blackboard. Paper presented at the 18th Annual Conference of the National Advisory Committee on Computing Qualifications, Tauranga, New Zealand.
Graf, S., & List, B. (2005). An Evaluation of Open Source E-Learning Platforms Stressing Adaptation Issues. Paper presented at the Fifth IEEE International Conference on Advanced Learning Technologies (ICALT), Kaohsiung, Taiwan.
Krantz, P. (2007). Evaluation of WYSIWYG editors (2007). Retrieved 16/11/2008, 2008, from http://www.standards-schmandards.com/2007/wysiwyg-editor-test-2/
Machado, M., & Tao, E. (2007, October 10 – 13, 2007). Blackboard vs. Moodle: Comparing User Experience of Learning Management Systems. Paper presented at the 37th ASEE/IEEE Frontiers in Education Conference, Milwaukee, WI.
Martin, L., Martínez, D. R., Revilla, O., Aguilar, M. J., Santos, O. C., & Boticario, J. G. (2008, 6-7 November). Usability in e-Learning Platforms: heuristics comparison between Moodle, Sakai and dotLRN. Paper presented at the 7th European Conference on e-Learning (ECEL), Agia Napa, Cyprus. (https://adenu.ia.uned.es/web/en/system/files/openACS08_lmdrmormmjaocsjgb.pdf)
Pacurar, E. G., Trigano, P., & Alupoaie, S. (2005). A QTI editor integrated into the netUniversité web portal using IMS LD. Journal of Interactive Media in Education.
Pierotti, D. (1995). Heuristic Evaluation -A System Checklist. from http://www.stcsig.org/usability/topics/articles/he-checklist.html
Rusama, M. (2006). Microsoft Word template for making Moodle questionnaires in GIFT format. from http://www.soberit.hut.fi/sprg/resources/moodle/GiftConverter.html
Storey, M.-A., Phillips, B., Maczewski, M., & Wang, M. (2002). Evaluating the usability of Web-based Learning Management Systems. Educational Technology & Society, 5(3), 91-100.
Yatskovsky, V. (2006). Microsoft Word template for making Moodle quizes. from http://www.finemetronome.com/moodle/
In reply to Eoin Campbell
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Tim Hunt -
Oooh! thanks for this list. Now I just need to make time to read them.
In reply to Eoin Campbell
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Tim Hunt -
Google finds the following links for the other usability papers:
Bremer, D., & Bryant, R. (2005) http://www.naccq.ac.nz/conference05/proceedings_05/concise/bremer_moodle.pdf
Graf, S., & List, B. (2005) http://moodle.org/other/icalt2005.pdf
Machado, M., & Tao, E. (2007, October 10 – 13, 2007) http://collaboration.becta.org.uk/servlet/JiveServlet/previewBody/1240-102-1-1540/Blackboard%2520versus%2520Moodle,%2520Comparing%2520User%2520Experience%2520of%2520LMS.pdf
Storey, M.-A., Phillips, B., Maczewski, M., & Wang, M. (2002) http://www.ifets.info/journals/5_3/storey.html
Comments: (I kept these in the original order, but read them in a different order)
Bremer, D., & Bryant, R. (2005).
Says lots of nice things about Moodle (1.4.5)
Makes some specific points about what works well and what doesn't.
Graf, S., & List, B. (2005).
Another paper that says nice things about Moodle (1.4.1) without many specifics.
Krantz, P. (2007). Evaluation of WYSIWYG editors (2007).
Evaluation of WYSIWYG editors In terms of the accessibilty of the HTML produced. TinyMCE scores slightly higher than HTML area, which I suppose is good news for Moodle 2.0, but a somewhat limited comparison.
Machado, M., & Tao, E. (2007, October 10 – 13, 2007).
Rough conclusion: three out of four people surveyed though Moodle was better than Blackboard.
Martin, L., Martínez, D. R., Revilla, O., Aguilar, M. J., Santos, O. C., & Boticario, J. G. (2008, 6-7 November).
Very hard to tell what these people did, or what their numbers mean.
Pacurar, E. G., Trigano, P., & Alupoaie, S. (2005).
Not about usability!
Pierotti, D. (1995).
This is just a check-list, says nothing about Moodle.
Rusama, M. (2006).
Not about usability!
Storey, M.-A., Phillips, B., Maczewski, M., & Wang, M. (2002).
This compares Blackboard and WebCT to a static HTML site. Does not say very much of interest to Moodle.
Yatskovsky, V. (2006).
Not about usability!
My main conclusion is that what academics mean by a usability comparison, and what a software developer wants in order to find practical ways to improve the software, are very different things.
Oh, and my other conclusion is that the less of value that an academic has to write, the more their writing becomes pretentious drivel.
Bremer, D., & Bryant, R. (2005) http://www.naccq.ac.nz/conference05/proceedings_05/concise/bremer_moodle.pdf
Graf, S., & List, B. (2005) http://moodle.org/other/icalt2005.pdf
Machado, M., & Tao, E. (2007, October 10 – 13, 2007) http://collaboration.becta.org.uk/servlet/JiveServlet/previewBody/1240-102-1-1540/Blackboard%2520versus%2520Moodle,%2520Comparing%2520User%2520Experience%2520of%2520LMS.pdf
Storey, M.-A., Phillips, B., Maczewski, M., & Wang, M. (2002) http://www.ifets.info/journals/5_3/storey.html
Comments: (I kept these in the original order, but read them in a different order)
Bremer, D., & Bryant, R. (2005).
Says lots of nice things about Moodle (1.4.5)
Graf, S., & List, B. (2005).
Another paper that says nice things about Moodle (1.4.1) without many specifics.
Krantz, P. (2007). Evaluation of WYSIWYG editors (2007).
Evaluation of WYSIWYG editors In terms of the accessibilty of the HTML produced. TinyMCE scores slightly higher than HTML area, which I suppose is good news for Moodle 2.0, but a somewhat limited comparison.
Machado, M., & Tao, E. (2007, October 10 – 13, 2007).
Rough conclusion: three out of four people surveyed though Moodle was better than Blackboard.
Martin, L., Martínez, D. R., Revilla, O., Aguilar, M. J., Santos, O. C., & Boticario, J. G. (2008, 6-7 November).
Very hard to tell what these people did, or what their numbers mean.
Pacurar, E. G., Trigano, P., & Alupoaie, S. (2005).
Not about usability!
Pierotti, D. (1995).
This is just a check-list, says nothing about Moodle.
Rusama, M. (2006).
Not about usability!
Storey, M.-A., Phillips, B., Maczewski, M., & Wang, M. (2002).
This compares Blackboard and WebCT to a static HTML site. Does not say very much of interest to Moodle.
Yatskovsky, V. (2006).
Not about usability!
My main conclusion is that what academics mean by a usability comparison, and what a software developer wants in order to find practical ways to improve the software, are very different things.
Oh, and my other conclusion is that the less of value that an academic has to write, the more their writing becomes pretentious drivel.
In reply to Tim Hunt
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Syndia Lengyel -
Hey Tim,
thanks for the references. I am aware that the understanding / meaning of different words or activities is quite complex. In my work, I always experience that people are scared by databases because they worked with non-user friendly applications or software. What is "on top" of the database is something completely different, but it is quite hard to make them to understand that...
Anyway, I would not consider myself as a pure academic as I love to keep things practical and manageable. The guidelines for writing a Master dissertations are quite strict and one of them is to be focussed on a very specific sample or task and draw up recommendations from there. Personally, I would have liked to worked a bit in a different way. As it is know what features are available, it would have been of interest to embed these in the normal studies and then see how students feel about that. Currently, on our Uni, there is no need for students to use most of the features as students and lecturers are not aware of their existence or feel that this is too complicated. Therefore, feedback can be only gathered by "first time users". It is a bit the question, if this makes sense in a long term development for the application Moodle itself, although I am aware that you should use first-time users for a Usability Study. But more experienced users might have better ideas how to improve applications as many different factors influence the studies itself.
We shall see... Using Morae for User Testing will be quite interesting. But again, I do expect a lot of students from a non-european background (although I am based in UK) which could have an impact on Computer Literacy. So measuring the time and see how they perform, is not enough in my opinion. I feel that the learning style as well as the background impacts on the performance.
All the best. Syndia.
We shall see... Using Morae for User Testing will be quite interesting. But again, I do expect a lot of students from a non-european background (although I am based in UK) which could have an impact on Computer Literacy. So measuring the time and see how they perform, is not enough in my opinion. I feel that the learning style as well as the background impacts on the performance.
All the best. Syndia.
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Tim Hunt -
It's OK. once upon a time I did a PhD, so I know academics are taught to write like that. But then, having left that world, and looking back, it does seem a bit anachronistic.
It is probably OK (from the point of view of the Moodle project) if usability studies focus on first time users. We have plenty of experienced Moodle users participating in these forums, and we find out what they think. But there are very few first-time users here.
If you find any interesting usability problems, please to post about them here. We would like to know.
Good luck in your studies.
It is probably OK (from the point of view of the Moodle project) if usability studies focus on first time users. We have plenty of experienced Moodle users participating in these forums, and we find out what they think. But there are very few first-time users here.
If you find any interesting usability problems, please to post about them here. We would like to know.
Good luck in your studies.
In reply to Tim Hunt
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Olli Savolainen -
Thank you, Eoin for the references and Tim, for the summaries! Tim already mentioned last year's usability project I worked on, and I am doing a master's thesis on that now so those references will probably be of use! Currently the main research question is forming as What are the special characteristics of applying usability methods in an open source software (for example, compared to in-house) development project, where the knowledge about the users and their circumstances is distributed among developer-users. I have yet to further narrow it down though, it is unsure how much of the details of the actual usability work done will be included.
So Syndia, I would indeed be interested in hearing more about how your work advances. Also, I would be interested in knowing just what criteria they (who?) have set for your thesis, since nothing very strict has been provided to me, aside from the basic conventions of scientific writing. Thanks.
So Syndia, I would indeed be interested in hearing more about how your work advances. Also, I would be interested in knowing just what criteria they (who?) have set for your thesis, since nothing very strict has been provided to me, aside from the basic conventions of scientific writing. Thanks.
In reply to Tim Hunt
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Eoin Campbell -
The interesting point about "Rusama, M. (2006)"
and "Yatskovsky, V. (2006)" is that
they describe Word templates that are essentially work-arounds to overcome the lack of usability of certain features in Moodle (the UI for entering questions and answers).
and "Yatskovsky, V. (2006)" is that
they describe Word templates that are essentially work-arounds to overcome the lack of usability of certain features in Moodle (the UI for entering questions and answers).
In reply to Tim Hunt
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
Hi Tim and all,
I'm David Roldán one of the authors of the papers about usability named here: Martin, L., Martínez, D. R., Revilla, O., Aguilar, M. J., Santos, O. C., & Boticario, J. G. (2008, 6-7 November).
What we did was the following. We designed a usability test which was passed to each of Moodle, Sakai and dotLRN by different users and selected users evaluated the three platforms. Once we got these results, using heuristics, we calculated usability level of the three LMS under study.
If anybody on the community needs/wants further information about the study, it will be a pleasure for me to help you. You can ask me at darolmar@upvnet.upv.es o to http://moodle-vs-sakai.blogspot.com.
Thank you very much.
Dr. David Roldán Martínez
I'm David Roldán one of the authors of the papers about usability named here: Martin, L., Martínez, D. R., Revilla, O., Aguilar, M. J., Santos, O. C., & Boticario, J. G. (2008, 6-7 November).
What we did was the following. We designed a usability test which was passed to each of Moodle, Sakai and dotLRN by different users and selected users evaluated the three platforms. Once we got these results, using heuristics, we calculated usability level of the three LMS under study.
If anybody on the community needs/wants further information about the study, it will be a pleasure for me to help you. You can ask me at darolmar@upvnet.upv.es o to http://moodle-vs-sakai.blogspot.com.
Thank you very much.
Dr. David Roldán Martínez
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Wen Hao Chuang -
Hi Syndia you might want to talk to people who is working on FluidProject
They might have something for your literature review part. Hope this helps!
I was trying to find if there is a podcast or mp3 file for the "usability panel" that me, Anthony Borrow, Fluid project director Mara, and Gary Anderson did for the SF Moodlemoot 08, but I can't seem to find it. If I could find it I will post it here again.
Wen
They might have something for your literature review part. Hope this helps!
I was trying to find if there is a podcast or mp3 file for the "usability panel" that me, Anthony Borrow, Fluid project director Mara, and Gary Anderson did for the SF Moodlemoot 08, but I can't seem to find it. If I could find it I will post it here again.
Wen
In reply to Wen Hao Chuang
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Syndia Lengyel -
Hey Wen,
many thanks for the hint. I will try to find out more and if I find something, I can post it here.
Enjoy your weekend.
Syndia.
many thanks for the hint. I will try to find out more and if I find something, I can post it here.
Enjoy your weekend.
Syndia.
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Syndia Lengyel -
Dear All,
just to let you know that this dissertation was submitted on time, marked with an A and would be available on request. I am quite happy to upload this dissertation somewhere and link it to the Moodle Buzz, if someone could tell me where.
Please note:
This dissertation is 255 pages long (although more than half are appendices which might be of interest for developers). Also, I tried to capture progress on the Moodle Blog - so, please feel free to check there as well.
Title of the dissertation:
Make the Most of Your Studies: A Case Study and Web Usability Study Using the Collaborative Virtual Learning Environment Moodle with Postgraduate Students from Oil & Gas Related Courses.
Abstract:
The overall aim of this study was to draw-up recommendations on how the collaborative virtual learning environment ‘Moodle’ could be integrated and embedded more effectively in the studies of postgraduate Oil & Gas related courses at The Robert Gordon University. A single intrinsic case study approach was chosen. Data was collected using usability testing and observation, interviews, questionnaires, Freedom of Information Requests and additional data sources such as relevant conferences and secondary research for literature review.
Background information was provided by introducing the University’s infrastructure, relevant student demographics related to the investigated module ‘Business Essentials’ and analysing the Moodle implementation. By focussing on students’ and academics’ information needs, an evaluation of current processes and setups within the University’s network and Moodle was undertaken. Navigation and consistency as core usability issues were investigated and examples of a potential re-design provided. The e-learning aspect was evaluated by investigating interactivity in ‘Business Essentials’. By adapting existing learning content, a demonstration of using Moodle feature in the investigated module was given. Additionally, challenges and opportunities outside the Moodle environment were identified. Within the analysis recommendations were made as appropriate in the relevant sections.
Key findings of this study were that usability issues need to be addressed in order to provide meaningful information to individual user groups. Furthermore, existing Moodle features need to support interactive learning and address students’ learning style. Additionally, staff training regarding Moodle features, e-learning facilitation and search skills needs to be addressed. Finally, it was found that the University would benefit from establishing a Knowledge Management strategy. A summary of recommendation was detailed in a separate chapter.
Please do not hesitate to contact me, if you have futher questions.
All the best.
Syndia.
just to let you know that this dissertation was submitted on time, marked with an A and would be available on request. I am quite happy to upload this dissertation somewhere and link it to the Moodle Buzz, if someone could tell me where.
Please note:
This dissertation is 255 pages long (although more than half are appendices which might be of interest for developers). Also, I tried to capture progress on the Moodle Blog - so, please feel free to check there as well.
Title of the dissertation:
Make the Most of Your Studies: A Case Study and Web Usability Study Using the Collaborative Virtual Learning Environment Moodle with Postgraduate Students from Oil & Gas Related Courses.
Abstract:
The overall aim of this study was to draw-up recommendations on how the collaborative virtual learning environment ‘Moodle’ could be integrated and embedded more effectively in the studies of postgraduate Oil & Gas related courses at The Robert Gordon University. A single intrinsic case study approach was chosen. Data was collected using usability testing and observation, interviews, questionnaires, Freedom of Information Requests and additional data sources such as relevant conferences and secondary research for literature review.
Background information was provided by introducing the University’s infrastructure, relevant student demographics related to the investigated module ‘Business Essentials’ and analysing the Moodle implementation. By focussing on students’ and academics’ information needs, an evaluation of current processes and setups within the University’s network and Moodle was undertaken. Navigation and consistency as core usability issues were investigated and examples of a potential re-design provided. The e-learning aspect was evaluated by investigating interactivity in ‘Business Essentials’. By adapting existing learning content, a demonstration of using Moodle feature in the investigated module was given. Additionally, challenges and opportunities outside the Moodle environment were identified. Within the analysis recommendations were made as appropriate in the relevant sections.
Key findings of this study were that usability issues need to be addressed in order to provide meaningful information to individual user groups. Furthermore, existing Moodle features need to support interactive learning and address students’ learning style. Additionally, staff training regarding Moodle features, e-learning facilitation and search skills needs to be addressed. Finally, it was found that the University would benefit from establishing a Knowledge Management strategy. A summary of recommendation was detailed in a separate chapter.
Please do not hesitate to contact me, if you have futher questions.
All the best.
Syndia.
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Frank Ralf -
Hi Syndia,
Congratulations! I really would like to have a look at your work. You could upload it under http://moodle.org/mod/glossary/view.php?id=2757 and link to it at Usability FAQ and/or Development:Usability.
Cheers,
Frank
Congratulations! I really would like to have a look at your work. You could upload it under http://moodle.org/mod/glossary/view.php?id=2757 and link to it at Usability FAQ and/or Development:Usability.
Cheers,
Frank
In reply to Syndia Lengyel
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Martin Dougiamas -
Hey Syndia - excellent, well done!
I'm really looking forward to reading this. If you email a pdf to me I'll be happy to host it on moodle.org and list it under Buzz.
I'm really looking forward to reading this. If you email a pdf to me I'll be happy to host it on moodle.org and list it under Buzz.
In reply to Martin Dougiamas
Re: Usability: Moodle - Usability Study (for an MSc): Anyone experience?
by Syndia Lengyel -
Dear All,
the dissertation is can be accessed via Moodle Buzz. Martin, thanks for hosting it. Frank, could you please link the Moodle Buzz page to the pages you suggested - I didn't want to "crash" anything.
As the dissertation is quite long, I feel it is more appropriate to link to a summary from which people can chose, if they which to download it.
All the best.
Syndia.