CBM is particularly valuable in formative exercises, rewarding students who can distinguish between those of their answers that are reliable and those that are uncertain. The idea is to encourage deeper reflection about the basis of their knowledge and reasoning. Students do need practice with CBM to optimise performance, but given this (and even without it in some research studies) it has improved the statistical reliability of exam data substantially. The remarkable thing about it is how easily students immediately understand what it's all about and find it an aid to critical study.
Here now is a proper 'export' plugin module for running Moodle questions with Certainty-Based Marking. This handles True/False, Multichoice, Short Answer, Numerical, Match, Order and Description Qs and has been developed and tested with Moodle 1.6 and 1.7. Please of course report any problems if you try it ( to email@example.com , or here).
Expand the attached file (cbm.zip - also available with any future corrections or updates at www.ucl.ac.uk/lapt/moodle/cbm.zip ) into your folder ../question/format/ to make a sub-folder called cbm, which will contain two files (format.php, run.php).
Now use the developers' "export" facility to try out Moodle Qs directly: the option 'cbm' appears under 'export questions to file' in editing mode. You can save the formatted (.js) file for future editing and use with CBM.
To use CBM with your students you can place a .js exercise file (with any locally referenced graphic files) onto a website and reference it by creating a suitable URL: http://www.ucl.ac.uk/lapt/?mySite/myFile.js You can include the student's username using the 'parameters' options when setting up the link in Moodle, with the name 'u'. Marks are saved on the CBM-LAPT system and can be accessed with suitable authorisation. If, as I imagine, Moodle has a way of uploading these, then they could be entered into a Moodle grade book. [I simply haven't investigated this at present - we do this at UCL and Imperial with WebCT.] If you want to run CBM files locally rather than from UCL, then contact me. There are many options.
There are lots of examples (in several disciplines) in UCL's CBM open dissemination site at www.ucl.ac.uk/lapt (click Exercise Menu and then e.g. for calculation or critical thinking Qs, try "Maths in Med Sci" or BMAT1).
For anyone not sure how to follow the instructions for trying out CBM with your own Moodle Qs (unzipping the export plugin into your Moodle system and doing export - cbm) - please send me a file with your Qs e.g. in XML or GIFT export formats or even just plain text (to firstname.lastname@example.org ) and I'll easily enough set up a (private) link for you to try. It is surprisingly little work.
CBM isn't integrated in Moodle as a presentation and scoring procedure at present - you need to link outside Moodle. It may be worth trying to size up how much work would be involved in this, and I'd be happy to work on this with anyone with more Moodle experience. There was preliminary discussion at http://moodle.org/mod/forum/discuss.php?d=59739#277014 but a smooth CBM option available for any Q type would need some careful strategic thinking.
Tony I am interested in using moodle and CBM. I have studied works of Bruno, Hunt and Leclercq with much fascination. I have recently viewed your web page and your research. I am interested in a moodle plug-in for CMB as well. I am trying to find a way to use CBM in my Economics class in an efficient way, but I have had little success. What progress have you had in working on a plug-in for CBM?
Hi! I'm working solidly on this at present. Relevant posts have shifted mostly to a thread http://moodle.org/mod/forum/discuss.php?d=77290 along with related issues. There is an open trial site at www.ucl.ac.uk/lapt/moodle19/moodle which you should look at, and an earlier one for Moodle 1.7 at www.ucl.ac.uk/lapt/moodle17/moodle that could have all the same features, but I simply haven't updated yet. Both are visible (as a student) with userid=password=moodler, and parts of the 1.9 site as a teacher with userid=mdlteacher password=mdl19 . Send me a backup/XML export of a course or quiz (ucgbarg at ucl.ac.uk) and I'll put it up for you, with private key access.
CBM requires changes in core code, and can't be handled simply as a plugin unfortunately. I'm working on 1.9 so that people can consider including these changes in the core. The current 1.9 modified files are zipped at www.ucl.ac.uk/lapt/moodle19/moodle.mod.zip . There are actually a number of changes and bug fixes there that could be relevant beyond CBM, though at present I've written most of the code so that functionality is unchanged if CBM is switched off. Note that I'm doing this work for Moodle users in general, not for UCL and Imperial College, where LAPT remains the CBM workhorse.
Another strategy for people with Moodle is to use the simple CBM export plugin at http://www.ucl.ac.uk/lapt/moodle/cbm.zip . If you expand this (at moodle/question/format/type/ ) it allows you to export a category with your Qs to run immediately using the LAPT programs from UCL's server. You can also copy or download the exercise text in LAPT format for editing and tailoring for use in a specific context. Moodle is great for a multitude of admin features, but for formative exercises LAPT is usually more convenient, flexible and efficient. You can't at present return LAPT grades automatically into a Moodle gradebook as we do with WebCT4.1, and with a cursory look at 1.9 I'm worried that it isn't clear that this is going to be at all straightforward. I hope I'm wrong. Moodle mustn't go the route of thinking that the only things worthy of attention happen within Moodle!
Have you noticed problems when trying to add the quiz activity to a course? Post-adding your CBM mods?
You made a great work ! Do you know if you code will bee integrated in de core of Moodle ?
I am not a specialist and I have a question : why do you use a three levels scale to measure the confidence. Is it possible to configure that ?
Hi! Glad you find this potentially useful. The aim at the moment is to try and get it into Moodle 2.0 core. I'm working on combining a facility for CBM with various related facilities that seem useful even when not using CBM (e.g. immediate feedback, simple negative marking, distinguishing unresponded items from incorrect responses, comments on Qs).
There will be scope to configure the CBM marking scheme somehow, but I want to keep this simple and not allow people to set CBM schemes that are irrational, in the sense that students might be offered options that it is never in their interests to use. This is discussed in a number of papers on the UCL CBM website: www.ucl.ac.uk/lapt . I'd be interested to know what you have in mind.
Rationale for the options we use is that 3 is a manageable set: mid, high and low. Two seems a bit minimal, 4 becoming a tax on the brain! As for the specific marks (1,2,3) and penalties (0,-2,-4), they set simple criteria for using higher confidence levels: odds > 2:1 (P>67%) for C=2 (mid), and > 4:1 (P>80%) for C=3. These levels work for True/False as well as MCQ or open question types (NB you can't ever believe P(correct)<50% for a T/F question). If you don't think the odds on your being right are > 2:1, then you don't really know the answer in anybody's book - even if you do get it right!
Anyone who wants to suggest how best to develop these issues - do please contribute your suggestions, in the forum or to me. The code I use at www.ucl.ac.uk/lapt/moodle19/moodle is available there (though it needs care to patch to installations of different vintage). If you want more easily to try out CBM with your own exercises, the simplest things is just to email me a course backup and I can put it onto that site for you with a private key.
Something I don't understand. What software does one need to apply a patch to an existing set of files? I can make patches with "cvs diff" OK, using Tortoise, but Tortoise doesn't seem to have a command for applying the patch, except 'commit' which so far as I can see only works if you have access to a repository. It's not the most friendly software in the world!
Eclipse certainly can.
There is a program called WinMerge which is very nice for comparing sets of files. I think it does now have apply patch functionality, but I think it was a recent addition.
There is always the command line 'patch' program.
The safest way is doing it manually with a compare file software.
You have not so many files in yor project.
I just download your code to see how it modify the actual moodle code and I notice that you made several modifications for your CBM project but also for "bugs" in moodle code.
Can you build a version where the modifications are just the ones necessary for CBM with some docs or comment lines in the code.
Moodle is always in development of new e-learning features as long as they are useful and more or less compatible with the actual code.
P.S. more compatible means less time to wait for their implementation in Moodle code...
Thanks all of you. I do agree with Pierre that if changes are well documented it's OK & reliable making patches by hand. But that's what I hoped to get away from with cvs patches. With all the different Moodle versions, handwork would have to be repeated carefully many times over, and patch software should really be doing most of this. The command line 'patch' is only unix isn't it - not in Windows command line. I develop mostly on Windows, but maybe I should aim to try out patches separately just on a server with unix. Bit tiresome. Eclipse does look powerful, but seems on the download website mighty intimidating simply for this job, and other people who would aim to use the patches may not in any case have Eclipse. Perhaps really unix is the thing common to all, and the way to go. I'm just puzzled why there isn't a simple Windows equivalent.
This was not what I try to explain.
You code your CBM in moodle code on one version and this a quite a job to synchronize everything when new versions of moodle appears.
I don't know how to do this automatically and when I merge changes from moodle head to older versions like 1.6 (we use 1.6 at my university) I do all changes manually using a Compare file software. As i merge in STABLE version this is a must.
On the other hand, i want to explore what changes are necessary to have CBM work on Moodle. If you could set a version of your code with just the CBM changes (not the other bugs) this will be useful and even more if write either a doc on this or at least //line comments in the code.
I am just as puzzled as you. I personally use Eclipse, which I find quite user-friendly both for developing php and committing to CVS. As you say it looks a bit intimidating at first, but Tim has provided excellent guidelines to it. However, it is perhaps not so useful if you do not have committing rights to moodle or contrib.
As for Unix-like command line patch commands, you can use them even in the Windows environment. You need to do a google search for GnuWin32 Packages. The programmes you need are diff.exe or diffutils.exe and patch.exe. Be aware however that for a Windows user there is a steep learning curve and it's hard to find help. For me it was like suddenly being thrown back into the prehistoric , pre-Windows past of the DOS command-line. I am truly amazed that still nowadays the "computer people" (at my university, for example) consider the command line the nec plus ultra of computing and thoroughly despise the more user-friendly user interfaces. Well, one reason may be that mastering all those arcane procedures gives them power over the "ordinary" Windows-attached user.
Have you tried putting a wet towel round your computer?
About the number of option, I work with teachers who uses 4 levels of confidence. Thus students may not always choose the middle.
They said they are based on the work of Dieudonné Leclerq (University of Liege)
But I would like to squeeze that option (student ->teacher feedback) in here and I believe that it fits: There are cases where the teacher is "too certain" how the question will be interpreted or should be graded! We teachers somethings give "right" for simplified solutions that maybe some students can rightfully challange!
I suggest an option like this:
If the teacher checks "[_] Invite feedback from students" a suitable button or link would be created. This setting could be at the question, quiz, course and site level where I guess at higher level it would need "[_] default; [_]disable" where neither would wholly leave it up to a lower level. There would also be a choice of where to send the feedback (if it doesn't just go to the quiz database.) with default being to quiz editor if set at quiz level or to question author if set at question level etc.
A button like this: , with explaining "title popup", wouldn't take up much question page "real estate" but be connected to a pop-up or expanded form for the feedback.
In my wish-list for numerical questions, I also mention my hope for feedback from students also for pages in general, though I think it is extra important on quiz questions. These two feedback-to-teacher functions ought to be able to share some code.
Jeff - I'm absolutely with you on this need for student->teacher feedback and the concept was very much intended in what I thought was needed, under the heading 'comments on Qs'. One of the unexpected spinoffs from using CBM has been the stimulus that it gives (when a student claims certainty and then feels unfairly marked and penalised) for providing such feedback to teachers - REALLY useful.
I have actually incorporated a comment system into www.ucl.ac.uk/lapt/moodle19/moodle (where students are always invited to enter or look at comments after they have been marked on a Q). These comments are emailed to a staff member as well as being visible to other students. This is a crude version of what has been so successful on LAPT. I would encourage you to look for example at www.ucl.ac.uk/lapt?bmat1 (a critical thinking exercise for medical school admissions) and click View/Make Comments at the bottom of the screen.
The code in my Moodle CBM demo site actually accesses the pre-existing external LAPT code for the comment arrangements in a way that certainly isn't satisfactory for more than a demo. I haven't had time yet to look at the proper ways to run such a facility using the Moodle database, and would most welcome help from anyone familiar with the Moodle database and discussion code. It would be very much a second best option I think just to have an "email to teacher" option, though this would be better than nothing. It is important, in my experience, to have automatic inclusion of info about context and exactly what the student has entered as an answer, since often what the student says they have done (especially with numerical answers) is no more than what they think they have done!
The "Make Comment" arrangement was interesting. I'd never considered having it like a forum. But I did find it hard to orient in. Comments for all questions for one quiz mixed together on one page.
I'm also interested in seeing how the student has actually answered. In an interactive question type I have used a lot BM (Before Moodle) I used a Perl script that packaged student comments together with an URL that would take me to the question in the state it was when they answered. It would also include their email address. My stuff is in Swedish but you can see an translated version at: http://www.tupo.biz/cgi-bin/tut/tuteng.pl/kurser/MaC/StudArb/QuadEq1.htm
In this version I'm no longer using the Perl script to do it but a Java-Script feedback form that catches the referrer (and also harvests info about the OS, browser, screen-size that might influence how the question appears for the student.) Unfortunately the .js I use won't work in a Moodle quiz. The referrer won't capture the state of the question. (Maybe if it would also capture a session cookie? I'm not sure what is done in that way and what is done in the server on Moodle.)
An arrangement in Moodle that would give the teacher a URL which recreates the student's question page at time of comment would be good. When the teacher is there it is so easy to click the edit icon to get to question editing to trim the feedback or grading.
There are pros & cons between having comments presented chronologically for the whole quiz (as in LAPT), or separately for different Qs (like the Moodle demo). I don't think I would want to change it for LAPT - where an exercise is often much more an integrated learning resource than a quiz-like compilation of independent Qs. Editing of resources in LAPT is done with the text for the whole exercise visible, not just individual Qs.
Another difference is that the comments in LAPT are anonymous unless the student gives an identification or email. I know people have different views about this, but I don't like the idea that anyone would be inhibited from making a comment by potential embarrassment. It's never led to any abuse.
The forum format allows students to respond to comments. This is often useful since the hardest thing in my job has usually been to get staff to respond to them.
I've done a multi-media interview on UCL's initiative in Certainty-Based Marking for an American journal 'Innovate: journal of online education' which is linked at http://www.ucl.ac.uk/lapt/Innovate . I'd be keen to hear any reactions to what I had to say there. There is also a transcript available on this link. There will be an open webcast and opportunity for online 'chat' at 5pm GMT (12 EST) on Tue 19th Feb.
Although the interview doesn't specifically mention Moodle, I shall probably use the Moodle (demo) installation with CBM at http://www.ucl.ac.uk/lapt/moodle19/moodle for illustration in the webcast.
Good interview. I would like to know what kind of interest it generates in CBA?
I heard Tim speaking of CBM and incorporating it in Moodle 2.1 at mootuk2011. At the same time a client found out about it. We're both keen on using it in the new Moodlesite we are building. Tim, do you think we can count on release of 2.1 with CBM in it on 30th of june?
I can't promise anything yet. You need to watch MDL-20636.
We will know by 31st May whether it is complete enough to be included in the release. I am still hopeful.
Thanks thats promising.
I am nog very familiair with the MDL-stuff. Where on he page should I look?
You can already use CBM in Moodle1.9(.12) or Moodle 2.0(.3) by downloading code modifications and following the instructions available on the UCL/LAPT Moodle-CBM demo site at http://www.ucl.ac.uk/lapt/moodle19/moodle/mod/forum/discuss.php?d=2 . Login with id=mdlteacher and pwd=mdl19.
This code (unlike Tim's new code) doesn't need changes in the database structure, so is easily copied in and restored reversibly if you want to test it yourself. You can see how it works from both teachers' and students' perspectives on the demo site. The code also allows you to export Moodle Qs for use with the LAPT system that is specially designed for self-tests with CBM.
I haven't yet reviewed Tim's code for 2.1, because of problems installing it on my setup. But I'm sure it will eventually run smoothly even if it needs tweaking. Please contact me ( email@example.com ) if you want to try CBM on 1.9/2.0 or LAPT. NB I don't regularly watch this forum. Tony GM
please help me to enable automatic backup process in moodle 1.9.
its not working .
i find myself successfully in course backup,
but did not get good news in complete moodle backup.
Is there any Moodleman who has done it successfully.
i hope moodle people will solve this problem soon.