Quiz enhancements - Programmers wanted - Funding available

Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Number of replies: 33
Dear Quizzers,

I have obtained (limited) funds for the development of two or three specific enhancements to the Quiz module that many of the users at our university consider a "must" for a larger-scale adoption of Moodle as a testing platform.

I'd be extremely grateful for any feedback on the feasibility of these enhancements and for responses from potential programmers. Please feel free to contact me directly via PM or through the forum. I'd also be delighted to receive general proposals/comments/suggestions regarding the enhancements so that any work undertaken is of maximum benefit to everyone who uses Quiz to administer exams. Below are the enhancements in order of our local priority:

Enhancement 1: Negative marks (critical)

The lack of a fully functional negative marking option and the difficulties associated with proper implementation have been discussed at length in various threads. What about the following "quick fix": Make the adaptive mode and real negative marks mutually exclusive, i.e. allow for a quiz configuration using either negative marks or adaptive mode. Is this feasible from a programming perspective, and if so, how much would it cost?

Enhancement 2: Improved reporting facility (urgent)

We have to archive examination results for a minimum of ten years (legal compliance). The archiving could be handled much more comfortably by the instructors themselves if individual full reports of all attempts (type "show all questions on one page") could be downloaded as static html or pdf files with automatic naming (being able to select/combine course/quiz names, user names, student ID or real names, attempt no. + timestamp for naming would be really sweet). Feasible? Price tag?

Enhancement 3: Auto save facility in quiz setup (nice to have)

Some students simply forget to "save without submitting" once they have completed a substantial portion of work in a quiz. On a few rare occasions this has led to complications (and even tears) when browsers terminally crashed and the attempt had to be continued virtually "from scratch." Is it feasible to implement an autosave feature hooked to the quiz timer (perhaps using the same mechanism that saves and submits when time is up)? Feasible? Price tag?

Terms & conditions

I am aware that some of the above issues have been discussed extensively and that the enhancements may not be the cleanest solutions. However, there is an opportunity here to get some development funded, provided work commences relatively soon. The exact status of the solutions (core, patch/hack, add-on) does not matter as long as it is designed to be used with 1.8+ and 1.9(+) and is GPL.

I am looking forward to receiving your feedback.

Kind regards,

Peter

P.S.: Estimates preferably in EUR.

Average of ratings  -
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Anthony Borrow,
Avatar Core developers Avatar Plugin developers Avatar Testers
Peter - You may be able to get a more accurate response by contacting Moodle.com directly and requesting a quotation for each of the projects:

(http://moodle.com/helpdesk/index.php?departmentid=5&_m=tickets&_a=submit&step=1).

Peace - Anthony


In reply to Anthony Borrow

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Dear Anthony,

Thanx. Submitted the request as advised and hope all goes well. If anyone has a similar request, pls. feel free to latch on before final specs are completed (budget permitting).

Kind regards,

Pete
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Pierre Pichet,
I have looked at the MDL-5324 allow negative marks for questions and as this appears to involve other projects I am working on (multianswer), I will see if I can applied valid patchs as Tim wrote in 5324 "if someone can come up with a patch that covers all cases reliably, then I would be delighted to check it in. ".

Pierre
In reply to Pierre Pichet

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Hi Pierre,

Many thanks for the quick reply. It would seem that the best course of action is to talk to moodle.com (see Anthony's post) as well as the general community. Do you think the idea of making the adaptive mode and "real negative" negative marks mutually exclusive might work reliably as a quick fix?

I'll stay on it as collaboration develops.

Kind regards,

Pete
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Pierre Pichet,
"adaptive mode and "real negative" negative marks mutually exclusive"
It seems a good way to fix it as Tim agrees.

Pierre
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Pierre Pichet,
To what questiontypes do you want to apply negative marking?

All question types or just the ones where the student is given a choice so he has no advantages to play loto...like multichoice, true false question.

Otherwise having a 0 ...1 scale is equivalent to -0.5 ...0.5 or -1 ...1 if it was not that students are more at ease with a 0-100% scale in USA , 0-20 in France etc..

Actually the following question types do not allow negative grading to a bad answer : shortanswer, numerical, calculated, match, truefalse.

Pierre


P.S. But effectively we as teacher are sometimes more angry (i.e. he should get a negative score!!) with a student that is answering a question with the error that was the main subject of the last three hours course than with one that do not answer at all smile.

In reply to Pierre Pichet

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Hi Pierre,

Just a brief reply for now. You are right, negative marks only make sense in multiple choice-type questions as they are a popular (but certainly not the best) means of reducing/correcting for guessing. This would include MC, TF and, perhaps (might be out of project scope, though), Cloze questions with predefined answer options.

Kind regards,

Peter
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Pierre Pichet,
As I am working on the Cloze questiontype, I could include it.
for truefalse it would include a revision of the edit_truefalse_form.

Pierre
In reply to Pierre Pichet

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
This sounds pretty good. Most of my users could live without cloze, but classic MC and T/F are a must. In the case of T/F it would also be useful to include an optional "don't know" radio button to give exam takers a way out if they have accidentally selected one of the real answer options and then change their mind. For MC, the "don't know" is on board already.

I don't really have any idea how difficult it is to do Cloze. I think you mentioned in another thread that your proposed rewrite of the code might imply substantial changes to the way the question type is handled.

On a general methodological note I have to admit that I personally (but obviously not my users) think that negative marks can be overkill with the exception of T/F. A well-designed four-option MC-question shouldn't need negative marking if you work out the probabilities, and there are actually better ways of correcting a test score for guessing, such as a relatively simple model which assumes that the amount of guessing per examinee increases with the proportion of wrong answers and correction (negative marking) is scaled up accordingly (i.e. somebody who produces many correct answers some no-answers and a few wrong answers incurs a proportionally lower penalty per wrong answer than somebody who produces lots of wrong answers). However, this is only implemented easily if items can be grouped into "blocks" with equal numbers of answer options, and the calculation has to be performed individually on each block. I think this is actually a lot better (not anywhere near as good as IRT, though) but too difficult to implement at this time. CBT is also an option, but it won't help people who need to emulate types of tests/exams that use simple negative marking or are subject to legal constraints (our -- rather conservative -- ministry of education would go ballistic if they found out we used CBT in real exams). Isn't bureaucracy great?

Apologies for the rant & best regards,

Pete


In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Pierre Pichet,
A first remark about TF and MC. If you need more than two options wich is the T/F case, just use the MC to put your "Don't know" case.
In reality we must question the existence of a T/F type as a specific type. It could be a subset of the MC. It is however very useful in lessons which at this time remain separate from the quiz.
So actually the real case for negative marking is the multichoice among the "standard" questiontypes.
This could simplify the problem. For new questiontypes (plug-ins) the quiz should know if negative grading is used and the questiontype should know the quiz options so that he can or not use the negative grading. The $state appears to be the object to use for such information exchanges.

Pierre
In reply to Pierre Pichet

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Hi Pierre,

I agree with your view on TF and MC. In fact, we are using MC on a "hacked" installation of 1.6+ as a workaround to achieve negative marking. There are, however to problems with the workaround:

  1. 1.6 is being phased out developmentwise, and
  2. negative marks have to be recalculated manually if students use multiple attempts (we allow for two attemps in case something goes "wrong").
Cheers Peter
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers
These are certainly some of the major wishlist items for the quiz, so it would be wonderful if someone has the time to solve in a way that is compatible with the rest of the quiz module.

The normal procedure for someone doing a major change like this is:

1. Write a document, linked from http://docs.moodle.org/en/Development:Developer_notes that outlines
a. The requirements - that is, it specifies what the proposed change is meant to achieve in language a teacher or administrator might understand.
b. The propsed design - how you plan to modify Moodle to achieve this aim, including any changed to the database, code and user-interface, in language another developer would understand.
this document should be as short and easy to read as possible, consistent with containing enough detail.
2. Start a thread here, or in the General Developer Forum (whichever seems more appropriate) to discuss the merits of the plan.
3. Based on the discussion, improve the plan until a consensus is achieved.
4. File issues in the tracker for each part of the work - perhaps one top-level issues with sub-tasks.
5. Write the code following all the Moodle coding guidelines, and attach it as patches to the tracker issues for review.
6. Respond to the reveiw comments by improving the code until it is good enough to be checked in.


Three recent and successful examples of this have been http://docs.moodle.org/en/Development:Plan_to_Improve_Flexibility_of_Question_Category_Sharing_and_Permissions, http://docs.moodle.org/en/Development:Email_notification_when_a_quiz_is_submitted and http://docs.moodle.org/en/Development:Plans_for_enhancing_import/export_in_questiontype_plugins.

What I found particularly interesting, as a participant in at least steps 2 and 3 of these examples, was that in the process of discussing the details of the proposals with other developers, we collectively came up with solutions that where better than anything I could have come up with thinking about the problem on my own. In particular, this amazing 100+ post thread, between Jamie, me, and others, which started off with an idea of mine that turned out to be totally wrong, and ended up with a technically solid plan that lead to the new question bank in Moodle 1.9.

But note that discussion took about a month - time well spent, but a lot of time. And for anyone intersted in bidding for any of this work, note that Jamie, an experience Moodle developer, originally estimated 4 weeks for this work, and it ended up being at least double that. The quiz code has been around for over 5 years, and many different people have contributed to it. In places that shows. Some corners of the quiz code are rather messy and hard to work with. Of course as you study the code in detail to make changes, it is a good time to do some cleaning up. Certainly, Jamie cleaned up large chunks of the code while rewriting the question bank, which I am very grateful for.


There was a thread in this forum some months ago in which someone gave a remarkably small amound of code that implemented auto-save. Regrettably, I have not had the time to investigate it yet. If other people have time to look at it, try it, and tell me whether it works, then that would be great, and let us decide sooner whether (or not) it should be incorporated into the official Moodle release.


Also, you are almost cetainly right, that negative scoreing and adaptive mode need to be mutually exclusive. Indeed, if Tony Gardner-Medwin's plan for certainly-based marking in Moodle comes off, it will may end up being three mutually-exclusive options, since CBM uses negative scores in yet another way.
In reply to Tim Hunt

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Hi Tim,

Many thanks indeed for your very comprehensive reply and particularly for outlining the procedure. Moodle is such a complex project that just finding out where to start things and how to find information and work already available can be a challenge, at least the first time round.

  1. I will follow your advice and write a more detailed requirement/ specification document as soon as possible (could take a day or two, though). a) - Description of minimum requirements/functionality: no problem; b) - Description of proposed design: third party input needed (I am not a programmer and have looked at/dealt with Moodle code only when absolutely necessary.
  2. Thread started. Will be linked to developer forum when detailed requirements have been written.
  3. I suppose discussion will really start when the first reqts. doc has been written. Stupid question: How does one avoid making the requirements a "moving target" if developers are to bid?
  4. , 5. & 6. We'll cross that bridge when we come to it sorire
One final issue/piece of information & one more question for now. Even if there were to be substantial discussion regarding the enhancements I'd need to start spending relatively soon (end of budget year), and some ballpark estimate of the financial order of magnitude of the project(s) would be helpful. The question: Should I split the enhancements up into three separate projects? Would make sense to me.

Best regards and again many thanks,

Pete


In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers
1. As the customer/client for this work, the bit you are responsible for is the requirements. How exactly do you want it to work? Anyone bidding for this work will need to know, so they can estimate it.

The proposed implementation is the business of the developer you hire. They will need to think briefly about their proposed implementation before they quote you a price, and will need to spend more time on a detailed desing once they know they have got the work.

2 & 3, we can discuss the requirements now. As you say, discussion of the design has to come later.


The three proposed enhancements are distinct pieces of work, and it would make sense to specify and implement them as separate changes.

From a business point of view, however, it would be easier to find just one developer to do all three, and build a good working relationship with them. But if you were planning to comission lots more Moodle development work in the future, you might want to invest more effort in outsourcing this work to three different developers now, so you can decide who you might, or might not want to work with again.


Some comments on the individual projects:

1. The main thing to decide when writing your requiremesnt is how may question types you expect this to work with, and some idea about how you, as a teacher, would expect to enter the scoring information. Obviously it is best if you can come up with one general idea the works for all question types (but perhaps with a few tweaks for some of the question types). Is negative scoring controlled on the quiz editing form or the question editing form? If you get a negative score for each question, do you get a negative score for the whole quiz, or is the quiz score always rounded up to zero in this case. Note: I am not sure that the Moodle gradebook can cope with negative scores. I think the final score should be limited to be between 0 and the maximum score for the quiz. (Scoring zero on a quiz is probably demoralising enough for the student already.)

This is proabaly at least 4 weeks work. 1 week to thrash out a detailed design (probably with more than one week elapsed time); 1 week to go round all the question types making the necessary changes to the editing interface and answer marking; one week working on the main quiz and question bank code to make sure the negative scores returned by questions are processed properly; and at least a week testing quizzes that use all question types, with one or many pages, with and without each attempt builds on last, checking that you have not broken adaptive mode, etc.


2. This is smaller and more self-contained. I would do this a quiz report, which is one of the smaller and simpler types of Moodle plugin. The report could just request the review page for each quiz attempt, with 'Show all on one page' selected, and save the results in a sub-folder of the course files area. Probably in a subfolder of the backupdata folder, which is restricted to teachers only. (That is why question exports are stored in there.) Then finally, it could combine all the exports into a zip file. Obviously, there would need to be a bit of user-interface to controll the process. The only tricky bit is that the quiz review pages need various images and stylesheets to display properly. Perhaps the plugin should rely is wget or equivalent to do the downloading. Depending on how robust the finaly solution is, and what sort of other tools it relies on, it might not be incorporated into the quiz, but instead live as a third-party plugin in the modules and plugins database and contrib CVS. Anyway, this might only be a week or two's work.


3. This is the hardest one to estimate. If you can find the forum thread I alluded to above, then there might be very little to do beyond a bit of testing. However, if you have to start from scratch, I don't really know what the best solution might be, and so can't guess how hard or easy it might be.
In reply to Tim Hunt

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Thanks Tim,

I'll get cracking on the specs document tomorrow. Too tired for anything but a semi-automatic rant after a full day of lecturing.

Kind regards,

Pete
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Dear All,

A developer document for quiz enhancement #1 (neagtive marks) is now available here.

I will produce the documents for the remaining enhancements as soon as possible.

Apologies for the delay and keep the good ideas coming.

Regards,

Pete

In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers
Two comments:

1. clean integration with multiple attempts and at least the option set "each attempt builds on the last" & "highest grade"



If htis is going to go in to Moodle core, then it needs to work correctly (or at least not be incompatible with) any combination of quiz settings. However, I can't think of any other settings it needs to be compatible with right now.


2. You have not addressed the issue of what happens if the student gets all the questions wrong. Do they get awarded a negative score for the whole quiz? I think not. I think that the total score for the whole quiz should be between 0 and the maximum available marks. (And I have a feeling that if this is not the case, then they Gradebook will need to be changed.) I also think that this is someting that will not affect many students on real quizzes.
In reply to Tim Hunt

Re: Quiz enhancements - Programmers wanted - Funding available

per Peter Möbius,
Hi Tim,

1. Totally agree. I simply decribed our local minimal requirement but think we should go for the comprehensive solution if at all possible, which means full compatibility with all possible option sets.

2. More or less agree, but don't think it is a fundamental problem. -- Input by other users much appreciated here. True negative raw scores (negative percentages don't seem to make much senseclinhada should be possible or at least documented somewhere (e.g. in the detailed response report) before they are truncated to 0. This would make it easier to gage the extent of the student's "problem" in one glance and provide some kind of audit trail.
What do you think?

Regards,

Pete
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers
Hmm, perhaps we can have negative attempt scores, but the final score for the quiz as a whole (highest, average, ...) can be restricted to between 0 and max.

But as you say, imput appreciated.
In reply to Tim Hunt

Re: Quiz enhancements - Programmers wanted - Funding available

per John Ennew,
Hi,

Is anyone aware if negative marking for quiz questions are being worked on?

Is it something planned for Moodle 2.0?

Thanks,
John
In reply to John Ennew

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers
No.

I would like to solve this and various other problems in Moodle 2.1, but whether I get to do that is up to the OU, naturally.

There are hacks around that get you negative marking if you don't mind breaking adaptive mode.
In reply to Peter Möbius

Απάντηση: Quiz enhancements - Programmers wanted - Funding available

per Anastasios Pournias,
I think that the Enhancement 2 is implemented in Quiz Report Analysis tool. Check the patch here
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Clare Denholm,
Hi I am looking for the enhancement listed as option 2, about being able to export the detailed information of completed quizzes. As stated in the example I need to store them as we would the equivalent paper exams and the student answers are required, not just the marks. has this been addressed anywhere as I haven't been able to find it and the link mentioned above seems to be related to a different notion (using stats for students to review progress).

thanks
In reply to Clare Denholm

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers
I don't think this has been done, unless the output from the detailed responses report (in the modules and plugins database) is good enogh.

In reply to Tim Hunt

Re: Quiz enhancements - Programmers wanted - Funding available

per Clare Denholm,
Great, that plug-in does the job, sorry I hadn't found it.
Thank you for the swift response, much appreciated sorire
In reply to Peter Möbius

Re: Quiz enhancements - Programmers wanted - Funding available

per Jibran K,

Hi All,


Some part of the requested functionality has been developed by us. It is the functionality of bulk exporting student attempts in the form of PDFs in exactly the same UI as displayed on Review attempt page. Please find details in the following:

http://shispare.com/portfolio-view/moodle-export-student-work-record-to-pdfs/

In reply to Jibran K

Re: Quiz enhancements - Programmers wanted - Funding available

per Tim Hunt,
Avatar Core developers Avatar Documentation writers Avatar Particularly helpful Moodlers Avatar Peer reviewers Avatar Plugin developers

Please to not make duplicate posts in many threads. (I have delete the copies now.)

If you want to share your plugin, one post is sufficient. Any more makes you look like a spammer.

Also, why not upload your plugin to the Moodle plugins database, or to github?

In reply to Tim Hunt

Re: Quiz enhancements - Programmers wanted - Funding available

per Jibran K,

Hi Tim,

I feel none of my messages were actually duplicates, they were all very specific to the threads and none of them was irrelevant to the topic being discussed in the thread. All the threads I posted to were open threads without any solution of the issue being discussed. So I tried giving the people a possible solution. I don't know how  you count them as spam. A google search should help, (https://www.google.com/search?q=what+is+spam&ie=utf-8&oe=utf-8&client=firefox-b-ab#). The keyword here is "irrelevant".

Still if you restrict me to one post that's fine I will make one post to just one thread of my choice. You can delete the one here.

Yes we plan to upload this plugin to Moodle plugins database (along with some others that are under development) but it needs some polishing before we publish it for the public. The current state is very specific for our needs and not even easily installable as it depends on some utilities that need to be installed at the server. Also the plugin publishing process seems to be a lengthy one here. So we are working on that and we will share it with the community but that will take some time. Currently what we can do is that we can tailor it according to the needs of some other schools who need similar functionality and install it on their Moodle ourselves which was the purpose of my posts in the threads where people were looking for a similar thing.

Thanks.


In reply to Jibran K

Re: Quiz enhancements - Programmers wanted - Funding available

per Anh Tuan Bui,

There is a similar plugin already developed and shared among the community at this URL: https://github.com/elccHTWBerlin/moodle-quiz-export.

However, it requires installation of wkhtmltopdf utility on the system to generate quiz attempts in html to pdfs (either individual or group attempts. And it's a bit complicated to configure Unicode font settings to generate the output correctly.

I see the certificate plugin use TCPDF that can generate pdf file directly without any utilities on the server. Does your plugin use the same mechanism?

Thanks

In reply to Anh Tuan Bui

Re: Quiz enhancements - Programmers wanted - Funding available

per Jibran K,
Hi Anh,

Our plugin also uses wkhtmltopdf utility. Actually wkhtmltopdf is superior compared to TCPDF and other open source PDF generating libraries because it uses webkit rendering engine to convert HTML to pdf. Therefore the output generated using wkhtmltopdf is exactly similar to the generated HTML. On the other hand libraries like TCPDF can be very difficult to handle with complex HTML/css combinations.

We are working to make it convenient for the public to install. Thanks for the link, I will have a look into this one also to make sure we come up with something better.
In reply to Jibran K

Re: Quiz enhancements - Programmers wanted - Funding available

per Anh Tuan Bui,

Dear Jibran,

Thank you for your explanation.

I've tried the Quiz Export plugin on local test server but have difficulties in setting the correct font and utf-8 encoding, which makes the output pdf unreadable, even with non-Asian language (a weird Cursor font, along with Utopia, are embedded in the PDF otput).
As you also use wkhtmltopdf, could you have a look at the attached file and give me some hints how to setup the .css file to make the font display correctly?

Kind regards,
Tuan





In reply to Anh Tuan Bui

Re: Quiz enhancements - Programmers wanted - Funding available

per Jibran K,
In reply to Jibran K

Re: Quiz enhancements - Programmers wanted - Funding available

per Anh Tuan Bui,
Dear Jibran,

I've tried a fresh installation of Moodle on Ubuntu Desktop 16.04 and latest wkthmltopdf (0.12.3) and it works.

I think I had forgotten to install some packages that wkhtmltopdf depends on as noted on the wkhtmltopdf site below:
Linux 0.12.3 32-bit / 64-bit depends on: zlib, fontconfig, freetype, X11 libs (libX11, libXext, libXrender)

Thank you very much for your help.
Tuan