1.6.2 Essay Question Bug - what lessons to be learned?

1.6.2 Essay Question Bug - what lessons to be learned?

by Don Hinkelman -
Number of replies: 11
Picture of Particularly helpful Moodlers Picture of Plugin developers
This discussion moved from Issue Tracker [MDL-6649]

Scenario: In January, during my end-of-term tests, students spent one hour doing 30 multiple-choice questions and 2 essay questions. Generally, they spent about half an hour writing the 200-400 word essays. Upon submitting, their essays disappeared. All text was gone. I put on a calm, smiling face and said no problem. And thanked them for their hard work. They assumed the data was there, and I pretended so as well. blush mixed

Investigation: Searching the issue tracker, I found that yes, indeed this was a bug that was fixed in version 1.6.3. When the HTML editor is used more than once in IE, "unreliable" things happen. Unfortunately, at our school we installed version 1.6.2+ at the beginning of the semester, in September. The bug was fixed a few weeks later unknown to us. Once we install a server for a semester, we never change it until the end of semester, so we choose stable releases only.

Questions: At first I was angry that a stable release (actually three stable releases 1.6.0, 1.6.1, 1.6.2) used with default settings would still have a catastrophic bug ('catastrophic' meaning either a script crashing or a critical/total loss of student data). I have used Moodle since version 1.0 and found stable releases to always be solid. I suggested that 1.6 should have remained as beta status as a warning not to be used with students in critical situations. Tim replied that version 1.6 had been delayed in beta from January to June 2006, and that "delaying releases indefinitely was not a solution". So I am wondering what the solution is. If this was just an odd, rare situation, we could forget it. However, IE/Windows and essay questions are all very standard things used in education. I am especially curious why it took so many releases (over about 9 months?) for this problem to surface. So here is my list of questions:
  • Am I one of few teachers who use essays in the quiz module? Is everyone using multiple choice?
    • (if so, maybe this post should be moved to a pedagogical forum) smile
  • Were other teachers confused or hesitant to report the problem?
  • Was the new essay question type not user-tested in its beta phase?
  • Or was it simply tested with only one essay question?
  • Was the problematic switch from a plain text editor (Humboldt version) to an HTML editor (1.6 version) not well noticed in the nine months of integration into standard Moodle?
  • Should we keep new releases in beta longer? Wait for more user testing?
  • Should we warn that stable releases may not be stable?
  • Should we strongly recommend to schools that servers need to be updated continually, even while classes are in session?
Finally, on behalf of the Moodle community, I would like to apologize to my 60 students for my failure to trustfully handle their data, and apologize to perhaps many more students in other schools who also lost their essays this way.
Attachment aisatsu.gif
Average of ratings: -
In reply to Don Hinkelman

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by N Hansen -
Don-I know there were some complaints about spotty behavior in IE using the html editor, perhaps six months ago. But I don't recall reading about data loss, just that the html editor wouldn't load.

I use essay questions almost exclusively. For my students, I put 3 per page, but when correcting them I may have 50-60 loading on one page. But I and my students need to use Firefox (for other technical reasons) so I have never had this problem. I know from time to time my students complain that their answers disappeared from a page but it is so infrequent that I think it might just be a temporary internet connection problem.

However, 1.6 was released with a lot of bugs in quiz. I spent the first two weeks using 1.6 reporting bugs in quiz left and right. Part of the problem is it was handed over to Tim at a crucial time, right before release, and Tim just didn't have the time yet to get up to speed on how it worked, and that isn't his fault at all, but still it was a big problem. When 1.6 was first released, clicking on regrade could destroy an entire quiz results, and also until recently I believe, teachers' manual comments on essays and other quiz question types were not getting backed up.
In reply to N Hansen

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
With the new forms library in Moodle 1.8, lots more pages now have multiple right text editors on them. It will be interesting to see if they still cause problems.
In reply to Tim Hunt

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Jörg S. -
  • It will be interesting to see if they still cause problems.
Have you seen those problems yet? If not - is it possible to turn on multiple rich-text-editors on essays?
-Jörg
In reply to Don Hinkelman

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers
Hi Don -

There are a number of questions/issues here.

From a purely developmental standpoint, I believe the essay question was first added as a standard feature for quizzes in 1.6. I think prior to that it was a 'hack', so it didn't have a long testing period.

And therein lies the problem. The developers can only test these elements so far. A lot of these problems don't surface until they're used in the wild (this is true of commercial products too, by the way). The good news is that they are usually fixed quickly.

I don't think leaving it in beta would have helped. Institutions like yours would not likely install a beta version and thus we would not get the real-world testing.

The community really needs organizations to test these systems in the real world, doing things the way they need to. And, its probably best that institutions run their courses/activities through a rigorous test before counting on them in a real situation.

For the specific problem, if it was indeed caused by the bug you quoted, that would mean that you must have had both essay questions available on the screen at one time. Its likely you would have caught this problem ahead of time yourselves had you done a run through with a couple of "test" students (but maybe not).

By the way, I have seen quiz submissions get "lost", although its usually caused by use of the browser "back" button. Quite often they're in the database, just disconnected. I think these issues have been fixed over time.

As to your update schedule, not upgrading your system until the end of a semester means that you're stuck with whatever bugs there are. Its probably best to keep an eye on releases to see if they fix bugs that affect you.

Lastly, as a general rule, if I'm entering a lot of text (like this post for example), its always a good practice to have an off-line copy. Many things can go wrong when you press that "submit" button, including connection drops. You may want to get students to type their essays in another editor and paste them into the essay form when they are done.

mike (hitting ctrl-a, ctrl-c before posting)
In reply to Mike Churchward

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Tony Beld -
Mike says...<snip>if I'm entering a lot of text (like this post for example), its always a good practice to have an off-line copy. Many things can go wrong when you press that "submit" button, including connection drops. You may want to get students to type their essays in another editor and paste them into the essay form when they are done.</snip>

Yes indeed, I learned this the hard way myself. Several times while working on long explanations to students or in detailed post here I have lost messages after hitting submit. There are problems with using "another editor" though. What if the test is in "secure mode" (or the "more" secure mode as the case may be)? What if you are teaching and testing language as is the case with Don and I? You probably don't want students working in Word or OpenOffice where they can easily spell check and somewhat grammar check their submission. Some of our testing we do in a lab where we can be sure the student isn't using such tools.

Mike says...<snip>(hitting ctrl-a, ctrl-c before posting)</snip> I think this is the best practice to impart to students/teachers/Moodlers. That way if there is a hicup, you still have your text.
In reply to Tony Beld

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Paul Nijbakker -
In addition, copying non-Latin text to the Moodle HTML editor can cause it not to save the text (probably a UTF-8 issue), in our experience.

Rgrds,
Paul.
In reply to Mike Churchward

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Jussi Hannunen -
As to your update schedule, not upgrading your system until the end of a semester means that you're stuck with whatever bugs there are. Its probably best to keep an eye on releases to see if they fix bugs that affect you.

Agreed. You should try to update to the newest version of the branch you are using (1.6 is this case) as often as possible. If you stay in the same branch, the update is pretty quick and painless operation once you have worked out your routine. We update once a month; doing it more often (or automating the process) doesn't seem to offer that much more for the work that would be involved.

I once started to design/think about a system for harvesting a bug tracker for information about which versions are known to contain critical bugs. I stopped when it occurred to me that all old version are known to contain critical bugs. The only question is the exact date that the current version becomes a "old and known to contain bugs" version and that isn't quite as interesting question.


Jussi

In reply to Don Hinkelman

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Nicole: But I don't recall reading about data loss, just that the html editor wouldn't load.

If indeed I am the only one who lost student data, then this is a rare case and we don't need to spend time worrying much about it. It might also point to something odd in our system setup--but I think we are fairly standard.

Nicole: I spent the first two weeks using 1.6 reporting bugs in quiz left and right.


This is probably the most important lesson I have learned. Developers depend on users/teachers to test out new releases. Thanks, Nicole.

Mike: I believe the essay question was first added as a standard feature for quizzes in 1.6. I think prior to that it was a 'hack', so it didn't have a long testing period.And therein lies the problem. The developers can only test these elements so far. A lot of these problems don't surface until they're used in the wild (this is true of commercial products too, by the way).

It had a long testing period "in the wild" as a hack--well tested with lots of classes at California State. When added to 1.6, it was changed and thus a new bug introduced.

Mike: The good news is that they are usually fixed quickly.

Yes, it was fixed within two weeks--after some insistance. Curiously, it was not discovered for many months--going through three stable releases. That is what puzzles me.

Mike: Its likely you would have caught this problem ahead of time yourselves had you done a run through with a couple of "test" students.

Possibly in some cases, yes. However, it is usually not possible for me to have students test things because when we install a new release, we do not know what activities will be used in the coming semester. Courses are made on the fly. Exams are finalized minutes before class. Teachers trust the software to work and they do not know what is new and what is not. Frankly, there is so much changed or added, I as an admin often do not know what is new. I can read the release notes and start there, but still have a hard time guessing what things to suspect.

Mike: as a general rule, if I'm entering a lot of text (like this post for example), its always a good practice to have an off-line copy...

Really? mixed I had hoped browser-based applications had come along farther. How do new browser-based word-processing apps by Google manage? Yes, I can recommend an off-line Word window, but for testing, we have generally wanted students *not* to have any other windows open. Yet, next time I could have them do that (and make them spell-check as well).

I still think it would be prudent to have multiple beta releases (beta1, beta2,...beta5) and let the good wine age a little. Also since when a release goes "stable", it will get put into Fantastico, perhaps too soon, and then disappointing the less-technical users there. I used to be one that clammored for faster releases, and grew impatient with delayed schedules. Not any more. Let's take time, and even run our production servers on not one, but two prior versions ago (long live 1.6!). I believe MySQL and PHP take this approach of multiple stable releases.
In reply to Don Hinkelman

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers
Don: Possibly in some cases, yes. However, it is usually not possible for me to have students test things because when we install a new release, we do not know what activities will be used in the coming semester.

I wouldn't necessarily use students to test them, just use some student test accounts. Get staff to help out. Its always better to be safe than sorry.

Don: Really? mixed I had hoped browser-based applications had come along farther.

Its not a factor of how far browser-based applications have come so much as what can go wrong. I have lost several large blog posts because something failed as I hit submit - often nothing more than a connection drop. And, I believe there's a linear relationship between the likelihood of failure and the amount of text entered wink. I personally use CTRL-A / CTRL-C before pressing 'submit'. At least that way the text is saved in my clipboard.

Don: How do new browser-based word-processing apps by Google manage?

No idea. If its a local app, it may save locally. If not, and it doesn't save it as you type, then it has the same risk.

mike
In reply to Mike Churchward

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Mike > I personally use CTRL-A / CTRL-C before pressing 'submit'. At least that way the text is saved in my clipboard.

That is a very wise precaution. I do it too, especially before submitting a large chunk of text.wink

Joseph

In reply to Don Hinkelman

Re: 1.6.2 Essay Question Bug - what lessons to be learned?

by Debora Weber-Wulff -
Still not fixed in

Moodle 1.9.4+ (Build: 20090318)

And the same thing happens in Firefox, not just IE!

What happened in my case was that students had a < sign in what they were writing (code). This got interpreted as "malformed HTML" and just disappeared. When I put in a page break between questions, it works fine, each has their own editor. But both questions were linked, I needed them to be able to review what they had done on the previous exercise.

I assume that most people have one question per page - I normally do, and I've been using Moodle for final exams for at least 3 years now. Just happened to make sense this time.

In a quiz, nothing should ever be parsed away!

Oh, and I need a "Text is code" button that works better than just "preformatted" HTML, which puts in extra line feeds that are extremely annoying!

The editor needs more work, and should not delete stuff unasked!