Import/Export improvements

Import/Export improvements

by Howard Miller -
Number of replies: 24
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

There's a few tracker reports floating around and I've been having some discussions with Pierre offline about a couple of things, so I thought I'd note some of the things I want to do...

  • Fix some inconsistencies in the grade fraction handling in the xml format done
  • Make the xml format handle calculated questions (Pierre?)
  • Improve error handling. Have a more structured way of handling errors in the import file to provide better diagnostic feedback. Will particularly apply to GIFT format in the first instance. Don't have a firm solution to this just yet, but I'm working on it right now!
  • Related and relying on above, allow incomplete imports due to errors to be deleted - working on this too and Pierre has some suggestions.
  • Blackboard formats... Blackboard 5 and 6 formats are actually more the same than different but are handled completely differently which causes huge confusion. Both should handle either the zip file or the enclosed xml file silently. Both should handle multiple xml files within the zip file. Ideally this should be combined into one format which spots which xml format is being used. Enclosed data resources within the zip (e.g., pictures) should be uploaded (and linked to) too.
  • Create/add/complete documentation - http://docs.moodle.org/en/Import_questions
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Pierre Pichet -
To which I could add
  • adding supplementray functions as validating_import_data for each question type so that the code is not repeated in each format type.
  • adding a utf8 conversion function or better doing it silently for the user
more later...
Pierre
Average of ratings: -
In reply to Pierre Pichet

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
More validation - a good thing. But, should this not be (or possibly already is) in the questiontype code?


We had some considerable discussion about UTF conversion in the past and came to the conclusion that it was a bad idea. Unless you can completely silently detect and convert the text then it is ok to simply insist that the file is already in UTF-8. That happens everywhere else in Moodle after all. I guess it would be nice to have a utf-8 validity check though. I understand it is quite hard to do though. I tried to find a routine before but all I found was a massive amount of debate about how it should be done and no firm answers.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Thoughts on errors and delting questions...

In most cases there will actually be no need to delete questions at all. The import is a two part process - the file is parsed and *then* written to the database. Currently no check is made that the parse part of this process was successful. It makes sense to me to put in proper checks for that and simply not go on to the database write bit if that fails.

I think all I'm going to do is to add a method to call in format.php for reporting errors. This can flag a fatal error situation and allow the process to stop before the db write. I think this will actually stop most problems.

I have (Moodle) meetings for the rest of the day, but hope to have this implemented tomorrow. In the mean time any thoughts are appreciated.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Deleted user -

Howard Miller wrote "I think all I'm going to do is to add a method to call in format.php for reporting errors. This can flag a fatal error situation and allow the process to stop before the db write. I think this will actually stop most problems."

I think this suggestion was made in another post, but hopefully the error reporting will show you ALL the errors in file, as opposed to just the first one!!!

And I still like the idea of the user being able to choose whether or not they want to abort the import completely, or just skip the problem questions.

Average of ratings: -
In reply to Deleted user

Re: Import/Export improvements

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Howard, FYI, I've just changed how Moodle XML import slightly. I wanted to make creating, editing, previewing, exporting and importing, questions with the evil question text \ ' \\ \' < > & &lt; &gt; &amp; work. This is in branches 1.6 onwards. I hope I have done the right thing, but you might like to check.

In the process, I think I noticed that GIFT seems to always exports all categories, no matter which one you select in the menu on the export form.
Average of ratings: -
In reply to Tim Hunt

Re: Import/Export improvements

by Anthony Borrow -
Picture of Core developers Picture of Plugin developers Picture of Testers
Howard/Tim - My thumbs up on the import/export improvements. I had not noticed all of the categories being exported by GIFT. Tim, do you mean all of the subcatgories? Peace.
Average of ratings: -
In reply to Anthony Borrow

Re: Import/Export improvements

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think I had three categories, all top level, 'Default' (with 2 questions), 'Export' (1 q) and Import (empty). I tried to export the 'Export' category, and I got a file with 3 questions in.

But I was not paying attention at the time, so I may just be confused.
Average of ratings: -
In reply to Tim Hunt

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
ok - I'll check, *and* I'll try and make the error display / delete thing as flexible as possible. Tomorrow's job smile
Average of ratings: -
In reply to Deleted user

Re: Import/Export improvements

by John Isner -
Jeff,
Re the option to skip problem questions: Can you give a use case for this option? I'm hard pressed to think of one.
Average of ratings: -
In reply to John Isner

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm not sure what you are asking exactly John, but as far as I understand the thinking is...

  • If there is a syntax error in your file and you simply want to fix it then you don't want any of the questions to import, so you can try again.
  • If you are in a big hurry and are importing hundreds of questions you may well not care about a few getting missed (for example a truncated file) and you may be perfectly happy to import whatever you can get.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by John Isner -
Howard,
I was looking for practical situation in which partial import is preferable to "all or nothing." I guess your second bullet is one, but let's complete the scenario.

You (the teacher) went to a lot of trouble to write those questions, yet you were willing to let some of them drop on the floor because you were in a hurry. You'll get around to picking them up later, when you have time.
  • in the meanwhile, your GIFT (or whatever) file contains errors. How are you going to remember which questions had errors?
  • you're eventually going to have to isolate the offending questions in a file by themselves to avoid duplicates.
This is not a tradeoff I would make, no matter much of a rush I was in.

What if, after several tries, I still couldn't get my file to import under the "all or nothing" rule? Say I have a really complicated CLOZE question with LaTeX markup and lots of special characters and backslashes, and I'm not sure whether it's me (a syntax error) or Moodle (a parsing error). I think that's a different case. In that case, I'd want to remove the question from the file and post it to the quiz forum.

Average of ratings: -
In reply to John Isner

Re: Import/Export improvements

by Deleted user -

John Isner wrote "Can you give a use case for this option? I'm hard pressed to think of one."

Actually, no I can't... I just like the idea being able to choose!!!

As long as the import tells me where ALL the problems in my file are, then I'll be happy, even if it deletes all the questions!

Average of ratings: -
In reply to Deleted user

Re: Import/Export improvements

by Just H -
And I still like the idea of the user being able to choose whether or not they want to abort the import completely, or just skip the problem questions.

+1 for that.

I've just read a couple of threads where there seems to be a leaning towards deleting all questions that have been imported from a file that fails at one point or another. Not sure of the logic on that.

IMHO, if I was trying to import 100 questions (not sure why people choose 100 when discussing this smile ) I think it would make more sense for the import script to flag a question with an issue and let the user decide whether to skip or abort and whether to delete or keep questions successfully imported.

At the end of the import script a report stating what questions where skipped would then allow investigation at a later date. A lot easier to add 3 or 4 questions by hand than sitting there getting more and more frustrated because every time you try and import the 100 questions it bombs for what may be just one question.
Average of ratings: -
In reply to Just H

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I'm not sure (at the moment) how well this will work for all formats, but I mostly have GIFT in my head as thats the one that's likely to be hand coded.

Each question is parsed if it fails the question is printed out with a (hopefully) decent error message, the parsing continues.

At the end of the parsing if there are errors, "Errors found, do you want to try to store questions? Yes/No?". Simple as that.

If there are errors in the database storage then, I think, too bad. That's more likely to be a bug report in reality (in fact a nice message to that effect might be an idea).
Average of ratings: -
In reply to Just H

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Ok it's done - in 1.8 and HEAD (CVS).

It only really works for GIFT at the moment. But I will work on others over the next few days.

There is now a checkbox on the import screen called "Stop on Errors" which does pretty much what it says. The entire file is parsed and any (all!) errors that are detected are flagged up. I have tried to improve the display of such errors a bit too. If there are any errors then it will only go on to the write-to-database phase if the stop on errors box is unchecked. Otherwise, no questions are stored.

Simple as that. If anybody could have a look I would appreciated any feedback.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Anthony Borrow -
Picture of Core developers Picture of Plugin developers Picture of Testers
Thanks Howard! I have my test server updating from CVS every hour and I will try to go in later today and play with some GIFT importing. Peace - Anthony.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
On further investigation it seems that GIFT is the only format that grasps the idea that there might be errors in the input. A worry in itself sad
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Stephen Francis -
Just as a quick note in case this hasn't been spotted or sorted. GIFT importing does not like the degree symbol (ALT 0176) ° possibly others as I haven't tried others. All information after the symbol isn't imported correctly for that question. We're using 1.6.4+.

Steve
Average of ratings: -
In reply to Stephen Francis

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Can you post an actual example so that I can try it.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Stephen Francis -
This file (attached) is GIFT file with degree symbols in it (it is one question multiple choice question). Create a "TEST" category in the questions section. Look at the TEST category and then import this file using the GIFT format.

Steve

Average of ratings: -
In reply to Stephen Francis

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Stephen,

I checked out your file and it is not UTF-8. It is a requirement that all imported files are in UTF-8 format. It appears to be (as my Mac calls it) "Windows Latin 1" format. You have been got-at by the evil empire smile

EDIT.
I have converted your file to UTF-8 and re-attached it so you confirm/check it works now.
Average of ratings: -
In reply to Howard Miller

Re: Import/Export improvements

by Stephen Francis -
Howard,

Many thanks for this, just not sure how to make it UTF-8 format thats all, but I'll have a look.

Cheers

Steve
Average of ratings: -
In reply to Stephen Francis

Re: Import/Export improvements

by Stephen Francis -
Ignore that post, it's Unicode (UTF-8) that I need, silly me! and I know how to do that.

He he
Average of ratings: -
In reply to Stephen Francis

Re: Import/Export improvements

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
It depends what you created it with.

At a push a good editor should be able to convert it. I use a Mac and you obviously don't, so I can't recommend anything, but I would check out those free programmer's editors.
Average of ratings: -