Moodle in English: escaping problem in gift import?

escaping problem in gift import?

by A. T. Wyatt - Tuesday, August 8, 2006, 8:16 AM

I was working with my old plain version of the Timothy's much fancier(!) excel version of a gift creator.

I found that I had a big problem when importing question stems that had apostrophes. The importer displayed strange characters where the apostrophes were (but the right number of questions were imported), and then in the question preview, the question strings were truncated at the point where the apostrophe occured.

My question is, should we put the question strings in quotes?
Or is there a bug in the importer?
How shall I make sure that my instructors do not lose parts of their question stems?

I was going to try aiken and see if that worked any better, but I wanted to ask this question in the meantime.

I am using Moodle 1.6.1 from about July 25.

Thanks for any help!
atw

Re: escaping problem in gift import?

by Tim Hunt - Tuesday, August 8, 2006, 3:30 PM

It is a bug somewhere. It is related to the switch in Moodle 1.6 to using the Unicode (UTF-8) character encoding, which means that some of the quiz code that used to work doesn't any more.

The sympton is that bits of text get cut off at the first 'special' character.

Can you post one of you GIFT files that demonstrates the problem. That would help us investigate.

Re: escaping problem in gift import?

by A. T. Wyatt - Tuesday, August 8, 2006, 8:23 PM

This is plain US English! big grin

Here is the offending file. I think there are about 3 questions that have an apostrophe. We could reword to avoid apostrophes, but occaisionally that would be very awkward. Example: Surgeon General's Report. That is really what it is called, and the Report of the Surgeon General is just NOT the same thing!

Thank you guys for your help.

atw

chapter1.txt

Re: escaping problem in gift import?

by Joseph Rézeau - Tuesday, August 8, 2006, 8:55 PM

In your chapter1.txt file, just replace all apostrophes with their HTML entity: ’

Et voila!

However, I hope Tim or someone will find a fix for this bug... Moving Moodle to UTF-8 was supposed making dealing with special characters easier, not more difficult. wink

Joseph

Re: escaping problem in gift import?

by A. T. Wyatt - Tuesday, August 8, 2006, 9:40 PM

I am glad to know this workaround for the short term!
Thanks!
atw

Re: escaping problem in gift import?

by Howard Miller - Tuesday, August 8, 2006, 4:41 PM

I'll check....

Just to be sure - how is your GIFT file encoded? Are you using non-latin1 characters, or anything like that??

Re: escaping problem in gift import?

by Howard Miller - Tuesday, August 8, 2006, 9:25 PM

No fault found I'm afraid. Your file seems to be the problem. When I opened it on my machine it had strange characters in place of the apostrophe's. When you correct them it works just fine!

I have tested in 1.5 and 1.6 just to make sure.

Please see the attached image - this is what it looked like on my editor (I tried it in more than one editor too - no different).

I'm checking out the special character truncation thing - but that's not really the problem you were having.

Re: escaping problem in gift import?

by A. T. Wyatt - Tuesday, August 8, 2006, 9:43 PM

Following up--
Do you think it has something to do with excel? I had copied and pasted the output into notepad, then uploaded the .txt file.

Well, thank you for the hints. I will keep digging at it!

atw

-----------------------
edit: Additional thought. You know, the original text was in a Word file (the test bank provided by the publisher ONLY outputs to word or a proprietary format, so we were using the gift creator to try to get an importable file). We copied the question stems from word to excel, then got the concatenated output and put it in notepad. Perhaps the special characters were transfered from the office stage of the process? You never know about MS Word. . . .

Re: escaping problem in gift import?

by Howard Miller - Tuesday, August 8, 2006, 10:03 PM

I think that's a distinct possibility - Word does do some strange stuff with quotes and apostrophes at times.

I am not in a position to try this out I'm afraid. I don't currently have access to a Windows machine (happy Mac user

), and the Excel to Gift stuff doesn't work in the Mac version of Excel.

You may have to do a bit of research, as to where it is coming unstuck!

EDIT:
I saved some text from both Excel and Word with apostrophes to text files and no odd behaviour at all.

Re: escaping problem in gift import?

by Howard Miller - Tuesday, August 8, 2006, 11:06 PM

I've also been unable to reproduce the truncation problem. I just tried this in 1.6.1+ with the original test file supplied (with the 'bad' apostrophe characters). They come in as question marks, but are not truncated.

Possibly something has been fixed recently.

Re: escaping problem in gift import?

by Joseph Rézeau - Wednesday, August 9, 2006, 1:00 AM

Howard > [the apostrophes] come in as question marks...

Exactly, that's why I suggested replacing them with HTML entities, which does solve the problem (pending further fixes)...

Joseph

Re: escaping problem in gift import?

by Howard Miller - Wednesday, August 9, 2006, 1:44 AM

In this particular case, I don't think it's a problem - it just looks like an encoding mismatch. Imported GIFT files must be encoded in UTF-8 for Moodle 1.6 of course assuming you stray outside the latin1 characters - intentionally or otherwise

Re: escaping problem in gift import?

by Peter Sereinigg - Friday, August 11, 2006, 4:36 PM

Hi Howard,

its not only a problem with GIFT, you have the same situation with user bulkuploads ....

In an NON UTF_8 PC environment and MS-Office ... do you have to copy exported files in notepad2, change to utf_8, save it and then import this? You cant save from Excel or Word as an UTF_8 File ...or?

Or is there another more handsome way to do so,

thanks for hints!

Peter

p.s.: in config.php there is still from 1.5

$CFG->CSV_DELIMITER = ";";

$CFG->CSV_ENCODE = "59";

to translate "," to ";" for uploads ...

Re: escaping problem in gift import?

by Howard Miller - Friday, August 11, 2006, 5:29 PM

I'm a Mac user, so I can't be too specific, but yes you need to use your favourite tool to get the file into utf-8.

There was some discussion around this topic - we came close to having an option for the encoding of input files (in quiz importing at least) but decided against it. For one thing there are millions of the things, and the list depends somewhat on your environment and what libraries you have available. Secondly, we thought that if you usually use non-latin encodings you would be familiar with editing in and/or converting to utf-8. A dreadful cop-out I know

Re: escaping problem in gift import?

by Peter Sereinigg - Friday, August 11, 2006, 5:41 PM

Thanks for your answer, sometimes its a problem not to be an english nativspeaker and not a mac user ...

We will try to find a handsome workaround with an description and with downloadlinks, because most of the users have an idea what utf_8 could be ...

Peter

Re: escaping problem in gift import?

by Peter Sereinigg - Saturday, August 12, 2006, 1:24 AM

Here is one solution:

Use Notepad2, http://moodle.org/mod/forum/discuss.php?d=51569 BUT dont convert with datas in it (you may loose conversations!) , first change a blank form to UTF-8 THEN copy the text in notepad2 THEN save as UTF-8

Its just a workarround, but helps

Peter

DONT use windows notepad and save as UTF-8, because then a unknown sign will be in the first line - this raises problems when bulkuploading users with csv...

Re: escaping problem in gift import?

by Howard Miller - Saturday, August 12, 2006, 2:13 AM

I can feel a page in the docs wiki coming on with all this useful advice

Re: escaping problem in gift import?

by A. T. Wyatt - Friday, August 11, 2006, 5:42 PM

Apologies for my ignorance, but how do you
"copy exported files in notepad2, change to utf_8, save it and then import this?"

I am lost. I have not a clue about how to change a file to utf_8.
atw

Re: escaping problem in gift import?

by A. T. Wyatt - Friday, August 11, 2006, 7:07 PM

Aha, I found it! Well, this is to help anyone else. You can change the encoding when saving in notepad--ANSI, unicode, and utf-8. The drop down is at the bottom of the save as settings.

So I will try this and see if it solves our problem!

edit: yes, worked like a charm.

atw

Re: escaping problem in gift import?

by A. T. Wyatt - Wednesday, August 9, 2006, 1:11 AM

Right; mine looked like little black diamonds with white question marks. But they did truncate.

What a strange thing!

atw

Re: escaping problem in gift import?

by koen roggemans - Wednesday, September 13, 2006, 4:12 AM

I keep on strugling with this Word related apostrophy problem.
Creating new tests with notepad2 and GIFT seems like a very workable sollution, but I have loads of tests (should be over 2500 questions) with apostrophy problems in Moodle already (yes, some teachers prefer to use Word to produce text documents verrast

).

In IE I get a square and FF just ignores the apostrophy and shows nothing (like jai instead of j'ai)

Running Moodle 1.6.2 with migrated database (updated a minute ago goedkeurend

).

Any suggestions how to solve this ?

Re: escaping problem in gift import?

by koen roggemans - Wednesday, September 13, 2006, 5:03 PM

found it
Thanks to Yu!
see http://moodle.org/mod/forum/discuss.php?d=50821#245922

Re: escaping problem in gift import?

by Anne Ogborn - Friday, May 22, 2009, 11:47 PM

Turning off autocorrect fixes many of the problems with word and strange characters.
Word is trying to pretty up your text by replacing
apostrophe with open single quote and close single quote. Of course that means guessing which is which. Predictably it does an imperfect job.

Quiz

escaping problem in gift import?