UTF-8 conversion problems during Quiz import

UTF-8 conversion problems during Quiz import

by Mat Schaffer -
Number of replies: 9
Maybe this problem is with another module, feel free to recommend that I repost in another forum.

Basically, I'm trying to import a quiz file (GIFT format) that contains Japanese characters.  This file is in UTF-8, but the japanese text gets turned into garbage after I import.

If I create a quiz on the app containing japanese text, then export, the japanese text gets converted to stuff that looks like this:
りんご

So my question is either
(a) is there a bug here that should be fixed?
(b) am I doing something wrong?
(c) is there a workaround?  I know PHP so I could write a conversion script if I knew the functions to use.

Oh, I'm using moodle v1.5.2.

Thanks in advance,
Mat
Average of ratings: -
In reply to Mat Schaffer

Re: UTF-8 conversion problems during Quiz import

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I'm secretly hoping that Mr Takamoto answers this smile For a start, have you got your language set to UTF-8 Japanese? If not the characters will be stored correctly but not displayed correctly.
In reply to Howard Miller

Re: UTF-8 conversion problems during Quiz import

by Mat Schaffer -
It's set to "Do not force".

I tried changing the language of the course to jp_utf8. That makes the quiz import work, but the menus change to all japanese, which would be too much for my beginner students.
So I changed it to English (en). Now all the japanese text that used to look fine is garbage, quizes, titles, everything.
Finally, I went back to "Do not force", but everything is still garbage.

I decided to repeat the process and check the HTML charset. ja_utf8 is charset utf-8, but "Do not force" is using "iso-8859-1" as the charset.  Is there a way to not force the course language but still get "utf-8" as the charset?

I'm attaching a copy of the file I used, just incase you want to check.
Thanks!
In reply to Mat Schaffer

Re: UTF-8 conversion problems during Quiz import

by Timothy Takemoto -

Hello Matt
(Thanks Howard)

I am using Moodle in ja_uft8.

The file you attach looks fine. But as Howard says, it will not display unless the course is displayed using ja_utf8 encoding.

I think that it should display fine if you set your language to english and then manually change the encoding of your browser to "Unicode (UFT-8)."

Your browser may remember that encoding for you, so that it will always view your course with a unicode encoding. At other times in my experience, however, it will keep requiring you to reset the encoding UFT-8 each time you view the page. I am not sure why this is!

However, there is a language pack called
en_ja.zip (858 bytes) 2004-03-23 martin@moodle.com info cvs

This containts almost nothing except instructions to display the pages using a the "EUC-JP" Japanese encoding (till recently the encoding of preference for Japanese pages). Since the pack contains almost nothing all of the GUI strings default to their english variants, only the encoding changes. 

This is the contents...

<?PHP // $Id: moodle.php,v 1.265 2004/03/16 10:59:42 moodler Exp $
      // moodle.php - created with Moodle 1.2 development (2004013101)

$string['thischarset'] = 'euc-jp';
$string['thisdirection'] = 'ltr';
$string['thislanguage'] = 'English - EUC-JP';

?>

What you (and my non-Japanese speaking) teachers need is a en_ja_uft8 language pack, which I am making now....done.... I think that all it needs is the following:

<?PHP // $Id$
      // moodle.php - created with Moodle 1.6 development (2005060201)

$string['thischarset'] = 'UTF-8';
$string['thisdirection'] = 'ltr';
$string['thislanguage'] = 'English - UFT-8';

?> 

Yes. That works. My teachers will be pleased. Thanks for encouraging me to get around to this.

Please unzip attached, put it in your moodle/lang folder and turn off your "langcache" temporarily (or wait a few minutes) in admin settings, then select this en_jp_uft8 language.

Perhaps this might profitably be added to the list of language packs?

Tim
Timothy Takemoto (ne Williams)
PS Recently I remind myself of the Festinger's cognitive dissonance theory in "When Prophecy Fails," summarised here as "scholarship has tended to agree with Festinger's conclusion that adherents work hard in a post-failure moment"smile

In reply to Timothy Takemoto

Re: UTF-8 conversion problems during Quiz import

by Mat Schaffer -
Wow.  Thanks for the quick and very helpful response.  I can't try it yet because I don't have direct access to the system I'm using moodle on.  But hopefully the admin will come back to his desk sometime today and I can give it a try.  I'll post my results when I have them.
Thanks again,
Mat
In reply to Timothy Takemoto

Re: UTF-8 conversion problems during Quiz import

by Mat Schaffer -
Ok, so I just tried it.  We're almost there.
Most everything is working with the new package.  I think en_utf8 would probably be a better name for distribution, cause it doesn't have anything to do with japanese really.

BUT, there's still one (small) problem.  The quiz question preview is still coming up as iso-8859-1.  I guess it's ignoring the course language.

Again, not a huge problem.  Let me know if I should submit a bug for it.
Thanks again.
In reply to Mat Schaffer

Re: UTF-8 conversion problems during Quiz import

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
That does indeed sound like a bug.... please submit a report.
In reply to Howard Miller

Re: UTF-8 conversion problems during Quiz import

by Mat Schaffer -
Done and done. It's posted as bug #4312.

Thanks for the help everyone. Moodle is a great product and I applaud your efforts.
In reply to Howard Miller

Re: UTF-8 conversion problems during Quiz import

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Basically, you can't run multiple character sets at the same time. This is not really a restriction in Moodle it is a fact of life with web pages (you have to put an encoding at the top of the page). So, if your quiz data is in UTF-8 and your Moodle is set to English with an encoding of ISO-8859-1 you are going to have a problem, which encoding should Moodle use to display the page?

The answer, I think, is in using UTF-8 for *everything*, there is an ongoing project to produce a UTF-8 English language pack, I am not certain of the status of this, but you should certainly investigate and see if you can use it.

EDIT:
I think this discussion is *very* relevant to you...
http://moodle.org/mod/forum/discuss.php?d=29029