Moodle in English: Japanese Language Compliance - Please would you merge this with Moodle?

Japanese Language Compliance - Please would you merge this with Moodle?

by Timothy Takemoto - Wednesday, 6 October 2004, 6:43 PM

Several of the users over at the Japanese forums, particularly Mr. Kashiwagi, have made a Japanese language pack that cures all (?) of the character garbling problems that Japanese had. He is sad that this has not been taken up by the main Moodle install but that is my fault for not having relayed the information on that forum to this.

The following is a rough translations of Mr. Kashiwagi's site at
http://www.supertak.com/down/sample.htm

Moodle ver. 1.5 dev (rev. 9/1/2004)
changes to	graphlib.php / moodlelib.php / Worksheet.php / Format.php
Instructions	Environment：Windows 2K + Apache 1.3.31 + PHP 4.3.8 + Moodle 1.5 dev + GD2 （Checker Takahito Kashiwagi） Environment 2：WinXP Home + Apache 2 + PHP 4.3.8 cgi版 + Moodle1.5 dev + GD2 （Checker Mr. KAGO） The following improvements to the Japanese language handling abilities have been testsed on the above systems. ＜主な対応＞ Making GD2 work with Japanese. Unifying mail sent out by Moolde to iso-2202-jp, thus avoiding garbling when sending text mails even to mobile phones. (The standard release only succeeds with HTML mail) Date display in Japanese format (Mr. Kago) Correct display of Japanese in the TEXT/EXCEL worsheets (e.g. grades), and the acceptance of Japanese file names. When importing files to LESSON/QUIZ, Japanese the Japanese (encoding) (Mr. AOYAGI）. Adding the ability to NOT distinguish between two bite and singe bit roman characters inQUIZ marking. Making the WIKI module work with Japanese (Mr. Yoshida） Implementation and Instructions We use various FONTs from http://sourceforge.jp/projects/efont/ The Japanese library is jcode.php from http://www.spencernetwork.org/ We enclose a text file detailing copyright of fonts and code packages used. Please accept these limitations. First of all download langlib133.lzh and put the fonts and lib directory in the Japanese language folder lang/ja/ The following changes below are to instruct the main moodle code to check for the presence of langlibl. GD's Japanese language dispaly capability now uses 「lang_decode.php」 in lang/ja/fonts This is using the function extensions provided by Moodle. Hence the changes to graph.lib recommended previously are no longer required. Please make the changes indiciated in red below. locale is Japanese_Japan.20932（Windows） or ja_JP.EUC（UNIX）
Date	2004/9/6
Version	1.3.1
Author	Takahito Kashiwagi
HIstory	2004/9/1 first edition

Please see the above site for the code that follows the table.

The red code is NOT needed (is is from previous recommendations), the blue coded consists of the editions.

I have hacked my moodle so much that I am scared to touch it anymore. It is already going to take a day to move up to 1.4.2 Which I need for backups. So I have not tested the above, but the Japanese folks over there have and it would be really great, indeed essential to have a Japanese compliant Moodle if we are to use it for anything but English teaching here in Japan.

Tim
Takemoto

Average of ratings: -

Re: Japanese Language Compliance - Please would you merge this with Moodle?

by Paul Shew - Friday, 15 October 2004, 2:49 PM

Thank you for translating this, Tim, and bringing it to the attention of the wider Moodle community. I would also like to thank Mr. Kashiwagi and others who have contributed solutions to the problems.

As a 2-byte language, Japanese faces some particular problems that Western languages don't. As Tim points out, using an unmodified version of Moodle presents some real problems here in Japan.

I'm a bit concerned that this post hasn't been followed by any discussion, so I'm going to highlight a few reasons why this is so critical.

*Making GD2 work with Japanese.
If GD doesn't work, then the "Activity Report..." graph is just garble.

*Unifying mail sent out by Moodle to iso-2202-jp, thus avoiding garbling when sending text mails even to mobile phones.
The Japanese version of Moodle uses EUC-JP which is standarded for Lynux, but is not used natively on Windows, Mac, moble phones, web-mail, etc.. Some email clients display it properly in HTML mode, but many do not. Not being able to read the email is one of the primary complaints from my students.

*Date display in Japanese format (Mr. Kago)
I do not see a problem here. Back in versions prior to 1.0.8 we had this problem, but it's been solved as far as I know.

*Correct display of Japanese in the TEXT/EXCEL worsheets (e.g. grades), and the acceptance of Japanese file names.
This is a major problem. Excel cannot deal with Japanese text encoded as EUC-JP. For most Moodle-user, this problem renders the export function useless.

* When importing files to LESSON/QUIZ, Japanese the Japanese (encoding) (Mr. AOYAGI）.
I had been encoding my files to EUC-JP before uploading, but for most users, doing that kind of a text-encoding conversion on a text file will be well beyond their abilities.

* Adding the ability to NOT distinguish between two bite and singe bit roman characters inQUIZ marking.
I just caution my students to be aware of what text-mode they're typing in, and then mark it wrong otherwise. But adding this ability would really be slick.

* Making the WIKI module work with Japanese (Mr. Yoshida）
I've never used the WIKI module, so I cannot comment.

These are all problems that have solutions. The solutions have all been coded, and the code is available here:
http://www.supertak.com/down/sample.htm

Is there any reason not to include these in Moodle?

Paul.

Average of ratings: -

Re: Japanese Language Compliance - Please would you merge this with Moodle?

by Timothy Takemoto - Tuesday, 7 December 2004, 10:20 AM

Hi Paul, Moodle Developers

Thanks for your support Paul.

Is there perhaps some way that we could pay to have the code above included in moodle, and if so for how much? Perhaps I could set up a pool on the Japanese forums.

I am scared that as moodle advances the above code,that someone kindly created to make 1.4.1 Japanese compatible, and those persons may not be so kind as to go over the process again.

BTW, there are still a few places where the date is the wrong way around, when a short date is displayed, I think.

Tim

Average of ratings: -

Re: Japanese Language Compliance - Please would you merge this with Moodle?

by Don Hinkelman - Tuesday, 14 December 2004, 1:15 PM

Paul,
Have you tested the code on your server? If it works OK, there should be no reason not to include it in 1.4.x.
Don

Average of ratings: -

Re: Japanese Language Compliance - Please would you merge this with Moodle?

by Martin Dougiamas - Tuesday, 14 December 2004, 3:52 PM

My memory is hazy but last I looked at this I think there were going to be problems with non-Japanese languages .... is that right? I could well be wrong on this ...

Also, how does it relate to the move to Unicode for Japanese that Mits is working on? If we are going to use Unicode do we need much of this?

Average of ratings: -

Re: Japanese Language Compliance - Please would you merge this with Moodle?

by Timothy Takemoto - Wednesday, 15 December 2004, 4:18 AM

Martin

I will ask and get back to you soon.

I think that there would be no problem with non Japanese languages since the changes are packaged inside the ja file. I.e. I think that they put code inside the language file that handles things like creatinging images in Japanese fonts. Japanese languages code English fine. So if someone selected ja as their local then they would be able to display Japanese and English. Perhaps they would have problems displaying Japanese and French simultaneously.

I think that most of the changes would have to be made whether one uses unicode or not but I will get back to you on this once I have spokent to Mits.

Tim

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Don Hinkelman - Tuesday, 14 December 2004, 1:13 PM

Dear Tim, Paul, Kashiwaga-san, and others,

This is a wonderful piece of coding that solves serious problems nagging all Moodle users in Japan.

Tim, may I suggest that you post again (with code attached) in the bug/feature tracker? There it will be assigned to a programmer. I think this should go into 1.4.x (Martin Langhoff is now in charge of these stable branch improvements) as well as 1.5dev (Martin D. is in charge of that). If this coding has been tested on Kashiwagi's server and Paul's server with no bugs, then that should be explained and there is no need to consider fund-raising.

Don

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Paul Shew - Monday, 20 December 2004, 8:55 PM

I've been engulfed in teaching and other responsibilities at my school, and haven't had a chance to check in here in the past week. I'm delighted to see this issue getting attention. During the Christmas/New Year holiday, I'm planning on upgrading my server to 1.4.3. (Thanks, Martin, et al!

) When I do so, I'll try to go through and test as many of the Japanese code issues as I can.

My moodle server is now handling 8 classes and almost 700 students--all in Japanese. (I teach courses related to Christianity and comparative religions, not English.) Several Japanese faculty at my university have expressed interest in using Moodle too. But before I can recommend Moodle to them, these Japanese language issues have to be resolved. I'm glad that things look like they are moving in the right direction.

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Tuesday, 21 December 2004, 8:48 AM

Hi Don, Paul

Yes, I agree. The bug tracker is the way to go.

But the bug tracker is usuallly for seperate bugs. This comes as a complete solution with lots of links thoughout Moodle to check for the presences of a special set of php routines inside the language folder, when normally there are only a long list of passive translations inside that folder.

So perhaps the bugs need to be posted seperately?

I was hoping for confirmation from Mr. Kashiwagi that the Japanese language enhancements have no effect upon other languages. Alas, Mr. Kashiwagi and the others involved in making Moodle work with Japanese seem to have stopped checking their mail on the Japanese language forum. In any event I have not had a response yet.

Can you confirm that the patches do not effect Moodle's capability to handle other languages Paul? Please let us know when you can.

By the way, have you seen the book project on the Japanese language forums?

Tim

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Friday, 7 October 2005, 1:00 PM

Mr. Kashiwagi has alas left the forums but a kind and knowledgable professor Kita has taken up the patch work (npi) and published the following

An RMP combining Mr. Kashiwagi's extra files and patch for vine linux 3.1
http://t-kita.net/rpm/FC/moodle/

A read me file in English describing the rpm
http://t-kita.net/rpm/moodle/README-rpm-en.txt

A patch describing all the things that need to be done
http://t-kita.net/rpm/FC/moodle/patches/moodle-t-kita.patch

According to Mitsu, there are no affects on other languages.

However, with regard to the question of how much would need to be done when Moodle becomes unicode compatible, I am afraid that I have not understood the answers that I have been given. Mr. Kashiwagi had already left the forums, I have not found time to apply a patch of this magnitude, and I have, in my ignorance, not understood the answers that Mitsu has given. I am now asking Professor Kita the same questions.

It appears (IMHO) that
1) the biggest problem is the use of htmlentities in the moodle code.
2) the problem can be solved in part by the use of htmlspecialchars but some special characters are not converted correctly
3) Professor Kita's patch offers replace-htmlentities.pl which replaces both the above. I think.

Mail, excel files, graph displays all garble in moodle in Japanese at the momement.

It would be tremendous help if this patch were to be applied.

The patch is massive, and applies to many things. I am going to have a look in the bug tracker and see which of the current bugs are related to these issues and possibly post a new one.

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Saturday, 8 October 2005, 4:09 PM

According to Mitsu, the problem with htmlentities is one which will apply even after the move to unicode.

The problem with mail also. I know because all my mail is sent out in Unicode, but there are still many mail clients (particular that which is used by my university) that do not display unicode.

And the problem with excel too will remain after the move to unicode, unless microsoft gets its act together and allows excel to open unicode files.

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Haruhiko Okumura - Saturday, 8 October 2005, 7:28 PM

The problem with htmlentities(x) is that it is equivalent to htmlentities(x, ENT_COMPAT, ISO-8859-1). This fails for all encodings except ISO-8859-1. The bottom line is that all occurrences of htmlentities(x) should be replaced by htmlentities(x, ENT_COMPAT, get_string('thischarset')), whatever encoding you use.

Professor Kita kindly provided a useful script: http://t-kita.net/rpm/moodle/scripts/replace-htmlentities.pl

Average of ratings:Useful (1)

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Saturday, 8 October 2005, 8:24 PM

Thank you Professor Okumura. You sound like you know what you are talking about. I do not know what I am talking about. But I do hope that the patches being provided by Prof Kita and Mr. Kashiwagi are applied.

By the way though, the script that Professor Kita provides seems to be a perl script. Is that so? If so is there a good reason for re-writing it in php? And is there anyone that can do it?

muchi no

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Haruhiko Okumura - Saturday, 8 October 2005, 9:48 PM

Yes, replace-htmlentities.pl is a Perl script that converts every occurence of htmlentities(x) to htmlentities(x,ENT_COMPAT,get_string('thischarset','moodle')). This script is only useful to developers who only want to rewrite htmlentities() and see what happens. There are other encoding-related glitches which cannot be patched by running this script. Prof. Kita's patch data, http://t-kita.net/rpm/moodle/patches/moodle-t-kita.patch, incorporates these changes plus much more. Normally you only need this patch data. In short, there is no good reason for rewriting the Perl script in PHP.

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Monday, 10 October 2005, 3:34 AM

Dear Professor Okumura
Thank you. I understand. Prog Kita's script is just a quick way of changing the code.

Dear Developers,

Will the script be applied to the moodle code?

I don't think that it will cure the email problem but it will solve some issues such as, perhaps that with the current wiki.

Since I do not have server access I cannot apply Dr. Kita's patch automatically, and it is too long for me to attempt by hand. I would be sure to miss a ";" somewhere and anyway I would have to do it each time I upgrade moodle.

Importantly, apparently none of the changes would affect moodle in other languages. Nor would they be solved by the move to unicode.

'My hand is sticking out of my throat' for the functionality: a non-garbling moodle.

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Toshihiro KITA - Monday, 10 October 2005, 10:38 PM

I belive the htmlentities()-part modification should be essential to utf8-ized Moodle if they want to avoid the corrupted characters. So I do not think it will be long before all the htmlentities() are called with charset specification.

E-mail encoding issue is a bit different, I think. But on this issue I wish utf8-enabled mailers will be so common in Japan that we do not need care about encoding conversion to JIS. (Some think it is already so, it is not impossible to force students to use utf8-readble mailers if they want to join your course)
But for several years from now, it is better to embed email charset conversion to JIS in Moodle as my patch.

# Excel file issue would be fixed if I could modify Kashiwagi's to patch the current version of Moodle. I need some more time for it.

BTW, Takemoto-sensei, please tell me your environment where you ordinally hand-patch your Moodle and you are running Moodle.
I can make a patched Moodle or provide some easy way to semi-automatically patch for you (and maybe also for others).

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Paul Shew - Tuesday, 11 October 2005, 12:45 PM

Kita-sensei, I think you're right about Moodle's conversion to Unicode. It will solve a lot of problems for Japanese, but the email and Exel encoding are a different matters.

Until these issues are addressed in the standard Moodle package, I think that Japanese Moodle users need a patched version of Moodle for every major release. I know that Mitsuhiro Yoshida-san has made patches for most of the encoding problems and so have others. Rather than duplicating efforts, and making everyone apply patches on their own (which some people have a very hard time with), I think it would be very benefitial to make a patched version that could be downloaded from the Japanese users forum or even at Moodle.jp.

Unfortunately, Moodle still needs more than simple language translation to make it compatable with Japanese, and until those changes are made, Moodle's potential in Japan is limited.

Paul.

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Tuesday, 11 October 2005, 9:25 PM

Dear Paul,

Bearing in mind the fact that the proposed changes do not (so I am told) effect other languages, it seems to me that it would be appropriate for the stable branch of moodle to be patched. (Assuming that anyone has the time and generosity to do so). I would be very grateful.

As far as I know patch files can be run from the command line to change everything automatically. But I do not have command line access.

At the same time, the suggested patch are mainly or only of use to Japanese language moodle users. So perhaps you are right, for grace and simplicity, the main stable branch of moodle should not be patched to suit the needs of Japanese?

Japanese is the third most used language on the internet. It is a massive market. The Japanese are pants at software (other than game software).

Today I spoke to one of my bosses for the first time in a long time. He is the guy that spent the university budget on getting a programmer to write another moodle. He took the piss out of the name "moodle" (saying it sounded like "noodle" - as in "instant noodles") and closed by saying that I should purchase something from a Japanese company. Frigbat. Ardvark. Why would anyone want to dance with people that say things like that? Well...not all do.

I dream of non-garbling mail. I will try the patch manuallly soon.

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Toshihiro KITA - Tuesday, 11 October 2005, 10:23 PM

I can not imagine well your environment to run your Moodle, but
maybe you extract all the Moodle files on your PC and upload them to a server, right? If so, I will make a zip file of patched Moodle for with a bit of worry about someone might be unhappy to see a variant of the official Moodle package.

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Wednesday, 12 October 2005, 12:08 AM

Thank you very much. You are right. I unzip moodle and the use FTP to upload.

I don't think that there is any problem, under the licence, of making a variant of the official Moodle package, so long as we keep the licence and the copyright notices intact.

Timothy Takemoto

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Paul Shew - Wednesday, 12 October 2005, 9:30 AM

Kita-sensei,

I think it would be great ifyou can creat a patched version. There should be no problem at all with the license, since Moodle is GPL. That's the beauty of the GPL! Plus, we're not proposing the creation of a fork, just a patched version for Japanese language compliance. I'm sure that others would be willing to help too.

Here's a few of my quick thoughts on what would be nice to include in the "Japanese Compatible" patched version:

HTML entities:
Email
Excel
Bug #4132
Bug #4156 (include font)

Tim, after the patched version is created, can we include it in the files section of the Japanese forum. Then, can we create a new block on the Japanese forum main page with a link to download it?

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Wednesday, 12 October 2005, 1:07 PM

Dear Paul,

Thank you very much for taking this up. I can feel resolution -- an end to garbled moodle -- in the air.

It is no problem by me to put the patched moodle in the file space of the Japanese, but course is (like this) on Martin's server so I am not sure how he will feel about it. Martin?

Mitsu wrote to me recently, thanks to you, mentioning your suggestion of moodle.jp or perhaps his site.

But...

I am still failing to understand why the stable moodle branch can not be patched (why there needs to be two moodles). These patches do not affect the functionality of Moodle in any other language.

They do add to the download size (if the fonts are included) but as far as I know, moodle is distributed ready to run in many languages as part of moodle policy. Most of the changes would hardly add to the file size at all. IMHO the patch should be applied to moodle stable.

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Paul Shew - Tuesday, 11 October 2005, 10:50 PM

Yes, most things should be in the main branch. I think that full Unicode support in 1.6 will clear up most problems for Japanese unicode sites. But two issues unique to Japan will remain: Email text conversion and Excel file conversion.

I don't know if Martin is willing to make a special exception for converting email to iso-2202-jp for Japanese. And we need a similar special exception for converting text in the export routines for downloading the grades as an Excel file.

If Martin is willing to incorporate these necessary special considerations for Japanese into the main branch then great. But if not, then we need to make a seperate patched distribution for Japanese.

Frankly, since the code have already been developed by various people, I think we should create a patched version of 1.5.2+ now, and immediately make it available through the Japanese forum.

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Wednesday, 12 October 2005, 1:48 PM

Dear Paul

Oops, sorry, you replied to my earlier question as I was reiterating it above.

As far as I know, the patch to convert email to iso-2202-jp are quite small in comparsion with the size of moodle. But perhaps there are other reasons why the patch should not be incorporated into the main body of the code. I suppose the code might become riddled with
if ($CFG->locale = "'ja_utf8') {
...
}
...
if ($CFG->local = "xyz {
...)

I wonder what moodle / Martin's policy is on this issue.

Any way, I am really grateful to hear that I may be in reciept of a patch (both Mitsu and Professor Kita have offered to make me one) but also thinking of
1) Everyone else
2) The problem of having to keep patching new versions
3) The proliferation of localised versions of moodle on other sites
I wonder if a Japan-localised fork is really the way to go.

Tim

Average of ratings: -

Incorporating Japanese compatibility in 1.6

by Paul Shew - Wednesday, 12 October 2005, 4:33 PM

Yes I think these definitely can and should go into the main 1.6 release. These are long-term solutions for long-term problems, and it does not create problems for other languages as far as I know. Martin and/or Eloy, can you please comment on incorporating these changes into 1.6?

But it may be a while before 1.6 is released, so I'm proposing that we make available a patched version of 1.5.2+ ASAP as an interim solution.

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Tuesday, 11 October 2005, 9:41 PM

Dear Dr. Kita,

Thank you very much for your response.

I hope that the developers agree that there is a need to change html entities.
At the same time, I don't think that I am having a problem with htmlentities, other than in the wiki, which I am hoping will be replaced soon with the new DFwiki.

Email encoding and Excel files are the main problems I have.

Excel files: Teachers want to be able to download grades. They do not find it easy to convert the text file from UTF8 to JIS. I have to do it for all 20 or so classes. Some teachers want a copy of the file part way through the term.

Email: Email is the thing that causes ulcers. The special virus proof, 'deliberately low technology,' email client installed on my university's computers, MaiYU, is not UTF8 compatible. I am not sure when they will change but as you say it will be years before there is complete UFT8 compatibility. When one has several hundred students enrolled (e.g. about 600) then if only one in 10 is using an email client that does not cope with UTF8 then that is a lot of complaints. In my case about 50% of the students are using the university's comptures and getting garbled mail! Ulcers! I get around the situation by writing forum posts in English and then Japanese so that at least the students can see the English. I can do this since I am mainly teaching English. But moodle is just NOT going to work for normal courses taught in Japanese.

I have several moodles but the main ones are 1.5.2 on Free BSD and Mysql. I have FTP and SAMBA access.

If you would be so kind as to make a patched moodle that would be great.

But at the same time, I have to patch my moodle for other reasons (the Japanese privacy laws included) so I am really keen that the main branch of moodle become Japanese language tolerant.

Perhaps there might be enough people to put some money together. I am in debt recently but if it were not too much and there were enough of us that wanted a change...

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Toshihiro KITA - Tuesday, 11 October 2005, 10:50 PM

I see your situation a bit clearer. Thank you.
MaiYU will be hopefully UTF8 compatible if many people request so to the author in Yamaguchi University. Recently many applications send e-mails in UTF8. So does WebCT and others.

I guess you use Windows XP to unzip your Moodle, so I will make some semi-automatic batch file to patch the Moodle files in your PC (within a day or two, hopefully...)

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Timothy Takemoto - Wednesday, 12 October 2005, 12:02 AM

Professor Kita

Thank you very much indeed for your response.

I have contacted the programmer of MaiYU (one of our staff) about a year ago, but as yet he has not made MaiYU uft8 compatible.

I use Windows 2000 to unzip moodle and then upload it to a university server using ftp.

I hope that there can be some solution to the garbling problems. Thank you very much indeed for your help.

Timothy

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Eloy Lafuente (stronk7) - Wednesday, 12 October 2005, 8:32 AM

Hi,

interesting discussion about utf and problems. As moodlers running your servers under UTF-8 since some time ago, I think that all your experiences and ideas will be really important to improve the UTF-8 migration of Moodle.

Some days ago I started writting the "Migration to UFT-8 wiki page. It's unfinished for now (I hope to write all my initial thoughts ASAP, to discuss them properly) but perhaps it could be a good idea if you relate some of your well-known problems and how to solve them. The solution must work under UTF-8 for everybody, of course.

Perhaps, the Recoding PHP scripts page would be a good place.

TIA and ciao

P.S.: Just trying to get some free hours to finish the initial document. Please be patient...

Average of ratings:Useful (1)

Unicode implementation

by Paul Shew - Wednesday, 12 October 2005, 10:40 AM

Eloy, Good to see the work you've been putting into this.

You should considering adding information about UTF-8 pitfalls or incompatibilities that we're going to run into. That's one of the main problems for Japanese. Unicode is still not widely supported by email clients in Japan, so Unicode sites need to convert all outgoing email into an email-compatible text encoding. Excel files downloaded from the gradebook suffer a similar problem. We're obviously very eager to have these problems addressed as soon as possible.

Really the underlying problem is that the desktop environment (MS Windows or Mac) is not Unicode based, so all text importing (like quiz questions) and exporting routines (email, gradebook, quiz export, etc) need to compensate for that. For languages like English, it may not matter much, but in Japanese it's a show stopper!

I suspect that other languages may run into similar problems.

Average of ratings: -

Re: Unicode implementation

by Tim Allen - Thursday, 13 October 2005, 12:57 AM

I suspect that other languages may run into similar problems.

I am still testing these problems in Korean to be sure, but I am fairly sure that the same problems will occur. I suspect that all multi-byte languages are in a similar situation.

Average of ratings: -

Re: Unicode implementation

by Eloy Lafuente (stronk7) - Wednesday, 19 October 2005, 7:14 AM

Hi,

in this discussion an alternative solution for the htmlentities() problem is being discussed. It seems that a simpler htmlspecialchars() will be enough.

There is one small suggested change in weblib.php to test if the solution can be applied without problems to the rest of Moodle.

As you are advanced non ISO-8859-1 moodlers it would be really amazing to get some feedback about how such change works in your test servers.

Ciao

Edited: I've just discovered that BOTH discussions belong to the same forum so, obviously, you had received the other posts too. Sorry!

Average of ratings: -

Re: Japanese Language Compliance - New Code for 1.4.x

by Patrick Li - Thursday, 27 October 2005, 8:58 AM

I have had a quick view of the patch. Some points:

1. We need a more generic form of this patch as it is too Japanese specified (eg, hard-coded shift-jis encoding).
2. We shouldn't make changes to external libraries, for instance adodb.
3. We could create a new function to wrap htmlentities up. As get_text is a relatively expensive process we could use a static variable in this wrapper to cache the charset.

This problem should affecting other multi-byte languages besides Japanese. So if we could get this fixed then other languages should be happy too (with lib/ directory in their language pack).

Average of ratings: -

GD2 works in Japanese with unicode??

by Jamie Pratt - Wednesday, 26 October 2005, 4:17 PM

It seems that GD2 is working with Japanese on my server.

See attached screen shot of an access graph for one user (I whited out the name which was anyway in English).

I'm using the Japanese UTF-8 lang pack.

There are no Kanji in this graph. Are Kanji the problem? Would someone mind telling me where they are seeing problems with GD2?? Is this only an EUC-JP problem that will be fixed when we move or have moved to unicode??

Jamie

Average of ratings: -

Re: GD2 works in Japanese with unicode??

by Jamie Pratt - Wednesday, 26 October 2005, 4:22 PM

And here is the same graph after installing the true type font for gd2 to use that Mr. Kashiwagi recommends.

Mr Kashiwagi's patch is for an EUC-JP Japanese site. I notice that the quality of the letters is not so good. Perhaps most web servers can provide unicode Japanese fonts for GD2?

Average of ratings: -

Re: GD2 works in Japanese with unicode??

by Jamie Pratt - Wednesday, 26 October 2005, 4:25 PM

Looks like the font my server uses by default is superior in that it can also be rotated. See the text 'access' on the y access of the graph. It is not rotated when using Mr Kashiwagi's font.

Not sure if my default server font will be able to handle Japanese names with kanji in them.

Average of ratings: -

Re: GD2 works in Japanese with unicode??

by Jamie Pratt - Wednesday, 26 October 2005, 4:32 PM

Oh. I do have a problem with Chinese Unicode. I'm using three lang packs Japanese - Chinese and English. For Chinese I get the little blocks instead of kanji. See attached screen shot.

Average of ratings: -

Email in multilanguage site; 'Japanese utf-8 with support for older email clients' lang pack

by Jamie Pratt - Wednesday, 26 October 2005, 8:25 PM

I've applied Mr Kashiwagi's patch to fix the problem with Japanese email garbling. All users on my site who set their language settings to Japanese will now receive messages in ISO-2022-JP which should be properly handled even by older Japanese email clients.

I see a problem here on multilanguage sites everything sent to these Japanese language setting users will be converted to ISO-2022-JP encoding. English emails won't be affected by being converted to iso-2022-jp. But I expect I will have a problem though if users viewing the site in Japanese expect emails sent out to them of postings to forums and that kind of thing in Chinese. But I guess that these older email clients wouldn't be able to handle Chinese mails anyway.

Not sure it Is a problem that now noeone using the Japanese lang setting will not be able to read Chinese in emails.

I'm thinking I could name this patched lang pack 'Japanese utf-8 with support for older email clients' and make a new language pack for people who want their email sent out in unicode - a lang pack which doesn't convert email to iso-2022-jp. It is nice that the functionality for converting emails was moved into the lang packs by Mr Kashiwagi's patch (is this what Moodle will do in 1.6??) so that this sort of thing can be done fairly cleanly.

Average of ratings: -

Re: GD2 works in Japanese with unicode??

by Eloy Lafuente (stronk7) - Monday, 3 April 2006, 7:00 AM

Hi non-latin moodlers,

I was playing a bit in order to support the new lang packs from Moodle's graphic library, and after fixing it (will be available soon in CVS), I've started to search more and more info about Unicode fonts and how to solve it centrally.

Good news are that Moodle 1.6 sites running under UTF-8 won't need any more any laguage "transcode" or language "decode" artifacts in order to display properly non-latin chars, mainly because utf-8 strings are natively supported by GD-Freetype and it seems to work smoothly.

Also, I've found one updated version of the "/lib/default.ttf" font (from http://www.nongnu.org/freefont/) and it's reasonably small to be included in every Moodle package (300Kb over the previous one). It seems to support (after some limited tests) all the Latin languages plus some Cyrillic, Greek...

But, after looking for some "universal" unicode fonts, I must admit that I haven't been able to find one unique solution for all the languages (mainly because of the size of such "universal" fonts).

So, with the current approach, all the languages supported by the central "lib/default.ttf" font with work perfectly, but the rest should install one unicode font inside their "moodledata/lang/xx/fonts" directory. I've tried this with some Japanese and other "universal" fonts and everything seems to work smoothly with this approach.

How does it sound? Perhaps we should start one page in Moodle Docs in order to maintain the best font candidates for each non-centraly-supported lang?

As a reference, I've found this interesting sites about "universal" (a.k.a "Pan-Unicode Fonts") and specific unicode fonts:

http://www.unifont.org/fontguide/
http://www.travelphrases.info/fonts.html

Ciao

Average of ratings: -

Re: GD2 works in Japanese with unicode??

by Eloy Lafuente (stronk7) - Thursday, 6 April 2006, 7:51 AM

Hi,

I've patched graphlib so now it's able to find new language specific fonts. Also I've upgraded the central font and now it support some more languages, so I've deleted such specific fonts from language packs.

Anyway, some languages remain not working and they need to have its own font installed (basically, Latin, Cyrillic and Greek are working, the rest will need their own font- see links above).

Hope this was the final solution...ciao

Average of ratings: -

Re: GD2 works in Japanese with unicode??

by Jamie Pratt - Friday, 28 October 2005, 3:20 PM

Not sure if my default server font will be able to handle Japanese names with kanji in them.

The default font doesn't display kanji. So I've installed Mr. Kashiwagi's recommended font. But unfortunately this results in the 'Accesses' label on the y-axis of the graph not having the characters rotated (see screen shot above). So the characters of the word are reversed and the y-axis is gobbledy gook. Anyone have a fix for this - another font or a nice way to tell Moodle to put the characters in the right order?

Jamie

Average of ratings: -