Moodle in English: Translators, heads up!

Translators, heads up!

by koen roggemans - Thursday, 2 February 2006, 3:16 PM

Moodle 1.6 will change a lot for the translation business

1) There is the conversion from Moodle to UTF8 character encoding (done)
2) The language packs will not be included in the download anymore, but will be downloaded and installed with a new interface after installation. (to do extremely soon

, say friday 3 february afternoon GMT )
3) The lang/xx/docs folder will move to a http://docs.moodle.org/xx/ (to do)

These changes will ask from you a slightly different approach for contributing to the translations.

The first step from the moodle team will be

1) Removing all lang packs from the main CVS tree (moodle/lang) except en_utf8. This is ONLY in HEAD and 1.6 ... the lang packs in 1.5 will stay in that branch (MOODLE_15_STABLE) for continuity.
2) Checking in changes to admin/lang.php so that it only edits languages within dataroot/lang

After that, the consequences for Moodle Administrators are :

After you upgrade to Moodle 1.6 you will find that you only have the English language pack (en_utf8). You should use the new feature on the language page to download new UTF packs automatically from download.moodle.org. The upgrade process will tell you which ones you need.

The consequences for Translators are:

If you are using HEAD to do translation (and you should be!), then instead of modifying the lang pack within your main code tree as you used to, you will now be modifying the one within your dataroot/lang directory.

1) Check out the very latest Moodle 1.6 code
2) Upgrade your site. Of course this is a NON-PRODUCTION server, right?
3) Go to your dataroot directory and check out the new lang directory from CVS there like this on Unix/Linux

cd moodledata (for example)
cvs -z3 -d:ext:me@cvs.sourceforge.net:/cvsroot/moodle co lang

or like this with Tortoise on Windows,

right click on your dataroot folder, choose "CVS Checkout",
CVSROOT field: :ext:sfusername@cvs.sourceforge.net:/cvsroot/moodle
Module: lang

4) continue editing languages from within the Moodle GUI as you did before.

To check them into CVS, just go into the dataroot/lang directory and commit your changes.

If you really want to continue updating the language packs for 1.5.x then you should zip the folder of your langpack and mail it to translation@moodle.org. CVS write rights on moodle/lang/xx will be moved to lang/xx

Good luck to you all

Average of ratings: -

Re: Translators, heads up!

by N Hansen - Thursday, 2 February 2006, 9:27 PM

Koen-Will the language packs be stored in a different place in the actual Moodle installations? This is important for those of us who have our own custom language packs to know.

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Friday, 3 February 2006, 12:49 AM

I will test it, but I think it is the safest way to store the customised language pack in mooldedata/lang

Please be aware this all is very new, only for the Moodle 1.6 DEVELOPELENT version (wich means it isn't finished yet

)

So for production servers with customised lang packs, this isn't an issue yet.

Thank you for mentioning - I'll add something about this in the manuals as soon as I can test it thoroughly

Average of ratings: -

Re: Translators, heads up!

by N Hansen - Friday, 3 February 2006, 9:37 AM

That makes sense to me too. People who backup their sites may forget to backup the language packs if they are in the moodle directory. Putting them in a must-backup directory like the moodledata directory would be safer.

Average of ratings: -

Re: Translators, heads up!

by Ralf Hilgenstock - Sunday, 5 February 2006, 3:49 AM

The editing process of lang files searches the en-parent files in lang/en and not in the new lang/en_utf8 folder

The other lang files are searched in moodle/lang/language and not in moodledata/lang/language.

The problem is reported in the bug tracker.

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Sunday, 5 February 2006, 6:07 AM

Yes, 1.6 seems to run a little buggy today
It might be a little early to do translation work, but testing and reporting bugs always helps.

Average of ratings: -

Re: Translators, heads up!

by Ralf Hilgenstock - Sunday, 5 February 2006, 8:39 PM

Martin wrote that he is working on it.
You are right Koen, it's not the time to translate. I was only testing functions.

I'm not shure what happens if I have a customized lang package with a change in the parent string of moodle.php.
en_utf8 in folder moodle/lang/en_utf8
customized langfolder moodledata/lang/de_myown_utf8
parent langfolder for the customized folder moodledata/lang/de_utf8.

What are the settings for the moodle.php parent string?

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Sunday, 5 February 2006, 9:02 PM

I'd suggest translators to hold your horses at least until monday evening GMT and give moodle.com the chance to fix a few things up.
The new language editor from Yu needs to go in CVS, there was just a fix from Eloy for language conversion on php5 systems...
I'll report when I don't succeed in breaking things up anymore (and believe me I'm good at breaking things up blauw oog

)

Average of ratings: -

Re: Translators, heads up!

by Tim Allen - Monday, 6 February 2006, 12:00 AM

Thanks for your efforts in communicating these big changes Koen! You're doing a great job, don't worry about breaking things up, that can happen to anyone and always does.

I understand that now is not a good time to do any commits to CVS, but I have a question about what to do once everything is sorted out and translation can begin again.

I have just checked and found that the language packs were moved into moodledata/lang four weeks ago by Eloy. In the meantime, my translators have continued translating the files in moodle/lang/ko_utf8. The problem caused by this is that if I checkout the lang module in moodledata/lang via CVS and then continue translating (once the new language editor is in CVS), we will lose all the changes made in the last 4 weeks.

So how can I get those translations from moodle/lang/ko_utf8 into the moodledata/lang/ko_utf8 directory using CVS so that we do not lose all that good work? I don't think just copying the files across would work... thoughtful

TIA for anyone's help,
Tim.

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Monday, 6 February 2006, 1:12 AM

No worry, besides, Eloy and Yu are the big coders here, I do the messaging and the breaking things up clown

Eloy has written a nice script, converting all non-utf8 language packs in moodle/lang to moodledata/lang/XX_utf8.
Once the write access is closed on the moodle/lang files, he runs the script, so all languages in /lang will be up to date. It might be a good plan to check and notify him immediatly when there is something wrong.

If you have some translated material wich is not yet in CVS, upload it now, but hold then on translating until after the final conversion. Then continue with an up-to-date Moodle 1.6 and an upgraded database (the button in the admin screen).
Report any problems while upgrading the database and the result of it (no production servers please

)

Average of ratings: -

Re: Translators, heads up!

by Tim Allen - Monday, 6 February 2006, 2:56 AM

Ahh, I see, thanks Koen.

Average of ratings: -

Re: Translators, heads up!

by Eloy Lafuente (stronk7) - Monday, 6 February 2006, 2:23 AM

Hi Tim,

if everything goes fine, tomorrow (12 hours from now aprox.), just before "closing" the old moodle/lang packs, the conversion script would be executed again (that will be his last execution).

This will allow translators to have the new lang packages will all their changes applied so, data loss should be really minimal.

Anyway, about your current 1.5 packages (ko, and ko_utf8) I've seen that, the conversion script is using "ko" as origin to create the new 1.6 "ko_utf8" pack, so any string present in your 1.5 "ko_utf8" and missing in your 1.5 "ko" pack won't be processed, because only the "ko_utf8" lang pack is being generated. Looking my notes about the conversion evolution it seems that we chose the "ko" as origin because it was more complete that the "ko_utf8" flavour.

Is this correct? Can we assume that the 1.5 "ko" lang pack is a good origin to build the 1.6 lang pack? I hope so!

Ciao

Average of ratings: -

Re: Translators, heads up!

by Tim Allen - Monday, 6 February 2006, 2:47 AM

Hi Eloy,

Thanks for your attention to this.

I understand what you said about reconverting the language packs just before closing the old packs so that data loss is minimal. That's great, I now know I don't need to worry about that.

However, I am very glad you checked with me about which language pack is best to transfer to the new location, because actually it is the opposite way around. Because of the extra advantages of using unicode we have been translating into ko_utf8 by default. Then when I commit to CVS, I have been committing ko_utf8 first, and then I have been using a script to convert ko_utf8 unicode to ko EUC-KR and then committing that also.

If you use ko and not ko_utf8 as the origin, I will have to do that extra conversion again tonight (I will do my last commit tonight to get the latest translations in before the change over). If it was possible for you to use the ko_utf8 pack, I wouldn't have to do that second conversion (sometimes it takes a while to sort out conflicts).

That pack is of course already in UTF8 but I expect you could still run your conversion script on it to be sure.

However, if there is some technical reason that it would be easier for you to use ko as the origin, that is OK, I will convert ko_utf8 to ko and commit that too.

I hope you understand what I have said. If you have the time to tell me which pack you will use then I will know what to do when I commit.

Thanks a lot, I know you have spent so much time on this enormous improvement for Moodle, I really appreciate it! approve

Tim.

Average of ratings: -

Re: Translators, heads up!

by Eloy Lafuente (stronk7) - Monday, 6 February 2006, 3:58 AM

Hi Tim,

I've switched the conversion script to use the 1.5 ko_utf8 lang pack as origin instead of the ko one.

Once executed it seems that everything has run pretty well. Anyway I was reviewing results when I've found that the "firstdayofweek" string has some incorrect data within it. If I'm not wrong it should contain 0 for sunday, 1 for monday and so on (without other characters on it).

So, now, your ko_utf8 lang (1.5) will be used to build the ko_utf8 (1.6) lang pack and everything will be there tomorrow...

Ciao

Average of ratings: -

Re: Translators, heads up!

by Tim Allen - Monday, 6 February 2006, 4:32 AM

Thanks Eloy, I actually just committed the ko pack just in case, but I think it is better to use ko_utf8 so that is good.

Thanks for picking up that mistake too, I have fixed it now.

Tim.

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Wednesday, 8 February 2006, 7:58 AM

I sort of promised to report now and then on how the langconversion is going.

The documentation provided in my first post in this tread still goes cool

At the moment, a conversion looks good at the surface. One strange thing: since the langpacks aren't present anymore when you deleted 1.5 and copied an 1.6 in it, the upgrade performs in English.

Immediately after the upgrade, you do a database conversion by clicking on the link in the admin screen. It takes some time. After the conversion, you go to administration / language and download the required UTF8 languages for your site.

Things wich might catch your attention (and if so: report please!)

collation not converted to UTF8 (all tables must be converted)
Characters wich don't convert properly
missing directory 1 in moodledata causes error
other errors? please to bugtracker

I planned to do some further testing, but the byts aren't very coöperative

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Friday, 10 February 2006, 4:54 AM

So, I think the translation proces can continue. Bugs preventing this so far are all closed and I gave it some test rides.
Nicolas (fr) and Mits (ja) are already started, brave pioneers cool

.

Keep a few more things in mind:

no production servers yet!
no NCR-notation anymore in the 'thislanguage' string - we use Unicode everywhere.
all those funny strings, containing something NOT to translate, have now a more logical location in langconfig.php. This file should normally not be edited at all. For new languages, we will discuss the settings and will create it for you.
Since all characters should be Unicode, the best approach is to use the build-in language editor for strings and helpfiles.

Custom language files and language files from contrib modules

They should be converted to unicode too.

On unix flavours this can be done with the inconv command:

iconv -f original_charset -t utf-8 originalfile > newfile

On windows machines, the best thing I could come up with is to install Cygwin.
It supports running the iconv command on a windows machine in a unix like shell.
I include in attachement an examplescript to convert a language pack (in the folder /no in my example).
NOT TO BE USED FOR FILES ALREADY IN /LANG OR MOODLE/LANG (That 's already done for you)

Good luck

More reading on http://docs.moodle.org/en/Translation

utf.txt

Average of ratings: -

Unicode effects

by Mary Kaplan - Sunday, 12 February 2006, 12:58 AM

Hi Koen,

At the Savannah Moodle Moot, Martin told us about Unicode, and the changes it would bring for languages. It sounds really good, once it is in place, but I am a bit concerned for a Forum discussion course I have where there are posts in English and French, some posts even contain both languages in the same post. How will the 1.6 "know" which language to change things to? Or are the only current differences between French and English the French letters with accents?

We discussed waiting until the end of this school year to update to 1.6 if this looks like a potential problem, and we will obviously back-up and test and such. Do you have an opinion on what this will do?

Thanks for all you are doing! This really sounds like a wonderful improvement to world community of language learners and their mutual communication!

--Mary

Average of ratings: -

Re: Unicode effects

by Martin Dougiamas - Sunday, 12 February 2006, 5:27 AM

French and English (and German, among other European languages) use the same latin character set (ISO-8859-1), so there will be no problems.

Average of ratings: Useful (2)

Re: Unicode effects

by koen roggemans - Sunday, 12 February 2006, 7:23 AM

The whole unicode thing is actually not about the language, but about the characterset a language uses. So it is e.g. no problem to use English, French, Dutch, German and even Norwegian on the same Moodle page and upgrade to 1.6, since they all use the same characterset. Multiple languages using the same characterset shouldn't have problems after upgrade at all, because from a Unicode point of view, those languages are all the same.
Slightly more difficult is a site using multiple charactersets, like moodle.org, because the characterset used is not the same on every page (e.g. the moodle in other languages category). But, since it is not possible to display more then one characterset on the same page, it is enough to know the characterset of the teacher who creates the course. All elements (resources, posts, ...) belonging to that course must have the same characterset, otherwise they are displayed as garbage in Moodle 1.5 too.

After the conversion, every Moodle will use on every page UTF8, only one characterset for all languages. So typing a three collumn table with words translating Chinese into Hebrew into Arab should be possible, because they will share the same characterset.

I hope this small note clarifies a little bit of the utf8 conversion mistery.

Average of ratings: Useful (2)

Re: Unicode effects

by Tormod Aagaard - Sunday, 12 February 2006, 6:28 PM

I've got bilingual courses in French and Norvegian and it came out all fine after UTF-8 conversion (done on a copy of my site). So it seems like theory and practise agree on this issue

Average of ratings: -

Re: Unicode effects

by Mary Kaplan - Sunday, 12 February 2006, 7:13 PM

It does indeed help! As a matter of fact, it explains exactly what I was curious about. Thanks Koen and Martin! And Tormod, thanks as well for the example of a working model of exactly what I was concerned about. We're looking forward to all the exciting changes for 1.6 and understand that this advance will be important for us all.

Appreciated the quick turn-around to getting answers to my question!

Moodling on.....

--Mary

Average of ratings: -

Re: Translators, heads up!

by Ralf Hilgenstock - Sunday, 12 February 2006, 8:00 PM

Hi Koen,

I haven't tried it, but perhaps you can give an answer. What happens with a course backuped on an 1.5 or 1.4 sytem and restored in a new moodle 1.6 after conversion to utf8?

Ralf

Average of ratings: -

Re: Translators, heads up!

by koen roggemans - Sunday, 12 February 2006, 9:36 PM

That is not finnished yet (as far as I know), but is still on the todo list.

Average of ratings: -

Re: HTTP headers

by David Mudrák - Tuesday, 7 February 2006, 8:51 AM

I have got some problems in the current HEAD with HTTP charset headers. On some pages (e.g. /admin/) the page is sent with default HTTP charset headers instead of new utf8. HTML <meta> charset header tag is OK. But in Firefox, HTTP headers take precedence

Average of ratings: -

Re: HTTP headers

by koen roggemans - Wednesday, 8 February 2006, 7:30 AM

Please put it in the bugtracker David - here it might get lost

Average of ratings: -

Re: HTTP headers

by David Mudrák - Monday, 13 February 2006, 5:30 AM

I did more than it - I found the problematic piece of the code and the solution

See bug 4747.

Average of ratings: -

Languages

Translators, heads up!

Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Re: Translators, heads up!

Custom language files and language files from contrib modules

Unicode effects

Re: Unicode effects

Re: Unicode effects

Re: Unicode effects

Re: Unicode effects

Re: Translators, heads up!

Re: Translators, heads up!

Re: HTTP headers

Re: HTTP headers

Re: HTTP headers

Empowering educators to improve our world

Moodle

Support

Get Involved

Contributions

Downloads

Tracker

Development

Empowering educators to improve our world