Database is Converted to UTF-8, but the output becomes gibberish

Database is Converted to UTF-8, but the output becomes gibberish

by N Hansen -
Number of replies: 6
I got my conversion to work. Everything is fine in the database. However, I can't get the characters to display correctly when it outputs them. All the unicode characters come out as question marks. Here is the situation:
  1.  I couldn't access the languages from the menus at all, so I manually went in the database and changed the language for the course in which I am having the problem to the proper utf8 language, called en_utf8_oe, which is stored in moodledata/lang.
  2. The language strings from that pack are displayed in the course, so that aspect is working.
  3. However, the question marks still appear, and when I check the page source, I saw that it is setting the language to "en" and the encoding to iso-8859-1, hence the question marks for characters not in that language.
  4. The en in question seems to be coming from the "en" folder in the moodle/lang folder, so I went in and changed the encoding for that language to UTF-8. This made UTF-8 appear in the page source. However, the question marks still appear.
I don't have time to bang my head over this conversion at the moment as I still have other things that broke during the upgrade to fix and it isn't my top priority at the moment, so I'm going to restore my non-UTF8 database and wait until there are some better documentation available on how to do this conversion. But if anyone has any clue why this is  happening and what I might be doing wrong, I would appreciate it.
Average of ratings: -
In reply to N Hansen

More odd language behavior

by N Hansen -
I really don't know what to make of this. I reverted back to the non-Unicode database. Still can't get the languages on the menu. I have two active courses and the site. One course and the site have no language listed in the mdl_course table but are displaying the proper language pack. The other has its language listed and is displaying the proper language pack...but in the list of participants for this course all I get are the country codes, e.g. (AE), (US). The site and the other course are showing the countries properly. When I remove the language name from the problem course, it no longer displays its proper language pack, but now the countries are listed as (AE) United Arab Emirates, (US) United States of America etc., i.e. both the code AND the country! The individual profiles only list city, no country. I don't know what to make of this all. Is the information about language stored somewhere else besides mdl_course?
In reply to N Hansen

Re: More odd language behavior

by Tim Allen -
Hi, I am interested in the problem with the broken text rather than the sorting and display of languages within the profiles, I think they are probably separate issues.

Still can't get the
languages on the menu

Do you mean the language selection menu on the front page, or the language menu in course settings within a course (or both)?  Probably this is related to the language names being in a different encoding to that set to the page in which it is displayed. 

Your en_utf8_oe is a customized language pack, right?  So you should have one pack en_utf8 and the other one  en_utf8_oe...

What is the site-default language?  en_utf8_oe?  And are the other courses intended to be in that language also?  Is the course that is acting differently set to a different language to the site default?  Anyway, I guess that in your site you have text stored in mixed encodings, including some text already in unicode?

The reason I ask these questions is that I am thinking that text in the database that is already stored in unicode (in sites that were already using unicode language packs due to the need to use mixed languages) but which is pre-1.6 Moodle unicode conversion may not be being converted properly, hence the question marks... thoughtful

I also asked about this in another thread.

Tim.


In reply to Tim Allen

Re: More odd language behavior

by N Hansen -
Tim-I think you may be right. I had unicode in my site already and it got messed up on conversion so I reverted...although what was in the database seemed ok. It just couldn't output it. In any case, I am having enough troubles just getting my language packs to show up on the dropdown list that I am not even going to worry about conversion again until I get that fixed first.

I seem to recall when they were doing the UTF-8 conversion test, they told us to set our language to whatever we wanted to enter before we entered it. My problem was that I was entering unicode already without using a unicode language pack. The output of the test was mostly ok for my stuff, but it did insert an odd character in the middle of a string but this problem was dismissed as being just because of my peculiar settings.
In reply to N Hansen

Re: More odd language behavior

by N Hansen -
Just an idea Tim...Not quite sure if this would work, but what if we removed the utf8 conversion scripts for our tables that we know contain unicode already? That way the conversion will happen and Moodle will think it has converted everything and switch over but those tables will remain untouched. 
In reply to N Hansen

What is the point of converting? Everything looks fine already.

by N Hansen -
Tim-One more point. I'm not quite sure what the benefit to us is of converting our sites to Unicode at all. One of the benefits I thought would come from the conversion actually came from simply upgrading to 1.6. That is the ability to search forums using special Unicode characters. I just tried it now and it works without the conversion. It didn't work in 1.5, so it must be some change in 1.6 itself not because of conversion.