гÑабÑж
rather than this:
грабёж
Any thoughts? I've tried to view it in both IE and Firefox. The encoding for the pages is set as Unicode (UTF-8).
The relevant portions of MySQL's my.conf file:
[mysql]
character-sets-dir=utf8
default-character-set=utf8
[mysqladmin]
character-sets-dir=utf8
default-character-set=utf8
[mysqlcheck]
character-sets-dir=utf8
default-character-set=utf8
[mysqldump]
character-sets-dir=utf8
default-character-set=utf8
[mysqlimport]
character-sets-dir=utf8
default-character-set=utf8
[mysqlshow]
character-sets-dir=utf8
default-character-set=utf8
[myisamchk]
character-sets-dir=utf8
[myisampack]
character-sets-dir=utf8
# use [safe_mysqld] with mysql-3
[mysqld_safe]
err-log = /var/log/mysql/mysql.err
# add a section [mysqld-4.1] or [mysqld-5.0] for specific configurations.
[mysqld]
character-set-server = utf8
default-character-set = utf8
I also have "AddDefaultCharset off" in my httpd.conf file on both machines. The only difference between the two machines is that I compile MySQL with the "USE=utf8" flag on the testbed server.
The language file for my site is en_us_utf8 which sets the encoding to utf8. I'm also attaching a screenshot. The top portion is the correct way it should be displayed and the bottom portion is the way it appears on the testbed machine. Any help would be appreciated.
Hello Robert.
Did you do a mysqldump to move data from production to testbed? Or are you using the same DB from testbed (connecting testbed to production DB)?
If you choose the first way, there is an option you must use at dump time:
--default-character-set=
charset_name
Use charset_name
as the default character set. See Section 5.10.1, The Character Set Used for Data and Sorting. If not specified, mysqldump from MySQL 4.1.2 or later uses utf8
, and earlier versions use latin1
.
Have a look at MySQL WEB site.
Good luck.
I tried upgrading to 4.1.16 of MySQL and the results were the same when compiling with the USE="utf8" flag..
and what was the DB encoding of your original DB? If nothing was specified when you created it (previously to Moodle 1.6), it should be, by default, 'latin1' and PHP DB connection is also set to 'latin1' by default (if you didn't modified php.ini nor my.cnf in your original server).
This means that everything in you old server was sent over one 'latin1' channel (between PHP and MySQL) and stored in 'latin1' tables in MySQL, although it seems that you've used non 'latin1' characters in your old server.
All this implies that the data currently stored in you DB is really wrong (from a purist technical point of view) because it isn't proper 'latin1' data (because you have stored some non-latin1 data) nor proper 'utf8' data (because you have used one 'latin1' communication channel to send the data).
Luckily, this is only from a "purist technical point" of view and MySQL allows to store non-latin1 data in latin1 tables and, when data is retrieved from DB back to PHP the inverse encode is performed and everything seems to work pretty fine.
But this imposes serious limitations. For example, you cannot change the communication channel encoding (as you are doing in you edited my.cnf example above), nor can force any encoding in your web server (because its' possible that you have one mix of different encodings in your DB).
This was one of the main reasons to the new UTF-8.-ized Moodle ASAP, because DB internals were mixing contents in really different encodings and everything was based in the user lang and difficult to handle, more every day, with Moodle trying to communicate with other systems.
Obviously, as that nightmare of different encodings co-exist in a lot of servers, the conversion process isn't as simple as reconvert all the fields from 'latin1' to 'utf8', set the communication channel encoding to 'utf8' and to continue working, because your data isn't proper 'latin1' data as I explained 2 paragraphs above.
This implies that, if your site was being used by users using different encodings, every content (every field, every record!) has to be transformed to 'utf8' from its ORIGINAL encoding (user based) and, well, it cannot be performed by hand.
So, who execute/handle this really expensive task? Can you imagine it? Yes, it's Moodle 1.6. Once installed it will detect that your DB isn't running if the proper 'utf8' mode and will offer you the possibility to process all the info in order to convert every content to 'utf8' introspecting in each field, analysing who sent that content to DB and performing the required conversion. (great job, Yu!)
At the end of the looong process (depending of its size), once everything has been converted, Moodle itself will put the communication channel to 'utf8' and since that moment, everything (database, communication channel and http) will be running under 'utf8'.
Sounds simple, eh?
So, for sites having 'mixed' contents (contents stored in DB under different encodings) like you, the upgrade path should be something like:
- Backup everything before upgrade!
- Use MySQL 4.1.12 (and upwards), avoiding to set anything in their configuration files to force any encoding at all (mysql, php, apache). It must work exactly the same than the old DB.
- Standard Upgrade Moodle with the newer version.
- With this, you'll have one non-utf8 Moodle site running and you should be able to see everything exactly as it was before.
- Backup everything again!
- Execute the utf8 migration utility. If something stops the migration process it can be safely continued by launching it again (it remembers where it ended).
- At the end of the 'utf8' migration, one message will appear telling you what languages you have to install (annotate them).
- Login to your new site and install the required languages.
- Voilà, everything should be utf8 and be working like a charm!
Sites being 100% sure about they are using ONLY ONE encoding can specify it in the process above, and it would save a lot of CPU cycles to the looong migration process. But do it, ONLY if you are sure of the encoding of all your DB contents. Else, data loss could arrive!
And this is all, I hope it had explained a bit more about the utf8 thing, do's and dont's...
Ciao
Is a check built in that to make sure the data isn't already UTF-8?
Gavin
if you DB was set to UTF-8 and all your contents were properly stored as UTF-8 strings, you are really lucky, because, if I'm not wrong, all you have to do is to upgrade to 1.6 (standard upgrade).
Once 1.6 was running it'll autodetect DB encoding and everything will work automatically. You'll detect if Moodle 1.6 has detected your UTF-8 encoding if you don't see the option to perform the UTF-8 DB migration in your admin page.
Personally, I haven't tested this situation, but theory says everything should go ok. Obviously I would recommend you to do the test against one copy of your DB first (or having the mandatory backup previous to upgrade).
Ciao
if you are 100% sure that all the contents in your database are using the same encoding (i.e. all the users in your servers were using the same encoding), you have two methods to migrate, gaining time:
1.- Use the standard migration utility inside Moodle, specifying the default encoding. This really helps the migration process to be quickly, reducing execution time to 1/3 of the original one (this is one "brute" approximation). This is the official method.
2.- Dump, your entire DB, convert it fro your UNIQUE encoding to UTF-8, create one new DB (with its default encoding to 'utf8') and load your "translated" dump file to the new DB. Everything should be loaded properly, although, perhpas, some indexes adjustments performed by the 1st alternative were lost. If they give you problems and your DB skills are enough you'll be able to rebuild such indexes without problems. This is the non-official method and I only have performed some basic tests for this (mainly because we have focused our efforts in the 1st method).
I really would recommend to use the first one, because it has been tested against a lot of configurations/sites and it always ends with everything ok. Also, I've executed the migration of moodle.org (ant it's a BIG server) in my 3-years old cheap laptop in 36 hours, more or less. If I apply the 1/3 rule above to it, time would be 12 hours, really not too much for my slooow PC laptop running Win32. Also, moodle.org is really one special case because the users table is the biggest in the world (in the world of registered sites ) and the migration spends practically the half of the time processing it (because the high number of string-based fields it contains).
Hope this helps, ciao
Hi guys,
I desperately need help, I upgraded my moodle from 1.6dev to 1.6.1 and database has been migrated. But all my Arabic courses after the Unicode has been applied became rubbish, I can not read any word, it looks like this ÇáÞÇåÑÉ ãÔÑæÚ ÇáÊÚáã . So is there any solution to retrieve these courses or to cancel that upgrade. Pleas save me.
Thank u in advance,
Many thanks,
Paolo