Moodle 2.0 upgrade db error

Moodle 2.0 upgrade db error

by Brian Lockwood -
Number of replies: 5

Upgrading a copy of one of our modles I get an error.

Luckily it is just a copy as I wanted to practice upgrades as well as be careful. The server is a more or less vanilla ubuntu server

Linux voyager 2.6.32-24-server #43-Ubuntu SMP Thu Sep 16 16:05:42 UTC 2010 x86_64 GNU/Linux
Ubuntu 10.04.1 LTS

mysql  Ver 14.14 Distrib 5.1.41, for debian-linux-gnu (x86_64) using readline 6.1

PHP 5.3.2-1ubuntu4.7 with Suhosin-Patch (cli) (built: Jan 12 2011 18:36:55)
Copyright (c) 1997-2009 The PHP Group

 

I have configured mysql to utf8 and restored the db from a dump.

The initial pages pass every except phpis 5.3.2 instead of 5.3.3 which is just a yellow check anyway.

Error is

+++++++++++++++++++++++++++++++++++++++++++++++++++++

Debug info: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (utf8_general_ci,IMPLICIT) for operation '='
SELECT DISTINCT c.id AS courseid, ra.enrol, c.timecreated, c.timemodified
FROM mdl_course c
JOIN mdl_context ctx ON (ctx.instanceid = c.id AND ctx.contextlevel = 50)
JOIN mdl_role_assignments ra ON (ra.contextid = ctx.id AND ra.enrol <> '')
LEFT JOIN mdl_enrol e ON (e.courseid = c.id AND e.enrol = ra.enrol)
WHERE c.id <> ? AND e.id IS NULL
[array (
0 => '1',
)]
Stack trace:
  • line 391 of /lib/dml/moodle_database.php: dml_read_exception thrown
  • line 728 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
  • line 4082 of /lib/db/upgrade.php: call to mysqli_native_moodle_database->get_recordset_sql()
  • line 1309 of /lib/upgradelib.php: call to xmldb_main_upgrade()
  • line 252 of /admin/index.php: call to upgrade_core()
Average of ratings: -
In reply to Brian Lockwood

Re: Moodle 2.0 upgrade db error

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Brian,
it's unfortunately a not so rare issue: take a look at http://tracker.moodle.org/browse/MDL-24845#comment-97694 and below.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Moodle 2.0 upgrade db error

by Brian Lockwood -

Well, thanks, an amusing little discussion. I will look into it tomorrow. Seems possible that adding one of the collation bits to my my.cnf will help with this.

Is it probably difficult to test for this on install due to the fact that there are a really huge number of possible compares that could have gone wrong?

In reply to Matteo Scaramuccia

Re: Moodle 2.0 upgrade db error

by Brian Lockwood -

Well the sql database tables are all in utf8, regardless of skipping the character set on restoring to the new server.

The problem is a disaggreement between utf8_unicode and utf8_general somewhere in the data.

I will do some more work.

Hurray for backups

In reply to Brian Lockwood

Re: Moodle 2.0 upgrade db error utf_General utf_unicode Solution

by Brian Lockwood -

Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (utf8_general_ci,IMPLICIT) for operation '='

There is an argument to be had about whether the scripts should be able to cope with this however, they could cope with my particular flavour of the error if the script writers detected the unicode requirement and converted/sifted the tables.

In a spirit of being helpful rather than just blaming mysql and washing my hands  of it. Here goes.

1. utf8 comes in  flavours, "unicode" and "general". For most of us using Western European languages, general is probably fine. utf8_unicode is for languages with joint characters and such like where it might be important to have sorting work correctly for that language. When I started this process I did not know that and so I think I came across this problem because I chose unicode. (That is an assumption and not necessarily true).

2. The --skip-character-set switch on a mysqldump restore does not (or in my case didn't) correctly restore all data to utf8_unicode, It was certainly all set to utf8 but existing data may have been utf8_general, and so I think I ended up with  a mix of collations.--skip-character-set did not convert existing data

This may be because I do not know enough about switches or because mysqldump needs improving. One of many other possibilities is that my database was all general and new stuff from the upgrade was utf_unicode because of my my.cnf being set to unicode rather than general.

3. This bit of php code (from here http://drupal.org/node/298820).

<?php
// your connection
mysql_connect("localhost","username","password");
mysql_select_db("databasename");

// convert code
$res = mysql_query("SHOW TABLES");
while ($row = mysql_fetch_array($res))
{
foreach ($row as $key => $table)
{
mysql_query("ALTER TABLE " . $table . " CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci");
echo $key . " =&gt; " . $table . " CONVERTED<br />";
}
}
?>

 

Will fix your database by converting all the tables to utf8_unicode properly. I applied this after I had done the sql restore using --skip-character-set from a sql dump imto a mysql with utf_unicode set up as the defaults in my.cnf.

To use this code,

a) In the code, change the login details for the database to whatever yours should be.

b) save it with the name convert.php (or <whatever you choose>.php into the root of your moodle.

c) Call the program  by visiting http://...<myMoodleSite>..../convert.php

d) Wait until loads of messages saying CONVERTED appear for each table.

Now continue with the normal upgrade process by visiting

http://...<myMoodleSite>.../

In reply to Brian Lockwood

Re: Moodle 2.0 upgrade db error utf_General utf_unicode Solution

by Chris Craft -

Hi Brian,

I just wanted to thank you for this code. This saved my neck as I was trying to upgrade from 1.9.x to 2.0.2 tonight.

I think we've interacted on Twitter, aren't you in Japan? At an International School?

Either way, I very much appreciate the help.


Chris (@crafty184)