Problems upgrading from 1.9.8

Problems upgrading from 1.9.8

by William Pearson -
Number of replies: 21
Hey everybody, I'm having trouble when I upgrade from my current 1.9.8 moodle site to any of the moodle preview versions. The latest I have tried is moodle 2.0 preview 4. During the upgrade I get the "Error reading from database" error, and with debugging enabled I get the following:

Debug info: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (utf8_general_ci,IMPLICIT) for operation '='
SELECT DISTINCT c.id AS courseid, ra.enrol, c.timecreated, c.timemodified
FROM mdl_course c
JOIN mdl_context ctx ON (ctx.instanceid = c.id AND ctx.contextlevel = 50)
JOIN mdl_role_assignments ra ON (ra.contextid = ctx.id AND ra.enrol <> '')
LEFT JOIN mdl_enrol e ON (e.courseid = c.id AND e.enrol = ra.enrol)
WHERE c.id <> ? AND e.id IS NULLDebug info: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (utf8_general_ci,IMPLICIT) for operation '='
SELECT DISTINCT c.id AS courseid, ra.enrol, c.timecreated, c.timemodified
FROM mdl_course c
JOIN mdl_context ctx ON (ctx.instanceid = c.id AND ctx.contextlevel = 50)
JOIN mdl_role_assignments ra ON (ra.contextid = ctx.id AND ra.enrol <> '')
LEFT JOIN mdl_enrol e ON (e.courseid = c.id AND e.enrol = ra.enrol)
WHERE c.id <> ? AND e.id IS NULL
[array (
0 => '1',
)]
Stack trace:
• line 378 of /lib/dml/moodle_database.php: dml_read_exception thrown
• line 591 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
• line 4057 of /lib/db/upgrade.php: call to mysqli_native_moodle_database->get_recordset_sql()
• line 1247 of /lib/upgradelib.php: call to xmldb_main_upgrade()
• line 248 of /admin/index.php: call to upgrade_core()

[array (
0 => '1',
)]
Stack trace:
• line 378 of /lib/dml/moodle_database.php: dml_read_exception thrown
• line 591 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
• line 4057 of /lib/db/upgrade.php: call to mysqli_native_moodle_database->get_recordset_sql()
• line 1247 of /lib/upgradelib.php: call to xmldb_main_upgrade()
• line 248 of /admin/index.php: call to upgrade_core()


I've read up on the issue and it seems like there is a problem with my old imported 1.9.8 database. I tried to change the collation of the offending tables listed in the debug statement using the alter table syntax:
ALTER TABLE xxx DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

This didn't fix my problem so I tried changing the collation on all of the tables in my moodle database. I used the following perl script to do so. (I know it's ugly, no comments)

#!/usr/bin/perl
# Script to change the collation of all the moodle tables at once.

#Read in the Database password
print "Enter DB Password";
system("stty -echo");
$DBPASS = <STDIN>;
system("stty echo");
chomp($DBPASS);

#Find all fragmented tables
$TABLES=`mysql -u root -p'$DBPASS' -N -e 'use information_schema; select TABLE_NAME from TABLES where TABLE_SCHEMA="moodle";' | grep -v "^+"`;
@TABLES=split(/\n/, $TABLES);

# Prompt the User to continue
print "\n\nThe following tables will be changed to a character set of utf8 and a collation of utf8_unicode_ci: \n";
print $TABLES;
print "Do you want to continue (y/n)? ";
$CONTINUE = <STDIN>;
chomp($CONTINUE);
#print "\nInput Value: " . $CONTINUE;

while ($CONTINUE ne "y" && $CONTINUE ne "n") {
print "Please enter \"y\" or \"n\"";
chomp($CONTINUE = <STDIN>);
}

$COMMAND = "mysql -u root -p'$DBPASS' -N -e 'use moodle; ";
if ($CONTINUE eq "y") {
foreach $TABLENAME (@TABLES) {
$SQL = $SQL . "ALTER TABLE $TABLENAME DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci; ";
}
$COMMAND = $COMMAND . $SQL . "'";
#print $COMMAND;
$RESULT = `$COMMAND`;
}
else {
print "Quitting...\n";
exit;
}
exit;

This didn't fix my problem either. Has anyone else experienced this problem? Also, is my method of changing the table collation correct, or is there another way I should go about it. Thanks.

-Will






Average of ratings: -
In reply to William Pearson

Re: Problems upgrading from 1.9.8

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Perhaps you'll need to go one step ahead in your ugly script wink and apply the same collation not only to the database and the tables but to the fields.

Once all the varchar/text fields have the same collation, I bet the problem will be out.

Ciao smile

P.S: In fact, both the database and the tables collations aren't anything but the defaults that will be applied when creating new fields. Hence, the important bit is to have the all the fields with the same collation.

P.S.2: Also, note that perhaps, after changing the collation of a lot of fields, it can be useful to regenerate all the indexes on those tables, as far as some of them will perform better with the last collation set. It is not a big difference but ordering clauses will be quicker with rebuilt indexes.
In reply to Eloy Lafuente (stronk7)

Re: Problems upgrading from 1.9.8

by William Pearson -
Thanks for the help Eloy, everything is working now. Based on your suggestion I went back and read the mysql documentation and changed the sql in my script to read:

ALTER TABLE $TABLENAME CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;

After running the new script the install finished without any further problems. I have attached my somewhat cleaned up perl script if anyone else has the same problem. Obviously, the attached script comes with no guarantee that it will fix your problem or not cause data loss/corruption. Use it at your own risk.

Thanks.

-Will

In reply to William Pearson

Re: Problems upgrading from 1.9.8

by Larry Elchuck -
Will

There is one error in your script ... there should be a ; at the end of line 13
$CHARSET="utf8" should read $CHARSET="utf8";


Can this script be adapted to run under Mac OS X using MAMP Pro?

I changed the db name to the one I want to access in the script; saved it and ran
perl courses2_change_collation.pl

I get
"Can't connect to local MySQL server through socket '/var/mysql/mysql.sock' (2)"

I think it is because in MAMP Pro, mysql.sock is located at /Applications/MAMP/tmp/mysql/mysql.sock

Any suggestions on how to get around this issue?

thanks
In reply to Larry Elchuck

Re: Problems upgrading from 1.9.8

by Larry Elchuck -
Found 2 other solutions to this collation problem ...

1. Backup the database; Dump the database to a .sql or text file; Perform a global search and replace all instances of utf8_general with utf8_unicode; Run a query on the database replacing the existing code

2. In the following script, replace "database_name" with the name of your database and edit the statement template in the parameters of the CONCAT function to have the target character set (and/or collation) of your choice. After that you can just take the statements and batch execute them in the MySQL client of your choice (eg. phpmyadmin, Navicat, the official MySQL client, etc.). Of course, you're strongly advised to make a backup of your database before you start messing around with character set (and/or collation) conversions.

SELECT CONCAT('ALTER TABLE `', t.`TABLE_SCHEMA`, '`.`', t.`TABLE_NAME`, '` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') as stmt
FROM `information_schema`.`TABLES` t
WHERE 1
AND t.`TABLE_SCHEMA` = 'database_name'
ORDER BY 1


I used the first option and later located the latter.
Average of ratings: Useful (4)
In reply to Larry Elchuck

Re: Problems upgrading from 1.9.8

by William Pearson -
Sorry to let your posts go cold. These are some great suggestions for changing the database collation. I especially like your second method as it doesn't require any outside scripts or database dumps. Also, Good find on the typo. Hopefully any future users will see your note, or better yet use one of the two methods you suggested.

I don't have any experience with the MAMP Pro stack, here's a link to a potential fix for your problem:
http://discussions.apple.com/thread.jspa?messageID=5059320



In reply to William Pearson

Re: Problems upgrading from 1.9.8

by Diego Maté Cuñado -

Hi:

The error is in a table of moodle db, the table is 'mdl_role_assignments'.

You have to update the collation of 'roleid' field to utf8_general_ci.

This is the solution, but previously you may change the collation of every tables in moodle db.

Hope this could be fine for your migration.

In reply to Diego Maté Cuñado

Re: Problems upgrading from 1.9.8

by Chad Outten -

agreed, you also need to convert the collation of table fields. as per the debug info and more specifically, run the following queries...

ALTER TABLE `mdl_course` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

ALTER TABLE `mdl_role_context` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

ALTER TABLE `mdl_role_assignments` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

ALTER TABLE `mdl_enrol` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

---

HTH

In reply to Larry Elchuck

Re: Problems upgrading from 1.9.8

by Phaessuh Kromah -

Thanks Larry... Solution 2 worked for me... This post saved lots of time

In reply to Larry Elchuck

Re: Problems upgrading from 1.9.8

by John White -

Larry,

Thanks for your post, that really helped; option 2: convert the Collation seems to work a treat!

But I also note a couple of pointers to foresee the problem. It seems to me that if in phpmyadmin you see tables with one collation e.g. utf8_general_ci like this:

...but when you then click the Operations tab and it suggests creating a new table with a different collation e.g utf8_unicode_ci like this:

You can be pretty certain that the Moodle 2 conversion is going to hiccup!

Regards, John

In reply to John White

Re: Problems upgrading from 1.9.8

by Rob Johnson -

This can be changed in the same place.  If you click "Operations," you can change the collation field to utf8_general_ci.  After that, new tables should use that collation.

Average of ratings: Useful (1)
In reply to Rob Johnson

Re: Problems upgrading from 1.9.8

by John White -

Thanks Rob. Of course that would mean, say, the existing 'general' collation being maintained, and new tables matching that, rather than the existing tables being modified to match the currently selected new table collation!!

I suppose the choice just depends on which direction you want to go it.

Regards, John

In reply to Larry Elchuck

Re: Problems upgrading from 1.9.8

by Bob McKay -

Larry, thank you for the solutions. Unfortunately I can't get it to work for me - any suggestions would be greatly appreciated. I tried the second solution with utf8_unicode_ci (because I may need to handle Korean-script names at some point):

mysql> SELECT CONCAT('ALTER TABLE `', t.`TABLE_SCHEMA`, '`.`', t.`TABLE_NAME`,
    -> '` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') as stmt
    -> FROM `information_schema`.`TABLES` t
    -> WHERE 1
    -> AND t.`TABLE_SCHEMA` = 'moodle'
    -> ORDER BY 1;

It appeared to work:

| ALTER TABLE `moodle`.`adodb_logsql` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;                     |
| ALTER TABLE `moodle`.`mdl_assignment_submissions` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;     

etc. until

245 rows in set (0.00 sec)

mysql> exit

I then restarted mysqld and httpd.

In the moodle upgrade I am still getting "Error reading from database":

Debug info: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (utf8_general_ci,IMPLICIT) for operation '='
SELECT DISTINCT c.id AS courseid, ra.enrol, c.timecreated, c.timemodified
FROM mdl_course c
JOIN mdl_context ctx ON (ctx.instanceid = c.id AND ctx.contextlevel = 50)
JOIN mdl_role_assignments ra ON (ra.contextid = ctx.id AND ra.enrol <> '')
LEFT JOIN mdl_enrol e ON (e.courseid = c.id AND e.enrol = ra.enrol)
WHERE c.id <> ? AND e.id IS NULL
[array (
0 => '1',
)]
Stack trace:

    line 391 of /lib/dml/moodle_database.php: dml_read_exception thrown
    line 753 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
    line 4085 of /lib/db/upgrade.php: call to mysqli_native_moodle_database->get_recordset_sql()
    line 1378 of /lib/upgradelib.php: call to xmldb_main_upgrade()
    line 273 of /admin/index.php: call to upgrade_core()

 





In reply to Bob McKay

Re: Problems upgrading from 1.9.8

by Bob McKay -

Taking a clue from your first solution, Larry, I tried dumping the database. As expected, every table shows up as utf8_unicode, and the words utf8_general do not show up anywhere in the dump. So it looks like the original problem may have been compounded in more recent versions of moodle (that is, it seems like someone somewhere must have utf8_general hard-coded).

In reply to Bob McKay

Re: Problems upgrading from 1.9.8

by Alice Knox -

Hi Bob,

I'm upgrading from 1.9.9+ to 2.1 and having the same problem you are - tried Larry's solution # 2, it appeared to work, but the error didn't go away. I too did a SQL dump of my database, and searching for 'utf8_general' doesn't turn up anything.

I'll keep messing with it and let you know if I figure anything out... I might try upgrading to 2.0 and then 2.1.

In reply to Bob McKay

Re: Problems upgrading from 1.9.8

by Alice Knox -

Well, I got past the collations error, so I'll share how I did that. However, I'm stuck on a new error, so be warned that this may not fix all your problems (and may, in fact, cause new problems).

I found this bug report: http://tracker.moodle.org/browse/MDL-24845 and followed the loose instructions provided by Matteo. See that report for his explanation and to download the attachments: fixmycharsetcollation.php, settings_unsupported.patch and adodb-mysql-drivers.patch. Here are the specifics of what I did:

  1. Reverted to Moodle 1.9 on my test site and made sure I could log in (in my case, I saved my web.config and config.php files).
  2. Added the 'fixmycharsetcollation.php' file to the admin directory. I think this page does the same thing as Larry's method 2, but also applies the collation/charset correction to the fields in each table.
  3. Opened the 'settings_unsupported.patch' file in a text editor and manually made the change indicated. This change allows the 'fixmy...' page to be added to the admin section of your moodle site.
  4. Logged into the test site, went to [my URL]/admin/fixmycharsetcollation.php, and ran it.
  5. Backed up the database, once it was collated correctly. That way, when my upgrade failed for a different reason, I could revert to this copy of the DB and not have to jump through the collation-fixing hoops again.
  6. Replaced the Moodle 1.9 files with the Moodle 2.1 files, except for web.config and config.php.
  7. Opened the 'adodb-mysql-drivers.patch' file in a text editor and manually made the changes indicated.
  8. Tried the upgrade again

Unfortunately, though this got me past this error, I am now stuck on another one, under "mod_feedback": "Error writing to database. Debug info: Field 'search' doesn't have a default value"

I believe the the adodb-mysql-drivers patch was supposed to fix this error, so I'm not sure where to go from here.

I'm about ready to throw in the towel - I've spent way too much time on this. I'm going to try upgrading to 1.9.13 instead, and if that goes smoothly I'll back it up and try upgrading from there to 2.0, and then try 2.1. I'd love to be up to date, but the most important thing is the security fixes, which I believe are implemented in 1.9.13.

 

In reply to Bob McKay

Re: Problems upgrading from 1.9.8

by Alice Knox -

OK, here's what finally worked: I bit the bullet and upgraded my production site, rather than my test site. Since it's recommended, I upgraded to 2.0.4 and then to 2.1.1+. No errors, no problems - easy as pie.

As for my adventures trying to upgrade the test site: it turned out to be easy enough to get past my last error about field 'search' not having a default value.  Turned out, it didn't even have an empty string in there, and adding one solved the problem. Unfortunately, though the installation completed "successfully", the site did not work at all afterward. Just a blank page.

While adding a default value to 'search', I noticed other fields in the same table that didn't have default values either, but those didn't generate errors. I eventually determined that those fields originally had default values, but they disappeared when I ran fixmycharsetcollation.php. I can only assume that the upgraded site didn't work because of these missing default values?

This was my first Moodle upgrade, and it was pretty hairy. Trying out the upgrade in a test environment is recommended, but also (for some reason) impossible to get working. Apparently it has something to do with losing the charset/collations when you make a copy of the database, but I still have no idea why that happens. Anyway, my best advice to anyone who's having this error: just back up your production site and run the upgrade there. Just go for it. I wasted hours trying to do it the "safe" way - I learned some things, but overall it's not worth it.

Happy Moodling and good luck to you!

In reply to Alice Knox

Re: Problems upgrading from 1.9.8

by Rob Johnson -

The collation differences are likely due to new tables being created during the upgrade with the database set to a different collation than the tables that were imported.  I ran into this when upgrading a test site.  The tables I imported were UTF8-general, but the database was set to UTF8-unicode.  I would get a mixed collation error when new tables were created during the upgrade (unicode).  It was not until I set the imported database to "general" that I was able to upgrade without an error.  I have a hunch your production database worked because the database collation matched the existing tables.

In reply to Rob Johnson

Re: Problems upgrading from 1.9.8

by Rufus Chaney -

Is there a solution here or a patch to fix the problem?  I've manually corrected every table from "utf8_unicode_ci" to "utf8_general_ci", but I still get the error message.

In reply to Rufus Chaney

Re: Problems upgrading from 1.9.8

by Trent Miller -

I had this same problem. After many hours of googling and reading I came across this fix.

1) Open the attached file.
2) Change "DBUSERNAME", "DBPASSWORD" and "DBNAME" to suit your environment. Notice it also says "localhost" I think if you were running this remotely you could put an IP or URL in it.
3) Save the file under C:\. Make sure the extension is still "php"
4)Open a command prompt and navigate to your php install. Mine was "C:\PHP".
5)Tell "PHP.EXE" to use your script. I typed in...
   "C:\PHP\php.exe -f "c:\collatechange.php"

The script will run, you'll see some output jibberish in your command window. It took roughly one minute to complete. Restart IIS and try your upgrade again.

In reply to Larry Elchuck

Re: Problems upgrading from 1.9.8

by Frederic Nevers -

Thanks for the tip. I thought I'd share a simple solution that worked for me.

I ran the SQL found in your second tip (it took me a while to work out that it only checks which tables need a change in collation, without actually fixing them...), exported the results to Excel, copy/pasted the stuff from Excel into a SQL window in PHPmyAdmin, ran that. It worked a treat. This only took 1 minute total and only requires basic SQL skills.
Cheers,

Fred