Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Jerry Lau -
Number of replies: 36

I did the optimize and repair and I still get this message after I login.

Did a test upgrade from 3.4.6+ to latest 3.6.x ... MySQL 5.7.20

My charset is:  utf8mb4

My collation is: utf8mb4_unicode_ci


Debug info: COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8'
SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin = LOWER(?) AND deleted <> 1 AND mnethostid = ?
[array (
  0 => 'lng',
  1 => '4',
)]
Error code: dmlreadexception
Stack trace:

line 486 of /lib/dml/moodle_database.php: dml_read_exception thrown
line 1245 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()

line 1571 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->get_records_sql()
line 1543 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
line 4810 of /lib/moodlelib.php: call to moodle_database->get_record_select()
line 4319 of /lib/moodlelib.php: call to get_complete_user_data()
line 143 of /login/index.php: call to authenticate_user_login()

Did a ../admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci

and the mdl_user table came back with

   - Error: Tried to convert mdl_user, but there was a problem. Please check the details of this  table and try again.

Where do I check the details of this table? and how do I troubleshoot?

I think I need to replace something in my .sql file that I was using to import it into its database. A search or replace of some sort?

 SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
+--------------------------+--------------------+
| Variable_name            | Value              |
+--------------------------+--------------------+
| character_set_client     | utf8               |
| character_set_connection | utf8               |
| character_set_database   | utf8mb4            |
| character_set_filesystem | binary             |
| character_set_results    | utf8               |
| character_set_server     | utf8               |
| character_set_system     | utf8               |
| collation_connection     | utf8_general_ci    |
| collation_database       | utf8mb4_unicode_ci |
| collation_server         | utf8_general_ci    |
+--------------------------+--------------------+



Average of ratings: -
In reply to Jerry Lau

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Jerry Lau -

Good news I fixed it by simply deleting the index associate with the mnethost and then the alter table and collation fix now works


smile

Average of ratings: Useful (1)
In reply to Jerry Lau

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sergio Teixeira -

I have the same problem, after upgrading from 3.6.2+ (Build: 20190201) to 3.6.3.

Someone help with the necessary steps.



Average of ratings: Useful (2)
In reply to Sergio Teixeira

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sylvain Borde -

Updating this morning, I've experienced this problem too with 3.6.3. I resolve this problem changing 'dbcollation' in config.php from utf8mb4_general_ci to utf8_general_ci. Now, I can connect again to moodle 3.6.3.

Note : I've now a warning with admin/environment.php, but it work well.

Average of ratings: Useful (4)
In reply to Sylvain Borde

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sergio Teixeira -

I change in config.php utf8mb4_unicode_ci to utf8_unicode_ci and work. But I think it must have some problem  if the data collation is in utf8mb4_unicode_ci.

In reply to Sylvain Borde

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sean Marx -
Picture of Testers

Thanks Sylvain, this solved my collation issue going from 3.4 tot 3.5

Average of ratings: Useful (1)
In reply to Sylvain Borde

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Michael Shanahan -

Thank you this fixed my issue!

In reply to Jerry Lau

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi,

while both utf8 and utf8mb4 are, stictly speaking, unicode (UTF-8), in MySQL, the former is really limited and does not support a lot of characters (for example, emojis). The later does support the complete unicode charset, but has some extra requirements in the MySQL side.

More info: https://docs.moodle.org/en/MySQL_full_unicode_support

So, it's really important to ensure that, for any MySQL installation, all the database / table and column collations match and, also, that config.php, when configured... has the very same dbcollation option (it may not be needed if MySQL defaults are ok).

There is a utility that will help you to get all the tables/columns converted to the collation that you want: admin/cli/mysql_collation.php.

But first, it's good to read the Docs above in order to understand the problem and then choose the collation that better suits your needs (moodle contents and mysql configuration). And make all them to match with the config.php one (if set).

Hope it helps, ciao smile

Average of ratings: Useful (2)
In reply to Eloy Lafuente (stronk7)

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Jerry Lau -

So which "collation" is the "correct" or preferred one?

I ran the collation.php and to no avail.. the error message is too generic... "something wrong with this table, have a look.." pretty clear as mud LOL

my config.php had db collation settings as well though I commented it now and even when turned on, made no difference.. mine was a duped index


$CFG->dboptions = array(
  'dbcollation' => 'utf8mb4_unicode_ci',
 );




In reply to Jerry Lau

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Ken Task -
Picture of Particularly helpful Moodlers

** MAJOR CONTRIBUTING FACTOR FOR CONFUSION! **

Failure of OP to check environment.  Optional reference in environment.xml since 3.0 of Moodle.
Out of sight ... out of mind.  Environment check should have a link in Notifications page for it's there
that OP get's the 'itch' to upgrade.


But even so, the langugae/description of the opional reference had no link for further
explanation to an admin who was NOT a certfified DB admin.

"The current setup of MySQL or MariaDB is using 'utf8'. This character set does not support four byte characters which include some emoji. Trying to use these characters will result in an error when updating a record, and any information being sent to the database will be lost. Please consider changing your settings to 'utf8mb4'. See the documentation for full details."

IF DB version known to Moodle discovery, couldn't a link be made to 'official' MySQL docs?
https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-conversion.html
Or Moodle's docs ...

https://docs.moodle.org/31/en/MySQL_full_unicode_support

Unsolicited historical info ... we learn from out past:

The dbcollation variable in config.php made it's first appearance as far back
as 3.1.

 fgrep 'dbcollation' ./moodle??/config-dist.php
./moodle31/config-dist.php:    'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8
./moodle32/config-dist.php:    'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8
./moodle33/config-dist.php:    'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8
./moodle34/config-dist.php:    'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8
./moodle35/config-dist.php:    'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8
./moodle36/config-dist.php:    'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8

No mention of this 'signficant' DB change in release notes of 3.1 (may 2016)
https://docs.moodle.org/dev/Moodle_3.1_release_notes#Database_requirements
But it only pertained to MySQL.

MariaDB
https://mariadb.com/kb/en/library/supported-character-sets-and-collations/

MySQL
https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-conversion.html

Changing my.cnf config applied to new databases only ... didn't convert existing DB's.
Using Global commands to accomplish good only for the DB server session instance.   DB server
restarted, then global commands no longer in affect.

Databases that needed to be changed needed changes at all levels:
DB, the tables, and the columns in every table ... which is why
code/admin/cli/mysql_collation.php
Interesting to note that that script first appeared in version 2.3 of Moodle.

mysql_compressed_rows.php first appeared in 26 of Moodle.

The general method which describes upgrading moodle says to keep the config.php file of existing site.
https://docs.moodle.org/36/en/Upgrading

Standard install package
3. Copy your old config.php file back to the new Moodle directory.


Same page:
After upgrading

The config.php file from your installation should work fine but if you take a look at config-dist.php that came with Moodle 3.0 there are more/different options available (e.g. database drivers and settings). It's a good idea to map your old config.php settings to a new one based on the 3.0 config-dist.php.

Note that says 'after upgrading' ... but the catch 22 came during upgrading.

It has been advised more than once in community forums to use a copy of config-dist.php as the new
config.php but edited inserting variables known to OP as required.
No reference to that 'trick' in docs.

Even if folks followed that 'trick', still a 'catch 22' if variable included and database (+tables +columns in those tables) hadn't been converted.

in the 'spirit of learning from our past' ... and 2 cent opinion, Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sergio Teixeira -

Thank you for your explanation.

I read your answer carefully and checked all the steps and discovered the problem.

I had all the tables converted, but did not have the columns of those tables converted into utf8mb4_unicode_ci. 

To solve, I run the CLI script to convert: 

$ php admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci


and this work for me. Thanks

Average of ratings: Useful (4)
In reply to Sergio Teixeira

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Dashamir Hoxha -

Just running this solved the problem for me:

$ php admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci


Average of ratings: Useful (4)
In reply to Dashamir Hoxha

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sylvain Borde -

It worked well for me too, thank you for the tips !

In reply to Dashamir Hoxha

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Dario Murgia -

Thank you so much! now it works! it's incredible ...

I didn't understand the problem well... 

after updating from 3.5.5 to 3.6.3 in the DB there were 3 different collations !!! Now everything is standardized to utf8mb4_unicode_ci.

Thanks again ;)

In reply to Dashamir Hoxha

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Kate Rhodes -

Hey there, 

Sorry this sounds incredibly stupid... but where did you run this:

$ php admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci


I can go in and see my database using phpadmin through my cPanel.... I'm getting confused as to what I suppose to change the tables to?  I get the same error that everyone else in this thread had.... 

but I'm not that familiar with running scripts....


In reply to Kate Rhodes

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Ken Task -
Picture of Particularly helpful Moodlers

@Kate

Not 'stupid' at all!   No such questions are 'stupid' - stupid questions are the questions one doesn't ask!!! smile

Run it from the moodle code directory.   If remotely hosted that could be in /home/customerlogin/public_hmtl/  ... or even a moodle directory of that path.

Or could be in /var/www/html/

So it depends on where and how hosted.

Got cPanel?   Use it to explore the location of moodle code and files contained therein.

'spirit of sharing', Ken

In reply to Ken Task

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by SYPEO Canarias -

Hello!

How can I run this command without access to any command line?


I am using a Shared Hosting provider which does not allow using command line.


Would it be possible to run it through a Cron job?


Edit: I was able to execute the script by using a Cron job set to run every minute. Here is the command:

/usr/local/bin/php /home/USERNAME/path/to/mysql_collation.php collation=utf8mb4_unicode_ci

In reply to SYPEO Canarias

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Ken Task -
Picture of Particularly helpful Moodlers

@SYPEO Canarias ...

Congrats!   But ... hopefully you removed the cron job now that DB is 'fixed'.   Let's hope you don't have to run other cli scripts that use switches ... like -l or -f or other.

Since you've dodged a bullet with shared hosting this time, think maybe you need to move up one rung with hosting ... such that you have a Terminial icon in cPanel?

'SoS', Ken


In reply to Kate Rhodes

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Jerry Lau -

You run it through the command line... like you would with "DOS" command line.

make sure you are using the correct php version you are using...

you can check that if you have access to the command line (aka terminal prompt), by entering "which php" without the quotes. That will show you the full path of the php you are using.

To be safe, I would fully qualify the path of the scripts you want to run so there less chance of an error.

For example, instead of running:

$>  php admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci

I would do this instead and replace the italicized and bolded text with yours.

$>  /where/your/php/is/located/php   /where/your/moodle/folder/is/located/admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci

This would eliminate any errors so you can run it in any folder you are currently in.

Good luck





In reply to Jerry Lau

Re: Worked OK for 3.6.2. upgrade to 3.6.3 (Installatron update -> error reading database) error

by Antti Peltonen -

Hi,

Thanks for the tips...

This worked for me:   (ISP server SSH-connection and there to the right folder)...

 moodle/admin/cli$ php mysql_collation.php --collation=utf8mb4_general_ci

-Antti-


In reply to Dashamir Hoxha

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Katrina Belcher -
Hi Dashamir - when you say to just run it, what does that mean? Run it where and how? Thanks!
In reply to Katrina Belcher

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Jodi Selander -

I upgraded from 3.3.6 to 3.11.3 (in stages, not all at once - 3.4, 3.5, 3.6... etc). 

I am on a shared server, so I am unable to run the (apparently very helpful) moodle scripts that have been described here. I also cannot use Putty or SSH to access my host server, or edit the etc/my.cnf files.

I have cPanel and phpMyAdmin so I make all of my database adjustments through those. 

I had the same exact error as has been described here. It took a lot of hours of searching, but I was able to resolve it with the following methods. 

1) Go in through phpMyAdmin and choose Operations. At the bottom, there will be a "Collation" area with a dropdown list of options to change your tables to. Choose utf8mb4_unicode_ci.
(If links are allowed, I found a detailed description here: https://mariushosting.com/phpmyadmin-how-to-convert-utf8_general_ci-database-to-utf8mb4_unicode_ci/

2) I also ran this as a sql query (link to source is below). Use at your own risk, but I believe it helped me. 

****
SELECT
    /* Disable foreign key checks temporily to be able to make these changes */
    'SET FOREIGN_KEY_CHECKS = 0;' AS alter_statement 
UNION SELECT 
    /* Alter the default character set of each database */
    CONCAT('ALTER DATABASE `', SCHEMA_NAME,'` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS alter_statement 
FROM 
    information_schema.SCHEMATA 
WHERE 
    DEFAULT_CHARACTER_SET_NAME!='utf8mb4' AND 
    SCHEMA_NAME NOT IN('mysql','information_schema','performance_schema','sys')
UNION SELECT
    /* Alter the default character set of each table .
      This also converts all text columns in the table,
      So there is no need to have a statement to alter each
      column individually */
    DISTINCT CONCAT('ALTER TABLE `', TABLE_SCHEMA,'`.`',TABLE_NAME, '` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS alter_statement 
FROM
(
    SELECT
        /* Find all tables with a text column that isn't utf8mb4 */
        TABLE_SCHEMA, TABLE_NAME
    FROM
        information_schema.COLUMNS 
    WHERE
        TABLE_SCHEMA NOT IN('mysql','information_schema','performance_schema','sys') AND 
        CHARACTER_SET_NAME IS NOT NULL AND 
        CHARACTER_SET_NAME!='utf8mb4'
    UNION SELECT
        /* Also find all tables that don't have the correct default character set */
        TABLE_SCHEMA, TABLE_NAME
    FROM 
        information_schema.TABLES AS T
    JOIN 
        information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` AS C ON C.collation_name = T.table_collation
    WHERE 
        CHARACTER_SET_NAME!='utf8mb4' AND 
        TABLE_SCHEMA NOT IN('mysql','information_schema','performance_schema','sys')
) AS TABLE_UPDATES
UNION SELECT
    /* Re-enable forign key checks */
    'SET FOREIGN_KEY_CHECKS = 1;' AS alter_statement 
;
https://dba.stackexchange.com/questions/257962/upgrade-all-mysql-columns-tables-and-databases-from-utf8mb3-to-utf8mb4
***

My database is pretty large, so it could have been that I just wasn't giving it enough time to finish all the updates via the Operations suggestion above. 

In any case, new students can now create an account without error. 

I hope this helps save someone some time. 

In reply to Dashamir Hoxha

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Mark Pape -

Thank you Dashamir Hoxha this was an essential step in the move from 3.1 to 3.5 

In reply to Dashamir Hoxha

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Gloria del Pilar Villarreal Cuellar -

Can you help me out? Just had this error and I am not sure how or where to fix it. My Moodle doesn't allow me to register new users. I would like to know if you can tell me if the same you ran could help me and I would also need to know where it should be run (to be able to tell a friend who helps me with programming). Thank you in advance. Any help is greatly appreciated.  

Attachment descarga.png
In reply to Gloria del Pilar Villarreal Cuellar

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Sean Marx -
Picture of Testers
Hi Pilar

I had this exact same issue today with a Moodle site I took over and upgraded to Moodle 3.8.

When viewing the environment variables, I saw the warning as below.

I then followed the instructions as outlined here: https://docs.moodle.org/38/en/cli#Converting_InnoDB_tables_to_Barracuda
Viewed the DB tables, and then compressed them.


That alone did not fix the issue. I then made sure that my config.php file has the collation correctly defined as utf8mb4_unicode_ci

$CFG->dboptions = array (
  'dbpersist' => 0,
  'dbport' => '',
  'dbsocket' => '',
  'dbcollation' => 'utf8mb4_unicode_ci',
);

With that confirmed, I ran php mysql_collation.php --collation=utf8mb4_unicode_ci to update the mixed collation on all the tables

You'll get an output like this, showing the changes.
mdlap_workshopallocation_scheduled       - CONVERTED
    settings                             - NO CHANGE
    resultmessage                        - NO CHANGE
    resultlog                            - NO CHANGE
mdlap_workshopeval_best_settings         - CONVERTED
mdlap_workshopform_accumulative          - CONVERTED
    description                          - NO CHANGE
mdlap_workshopform_comments              - CONVERTED
    description                          - NO CHANGE
mdlap_workshopform_numerrors             - CONVERTED
    description                          - NO CHANGE
    grade0                               - NO CHANGE
    grade1                               - NO CHANGE
mdlap_workshopform_numerrors_map         - CONVERTED
mdlap_workshopform_rubric                - CONVERTED
    description                          - NO CHANGE
mdlap_workshopform_rubric_config         - CONVERTED
    layout                               - NO CHANGE
mdlap_workshopform_rubric_levels         - CONVERTED
    definition                           - NO CHANGE

Once that was done, I had no problem creating a new user.

Hope that helps.
Average of ratings: Useful (1)
In reply to Sean Marx

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by YOURCODE Soluções Web & Mobile -

Thanks!! Work fine for me!!

In reply to Sergio Teixeira

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Алексей Подолинный -

thank you for the tips ! 

In reply to Ken Task

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Miguel Brostrom -

Hey Ken,

Thanks again for your tips, I see your posts on probably every Moodle community forum post thread I run into.

I did the "trick" of combining my config.php and config-dist.php and that finally got my debug working. I used the mysql_collation.php script, starting with the -l option:

# php mysql_collation.php --l

...

Table collations summary for (linktowebsite):
utf8mb4_unicode_ci: 1296
utf8_general_ci: 207

Noticing there were utf8_general_ci still there I tried the conversion. The script returned an error after the first table it tried to convert:


# php mysql_collation.php --collation=utf8mb4_unicode_ci

...

mdl_survey_questions - NO CHANGE
text - NO CHANGE
shorttext - NO CHANGE
multi - NO CHANGE
intro - NO CHANGE
options - NO CHANGE
mdl_tag - NO CHANGE
name - Default exception handler: DDL sql execution error Debug: Duplicate entry '1-however' for key 'mdl_tag_tagnam_uix'
ALTER TABLE mdl_tag
MODIFY COLUMN name varchar(255)
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT ''
Error code: ddlexecuteerror
* line 492 of /lib/dml/moodle_database.php: ddl_change_structure_exception thrown
* line 1070 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 203 of /admin/cli/mysql_collation.php: call to mysqli_native_moodle_database->change_database_structure()

!!! DDL sql execution error !!!
!! Duplicate entry '1-however' for key 'mdl_tag_tagnam_uix'
ALTER TABLE mdl_tag
MODIFY COLUMN name varchar(255)
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT ''
Error code: ddlexecuteerror !!
!! Stack trace: * line 492 of /lib/dml/moodle_database.php: ddl_change_structure_exception thrown
* line 1070 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database-&gt;query_end()
* line 203 of /admin/cli/mysql_collation.php: call to mysqli_native_moodle_database-&gt;change_database_structure()
!!

I Googled my way through MySQL syntax to find out that the table has a unique row that might be the cause.. Changing the 'dbcollation' in my config.php file to 'utf8_general_ci' fixed my issue, but now getting the environment warning like others have mentioned. This is OK for me to get updates done and is not a mission critical issue, however I would love to find a solution. I was hoping you had any insight on this or where to go from here? I am not MySQL expert and I don't have any DB admins where I work. sad

Thank you again for all the information you've shared in this forum.

-Miguel


In reply to Miguel Brostrom

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Ken Task -
Picture of Particularly helpful Moodlers

Welcome ...

An not a certified DB admin either!  Like most folks, I have to dig and tinker.

utf8mb4_unicode_ci: 1296
utf8_general_ci: 207

The table mdl_tag has been altered for utf8mb4_unicode_ci it appears.
That script 'recurses' (not the correct DB admin term) into all tables and all columns - which is much better/easier than issuing individual SQL queries to alter one table and another to alter each column one at a time.

The change to he db collation in config.php is telling Moodle how it should handle all.

Insights only gained through past experiences and worked for me, but this might not work for all, so
backup what you have ... even if in it's current state.

What I did with one system ... dedicated to Moodle.

edited my.cnf and made the defaults/settings compatibile for utf8mb4.

https://docs.moodle.org/36/en/MySQL

[client]
default-character-set = utf8mb4

[mysqld]
innodb_file_format = Barracuda
innodb_file_per_table = 1
innodb_large_prefix

character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
skip-character-set-client-handshake

[mysql]
default-character-set = utf8mb4

Those settings are in affect for *new* databases.   Won't convert old/existing databases ...
Restarted mysqld and checked for errors/complaints, etc.

From mysql client/prompt created a new database for moodle
create database moodlenew character set utf8mb4 collation utf8mb4_unicode_ci;

Then imported the sql dump into new DB.
Changed config.php (commented out lines in case I needed to step back)
Inserted new lines with the changed parameters values.

Went to code/admin/cli/ and used the scripts in there agsin to check/see/fix.

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Miguel Brostrom -

Hey Ken,

Thanks for the speedy reply.

I'm sorry I forgot to include the my.cnf changes. I actually made all the changes you mentioned above:

[client]
default-character-set = utf8mb4

[mysqld]
innodb_file_format = Barracuda
innodb_file_per_table = 1
innodb_large_prefix

character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
skip-character-set-client-handshake

[mysql]
default-character-set = utf8mb4

These configs have been present for several version updates of Moodle and I haven't had an issue until recently. I did as suggested and created a new MySQL db through the SQL cli.

create database moodlenew character set utf8mb4 collate utf8mb4_unicode_ci;

Checked to be sure my.cnf had the above settings and restarted MySQL. Imported the dump of old Moodle db into the newly created one with Modified my config.php for the new database.

Ran the mysql_collation.php -l script again to list the tables: No change.
Ran the mysql_collation.php script to with the --collation=utf8mb4_unicode_ci: Errors at the same place.

I found someone had the same error as I did on a separate thread. I would post the link here, but the post is being marked as spam. (My account is too young).

The error:

Error reading from database
More information about this error

Debuf info: COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8'
SELECT * FROM mdl_user WHERE LOWER(username) COLLAGE utf8mb4_bin = LOWER(?) AND deleted <> 1 AND methodstid = ?
[array (
0 => 'miguel',
1 => '1',
)]
Error code: dmlreadexception

My solution:

I took table name "mdl_user" and ran the SQL commands that the mysql_collation.php script attempted on just that table:

ALTER TABLE mdl_user
MODIFY COLUMN username varchar(255)
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '';

After this, it allowed me to login without getting the database errors. Not exactly sure what the repercussions are doing it this way, but hopefully this could be useful to those with mixed collated tables.

Thanks again for all your activity and knowledge.


Miguel

Average of ratings: Useful (1)
In reply to Miguel Brostrom

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Ken Task -
Picture of Particularly helpful Moodlers

@Miguel

Really dunno what to tell ya, cept mnethost has been a problem in the past.  It just happened to rear it's ugly head now for some reason. :|

But, good that you used your head ... didn't panic ... and did the alter.

Alls well that ends well, right? smile

'spirit of sharing', Ken


In reply to Ken Task

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Miguel Brostrom -

Hey Ken,

Yeah, I'm not sure why this is happening on this particular Moodle site, as I maintain 2 additional Moodle sites (1 of which is located on the exact same server!) and neither of them have encountered this problem. Glad it's working now though, I suppose I'll write back should something strange happen.

Thanks again Ken,

Miguel

In reply to Jerry Lau

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Franz Medrano -
Hi Jerry Lau, I was also presenting the same problem, I was migrating a moodle on an old server to a new server, I did the whole update process, but at last I had this problem (luckily I had not yet presented my client in production) related to utf8mb4_unicode_ci, likewise, use the moodle tutorial:
https://docs.moodle.org/38/en/MySQL_full_unicode_support

My solution for me was that when entering:

 php admin / cli / mysql_collation.php --collation = utf8mb4_unicode_ci

I was starting to use more temporary file space on the server hard drive and that caused that message to appear in an mdl table .... (- Error: Tried to convert mdl_user, but there was a problem. Please check the details of this table and try again.)

Then you should see how to get more space for the task to be performed, these commands may help you to check the space of your server:

Displays information about total space and available space on a file system
df -h

I had arrived at that solution because I realized that manually checking the table where the error was throwing me, I could not repair it manually, I looked for the error and found this:
https://planet.mysql.com/entry/?id=32629

In that article. I was able to check that the error was in space.
I share the article of the process as it should for the change of utf8mb4_unicode_ci
https://mathiasbynens.be/notes/mysql-utf8mb4

I hope that helps
In reply to Franz Medrano

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by Guillaume Vaslin -

Hi Franz,

Thanks for your solution ! It solved the problem for me !👍🏻

In reply to Guillaume Vaslin

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by MN MN -

one running the command  php admin/cli/mysql_collation.php --collation=utf8mb4_unicode_ci

i get Could not open input file: admin/cli/mysql_collation.php

i run that command with sudo rights

how can i go about this?


In reply to MN MN

Re: Error reading from database - COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8' SELECT * FROM mdl_user WHERE LOWER(username) COLLATE utf8mb4_bin ..

by MN MN -
it finally worked out.
I had to run the command inside the folder that contains moodle files.