Converting to UTF8 - what Moodle Docs procedure to use?

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Helen Foster -
Number of replies: 10
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Thanks Ray for highlighting documentation needing improvement, and thanks Howard and David for your advice.

As David mentions, the page "Converting your MySQL database to UTF8" dates back to Moodle 1.6, so I have deleted the page in the 3.3 and 3.4 docs wikis. (It's still available in the 3.2 docs wiki though - https://docs.moodle.org/32/en/Converting_your_MySQL_database_to_UTF8 )

I also noticed that MySQL full unicode support linked to it, but found it unclear which bit of text it referred to. Thus I removed the link and am hopeful that if further information is required, someone knowledgeable will add it. smile

Average of ratings: Useful (3)
In reply to Helen Foster

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Ray Hinton -

Hi Helen et al.,

I really appreciate everyone's careful attention to this nitty-gritty issue! Those changes sound about right to me.

The only final thing I would say, which I think you alluded to ("hopeful that if further information is required, someone knowledgeable will add it. smile"), is that that old page did have examples of some useful find/replace commands that could be used with the "dumpfile" method.

Specifically, I am talking about the commands in the Converting a database containing tables > Linux & Mac section. My understanding is that, to modernize those commands, one would change instances of `latin1` to `uft8`, or something like that. Since I know the old versions of that page exist, I can find the example commands and use them, but other people would not know to look for the old 3.2 version. All that said, the PHP script exists to do the same thing, and perhaps it is appropriate to expect that power-users with huge sites will seek out the appropriate information elsewhere. We can't expect the Moodle Docs to explain everything about everything!

Anyway, thanks again for helping resolve this.

- Ray

Average of ratings: -
In reply to Ray Hinton

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm a bit lost... you shouldn't be seeing 'latin1' encoding at all. It's nearly 2018!!


Average of ratings: -
In reply to Howard Miller

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Ray Hinton -

Hi Howard,

Yes, that is correct! (and wow, 2018 snuck up on me!) My database never used `latin1` encoding. But if you read this example find/replace command in the Linux & Mac section of "Converting your MySQL database to UTF8" (the old document that is now deleted for 3.3+), you will see that it says:

:%s/DEFAULT CHARACTER SET latin1/DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci/
:%s/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8mb4/

When we used these commands, we just replaced `latin1` with `utf8`. So, we found those commands useful as a suggestion of what to do, but we needed to tweak them. If a new person comes along and only reads the 3.3+ versions of these wiki pages, they will never see these useful commands (since that page is deleted). Again though, they are directed to use the script, which does the same thing.

Therefore, in the current state, I think the 3.3+ version of these wiki pages is technically 100% correct (in my humble opinion) and does not lead anyone astray, but it could have one more piece of useful information if it somehow included an up-to-date version of the find/replace commands above. Nit-picky, I know, but I just want to make sure everyone understands what I mean.

-Ray

Average of ratings: -
In reply to Ray Hinton

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I may be missing something... why would you still need these commands?

Moodle has a built-in tool to do the conversion.

Average of ratings: -
In reply to Howard Miller

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

Because those in-built commands were reported to take a while to run so it may be faster to actually do it this way (that's how I understood the reasoning).

Average of ratings: Useful (1)
In reply to David Mudrák

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Ye-es. In fact I think it was me that reported it. On a copy of my live site it took several days (and then I gave up). I thought that some work had been done on the script to improve it?

Average of ratings: -
In reply to Howard Miller

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

I don't know, sorry. I haven't been involved in that issue.

Average of ratings: -
In reply to David Mudrák

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Answering my own question - MDL-58729

It would seem that substantial improvements were achieved. approve

Average of ratings: Useful (1)
In reply to Howard Miller

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Ray Hinton -

Hi Howard and David,

Yes, I was going by the recommendation on MySQL full unicode support that said "For large sites, this script will be slow" and in the old version of that page, it then included a link to the (now-deleted) page with the find/replace commands. Perhaps that sentence ("... this script will be slow...") could be removed? Then there would be no more possible confusion. Based on that tracker issue, it seems like the tests that David Monllao reported indicate that the script runs much much faster! Taking less than 10% of the unpatched time, that's amazing!

Anyway, I think all of you are caught up as to exactly what I was asking and experiencing. Thanks again!

- Ray

Average of ratings: -
In reply to Ray Hinton

Re: Converting to UTF8 - what Moodle Docs procedure to use?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I think it might still be slow on large sites. But there's slow and then there's S L O W wink

On the 3.3 and 3.4 doc pages I have updated that line to say...

On very large sites this may take a long time to run. You should probably establish how long on a test install before taking your live site offline. In some cases you might consider dumping and re-importing your data.
Average of ratings: Useful (2)