Bug in Text.

Bug in Text.

yazan Jonathan List -
Yanıt sayısı: 23

I am doing a lesson module which requires text from a wikipedia entry.  The interesting thing is that the text makes Moodle freak out when I add it to the lesson module.

The offending text is:

The lowest grade level score in theory is −3.40, but there are few real passages where every sentence consists of a single one-syllable word. Green Eggs and Ham by Dr. Seuss comes close, averaging 5.7 words per sentence and 1.02 syllables per word, with a grade level of −1.3. (Most of the 50 used words are monosyllabic; "anywhere", which occurs 8 times, is the only exception.)

-----

I had removed links and edited the HTML so there were no hidden bits, and it wouldn't work. This is really strange behavior. I'm using the latest Simple Scripts install of Moodle.

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Jonathan,

  1. When you say "I am doing a lesson module" do you actually mean you are using the Moodle Lesson activity?
  2. What type of page or question are you pasting your text into (MCQ question, or "branch table" or what else?
  3. Which version of Moodle are you using : 1.9.x or 2.0.x?
  4. What do you mean exactly by "it wouldn't work"? What happens?
  5. "I'm using the latest Simple Scripts install of Moodle." No idea what that means.

Joseph

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Mary Cooch -
Documentation writers 'ın resmi Moodle HQ 'ın resmi Particularly helpful Moodlers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Sorry- I don't understand what the problem is? What is making Moodle "freak out"?

Joseph -just realised you posted at the same time as I did - Simple Scripts I believe is another automated installer like Fantastico.

Mary Cooch yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

The text itself is making moodle throw an error.

I am attempting to add a question to a lesson activity, and it apparently doesn't matter if it's a multiple choice or numerical response; I didn't try any other types of questions.  When I submit the question, I get an error, but it doesn't give me any error messages with it.

I finally got the question to submit when I removed that paragraph from the body of the question.  I suspected that it might be a PHP error; that the semi-colon wasn't getting escaped properly, but I added a semi-colon to the other text and it worked fine.

And you're right, Simple scripts is the installer.  I'm using the most up to date version they offer (2.0).

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Cannot reproduce this "bug" and am skeptical that the text you mention would cause a bug anyway.

Please Jonathan, can you be more specific. Could you post here exactly the full contents of Lesson question that you say causes the problem:

  1. question type:
  2. question text ("Page contents"):
  3. Answer1 Answer:
  4. Answer1 Response:
  5. Answer2 ...
  6. etc.

"When I submit the question, I get an error, but it doesn't give me any error messages with it."

What do you mean by "I get an error" ? Have you tried to switch to "Display PHP errors ALL" in Admin / debug?

Post a screenshot showing the "error"...

As far as I know semi-colons do not need to be escaped.

Joseph

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

1: Already answered

2: Already given

3: Irrelevant

4: Also Irrelevant

5: etc.

"As far as I know semi-colons do not need to be escaped." -- In which case I am unsubscribing from this forum.  And you should probably put the following into your unescaped application:

username; SET FOREIGN_KEY_CHECKS = 0; DROP DATABASE;

And yes, I apparently did imagine the entire two hours it took to troubleshoot the issue.

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Hi Jonathan,

Sorry if I made it sound like I did not take your problem seriously.utanmış We are all volunteers here on the Moodle forums, and are very willing to help, but you must understand we need precise information and we need to be able to replicate your problem, otherwise I don't see how we can provide a) a diagnostic and b) a possible remedy.

1.- Could you please tell us where you type (or paste?) what you call "the offending text": The lowest grade level score in theory is −3.40, but there are etc.

a. In the question's text (called "Page contents")

b. or in Answer 1 (2... etc.) Answer field

c. or Response field.

1a.- Did you actually type that text in Moodle's HTML editor or did you copy-and-paste it from somewhere else (from your Word processor, from a Website, from ?).

2.- If (b) or (c), which type of format did you select from the Format drop-down list (see attached screenshot).

3.- Sorry to insist, but it would be really useful to know how you can tell that there is an error, or, to put it your way, "makes Moodle freak out". It would help if you could tell us what you mean by "freak out".

Looking forward to your reply,

All the best,

Joseph

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

Ok, for whatever the reason, the debugging options didn't show everything yesterday.  (I wonder if... no matter) so I got the debugging scripts to show all errors, and here is the dump.  I will do a screencast of the error later.  It is very obviously a bug of some kind, but I can't figure out why it's happening.

Debug info: Incorrect string value: '\xE2\x88\x923.4...' for column 'contents' at row 1
INSERT INTO mdl_lesson_pages (title,contents,contentsformat,lessonid,timecreated,qtype,qoption,layout,display,prevpageid,nextpageid) VALUES(?,?,?,?,?,?,?,?,?,?,?)
[array (
0 => 'Flesch-Kincaide Scale',
1 => '<p>These readability tests are used extensively in the field of <a href="http://en.wikipedia.org/wiki/Education">education</a>. The "Flesch–Kincaid Grade Level Formula" translates the 0–100 score to a <a href="http://en.wikipedia.org/wiki/Education_in_the_United_States#School_grades" title="Education in the United States">U.S. grade level</a>, making it easier for teachers, parents, librarians, and others to judge the readability level of various books and texts. It can also mean the number of years of education generally required to understand this text, relevant when the formula results in a number greater than 10. The grade level is calculated with the following formula:</p>
<dl><dd><img class="tex" alt="0.39 \\left ( \\frac{\\mbox{total words}}{\\mbox{total sentences}} \\right ) + 11.8 \\left ( \\frac{\\mbox{total syllables}}{\\mbox{total words}} \\right ) - 15.59" src="http://upload.wikimedia.org/math/a/3/a/a3a80e6e52fda2b5f7647a451c9c6c13.png" /></dd></dl>
<p>The result is a number that corresponds with a grade level. For example, a score of 8.2 would indicate that the text is expected to be understandable by an average student in 8th grade (usually around ages 13–14 in the United States of America). The sentence, "The Australian platypus is seemingly a hybrid of a mammal and reptilian creature" is a 13.1 as it has 26 syllables and 13 words. This page has the score of 12.1.</p>
<p>The lowest grade level score in theory is −3.40, but there are few real passages where every sentence consists of a single one-syllable word. <i><a href="http://en.wikipedia.org/wiki/Green_Eggs_and_Ham">Green Eggs and Ham</a></i> by <a href="http://en.wikipedia.org/wiki/Dr._Seuss">Dr. Seuss</a> comes close, averaging 5.7 words per sentence and 1.02 syllables per word, with a grade level of −1.3. (Most of the 50 used words are <a href="http://en.wikipedia.org/wiki/Syllable" title="Syllable">monosyllabic</a>; "anywhere", which occurs 8 times, is the only exception.)</p>',
2 => '1',
3 => '2',
4 => 1303150390,
5 => '3',
6 => 0,
7 => 0,
8 => 0,
9 => '34',
10 => '0',
)]
Stack trace:
  • line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown
  • line 874 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
  • line 916 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw()
  • line 1760 of /mod/lesson/locallib.php: call to mysqli_native_moodle_database->insert_record()
  • line 121 of /mod/lesson/editpage.php: call to lesson_page::create()
Jonathan List yanıt olarak

Re: Bug in Text.

yazan Yvette Martinez -

The incorrect string value usually refers to character encoding in the database.

I don't know but I suspect that your database is not set to unicode as moodle 1.9+ requires.

Yvette Martinez yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

wouldn't that influence any data added to the moodle?  This is the only time it has happened.

Double Checked & it's unicode.  So that's not it.

Yvette Martinez yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

I expect Yvette is right in her interpretation of the error message. Here is some more info about the problem at hand.

1.- It appears that the chunk of text copied from the Wikipedia article (Flesh-Kincaid readability test) is formatted as utf-8. This is not immediately obvious, but there are two utf-8 characters in the last paragraph, the minus signs in "The lowest grade level score in theory is −3.40,..." and "... with a grade level of −1.3." This, by the way, explains why - as mentioned in your screencast - when you remove that last paragraph your question text saves OK in the Lesson! And I was wrong in blaming the Latex bit in the Wikipedia text.utanmış

2.- Further to Yvette's remark I conducted the following experiment.

a. As reported in my previous messages, on my local Moodle 2.0 site I do not have the error you mention and I can save my question including the Wikipedia text as a Lesson question text.

b. This is what I did to replicate your problem, on my local Moodle 2.0 test site. My local site is set to utf-8, as required by Moodle. I changed the mdl_lesson_pages table from utf8_general_ci to latin1_general_ci.

c. Then I edited my Lesson question page contents by pasting the last paragraph from the Wikipedia text:

The lowest grade level score in theory is −3.40, but there are few real passages where every sentence consists of a single one-syllable word. Green Eggs and Ham by Dr. Seuss comes close, averaging 5.7 words per sentence and 1.02 syllables per word, with a grade level of −1.3. (Most of the 50 used words are monosyllabic; "anywhere", which occurs 8 times, is the only exception.)

And saved. Got the error message (similar to the one you get):

Error writing to database
More information about this error
Debug info: Incorrect string value: '\xE2\x88\x923.4...' for column 'contents' at row 1
UPDATE mdl_lesson_pages SET qtype = ?,title = ?,lessonid = ?,qoption = ?,contents = ?,contentsformat = ? WHERE id=?
[array (
0 => '3',
1 => 'Flesch–Kincaid Scale',
2 => '1',
3 => '0',
4 => '<p>The lowest grade level score in theory is −3.40, but there are few real passages where every sentence consists of a single one-syllable word. Green Eggs and Ham by Dr. Seuss comes close, averaging 5.7 words per sentence and 1.02 syllables per word, with a grade level of −1.3. (Most of the 50 used words are monosyllabic; "anywhere", which occurs 8 times, is the only exception.)</p>',
5 => '1',
6 => '1',
)]
Stack trace:
line 394 of \lib\dml\moodle_database.php: dml_write_exception thrown
line 980 of \lib\dml\mysqli_native_moodle_database.php: call to moodle_database->query_end()
line 1012 of \lib\dml\mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->update_record_raw()
line 2133 of \mod\lesson\locallib.php: call to mysqli_native_moodle_database->update_record()
line 119 of \mod\lesson\editpage.php: call to lesson_page->update()

Note.- "Incorrect string value: '\xE2\x88\x92" refers to the Unicode value of the "minus" sign used in the Wikipedia article. See UTF-8 encoding table and Unicode characters, page with code points U+2200 to U+227F.

d. Then, I reset my mdl_lesson_pages table back to utf8_general_ci and re-edited the question text, and saved it without any problem, as before the coding switch experiment.

3.- From this little experiment it seems likely to me that your moodle site's database is not set to Collation = utf8_general_ci. Or the mdl_lesson_pages table is not set to Collation = utf8_general_ci (which would be surprising, but...).

I know you have double-checked, but you might like to treble-check (& maybe post a screenshot).

4.- Have you tried the little experiment I suggested in my post dated Monday, 18 April 2011, 11:43 PM:

Try creating a simple "Page" (from the Add a resource menu) in your moodle course, and paste that Wikipedia chunk of text and see what happens.

5.- Is your current Moodle 2.0 site a brand new installation or an upgrade from a previous 1.9 version site?

ATB

Joseph

 

 

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Yvette Martinez -

Thank you for testing my hypothesis Joseph. I suspect this is an upgraded installation.

I'd like to also add that there are three places in which to change encoding. The database itself, the database tables and the individual fields within the tables. It is necessary for these updates to be made when upgrading from older versions to 1.9.9 and 2.0. Actually I'm not exactly sure which versions before 1.9.9 might have required this. I only know that I had to make these updates for my 1.9.9 upgrade and when setting up a test 2.0 environment using my existing database.

Changing the db and table is easy enough however the individual collation fields within each table have to be changed manually.

Yvette Martinez yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

Nope, this is a straight-up 2.0 install.

utf8

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Hi Jonathan,

1.- Could you please try the experiment I suggested in my post dated Monday, 18 April 2011, 11:43 PM:

Try creating a simple "Page" (from the Add a resource menu) in your moodle course, and paste that Wikipedia chunk of text and see what happens.

You might also try to paste that Wikipedia text (or at least the last paragraph which features the utf8  _ signs) into some other activities or resources on your Moodle site (e.g. a quiz question, a glossary entry, etc.). If all goes well, then we can confirm that the problem is restricted to your Lesson module.

2.- Unfortunately the screenshot of "utf8_general_ci" column you posted does not show the table names column. Could you please post a screenshot similar to the one attached to this post?

3.- What kind of database do you have on your moodle site (mysql, postgres, etc.)?

Thanks in advance,

Joseph

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

1) Test page produced an error.

2) I am a moron.

3) mysql

2a- I was looking at the media wiki database (repeatedly).  Since this is a fresh install of Moodle, why would it create a latin1_swedish_ci database?  I don't really want to dump my Moodle install and reinstall, is there a simple way to switch it all to utf8?

Seems like this is a problem from simplescripts... I'll send them an e-mail letting them know their install has a bug.

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

"... is there a simple way to switch it all to utf8? Seems like this is a problem from simplescripts..."

There might be a way, but I am no specialist of databases. Please submit the problem to the "simplescripts" people. They should be able to help.

I'm glad the problem has been diagnosed at last and you know how to cure it. Many thanks to Yvette Martinez for putting us on the right track.onaylamış

Joseph

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

Not to be a nidge, but to prevent this problem from reocurring, can't there be a line of code in the setup that checks for utf8?  If It's not utf8, the install fails, no matter what.

Jonathan List yanıt olarak

Re: Bug in Text.

yazan Yvette Martinez -

I believe that this could partly be a webhost issue if the mysql server itself is set to swedish.

Since your database is already installed the following two links will help with the first part.

How can I change my MySQL database collation?

http://kb.siteground.com/article/How_can_I_change_my_MySQL_database_collation.html

How to change the collation for all tables in a MySQL database to UTF-8?

http://kb.siteground.com/article/How_to_change_the_collation_for_all_tables_in_db_to_UTF8.html

[btw, I am not affiliated in any way with siteground, I do my own hosting, but I am grateful to them for posting these solutions in their knowledge base.]

Unfortunately I haven't found any script that will help with changing the collation of the actual fields. I've seen it asked many times but never answered. What I am talking about is represented in the following screenshot.

 

If you can start over and dump the database then create the empty db first, check/update the collation, then install moodle.

If you can not dump the database these fields will have to be changed individually. It's tedious and boring with over 180 tables and many fields to change. It may seem daunting but it will only take a couple of hours once you get on a roll.

I am glad I was able to help!


Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Jonathan:

to prevent this problem from reocurring, can't there be a line of code in the setup that checks for utf8?  If It's not utf8, the install fails, no matter what.

Well, that feature is provided by the moodle 2.0 install script. If you try to install moodle 2.0 with a database whose collation is not set to utf8 (e.g. latin1_swedish_ci, etc.), then the install script does stop with the following warnings:

Moodle 2.0.2+ (Build: 20110413)
Server checks
Name     Information     Report     Status
unicode         must be installed and enabled
It is required that you store all your data in Unicode format (UTF-8). New installations must be performed into databases that have their default character set as Unicode. If you are upgrading, you should perform the UTF-8 migration process (see the Admin page).
[...]
You must solve all the environmental problems (errors) found above before proceeding to install this Moodle version!

And you cannot proceed with the installation. Just like you requested.

The problem in your case seems to be that you did not install moodle yourself but you rely on the "simplescripts" organisation (of whom I do not know). Again, you should take your problem to them, not to moodle.org, and ask them a) why they set your moodle database to latin1_swedish_ci rather than to uf8_general_ci and b) how they could have "installed" a moodle 2.0 version with that database collation, since it is not possible to do so.

Hope that helps.

Joseph

PS.- To show your appreciation of help given in these forums, you may either say "thank you" in one of your replies or select "Useful" in the Rate dropdown list (or both) gözkırpıyor.

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Jonathan List -
Jonathan List yanıt olarak

Re: Bug in Text.

yazan Joseph Rézeau -
Core developers 'ın resmi Particularly helpful Moodlers 'ın resmi Plugin developers 'ın resmi Testers 'ın resmi Translators 'ın resmi

Thanks for the debug info and the screencast, Jonathan.

In the chunk of text that you are copying from Wikipedia, there seems to be a Latex or Tex "formula" in the alt tag of the image as well as the resulting "translation" of that formula into a *.png image file.

When I paste the exact same chunk of text from Wikipedia into the question text (page contents) of a MCQ question of a Lesson on my local Moodle 2.0 test site there is no problem, everything is copied and saved fine and I can view the question text, with the image formula when I preview the question.

I am not familiar with Latex and Tex formulas and never use them in Moodle, so all I can say is that maybe there might be a kind of incompatibility problem between that formula and either your version of Moodle or something installed on your server.

This is what I suggest:

1.- Try creating a simple "Page" (from the Add a resource menu) in your moodle course, and paste that Wikipedia chunk of text and see what happens. If you are not getting any error and the text displays fine - including the image - then there must be something wrong with the Question editor in the Lesson module (or your Moodle server).

2.- Try this workaround. Paste the following "cleaned" text - where I have removed all links and the "latex formula" from the alt tag - into your question text and try to save and see what happens. I am attaching the text in a Flesch–Kincaid.txt file.

3.- In your screencast, towards the end, you say that if you remove the last paragraph from the Wikipedia text, you can save the question. Unfortunately your screencast does not show what a preview of the question looks like. It would be interesting to see if the formula is there. I am surprised that simply removing the last paragraph (which only contains "normal" text) would solve the problem...

4.- In general it is not a recommended procedure to copy-paste anything into Moodle's HTML editor, either from a MS Word document or a Web site, since you never know what "hidden" code you might be pasting, with unpredictable results. If you need to copy chunks of text, I recommend pasting into a simple text editor such as Windows' Notepad, then copy-pasting from that simple editor into Moodle's HTML editor and finalising your formatting in the HTML editor. Much safer (and time-consuming, unfortunately).

ATB

Joseph

Joseph Rézeau yanıt olarak

Re: Bug in Text.

yazan Jonathan List -

The funky part about the whole thing is that I removed the Latex, assuming that you were right, and it didn't work.  I also went into the underbelly of the text and checked it out.  I have no honest clue why the text didn't work.

Oh well, I'm sure it'll pop up again.