Glossary Import problem

Glossary Import problem

by J B -
Number of replies: 34
Since upgrading to 1.6.1 I have been experiencing problems with creating and updating glossaries.

If I try to import an xml file, the glossary will import (up to) the first two entries, or just act as if nothing had been uploaded. It does not seem to matter if the file is a freshly exported one, from the moodle exchange, or one I prepared using my text editor.

As a workaround I tried the new "import" feature, which allows editors to import content from their other courses, and this was successful (I tried zipping and restoring the course, adding the content to the destination, but this didn't work).
Average of ratings: -
In reply to J B

Re: Glossary Import problem

by Simon Clegg -

Hi James,

Iv'e just posted exectly the same problem. Did you disover the cause?

Simon

In reply to Simon Clegg

Re: Glossary Import problem

by J B -
Hi Simon

I am sorry to say I did not, which is a shame as it does restrict the portability of content. I just tried again and it does exactly the same thing on our (now 1.6.3+) installation.
In reply to J B

Re: Glossary Import problem

by Simon Clegg -

Hi James,

I suppose it could be possible that we are the only two people experiencing this problem.

I would just like to be clear that we are are suffering from the same issue. My installation is refusing to import some glossaries but allowing others. There is no clear way of knowing why because no error message is being delivered, it just acts as if nothing has happened. Does this match your situation?

I am confident I have discovered Moodle's selection criteria and it seems to be based on file size. However, essentially this is a guess for as I mentioned there are no error messages. It is further confusing because the file size restriction is based on because the glossaries I try to upload are all well below the system limit.

The solution I have come up with is to chop the large glossaries up into small files and hey presto! They now import. I don't know how and I don't know why. I have no way of telling what the precise file size limit is. I am just divide the files up arbitrarily and hope for the best.

Although this solves my ultimate problem of getting the glossaries into my courses, it is far from adequate. Could anybody shed any light on this? In the meantime, James, I hope you find this of some (small) use.

Simon

In reply to Simon Clegg

Re: Glossary Import problem

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Hi Simon,
If you would post (on this forum or attached to a private e-mail to me) one of the larger glossaries which you have problems importing into Moodle, I might be able to test it on my 1.6.3 install and report back.
Joseph
In reply to Joseph Rézeau

Re: Glossary Import problem

by Simon Clegg -

Thanks Joseph,

You can take your pick from any of the larger glossaries in the Moodle Exchange. But the one I am particularly interested in in the Japanese-English dictionary.

Again, many thanks.

Simon

In reply to Simon Clegg

Re: Glossary Import problem

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Simon,

Just downloaded the Japanese-English dictionary and tried to import into a glossary of my Moodle 1.6.3 installation. Here are the steps I followed.

  1. Unzip japanese_-_glossary.zip to japanese - glossary.xml
  2. In one course, create a new glossary, call it e.g. Japanese
  3. In (empty) Japanese glossary view, click on Import Entries tab
  4. File to import: japanese - glossary.xml
  5. Destination: select Current Glossary (selecting New Glossary does not work sad).
  6. Click Save changes
  7. Wait...
  8. It works!
Can you please follow these steps and tell us where things go wrong for you.

Joseph

Attachment image000.jpg
In reply to Joseph Rézeau

Re: Glossary Import problem

by Simon Clegg -

many thanks for your time Joseph,

I followed your steps which are the steps I always take. As I mentioned in one of my other posts, I had no problem with this when my site was on a different server. But I am getting no hints as to the cause. Furthermore, I can import this file if I chop it up into smaller pieces (?)

Attachment before-after.gif
In reply to Simon Clegg

Re: Glossary Import problem

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Maybe the problem has to do with the Japanese characters included in the word definitions? Have you tried with a "normal" glossary, i.e. plain English words?
Joseph
In reply to Joseph Rézeau

Re: Glossary Import problem

by Simon Clegg -

I'm quite certain it is not that. I am convinced now that it is something to do with the size of the import. I have managed to import the whole glossary, but I did this by slicing it up into smaller files. Pretty laborious.

Can't for the life of me imagine what is happening. As you can see from the screen shots the upload setting gives us ample to play with. The encoding can't be to blame because the database is set to unicode. And besides, I have now managed to import the file piecemeal. The system is giving me no clue. Just decides to ignore the import if its not happy with the size of the file. Haven't yet worked out the exact threshold...

In reply to Simon Clegg

Re: Glossary Import problem

by Just H -
Hi Simon

This may be a really dumb question (no reflection on your admin/server knowledge inferred) but have the upload limits on your new server been set up correctly?

Reason I ask is I believe I saw someone else having upload issues in general a fair while back and they hadn't changed the post_max figure if I remember correctly.

H
In reply to Just H

Re: Glossary Import problem

by Simon Clegg -

Hi Harry,

Perfectly valid question. Yes, server upload limit is set to 20Mb which is reflected in my Moodle global settings. As you can see from the screen shots, the limit for the course into which I have been trying to import the glossary is set to 10Mb.

Besides, shouldn't I expect a warning if an upload/import exceeds the limit?

Baffled mixed

In reply to Simon Clegg

Re: Glossary Import problem

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Simon,
Have you tried setting debug on in Admin to see if you get extra warnings/error reporting that way?
Joseph
In reply to Joseph Rézeau

Re: Glossary Import problem

by Simon Clegg -

Thanks Joseph,

Yep. Done that. No message.

Simon

In reply to Simon Clegg

Re: Glossary Import problem

by David Fountain -
If it is the larger files only and only on one server might I suggest that there is a time out happening on the server?
In reply to J B

Re: Glossary Import problem

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
Hi James,
It is not quite clear what you are intending to do. Do you mean that you want to create from scratch a new Glossary outside of Moodle, using an XML editor and then import that file into a Moodle Glossary? Are you quite sure you have the right template to do that? Which XML editor are you using?
Can you post on this forum a sample glossary thay you have created and cannot manage to import?
Joseph
In reply to Joseph Rézeau

Re: Glossary Import problem

by lesley beasant -

Not sure if this might help

In summary this is simple to do

  • In Excel copy and paste the glossary terms into a particular spreadsheet (this is attached) then follow the spreadsheet macro instructions to create an import file
  • In Moodle create an empty glossary in the appropriate course/topic and upload the file of terms created via the spreadsheet

Step 1 – Create a Spreadsheet Glossary

Within the Moodle Training Course Glossary Topic is a spreadsheet attachment called Moodle Glossary Prep Sheet

  • Click on this spreadsheet to open it.
  • Click on the enable macros button in the message box that appears
  • Use File and save as to create your glossary as your own spreadsheet and save it in your curriculum shared area appropriately named
  • Open up the document or spreadsheet that holds your glossary terms
  • Copy and paste the Glossary terms and definitions into the first 2 columns of your copy of the Glossary prep sheet
    Note – at this point you may need to tidy up the glossary spreadsheet content to ensure that each row has a term and a definition.

Step 2 - Run the spreadsheet macro to create an XML file

Select

  • Tools
  • Macro
  • Macros

Click the Run button

Ensure there is a tick in each of the 3 tick boxes (if not then click in each to put a tick in)

Click the Assign Values button to update the spreadsheet prior to exporting it.


Click the Create XML file button

In the Save as window that opens

  • Browse to find the appropriate folder in your curriculum shared area
  • Amend the file name to describe your glossary contents
  • Click on the save button
  • Close the Moodle Glossary Preparation for Import dialogue box (shown above).

The XML file is now available to import into a Glossary.


Step 3 - Create a Glossary

Go to the required Boodle course and turn editing on.

In the required topic select the Glossary activity.

Name the Glossary

Enter a description

Scroll down the screen

Initially you can leave the system defaults as they are.

Note - If you want to know more about Glossaries in detail then see the ‘Fareham College Guide to Glossaries’ in the Boodle Training course, Glossary Topic.

Scroll down

Save the glossary by clicking on
Save changes


The Glossary is then generated

Step 4 – Populate the Glossary with your XML file

To import entries click on the Import entries link

The screen below will appear.

Click on the Browse button and find the XML file that you created in step 2.

Click on the Save changes button to import the glossary terms.

A report will then be generated telling you how many entries were imported or possibly list any that were rejected and why.

Glossary entries will be rejected if they are duplicates or if they contain special characters such as the degree symbol .

Review the Import and rejection report. If necessary correct the spreadsheet entries and re run the spreadsheet macro (follow step 2).

Click the continue button at the foot of the import report and you will be returned to the glossary.

You can now either review the glossary (see the Fareham College guide to glossaries) or import more glossary entries.

It is OK to re import a similar XML file (e.g. after correct errors as duplicate entries will be rejected and only new entries (not previous imported) will be brought in.

In reply to lesley beasant

Re: Glossary Import problem

by Sarah Dutcher -
Hello,
This is a great process and it works just fine unless my spreadsheet is displaying the characters in Japanese. For whatever reason if my spreadsheet is showing the Japanese characters as gobbledy-gook, it works just fine. I've attached the spreadsheet that works (for the one that displays Japanese characters correctly, I just get question marks instead of characters in the XML file).
I have a feeling this is some kind of encoding issue, but I don't know how to fix it. Any advice would be greatly appreciated.

Thanks!

Sarah Meadus
In reply to Sarah Dutcher

Re: Glossary Import problem

by David Fountain -

Sarah,

If it is my spreadsheet your are using then it is most certainly a problem with the encoding (utf-8) for which Hellmuth Sole has found a solution for accented charaters and which might work with yours…if you try it and it does please let me know. http://moodle.org/mod/forum/discuss.php?d=79120#p442126

Otherwise there is an alternative solution created by Yasu Imao about which I have heard only good things but I have not personally used. http://moodle.org/mod/forum/discuss.php?d=91224#p404192

Regards

Dave

In reply to David Fountain

Re: Glossary Import problem

by Sarah Dutcher -
Thank you so much for the quick response Dave!

I found that using the Glossary XML converter works best. I got it here http://moodle.org/mod/forum/discuss.php?d=91224#p404192. Thanks for giving me the idea.

I can just copy and paste whatever I have in Excel to the left hand side and then convert it to XML and it keeps the Japanese characters with no problem.

I REALLY appreciate the help. Thanks again!

Sarah
In reply to lesley beasant

Re: Glossary Import problem

by Linnea Hannigan -

Have followed your directions to a T, and I still cannot get the file to import into my Moodle correctly.  It either gives me the Parsing Error message, or it only imports 30 out of 60 entries.  I actually had 120 entries, but I cut the file in half thinking it might help.

I have Moodle 1.9.2

There are no non-English characters - it is a Business English glossary.

Any help would be appreciated.

Linnea Hannigan

In reply to Linnea Hannigan

Re: Glossary Import problem

by David Fountain -
Hi Linnea
Can you tell which is the last entry that is imported, there may be something unusual in its concept or definition - by that I mean a character that is one of the special characters, look for <>!"£$%^&*(_) etc.Are you using the latest version of my spreadsheet?
Failing that post me a copy of your spreadsheet file and I'll have a look.

Dave
In reply to David Fountain

Re: Glossary Import problem

by Linnea Hannigan -

Hi Dave,

Thanks for your quick reply.  I don't know if the spreadsheet I am using is yours. I got it off of a post on the Glossary thread by:

Re: Glossary Import problem
by lesley beasant - Monday, November 19, 2007, 08:36 PM
  Copy_of_MoodleGlossaryPrepSheetWithCategory.xls

Her post was so detailed and helpful, I know I must be doing something wrong.

I can't figure out what makes the uploading stop.  None of the characters seem out of order (special), but I am SO not good at this tech thing...or at Excel.  My glossary was "cut/paste" definition by definition from the web. In other words, I didn't "type in by hand" the concept and the definition.

Sometimes, when it stops importing, if I open the xlm version of the file and scroll down to where it seems to have ended, it will give a message like:

The XML page cannot be displayed

Cannot view XML input using XSL style sheet. Please correct the error and then click the Refresh button, or try again later.


An invalid character was found in text content. Error processing resource 'file:///C:/Documents and Settings/Linnea/Desktop...

/table>

IV>

  <FULLMATCH>1</FULLMATCH>
I realize that something is wrong, I got to the definition (as in "positions customer" listed above), but I don't see what is wrong. It is no different from other definitions.  I will attach my EXCEL file for you.  If you want to run it through the Tools>Macro>etc and then see if it uploads for you to a Random Glossary Block, go ahead!  I am trying so hard to learn this Moodle stuff, but I am a grammar teacher, and Moodle is so insanely frustrating for me !!! 
Thanks so much,
Linnea
I
In reply to Linnea Hannigan

Re: Glossary Import problem

by David Fountain -
Hi Linnea
Yes that's mine, Lesley made a step by step guide for it which she very kindly shared. The problem is with 'special' characters, '&' in the definition of prime time and the concept R & D. Also the (R) in registered.

Removing these allowed it all to go through.

I've attached the latest version of the spreadsheet to this post.
In reply to Linnea Hannigan

Re: Glossary Import problem

by David Fountain -
and a copy of the xml file for you to import.
In reply to David Fountain

Re: Glossary Import problem

by Linnea Hannigan -

Hi David,

Bless you a million times over.  This is such a huge favor to me!  My project is due today, and I've been struggling with this glossary for so long. Thank you for taking the time to teach me what was wrong, as well as to run the macro.

Linnea wink

In reply to Linnea Hannigan

Re: Glossary Import problem

by David Fountain -
Thank you Linnea, I really must get round to adding in something to take care of those irritations.

Dave
In reply to David Fountain

Re: Glossary Import problem

by Linnea Hannigan -

Dave,

Again, thank you so very much for your help. 

Please allow me another question or two wink

Can I now add glossary terms to the Excel spreadsheet and rerun it through the Tool>Macro>etc and the formerly loaded terms will be rejected and the new terms will be added? (I've put "no duplicate entries") on the settings.

The "&" that you found was in a definition, I take it. I was copy/pasting from the web. The new entries I would like to add would be typed in by hand by me.  What symbols must I avoid?  Can I use a semi-colon (;) a dash (-)?

Finally, regarding glossaries, if I simply add the new words right on the moodle, and then click on Export, how are they exported? As a xlm file or as an excel file?

Thank you sincerely for your time and help.

Linnea

In reply to Linnea Hannigan

Re: Glossary Import problem

by David Fountain -
No problem!

You can add them, any definitions already in the glossary will be ignored as duplicates.

The symbols to be ignored are those that in html have to be represented by something other than themselves, I found a list at http://www.natural-innovations.com/wa/doc-charset.html the column I look in is the one called entity - the & for instance is &amp; One way to check would be to open a web editor, type in the symbol and see what code it produces.

On this list it looks like ; and - (hyphen) are acceptable. I'll have a look at incorporating some translation into the spreadsheet.

As for the export - try it and see, you won't damage anything by running an export! wink (but I think you get xml)

Dave
In reply to Linnea Hannigan

Re: Glossary Import problem

by David Fountain -
I think that I have sorted this problem in the latest version of the spreadsheet (found here). This version will replace the special characters with their named entity equivalents before creating the xml file. I have not tested every combination so please try importing to a test glossary before commiting to any 'real' version!

Dave
In reply to Joseph Rézeau

Re: Glossary Import problem

by J B -
Hi Joseph. belatedly

As I posted earlier, I have given up on this one, but it may be of some help to clarify the situation. Glossaries can be moved well enough for me using a selective restore.

> It is not quite clear what you are intending to do.

I was trying to import content in moodle glossary XML format in all three scenarios I had mentioned:
  1. if the file is a freshly exported one, (from a glossary instance)
  2. from the moodle exchange, or (like the FPP glossary I created in 2004 and uploaded to the moodle exchange)
  3. one I prepared using my text editor (usually SCITE, but often generated using more complex means, and yes, the template was correct)
I tried to attach a file that for experimentation's sake I exported today from a live course and tried to import, only successful for the first 8 entries. Unfortunately its size (157KB) exceeded the 100KB limit for the forum.

In reply to J B

Re: Glossary Import problem

by Neal Durando -
Sorry if anyone objects to such blatant necromancy, but I'm really put out with this problem. Like the above posters only the first two entries of maybe a thousand import. I've followed the detailed process outlined above, used the latest (circa 2008) spreadsheet template. No dice.

Unfortunately the XML file is 300K, so it won't upload here. Anyone have any ideas before I give up on this?
In reply to Neal Durando

Re: Glossary Import problem

by Frank Ralf -
Hi Neal,

This might be an encoding problem. See Import_and_export_FAQ#What_encodings_should_I_use.3F for further details.

See Importing Entries Using a Text File for an example where an additional BOM caused some trouble, see UTF-8_and_BOM.

Spreadsheet software is known to add some hidden unwanted characters to CSV files.

If the problem is with the first entries of your file you could just post an excerpt.

hth
Frank
In reply to Neal Durando

Re: Glossary Import problem

by David Fountain -

Hi Neal, just posting here following our off-board communications so others can see what we've been up to.

The problem is a little bit of encoding but also related to the import of special characters such as the

é

(e acute)

not sure why, but probably due to the proximity of other characters, but the substitution in the xml with &eacute; etc appears to have stopped working. The workaround is to look at the code for CommandButton2 and comment out the 'Call replaceSpecials line and then open the resulting xml file in notepad and with the UTF-8 encoding.

In reply to David Fountain

Re: Glossary Import problem

by David Fountain -

Problem was with the named variations of the special code replacements. By altering it to use the numerical codes it appears to be fixed though the XML output will still need to be opened in Notepad and saved with the UTF-8 encoding.

Latest version