Restore breaking image references?

Restore breaking image references?

by John Isner -
Number of replies: 7
I recently restored a course backup from a moodle 1.5.4 site (the "source site") into a course on 1.6+ site (the "target site").   The course contains about 1800 questions.   Most of the questions contain images.

I'm not sure if this is relevant, but the course, which is #58 on the source site, was restored into course #1 of the target site.  This was done so that the images would become Site Files and would be visible everywhere in the target site  (all question categories are published in the target site).   This is supposed to work, and it seems to work just fine.

After the restore, I logged in as a teacher in one of the courses in the target site to check out the questions.   To my surprise, many of the images appeared as broken links, while others displayed properly.  I looked for a pattern and discovered the following.

Consider what happened to one of my images.  On the original site, the img src attribute looked like this:

http://sourceSite/file.php/58/images/CBOS/CBOS  1-3-18.jpg

Notice the space in the file name?  Yes, I know: a terrible practice, but it seemed like a good idea at the time smile

After restoring the backup into course #1 in the new site, the img src attribute looks like this:

http://targetSite/moodle/file.php/1/images/CBOS/CBOS%201-3-18.jpg

So the restore re-mapped the image src URI very nicely, EXCEPT for the way it handled the space in the filename.  It replaces blanks by 20%.  In fact, all image files with names containing spaces have this problem and display as broken links.  Unfortunately, there are over a thousand such files.

I have two questions:
  • Is this a bug in backup/restore?
  • is there a relatively quick way for me to repair my target site?  I don't want to have to go back to the source site and rename all my images.  This would take days!  For example, what if I manually rename target site file names by replacing space by 20% ?
Average of ratings: -
In reply to John Isner

Re: Restore breaking image references?

by John Isner -
I timed out before I could add one more bit of information....

The original image files were uploaded to the source site as zip files and were unzipped through Moodle's files interface.  Moodle Unzip did not replace spaces in filenames by underscores, which (I am told) it should have done.  So the spaces remained as spaces.
Average of ratings: -
In reply to John Isner

Re: Restore breaking image references?

by Chris Throup -

Two ideas for you, John:

  1. Do the images appear in the main site files? Course no.1 is the main site; if you are expecting these images to be inside a course, it will be a different number (2, 3, 5 or 7 by the look of your site);
  2. If the images are in the right place, are the filenames written in the correct case? Some servers care about upper/lower case whereas others do not. For example, your old server may have treated CBOS 1-3-18.jpg identically to cbos 1-3-18.jpg, whereas your new server might be slightly pickier smile.
Average of ratings: -
In reply to Chris Throup

Re: Restore breaking image references?

by John Isner -
Hi Chris,
The files were restored into the correct directory (moodledata/1) and the filenames are OK.  The problem turns out to be something entirely different.

When I examined the contents of moodledata/1 more closely, I discovered that not all the image files in the zip archive had been restored, even though the quizzes that reference those images were restored.  The missing images were causing the broken links, not spaces in filenames!

Now the mystery is why some of my files (actually, most of them) were not restored.

Fortunately, I kept detailed records at each step of the backup and restore.   Here is the structure of the course files on the old system (the course was number 58 on the old system):

       moodledata
       |
       58
       |
       +----------------- images
       |                     |
   78 images   +-------- C
               |         |
               |         214 images
               |
               +-------- K
               |         |
               |         91 images
               |
               +-------- Q
               |         |
               |         28 images
               |
               +-------- T
                         |
                         39 images

I checked that the zip archive created by backup contained exactly these files and directories.

Here is the structure of the Site Files after the restore

       moodledata
       |
       1
       |
       +----------------- images
       |                     |
   78 images   +-------- K
                         |
                         14 images
               

In other words, three entire subdirectories were omitted and one directory (moodledata/1/images/K) was only partially restored.  All 78 images residing directly in moodledata/1 were restored.

I do not see any pattern in what files were restored or omitted.  For example, some of the restored images have names with blanks and some do not.

Once again, every image is referenced by exactly one quiz, and all the quizzes were restored.

Curious note: During the restore, this message was displayed:

Copying course files.....
    File/Folders: 80

I have no idea where the 80 comes from.  There were no warnings or error messages during the restore that would have indicated anything was being omitted.


 
Average of ratings: -
In reply to John Isner

Re: Restore breaking image references?

by Chris Throup -

Hi John. I have had two thoughts on this; one to get your site working now and one to try to solve the problem:

  1. If you have access to the file-systems for both servers then you could copy the files manually. Download the contents of moodledata/58 from the old site onto your own computer, and then upload those files to moodledata/1 on the new server.
  2. Are there any files with punctuation in their names? I have managed to totally break a Moodle course in an earlier version just by using an apostrophe (') in a user's name—it is possible there is a similar weakness in the backup/restore code.
Average of ratings: -
In reply to Chris Throup

Re: Restore breaking image references?

by John Isner -
Hi Chris,
That is exactly what I plan to do next.  It will be interesting to see if the quizzes link to the copied images when the filenames contain spaces.

This may be relevant.  In restoring my 1.5.4 backup into my 1.6+ course, I ignored the following warning:

Warning: this backup is from a non-Unicode version of Moodle (pre 1.6).  If this backup contains any non-ISO-8859-1 texts then they may be corrupted if you try to restore them to this Unicode version.

because I thought all my characters were ASCII.  I wonder if this could be the problem?
Average of ratings: -
In reply to Chris Throup

Re: Restore breaking image references?

by John Isner -
Hi Chris,
A while ago you made the above suggestion, and I just got around to trying it.  I made a really interesting discovery -- possibly of a bug.

Recall that I have a quiz question that has image reference with an encoded space (html %20):

<img src="http://mysite.com/moodle/file.php/1/images/CBOS/CBOS%201-3-18.jpg"/>

Following your suggestion, I downloaded the image from the old site and uploaded it to the new site (using the moodle "upload" button).  I checked the quiz but the link was still broken.  Then I noticed that moodle had replaced the space with an underscore when I uploaded the file:

moodledata/1/images/CBOS/CBOS_1-3-18.jpg

Here's what I did next:
  • Delete CBOS_1-3-18.jpg from the files area
  • Zip CBOS 1-3-18.jpg (creates CBOS 1-3-18.zip)
  • upload CBOS 1-3-18.zip (creates CBOS_1-3-18.zip)
  • unzip CBOS_1-3-18.zip (creates CBOS 1-3-18.jpg)
In other words, moodle replaced space by underscore in the zip file name, but not in the filenames of the contained files. 

I rechecked the quiz and the image was intact.

So the solution is for me to make the transfer using zip files in order to preserve the spaces in their names.  I probably would have done this anyway, but this was an interesting discovery.

Do you think this is a bug, or was it a conscious moodle design decision (or an intentional loophole)?
John

Average of ratings: -
In reply to John Isner

Re: Restore breaking image references?

by Anthony Borrow -
Picture of Core developers Picture of Plugin developers Picture of Testers
I am not an expert, but I would imagine with a carefully crafted SQL statement you should be able to clean this up in the database. Thanks for reminding me of why it is good practice to avoid spaces and for reporting this behavior. Perhaps someone with a little more experience can point you in the right direction.
Average of ratings: -