file problem after upgrade - update with debugging

file problem after upgrade - update with debugging

by Richard Kingsley -
Number of replies: 21

I use a shared hosting service.  After upgrading, none of my private files were available but I can do the following:

All my private files are available as server files  (I have never used this option before).

I can access all other repository files including those I set up in my moodle data file.

All files added via a forum are also available.

On trying to access my private files, I get the following error message:

Debug info: [dataroot]/filedir/4e/89/4e89d6915f1c57f2e243b889f7da8cdc13741d83 Error code: storedfilecannotread
Stack trace:
  • line 175 of /lib/filestorage/stored_file.php: file_exception thrown
  • line 846 of /lib/filestorage/stored_file.php: call to stored_file->update()
  • line 399 of /lib/filelib.php: call to stored_file->set_source()
  • line 260 of /lib/filelib.php: call to file_prepare_draft_area()
  • line 65 of /user/files.php: call to file_prepare_standard_filemanager()

I have no idea what this stuff means so hopefully, there is someone out there that does.

Thanks

Richard

Average of ratings: -
In reply to Richard Kingsley

Re: file problem after upgrade - update with debugging

by Ken Task -
Picture of Particularly helpful Moodlers

Using some other tool to browse files, to go the moodledata directory.
filedir then 4e then 89

See if there is a file with a long hash name:
4e89d6915f1c57f2e243b889f7da8cdc13741d83

storedfilecannotread might indicate a permissions problem or other.
Looks like it's trying to update information on it.

When was the last time your site ran cron?

Run cron … maybe several times.

Shared hosting services sometimes do nasty things … what's the max on file space?  What's the max on how much time a script can run?  What's the max on how much memory a script can use?
The above can be seen (or should be see) in the phpinfo link from the Site Admin menu.

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: file problem after upgrade - update with debugging

by Richard Kingsley -

Ken,

Thanks for responding.

 

I ran cron a couple of times last night, so I know that everything is clear.

There is a folder 4e, but no sub-folder 89.

What do you make of that?

Richard

In reply to Richard Kingsley

Re: file problem after upgrade - update with debugging

by Ken Task -
Picture of Particularly helpful Moodlers

What do I make of that?  Computer error ... glitch ... hickup!

There is data in the mdl_files table that says a file should be there and it's not physically present.  Sooooo ... find that record in mdl_files table and remove it.

Use phpmyadmin or whatever you have of a graphical nature to remove the one record that refers to a file whose contenthash reference is:

4e89d6915f1c57f2e243b889f7da8cdc13741d83

Not that you did anything wrong, but by chance did you rename a file in private files or move one recently?  Just curious ... cause we all know the new file system is 'perfect', right? ;)

'spirit of sharing', Ken

Average of ratings: Useful (2)
In reply to Ken Task

Re: file problem after upgrade - update with debugging

by Richard Kingsley -

Ken,

 

Well I do not remember changing anything, but,  I found 19 records with that file name and deleted them all.  Guess what?

It worked!

Everything is running beautifully now.  Thank you so much for your help.

Cheers

Richard.

In reply to Richard Kingsley

Re: file problem after upgrade - update with debugging

by Ken Task -
Picture of Particularly helpful Moodlers

19 records indicates that the file might have been linked elsewhere.  Do you happen to recall what the component or filearea columns had for values?  No biggy.  It gets by the error you've seen, but wonder if there won't be others that show else where.  Hope not.

'spirit of sharing', Ken

In reply to Ken Task

Re: file problem after upgrade - update with debugging

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators

I seem to have the same problem, not on shared hosting. Some random students lost there files and there access to the private files area. At least 10 (out of 1700 accounts). We upgraded from 2.4 to 2.5. No idea what else happened sad

What I did is recreate the missing file:

root@scorpio:/var/moodledata/filedir/21/ff# ls
21ffac38fbdf20e9d3bfa0dd9e089632dbf66462
root@scorpio:/var/moodledata/filedir/21/ff# touch 21ff8b91fe5cc7deac8fc3d730fd0cc0310384f0
root@scorpio:/var/moodledata/filedir/21/ff# ls
21ff8b91fe5cc7deac8fc3d730fd0cc0310384f0
21ffac38fbdf20e9d3bfa0dd9e089632dbf66462
root@scorpio:/var/moodledata/filedir/21/ff#

This solved the problem too, although the data in the file is lost of course.

In reply to koen roggemans

Re: file problem after upgrade - update with debugging

by koen roggemans -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Translators

I know how it happened: I deleted users by accident. They were deleted by the invalid accounts cleanup script running in cron. They became invalid by deleting duplicate email addresses with  "upload users from file". Their files also get cleaned up by the cron script.

When I noticed, I restored only the database and I forgot to restore a backup of the files, so there were files in the database from the users that previously had been deleted, but on disk they didn't exist anymore.

I occasionally restore a deleted user by altering in the mdl_user table the deleted, username and a few other changed fields, but learned today that this is bad practice and can mess up your database integrity concerning the files. 

Average of ratings: Useful (2)
In reply to Ken Task

Re: file problem after upgrade - update with debugging

by Murphy Wong -

Is there a way (PHP scripts) to check the consistency of the database table mdl_files with the filesystem at ~/moodledata?  If the filesize is 0 in the database, can we simple remove them from the mdl_files table?

In reply to Ken Task

Re: file problem after upgrade - update with debugging

by Jeff Geronimo -

Hi Ken and everyone else!

I'm having the same issues this morning with some profile pictures going missing and error messages stating "Can not read file, either file does not exist or there are permission problems" when some faculty are trying to edit content with files. We upgraded our production server from 2.3 to the latest stable build of 2.6 and running PHP 5.5 and Apache 2.4.10 on Ubuntu Server 12.04. Our moodledata directory is on the same production server, but I have an rsync process that automatically backs up the moodledata files to an external server for archival purposes.

I'm not seeing the contenthash name in our database that Ken suggested, and I'm not sure how else to fix this. Are these files simply lost? Or should I copy all of the backed up moodledata files from the external server back to the original moodledata folder (though I'm not sure this would work)?

Any help would be much appreciated!

Thanks,

Jeff

In reply to Jeff Geronimo

Re: file problem after upgrade - update with debugging

by Ken Task -
Picture of Particularly helpful Moodlers

Reponse to @Jeff via regular mail ... for those who might also need clarification/assistance ...

There are references in the DB for those profile pics that point to
filedir/XX/XX/somelongname.  Are you sure the permissions on moodledata/ is set such
that the apache user/group ... on Ubuntu that might be www-data:www-data ... can
read/write files/folders in /moodledata?

The content hash would be unique to each Moodle server ... not the same on ALL. If you were using the content hash in my response, that was for the other persons situation/server - not yours.

So what, exactly, does your error show?

Don't use a meat cleaver when a scalpel is needed!!!! The error you see reports the folders/files that can be found in your rsync'd backup of your moodledata. Using whatever tool you have to browse files in the rsync backup, find the contenthash named file, copy it out somehere to a file so you can check to see IF it is a f1.png (those are the profile pic file types/names). And then see IF one can open that file with some image editor. Reason for the test ... to assure the file itself is valid (not corrupt). IF it's a good file, one can copy the contenthash file back into the *production* moodledata/filedir/somesubdir/somesubdir/contenthash file name. Since you would be doing that as root user, make sure the owership/permissions are of the apache user/group of your system (again, think those to be www-data:www-data on your Ubuntu system). And if a subdirectory/folder cannot be found in that reported path, create it and set ownership/permissions to that of apache user group. Example path shows moodledata/filedir/XX/yy/somelongcontenthash and yy doesn't exist, make a 'yy' directory in XX. Then move the somelongcontenthash named file into that directory. Don't ask me why/how it happened ... can't do a mind meld with servers!!! ;) However, one potental snafu may have occurred ... that with the use of rsync. Someone may have tried to do exactly what you asked about doing ... never a good idea as the database information/meta data has un-doubtedly changed. DB info *must match* the physical filedir/XX/YY/longcontenthashnames.
Average of ratings: Useful (1)
In reply to Ken Task

Re: file problem after upgrade - update with debugging

by Ana Heyn -

Hi All.

We have been working in this issue quite intensively and it looks we finally fixed it.

The issue started after upgrading from 2.4 to 2.6 but it was intermittent which make it quite difficult to diagnose.


It looks like existing hash folders (in version 2.4) had permissions  drwxr-xr-x and if a new file had same 4 characters the file was uploaded to that directory  with .tmp extension which was causing the issue.

Hash file 95c7bd8b300c09ed90308a9d2734bda1e883854e
will be uploaded to existing folder
/moodlepath/moodledata/filedir/95/c7/ which had drwxr-xr-x

So the solution is to change the permission on folder and subfolders under filedir to drwxrwxr-x (chmod 774).

New folders created under 2.6 has 744 permissions already.

Since then, no  new .tmp files have been created. Also we needed to remove .tmp from all files.

Hope this helps.

Cheers.

Ana

Average of ratings: Useful (1)
In reply to Richard Kingsley

Re: file problem after upgrade - update with debugging

by Mark Findlay -

Hello Folks,

This has been an interesting thread...


We have seen similar issues after an upgrade from 2.4 to 2.6, however, in our case the issue seems to have been limited, and random in nature, BUT as a result of new resources being created. The behaviour however, is identical to that reported here by others.


In our case there seems to have been an issue somewhere in the upload process, and after some tracking of the resource location in the filestore, and the database, the issue seems to be that there is a mismatch between DB content hash, and the filestore content hash. Thy are the same except that in the filestore the file is stored as 'content hash.tmp'.


We have managed to recreate the issue on our DEV system by introducing the .tmp to the content hash in the file store, and removing it fixes the issue. However, before we start to remove all of the .tmp instances on our LIVE install we would like to know the mechanics of how the upload process works, and whether removing .tmp extensions on 'original broken' resources might have any unintended consequences?

Thanks in advance, we are relatively new to Moodle.

Mark.

Average of ratings: Useful (1)
In reply to Richard Kingsley

Re: file problem after upgrade - update with debugging

by Mark Pearson -

I have a similar problem which combined an update from 2.6 to 2.8 and a restore of Moodle production data into my sandbox server (just to test whether it would actually restore smile. Here's the actual problem and how I fixed it :

Home / ► Site administration / ► Appearance / ► Themes / ► More

More

Can not read file, either file does not exist or there are permission problems

More information about this error

Debug info: [dataroot]/filedir/41/cf/41cfeee5884a43a4650a851f4f85e7b28316fcc9


1. Ran the following in phpmyadmin : SELECT * FROM `mdl_files` WHERE `contenthash` LIKE '41cfeee5884a43a4650a851f4f85e7b28316fcc9' LIMIT 0, 25 ;

which extracted 9 records relating to the theme More.

2. Contents of containing dir :

filedir/41/cf % ls -lh

total 892

-rwxrwxr-x 1 theFN www 722K Dec 2 2013 41cf00ce91593edf51178a80de0279178615e66d

-rwxrwxr-x 1 theFN www 3.3K Apr 30 2013 41cf3a89a0900741fad47822af24bee0ca3020bd

-rwxrwxr-x 1 theFN www 118K May 18 2014 41cf8b2005e023d2d5d08a4264963e73263be24e

Notice that there's no file 41cfeee5884a43a4650a851f4f85e7b28316fcc9 which is what the error is complaining about.

3. How to fix? Had a brainstorm – Uninstall the More theme via Home / ► Site administration / ► Plugins / ► Plugins overview . This should remove any crap in the database. Since the code is still there when [Upgrade database] is clicked the records should be re-created & the files as well (in theory). So that is what I did.

4. WORKED. More is back and no errors.

Contents of filedir :

filedir/41/cf % ls -lh

total 900

-rwxrwxr-x 1 theFN www 722K Dec 2 2013 41cf00ce91593edf51178a80de0279178615e66d

-rwxrwxr-x 1 theFN www 3.3K Apr 30 2013 41cf3a89a0900741fad47822af24bee0ca3020bd

-rwxrwxr-x 1 theFN www 118K May 18 2014 41cf8b2005e023d2d5d08a4264963e73263be24e

-rw-rw-rw- 1 www www 4.3K Jul 10 12:45 41cfeee5884a43a4650a851f4f85e7b28316fcc9

Note change of permissions!

Did the same SQL select as above – extracted 16 records! All is good.

Hope this helps someone.

Mark

Average of ratings: Useful (1)
In reply to Mark Pearson

Re: file problem after upgrade - Migrated to a new server and files showing error

by Andrea Samadi -

Hello, thought I would post here incase I can find a faster solution.


I just migrated my moodle site to a new server, and get the error message when I try to click on any file.


I have asked my webhost (hostgator) to look again at the migration process, but is there something else I can do to fix this? I tried to go in and add the files back (I would do this if I could for a faster fix) but I cannot do this.


Any help, advice, tips? I'm hoping I can fix before students login.


Andrea Samadi



Attachment 2015-09-16_0535.png
Attachment 2015-09-16_0536B.png
In reply to Andrea Samadi

Re: file problem after upgrade - Migrated to a new server and files showing error

by Karrie V -

did you find an answer?  I just moved from hostgator as well and I'm having a similar issue.  

In reply to Karrie V

Re: file problem after upgrade - Migrated to a new server and files showing error

by Fernando Navarro Páez -
Picture of Testers

+1, from 2.5 to 2.8

In reply to Andrea Samadi

Re: file problem after upgrade - Migrated to a new server and files showing error

by Ken Task -
Picture of Particularly helpful Moodlers

The migration from one server to another ... what was old servers fully qualified domain name?

http://oldhost/

Is that different from the new location/hostname?

http://newhost/

IF this is the case, the URL's pointed to files in Moodle are built by taking the value of the wwwroot for the site thus in the DB one has http://oldhost/....blah/blah/file.   You can check this by placing mouse over the link and look at where it's going to retrieve that file.

When migrating to a new site/domain one must change all links to files.

So ... IF above is true ... do a dump of the DB (ie, get a backup of it before doing the following).  Might also research how to restore your DB should the following not work or you do this in-correctly.   The search and replace has to be 'exacting' and accurate or you'll mess up a bunch of stuff making it harder to fix.

There is a search and replace tool

http://newsite/admin/tool/replace/

You'll see a warning ... that's why the DB backup is important!

Search box: http://oldsite/ - make sure you include the full URL to the old site ... if site was running https then use https://oldsite/ ... make sure to include the trailing '/'.

Replace box: http://newsite/ or https://newsite/ - same thing include the full URL ... include trailing '/'.

That does a search and replace for URL's in a bunch of tables.

Once it finishes, navigate to one of those former errant links to see if that didn't fix it. ;)

This will not fix links one may have had in user created HTML blocks ... those you have to edit manually.

'spirit of sharing', Ken


In reply to Mark Pearson

Re: file problem after upgrade - update with debugging

by Matt Polaniecki -

I have the same problem after trying to switch over to another database.

1. I cloned MYSQL over to another database

2. I changed the info in config.php.

3. My courses and users were there, but I kept running into this error whenever I tried going to Site Home or to edit a course.

Debug info: [dataroot]/filedir/0a/32/0a3201a56aac42e796fd5e929bf0c2b8d85ce22d
Error code: storedfilecannotread
Stack trace:
  • line 579 of /lib/filestorage/stored_file.php: file_exception thrown
  • line 606 of /lib/filestorage/stored_file.php: call to stored_file->get_imageinfo()
  • line 55 of /theme/interactive/renderers/course_renderer.php: call to stored_file->is_valid_image()
  • line 83 of /theme/interactive/layout/frontpage.php: call to theme_interactive_core_course_renderer->new_courses()
  • line 1015 of /lib/outputrenderers.php: call to include()
  • line 945 of /lib/outputrenderers.php: call to core_renderer->render_page_layout()
  • line 112 of /index.php: call to core_renderer->header()

Any idea? I did the SQL call and found 8 records relating to 0a3201a56aac42e796fd5e929bf0c2b8d85ce22d


Should I just delete them? Please help!

In reply to Matt Polaniecki

Re: file problem after upgrade - update with debugging

by Ken Task -
Picture of Particularly helpful Moodlers

What see in DB is metadata, so rather than just removing records manually without really knowing what one is doing (never advised) think I'd investigate a little more about the file that's in the DB but does NOT appear to be in moodledata/filedir/

So go to your moodledata/filedir/

Look for a directory: 0a ... change into it.

In side that diretory look for another directory: 32 - change into it.

Now list to see if there is a file by the following name: (command: ls -l)

0a3201a56aac42e796fd5e929bf0c2b8d85ce22d

We don't know what it is but from errors appears to be front page related so ... got any images on the front page NOT generated by theme?

While in moodledata/filedir/0a/32/ issue the following command:

file -b 0a3201a56aac42e796fd5e929bf0c2b8d85ce22d

That will give some brief info about the file.  If that info says something about .jpg or other image info, see what it is.

cp 0a3201a56aac42e796fd5e929bf0c2b8d85ce22d /path_to_web_server/document/root/where_modle/code resides/some.jpg (or .png - whatever file info said it's file type was).

Then, using a browser, go directly to that image ... http://yoursite/some.jpg

Recognize the image?

That same image could be linked in any of the courses.  Then the fun begins to find the course that used it.

'spirit of sharing', Ken

In reply to Ken Task

Re: file problem after upgrade - update with debugging

by Matt Polaniecki -

Thank you so much for your help. In PHPmyadmin I was able to see that the errors were all related to the .jpgs i had set for the Course Summary files. Since "Site Home" and "Edit settings" all call those course summary files, I was getting scary errors on those pages.

The fix was I simply deleted the courses and then re-imported them. Another reminder of how important backups are!

Thank you very much for your help. I gotta bookmark this page so I remember these commands for the future.