Corruption of uploaded files

Corruption of uploaded files

by Md. Nazmun Sadat Khan -
Number of replies: 20
Problem:
Any binary files (like images or excel files) when uploaded in any course, have broken previews, and when they're downloaded gives corrupt files. Text files or other human readable files upload just fine and gives back the original files when downloaded. Oddly, PDF files aren't affected and zip files get converted into RaR files (although the extension doesn't change). I've uploaded one jpg file and when I downloaded it again and opened with a text editor, weirdly parts of it contained XML (.plist).

Interestingly, it seems in the moodledata/filedir/ folder, the original uploaded file gets stored without any problems. But the previews and the download options pointed toward the converted/corrupted files.

I can't make head or tails of the situation, especially since this problem started quite recently. Moodle used to be able to upload user submitted content perfectly before. What should I do?

Environment and Version:
Windows server
Moodle 2.6.11 (I know it's old, but I've no choice but to use this version at the moment. I'm planning on updating in near future)
Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Colin Fraser -
Picture of Documentation writers Picture of Testers

I would suggest that you plan on updating sooner rather than later. You could spend ages looking for the problem, find it, solve it and then update. Waste of time, I think. Just update now, adding in a fresh codebase. If the problem persists then look to your PHP version, 5.x or 7.x? You might want to update that as well, prepare for the time that you will need to be using that version of PHP. If the problem continues to exist, then likely something has bitten your server and that's outside my knowledge. Don't update in too big a step, go to v2.8, then v3.0, two version steps is likely to work best. 

BTW, there are likely to be others who would suggest don't update until you find the problem and fix it. They will offer good advice, and encouragement, and might even find a solution, but it is your decision.  


Average of ratings: -
In reply to Colin Fraser

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -
Unfortunately I cannot update it now. We don't host the servers and as a non-profit organization, it's hard to get funds. We're planning to get a better ISP, but at the moment there is no way for me to update.

As for the problem, like I said it's pretty weird, happened only a few days ago. Note that the original file still gets uploaded in the moodledata directory. You said that something might have 'bitten' my server? You don't mean to say malware?
Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

Incentives to upgrade?

https://www.cvedetails.com/vulnerability-list.php?vendor_id=2105&product_id=3590&version_id=158708&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=1&trc=70&sha=b42aa2f9c36f0252a20f73492f01d395cde20268

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Colin Fraser -
Picture of Documentation writers Picture of Testers

Interesting list Ken, is it a replay of the Moodle Tracker? 

Average of ratings: -
In reply to Colin Fraser

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

@Colin ...  are you informing me of Moodle Tracker, or what?

Fine ... for the OP.

https://tracker.moodle.org/browse/MDL-43655?jql=text%20~%20%22version%202.6%22

Same incentives, I would think.

As far as who post what where and when, I would hope 'responsible reporting' (got a 'finger wag' on that one recently) would mean Moodle Tracker first ... CVE second .... or automagically posted from Tracker to CVE.

Tracker does have more Moodle specific info ... and stuff like ... cannot replicate ... etc..

So it might be preferences of seeker?

'spirit of sharing', Ken




Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -

Thanks for the list. I realize that using an old version of Moodle is a security risk. We already have a plan for updating, but right now we have courses depending on the old version of Moodle that we cannot delay at all.

Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

Consider that the issue might be on the workstation you are using to access.

Same issues on other workstations using same OS and browser?

Got a Mac or know someone who does?   See what happens to them.  Know this might be far fetched, but how about a Linux OS workstation?

Resisting 'finger wag' concerning version as you have already stated you do need to upgrade. ;)

Does Windows still use registry and Reg Edit?

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -
I'm using a Windows PC to access the site, and all my colleagues do so too. The problem persists for all of them, and it's unlikely that the problem is just on my end. And although I do not have access to a Linux workstation, uploading any image files from Android still gives a broken preview and the download button points towards the corrupted file.


Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

In the past, it was not un-common to have issues with workstation registry.   Had to ask.

So if, one were to work right on the server, in the OS interface, NOT moodle, and find a file that was uploaded to moodle via whatever, does that file open correctly?

For that you'd need to make queries of your mdl_files table to find the humanly recognizable name and it's contenthash ....

contenthash shows path to the file in the sea of files in moodledata/filedir/

Example:

humanly recognizable name: mycool.doc or mycool.zip

contenthash has qfwqwehjgrwelqwedhfalslw

That file is located in moodledata/filedir/qf/wq/ and is named with the contenthash ... so the file doesn't have any extension on it.

Copy that file (qfwqwehjgrwelqwedhfalslw) out to desktop of the server, rename it for the humanly re-cognizable name (mycool.doc or mycool.zip)

and see if the server OS can open the file.

Don't run Windows anything anymore myself .... server/workstation ... so above is 'troubleshooting' and trying to find out where the issue might be.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

Also ... one other thought if using Apache ... what is apache setting for xsendfile and mime types?

Think you could see some of that info via phpinfo in admin -> Server -> PHP Info

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -

As I mentioned in the post, the original file does get uploaded. Like if I upload a PNG file say p.png with a content hash of abcdefghij, it does get stored in the moodledata/filedir/ab/cd/abcdefghij, which is exactly the same as the file I uploaded. But, the preview image in Moodle and its associated download button, gives me a different corrupted image. If I download the same image from the Moodle interface and call it say p2.png, it has a different hash, say efghijklmn. Looking at the moodledata dir again, there is another file in moodledata/filedir/ef/gh/efghijklmn. That's the issue here.

I don't know if it'll help, but the corrupted file and the original file always seems to have the same filesize. Looking at the one corrupted jpg file with a text editor, I notice the following:


Ǿˡƿ Ÿϊ۠ʶ̝Щ��j荺ݑ^?PٽȒޞliеı?ԳƷ}ۯ޴��ҋߜɠdɭ$?޲?㡈ңԁ튉ܕ!θ{ثغbw{]?[��οR뵕荶޾ĝU    2}ͳXಹ_ڃޜL{1+ɣlև뫰~n#��Ѯ݊У,Ǻκjףa����5?߻5Փم(ǡUࠇ%ؙٖ6��ܴ#۽˭ߡȿ ҚΚʦʛѽ?��?䀲IεۣuSȳևࠪ(̥#fˤcʍ Ɣȹ5ɟ?��?м۩vޘ:cE؈ϢɉϘӚϟɺ ׾vև㴨����¯Ƞh$Ә����Ƶ۝\    ㋝Ȯ�%EZw��Aɐhotoshop 3.0 8BIM         Untitled-2a 8BIM%     ԂrW.G΅Ჟ湂IMꠠ  ��ml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.print.PageFormat.PMHorizontalRes</key>
    <dict>
        <key>com.apple.print.ticket.creator</key>
        <string>com.apple.jobticket</string>
        <key>com.apple.print.ticket.itemArray</key>
...

That plist shouldn't be here in a jpg file format. Don't know where or why Moodle put this in the file.

Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

If files were uploaded by a mac user .plist might be there.  Looks like photoshop header.

Got any apple products (applications) installed **on the server**?   On the workstation?

mysql> select contenthash,filename,userid,mimetype from mdl_files where filename like '%.jpg';

Example of what is returned:

| 41cfeee5884a43a4650a851f4f85e7b28316fcc9 | background.jpg                                        |      2 | image/jpeg |

So if I go to the moodledata/filedir/41/cf/ directory do I see a 41cfeee5884a43a4650a851f4f85e7b28316fcc9 file?

And if I copy that file to desktop *ON THE SERVER* and rename it to background.jpg can I open the file *ON THE SERVER*?

Find out who is user id is and if they are using photoshop.

What php extensions do you have installed? and version of PHP?   User pics uploaded by users are converted to two images .... sized to fit and a smaller version for display in certain areas of Moodle. Do those user pics look ok for a user?

PHP info page will show php info.

Agree this is strange ... don't know that anyone has ever posted anything similar. :\

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -

Pretty sure there's no apple product installed on the server. As part of the course work, we do have students submitting images using Adobe Illustrator, but they submitted only the final images. And yes, when  you upload a file, you do have the original file stored according to its checksum and when you rename it, it does open on the server and it displays the original image.

But the preview image looks broken. From the Moodle interface, you get a broken image and the download option points towards the broken file, which has a different checksum stored in a different folder in moodledata/filedir.


Really strange behaviour from Moodle. Haven't got the slightest clue why this is happening.

Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

There is no column in mdl_files for 'checksum' ... there is a size column.

Did you do the example query on your DB?  Here's another query:

mysql> select contenthash,filename,filearea,filepath,userid,mimetype,filesize from mdl_files where filename like '%.jpg';

Run it ... if you know the name of the .jpg file change the ending of the query to the actual humanly recognized filename.

Explain what you mean by 'preview' .... and if the downloaded file has a different contenthash have you tried the same test ON THE SERVER with the 'different' contenthashed file?  

There is a preview image and a full image file? How are students uploading files?  I take it this is in assignments?   In site admin menu, do you have a tool to convert assignments?

Adobe illustrator ... that plist file didn't show that!

I take it that your cron job is running ok and frequently.

And I asked about PHP and extensions?  Are you missing any?

'spirit of sharing', Ken



Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -


Let me provide some screenshots. Below is the screen for the course
Course page
Say I'd like to add the following image as a resource:
Smiley, before the glitch
(Checksum SHA-1: 58b8d77aac35c792ea14f9b359be35eee3a54707)

I click on the add activity link, check the File radio button on the left, and I try to upload this smiley1.jpg
File Picker
and here's what I get:
Uploaded file has no preview
In the Content pane, I can clearly see that 'smiley1.jpg' has been uploaded but for the preview is broken. So I click on and try to download it.
Say I downloaded the file as smiley2.jpg.

Now smiley2.jpg will come out as broken, with a different checksum (325f199b5d38eb495730c8a1ec6d824b5aa9124a).

Now when I look inside the moodledata directory, I find that only
moodledata/filedir/58/b8/58b8.... exists, which is the original unmodified smiley.jpg and it opens just fine when I name it such. There is no moodledata/filedir/32/5f/325f..... corresponding to the
smiley2.jpg.

The weirdness doesn't end there, for some reason PDF files are exempt from this strange behaviour, as are html, css, js files (and possibly all text files). Also when I upload a zip archive, say compressed1.zip, and download it again as compressed2.zip from the Moodle interface as before, compressed2.zip doesn't open as a zip archive. But if I rename compressed2.zip as compressed2.rar, it CAN be opened, with all the original files inside it intact. Like before, the uploaded compressed1.zip exists on the server, but the compressed2.rar doesn't, suggesting that the conversion takes place on requesting the original file.


Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

Are we still confusing 'checksum' with 'contenthash' .... they are not the same.

The contenthash value stored in the DB is NOT a checksum .... it's just an obscured path and file name in moodledata.

Think the issue with the .zip file is on the workstation end.  Registry?  Have never heard of Moodle converting an uploaded .zip to a .rar.   That would mess up things like SCORM's in Moodle.  You must have WinRAR or something else installed on the workstation that does that 'automagically'.

What happens when you access that course/file using a smartphone .... like an Android ... and you tap on the link to the .zip file?   Does it download as a zip?

Ken





Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Md. Nazmun Sadat Khan -

Running the SQL query you provived gives me the following output

SQL result

Host: 204.93.172.30:3306
Database: cmtphys_moodle
Generation Time: Jul 01, 2018 at 08:41 AM
Generated by: phpMyAdmin 3.3.1 / MySQL 5.1.66-community
SQL query: SELECT contenthash,filename,filearea,filepath,userid,mimetype,filesize from mdl_files where filename like 'smiley1.jpg' LIMIT 0, 30 ;
Rows: 2

contenthash filename filearea filepath userid mimetype filesize
58b8d77aac35c792ea14f9b359be35eee3a54707 smiley1.jpg draft / 4 image/jpeg 7095
58b8d77aac35c792ea14f9b359be35eee3a54707 smiley1.jpg content / 4 image/jpeg 7095

Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

See the filearea the DB returned .... 'draft'.   Means it's not quite there all the way yet.

Rather than worry about the preview, go ahead and SAVE.

Make sure the cron job runs.  You'll have to research how to do that one windows command line ... if on linux, one could cd admin/cli/ then issue: php cron.php

In the output of running cron, look for anything related to files.   Since 2.6 is soooooo old there is much that has changed in the running of cron so I cannot provide an example of what it would look like to you as that version.

After cron job runs, that 'draft' should change to things like 'section' or 'content'.  Moodle then has been given time to gen a preview if that version actually gen's a preview.   That preview might be coming from .js code.

IF, after running cron do the DB query again to see if that 'draft' status has changed to something else ... 'section', 'content', whatever.  There really should be only one reference to that file.  Your query shows 2 .... one draft and one content!!!!

????  https://moodle.org/mod/forum/discuss.php?d=185839

Site might have hit something that can only be fixed by upgrading.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Corruption of uploaded files

by Ken Task -
Picture of Particularly helpful Moodlers

Correction ...

The query will show two rows ... the contenthash is the same so that means there is only ONE file in the meta data.

At some point in the future, draft is no longer be true as you've not edited the file link created and that should/would disappear leaving on the row that has 'content'.

Add filesize to the query .... that value should be the same ... it's the same file.   There is no 'preview' sized image in the DB.   That only shows when in the process of editing or adding the resource.

What's important is not the  preview that might have been trying to acquire via right click ... but the actual link to the resource in the course.

'spirit of sharing', Ken


Average of ratings: -
In reply to Md. Nazmun Sadat Khan

Re: Corruption of uploaded files

by Dasu Gunathunga -

I faced the same issue with Moodle 3.5 and filemanager field. Image preview has got corrupted for some reason only for some occasions. When I cleaned the preview file record in database, it regenerated without any issue. However I need to clean records in database using following sql,

// Get contenthash   

select contenthash from mdl_files where itemid='688186689';


// Get preview - filename is the contenthash from previous sql

 DELETE from mdl_files where filename='53002d018e1ac3a82326882386f9e5186a34f730' AND filearea='preview' AND filepath='/thumb/';


Average of ratings: -