Submissions in Moodle assignments are corrupted upon download

Submissions in Moodle assignments are corrupted upon download

by Richard Flurey -
Number of replies: 13

Hi folks,

We have recently moved to Ubuntu 14.04 LTS and installed Moodle 2.7.1 and discovered during the migration of our assignments that all downloads from assignments are corrupted when using the pluginfile.php method. This also seems to apply newly created assignments too. However, if a submission is in draft mode and using the file chooser to download the file then it works and there is no file corruption. It seems that a blank space (hex code 20) is added to the beginning of every file, thus corrupting the download.  

We can still retrieve the files directly from the server once we query the database to find the actual contenthash name of the file, then take a copy and rename it accordingly.

I've seen a number of posts reporting similar issues throughout various versions of Moodle v2 and v1.9 with different operating systems but with no definitive answer. Has anyone else encountered this on Ubuntu 14.04 and know specifically how it might be fixed? I am doubting it has anything to do with white space in the config.php for example since it is a brand new installation and it looks normal.

We also have a copy of Moodle 2.7 running on Debian 6.0 and assignments work just fine there. It is currently running PHP 5.4.28 whereas Ubuntu is running PHP 5.5.9 with Zend Opcache, which I have a slight suspicion might be part of the problem?  


Regards,

Richard

Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

I dont think it is Ubuntu per se, simply because you said it works on Debian, and I have Mint17 on my localhost testbed, which also works fine and as Debian > Ubuntu > Mint it does seem unlikely that Ubuntu would cause issues that are not present in either Debian or Mint.  I realise thats a simplistic view of the distro process, but its indicative that the issue lies in a specific component such as php rather than the distro itself (yes I know that a distro is basically the sum of all those component parts, but it would be useful to determine the component rather than just the whole distro - if that is the issue smile ).

Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by Martin Arman-Addey -

Hi Richard,

We are also running with 2.7.1 on Ubuntu and have noticed the same error when retrieving word documents from assignment and their being corrupted.

I only just found this yesterday, I have also tested this on our old 2.3 system and found the same issue.

If there is anyone out there who has a fix, it would be greatly appreciated.

Average of ratings: -
In reply to Martin Arman-Addey

Re: Submissions in Moodle assignments are corrupted upon download

by Richard Flurey -

Hi Martin,

I finally got to the bottom of the issue and figured out a workaround myself. It's nice to get to the point where you can answer your own questions. I know the problem isn't with the flavour of Linux per se (as lecturered to us by Richard) but I mentioned that we recently moved to Ubuntu 14.04 and on the latest Moodle just to see if anyone else has a similar experience with the same file corruption problem in assignments. It looks like we are not the only one!


I suspected the problem had something to do with caching and at first I thought it might be the Zend opcache. However, I turned this off during a quiet moment and restarted Apache and this made no difference. Then I had a sudden brainwave pointing me in the direction of output_buffering, a setting in php.ini which I'd left untouched.


I know in the past this setting is preferable to be turned off but what I figured about php 5.5 is that it's often set to 4096 by default. I tried turning it off and it did make some difference to the way files download from assignments. Unfortunately, it broke the forcedownload mime type functionality and loaded the files into the browser but I did notice the extra space in the front of the file header had disappeared. It might have solved the corruption issue but how to download the file? So I tried different output_buffering settings which made no difference and put it back to the default of 4096.


At this stage I reckoned it had to be enabled but there must be another way to prevent the corruption in the buffering. Now this behaviour does seem specific to the updated patch/release version of PHP 5.5.9/Apache 2.4 used within Ubuntu 14.04 since the corruption doesn't seem to occur in other PHP/Apache versions being used by other Linux distributions. After doing a bit more searching online I found the following suggestion which worked a treat when I figured where it was best placed in the Moodle code.


Find the following 2 lines in /lib/filelib.php just after (line 1977 in Moodle 2.7.1):

function readfile_allow_large($path, $filesize = -1) {
    // Automatically get size if not specified.


Add the following 2 lines directly after the lines above:

ob_clean();   // discard any data in the output buffer (if possible)
flush();      // flush headers (if possible)

Make sure you take a backup copy of the filelib.php of course and be sure to flush the opcache, or simply restart Apache if you able to. Once a new copy of filelib.php is running in the cache it's good to go. I was able to download ZIP files from the assignments and open them with no problem with no leading spaces before the 'PK' file header that previously stopped them from opening. Same applies to all the other file types that appeared corrupted because of the leading white space.


I hope this helps you and potentially others too.


Cheers,

Richard

Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

My sincere apologies if the tone of my post came across as 'lecturing' anyone - that was never my intention sad ! I was simply trying to help and point out my experiences with another member of the Ubuntu family of distros.

Average of ratings: -
In reply to Richard Oelmann

Re: Submissions in Moodle assignments are corrupted upon download

by Richard Flurey -

No worries Richard, I was being a little bit sarcastic. I could guess from your reply that you must have been a teacher, hence the 'lecturing' quote, without even looking at your profile smile   I have worked with academics for about 18 years and used to them coming to me for help, it is not often I call out for help but I was stumped.


I was just hopeful that people here on Moodle forums might come back with possible techie solutions rather than a discussion of Linux distros. It's all good now, my bosses will be pleased that everything is back in working order and I can resume the mountain that is my outstanding workload waiting for me on Monday.


Apologies for my bluntness, I guess I can be a straight to the point kind of person.


Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by dan attwood -

I've been working with Martin on this as we were seeing the same problem.

This is moodle 2.7 on ubuntu 14.04.

I've just applied your fix and it seems to be working fine now.

Has there been a tracker issue raised for this?

As this is a fix that requires a fix to core code I think someone at Moodle HQ needs to be aware of this and review it for inclusion into Moodle.

If you post a link to the tracker issue here we'll add our comments to it.


Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by Martin Arman-Addey -

Hi Richard,

You will have seen the post from Dan. Thanks very much for the information, and the coding. That has worked a treat.

All the best

Martin

Average of ratings: -
In reply to Martin Arman-Addey

Re: Submissions in Moodle assignments are corrupted upon download

by Richard Flurey -

Martin/Dan,

I didn't think the issue was necessarily a 'bug' , more of a failing of Ubuntu 14.04's version of PHP to handle the output buffer cleanly, so I thought it best to post it on here instead. 

I suppose it could be seen as a feature request or enhancement?

Also, after applying this patch I noticed files over 10Mb are still affected and checked filelib.php again and found the culprit:


if ($filesize > 10000000) {

        // for large files try to flush and close all buffers to conserve memory

        while(@ob_get_level()) {

            if (!@ob_end_flush()) {

                break;

            }

        }

    }


It seems that after flushing the buffer, it causes the corruption issue to arise again. Therefore, I simply decided to comment it out and all files including much larger files (under 2 Gb) downloaded and opened successfully. I'm not sure how critical it is to flush the buffer since it's probably being re-used with each new request and the default buffer size isn't particularly large so I'm sure most servers are not sending files simultaneously in split-seconds and can handle it?

I suppose it might still be worth making a bug report and see if it is given any consideration to be factored into the file handling subroutines when releasing new builds of Moodle.

Out of interest, can you output the results of 'php -v' from the command line and share your version of PHP and we can see if our versions match?


Cheers,

Richard

Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by dan attwood -

PHp version infomation below:

PHP 5.5.9-1ubuntu4.4 (cli) (built: Sep  4 2014 06:56:34)

Copyright (c) 1997-2014 The PHP Group

Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies

with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies


Martins will be the same


I would say that this is definitely a bug, as to if it's Moodles or Ubuntu fault then that that's less clear.

Given that your fix works though and other Moodles will be running atop Ubuntu it's probably worth highlighting it to HQ regardless. 






Average of ratings: -
In reply to Richard Flurey

Re: Submissions in Moodle assignments are corrupted upon download

by Feng Sun -

Can anyone help our problem that an online assignment submission set up for students to load the file before the due date, but Moodle will not allow student to load the file? There is still two hours left to the due date and cut off date, Moodle shows the message at the submission status area with the message of "This assignment is not accepting submissions".

Thanks.

Average of ratings: -
In reply to Feng Sun

Re: Submissions in Moodle assignments are corrupted upon download

by ben reynolds -

Check that "allow submissions from" is not the same as the due date.

Average of ratings: -
In reply to ben reynolds

Re: Submissions in Moodle assignments are corrupted upon download

by Mary Cooch -
Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

I did a forum search for the terms "This assignment is not accepting submissions" and came up with some useful posts, one or more of which I am sure will put you on the right track.

Average of ratings: -