Onedrive file size limit?

Onedrive file size limit?

by Simon Lewis -
Number of replies: 1

Moodle 3.9.6 / PHP 7.4

Hello,

I've set up onedrive as a repository to use for file submissions on assignments. And it's mainly working fine except, if I try and upload anything over 250MB it fails. 

The logs point to this function, and when I've outputted the data, it seems to be the error/lack of response from onedrive returning 'error' here: 

$response = $curlinstance->put($created->uploadUrl, $options);
returns: {"error":{"code":"invalidRequest","message":"Invalid request"}}

 /repository/onedrive/lib.php

protected function upload_file(\repository_onedrive\rest $service, \curl $curl, \curl $authcurl, $filepath, $mimetype, $parentid, $filename) {

        // If that fails, try an upload with the auth headers (will work for work onedrive accounts).

        $curls = [$curl, $authcurl];

        $response = null;

        foreach ($curls as $curlinstance) {

            $curlinstance->setHeader('Content-type: ' . $mimetype);

            $size = filesize($filepath);

            $curlinstance->setHeader('Content-Range: bytes 0-' . ($size - 1) . '/' . $size);

            $response = $curlinstance->put($created->uploadUrl, $options);

            if ($curlinstance->errno == 0) {

                $response = json_decode($response);

            } .......

        if (empty($response->id)) {

            $details = 'File not created';

            throw new repository_exception('errorwhilecommunicatingwith', 'repository', '', $details);

        }


        }
     return $response->id;
}


Like I saw, under 250MB there is a valid response:

{"@odata.context":"...... ","@content.downloadUrl":"longstring &ApiVersion=2.0","createdBy":{"application":{"id":"973f7419-4dd0-48a7-b2c2-2a73121a696e","displayName":"vledev2122"},"user":{"email":"S....@lcm.ac.uk","id":"....","displayName":"Lewis, Simon"}},"createdDateTime":"2021-06-25T13:34:49Z","eTag":"\"{5977D4B5-9856-488B-B45A-3D706D4A7669},3\"","id":".....J","lastModifiedBy":{"application":{"id":"973f7419-4dd0-48a7-b2c2-2a73121a696e","displayName":"vledev2122"},"user":{"email":".....@lcm.ac.uk","id":".....","displayName":"Lewis, Simon"}},"lastModifiedDateTime":"2021-06-25T13:34:50Z","name":"myportfolio1-2.zip","parentReference":{"driveId":".....","driveType":"business","id":"0....","path":"....._id_1372/Assignment+onedrive+ass+-+1st+dec_id_116678/assignsubmission_file/submission_files/154785"},"webUrl":"...","cTag":"\"c:{.69},2\"","file":{"hashes":{"quickXorHash":"..."},"irmEnabled":false,"mimeType":"application/zip"},"fileSystemInfo":{"createdDateTime":"2021-06-25T13:34:49Z","lastModifiedDateTime":"2021-06-25T13:34:50Z"},"size":509993} [] []

Our PHP max file upload is 2G.

Is there a file submission limit for onedrive? I can't find one anywhere and not sure if the issue is within moodle or one drive settings. Or is there a size limit for cURL?

thanks in advance,

Simon


Average of ratings: -
In reply to Simon Lewis

Re: Onedrive file size limit?

by Rob Davenport -
Hi Simon,

We have been testing the OneDrive Repo recently and have found the exact same issue (and confirmed with Microsoft) when using "Access Controlled Links". Our host provided this explanation, we may be going back to them soon to see if any development work is possible to solve, as Microsoft themselves don't appear too interested in supporting any of their own plugins for Moodle:

______________________________________________________ SNIP________
The error you have noticed is directly caused by uploading a sizeable file and the curl (data transfer protocol) process exceeds the php execution timeout default of 30 seconds. The particular logic of this is detailed here https://docs.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0, but the relevant detail states: 

Upload bytes to the upload session

To upload the file, or a portion of the file, your app makes a PUT request to the uploadUrl value received in the createUploadSession response. You can upload the entire file, or split the file into multiple byte ranges, as long as the maximum bytes in any given request is less than 60 MiB.


The fragments of the file must be uploaded sequentially in order. Uploading fragments out of order will result in an error.


Note: If your app splits a file into multiple byte ranges, the size of each byte range MUST be a multiple of 320 KiB (327,680 bytes). Using a fragment size that does not divide evenly by 320 KiB will result in errors committing some files.



Large files are liable to not meet these requirements. Previous efforts with another client to bump the default time window up also did not work because you cannot increase the php timeout on various location/functions and have it take any effect that matters with this kind of problem. The support tech that worked that case also mentioned this: 

The Curl call request is made from the lib/filelib.php script ("function request") and the script encounters the php execution timeout before the Curl request can send a response back... The response is then interpreted as empty hence the exception error    


In other words,  the large amount of data being sent is timing out before the destination can send back receipt confirmation of said data.

This appears to be a somewhat unavoidable issue when uploading large data files via this method to OneDrive.
____________________________________END SNIP____________________

Hope that helps. Cheers, Rob.
Average of ratings: -