Simple proxy script workaround for Moodle's file API

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Number of replies: 11
Picture of Plugin developers

Hi Matteo,

Thanks again for all your help.

I've implemented some the of the recommendations you made and it seems to be working. However it's forcing file download when I call /lib/filelib.php::send_file() Here's the code I'm working with now:

require_once('../../config.php');
require_once('../../lib/filelib.php'); // contains mimeinfo() and send_file() definitions

require_login(); // Users must be logged in to access files

global $CFG;

// If no value is provided in $_GET, we get an empty string.
// See: http://ca2.php.net/manual/en/reserved.variables.get.php
if($_GET['file'] === '')
{
    header('HTTP/1.0 404 Not Found'); // Send back a 404 so that apps don't wait for a timeout
    echo '404 Error: File not found';
    die;
} else {
    // Strip out special characters, extra slashes, and parent directory stuff
    $swf_disallowed = array('../','\'','\"',':','{','}','*','&','=','!','?','\\','//','///');
    $swf_replace = array('','','','','','','','','','','','','/','/');
    $swf_file_path = str_replace($swf_disallowed,$swf_replace,$_GET['file']);
}

// Make sure all the condtions are met before attempting to serve file:
// must have a "." within 5 chacters of the end
if(strrpos($swf_file_path,'.') > strlen($swf_file_path) - 6)
{
    $swf_file_array = explode('/',$swf_file_path);
    $swf_file_name = $swf_file_array[count($swf_file_array) - 1]; // get file name to serve
    // Build full path to file e.g. /home/sites/example.com/moodledata + /repository/swfcontent/ + videos/myvideo.mp4
    $swf_file_path = $CFG->dataroot.$CFG->swf_content_dir.$swf_file_path;
    $swf_mime_type = mimeinfo('type', basename($swf_file));
} else {
    header('HTTP/1.0 404 Not Found');
    echo '404 Error: File not found';
    die;
}

send_file($swf_file_path,$swf_file_name,'default',0,false,false,$swf_mime_type);

Any help? Thanks in advance smile

In reply to Matt Bury

Re: Simple proxy script workaround for Moodle's file API

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Matt,
that's strange since you've put false in $forcedownload.
Could you share how the HTTP Headers of both your Request and Response look like? You can access them through e.g. Fiddler as well as the F12 Dev Tools of your browser.

Matteo

P.S.: FYI, send_file_not_found() send HTTP 404 in a nice way (die management included) i.e. a response with both headers and human readable body.

In reply to Matteo Scaramuccia

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Picture of Plugin developers

Hi Matteo,

Re using send_file_not_found(), the requests are being sent and and responses received by Flash clients so HTML formatting is redundant and can cause problems for Flash developers who don't know how to handle unexpected HTML responses, as well as existing Flash apps, e.g. JW Player, that just don't bother and throw a fatal error.

Also, I've noticed that mimeinfo('type', $swf_file_name); returns video/quicktime for .mov files. Not a big deal; MP4, M4V, and MOV are almost identical media containers; but it should work better in Flash with video/mp4.

It's working in the browser now, thank you (I think it was a caching issue). However, the Content-type headers appear to be wrong, they're always text/html; charset=utf-8. I've tested them both with Strobe and JW Players. The first one (my script) doesn't work but the second one (Moodle API) does. I think perhaps the ? query string is confusing the players. Do you know of a way to fix that? e.g. just use slashes?

Thanks again!

Here's the response headers for http://localhost/m2/mod/swf/content.php?file=video/subdir/video.mp4 (doesn't work in Flash)

Date: Mon, 08 Jul 2013 16:35:41 GMT

Server: Apache/2.2.21 (Win64) PHP/5.3.10

X-Powered-By: PHP/5.3.10

Set-Cookie: MoodleSession=8519qr467t8e9mvc9fj0rmkti2; path=/m2/

Cache-Control: private, pre-check=0, post-check=0, max-age=0

Pragma: no-cache

Content-Language: en

Content-Script-Type: text/javascript

Content-Style-Type: text/css

X-UA-Compatible: IE=edge

Accept-Ranges: none

X-Frame-Options: sameorigin

Keep-Alive: timeout=5, max=99

Connection: Keep-Alive

Transfer-Encoding: chunked

Content-Type: text/html; charset=utf-8


200 OK

And here's the response headers for the same file served through the regular file API: http://localhost/m2/pluginfile.php/137/mod_swf/content/0/video.mp4 (does work in Flash)

Date: Mon, 08 Jul 2013 16:41:36 GMT

Server: Apache/2.2.21 (Win64) PHP/5.3.10

X-Powered-By: PHP/5.3.10

Set-Cookie: MoodleSession=spi38euufu2n8ov3rlchqvc8k2; path=/m2/

Cache-Control: private, pre-check=0, post-check=0, max-age=0

Pragma: no-cache

Content-Language: en

Content-Script-Type: text/javascript

Content-Style-Type: text/css

X-UA-Compatible: IE=edge

Accept-Ranges: none

X-Frame-Options: sameorigin

Keep-Alive: timeout=5, max=99

Connection: Keep-Alive

Transfer-Encoding: chunked

Content-Type: text/html; charset=utf-8

In reply to Matt Bury

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Picture of Plugin developers

Hi again Matteo,

You've been a great help and I've got it to play videos through media players. I've experimented and tried a few things out and in the end, this is what works:

require_once('../../config.php');
require_once('../../lib/filelib.php');

require_login();

global $CFG;

$swf_relative_path = get_file_argument();
$swf_data_path = $CFG->dataroot.$CFG->swf_content_dir.$swf_relative_path;
$swf_data_info = pathinfo($swf_data_path);
$swf_mime_type = mimeinfo('type', $swf_data_info['basename']);
send_file($swf_data_path,$swf_data_info['basename'],'default',0,false,false,$swf_mime_type,false);

The paths look like: http://localhost/m2/mod/swf/content.php/video/test/video.mp4 It looks like the /content.php?file=etc. bit was confusing the media players after all.

However, I don't recommend serving large video files via this method. It seems to use up a lot of memory for just me playing one video at a time (165MB and 258MB each). I reckon it'd only take a small number of concurrent users to overload a server. I think it'd be better to use a media server to stream the files instead of using progressive download/pseudo streaming.

Since the files are intended for Flash clients, not download, and I don't need to tell users how big the files are, would it be possible to get some performance gains by not calculating header('Content-length: '.(string)filesize($swf_data_path)); ? Is there a way to do that?

Many thanks Matteo!

Matt

In reply to Matt Bury

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Picture of Plugin developers

Final question: What security measures do I need to take with the above script? Are there any possible exploits with it, like accessing outside of /moodledata/repository/swfcontent/ or does the code I've used already check for that?

Matt

In reply to Matt Bury

Re: Simple proxy script workaround for Moodle's file API

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Matt,
glad to read that you've been successful: PATH_INFO is the way to go even when players do not trust the HTTP Headers and look and the path name to get some informations.

  • About file size:
    • reading the file size should be a fast call being served by the OS file system API. If you want your system to be efficient in serving big media files please stick with Files API and exploit the X-Sendfile opportunity: it gives your web server - not all the web servers are able to do it - the capability to serve the file on behalf of PHP, saving a big amount of memory per request; unfortunately there's no documentation in Moodle Docs yet and you need to look at config-dist.php as well as e.g. in some notes here. Yes, a media server is the right solution but this will increase the complexity of your deploy: maybe you could add an option to support it but, I suggest, in a second release wink
    • If you omit that information, your client (here the player) will never know how much content should be served and it will end up with "strange" behaviors;
  • About MIME types: you should increase the MIME mapping coded in Files API; I'll look at it;
  • About security: I'll give a better look at your final code one of the next evenings. The key point is if you want your files to be not in the Moodle Files Pool, to get benefits from having a separate real folder to upload them e.g. via FTP: outside of the poll, security must be managed by your code both to avoid traversal path browsing - and give a malicious user the possibility to browse any folder of your server - as well as giving access to those e.g. already logged in, as part of the auth logics, required even for the pool.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Picture of Plugin developers

Thanks again Matteo,

Users will also be able to deploy videos via the standard Moodle filemanager API. The main reason for having the proxy is to use consistent, predictable file URLs that can be referenced in SMIL and XML files and that can be transported easily from one Moodle (or other platform) installation to another without having to rewrite the URLs, which can be hundreds or even thousands of images, animations, audio, video, and captions. In these cases, videos are typically short and small and organised into playlists. Support for .m3u (https://en.wikipedia.org/wiki/M3U) playlists (provided by StrobeMediaPlayback.swf) will be helpful for those who are shy of writing/editing more complicated playlist formats and some desktop media players/libraries offer easy WYSIWYG GUIs to drag'n'drop and organise them. .m3u doesn't seem to be listed in Moodle's MIME types array (audio/x-mpegurl).

# This is a simple playlist.
http://mediapm.edgesuite.net/osmf/content/test/manifest-files/progressive.f4m


http://mediapm.edgesuite.net/strobe/content/test/AFaerysTale_sylviaApostol_640_500_short.flv

Save it with the .m3u file extension and you're good to go. Easy huh?

Re MIME types, I think the video/quicktime for .mov files makes it more compatible with HTML video source tags, prompting a download instead of just failing in some browsers. Seems to be a logical choice to support HTML over Flash in these cases. The same thing happens with .smil files because Moodle sets the MIME type to application/smil. If it's text/xml, browsers without SMIL support (most) just read it as XML and it works fine (no need to download and then open it with a text editor). These days SMIL is mostly used for playlists, captions (TimedText/RealText is a subset of SMIL), and automatic adaptive bandwidth switching files. If it's used for multimedia presentations, more than likely it'll be through a Flash, Java, or Javascript app which shouldn't make a difference.

BTW, these days it seems best to simply add direct download links and let users choose their own media players for maximum compatibility/reach. Where Flash support isn't available, deploying video's like taking a time machine back to the year 2000.

Anyway. getting too carried away with the possibilities...

Thanks for helping to make a kick-ass multimedia module for Moodle! smile

In reply to Matt Bury

Re: Simple proxy script workaround for Moodle's file API

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Matt,
here is my proposal, based on both Moodle patterns and your logics:

require_once('../../config.php');
require_once('../../lib/filelib.php');

require_login();

global $CFG;

// (Optional) Clean the SWF content directory setting.
$CFG->swf_content_dir = clean_param($CFG->swf_content_dir, PARAM_PATH);
// Remove trailing slash(es).
$CFG->swf_content_dir = rtrim($CFG->swf_content_dir, '/');
// Get the relative path of the requested content.
$swf_relative_path = get_file_argument();
if (empty($CFG->swf_content_dir) || !$swf_relative_path) {
    print_error('invalidargorconf');
} else if ($swf_relative_path{0} != '/') {
// Relative path must start with '/'.
    print_error('pathdoesnotstartslash');
}

$swf_data_path = realpath($CFG->dataroot . $CFG->swf_content_dir . $swf_relative_path);
// (Paranoid) Content will be served just from the SWF content dir.
if (strpos($swf_data_path, realpath($CFG->dataroot . $CFG->swf_content_dir)) === 0) {
    $swf_data_info = pathinfo($swf_data_path);
    $swf_mime_type = mimeinfo('type', $swf_data_info['basename']);
    send_file($swf_data_path, $swf_data_info['basename'], 'default', 0, false, false, $swf_mime_type, false);
} else {
    send_file_not_found();
}

Regarding with MIME types, you can optionally change the Moodle ones using a switch to force some of them.

Besides, since you're developing a module keep care of providing $plugin->component (details here: http://docs.moodle.org/dev/version.php ) to let users receive an error in case they'll put your code in a different folder.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Simple proxy script workaround for Moodle's file API

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Matt,
since you like a simple HTTP 404 with no payload I missed to give you that option. You can do that in the Moodle way as sampled below:

} else {
    send_header_404();
die;
}

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Picture of Plugin developers

Hi Matteo,

Thanks again.

I'm putting together some demos that show a few of the things the new SWF Activity Module can do (a work in progress and pretty incoherent so far ). You can see how it is here: http://m2.matbury.com/course/view.php?id=3

To recap, the mod/swf/content.php code I'm experimenting with now is pretty much what you've suggested, but with different error reporting:

require_once('../../config.php');
require_once('../../lib/filelib.php'); // for mimeinfo() and send_file()

require_login(); // Users must be logged in to access files

global $CFG;
    
// (Optional) Clean the SWF content directory setting.
$CFG->swf_content_dir = clean_param($CFG->swf_content_dir, PARAM_PATH);
// Remove trailing slash(es).
$CFG->swf_content_dir = rtrim($CFG->swf_content_dir, '/');
// Get the relative path of the requested content.
$swf_relative_path = get_file_argument();
if (empty($CFG->swf_content_dir) || !$swf_relative_path) {
    header('HTTP/1.0 404 Not Found');
    exit('404 Error: File not found. SWF settings and/or path to file is/are not set correctly.');
} else if ($swf_relative_path{0} != '/') {
    // Relative path must start with '/'.
    header('HTTP/1.0 404 Not Found');
    exit('404 Error: File not found. Relative paths must start with a /');
}

$swf_data_path = realpath($CFG->dataroot . $CFG->swf_content_dir . $swf_relative_path);
// (Paranoid) Content will be served just from the SWF content dir.
if (strpos($swf_data_path, realpath($CFG->dataroot . $CFG->swf_content_dir)) === 0) {
    $swf_data_info = pathinfo($swf_data_path);
    $swf_mime_type = mimeinfo('type', $swf_data_info['basename']);
    send_file($swf_data_path, $swf_data_info['basename'], 'default', 0, false, false, $swf_mime_type, false);
} else {
    header('HTTP/1.0 404 Not Found');
    exit('404 Error: File not found');
}

Love the "Paranoid" bit smile

Re: error reporting, I'm going to think about a human and Flash friendly way to manage that. Pure text that is meaningful to both users and Flash, and that Flash apps can display when things go wrong... maybe with language support?, i.e. get_string('filenotfound','swf'), /swf/lang/en/swf.php::$string['filenotfound'] = 'FILE_NOT_FOUND|404: File not found'; so that Flash apps can split at "|", read the type of error (FILE_NOT_FOUND) as a constant and, if desirable, display the message (404: File not found) to users.

Hopefully, this script will also be useful to others who want to implement manageable, stable, reliable, predictable media libraries within Moodle.

In reply to Matt Bury

Re: Simple proxy script workaround for Moodle's file API

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

big grin

Nice job and great idea about the way to localize errors: besides even the images are served via content.php e.g. mod/swf/content.php/mmlcc/commonobjects/pix/purse1.jpg.

Keep us updated using this thread,
Matteo

In reply to Matteo Scaramuccia

Re: Simple proxy script workaround for Moodle's file API

by Matt Bury -
Picture of Plugin developers

Yep, all the files have to go through content.php. AFAIK, the only way to get files into /moodledata/repository/ is via FTP so organisations should be able to implement some kind of standardised file structure or best practice for organising their media libraries coherently (one of the problems with Moodle 1.9's file management), which can add up to 10,000's of files very quickly. I'm offering some suggestions here: https://github.com/matbury/SWF-Activity-Module2.5/wiki/Content-Library Other suggestions and ideas would be most welcome smile