303 + Logon Redirect for most external SCORM course resources

303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
Number of replies: 21
Hey there,

I'm a bit of a Moodle newb, but I'm a programmer so I've dug around a ton on this issue and can't figure it out. Hopefully someone can help me.

I've installed Moodle 1.9.8 on a Rackspace Linux/mysql setup. Starting from this bone stock install, I create a single course, add a SCORM/AICC activity, and upload a super simple SCORM package consisting of an HTML page and a few external resources (js, css, swf). That's it.

This is a direct download link to the SCORM sample file: http://pipwerks.com/d/9

I have, however, tried this with a dozen other files, some found, some created from scratch, all with the same issue.

When I try to run the course, I can see in Firebug it works fine for the html page, but when it tries to access any other resource file, Moodle throws a 303 ("See Other") HTML status code response and redirects to the login page.

If I type the address of the html file directly (while logged in, and via file.php), it's also no problem. When I try to hit one of the other resource files, it still does the 303/login thing. If I log in, it just tries to hit the file again and redirects once more with a 303 to the login page.

I do have the Multimedia Plugin/Filter for swf enabled, but it doesn't seem to make any difference. But it almost seems like Moodle is allowing/failing on files depending on the file type.

I've also tried slasharguments on/off and it makes no difference. admin/health.php tells me no problems found, and file.php/testslasharguments also tells me I'm good to go.

I've also tested Rackspace's Apache setup, and AcceptPathInfo is indeed set.

I can hack/solve this in four ways, none of which are acceptable:

1. Adding any query string (e.g.: ?aaa=0) to the end of the external filename makes it work. I'm not sure why -- maybe it tricks Moodle into thinking it's not a "restricted" file type? If so, this is probably a security concern. Anyway, I don't necessarily have access to every module I'm going to add to put a querystring on the end of each filename.

2. Changing the path from xxx/file.php/11/etc/etc to xxx/file.php?file=/11/etc/etc also fixes it, but I'm not sure why. Again, I can't necessarily modify the file paths of everything I'm uploading, so also not a good solution.

3. I can hard-code the external file paths to bypass file.php. Again, I can't necessarily modify the file paths of everything I'm uploading, so also not a good solution.

4. With enough digging, I've figured out I can comment out lines 70-71 in file.php and it will load the files just fine. But those lines are security code and probably very bad to remove:
} else if ($course->id != SITEID) {
require_login($course->id, true, null, false);

So a lot of these solutions I can add manually to the code, but that's a huge hack just for my purposes and will be blown away with each code base update, so I want a real solution. Either I need to configure something I've missed, or there's a bug in Moodle I'd like to get solved.

I ended up tracing the problem down as far as I can go, and I've determined that in lib/setup.php, when @session_start(); is called, $_SESSION['USER'] is retried and populated fine on the files I can access, but it's not populated on the files I can't. It's just empty, and I can't figure out why that is because I'm not sure what code is running to populate/not populate the USER session variable. I thought I traced through every piece of code that runs from the moment I request a file to the moment it fails, but I must be missing something.

So hopefully someone can help me. What is it about these (.js, .css, .swf) files that cause Moodle to not retain the USER session variable, and therefore throw me into a circular, eternal "303 See Other" forward to the login page?

Thanks in advance.


-=Cam
Average of ratings: -
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Amy Groshek -
Hi Cam,

I wonder if this is what you're seeing:
http://docs.moodle.org/en/SCORM_FAQ#File_not_found_error

If so, it has to do with the scorm packages, specifically, the manifest file, and not the other stuff you're looking at.

Have all of your scorm packages been zip directories, with an imsmanifest.xml file in the root directory? If so, does the manifest list every single file to be used, with the correct relative path?

Moodle duplicates all of the files in the scorm package into /moodledata/{courseid}/moddata/scorm/{no}/ for playback. You can find that dir in the course files and make sure all of your scorm package files made it there. If not, take a look at the manifest you're delivering up to Moodle, and make sure that you're packaging and installing the SCORM package correctly.

-A
In reply to Amy Groshek

Re: 303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
Hi Amy,

Thanks for replying!

Sadly, I don't think that is the problem. In the insanely simple planets demo I linked to, the index.html and planets.swf files are both in the root, and both relatively referenced properly. It displays index.html no problem, but not planets.swf.

Also, it's a 303 (See Other) error, redirecting to the login, implying a security issue, vs. a 404 (File Not Found) error.

I was hoping your solution worked, too, because it would have been such an easy fix!


-=Cam
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Matteo Scaramuccia -
Hi Cam,
it seems an issue strictly related to the PHP session being broken since you're able to access the resource using a new tab within a browser session with an already logged Moodle session and commenting out require_login() over sites different from the main one fixes everything. Please:
  1. Look at the HTTP headers of the request to external files to see if the PHP cookie session is there. I guess no;
  2. Check-out the Only HTTP cookies setting: if enabled, disabled it. I guess it is enabled.
Question: has the published PIF being generated using the content just under /completed folder? Time permitted, I'll give it a try since your (1) seems not to fit with my suggestions.

HTH,
Matteo
In reply to Matteo Scaramuccia

Re: 303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
Hi Matteo,

I'm pretty sure we're talking about the same thing, but what do you mean by "able to access the resource using a new tab within a browser session with an already logged Moodle session"? If you're talking about the .swf file, I can't access it under any conditions save commenting out the require_login() function.

Also, you mention "you're able to access the resource ... [when] commenting out require_login() over sites different from the main one". I'm not sure what you mean by "sites different from the main one", as it's all the same site/course.

As for your questions, I *am* seeing the cookie for both the .html file (which works fine) and the .swf file (which results in the 303/login redirect).

The one thing I find curious is no matter what file I request in the course, it always lists the index.html file as the referrer, even if I'm accessing a different file in the course directly in a new tab. It doesn't matter whether I'm accessing a file I *can* get to or one that throws a 303, they all say the index.html file is the referrer. I've noticed this in both Firebug and while tracing through the php code.

As for the cookies, it was set to "Secure cookies only: No". I switched it to yes and tried again with no luck.

The one thing I'm still really stumped on is when the $USER variable gets set with login information. That seems to be the root of the problem. The .swf file's $USER variable is NOT set, causing it to be redirected to the login page, whereas the .html file's (and other files') $USER variable IS set, so it's fine. I can't figure out where in the php code it's being set/not set for either file, so I can't keep tracing it back and find the root of the problem.

Thanks so much for replying!


-=Cam
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
Quick general question (to Dan?): should I cross-post this to a more core Moodle developer forum? I'm getting this impression this is a security issue rather than a SCORM issue.


-=Cam
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
Hi Cam - is this specific to SCORM? - or does it occur when you add a normal resource as well?

What browser/version are you using? - does it occur using other browsers? - can you reproduce it on another machine?
In reply to Dan Marsden

Re: 303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
Hi Dan,

Good call -- it seems to be happening with any files of certain type, in my case specifically .jpg and .swf files.

It seems fine with .html, .js, .css...

I keep feeling like this is some REALLY common newb thing I'm just screwing up, like a security setting, but there's no reference anywhere to this symptom (303 "See Other" redirect).


-=Cam
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
In reply to Dan Marsden

Re: 303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
DAN!

That is EXACTLY what is happening!

Thank you so much for finding that -- I'm surprised I haven't run across it yet.. I've been working on this off and on for a month.

Hopefully this thread will save someone else a boatload of time if they hit any of these symptoms.

I'm off to badger Rackspace into reconsidering their "nothing we can do" stance.

Thank you so much.


-=Cam
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
np - good to see there's an easy solution to bypass the cache too!
In reply to Dan Marsden

Re: 303 + Logon Redirect for most external SCORM course resources

by Cam Marsollier -
Well, to end off this discussion, Rackspace pointed me to a FAQ they have detailing possible solutions:

http://cloudsites.rackspacecloud.com/index.php/How_can_I_bypass_the_cache%3F

I've jointly worked through all of their solutions in that document, and the only one that works is adding a querystring to the end of the file. It's not a very good solution unless it's a one-off with a simple course or you built it yourself. If you have a ton of courses to convert, your content is user-submitted, or you don't have access to the course code, there's a good chance that solution may not work for you.

The main problem is they have a media caching server called Varnish, and before their servers even look at .htaccess, it sends it to Varnish. So even if you have caching turned off, it STILL goes to Varnish, then reports a cache miss ("caching is turned off") and fetches your file from elsewhere. But by that time it's already blown away your PHP session, making Moodle think the user isn't logged on and redirecting you (for that file) back to login.

Apparently this problem also affect the ability to host MediaWiki on Rackspace.

I hope time will provide a solution to this one. Currently I'm investigating using Moodle 2.0's new Amazon S3 module so I can host the courses on Amazon, which is not only cheap and convenient, but would bypass this entire caching issue.
In reply to Cam Marsollier

Re: 303 + Logon Redirect for most external SCORM course resources

by Franco Pomidoro -

Hi Cam, I've bumped into this issue testing the new moodle 2.0 RC1 in Rackspace Cloud Sites.

I've managed to make a workaround with some .htaccess rewrite rules and an extra PHP file following the basic idea provided by "initall" here http://stackoverflow.com/questions/2248702/rackspace-cloud-strips-session-if-url-has-certain-file-extensions

Basically the rewrite rule picks up every /pluginfile.php/... link that has no querystrings and which extension is within the list of files that are routed to the MA servers (gif|jpe?g|png|mp3|wav|wma|wmv|swf|flv|f4v).

When a match is found, the redirect rule sends the browser to look after the extra PHP file, "bypassvarnishcache.php" adding the original URL as a querystring.

"bypassvarnishcache.php" just take the original URL from the querystring and adds "?cache=0" to it, then redirects the browser to follow that link and presto!

So, if the original link was h**p://www.domain.com/pluginfile.php/82/mod_resource/content/1/images/footer.jpg, it will be rewrite it to h**p://www.domain.com/pluginfile.php/82/mod_resource/content/1/images/footer.jpg?cache=0

This concept should be easily adapted to "file.php" for 1.X versions.

I have not tested it yet with SCORM packages but it should work or need minor changes, when I tested it if changes are needed I'll post the update.

So far the only problem I have is with WMV files and the WMP plug-in, somehow the "?cache=0" trick do not work.

Hope it helps!

In reply to Franco Pomidoro

Re: 303 + Logon Redirect for most external SCORM course resources

by Matteo Scaramuccia -

Hi Franco,

I think redirection (HTTP 302) to be not always the right solution when the URL points to a media asset like audio and video. I'm not using Rackspace but interested in understanding the issue.

Rackspace suggests Option Four among its options and it could be easily addressed by Moodle code:

Index: filelib.php
===================================================================
RCS file: /cvsroot/moodle/moodle/lib/filelib.php,v
retrieving revision 1.50.2.34
diff -u -r1.50.2.34 filelib.php
--- filelib.php    7 Oct 2010 08:42:47 -0000    1.50.2.34
+++ filelib.php    25 Nov 2010 11:11:56 -0000
@@ -731,6 +731,9 @@
}

if ($lifetime > 0) {
+        if (!empty($CFG->disablereverseproxycaching)) {
+            @header('Cache-Control: private');
+        }
@header('Cache-Control: max-age='.$lifetime);
@header('Expires: '. gmdate('D, d M Y H:i:s', time() + $lifetime) .' GMT');
@header('Pragma: ');

Try the patch above, adding a new line $CFG->disablereverseproxycaching = true; to your config.php and report back if it works.

Ciao,

Matteo

In reply to Matteo Scaramuccia

Re: 303 + Logon Redirect for most external SCORM course resources

by Franco Pomidoro -

Hi Matteo, thanks,

I've tested this just in case but unfortunately this approach does not work. It should, according to the documentation you mentioned, as well as the .htaccess approach mentioned in that same documentation.

The thing is the problem is with the request and the PHP session being lost when routed to the MA server.

As Cam posted, the following happens when a request is made:

"The main problem is they have a media caching server called Varnish, and before their servers even look at .htaccess, it sends it to Varnish. So even if you have caching turned off, it STILL goes to Varnish, then reports a cache miss ("caching is turned off") and fetches your file from elsewhere. But by that time it's already blown away your PHP session, making Moodle think the user isn't logged on and redirecting you (for that file) back to login."

For other people with this issue and thinking on contacting Rackspace support I'm pasting their response about this:

"[...] If I am reading this correctly your website uses sessions to know if it's ok to get these images. But all image requests go through our MA cache server like a media proxy. The issue is that there is no image in MA cashe and because of this MA cashe attempts to get the image from your site. And your site refuses to talk to our MA cashe because it's not authenticated.
So Logically speaking it does not matter if your trying to dissable caching here. This is because what you are doing is only setting a (do not save) flag on the response. When the request is the problem.
This is also why links of the nature .jpg?a=0 work. Because only urls that match (gif|jp?g|png|mp3|wav|wma|swf)$ are routed through our Media accelerator box. and this is done as the request comes into our infrastructure before your .htaccess rules are checked."

"[...]Unfortunately there is no way for us to configure varnish just for your site. It's used to host thousands of websites actually."

Another valid approach, I think better also, would be modifying Moodle code to add a querystring to the URLs but not sure where to start here, I see you are familiar with filelib.php, can you point to a start point?

Thanks! best! Franco.

In reply to Franco Pomidoro

Re: 303 + Logon Redirect for most external SCORM course resources

by Matteo Scaramuccia -

Hi Franco,

understood: so they have configured Varnish to be friendly with Cache-Control: private but they keep on stripping down cookies even if Varnish will ignore to cache the asset... sounds like a misconfiguration on their Varnish configuration or an issue on their assets redirection to MA (querystring disallows redirection so it could be feasible for them to take a look at headers too, before going for a redirection), which, I guess, is used to drop cookies to avoid some sort of security injection during the lifetime of the asset, greater than a logged session.

BTW this is off-topic (but quite interesting), backing on topic ;): Moodle Files API is able to manage security checks (it acts like a filter at the architecture level) as well as HTTP headers settings, redirection is not a good solution in case of media assets because:

  1. it could be not correctly supported by some players
  2. it adds an HTTP round trip which could slow down things

Besides adding a querystring to implement a cachekiller approach, useful e.g. for Flash based clients loading external content, prevents some browsers to cache the content so we are adding latency and bandwidth overload but... to answer your question and implements the same behaviour of your varnish.zip it could be done with (changed configuration param due to the nature of this ad hoc approach):

Index: file.php
===================================================================
RCS file: /cvsroot/moodle/moodle/file.php,v
retrieving revision 1.46.2.5
diff -u -r1.46.2.5 file.php
--- file.php    9 Apr 2009 09:30:32 -0000    1.46.2.5
+++ file.php    25 Nov 2010 20:51:44 -0000
@@ -37,6 +37,19 @@
error('No valid arguments supplied, path does not start with slash!');
}

+    // Hack: add a query string, if missing, to those assets managed
+    //       through the Rackspace web accelerator environment
+    if (// Available if properly enabled in config.php
+            !empty($CFG->disablerackspacecaching)
+        // No query string?
+        && (empty($_SERVER['QUERY_STRING']))
+        // Are we going to serve a common media asset?
+        && (preg_match('/\.(gif|jpe?g|png|mp3|wav|wma|wmv|swf|flv|f4v)$/i', $relativepath))) {
+        // Kind redirection: 302
+        @header('Location: '.$_SERVER['REQUEST_URI'].'?cache=0', true, 302);
+        die;
+    }
+

The goal of my reply was to understand if there was room to ask for a new feature in Moodle but IMHO redirection can never be implemented into the main stream for the reasons above.

Let me know if it works as per your needs,

Matteo

In reply to Matteo Scaramuccia

Re: 303 + Logon Redirect for most external SCORM course resources

by Matteo Scaramuccia -

Hi Franco,

my previous reply did not take into account that your .htaccess talks about Moodle 2.0. My patch is for 1.9 and it is useful to address the redirection on any OS regardless the web server.

Your goal could be addressed by simply reviewing your .htaccess to use just 1 redirection, potentially fixing your video issues:

RewriteEngine on
RewriteCond %{QUERY_STRING} ^$
RewriteRule varnish\.php/(.*)\.(gif|jpe?g|png|mp3|wav|wma|wmv|swf|flv|f4v)$ %{REQUEST_URI}?cache=0 [R=302,NC]

HTH,

Matteo

In reply to Matteo Scaramuccia

Re: 303 + Logon Redirect for most external SCORM course resources

by Franco Pomidoro -

Hi Matteo, yep, I agree with you, it seems the varnish cache server is not properly configured, but they say it's like that and there is nothing they can do so...here we are =P

You are right about the 2 redirections, there is no need for that, my head was a little burned out after trying several different ideas to make moodle 2.0 work on rackspace  =)

I'm uploading the new .htaccess file. The problem with WMV and Windows Media Player continues though, just for the record, it seems a WMP issue, Firefox running in Ubuntu has no issues opening WMV files with this rewriterule.

Thanks for your answer! but I think I did not explain myself correctly...

What I'm aiming for as a more valid "workaround" for this issue, maybe something that could be added to moodle code in future releases as a config option, is the following:

I'm looking for a way to add the "?cache=0" querystring to the URL when Moodle parses it, before the HTML is sent to the browser, before the request. This way no redirection would be necessary the link will already be "able to bypass" the varnish cache server.

Following the logic of your example, something like:

if (// Available if properly enabled in config.php
!empty($CFG->disablerackspacecaching)
// No query string?
&& (empty($_SERVER['QUERY_STRING']))
// Are we going to serve a common media asset?
&& (preg_match('/\.(gif|jpe?g|png|mp3|wav|wma|wmv|swf|flv|f4v)$/i', $filename)))
{
// Add querystring to disable caching
$filename = $filename.'?cache=0';
}

The thing is where to put it! I'm not that familiar with Moodle 2.0 code =( I've done a quick search and I found out there are a lot of "rewrite" functions, do you know if they all end up some place where this piece of code could be added?

Thanks! Franco.

In reply to Franco Pomidoro

Re: 303 + Logon Redirect for most external SCORM course resources

by Matteo Scaramuccia -

Hi Franco,

I'm quite new to Moodle and "newer" to 2.0: please take my advices under this premise ;)

It will be "somewhat" time consuming to try to address your request. There are plenty of points where the hack should be added, not to talk about contribs. It could be done but really difficult to be tested (regressions) and fully adopted.

IMHO this will be not feasible at least until Moodle will be able to manage prepared HTML anchors e.g. for SEO goals: at that time there will be some code (kind of Links Manager) that all the links will (be forced to?) run to optionally have SEF URLs... really far from it right now.

Option Four (the one allowing the usage of local browser cache but IMHO even Option One is nice and maybe better), being under the domain of HTTP headers manipulation, can be easily addressed by Moodle Files API right now.

IMHO if Rackspace is claiming that the set of HTTP header based Options is equal to Option Two, they need to fix the incorrect stripping down of cookies... otherwise they need to reduce the number of their successfully running options up to 1, just Option Two. If I have correctly understood, any PHP app controlling the streaming of files for security - read: authorization - reasons will fail to run under Rackspace cloud hosting.

Matteo

In reply to Matteo Scaramuccia

Re: 303 + Logon Redirect for most external SCORM course resources

by Franco Pomidoro -

Hi Matteo, sorry for the delayed answer,

I understand, thank you very much for your help and ideas, it was good to have someone to "ping pong" ideas with.

Yesterday I opened a ticket requesting Rackspace to change how all this in handled so applications that protect media assets with sessions will work. Silence so far...

I'll prepare a new post explaining the problem and situation and see if other people have some ideas on how to modify moodle 2.0 code to achieve what we've talked about.

Thanks again, best. Franco.

In reply to Franco Pomidoro

Re: 303 + Logon Redirect for most external SCORM course resources

by Franco Pomidoro -

Hi everybody, just to update on this,

I have created this request for rackspace to take this into consideration based on how many people vote for it, so please vote! big grin

http://feedback.rackspacecloud.com/forums/71021-product-feedback/suggestions/1261723-ability-to-bypass-media-assets-redirection-to-ma-c

thanks! best! franco.

In reply to Franco Pomidoro

Re: 303 + Logon Redirect for most external SCORM course resources

by Franco Pomidoro -

Hi everybody, just bringing this back to your attention, if you could please drop by and vote this will help have an interesting viable cloud option for hosting Moodle.

Best, Franco.