Using ARR Reverse Proxy - broken content

Using ARR Reverse Proxy - broken content

by Tom North -
Number of replies: 8

Hi all,

I have been tasked with setting up a new Moodle site for my organisation. Given our existing infrastructure, this will be served in a 3 tier environment. Topology is roughly as follows:

2x IIS with ARR for reverse proxy in DMZ connected to a - 

HA pair of hardware load balancers

2x IIS application servers connected to - 

A different pair of HA load balancers

Seperate database server that the App servers can connect to.


The aim has been to use the DMZ web servers purely to reverse proxy requests to the application tier. This is currently done with a simple catch-all wildcard rule. This works fine in most scenarios that we use it. 

The application servers are fine. Here I will call them http://app.moodle.local. The site is configured and working correctly. However, when viewing them through the reverse proxy, some of the javascript and css fails to load correctly. I have the requests rewriting correctly, but the application servers are resulting in a 404 when they receive the request, which can present itself as a 500 from the DMZ. 

I'm trying to understand why these assets won't resolve correctly if passed through a proxy. I'm not a developer so I'm not great at understanding what the javascript is for or whether there is a degree of client participation required that has now been removed by going through a proxy. I gather that yui_combo.php is a compiler of some sort. I do have an outbound rule to rewrite the host name of the app servers so that the URLs appear relative. If I remove that rule then it all works fine, because they will load with http://app.moodle.local, which I have access to on my network. However, when it goes into production, this won't be OK because the application servers are not internet facing. 

So the files in question are generally under /theme/ and /lib/. I can also see in the source code that some JQuery is allowing the http://app.moodle.local URL through. 

I'm guessing other people have used a 3 tier architecture to host Moodle, even if not in Windows environment (please don't yell at me). I'm interested to know the success people have had or any considerations that they have had to make to how they handle requests. 

Of course the real techies here may need more info, so please ask if there's something else I can provide. 


Thanks,
Tom


Attachment Capture.PNG
Average of ratings: -
In reply to Tom North

Re: Using ARR Reverse Proxy - broken content

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Tom,
you need to configure the nodes with the actual FQDN with which your users will access the Moodle clustered instance; the configuration is available in the config.php file of each Moodle instance in $CFG->wwwroot and add this line below could help:

$CFG->reverseproxy = true;

That being said, yui_compo.php and javascript.php files are charged to serve the required files in a "cache killer" fashion i.e. the guarantee that each time you update a component in Moodle they will change accordingly to let the browser to not cache the "static" files. They need PATH_INFO support to work i.e. you need to configure cgi.fix_pathinfo=1. Refs: https://docs.moodle.org/30/en/Internet_Information_Services#Slasharguments and https://docs.moodle.org/30/en/Internet_Information_Services#Optional_UTF-8_file_name_fix.

HTH,
Matteo

Average of ratings: -
In reply to Matteo Scaramuccia

Re: Using ARR Reverse Proxy - broken content

by Tom North -

Hi Matteo,

Thanks very much for the response. Today I have had some more success following your instruction.

I can confirm that the cgi.fix_pathinfo=1 was already set and the application itself has no errors.

Regarding the presentation through the ARR reverse proxy, I managed to get this working, but it was a tad unorthodox. Due to the way that we have configured our environment, I needed the application server to have a different host name binding to that of the front end reverse proxy servers. So I unfortunately had to set the site up under http://app.moodle.local. To get it to then use the URL that users would access the site with, I had to change the config.php after installation, then do a find and replace on the database to change http://app.moodle.local to http://www.moodle.local for example.


The rewrites were still an issue, as some of the content appeared to be compressed, which will by default stop IIS from rewriting outbound responses. I got around this with a well known HTTP_X_ORIGINAL_ACCEPT_ENCODING server variable. Since I had compression turned off on all web servers, I presume that it was Moodle doing this, because it was only a couple of JavaScript files affected.


All URLs are now serving relative and all content is rewritten through the reverse proxy. It seems to be browsing OK at the moment, but once it is put into use by developers and testers we will see. The only problem is that with the $CFG->reverseproxy set, I can no longer access the application server directly. I HAVE to access through the reverse proxy, which does remove a degree of troubleshooting ability in the future.


I am going to pay for an expert in Moodle to come in and sanity check once I have SSL applied to front end and stuff to sanity check it anyway, but if the above sounds completely messed up or flawed to you, please shout!

Thanks for the help, much appreciated.
Kind regards,

TomN

Average of ratings: -
In reply to Tom North

Re: Using ARR Reverse Proxy - broken content

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Tom,
happy to read the good news!

About $CFG->reverseproxy: shortly, you can't, unless some code hacking and understandings on the way the clustered instance will be used in terms of activities and resources.
The goal of Moodle code is to create constraints for the deployment in order to have a unique way to access the instance since in some places there is the need to write the URLs using the FQDN, which must be obviously unique.

That being said there is a way to do what you want to do IF you'll access the system for troubleshooting administration purposes and not e.g. to check for bugs when back-end users like teachers are not able to store some content: add a third back-end node, being not part of the farm, with the config.php file equals to the others but the $CFG->wwwroot value and $CFG->reverseproxy the setting. Then you could even play with the hosts file of the PCs used to perform administration troubleshooting or creating different views of your main DNS zone... that depends on the way you have created the site hosting this "back-end access" Moodle instance and on the way you're managing your DNS zones.

HTH,
Matteo

 

Average of ratings: -
In reply to Matteo Scaramuccia

Re: Using ARR Reverse Proxy - broken content

by Tom North -

Hi Matteo,

Understood - I don't think this causes us any real functional problem - in WordPress we would use this "back end access" to restrict access to admin pages from outside the local network, but we can control this from the front end with URL rewrite if need be by capturing the {Remote_Address}. I'm also please you haven't reacted in horror regarding me doing a search and replace on the database post installation, just because its a bit unusual and I was worried I was over complicating things. 

I have one final issue to chuck out just in case anybody wants to shout, but I will go and post it on the IIS forums where it's more at home. 

I have found that having ARR caching on breaks the javascript in Moodle. So the page will load fine, but clicking menu items etc has no effect. However, I need ARR caching turned on for our WordPress sites, which are accessed through the same external facing web servers for the reverse proxy. For now, because I'm under pressure to get a test system fully functional, I have had to deny read access to the cache location for the application. This is not something I am particularly proud of, but the cache control rules I put in to prevent ARR from caching the moodle site are not working. As said, I will go to the IIS forums for this, because denying NTFS read permissions to an application because I can't fix the cache control rules is a tad ridiculous haha. 


Thanks for your help Matteo.

Kind regards,

Tom N

Average of ratings: -
In reply to Tom North

Re: Using ARR Reverse Proxy - broken content

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Tom,
replace is part of the game when migrating an instance to another FQDN big grin, read more at: https://docs.moodle.org/30/en/Search_and_replace_tool.

Keep us updated: I'll try to install a W2K12R2 server with the ARR feature to try to test it by myself: Moodle is supposed - read: it should be reverse proxy friendly - to correctly manage its cache control headers including the way it compress some outputs.

Matteo

 

Average of ratings: -
In reply to Matteo Scaramuccia

Re: Using ARR Reverse Proxy - broken content

by Tom North -

Hi Matteo,


To all intents and purposes it seems perfectly friendly so far once I added the server variables to strip and store the encoding for the rewrite, just that if the client pulls from the ARR cache it breaks. Well, it breaks in our environment anyway! I didn't get around to posting on IIS forums yesterday, and due to impending annual leave I have jumped in to get technical consultancy in today, so will run it past them too. When I have a solution or information that helps others, I shall report back! I'll need to get my head around ARR a bit more, because the client cache doesn't have any negative affect on these assets.

Thanks,

Tom N


Edit- Noticed its an AJAX POST method when expanding the menu, so hadn't searched for that before. I found the below post which I may or may not be able to try today depending on how time goes with the guys I have coming in to check our configuration. Hopefully I will get a moment when the designers poach them for a chat:

https://forums.iis.net/t/1182301.aspx


Average of ratings: -
In reply to Tom North

Re: Using ARR Reverse Proxy - broken content

by Tom North -

Hmmm....so it would appear that the yui_combo.php was the cached item that stopped the menu expand/collapse from working. My colleague tells me it looks like some sort of compiler, but I am not a dev (unfortunately) so don't really get it. 

I have now added an ARR cache control rule to prevent this from caching (I noticed in the file it's cache control header is set to public, hence ARR is grabbing it). The ARR Cache control rule is dead simple, not sure why I haven't been able to get it working before. See below:



If this file doesn't cache, my menus work fine. We'll see over the next few days whether any of the other cached items cause issues. I don't suppose anyone would like to elaborate on why retrieving a cached version of this file may cause issues, but this is the workaround I have. 

As a bonus, the technical assistance who visited today were very happy with my environment, which is nice! 

Thanks for all the help Matteo, much appreciated.

Tom N

Average of ratings: -
In reply to Tom North

Re: Using ARR Reverse Proxy - broken content

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Tom,
yui_combo.php combines several files for the UI running on the browser side (CSS and JS), to reduce the HTTP hits required to "bootstrap" the UI and, yes, it helps caching the resources in both clients and proxies: it uses public and max-age to force a renewal and it uses a "revision number" in the "path" to invalidate the cache when 1) the theme cache is purged and 2) each time the Moodle instance will be updated in any of its components.

To do its work it makes usage even of HTTP GET params to describe a list of files to be served in one HTTP request so you could include query strings in your caching strategy and this could be an alternative option, to be tested (I still need to create the farm to play with).

HTH,
Matteo

 

Average of ratings: -