HTTP to HTTPS converter for external content

HTTP to HTTPS converter for external content

by John Okely -
Number of replies: 14

To ease the process of updating a moodle site from http to https (enabling ssl) we are going to develop a tool or filter to convert external embedded content (images, scripts, frames) from http to https.

We have a four options:

  1. Filter with manual blacklist
    A filter will be run, replacing urls on every page load. Admins will need to manage a blacklist of sites that do not support SSL, so that the filter will not change them to https.
    Pros: Database is left unchanged. If you later add http content, it will still work. You have more flexibility to change your site back and forth between http and https.
    Cons: Slight performance hit from running on every page load

  2. Filter with automatic blacklist
    Same as above but the blacklist will be updated automatically every night by a scheduled task. It will search the database for every external embeded link and check if the site supports SSL. Any site that does not will be added to the black list.
    Pros: Admins do not need to manually update blacklist
    Cons: Performance impact (although it can be scheduled outside of peak times). Longer development time

  3. Admin tool with manual blacklist
    An admin tool would be available to replace the links in the database. This will only need to be run once (when the site is changed from http to https), so will not impact performance.
    Pros: Cheaper performance
    Cons: If you add http content after running the tool, it will not work so this option would require a warning in editors.

  4. Admin tool with automatic blacklist
    Same as above, but the tool would check what sites do and do not support SSL when it is run.
    Pros: Admins do not need to manually update blacklist
    Cons: Slightly slower than #3

Another consideration is whether we should include the content from sites that do not have SSL anyway (potentially releasing session information) or just not include the content at all. It is also possible this should be configured on a per-site basis.

Personally I am leaning towards option number 4 with options to either include or skip content from non-SSL sites on a per-site basis

For those interested, this has become a priority due to the impending removal of loginhttps, which is expected to cause admins to upgrade their moodle sites from authentication-only SSL to site-wide SSL. This feature will make the transition easier. You can read the discussion about loginhttps here.

Average of ratings: Useful (3)
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Damyon Wiese -
We just had a discussion about this in the office and suggested a way forward.

The plan is:

* Install a filter with a "whitelist" of domains to rewrite. By default only internal links and supported links to our known repositories will be converted by the filter. The white list of domains should be configurable by the admin
* Enable the filter by default for sites that previously had loginhttps enabled
* If loginhttps was enabled during upgrade, send an email to the admin suggesting that they run the admin tool to convert all their links and disable the filter
* Add an admin tool to do the same rewriting and update the database. Support the same whitelist settings as the filter. The admin tool should work from cli as it may take a long time to complete. The optionally the web version should allow the actual rewriting to be done as an adhoc task (and send an email when its finished).

Please let us know if this plan does or does not sound like it will work for you.

Average of ratings: Useful (3)
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators
Hello,

At Enovation we have created a script for changing hardcoded urls in the Moodle database. Our approach is a bit different than the options above. We edit records directly in the database but we use whitelist for columns and tables. This is basically to make sure we know exactly what we edit. This alone could leave some content unchanged, so in the next step script does a search for hardcoded URLs across the whole database this time with blacklist. When records are found you can add them to wishlist and replace or decide to ignore them and add to search blacklist.

When changing this kind of data you also need to consider base64 encoded and url-encoded data and handle it appropriately. You want base64 search to be efficient - that is work using database query and not have to decode each record.

Finally our script works as a CLI but it's not Moodle admin tool - we wanted it to work even with old 1.9 Moodles.

We would be happy to share it publicly on github if you're interested. You can obviously use it to change url like http://example.com into https://example.com.

Cheers,
Tomek
Average of ratings: Useful (1)
In reply to Tomasz Muras

Re: HTTP to HTTPS converter for external content

by John Okely -

Sounds like a good solution. It would be great to see that code, thanks!

Average of ratings: -
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators
Our script is here: https://github.com/enovation/moodle-url_change . If you decide to do Moodle admin tool for 2.8 it may be worth moving some code into a shared library to share the code - as we will definitely want to use it across all Moodle versions.


cheers
Average of ratings: -
In reply to Tomasz Muras

Re: HTTP to HTTPS converter for external content

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

How does your proposed tool differ from tool_replace in standard Moodle? Wouldn't it be better to improve that tool than to build a separate add-on?

Average of ratings: -
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by John Okely -

Tracker link: MDL-46269

Average of ratings: -
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

Personally I think the "new content" issue should be resolved in the editor on saving content - it would be nice if it could do an automatic check and fix anything it can fix automatically and then warn the user perform some validation to report to the user that to embed the content they must find an https link.

Then we can just use an admin tool to fix "existing" content and don't need to rely on a filter that adds performance overheads to the site.

Average of ratings: -
In reply to Dan Marsden

Re: HTTP to HTTPS converter for external content

by John Okely -

Yeah the filter is added overhead. Then again, if we don't implement a filter, we will need the admin tool to run during upgrade, then again once the admin finishes putting together their whitelist.

The filter would work nicely for existing content, allowing the admin to locate items that don't work and gradually add them to the whitelist, without having to run the tool each time.

Average of ratings: -
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Kevin Wiliarty -

Is my understanding correct that a filter would miss URL resources set to display as embedded content? It seems to me that the external URL is not exposed to a filter in that case.

Average of ratings: -
In reply to Kevin Wiliarty

Re: HTTP to HTTPS converter for external content

by John Okely -

A filter could modify the embedded url, if it was programmed to do so. But it would not be able to change the url in the URL resource's settings. Only the admin tool could achieve that.

Average of ratings: -
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Kevin Wiliarty -

My attempt to write such a filter succeeded in influencing the intro contents of a URL resource -- so I know it is working -- but did not affect the value of the URL itself that was used to build the iframe markup. Looking at /mod/url/locallib.php I can see where some of the settings are being passed through filtering, but I can't see where $exteurl gets filtered. Is there some way, from within the filter (without hacking mod_url) to invoke a filter on that value?

I've tried the filter you proposed in connection with MDL-46296 and while it will modify an iframe src that is added to a textarea, it does not modify the "External URL" value for an embedded URL resource. I know that it won't change the setting, but it seems not to filter it, either.

Average of ratings: -
In reply to Kevin Wiliarty

Re: HTTP to HTTPS converter for external content

by John Okely -

In that case, perhaps a tool like enovation's would help? It could modify the external tool URL in the database.

Average of ratings: -
In reply to John Okely

Re: HTTP to HTTPS converter for external content

by Gyordee Lawe -

Hey John!! Did you guys ever complete that filter? I'm trapped in the crazy situation where I'm mandated to have https, but the users are refusing to change thier links to include the s in https. If there is a working filter or tool I can use to do this, that would be so amazing!!

Average of ratings: -
In reply to Gyordee Lawe

Re: HTTP to HTTPS converter for external content

by John Okely -

The filter has not been worked on. Edit: I forgot, but work had begun on MDL-46269. However this tool is not thoroughly tested at all. If you want to help create the filter, you can continue work on that issue, starting from my branch.


In your situation my encouragement would be to explain to the users why they should only include content with https. As discussed in this thread, the filter fix was a flawed solution because there is no reason that all content available over http will be available over https too. However, it will work for a majority of sites.


You can look at enovation's plugin and see if it is right for you https://github.com/enovation/moodle-url_change

Average of ratings: -