Security and privacy

HTTP to HTTPS converter for external content

 
 
Picture of John Okely
HTTP to HTTPS converter for external content
Group Moodle HQGroup Testers

To ease the process of updating a moodle site from http to https (enabling ssl) we are going to develop a tool or filter to convert external embedded content (images, scripts, frames) from http to https.

We have a four options:

  1. Filter with manual blacklist
    A filter will be run, replacing urls on every page load. Admins will need to manage a blacklist of sites that do not support SSL, so that the filter will not change them to https.
    Pros: Database is left unchanged. If you later add http content, it will still work. You have more flexibility to change your site back and forth between http and https.
    Cons: Slight performance hit from running on every page load

  2. Filter with automatic blacklist
    Same as above but the blacklist will be updated automatically every night by a scheduled task. It will search the database for every external embeded link and check if the site supports SSL. Any site that does not will be added to the black list.
    Pros: Admins do not need to manually update blacklist
    Cons: Performance impact (although it can be scheduled outside of peak times). Longer development time

  3. Admin tool with manual blacklist
    An admin tool would be available to replace the links in the database. This will only need to be run once (when the site is changed from http to https), so will not impact performance.
    Pros: Cheaper performance
    Cons: If you add http content after running the tool, it will not work so this option would require a warning in editors.

  4. Admin tool with automatic blacklist
    Same as above, but the tool would check what sites do and do not support SSL when it is run.
    Pros: Admins do not need to manually update blacklist
    Cons: Slightly slower than #3

Another consideration is whether we should include the content from sites that do not have SSL anyway (potentially releasing session information) or just not include the content at all. It is also possible this should be configured on a per-site basis.

Personally I am leaning towards option number 4 with options to either include or skip content from non-SSL sites on a per-site basis

For those interested, this has become a priority due to the impending removal of loginhttps, which is expected to cause admins to upgrade their moodle sites from authentication-only SSL to site-wide SSL. This feature will make the transition easier. You can read the discussion about loginhttps here.

 
Average of ratings:Useful (3)
Me!
Re: HTTP to HTTPS converter for external content
Group DevelopersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers
We just had a discussion about this in the office and suggested a way forward.

The plan is:

* Install a filter with a "whitelist" of domains to rewrite. By default only internal links and supported links to our known repositories will be converted by the filter. The white list of domains should be configurable by the admin
* Enable the filter by default for sites that previously had loginhttps enabled
* If loginhttps was enabled during upgrade, send an email to the admin suggesting that they run the admin tool to convert all their links and disable the filter
* Add an admin tool to do the same rewriting and update the database. Support the same whitelist settings as the filter. The admin tool should work from cli as it may take a long time to complete. The optionally the web version should allow the actual rewriting to be done as an adhoc task (and send an email when its finished).

Please let us know if this plan does or does not sound like it will work for you.

 
Average of ratings:Useful (3)
Picture of Tomasz Muras
Re: HTTP to HTTPS converter for external content
Group DevelopersGroup Particularly helpful MoodlersGroup Translators
Hello,

At Enovation we have created a script for changing hardcoded urls in the Moodle database. Our approach is a bit different than the options above. We edit records directly in the database but we use whitelist for columns and tables. This is basically to make sure we know exactly what we edit. This alone could leave some content unchanged, so in the next step script does a search for hardcoded URLs across the whole database this time with blacklist. When records are found you can add them to wishlist and replace or decide to ignore them and add to search blacklist.

When changing this kind of data you also need to consider base64 encoded and url-encoded data and handle it appropriately. You want base64 search to be efficient - that is work using database query and not have to decode each record.

Finally our script works as a CLI but it's not Moodle admin tool - we wanted it to work even with old 1.9 Moodles.

We would be happy to share it publicly on github if you're interested. You can obviously use it to change url like http://example.com into https://example.com.

Cheers,
Tomek
 
Average of ratings:Useful (1)
Picture of John Okely
Re: HTTP to HTTPS converter for external content
Group Moodle HQGroup Testers

Sounds like a good solution. It would be great to see that code, thanks!

 
Average of ratings: -
Picture of Tomasz Muras
Re: HTTP to HTTPS converter for external content
Group DevelopersGroup Particularly helpful MoodlersGroup Translators
Our script is here: https://github.com/enovation/moodle-url_change . If you decide to do Moodle admin tool for 2.8 it may be worth moving some code into a shared library to share the code - as we will definitely want to use it across all Moodle versions.


cheers
 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: HTTP to HTTPS converter for external content
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

How does your proposed tool differ from tool_replace in standard Moodle? Wouldn't it be better to improve that tool than to build a separate add-on?

 
Average of ratings: -
Picture of John Okely
Re: HTTP to HTTPS converter for external content
Group Moodle HQGroup Testers

Tracker link: MDL-46269

 
Average of ratings: -
Picture of Dan Marsden
Re: HTTP to HTTPS converter for external content
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Particularly helpful MoodlersGroup Translators

Personally I think the "new content" issue should be resolved in the editor on saving content - it would be nice if it could do an automatic check and fix anything it can fix automatically and then warn the user perform some validation to report to the user that to embed the content they must find an https link.

Then we can just use an admin tool to fix "existing" content and don't need to rely on a filter that adds performance overheads to the site.

 
Average of ratings: -
Picture of John Okely
Re: HTTP to HTTPS converter for external content
Group Moodle HQGroup Testers

Yeah the filter is added overhead. Then again, if we don't implement a filter, we will need the admin tool to run during upgrade, then again once the admin finishes putting together their whitelist.

The filter would work nicely for existing content, allowing the admin to locate items that don't work and gradually add them to the whitelist, without having to run the tool each time.

 
Average of ratings: -