General plugins (Local): Data cleaner

local_datacleaner
Maintained by Picture of Catalyst ITCatalyst IT, Picture of Marcus BoonMarcus Boon, Picture of Brendan HeywoodBrendan Heywood, Picture of Dmitrii MetelkinDmitrii Metelkin, Picture of Daniel Thee RopertoDaniel Thee Roperto
This local plugin allows you to sanitise data from a production environment for use on testing/dev environments.
121 sites
36 downloads
11 fans
Moodle 2.6, 2.7, 2.8, 2.9, 3.0, 3.1, 3.2

DataCleaner Moodle Module

Moodle DataCleaner is an anonymizer of your Moodle data.


Supported versions of Moodle: 2.6 to 3.0 inclusive


How it works

Standard practice when hosting most applications, moodle included, is to have various environments in a 'pipeline' leading to production at the end. eg a typical flow might be 'dev' > 'stage' > 'prod' but there could be as many as you want for various reasons, like load testing, penetration testing etc.


To test properly it's often useful to have real production data in these other environments, but there are downsides:

  • Usually production can be quite massive, we don't need or want it all and disk space can be a pain with multiple copies.
  • There may be sensitive data we don't want to expose to developers or testers, eg personal data, grades, uploaded assignments etc
  • Moodle is integrated with 3rd party systems and we don't want test systems interacting with real systems, eg sending emails, or touching assignments in Turnitin etc, ie we want to remove any API keys and other related config

So we need a way to 'clean' the database after a refresh, to reduce the size of the data, to remove anything sensitive, and to ensure it's not going to touch any other real system. This also needs to be configurable because every moodle instance has different needs and there is no one-size-fits all approach. This could be configured outside moodle in the deployments tools, but over time we have found the most flexible and easiest approach is to have this configuration inside moodle itself, so our clients can directly make these decisions, and not be exposed to any of the complexity of our internal processes around continuous integration and deployment.


Practically this means the cleaning configuration needs to be added into the production system (which initially sounds scary but isn't), then you refresh the database to another environment where it can be washed. There are multiple levels of safeguards in place to ensure this never gets run in production, which would of course be catastrophic:

  • It can only be run from the CLI. There is no GUI.
  • We store the hostname in the cleaning configuration data. If the hostname matches production, DataCleaner will not run. If this data is missing then it will not run.
  • Typically a refreshed database will be from a nightly snapshot and so the data should be slightly stale. If a non admin user has logged in recently, that's a sign this moodle is being used, and the DataCleaner will not run.
  • If cron has run recently, DataCleaner will not run. This should only be run on a data washing instance, cron should not be needed here.
  • It can only be run if and only if a 'local_datacleaner_allowexecution = true;' has been added to config.php


Installation

The simplest method of installing the plugin is to choose "Download ZIP" on the right hand side of the Github page. Once you've done this, unzip the DataCleaner code and copy it to the local/datacleaner directory within your Moodle codebase. On most modern Linux systems, this can be accomplished with:

unzip ./mdl-local_datacleaner-master.zip
cp -r ./mdl-local_datacleaner-master <your_moodle_directory>/local/datacleaner


Once you've copied the plugin, you can finish the installation process by logging into your Moodle site as an administrator and visiting the "notifications" page:

<your.moodle.url>/admin/index.php


Your site should prompt you to upgrade.


Configuration

Once the installation process is complete, you'll be prompted to fill in some configuration details. Note that you MUST visit the DataCleaner config page to save the current wwwroot, or the cleaner will not run later in the other environments.


You have to add a 'local_datacleaner_allowexecution = true;' config item to your config.php in each of the environments you want the cleaner to run. DO NOT add that config setting to a Production environment!


There are multiple 'cleaners' which process different types of data in moodle. Each one can be enabled individually and may have additional config settings.


You can find the DataCleaner configuration via the Moodle administration block:

Site Administration > Plugins > Local plugins > Data cleaner


Sub-plugin options

Enable the sub-plugin options to clean the corresponding data area.


Cleanup core:

Enable this sub-plugin to clean core configuration settings.


Remove config:

Enable this sub-plugin to clean configuration settings. This has its own Settings page.


Remove standard logs:

Enable to truncate the standard log table.


Remove users:

This will remove users who have not logged in for a specific number of days. This has its own Settings page.


Remove courses:

Remove courses older than a specific number of days and/or in specific categories. This has its own Settings page.


Scramble user data:

Enable this sub-plugin to anonymise user data. This has its own Settings page.


Clean grades:

Enable to delete grade history or replace with fake data. This has its own Settings page.


Replace URLs:

Enable to replace all occurrences of the production URL with a another URl. This has its own Settings page.


Cleanup sitedata:

Clean orphaned files or replace with a generic file for the specific file type.


Running

After installing and configuring DataCleaner, copy your database and optionally your site data to another moodle instance.


From here run the cli script. On most modern Linux systems, this can be accomplished with:

sudo -u apache /usr/bin/php /<your_moodle_directory>/local/datacleaner/cli/clean.php --run


There are protections in place which prevent accidental running on this on your production system - which would of course be catastrophic!


More options

Run the cli script with --help for more options:

sudo -u apache /usr/bin/php /<your_moodle_directory>/local/datacleaner/cli/clean.php --help

Screenshots

Screenshot #0
Screenshot #1
Screenshot #2
Screenshot #3
Screenshot #4
Screenshot #5
Screenshot #6
Screenshot #7
Screenshot #8
Screenshot #9

Contributors

Picture of Catalyst IT
Catalyst IT (Lead maintainer)
Picture of Brendan Heywood
Brendan Heywood: Author
Picture of Dmitrii Metelkin
Dmitrii Metelkin
Picture of Daniel Thee Roperto
Daniel Thee Roperto
Please login to view contributors details and/or to contact them

Comments RSS

Show comments
  • Picture of David Mudrák
    Wed, Apr 20, 2016, 7:55 PM

    Thanks guys for sharing the plugin with the community. Can we please have the issue #4 sorted out before we move on with the approval review. Thanks.

    p.s. Better not to delete issues from github.

  • Picture of David Mudrák
    Fri, Jun 10, 2016, 4:58 PM

    Thanks for the new version. I can confirm the issue #4 has gone and all the rest looks ok to me. You are cleared to land, welcome to the Plugins directory!

  • Picture of Nico Declerck
    Sun, Jun 25, 2017, 4:29 PM
    The idea is very attractive, but if it is so important to add the wwwroot somewhere (since it is not enough that the setting is in the config.php-file), then why
    1) is it not shown on any of the pictures with this plugin?
    2) why is it not clearly documented where to find it? (for instance in the same sentence that emphasizes the importance of the setting)
    It would stop people from getting frustrated all the time
  • Picture of Brendan Heywood
    Mon, Jun 26, 2017, 8:50 AM
    hi Nico,

    The cleaner plugin needs to know both the wwwroot of your production environment as well as the wwwroot of the cleaning environment. It uses this for rewriting urls as well as one of several forms of protection to ensure that you don't accidentally clean your prod environment which would of course be catastrophic. You can find that setting in the admin gui page for the cleaner local plugin. You need to install the plugin into production, then configure it there to specify exactly what is cleaned and how, then clone the db from prod into a separate 'washing machine' moodle somewhere else and run the cleaning cli from there.

    https://github.com/catalyst/moodle-local_datacleaner#configuration
  • Picture of Leandro De Mestico
    Sat, Apr 13, 2019, 12:26 AM
    Does data cleaner work for moodle 3.5?
Please login to post comments