Plugins traffic

Duplicate user accounts is a nightmare for every good sysadmin. Especially if each of the accounts has been actually used for a while and is associated with recorded traces here and there (such as submitted assignments, attempted quizzes, published forum posts etc). This month, we are spotlighting a plugin that can help you to deal with these duplicates. It is an admin tool called simply Merge user accounts.

The tool tries to merge two Moodle user accounts into a single one. The solution is based on re-assigning all activity and records referring to the user A so they refer to the user B. This will give the effect of user B seeming to have done everything both users have ever done in Moodle.

Interview with the maintainers

The plugin maintainers, Nicolas Dunand and Jordi Pujol-Ahulló shared their thoughts about the project’s history and development background.

Hi guys. Can you shortly tell us something about yourself and your background?

Nicolas: Hi! My name is Nicolas, and I work in Lausanne, in the French speaking part of Switzerland. I have an MD in process engineering, but turned to IT shortly after graduating. I'm now working at the University of Lausanne, as an IT systems engineer. Since the adoption of Moodle by the university as our main online learning environment, my duties have mostly evolved around Moodle: developing the servers infrastructure, maintaining the software, developing plugins, and integrating Moodle with other information systems whenever feasible.

Jordi: Hello! My name is Jordi and I live in Tarragona, Catalonia, Spain. I am PhD in Computer Science with European Mention, in the field of Distributed Systems. Currently I’m working as software architect and software engineer at Universitat Rovira i Virgili. My University adopted Moodle in 2004 as its Virtual Learning Environment and we are responsible for Moodle integration with other University information systems. Besides, the University unit I’m working in, Servei de Recursos Educatius, is responsible for other teaching and planification applications and programs addressed to support to and increment teaching quality.

How did you get into Moodle and Moodle development?

Nicolas: As Moodle has taken more and more prominent position in my university's teaching support services, Moodle has become a central part of my work. What was initially just one service we provided among others became the central point of my job. I work in a service (RISET - Réseau Intefacultaire de Soutien Enseignements & Technologies) that values teacher input and requests a lot, so over time the list of third-party and home developed plugins grew quite a lot. We try as much as possible to follow a policy of sustainability when using third-party plugins, so our goal is of course the same when developing our own. In that sense, I try as much as possible to make any portable plugins public. This is for two main reasons: first, we profit so much from the whole Moodle community that it feels good to give something back, and second, the Moodle community is very efficient at giving feedback and requesting new features we sometimes hadn't thought of.

Jordi: Sincerely, I get into Moodle due to my job. One of my main tasks of my job is to support technically Moodle into my University, keep it up-to-date, planning and performing Moodle upgrades, as well as develop plugins (enrollment, blocks, locals or authentication) necessary for the University community. Apart from my contribution to the Merge Users administration tool, we contributed the Moodle community with other public plugins, like auth_ip, and other private plugins. Our aim is to make any plugin we develop as publicly available. However, private plugins were finally necessary since other University information systems we are synchronizing with, are so particular or complex that they do not fit into any Moodle standard plugin or they are related to private information systems.

I really like my job, actually, I enjoy it, and it allowed me to keep in touch with open source software and a live community like Moodle one, where people truly share and report issues as well as contributions (like the last ones in Merge Users plugin). I love that, I love contribute in something that helps and address needs from other people around the world, at the same time that they fit our needs.

Nicolas: Yes, I really like that, too! It's great to be able to collaborate with Jordi, even though we've never met in person. It is also very exciting to get feedback from people from all around the world about the work we're doing.

You both are set as the maintainers of the plugin. How does your co-operation look like?

Nicolas: I had first written a script for Moodle 1.9, which then got updated to support Moodle 2.0 by Forrest Gaston from Indiana. The script then evolved into a proper plugin with the help of Mike Holtzer from Pennsylvania. I was lucky enough that both of them – and some others – were kind enough to share back their own improvements, so I could move on and provide the first versions of the plugin via the Moodle plugins database.

I was even luckier later when Jordi stepped in, as he is really motivated by this plugin and improved it tremendously. I think he's now responsible for most of the code present in the latest versions. As I don't have any background as a software developer/architect myself, I'm glad I can rely on Jordi's experience. I think that thanks to him the quality of the plugin code and UI have improved a lot in the last year.

Jordi: We co-operate mainly via the Nicolas’ github repository, and then with very few private emails for long term planning and about particular things that need a mutual commitment to show publicly a common point of view.

As a team, we have advantages like: a) peer-review from the other part, and then, a major quality of the product we, with other contributors, are providing; b) shared responsibility since, most of the times, we only need that one of us replies to any contributor or question.

Working with Nicolas is always a good experience, since he is really positive and decisive.

As the thread at moodle.org shows, the plugin started as a simple script back in 2008 to solve Nicolas’ need to merge two user accounts at his site. It has significantly evolved since then. What’s your motivation for further development and maintenance of the plugin?

In our University we had a recurrent issue on multiple users for the same person, since there are several points of user registration. The good point is that we are provided with a list of merging actions.

So, we had to add into our Moodle instance something that should address that issue once for all. Making a research on Moodle and the Moodle community, I found the Merge Request plugin of Nicolas that really looked pretty good. I really liked the way it behaved: a search and replace of user ids throughout the whole database. We could see it as a “brute force” search and replace, but at the same time, efficient. I decided then to make a series of contribution to make it more modular and extensible, configurable, with the goal of having a cronable script.

Since then, the plugin has received quite attention and other contributions that both Nicolas and me are reviewing. When they are ok, Nicolas accept the pull requests in github, and finally the downloadable version in moodle.org got updated with last version.

What software and IDE do you use when developing for Moodle and how does your typical screen look like?

Nicolas: I'm using NetBeans as IDE, git internally on the command line, and TRAC to keep track of Moodle issues. We also use Redmine to track our team's work, but as each member of my team works on different things, we don't use a centralized issue tracking system. For administrative tasks, I rely mostly on the command line, editing text with vim and accessing databases with native clients.

Jordi: I use NetBeans as IDE, and Gitlab + Gitlab CI as project manager and continuous integration systems respectively. In addition, I use command line for git commits and conflict resolution, with customized git scripts for issue management that also interacts with Gitlab to create merge requests and accept them. Gitlab, in addition to project management, allows us to provide peer review to what we are working with, prior to accepting changes. We used Redmine before Gitlab, but since then we have noticed an increment on our software quality. With Gitlab CI we ensure the expected behavior of our software and provide automatic software quality testing. Finally, we use phpmyadmin for an easy and graphical access to databases. However, in the real infrastructure, we have only access via SSH and then command line is the only way to interact with the whole system, using vim to edit files and native clients and customized scripts to connect to the databases.

What is your development workflow and how do you organise the code in terms of branching, tags etc?

Jordi: In our plugin, we only have, by now, a master branch, but we have planned to build several branches to make sure our plugin will work and then, separate backward incompatibilities. Since we use github for code management, development workflow is as usual.

In general, in our work, we have mainly four statuses for issues:

• Developing, in which we are solving that issue (bug, feature, enhancement, etc).
• Peer reviewing, in which some other colleague review other’s code and continuous integration is passed.
• Testing, in which some third person tests that what is added is what was asked for and what is expected.
• Accepted and resolved, in which that contribution is ready to be deployed.

Nicolas: As I’m working alone in day-to-day Moodle operations, my workflow is much simpler. Issues only have three statuses: open, in progress, and closed. This workflow is very flexible and open to interpretation, as I’m the one opening and closing the tickets, as well as deploying the fixes to our production servers.

Is there some bigger thing you would like to see included in your tool in a near future?

Yes! Testing! Unit testing and behat tests should be included. Since our contribution is in work time, we have the pressure of making things done in the shortest time, so that testing is, differently to TDD, the last thing to add. We know that it is a big effort, but that will add software predictability and quality.

What communication channels do you use to stay in touch with users of your tool? How can they can ask questions and/or report issues with it?

We use Nicolas’ GitHub repository as the project management, and then any request, bug or contribution should be reported there. However, we also reply to any question posted in both the forum page and the official Moodle plugin Web page.

If you were starting now from scratch, what would you do differently in your plugin?

Jordi: It is hard to say, because this plugin is really hard to design. It affects whole database by “merely” update a user id by another one.

To augment software quality and predictability, I would like to have seen an API from Moodle to allow changing ownership of the user’s activity. That way, since there are Moodle modules interconnected between them, each consistent part will really know how to update a user id with another one.

Since there is no expected feature in Moodle core and API, the Merge Users plugin has to include such know-how and process each database table properly. To do so, the Moodle community experience is necessary to add the correct behavior of our plugin in any single Moodle part, even in third party, non-standard plugins.

Above all, I would have prioritized on plugin settings (for local configurability) and efficiency (caching and sql statements performance).

Nicolas: It’s very difficult to say for sure. One thing that was clearly not done here was to assess the needs of the Moodle community before developing anything. I just had this problem with multiple user accounts to begin with, saw that no one had proposed a solution yet, and worked out some hacky script to do it. Then things and people kicked in and shared their improvements based on my first try. I’m very thankful for that, but I think that this is really (if you’re not very careful) a recipe for very bad software design. However, and I would say thanks to Jordi stepping is as a plugin maintainer, things stayed under good control.

So, if we were starting now from scratch I think we would probably take things the other way around: think of what we need in terms of features and configurability, and then design the plugin. I’m not so sure that the plugin interface would have looked very different, but I’m sure the code would have!

Jordi: Reading the Nicolas last words, I MUST agree on his final remarks. I would like to add that, doing that together right now, the result would be amazingly promising. And I would like to thank you Nicolas to have created this plugin and let me join to it.

Thanks a lot.

Let the code talk

The overall idea of the plugin is quite simple. Find all places where the core user table is referenced and replace one user id value there with a new value. When doing so, the plugin relies on certain implicit rules - or rather habits - common in Moodle database scheme. For example, it expects the relevant fields to be named like userid, user_id, id_user or user, which is valid assumption in many cases. Beside that, there is a list of exceptions. For example, the fields createdby and modifiedby in the question table are explicitly listed to be processed, too.

The tool takes care of some cases that require special attention. Typically, when the user id field is part of an compound unique index. These special cases are well described in the README file shipping with the plugin. The tool aims to support additional (contributed) plugins, too. There is no inter-plugins communication API/callback to handle this automatically. If the additional plugin needs a special care, it must be hard-coded into the merge tool.

Even quite powerful in its simplicity, this approach has obvious disadvantages. The knowledge about the relationships to the user table has to be manually declared in the merge tool's code. If it is missing, the given area is not merged and it may be hard to spot it. For example, the Workshop module tables are not handled well by this plugin.

Suggestions

I believe that what could help is to try and make use of some sources that already contain the required knowledge about relations to the user table. One of them may be the moodle2 backup classes where plugins annotate their data that hold the user id. The second one is the database scheme files in the XMLDB format.

XMLDB provides a way to declare foreign keys and this information is stored in the plugin's install.xml file. An example of such declaration is the following part of the mod/workshop/db/install.xml file:

<TABLE NAME="workshop_submissions">
    <KEYS>
        <KEY NAME="overriddenby_fk" TYPE="foreign" FIELDS="gradeoverby" REFTABLE="user" REFFIELDS="id"/>
        <KEY NAME="author_fk" TYPE="foreign" FIELDS="authorid" REFTABLE="user" REFFIELDS="id"/>
    </KEYS>
</TABLE>

This says that the fields gradeoverby and authorid in the workshop_submissions table should be considered as foreign keys to the user table. As such, they must be processed by the merge tool when transferring the ownership from one user to another. It's true that not all additional plugin maintainers pay attention to these aspects of the XMLDB file. But some plugins have this done right, and it's a good opportunity to use this information. There is an API for accessing information in these install.xml files - such as database_manager::get_install_xml_schema().

The plugin's code is organised into classes with custom auto-loading feature implemented. Unfortunately, the mechanism does not take eventual naming collisions into account. The plugin uses classes like Config or Logger. Not only the capital letters go against naming conventions in Moodle. More importantly, these classes are loaded into the global scope without the proper component name prefix. So they should be either renamed to something like tool_mergeusers_config or refactored to use Moodle namespacing rules. Checking for expected and correct naming scheme is something we pay serious attention when reviewing newly submitted plugins into the Plugins directory. It's not a secret that plugins are not approved unless they have this done correctly. This issue itself could be a good reason to actually start thinking of more advanced branching model for this plugin, that would allow to maintain separate versions for particular Moodle release. That would allow to modify recent versions so they can profit from the in-built Moodle features (such as the mentioned namespaces and autoloading).

Some areas of the code are not fully cross-db compatible. For example, when searching for a user, the plugin searches couple of fields for the given value - including the 'id' column too. As the 'id' column is of type BIGINT, the LIKE operator can not be used (without explicit casting) on Postgres 8.3 and later. In this particular case, I would simply suggest to remove the 'id' field from the searched fields (as is does not make sense anyway to have users with id 1426 or 87427 returned when searching for 42). As usually, testing plugins on at least MySQL and PostgreSQL platforms is warmly recommended - and it's actually something that the community can help with a lot.

At some places, the plugin seems to re-invent what Moodle core already provides for no obvious (or inline documented) reason. For example, $DB->get_tables() seems to be a better alternative to how MergeUserTool::__construct() populates the list of Moodle database tables.

Since Moodle 2.6, the icon i/tick_green_big.* is not available any more (see MDL-40369 for details) so the broken image icon is displayed at the merging logs page (yet another example of when the "one branch for all Moodle releases" model reaches its limits).

Conclusion

The Merge user accounts tool has been available for a while. Originating back to a simple script posted to moodle.org forum, becoming a report and later rewritten into an admin tool, it raised attention of several contributors.

There are situations when something is better than nothing. Even if the tool does not do its job perfectly and may leave some database records still referring to the old user account, the final result is acceptable and may be an actual win for the poor student with multiple accounts created. The maintainers do great job in describing the known limits of their plugin's functionality and leaving a log of what the tool actually did. I wish Jordi and Nicolas good luck and enough energy to keep the thing moving on. It's definitely on the right track.