Support Composer dependency manager

Support Composer dependency manager

by George Mihailov -
Number of replies: 31

In order to be able to install Moodle using Composer dependency manager composer.json in root directory should have some additional data. Now it used to declare Behat dependencies of Moodle itself. Composer uses tags so current mirror on github will work perfectly matching version tags. To work with branches it's possible to declare each MOODLE_NN_STABLE as special "continuous" version like 2.7.x-dev / 2.8.x-dev / etc. This means that you may switch between stability and declare 2.7 or 2.7@dev (see composer tutorial). Once composer.json updated everything will start working without need to cherry pick something.

Then command like this will fetch latest 2.7.x release and put it in vendor directory.

composer require "moodle/moodle:~2.7"


Next step is to configure how to expose the moodle core to serve. One way is to point web root directly to vendor/moodle/moodle. Another option is to use events like post-install-cmd to execute command that performs particular logic with sources. Actually there is one more option -- to develop own installer like it did Magento.

Why it is important? I see it mainly for automation. It would be tricky to fetch latest release in 2.7 branch using bash/wget. In same time it is easy to define particular version and stick with it. Composer comes with some infrastructure behind like Packagist. At the end of the day sources could be cached locally or fetched from github. This offloads Moodle servers and makes distribution more flexible. So far I don't see any disadvantages using Composer smile

There is an issue on tracker https://tracker.moodle.org/browse/MDL-48114 with pull request. I'm following Dan's advise and want to discuss this first to get an agreement. Please read the Composer documentation https://getcomposer.org/doc/.

Average of ratings: Useful (2)
In reply to George Mihailov

Re: Support Composer dependency manager

by Dan Poltawski -

Hi George,

Thanks for the explanation and following my advice wink 

In your pull request you are prefixing the branch aliases with dev- i'm wondering if that is correct for the stable branches? (I'm not quite able to tell because the composer docs always use that in their examples)

Average of ratings: -
In reply to Dan Poltawski

Re: Support Composer dependency manager

by George Mihailov -

Composer distinguish branches from tags using this dev marker. More details you may find here https://getcomposer.org/doc/articles/aliases.md#why-aliases-

In brief when you define a master branch it looks like this

{"require": { "bar/zoo": "dev-master" }}

In Moodle context it would be

{"require": { "moodle/moodle": "dev-MOODLE_28_STABLE" }}

Using aliases you may use branches like versions

{"require": { "moodle/moodle": "2.8.x-dev" }}

or simply like this using stability flag

{"require": { "moodle/moodle": "2.8@dev" }}

Read more about stability falgs https://getcomposer.org/doc/01-basic-usage.md#stability

Average of ratings: -
In reply to George Mihailov

Re: Support Composer dependency manager

by Damyon Wiese -
I don't see enough reasoning for "why" we should do this in this thread. Moodle is not hard to install with git - and git give you more (conflict resolution) etc.

Composer is currently only used for additional developer tools (phpunit and behat) and we recommend git as the best way to install / track updates. What happens to composer when you start adding new plugins etc and then you go to update?

There are other (easy) ways to install moodle for developers (try mdk) - but we don't need to support them all in core.
Average of ratings: Useful (1)
In reply to Damyon Wiese

Re: Support Composer dependency manager

by George Mihailov -

I think it's obvious that every task has the right tool for it and every tool might be abused. Git is about source code management and it's the best for today. Composer is dependency manager and is aimed to solve different set of tasks. Installation using git has some inconveniences:

  1. Repository is big and it takes time to simple fetch everything
  2. Exposing .git directory is potential security breach and useless disk space
  3. Why there should be conflicts at all? As a non core developer I want to use new release without any additional headaches. I guess conflicts are happening because of custom core changes?
  4. Composer heavily uses Git and of course don't try to replace it

Okay, this is criticism, now I'll try a constructivist approach.

Here is alternative scenario to manage dependencies.

1. Init environment and install

Let's wrap Moodle core in order to separate it from project configuration. Run in empty directory e.g. project-acme command above

php composer.phar require "moodle/moodle:~2.7"


This will create composer.json and install latest version in 2.7 series (micro improvement: in git I should spent some time to figure out what is the latest). Then proceed with database installation and web server configuration. Cool, everything works fine.

2. Now I want to install custom theme or module

In case when theme supports composer it's easy as this

php composer.phar require "gjb2048/theme_essential:~2.6"

Otherwise project-acme/composer.json should be edited in order to manually add the package and define dependency. Then run 

php composer.phar update


Repeat this step for every additional module.

3. It's time to upgrade the core!

Change ~2.7 to ~2.8 in project-acme/composer.json and run `composer update`. It will install new core and most important is that additional modules are not lost. So my project has explicit declaration of 1) core version 2) additional modules and their versions and it has a tool that is able to resolve this declaration.

--------------------

In case when official Moodle team don't want to adopt and promote this way of installation and maintaining why other developers should't have alternative? If we drop branch aliases it's matter of couple of strings in composer.json to be added into repository and nothing to change in current internal dev process. I believe it worth it.

Average of ratings: Useful (1)
In reply to George Mihailov

Re: Support Composer dependency manager

by David Monllaó -

Thanks for opening this thread George; as a dev I like composer and in general all dependencies management systems that helps me to align all dependencies versions using a single command although in any case I think there are many aspects to clarify before doing any step towards composer.

I don't know about percentages, but there are many moodle customisations out there, I mean that a lot of people, at some point, hacks moodle codebase adding their own patches on top of HEAD because moodle does not have any central dependency injection system to change externally how the system behaves, so using componser in a 3rd party development environment (I imagine a school or a uni) may not be a common use case.

Other aspects to consider; will we be interested at some point to move our own 3rd party dependencies to composer? (knowing that we should still provide download packages that should contain all that is needed to run moodle) Will Moodle plugins authors write their own composer.json? (would be awesome if all of them do it, but if not, the example you exposed above would not be that ideal) If there are other systems (not moodle plugins) using moodle as a dependency how they use it? they will need to ln -s a config.php to vendor/moodle/moodle for example...

I don't think that the official Moodle team is not interested in composer.

Average of ratings: -
In reply to David Monllaó

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello, everybody


I am considering using composer to manage the plugins installed on some Moodle installations.

For what I've read,  composer seems to be a nice tool and it already (should) support installation of moodle plugins types in correct paths as listed here https://github.com/composer/installers and described here https://github.com/composer/installers and here https://raw.githubusercontent.com/composer/installers/master/src/Composer/Installers/MoodleInstaller.php

Everything should be working, but it is not.
Is there anybody else using composer to manage plugins dependencies?

I've opened an issue on composer-installer's github repo at https://github.com/composer/installers/issues/214


Kind regards,

Daniel

Average of ratings: Useful (1)
In reply to Daniel Neis Araujo

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello, again


I've writing bringing good news =)

People from composer community helped me solve the issue with my composer.json files and now everything is working fine =D

Composer seems to be a good tool to manage moodle-plugins as dependencies and i've already started to do some pull requests (14 until now) to developers of plugins I use on production.

You can see it is a simple addition and developers may friendly accept as Dan Marsden already done for https://github.com/danmarsden/moodle-mod_attendance/pull/67/files

I don't know yet if Moodle should consider moving it's dependencies to composer but the automagic installation of packages could benefit from it ;)


Kind regards,

Daniel

Average of ratings: Useful (2)
In reply to Daniel Neis Araujo

Re: Support Composer dependency manager

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm not yet convinced by this.

I'm not sure i want to pollute my plugin with an extra file, if it only benefits a few people. (Could the composer file not go in the db folder? How does composer know when there is a new version, etc?)

Moodle has its own built-in plugin-installation system for people who want easy plugin install, and that checks things like Moodle version compatibility.

Average of ratings: -
In reply to Tim Hunt

Re: Support Composer dependency manager

by Brendan Anderson -

Maybe an individual plugin developer should not, but I think Moodle as a community should. The reason we should is because the built in plugin installation system is inadequate for automation. As developers we need a way to automatically bring up new development environments and deploy changes.

I'm not sure i want to pollute my plugin with an extra file, if it only benefits a few people. (Could the composer file not go in the db folder?

The people looking at the actual files in the plugin are developers who are familiar with composer.json files. They will expect to find it in the plugin root folder. Right now, not many people can take advantage, so you are right that it will only benefit a few people. But imagine if this was the standard for Moodle. We could all develop more quickly and deploy more reliably. That's a big win.

How does composer know when there is a new version, etc?

Composer knows when there is a new version because you tell packagist.org that there is a new version. This can be easily automated with GitHub.

I am still fairly new to Moodle, but I have been really surprised with the lack of interest in this type of activity. If I bring another developer onto my Moodle project and tell him that he needs to manual download all the plugins I use, or worse, checkout a monolithic VCS repo with nothing but code I didn't write, I will get straight up laughed at.

Average of ratings: -
In reply to Brendan Anderson

Re: Support Composer dependency manager

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

You have one background, and one set of assumptions, but other people know different things. E.g. I am a developer who is always looking at file in my plugins, but I have no clue about these composer.json files. Also, I have no desire to tell packagist.org about my plugins. They are all in moodle.org/plugins. (If you were interested to build an automated link from the Moodle Plugins directory to some other repository system, well talk to David Mudrak, but whatever you do should require zero additional work for plugin maintainers.

Co-incidentally, yesterday just before I saw your other thread, I had just finished reading this blog post: http://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html wink

Average of ratings: Useful (2)
In reply to Tim Hunt

Re: Support Composer dependency manager

by Darko Miletić -

Docker is a great thing but it requires different mindset and will to try something new. Having an automated cli way to download and install Moodle plugins would also help automating stuff. With that implemented writing a docker script for whatever kind of setup would be fairly trivial.


Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Support Composer dependency manager

by Brendan Anderson -

You are right. Composer may be new to Moodle developers. And now the moodle plugin directory exposes an API for getting information about plugins in an easy to parse way (so we don't really need packagist). I am not married to composer. Composer is a tool for solving a similar problem, but not the exact problem we have. After all we are not autoloading anything, we just want to download the files.

And composer will introduce extra work for plugin maintainers.

I think it is a viable option because it already exists and is the standard for dependency management in PHP.

But Drupal has been facilitating automation without it using drush make. It's a concept that I think can work for us, the idea being that the core version and plugin versions are defined in a configuration file which is parsed when the command is run and the project is built.

Average of ratings: -
In reply to Brendan Anderson

Re: Support Composer dependency manager

by Just H -

"But Drupal has been facilitating automation without it using drush make."

Haven't looked into it but there's a similar plugin for Moodle inspired by Drush; see Moosh.

Average of ratings: -
In reply to Just H

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Just to note, Moosh uses composer to generate new plugins from templates that are on github and have composer.json files =)

And although you can use Moosh to install plugins for you,

there isn't a way (at least i haven't found) to have a file

like composer.json telling everything you want and just

install eveyrthing with one click.

For what i know, Moosh only let's you install plugins one by one.

I think "you can have a script to do that" is not a reasonable answer.

I've used git repo and git-submodules in the past to try to handle better my colletion of plugins

acrros different Moodle deploys, but composer seems to be better,

even if you need to add a composer.json file or even clone the original code

to your repo to have this file on it while you waiting for the original maitainer

to get convinced of the benefits ;)

FInally, in "container" and "virtualization" times, I've asked people of digital ocean to offer moodle as 1-click-app : https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/7606407-offer-moodle-lms-as-1-click-app
And there is a docker image for Moodle (https://github.com/playlyfe/docker-moodle) although it not worked on my debian-jessie with docker 1.3.3 , but fortunately is my fault and not a bug ;D

Kind regards,

Daniel

Average of ratings: -
In reply to Brendan Anderson

Re: Support Composer dependency manager

by Jay Knight -

I wonder if there could be a composer-like way to enumerate third-party plugins that you'd like to install in your config.php file, and have the installation/upgrade scripts install them and keep them up to date.  This would be a nice way to keep test/dev/prod deployments how you want them, without having to manually install plugins either via the web interface (I keep that disabled so other admins don't go trying to install stuff) or unzipping things in the right directories and confusing git*.

* As a bonus, it sure would be nice for third-party stuff to not be strewn about in various git-tracked subdirectories (mod/auth/block, etc). A configuration option to point to a directory of plugins outside of the moodle web root would be nice, but I understand that would require lots of changes to how modules work.

Average of ratings: -
In reply to Jay Knight

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


just to note, mod Scheduler

https://moodle.org/plugins/view.php?plugin=mod_scheduler


have just merged the code adding composer.json

https://github.com/bostelm/moodle-mod_scheduler/pull/21


=)

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Support Composer dependency manager

by Henning Bostelmann -
Picture of Core developers Picture of Plugin developers

...and I have removed the file from mod_scheduler again.

Sorry, but when I saw the pull request, I was (foolishly) under the impression that these files were part of some agreed recent API change and were being added to all plugins. As it turns out, this was not the case and the debate is still ongoing. I find these tactics of "pull request first, discuss later" rather questionable.

Henning, mod_scheduler maintainer


Average of ratings: -
In reply to Henning Bostelmann

Re: Support Composer dependency manager

by Juho Jaakkola -

There is no "debate". You either:

  • Support composer and make life easier for developers and sysadmins
  • Don't support it and make their life more difficult

It's as simple as that. What is there to debate about?

People in this thread have used a lot of time and effort trying to explain why it would be a very positive thing if Moodle and it's plugins supported composer. I don't understand why their effort is being neglected.

Average of ratings: Useful (1)
In reply to Juho Jaakkola

Re: Support Composer dependency manager

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
To me it seems that composer is not the ultimate solution, but instead just several new problems for core developers. I do not think there is any simple solutions here because contributors seem to not have much motivation to keep up with all the API changes and new versions. Moodle was not designed to work with composer.

I have been personally in several positions working on Moodle codebase, and my personal opinion is: Please not another new versioning and related APIs!
Average of ratings: -
In reply to Petr Skoda

Re: Support Composer dependency manager

by Juho Jaakkola - 
it seems that composer is [...] just several new problems for core developers

What are the problems? Could you make a list of them? I'd like to take on the challenge to try to find a solution to each one of them!

Please not another new versioning

Composer does not require you to change versioning. You can use existing git tags or even individual git commit hashes to define the version you want to use.

Please not another new [...] APIs!

Composer does not require you to add new APIs.

Average of ratings: Useful (2)
In reply to Juho Jaakkola

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello, everybody


I also can't see real problems for developers. If you are a new developer (or an old one) you already have to use "grunt" and setup things for it to work.

If you write and/or run tests, you also use composer to install the dependencies.


Also, besides all the other things already said about how composer can help Moodle, there is two more worth mention:

"Update Notifications

To receive notifications about new version releases you can sign up for VersionEye, a web service that can monitor your GitHub and BitBucket accounts for composer.json files and send emails with new package releases.

Checking your dependencies for security issues

The Security Advisories Checker is a web service and a command-line tool, both will examine your composer.lock file and tell you if you need to update any of your dependencies."

http://www.phptherightway.com/#composer_and_packagist

This is not just for plugin developers but also for Moodle own dependencies (https://docs.moodle.org/dev/Moodle_libraries_credits). If instead of relying on a wiki page to document the used libraries, we could use the composer.json file (the history will be on git) and use composer to update libraries when needed and also use the above services to help us know when such update is needed.

Here are the links to libraries used by Moodle that are on packagist (same order as link above):


Considering that we have 30 dependencies (and some are javascript libraries, at least one is "Spike PHPCoverage" that has an outdated link) and we have already 13 here (half of libraries),  maybe composer is still not the ultimate solution, but it is getting there. All these people would not use it if it was just "new problems and versioning". Also, we could start thinking about the quality and future of libraries that do not adhere to these new trends.

You may also like to take a look at http://thephpleague.com/

Kind regards,
Daniel
Average of ratings: Useful (1)
In reply to Brendan Anderson

Re: Support Composer dependency manager

by Matt Gibson -

This is is a really good idea. The use case I'm thinking of is where the individual plugins need to pull in outside dependencies.

I've had many problems with putting libraries into plugins, which are then also included in other plugins, causing a breakage (e.g. a class is already defined). For example, having multiple versions of the PHPCAS library included by different auth plugins, or wanting to use Symfony components, which some parts of core already include, but at different versions.

Composer should be able to sort all of this out by making sure that a version of the library which works for all of the plugins which require it (and core) is added to the class autoloading path via /vendor.

This would also be a better way to include libraries in core rather than bundling the actual code (assuming the libraries are available via composer).

I would imagine that running composer during each upgrade run, after telling it where the plugin composer.json files are located should be enough to make this work. I'm going to experiment with it over the next few weeks by running composer during 

It would be trivial to have the plugins directory include information on how to add a plugin via composer instead of downloading the code, if that was a route we wanted to support. The composer.json file can include links to a github repo, rather than packagist.org, so no extra work for plugin devs, really. https://getcomposer.org/doc/05-repositories.md#loading-a-package-from-a-vcs-repository

Average of ratings: Useful (3)
In reply to Matt Gibson

Re: Support Composer dependency manager

by Juho Jaakkola -

Matt, how did your experiments work out?

I'm completely absolutely totally in favor of making it possible to manage Moodle sites and community plugins using composer. It would remarkably reduce the development and maintenance overhead of developers and sysadmins.

And as stated - it is a PHP *standard*. You learn it once and you can use it in any PHP project.

Average of ratings: -
In reply to George Mihailov

Re: Support Composer dependency manager

by Michael Aherne -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

I'm a bit late joining in this discussion, but I've been hacking on generating a proof-of-concept Composer repository for the Moodle plugin directory. It's at http://micaherne.github.io/moodle-plugin-repo/ if anyone fancies trying it. There's some documentation at that URL.

The basic functionality works well enough - add the repo to your composer.json, add requires for plugins (either * or specific version), run composer install and you should get your selected plugin versions installed in the right place.

There is very little real dependency management at the moment but it makes a fairly good way of distributing Moodle with a specific selection of plugins. To make it work better, MDL-48114 would probably have to be resolved. Also, we'd need to interrogate the code for each plugin to get better metadata for it, which could be quite difficult. The existing repo is just a simple transformation of the plugin database dump (the script that made it is at https://github.com/micaherne/moodle-plugin-repo).

I'd be really interested in any discussion on this idea, as I think there is definitely some mileage in getting Moodle and Composer working better together.

Average of ratings: Useful (4)
In reply to Michael Aherne

Re: Support Composer dependency manager

by Juho Jaakkola -

Very interesting! Thanks for working on this. I'm currently swamped with work, but I'll definitely take a look as soon as I can.

Average of ratings: -
In reply to Michael Aherne

Re: Support Composer dependency manager

by George Mihailov -

Michael, you've made a great job to prove that it's more than possible!

In my opinion the main goal is to manage dependencies by composer, just plugin installation won't have enough value. 

Today it's pretty much messed out: in plugin you have 10 digit version, then you (maybe) have a release and finally you have git tags and branches and in most cases there is no relation between them.

One option that I see is to have some sort of a script that can translate 10 digit of core version in plugin dependency declaration into normal release version (e.g. 2.9). Then in packages.json you put require: { "moodle/moodle": "2.9" } and once you change the core version, composer will manage to install appropriate plugin versions.

Average of ratings: -
In reply to George Mihailov

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


linking two new discussions about implementing features already supported by composer dependency manager:

https://moodle.org/mod/forum/discuss.php?d=328380 (optional dependencies)

https://moodle.org/mod/forum/discuss.php?d=324520 (plugins conflicts)


It seems that is starting to be quite a rework using a custom version.php fie instead of a composer.json file with composer tool.


Average of ratings: -
In reply to Daniel Neis Araujo

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


taking another look at Moodle's dependencies, I've found that from the 33 listed on the "libraries credits", 13 says "This library is not currently used in Moodle", so we get only 20 libraries depencies being used right now.

From my last post here, there are 13 packages available on packagist.

I've also found other ones:


That is, there is only  two package dependencies (lib/graphlib.php and lib/horde) that are not on packagist.

I think it is time for Moodle to consider this better approach to dependency management =)


Kind regards,

Daniel


Average of ratings: Useful (1)
In reply to Daniel Neis Araujo

Re: Support Composer dependency manager

by Dan Poltawski -

Daniel, you are talking about a separate issue than the original thread (supporting moodle being installed by composer).

With regard to embeding third party libraries using composer there is MDL-49672 open discussing that. But I personally don't see a noticeable change to do this from an external POV:

  • Moodle has always bundled the required third party libraries. To stop bundling them would cause every single Moodle site in the world to do work to support the upgrades. I am not in favour of making Moodle upgrades harder..
  • The composer solution is nice to avoid having to bundle so much and think about it, but our Moodle repository structure puts everything in the web accessible directory. Not all composer packages are nicely laid out for that (i've found some with shell scripts there).
  • As Damyon mentions in the issue, often the library inclusions require some 'smarts' for our integration and are not straight embeds 

Its possible we might find ways to ease  upgrade process using composer in the future, but depending on composer for installations is a no no. 

Average of ratings: -
In reply to Dan Poltawski

Re: Support Composer dependency manager

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello, Dan


thanks for pointing the correct issue. I was really not aware of that one =)


"Moodle has always bundled the required third party libraries."


Ok, I agree with that, but this is not a reason to not change. There is several advantages in not "bundling" libraries together with your code (of course, there is some disadvantages too, as the recent case with the npm repository and a "rebel" maintainer, but these are most isolated examples).


Also, Moodle has a "command line upgrade tool", that can, of course, run the "composer update" command if that is too much for every moodle admin out there.


About the packages with shell scripts and everything else, well, this depends on what you are including. I am pretty sure all the cited Moodle core dependencies should not do anything like that. Also, you can tweak your webserver to deny thing in "/vendor" directory.


About the "not straight embeds", using composer, you can fork the libraries that moodle changes and use the forked libraries instead of relying on *_README.txt files to explain what was done and also keep the moodle core code history more clean.


Ah! At last but not least, let's not think our users/developers/admins can not do things. We need to raise the bar sometimes so people keep improving =)


Kind regards,

Daniel

Average of ratings: -