Hi Alistair,
Thanks for the feedback. I've tried my best to respond but I'm also going to ask for more information so that we can do our best to ensure we don't realise your fears.
I think the short term goals of moving some of the non-code elements out of the web exposed folder is a good move that improves security.
Great - let's keep up that positivity ;)
Some of the longer term goals seriously concern me however.
D'oh! Okay, let's go through things and see how we can help.
I think introducing a build stage would be a massive regression. We have to deal with that with Mahara and it nukes the ability to use git to manage the codebase. While in Mahara that's not that big of a deal because we might have 2 plugins and that's it, in Moodle we have dozens of core hacks, and 100s of plugins continuously updated via git.
It would be really helpful if you could explain why you think this will be a problem. From my point of view, there shouldn't really be any change to this -- that is, it will still be possible to use Git to manage your local plugins and customisations if that's your preference.
Firstly, we're not going to be making all of these changes in one go.
We'll be introducing them gradually starting with using Composer to install the third-party libraries currently installed in the lib directory (core). Unless you have local customisations to these libraries then there should be no change here.
Other steps (in no particular order) may include:
- Building JavaScript for plugins; and
- installation of Moodle plugins and their dependencies through Composer.
In the case of the JS builds, I would argue that this will make your workflows simpler because you will only have to maintain customisations to the source JS, and not need to handle conflict resolution on the built files.
In terms of the Moodle plugin installation via Composer, there are still a few details to work out. It may be possible to have both options -- that is to install plugins using Composer or as git submodules (or similar). I'm not entirely sure yet. It is definitely possible to get Composer to install packages into the relevant Moodle directories for each plugintype and for those to co-exist with existing (Git-installed) plugins. However, if you have a plugin which has composer dependencies I don't know whether Composer will resolve and install the dependencies for any plugin installed using git submodules.
Which similarly moving to managing plugins via composer instead of git, and having all plugins in one folder, I'm sorry I just don't get why that's a good idea.
I know random PHP devs have asked about it when joining the Moodle world but that doesn't mean it's a good idea for the existing users of Moodle who install and run the Moodle installs on a wide range of very minimal and very complex infrastructure.
So these are two separate things.
With installing plugins via composer the main benefit is that it allows plugins to use libraries more safely. At the moment any plugin can include any libraries it likes. Many developer use Composer to do so and include a bundled vendor directory in their plugin and load their copy of the Composer autoloader. That's works fine... until another unrelated plugin wants to use a different version of the same dependency. In this situation it's possible to get into a broken state for one or both plugins depending on the load order of those dependencies. We have seen this issue reported in community plugins. It has also been relatively common to see extremely large libraries installed, and these can go stale really quickly.
Supporting the use of composer in plugins from core means that we can use the Composer tooling to handle this case and to identify incompatibilities between Moodle plugins before it becomes an issue.
On the topic of a single plugin folder for Moodle I'll turn the question around and ask why we need to have 45 different plugintype directories. Why is that necessary? Historically it's probably at least partially because we want to have hierarchical URLs like mod/forum and so on, but with recent support for routing this is no longer a requirement (not that it ever really was).
The multitude of components also really impact adoption of Moodle by new developers, and adds unnecessary complexity because every plugintype seems to do things differently. Some have a lib.php, some have a classes/text_filter.php or various other magically named files, and it's very confusing -- especially for new developers.
I've been looking at an approach that I've nicknamed the single simplified plugintype for Moodle whereby you can have a single plugin (for example moodle_forum) which has a number of features (a forum activity, an 'Announcements' block, a 'Hot topics' block, and a couple of reports). Rather than each of these being separate plugins which have to be installed and kept matched and dependant upon the main forum plugin, but technically not allowed to use the internal APIs, JS, etc. we can have a single plugin with one set of JavaScript which is shared, and one set of APIs which are internal to the plugin and don't fall foul to our Component Communication Principles rules. Similarly it doesn't stop someone from creating plugins that interact with that forum, for example a 'Forum mentions' TinyMCE Editor plugin, a 'Forum mentions' filter to go with it, which are bundled in a single plugin and depend on the moodle_forum plugin.
Where I'd really like us to get to is:
- a single plugin type for all plugins
- located in a single folder
- which is not web accessible (it's not in the public directory)
I feel that this long-term goal allows us to build a more structured plugin system which kills off the old lib.php files, requires autoloading for all parts, makes use of routing, provides a consistent and well-documented approach to plugin development, etc. but also keeps the different inter-related parts of each plugin consistent and internal to that plugin. It makes it easier for new developers to on-board, and should make it easier for existing developers in the long run too.
I do appreciate the argument for wanting to better manage libraries in core and the workload / self-imposed limits though, so maybe there are ways of ensuring both ends of this can work optimally. We technically already do have a build process as a pipeline goes and clones the latest code and we zip it before sharing it around for unit testing & deployment, so if was limited to a couple of extra commands before compressing the folder, without the need to have a bunch more things installed on the prod servers to do the building then that could potentially be reconciled easily but it does need further consideration - but with Mahara it really wasn't that simple from what I can recall.
Ultimately it depends on your setup but your approach is exactly the kind of system that I was expecting. For most larger organisations, partners, and so on I imagine that you already have a build process to take the core product, and various plugins, and turn it into something for deployment.
Ideally you simply add a new build step into your existing build process which just runs the relevant composer install and node commands after fetching the latest code, and before any testing and zipping it up. Theoretically it's just a case of composer install at the right part of the process.
When are you planning on making these changes? I had assumed the public folder part was going in 5.0 being the big version where breaking changes / tech debt clearance take place but re-reading now there is no actual timeline listed and looking at the main branch / 5.0 beta it doesn't look like this was implemented.
The public folder part will be in 5.1. I was hoping to get it into 5.0 and it was ready but trying to get feedback from people takes time, and we decided it was safer in this instance to wait until Moodle 5.1. If you see the forum post you replied to there is a section titled "📅 When Will This Change Happen?" which notes this timeframe.
I also get there's brainstorming in the mix but are we seriously planning on creating a UI layer in React?
Again I know some devs have asked about it / done that in their projects but is that not an edge case?
Nothing is set in stone, but we are considering how we best manage the UI side of Moodle. A JS Framework for frontend is one of the more popular approaches (no particular framework selected at this time). I don't feel that this is an edge case, just not something that many Moodle developers are familiar with.
Best wishes,
Andrew
