Moodle in English: Plugins depending on other plugins

Plugins depending on other plugins

by Tim Hunt - Friday, 6 June 2014, 8:07 PM

Long introduction

Generally, in Moodle, we like different plugins to be completely independent and self contained. This is nice and simple, but not always possible, and there are various ways plugins can depend on other things. For example:

Well, to start with, no plugin is independent, becuase they all depend on Moodle core, and this is completely fine.
Then, again, when you have sub-plugins, like Quiz access rules, or Assignment submission types, which necessarily depend on the parent plugins like Quiz or Assignment module. Again, this is to be expected.
Now we start to get to more messy scenarios
Sometimes, you have an add-on that only makes sense if another add-on in installed. For example
- Two of my add-ons. Report workflow is only useful if Block workflow is installed.
- auth_cas requires auth_ldap to be installed, becase CAS extends the functionality provided by LDAP.
- filter_glossary requires mod_glossary, again this makes sense.
A few years ago we introduced $plugin->dependencies in plugin version.php files, to make this explicit, and so that the Moodle install system could make sure that you did not install an add-on without the other required add-ons. This case is now pretty well supported.
The final scenario is in the other directly. What if other plugins need to provide some code to work with this plug-in, if it is present. The topical example is Report builder, which they are planning to add to Moodle 2.8. The rest of this post is about this case, which is not currently well supported by Moodle.

Further summary of the problem

So here are some examples of the problem:

The new report-builder tool requires each plugin to describe its data, if data from that plugin is to be included in Report-builder reports. This is done by a class in mod_choice that makes the required information available. This class has to extend a base class in tool_reportbuilder.
I have an add-on, report_editdates, which lets you edit all the date settings in your course on a single page. This requires a class for each activity that we want to support, like report_editdates_mod_forum_date_extractor. At the moment we store all those classes inside report/editdates, but that is not extensible. It would be more flexible those classes could be stored inside each separate module. (Though we might still want to keep the classes for core mods inside
report_editdates.)
Another similar example is the Combined question type, one of the OU question-type add-ons. This lets you build one question out of several other, like qtype_multianswer/Cloze, but better. One way that it is better is that it is extensible. Another qtype can make itself addable to qtype_combined by defining some classes in a folder like question/type/myqtype/combinable, for example in pattern-match.

That last issues shows, that really we don't need anything special in Moodle core. You can just get on and do this, by making up your own rules. The point is that it is better for Moodle if we can come up with a standard way to do this for all similar situations.

The other point is, we now have a system of automatic class-loading, and we want to work with that. There were recently some rules defined about that, and we should fit in with those rules. I don't think the integrators have written up what was agreed in MDLSITE-2549 yet, so let me summarise.

Classes go in a path/to/my/plugin/classes folder. (For plugin with Frankenstyle name type_plugin.)
Classes can just go in the root of that folder, in which case the auto-loader will try to find the class type_plugn_class_name in the file path/to/my/plugin/classes/class_name.php.
Alternatively, you can use PHP namespaces, in which case the class \type_plugin\sub1\sub2\class_name is looked for in path/to/my/plugin/sub1/sub2/class_name.php.
In the name-space case, there are rules about what is allowed for sub1. Basically, you are only allowed to use the names of core components like event, backup, ... there, plus you can have sub1 == local, and inside local you can do whatever you like (so it is like the locallib.php file that a lot of plugins use).

Proposed solution

So, my proposed solution for the case where other add-ons want to optionally integrate with other plugins, is to extend the rules for what is allowed at leave sub1. Extend the rules so that you can use the frankenstyle name of another plugin there, just like you can refer to core APIs. So, the examples above the classes you would need to create to make the integration work are, for example

\mod_choice\tool_reportbuilder\data_description (in mod/choice/classes/tool_reportbuilder/data_description.php)
\mod_certificate\report_editdates\date_extractor
\qtype_pmatch\qtype_combined\renderer.php and \qtype_pmatch\qtype_combined\combinable.php

The plugin that looks for these classes would just find whether another plugin supports it with a class_exists call, and we already have ancore_component::get_plugin_list_with_class() API.

Rather than doing this at the sub1 level of namspacing, we could instead declare a new sub1=plugin, and then put the name of the other plugin at sub2 level, like \mod_choice\plugin\tool_reportbuilder\data_description, but I think it is impossible for the Frankenstyle name of an add-on to be the same as the name of a core component, so I prefer the flatter structure.

The other proposed solution

The reason I make this long boring post now, is that earlier, the Integration team had a secret meeting and came up with their own proposed solution, which so far they have only publised in a comment on a tracker issue https://tracker.moodle.org/browse/MDL-30193?focusedCommentId=293371&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-293371.

In short their solution is to define a new core API 'reporting' just for the use of plugin tool_reportbuilder. That is all very well for something like report-builder, which is going in to core, but it does not work for add-ons like Edit dates or Combined question types.

One of the good trends in Moodle in recent years has been increase the power of what add-ons can do without requiring changes ot Moodle core. The decision the integrators just took seems like a backwards step to me. Please reconsider.

Average of ratings: Useful (12)

Re: Plugins depending on other plugins

by dawn alderson - Sunday, 8 June 2014, 12:25 AM

understand.

Yes read the following:

https://tracker.moodle.org/browse/MDL-30193?focusedCommentId=293371&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-293371.

Sounds like a bit of streamlining. which I totally get, course could be wrong-teacher head/student head in mind here, am thinking in relation with for example, Accessibilty/UI/activities/resources/ modules etc-in terms of plug-ins/add-ons and compatibility affordances for L&T.

I do think, any compromise in terms of module functionality, for example, which I think I understood within Tim's post-then I will just add, if this is led in the wrong direction, it would be disappointing.

Cheers,

Dawn

Average of ratings: -

Re: Plugins depending on other plugins

by Simon Coggins - Monday, 9 June 2014, 4:22 AM

Great post Tim.

I think that your proposal is more flexible/extensible and I can't think of any obvious downside. Good to see some other concrete examples of where it would be used too.

Simon

Average of ratings: -

Re: Plugins depending on other plugins

by Eloy Lafuente (stronk7) - Monday, 9 June 2014, 11:21 AM

Hi,

nice post, and not boring at all. Really useful to put everything in context. Thanks!

Said, that, I can imagine my answer is going to be longer and more boring than your initial post, so here it's my sincerest apology in advance. Also, I'll be trying to split the information into "chapters", so everything gets more organized and readable.

The secret meeting

First of all, for the sake of transparency, let me share that we (the integration team - iTeam), after every major release, use to have a meeting, sort of retrospective, where we discuss about practically everything (in fact that discussion happened in the 5th session of the meeting, spread along the past 3 weeks, so you can imagine how long it is).

While those meetings are happening, new points are added to the agenda and that’s how the discussion about inter-plugin communication landed there (last point, if I’m not wrong), in the context of the report builder and its integration into core. I think the context is pretty important, please, don’t forget it.

The discussion was basically about clarifying some concepts of the report builder (that personally I haven’t “tasted” yet), and then, about which should be it’s ideal (in our opinion) way to land to core. Of course, constructive, looking-forward, open-minded (I think that term was exactly used) and not final. There is only one dictator in this Community, and the iTeam is not him.

And then, yes, we communicated it wrongly. More explanations and details, a proposal for discussion… were needed. So all I can say is sorry and that we’ll try to do it better next time.

In any case, that does not change much the opinion about how the report builder case should be integrated into core, let’s talk about that (at last!).

Components, plugins, subsystems and APIs.

Just to be sure that we are all taking the same language, I think it’s needed to briefly comment about all those concepts. While they are pretty simple (in theory), I see them continuously used/replaced wrongly here and there (for example the “event”, “backup”… in the initial post… are not components at all, but APIs). And that leads to confusion.

Components, (covering both plugins and subsystems), are the original effort to organize Moodle into differentiated areas. That included some agreed naming schema (the famous Frankenstyle) and some shared functionalities: support for own lang strings, (originally) support for own database schema, install & upgrade, prefixing all constants and functions, phpdocs @package, renderers support… and few more niceties.
Plugins (and subplugins), for sure the most famous components. Easy to identify, they are grouped into plugin types (auth, enrol, blocks, activities, filters… and many, many more). They are physically confined within a directory and, as far as they support easy installation & upgrade really good for everybody wanting to add/extend functionality to Moodle. So, there it is. The Moodle plugins database. Worth commenting that, no matter they all are plugins… since some months ago it was agreed that we were going to call “plugins” to the plugins distributed with Moodle releases and “add-ons” to the plugins not distributed with Moodle. All them are plugins, just we call them differently.
Subsystems, the 2nd type of components. I really don’t want to comment much about these ones, just imagine them as some sort of organization with some extra functionalities as commented above. My very-personal POV (unrelated to this discussion) is that, along the time, they should be phased out and just have plugins and APIs. But that’s another story to be written in the future. So keep them apart, please. For the rest of the post I’ll be using the “plugins” word exclusively.
APIs, first of all, they are not components. Ignore if there is any overlapping with the Moodle subsystems (some of them are APIs really) and everything you know about components. APIs are the core-core or Moodle, and they define how everything else must work and interact. Of course, they are legion, because Moodle is a big-mammoth (that’s one of the best secrets revealed, the real meaning of the “M”, LOL).

They define how things must work at all levels and provide the rest of Moodle (other APIs, plugins and add-ons) with a specification about how to perform a lot of tasks (install & upgrade, database access, files handling, langs support, logging, ws implementation, testing…) and also provide some functionality to be used where necessary (grades, comments, rating, rss…).

Finally, worth note that those APIs are becoming more and more important (in detriment of the subsystems, I’d say), and we gave them a really prominent position into the Namespaces agreement, reserving the whole “level2” for them (plus “local” that was decided to be allowed for plugins own stuff at 3rd level).

Basically the rule is that any namespaced class (apart from the “local” bits and bobs case commented) must, always, follow this simple scheme:

component_name\[api_name\]class_name

And this is all we need to know to continue…

95% of agreement

Yes, no surprises. I can say, without a problem, that I agree with 95% of the contents of Tim’s initial post. Indeed:

All plugins depend of core. Just let me add “and its APIs”. I suppose nobody will object.
Subplugins depend of their parent plugins. Correct, nothing to object. A plugin is responsible to control its children.
There may be “dependencies” between different plugins. Can be architectural ones (extension) or data-access ones. Note I’m not 100% happy with how we use to perform that “raw access” (database, files…) between them and IMO it should be more controlled but, for its simplicity, and being proved to work, it’s accepted. Plus we are able to control such dependencies at install/upgrade. Ok.
A plugin provides some functionality to other plugins by its own (aka, bypassing, core). Beep, beep, disagreement! Indeed!

I really hope that, at this moment, it’s clear for everybody that the 4th point is the controversy one and everything else, that already covers a lot of inter-plugin dependencies, are not a problem at all.

Let’s elaborate about that point a bit more by commenting about your 3rd case in the “Further summary of the problem”:

Another similar example is the Combined question type, one of the OU question-type add-ons. This lets you build one question out of several other, like qtype_multianswer/Cloze, but better. One way that it is better is that it is extensible. Another qtype can make itself addable to qtype_combined by defining some classes in a folder like question/type/myqtype/combinable, for example in pattern-match.

Pretty interesting, because it defines a new case, not explicitly defined in the 1-4 list above, but being a particular case of the controversial 4 (or perhaps a new case of 3). A plugin of a plugin type (qtype), combining with another plugins of the same plugin type (qtype). For that, you can use 2 approaches. Or you implement such combination abilities by controlling it at plugin type level (sort of API within the constraints of the plugin type). Or you implement that ability between plugins, straight between them, bypassing completely that qtype commons API.

Personally I’d prefer to bring the functionality to the (higher level) plugin-type level, so you can define there what is needed in order to allow that work for a given question type. But, no matter of that… still… I’d say. No objection there. As far as a given functionality is within the limits of a plugin type (qtype), dependencies can be implemented as you prefer.

So let’s add that point to the list of agreements and forget about it:

4a. A plugin type, constrained within its limits, can implement inter-plugin dependencies without a problem.

And, obviously, that keep us with the disagreement way more reduced:

4b. A plugin provides some functionality to any other, unrelated, plugin.

Do we agree that’s the problematic one? Hope so.

Ok, I’m going to sleep some good hours (05:00 AM here is really insane)… in the mean time it would be great if we all could:

Erase any previous knowledge / experience you can have about / with the report builder. That’s my current situation. And IMO it’s the key to see the forest behind the plugin current incarnation (the trees).
Have you been able? No? Try again, please. Thanks.
Great!

… to be continued… I warned it was going to be long, ciao

Average of ratings: Useful (3)

Re: Plugins depending on other plugins

by Marina Glancy - Monday, 9 June 2014, 12:59 PM

Hi Tim,

personally I would also like to encourage structured plugin-plugin communication which at the moment exists in the form of callbacks. I think you are aware of policy issue MDL-44078. Hooks were one of the suggestions for the problem, you may want to suggest your approach there as well, it sounds very reasonable.

At the same time I tend to agree with the majority of other integrators that in case of reportbuilder we should better introduce a new API because we expect almost all modules and many other plugins using it.

Average of ratings: Useful (3)

Re: Plugins depending on other plugins

by Tim Hunt - Monday, 9 June 2014, 11:43 PM

I agree with you about hooks. In fact, I think that now we have automatic class-loading, there is almost nothing left to implement in order to have hooks. It is really just some conventions about what we call certain types of classes that are used to communicate between core and plugins, and between plugins; and some helper functions to simplify with calling those classes.

(I must admit, I have not re-read the hooks spec recently, but I hope that is what it says.)

Average of ratings: -

Re: Plugins depending on other plugins

by David Mudrák - Monday, 9 June 2014, 5:01 PM

Thanks Tim for raising this up, and everybody else for the replies. In my opinion, this issue has several layers, or dimensions.

With a project of size like Moodle is, decision making may take time - and for a good reason. I believe that a policy issue in the tracker that would give our dev community time to gather enough arguments for a decision to be made, is the way to go. And it did not happen until now in this thread, as far as I know.

As Eloy admits, the iTeam could have really communicated things better - instead of announcing radical decisions like "We do not allow one plugin to define API that is used by another plugin". It actually does not make much sense in the light of Tim's summary. Regardless what our lovely integrators do or do not allow, these implementations have already been there for ages. And they appear to be vital so an attempt to standardize things even more is warmly welcome.

In this particular case, in the context of the reporting feature, I do not actually see things either white or black. As reporting is supposed to be essential feature of future Moodle versions, it only makes sense to come up with an API for it (which could finally clean up activity modules' lib.php from undocumented yet required callbacks we have now). So I have no objections against introducing a new core API for reporting which would then be reflected in the namespacing rules.

Said that, I really like Tim's proposal for the recommended inter-plugin communication scheme. It's not the first time I personally hit this use case. And I do not think there has been any evidence shown so far of why allowing it would be bad. The only thing I am not sure about is using the API provider's frankenstyle or the word "plugin" as the root node for these classes. What about something like "ext" (as opposed to "local") so that the namespace would read like

\mod_choice\ext\tool_reportbuilder\data_description

making it clear that this class implements a service provided by another plugin. I could also imagine "xpc" (for "cross plugin communication") to make it even more cryptic . If I had to prefer one of original Tim's proposed locations, it would definitely vote for the flatter structure (without the common "plugin" root) so it looks consistent with core APIs (in fact, if all the core sub1 levels had the core_ prefix, we would see nice list of frankenstyle components there).

Average of ratings: Useful (2)

Re: Plugins depending on other plugins

by Tim Hunt - Tuesday, 10 June 2014, 12:18 AM

I think my concern about trying to define a core API for reporting is this: At this point in history, it is impossible to get it right. It is not me saying this, 'Speculative Generality' is listed as one of the 'code smells' on page 83 of Refactoring: Improving the design of exsiting code by Martin Fowler. It is also sometimes called 'Premature Generalization' but others. Chris Atwook also refers to it here: http://blog.codinghorror.com/code-smells/

I certainly agree that, if it were possible to have a single API which all plugins would implement once, to supply all the metadata requried by all other reporting plugins, then that would be a very good thing to have. But then so would world-peace, and have have not cracked that one yet either.

The recommedation of the various writers about software design is that it is only when you come to implement something for the third time do you try to generalise. (E.g. Refactoring, page 58 "Three strikes and you refactor", attributed to Don Robers. See also http://c2.com/cgi/wiki?ThreeStrikesAndYouRefactor.)

If you shoot for trying to build a generic API now, your are higly likely to just end up with an overcomplicated mess that is not a perfect fit for what report-builder is trying to do, and that mess will be sitting in the report core API namespace, getting in the way of doing something better. I really think that you should concentrate now on building the API that is exactly what report-builder needs, as a report-builder API. If the need arises, we can think about generalising later, once we know if and how generalisation is necessary.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by dawn alderson - Tuesday, 10 June 2014, 1:36 AM

Hey,

It appears to me then that none of this impacts on L&T-so no drama there, great.

Tim, where I referred to disappointing, you refer to mess-and I think if we work with a 9 degree radius of agreement (this is the 9th post-I think) we are probably connecting in some way with that

OK am going to bullet some points here, which may be some interest for reflection-or not....

1. Quality-in streamlining/single API/how plugins are submitted: could all do with tightening up-NO?

2. Theory and practice (ideal world...akin to world peace) all variables need to be constant for effective application-Yes?

3. Theory Vs practice (pragmatics) context-bound for effective application

Point: maybe there is a comfortable midlle in there somewhere.

cheers,

Dawn

Average of ratings: -

Re: Plugins depending on other plugins

by Eloy Lafuente (stronk7) - Wednesday, 11 June 2014, 9:08 AM

Hi,

I was going to continue my "saga" with another long post about how some of the current APIs have found its place within Moodle, but your post about generalization has really saved me writing (and you reading), such a long beast.

About generalization, I agree early generalization does not use to be a good idea. Completely. Something cannot be generalized before specification, knowledge, scope and impact (and surely more things) . And I'm sure all we, beloved hackers, have generalized badly in the past. Whenever you see some over-engeeniering... 80% of times it's early generalization. Or, for the rest of the world, to kill a fly (poor animal) with cannon shots (Spanish saying).

But I cannot agree about your comments about generalization applied as a mantra to any development.

IMO generalization applies to systems (areas) growing from scratch where, later, duplication and repetition lead to generalization (that’s the way many of our APIs have landed to core, say comments, ratings and tons more). Yes, I know this is a simplification, lol, just showing a usual path.

But there are other areas that, by definition, must be generalized since day 0 (database access, authentication…) because they, simply, must exist on any system and their specification and scope is well defined.

And, of course, there are other areas that already are generalized enough (because they have had its own evolution), but in the incorrect place. And that is, I think and hope, the case of the report_builder.

In my previous post I asked to forget any previous knowledge about the report_builder, and that way, these are my thinkings:

I assume that the report_builder already has passed those early simple cases, duplication, repetition… and generalization (and hopefully optimization) has happened to it (within add-on-land restrictions).
This is just a case of where such already existing generalization should land.
And the answer is, yes, core API
The builder (consuming definitions - I think they are called sources - and building reports) is not a report, it's a tool (call it tool_reportbuilder). Database-less. It's only a UI using the reporting API. Pretty much like the xmldb_editor (completely dumb, using core stuff like xmldb, dml and ddl, sql generators… intensively).
The sources can be spread as we already spread a lot of things. I don't mind if they are XML files, php structures, classes or .yml files. I don't mind if they are loaded into some database structure, versioned or no (note I don't mind because I don't know). Final implementation will depend of their uses, indeed.
The reporting API will be the responsible of handling those sources, validate and combine them. Will be the responsible of building storing the reports, of publishing (injecting) them observing perms and of course, also of executing them.
Programatic access to all the responsibilities above, if created within an API will guarantee new functionalities like converting current reports to reporting reports. Or access via Web services, to the whole machinery, enabling support to things like on the fly "moodle-use-this-source-definition-build-a-report-this-way-and-give-me-the-information-in-xxxx-format”.
Anything else you can imagine (reporting_formats - renderers?, reporting_exporters, reporting_forms?..).

All them only can be implemented if it's an API in core. Cannot be done being an API in a plugin.

Back to planet Earth (from my thinkings).... in summary:

What’s the real advantage of having it @ plugin level?

Just to show some disadvantages, here there are a bunch (real ones, I think), of having it @ plugin level:

Never used before. Fear to the unknown.
New namespace levels and rules are needed to implement it .
Way more interdependencies. Can you imagine them graphically? I can. Just draw some rectangles and start throwing lines, lol.
Core cannot use that API. That would introduce a inverse dependency not allowed in the 1-2-3-4 cases agreed initially. What if we want comments, grades or logs to be sources for the tool? Those uses don’t sound crazy, but logical. What about them? core->requires->plugin? Are you serious?
Everything under a plugin API only can be implemented via subplugins. And only the (unique) “parent plugin” can access/use/control them. Way restrictive! Let’s make everything a 1st level plugin citizens so everybody can access to them (reporting formatters, reporting exporters… whatever).

And those are my reasons for the exact case of the report_builder. Once more, all those thoughts are based in my complete ignorance about its current implementation. Just hope its next incarnation, in core, will the best for all possible uses.

All I'm doing is imagining how it should look once promoted from add-on to core and for me it's a clear case of being an API since inception/landing. If it's correctly structured already (and has being generalized properly) it should not be so complex. I know, I'm optimistic. We have 3.5 months to achieve it.

Finally, about the plugins-intercomunication... and the mother of the disagreements (joking!):

4b. A plugin provides some functionality to any other, unrelated, plugin.

I must say that I don't oppose as a principle. But, still, I need to be convinced with a good case where it's really needed. And not a product of over-reponsabilising (does that word exist?) a plugin.

Ciao

Average of ratings: -

Re: Plugins depending on other plugins

by Marina Glancy - Wednesday, 11 June 2014, 10:13 AM

List of plugins that ALREADY introduce their mini-APIs that are implemented by other plugins:

block_course_overview (asks modules to provide information for the block)
block_recent_activity (asks modules to provide information for the block). This is combined with very similar callback in core for "pseudo-report" course/recent.php
report_participation (asks modules what log entries they consider to be "participating", deprecated in 2.7 with introduction of event->edulevel)
report_outline (again, asks modules to provide info)
tool_log (allows reports to notify if they require access to logs)

Actually now after all this discussion and looking closer to other instances in code, I'm not so convinced about introducing new core API for reportbuilder. Why should it be any different from other reports/blocks that we have?

Or let's create new core apis for all of them now: reporting1, reporting2, reporting3, etc.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Eloy Lafuente (stronk7) - Wednesday, 11 June 2014, 10:30 AM

Hi Marina,

About all those "ask modules" examples... really the problem is that we are lacking a real activities API. It's one of the parts of Moodle missing modernization, object-oriented switch... and other niceties. We have way better architectures in any other area than in activities. Funny coz they are the "main" plugin types in the learning experience. Don't get me wrong, they just work, properly. But I would not list any of them as a good example to be followed by any new component, API or intercommunication attempt.

Also, when logging landed I promised i was not going to talk about it, so allow me to avoid any comment about tool_log... or better, no. It's the best example of something in the incorrect place. I'm just trying to prevent that to happen again.

Finally, I'm not negating it's possible to implement it being plugin-centric. I'm sure it's possible, in fact that's its current organization. I'm just arguing it's not the best given its complexity, its relation with other core APIs and its probable evolution.

My very-personal POV, nothing more. Ciao

Average of ratings: -

Re: Plugins depending on other plugins

by David Mudrák - Wednesday, 11 June 2014, 10:15 PM

Why should it be any different from other reports/blocks that we have?

Because it sucks, and we know that. Activity modules are not only flag ship plugins we have. They are also the oldest ones we have, and it is apparent at many places.

The fact that some other plugin requires my module to implement a function, or it throws an error otherwise, is just wrong (see MDL-38210 for example). All plugins types should be equal technically. Why to have some of them more equal than the others? I believe the that time reasoning/argument "it's ok, they are all in core Moodle at the end" was wrong and we did a big step away from that in recent years.

I believe that all these mini-API you have mapped (and many thanks for that!) can form a good base for the discussions on designing the unified reporting API.

Average of ratings: Useful (2)

Re: Plugins depending on other plugins

by Tim Hunt - Wednesday, 11 June 2014, 2:05 PM

"But there are other areas that, by definition, must be generalized since day 0 (database access, authentication...)"

You have chosen an unfortunate way to make your argument. DB access in Moodle was not generalised since day 0.

Initially Moodle worked on MySQL only. Here is the original schema file from May 2002. https://github.com/moodle/moodle/commit/6fa8326dad19dbddf48b49752a81c998093b5025
Later, Postgres support was added, but that involved horrible duplication between mysql.sql and postgres.sql files, or something like that. https://github.com/moodle/moodle/commit/7104dfc9efd47f0fc3c3e4975d27e6c1409ddbe8 - Wow! that is as early as September 2002, before Moodle 1.0. I thought it was Catalyst IT who did this much later. What I remember was them fixing all the places where the MySQL and Postgres schemas had got badly out of synch. E.g. https://github.com/moodle/moodle/commit/4e980e5a924c10a7676369f24b89bf3fa240a55a
Only then in August 2006/Moodle 1.7 did some clever Spaniard come up with the XMLDB so we could support 4 different DBs. https://github.com/moodle/moodle/commit/67d7d694947ed0bec1f39f5989cc3ce4de98c1a6

Do you think we could have had the foresight to build XMLDB in 2002? I don't think so. By the time you came to work on it in 2006, we had learned a lot about what Moodle needed from its DB abstraction layer, and that helped you build XMLDB.

Average of ratings: -

Re: Plugins depending on other plugins

by Eloy Lafuente (stronk7) - Wednesday, 11 June 2014, 6:09 PM

Oh man,

I'm very-aware of the history and stories of that part of Moodle, LOL.

I agree it's not the best example (I could have picked many others) but, still, I see it like an example of generalization since day 0. No matter it only supported 1 DB and a few tables. It was already a generalization (deciding storage of information, interfaces to it... to be used everywhere). Really important stuff.

Anyway, my point was just to show that not all implementations come from a generalization process, but I'm not interested into discussing that (well I'm interested, but off-this-discussion).

Just current stuff. Delete that example, np here.

Ciao

Average of ratings: -

Re: Plugins depending on other plugins

by Marina Glancy - Tuesday, 10 June 2014, 1:10 PM

It was not really a "secret integration meeting", sorry that it does look like it now. On backend planning meeting I noticed that class namings in reportbuilder spec were not up to Moodle standards and very soon realised that there are no standards for this case. So, as a member of both backend and integration teams, I asked this question on the integration meeting that accidentally happend to be on the next day. This is also why I was the person delivering the decision to backend team.

I still have a feeling that the general case of plugin-plugin communication, callbacks, hooks, class files locations, or whatever you want to call it, is not finally decided.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by dawn alderson - Tuesday, 10 June 2014, 6:32 PM

Hello Marina,

I think we can safely say that the meeting is no longer a secret anymore ;0)

For what it is worth, my version of summary follows-hope helpful, if nowt else.

Tim packs a tight argument, like a pair of Rudolf Nureyev's tights (LOL!)

And Eloy can unpack such arguments with the grace of such a ballet dancer *big grin here*

It does seem that the whole thing appears to sway towards that of pragmatics...what is practical? what will work?...for the maintentance of standards (improved standards even).

So, based on the following:

And, obviously, that keep us with the disagreement way more reduced:

4b. A plugin provides some functionality to any other, unrelated, plugin.

Do we agree that's the problematic one? Hope so.

Maybe a swing towards the pragmatic approach for practice is the required direction....obviously keeping peeps in the loop as to the 'how' would be a useful thing to do...as others may have comments along the way-to inform process for change.

that is me done with this!

Cheers,

Dawn

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Tim Hunt - Tuesday, 10 June 2014, 9:09 PM

I still think it is more exciting to imagine a cabal of core Moodle developers plotting in secret, than the mundane truth of what really happend.

I agree that this issue is not finally decided, and it is good we have started the conversation, because reportbuilder may be a good moment at which to get it solved. However, I don't really have anything to add to what I already said, to try to move the debate forwards.

Is http://docs.moodle.org/dev/Hooks_spec is up-to-date and does it reflect your latest thinking? If not, do you have any time to work on it?

Average of ratings: -

Re: Plugins depending on other plugins

by Marina Glancy - Tuesday, 10 June 2014, 10:44 PM

Well the spec was sort of frozen. It was supposed to smoothly go into the release but some community members created a big noise about it. This community reaction scared management. Even though the issue quickly collected many votes, apparently one community member can change things by very loud objections. The same community member said:

There is no point doing this unless we actually clean up all the old ways of doing things. Your complaint seems to be that there are many different ways to do things in Moodle, and you plan to make it worse by adding another way.

Martin accepted this idea and asked me to prepare a list of existing callbacks (one can find the list in MDL-44078 comments). Then he became even more scared of the amount of work and the issue was delayed for unknown period of time.

Petr Skoda is planning to work on hooks implementation regardless afaik. I am not looking forward converting all existing callbacks, this is a massive work with possibility of regressions and no (or very little) performance benefit in the most of cases. So I'm not promoting the hooks anymore unfortunately.

Saying that I like your suggestion of the subfolder in /classes/ dir and I will support it. I won't even ask you to convert existing plugin-plugin relations such as recent activity, participation report, activity report, courses overview block, etc.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by dawn alderson - Wednesday, 11 June 2014, 1:18 AM

this might be better placed here.

https://moodle.org/mod/forum/discuss.php?d=261735

Average of ratings: -

Re: Plugins depending on other plugins

by dawn alderson - Thursday, 12 June 2014, 1:21 AM

Back again :0) LOOK OUT- BIG POST! What follows is a summary of the points from the debate because Eloy stated there are just 3.5 months left to get things moving for 2.8. Thus, surely things cannot stand still because of disagreement. I am able to read and pick up on points quickly (been trained to-bit like a chimp being trained to make tea) but nevertheless have applied my skills-for what it is worth! Do not shout at me! LOL!

Tim: Generally, in Moodle, we like different plugins to be completely independent and self contained. This is nice and simple, but not always possible, and there are various ways plugins can depend on other things.

Eloy: 95% of agreement Yes, no surprises. I can say, without a problem, that I agree with 95% of the contents of Tim's initial post. Indeed:

All plugins depend of core. Just let me add "and its APIs”. I suppose nobody will object.
Subplugins depend of their parent plugins. Correct, nothing to object. A plugin is responsible to control its children.
There may be "dependencies” between different plugins. Can be architectural ones (extension) or data-access ones. Note I'm not 100% happy with how we use to perform that "raw access” (database, files...) between them and IMO it should be more controlled but, for its simplicity, and being proved to work, it's accepted. Plus we are able to control such dependencies at install/upgrade. Ok.

The 5% disagreement (summary)

Tim:

Sometimes, you have an add-on that only makes sense if another add-on is installed. For example Two of my add-ons. Report workflow is only useful if Block workflow is installed. What if other plugins need to provide some code to work with this plug-in, if it is present.

Eloy-

A plugin provides some functionality to other plugins by its own (aka, bypassing, core). Beep, beep, disagreement! Indeed! Everything else, that already covers a lot of inter-plugin dependencies, are not a problem at all.

AN AGREEMENT FOR PRACTICE:

Tim: Another similar example is the Combined question type, one of the OU question-type add-ons. This lets you build one question out of several other, like qtype_multianswer/Cloze, but better. One way that it is better is that it is extensible. Another qtype can make itself addable to qtype_combined by defining some classes in a folder like question/type/myqtype/combinable, for example in pattern-match.

Tim: That last issues shows, that really we don't need anything special in Moodle core. You can just get on and do this, by making up your own rules. The point is that it is better for Moodle if we can come up with a standard way to do this for all similar situations.

Eloy: A plugin of a plugin type (qtype), combining with another plugins of the same plugin type (qtype). For that, you can use 2 approaches. Or you implement such combination abilities by controlling it at plugin type level (sort of API within the constraints of the plugin type). Or you implement that ability between plugins, straight between them, bypassing completely that qtype commons API. Personally I'd prefer to bring the functionality to the (higher level) plugin-type level, so you can define there what is needed in order to allow that work for a given question type. But, no matter of that... still... I'd say. No objection there. As far as a given functionality is within the limits of a plugin type (qtype), dependencies can be implemented as you prefer.

Tim: The other point is, we now have a system of automatic class-loading, and we want to work with that.

The iTeam's considerations nut-shelled: Marina: At the same time I tend to agree with the majority of other integrators that in case of reportbuilder we should better introduce a new API because we expect almost all modules and many other plugins using it.

Tim commented: I think my concern about trying to define a core API for reporting is this: At this point in history, it is impossible to get it right.

NOTE-this was based on theory/literature pertaining tothe concept of generalisation stated by software developers outside of Moodle. So, theoretically-based without context e.g. (moodle core devs context). Eloy provided that contextual framing at this point:

Eloy: my point was not all implementations come from a generalization process... I cannot agree about your comments about generalization applied as a mantra to any development.

Tim: I certainly agree that, if it were possible to have a single API which all plugins would implement once, to supply all the metadata requried by all other reporting plugins, then that would be a very good thing to have.

Tim's Proposed solution

\mod_choice\tool_reportbuilder\data_description (in mod/choice/classes/tool_reportbuilder/data_description.php)
\mod_certificate\report_editdates\date_extractor
\qtype_pmatch\qtype_combined\renderer.php and \qtype_pmatch\qtype_combined\combinable.php

The plugin that looks for these classes would just find whether another plugin supports it with a class_exists call, and we already have ancore_component::get_plugin_list_with_class() API. Rather than doing this at the sub1 level of namspacing, we could instead declare a new sub1=plugin, and then put the name of the other plugin at sub2 level, like \mod_choice\plugin\tool_reportbuilder\data_description, but I think it is impossible for the Frankenstyle name of an add-on to be the same as the name of a core component, so I prefer the flatter structure.

David M commented: If I had to prefer one of original Tim's proposed locations, it would definitely vote for the flatter structure (without the common "plugin" root) so it looks consistent with core APIs (in fact, if all the core sub1 levels had the core_ prefix, we would see nice list of frankenstyle components there).

Eloy: agreement: A plugin type, constrained within its limits, can implement inter-plugin dependencies without a problem.

Eloy's point for disagreement: A plugin provides some functionality to any other, unrelated, plugin.

A word of caution from Tim: If you shoot for trying to build a generic API now, your are higly likely to just end up with an overcomplicated mess that is not a perfect fit for what report-builder is trying to do, and that mess will be sitting in the report core API namespace, getting in the way of doing something better. I really think that you should concentrate now on building the API that is exactly what report-builder needs, as a report-builder API. If the need arises, we can think about generalising later, once we know if and how generalisation is necessary.

Dawn referred to the necessity to evaluate theory/practicalities...Big picture: quality....and pragmatics in order to oversee a do-able workload that ensures such quality......what works?....what is practical? for the core team.....

David M: Activity modules are not only flag ship plugins we have. They are also the oldest ones we have, and it is apparent at many places.

Eloy: suggestion for moving forward: If everything under a plugin API only can be implemented via subplugins, and only the (unique) "parent plugin” can access/use/control them. Way restrictive! Let's make everything a 1st level plugin citizens so everybody can access to them (reporting formatters, reporting exporters... whatever). If it's correctly structured already (and has being generalized properly) it should not be so complex. We have 3.5 months to achieve it. Finally, about the plugins-intercomunication... A plugin provides some functionality to any other, unrelated, plugin. I must say that I don't oppose as a principle. But, still, I need to be convinced with a good case where it's really needed.

David M: The fact that some other plugin requires my module to implement a function, or it throws an error otherwise, is just wrong (see MDL-38210 for example). All plugins types should be equal technically. Why to have some of them more equal than the others?

Marina: Provided a list showing substance for discussions on designing the unified reporting API.

Tim:I agree that this issue is not finally decided, and it is good we have started the conversation, because reportbuilder may be a good moment at which to get it solved.

Dawn: Posted a recent link: https://moodle.org/mod/forum/discuss.php?d=261735

NOTE: The link shows communication whereby it suggest anyone wanting to design a new plugin may not understand fully how to connect with what already exists in terms of pattern matching code etc, for example, this chap based his work on the glossary module/whereas database might have been better-I really do not know-not my specialist subject. But, surely this highlights an underpinning issue across this thread? Although, ironically, the process for getting a plugin through is in place....e.g. https://moodle.org/mod/forum/discuss.php?d=261769

I hope this is helpful....and I do hope you all end up playing nicely with each other

Dawn

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Marina Glancy - Thursday, 12 June 2014, 9:53 AM

This thread has too many words and too few solution suggestions.

In my opinion communication between plugins is absolutely essential. Reports and blocks are the best examples for it. Core very often needs to question plugins of different type about something as well. The most typical example is building navigation tree.

At the moment we have one current way to do it and two suggestions:

1. (current) Use callbacks

2. hooks proposal: read MDL-44078

3. Tim's proposal above to allow /classes/someother_plugin/ to store files for another plugin

If we can merge 2 & 3 we can win the world

Average of ratings: Useful (2)

Re: Plugins depending on other plugins

by Eloy Lafuente (stronk7) - Thursday, 12 June 2014, 10:18 PM

This thread has too many words and too few solution suggestions.

Too many words are always better than too few words. Indeed.

In my opinion communication between plugins is absolutely essential.

And there are a lot of cases where it's allowed (1, 2, 3, 4a, following the nomenclature). Only 4b is under discussion.

Reports and blocks are the best examples for it.

Which ones, the examples you listed above? Why a plugin should be able to decide the implementation of another plugin? All them replaceable by a reporting API (keeping the logs one apart) or by improving the "virtual/non existing" activities API.

Core very often needs to question plugins of different type about something as well.

Completely agree here. Core (and its APIs) has rights to dictate what's required in plugins. And ask for it.

The most typical example is building navigation tree.

The most typical example of core using plugins? Do you remember that "navigation" is an API?

At the moment we have one current way to do it and two suggestions:

(current) Use callbacks

hooks proposal: read MDL-44078

Tim's proposal above to allow /classes/someother_plugin/ to store files for another plugin

And I was not and I'm not against any (per se) at this stage. I'm against of (4b) plugin2plugin uses for any of them. More yet, the "hooks" proposal (that I did not dislike at all) was also an API candidate (from memory) and as a such, it has sense. I'm just worried about where people wants to use it.

More exactly, what I'm objecting is about all that stuff applied to the report_builder vs reporting case that, if I'm not wrong, is the case that ignited all these "words".

If we can merge 2 & 3 we can win the world

Beautiful phrase. pfff, LOL.

So, it seems we are discussing 3 (at least) things here:

A) the solution to have all Moodle hooks under control, hopefully phasing-out current callbacks and other "tricks" here and there.

B) the places/levels where A) will be allowed.

C) report_builder vs reporting API.

Perhaps we should split the discussions? Or do you prefer to continue with everything together? I say that because, sometimes, re-reading the discussion... it seems we are arguing when, really, we are just talking about some other (ABC) point.

I really think that the point of disagreement is small (but important): plugin2plugin yes/no. And from it, ABC will become determined, magically!

If the answer for that simple question cannot be agreed/determined... then we'll need to lower expectations, keeping the AB parked for some more (lunar-only) cycles and centre ourselves in the case under discussion (C) , the report_builder one, that needs to be addressed ASAP.

Ciao

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Tim Hunt - Friday, 13 June 2014, 2:40 AM

I agree with Marina's "If we can merge 2 & 3 we can win the world."

Eloy, in addition to Marina's list above, the list in in MDL-44078 is also helpful.

For standard plugins, we can always argue that a core API could be the solution, but since I have already implemented several add-ons that needed to interact with other add-ons, I am convinced we need plugin2plugin communication, but you are right to be very scared of this, and we need to do it in a way that does not just make a big tangled mess. I think a revised hooks spec can do that.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Eloy Lafuente (stronk7) - Friday, 13 June 2014, 3:02 AM

Tim,

I'm sure it's some sort of failure explaining @ my side, but have you really reviewed that list. Can you tell me which of them, apart from Marina's list above are plugin2plugin? Please don't mix things, let's keep them separated for the sake of ABC agreement.

Ciao

PS: And I reiterate (4-5th time in this discussion), that I'm not against (A) a proper hooks API. Never have been. Curiously others, now defending it to "win the world", were way more negative about it (sort of all or nothing) some months ago. I'm just reticent when talking about (B) where to allow it, specially now, before allowing (C) to flood the whole Moodle with new/wrong plugin2plugin unjustified hooks.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Damyon Wiese - Friday, 13 June 2014, 9:50 AM

Short post because I'm a rebel.

A) Hooks - +1 (especially for addon land)
B) Really the only hard rule I see in this discussion is that "hooks" are not possible when bits of core needs to call them (e.g grades).
C) APIs are more useful when this plugin will have to talk to every other possible plugin type + core.

"Why should it be any different from other reports/blocks that we have?"
"Because it sucks, and we know that"
+10 (referring to all those nasty lib.php callbacks that no-one knows exist)

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by dawn alderson - Friday, 13 June 2014, 2:33 PM

Good morning/afternoon you shiny, happy people! :0)

What a rebel you are indeed Damyon....far too succinct.

Anyway, have a couple of naive points....but I thought I would post with the aim of dragging this slug on its lead an extra inch or two forwards

Hope my thoughts are received the way they are intended to be-with a big smile-of course ha ha

We appear to be in a tunnel of detail eh.

..................A return to Big Picture.......

HALF OF THAT BIG PICTURE:

THE OTHER HALF:

Eloy-A plugin provides some functionality to other plugins by its own (aka, bypassing, core). Beep, beep, disagreement! Indeed! Everything else, that already covers a lot of inter-plugin dependencies, are not a problem at all.

How about someone comes up with a 'standard way suggestion'....theoretically speaking.....and who wants to bet hooks will be in there somewhere or not? :0)

Average of ratings: -

Re: Plugins depending on other plugins

by Dan Poltawski - Thursday, 26 June 2014, 12:39 AM

I am basically in agreement with Eloy, but I've been trying to quantify why it is that i'm in favour of APIs, particularly in the case as damyon says: "C) APIs are more useful when this plugin will have to talk to every other possible plugin type + core", so here is a stab at one rationale for it:

When we define an API in core we also establish the point where we define the rules for when the API is called how it gets dispatched to the plugins and when data exchange happens. For example, we might prohibit one plugin from being called depending on a capability or context. If we do that with hooks, we have to trust that the plugin implements this access control or business logic correctly (and if we add that logic into the hook dispatcher, then we may as well have just created the API in the first place).

Average of ratings: -

Re: Plugins depending on other plugins

by Damyon Wiese - Thursday, 26 June 2014, 11:15 AM

Some more quick points:

If we add a new plugin and have to make it a dependency of every single activity module in core (and plugins land), we have failed.

If I have to implement 6 different flavours of reporting API in mod_assign, we have failed.

Average of ratings: -

Re: Plugins depending on other plugins

by dawn alderson - Friday, 27 June 2014, 2:15 AM

got it so far.

Tim-I am keen-(well-you know what I mean-not quite To know what you think about this at this juncture. Yes, that is right..I want to know what you think-goodness knows why I got involved in this thread, but hey-ho...am intrigued to learn about your thinking with this(I get to learn too).

Average of ratings: -

Re: Plugins depending on other plugins

by Damyon Wiese - Tuesday, 8 July 2014, 11:35 PM

Yesterday a meeting was held to try and clarify the different opinions on this discussion and see if there was some general common agreement. The discussion is still open until we have a policy agreement on these issues.

Here are some notes from the discussion.

Attendees: Martin, Damyon, Dan P, Marina, Eloy, Petr, David Mudrak, Michael DeRaadt

The google docs notes are attached. They may or may not accurately reflect the discussion (I will try to).

First very important point. There are really 2 separate and only slightly related discussions here.

A) In what situations may a plugin directly call an api defined in another plugin?

B) Do we want to implement hooks for plugin communication and if so - what are the details that still need to be worked out?

I will clearly split the summary here into those 2 points mentioned above - because in reality the meeting was confusing because people kept jumping back and forth between these 2 points.

A) (to repeat) In what situations may a plugin directly call an api defined in another plugin?

So - it was noted that there are (currently) different styles and use cases for APIs - a defining feature of these being the "cardinality" of the caller / callee (ie one to one, many to one etc).

Examples

1-1 with strict dependency defined in version.php

Example: filter_glossary which depends on mod_glossary

Outcome:

No-one had any issue as long as the dependency is defined (in core code, core plugins and or contributed plugins)

1-1 with no dependency defined in version.php

Example: checking for class_exists on a specific class from another plugin

Outcome:

Never allowed in core code or core plugins. Allowed in contributed plugins.

1-many - no core api

Example: Overview block looking for a specific function in all installed plugins (no communication through core).

Outcome:

Never allowed in core code or core plugins. Allowed in contributed plugins.

many-1 - no core api

Example: Imagine locking with no core api. Each plugin would invent/duplicate it's own method of choosing the type of lock implementation to use.

Outcome:

Never allowed in core code or core plugins. Allowed in contributed plugins.

many-many - no core api

Example: Imagine logging with no core api. Each plugin would invent/duplicate it's own method of choosing the log store to log to and would call the log method directly on the store.

Outcome:

Never allowed in core code or core plugins. Allowed in contributed plugins.

all the above - WITH a core api

Outcome:

Allowed everywhere always.

Summary of all these examples. No use case could be given for any core plugin having hardcoded communication directly with another core plugin (not through an API) that would not be much better implemented through an API. Contributed plugins should be allowed to do this because they need flexibility and cannot easily add new core APIs. If multiple contributed plugins all start using an API from another plugin we should consider adding a core API for it.

Given that in contributed plugins are allowed to implement apis directly from other contributed plugins should we standardise the class names used when doing this? The agreement was no, the existing namespace rules were clear enough (but we noted to finally get those namespace rules documented in the coding style doc, which is now done).

We know there are lots of APIs existing that break these agreements above - but nobody had any defence for any of them. That said - we will not immediately deprecate all these old APIs because they are existing and they work for now and just forcing everyone to rename their functions is "busy work" that only benefits developers. Better to tackle each one when the time is right and create a new better API the same time.

B) Hooks (This is completely separate from discussion A - don't confuse them please)

You should re-read the hooks specification if you are not familiar with it.

http://docs.moodle.org/dev/Hooks_spec

A one liner is that hooks are a bit like "named" callbacks where the subscribers can modify the data passed (e.g. add fields to a form).

The hooks specification has matured and there are a number of benefits to using it over the existing callbacks methods.

The performance should be better because the list of hook observers can be cached
The list of observers for a hook can be programatically determined and listed on a page
Prevent code duplication (we re-implement the call backs system each time)
The list of available hooks can be easily found in the code for each plugin (or core).
We could improve hooks later e.g. to allow certain subscribers to be skipped over, or reordered.

There are some concerns about hooks being able to modify the data passed to them, and how much control there is of the order that hooks get executed. These were the only concerns noted and seemed in favour of hooks.

The simple case of 1-1 communication with a defined dependency will always be quicker/simpler to call a class directly.

Final note: Apologies if anyone felt they were not invited to the meeting (and apologies to anyone who was pulled into the meeting despite their howling protests) - it should have been advertised / organised better before-hand.

Agenda for callbacks_namespaces meeting.pdf

Average of ratings: Useful (3)

Re: Plugins depending on other plugins

by David Mudrák - Wednesday, 9 July 2014, 3:28 AM

Thanks Damyon for providing this summary. It definitely puts some light on the topic - even for me who was attending the hangout. Some random thoughts / reflection (although I am sure I'll only repeat what I already said elsewhere).

Firstly, I am very happy you agree that plugin to plugin communication and hooks are two separate issues. I had feeling that hooks are somewhat presented as the holy grail that would save all the issues we have here. Said that, I have no objection against them. Moodle core is a vital bazaar with many different patterns to solve software architecture challenges. Some of these patterns seem to be vital (such as more and more meta files in the db/ folder of a plugin that describe various interfaces of it). Some seemed to be a great idea but did not spread/evolve much for various reasons (such as the {plugin}_supports() thing that in fact was addressing similar case). There is no reason to not to give hooks a chance to become yet another way to solve things and see how it goes.

There is one thing in the conclusions I am not quite sure about conceptually. The fact that we discriminate that much between standard plugins (that ship with the core) and additional plugins developed by the community. I believe there should not be technical differences between these two. As I already said - both standard and additional plugins should be considered equal from the core's perspective. Both should follow the same good design patterns and coding style.

Keeping in mind where this whole discussion is coming from (MDL-30193), I am a bit afraid we might be a bit overreacting here. At the beginning there was a discussion whether certain API should be provided by a (standard) plugin or by the core. As reporting is essential feature we would like to see in Moodle, having a well defined core API for it sounds reasonable. But concluding that this means that we need core API for every possible interoperability between plugins sounds like an overkill to me. Do we really need such a Kragel here? Is not it more case by case type of thing?

Average of ratings: -

Re: Plugins depending on other plugins

by Damyon Wiese - Wednesday, 9 July 2014, 9:29 AM

Thanks David,

There are 3 key differences I see in contributed plugins (dont call them addons, dont call them addons) are:

1. It is harder for them to get changes accepted into core to facilitate communication between some of their own contributed plugins

2. It is a lot more work to design a generic reusable API than just calling a function from another plugin

3. The maintainers are responsible for maintaining their contributed plugins, so if they write sloppy code - it is for them to maintain it over time

A key example I can think of is editpdf where Davo created a very useful plugin that was used by lots of people for years before we added it to core. When we added it to core there was a lot of work involved in re-designing it in order to not do exactly the sort of inter-plugin communication we are discussing here. A lot of other things were done to meet our requirements too like converting the drawing code from Raphael to YUI. Should we have imposed these tougher restrictions on the version of editpdf in the plugins DB? IMO no - Davo created a good plugin and maintained it well and it benefited lots of people.

Average of ratings: -

Re: Plugins depending on other plugins

by David Mudrák - Wednesday, 9 July 2014, 3:47 PM

Should we have imposed these tougher restrictions on the version of editpdf in the plugins DB?

No. My point was not to be more strict on contributed plugins. Instead, I was thinking about being a bit more relaxed when it comes to standard plugins. Using Davo's plugin as the example again, maybe that giving it some time to stabilise and get mature (yet mellow ) in the plugin's scope allowed to design a better core API after the years. Rather than trying to design an API from scratch at the very beginning. Anyway, I admit my notes may be purely theoretical.

In any case, thanks a lot for all this discussion.

Average of ratings: Useful (1)

Re: Plugins depending on other plugins

by Tim Hunt - Wednesday, 9 July 2014, 5:44 AM

Well, it is obvious from this thread that I am interested in this topic, so I am still a bit miffed that I was not told about the meeting, but thanks for apologising.

Let us deal with A). Your summary gives, answer, but the key thing is the question:

Why?

You say lots of things are not allowed, without saying why they should not be allowed.

Also, ou try to give examples, but the key ones are very vague, and bad examples. Obviously locking is a core API. No one is proposing to replace that with a non-core API, so this is a straw man. There are plenty of good examples already in this thread.

I really don't see why a distinction is being made between core and add-ons. Good design is good design. Sometimes we will allow sloppy bad design in add-ons, when we have stricter rules in core. However, the decision reported here seems to be to mandate a worse design in core, and allow add-ons to do things in a properly modular way.

The key case for A) is one-to-many optional dependency. I notice you don't distinguish between optional and required dependency in your post, but it matters. Required dependency is always documented in version.php, and no-one objects to that, from what I have seen of the discussion. (Required dependency is case 3 in my original post in this thread.)

All the discussion is about what I called case 4 in my original post. One plugin wants to interact with any other plugin that wants to take part by implementing the required API, but no other add-on has to implement that API. Lots of people think this is bad. No has explained (in a way that I can understand) why it is bad.

Average of ratings: -

Re: Plugins depending on other plugins

by Damyon Wiese - Wednesday, 9 July 2014, 9:52 AM

Hi Tim,

"core and add-ons" - see my response to David.

"one-to-many optional dependency" - any time you do this you are basically hard coding the choice of the "one" because that is the only plugin that uses/provides that API/callback/hook/whatever. Plugins should not have egos and any plugin should be able to be replaced with a better version.

Consider the equation editor - I could have hard coded that to work with the MathJAX filter only - and my life would have been much easier - but some people prefer the Tex filter for various reasons - and at some point in the future there may be another way to render latex that no-one has thought of yet. So it only communicates through the "filter" API.

Same with all the hard coding of people directly calling tinymce functions that I have had to go through and un-ravel.

Also - saying that the outcomes in the previous post are not valid because of the choice of examples is also a case of straw man. Please understand I was documenting what was discussed in the meeting, not trying to create a perfect case for everything.

Average of ratings: -

Re: Plugins depending on other plugins

by Tim Hunt - Wednesday, 9 July 2014, 2:17 PM

I agree that when a certain sort of functionality can be abstracted into a generic core API, then that is the best way to go. Your filter API example is a good one, as is the Lock api example given earlier. (Thought, to be pedantic, filter is a plugin type. The API function is format_text, which is part of output API or something.)

And I agree we should allow add-on authors to be sloppy if they like.

The problem I have is with putting different requirements on well-written add-ons, and standard plugins. We know that sometimes add-ons go into the standard package, and sometimes they are removed. With the proposal on the table, when an add-on is added to core, suddenly it needs to have plugin-plugin communication converted to a Core API. And suppose we have a Core API that only exists to allow one standard plugin to work. What are you going to do when that plugin is moved out of the standard package?

It just feels better to me that we have a standard mechanism for this that is the same for all plugins, whether standard or not.

The other problem I have is, with the "can be abstracted into a generic core API" bit. Of course, we can make a core API for anything, but I don't think we want core APIs that are terribly idiosyncratic, and only exist to serve one plugin.

If you disagree, have a challenge for you: Let us suppose that the Edit dates tool is useful enough to be added to the standard Moodle package. (Note, this is not a well-written plugin that could go straight into core, but that is not the point.) What this does is provide a big form with all the date settings for a course, so you can see and edit them on one page. Since different Activities have different date settings, we need a per-module API.

My challenge is, how would you make a generic core API for this case (without re-writing a lot of Moodle)? Existing interface classes are here, if you want to see the information that needs to pass around: https://moodle.org/mod/forum/discuss.php?d=261673#p1134778. (Note, this is not currently extensible, with all the code in this report. If someone wants to add Edit-dates support to their add-on, they currently have to send me a pull request, which is not ideal.)

Average of ratings: -

Re: Plugins depending on other plugins

by Dan Poltawski - Wednesday, 9 July 2014, 8:16 PM

And suppose we have a Core API that only exists to allow one standard plugin to work. What are you going to do when that plugin is moved out of the standard package?

Like the question API? I assume we will only replace mod_quiz when a slicker alternative is developed.

But we can have APIs which do not have standard plugins using them - best example of that is the plagiarism API, which has no standard plugins, but supports add-ons.

The advantage of providing an API is that we allow new competitors in add-on land to compete on a level playing field as well as add-ons themselves to utilise this core functionality.

My challenge is, how would you make a generic core API for this case (without re-writing a lot of Moodle)?

I find this a bizarre challenge. You would have to re-write a lot of Moodle - that is what happens when we standardise and provide APIs for functionality in core! e.g. when we introduced the navigation API or repository API.

Average of ratings: -

Re: Plugins depending on other plugins

by Tim Hunt - Wednesday, 9 July 2014, 10:11 PM

Only one activity module?

It's not my fault that mod_lesson has never been coverted ot use the question bank.

As I already said "I agree that when a certain sort of functionality can be abstracted into a generic core API, then that is the best way to go."

report_dates is something relatively simple and useful. Should it be possible to do simple and useful things in core? You seem to be saying "No, things in core shoudl only be done if they involve a major rewrite." I don't believe you really mean that.

You say it should be a level playing field, while at the same time forcing the playing field to be completely un-level. Standard plugins can create new Core APIs. Add-ons can't.

Average of ratings: -

Re: Plugins depending on other plugins

by Dan Poltawski - Wednesday, 9 July 2014, 10:33 PM

Only one activity module?

Read your own post where you said 'standard', thats what I was replying to.

report_dates is something relatively simple and useful. Should it be possible to do simple and useful things in core?

But by your own admission, report_dates does not work with existing modules without a pull request to modify report_dates itself to support it..? So no, I do not see that as a very good example of something which should be put into core. It needs to be possible that a well-written add-on can implement support for the 'date changing API'.

Average of ratings: -

Re: Plugins depending on other plugins

by Tim Hunt - Thursday, 10 July 2014, 3:11 AM

I think we all agree that when a certain sort of functionality can be abstracted into a generic core API, then that is the best way to go. And I assume we all agree that the Question API falls into this cateogry. (If you disagree, speak now.) So, shall we stop discussing this point?

The case that needs discussion is when one plugin wants to do something specific with any other plugins that choose to contribute, but it does not require any other plugin to contribute, and where we don't yet have a good abstraction that should be a core API.

Report editdates is one example of this. It is useful, but as you say too hard to do with a truly generic API that it would never happen if we waited for that. qtype_combined is another good example. It handles the use of other plugins much better than the standard qtype_cloze. Note that it has proven impossible to fix MDL-6371 - let add-on qtypes be part of Cloze questions - in 8 years. (Cloze does hard-dependency on the other qtypes that can be subquestions, declared in version.php, which is all agree is OK, but it is absolutely not a level playingfield for add-ons.)

In contrast, qtype_combined defines base classes https://github.com/moodleou/moodle-qtype_combined/tree/master/combinable which other qtypes override if they want to be combinable. E.g. https://github.com/moodleou/moodle-qtype_gapselect/tree/master/combinable. (This was done before the classes directory was a standard, but would be trivial to update to modern standards if we did not mind breaking backwards compatibility.) This works well. As is clear from this thread, I don't see a problem with allowing this pattern in core.

Indeed, I would expect that allowing this pattern in core, for those cases were we don't yet understand enough to implement a good generic core API, is probably a good way to learn from specific cases, and get good functionality easily today, that may one-day inform a good generic core API that can only be designed once we have seen enough specific examples working in practice.

Back the edit dates report, to explain how it could be made properly pluggable. Instead of putting classes with names like report_editdates_mod_assign_date_extractor in report/editdates/mod/assigndates.php, I would allow either

\report_editdates\local\mod_assign\date_extractor in report/editdates/classes/local/mod_assign.php/date_extractor.php; or
\mod_forumng\local\report_editdates\date_extractor in mod/forumng/classes/local/report_editdates\date_extractor.php

(1. is only necessary because we expect report_editdates to remain a contrib plugin, and we want to edit dates for core modules without changing core. 2. is what lets add-ons participate in the report, without needing me to merge a pull request.)

Average of ratings: -

Re: Plugins depending on other plugins

by Tim Hunt - Thursday, 10 July 2014, 3:53 AM

Hmm. The proposed new reports API could conceivably make enough information available to implement report_editdates without any p2p communication. If so, that would be a win for everyone.

(Except repot API is unlikely to expose the validation rules like timeopen < timeclose. Hmm.)

Average of ratings: -

Re: Plugins depending on other plugins

by dawn alderson - Thursday, 10 July 2014, 4:47 AM

A simple response,

It may be that a lot of deep thought needs to be given to point/option 1...then implement it.

Could be wrong.

Average of ratings: -

Re: Plugins depending on other plugins

by dawn alderson - Wednesday, 9 July 2014, 9:06 PM

Hi all,

Tim, I think, honestly-your point is pertinent here-in that you ask why?

Therefore, when you state:

'You say lots of things are not allowed, without saying why they should not be allowed'.

Yep-I see that too.

I am going to cut across the enormity of detail coz, it feels like this thread at the mo is becoming circular-as in we are all running around in circles....and some more circles...and ummmm more like

It appears to me that the point/purpose/rationale/reason for all this stuff and in turn the need for change= moodle being the engine for development.....anything that cuts out the engine and runs as a plug-in with another plug-in (non-core dependency) defeats the whole notion of moodle as a collective and ever evolving project.

Course, there maybe some features that are extremely well designed in terms of the latter, but it does appear the concept of standardisation needs to apply across the board....it may be that time is a key factor in applying a one-rule to all features and what-not....identifying a timeline for those features concerned with regrd to API changes....might enable the more succesful/used features to be affected after seeing how things work with less-used features....I guess I am suggesting a roll-out for change as opposed to a fast-blanket change.

Now, I could bloody well be wrong!

Oh and if there is a need for an SAS pass to these meetings can you please let Tim know-out of common courtesy ( LOL!)

Average of ratings: -

Re: Plugins depending on other plugins

by Dan Poltawski - Wednesday, 9 July 2014, 11:15 PM

One plugin wants to interact with any other plugin that wants to take part by implementing the required API, but no other add-on has to implement that API. Lots of people think this is bad. No has explained (in a way that I can understand) why it is bad.

Note that its worth reiterating from Damyons post that explicitly defined ($plugin->dependencies) relationships were agreed to be fine.

You asked for a response to this specific point, but I find this a hard thing to express to you because I think you agree with the principles of what would be bad design. Furthermore Moodle core is built on the principle of strong core apis, not a thin core where modules all independently provide functionality. We provide the repository API, not the repository module and the question API, not the question Module. Thats significantly different to say drupal which (from memory) acts much more like 'everything is a module' approach.

1) Communicating p2p naturally leads to a many <-> many API relationship. Many <-> Many communication is bad:

* If there are 3 global search plugins, we shouldn't need to implement our search hooks 3 times

* By doing it 3 times in subtlety different ways, we eliminate the ability to cache/optimise these requests in an intelligent way that only core can.

2) By communicating in plugins and not involving core, we remove the ability for core to be an arbiter of communication. Say some fancy new availability feature is introduced - that needs to be implemented in every module which communicates together this way. Rather than being taken care of by core APIs.

3) The goings on of plugins should not affect other plugins. i.e. if a new reportbuilder_for_schools is added to join the existing one, it should not require all modules to implement support for that too. Or if one is removed it also shouldn't have an impact.

Can you think of a plugin 2 plugin communication where a core API would not make sense (because each time I try, it seems unreasonable to me).

Average of ratings: -

Re: Plugins depending on other plugins

by Tim Hunt - Thursday, 10 July 2014, 3:21 AM

I think the problem is that you are living in an ideal world (lucky you ) but I am aware of my own limitations and want to get useful work done, even if I cannot produce perfect understanding, desing and code today.

It is interesting that you mention global search, because I feel that it proves my point. How many years have we been waiting for that? And, for all that time, we don't have the ability to search for questions in the question bank by the full text of the question. (Though I agree that could be implemented with SQL, if you were happy to only search the question text and general feedback, but not things like the multiple choice choices.)

In the meantime we have had https://moodle.org/plugins/view.php?plugin=local_ousearch for years, and several other add-ons that use it. Not to mention years of live use at the OU. (Note that, other add-ons using that is a specifically declared dependency in version.php, so not problem there.)

Yes, a core search API would be better, but we don't have one yet.

Note, we don't yet have an 'recent activity API' to cover some of the of cases that Marina identifies in the useful post. Do you not think that when it comes to cleaning up the legacy code by creating such an API, it will be much easier to work out how that API should work by having an number of specific working examples to refer to?

So, to summarise what I am advocating:

Where we can define a good generic core API, this is the preferred approach instead of plugin-to-plugin communication.
In other cases we might acutally be looking at a new plugin type. (Another recent example is question bank columns MDL-40457 - should that have been a new plugin type?)
But, in other cases where direct plugin-to-plugin communication lets us implement some useful functionality today, than a core API that would be too hard or prohibitively expensive to implement now, then it should be allowed (with an understanding that we might want to re-design to a core API in future, when we work out how).

Average of ratings: -

Re: Plugins depending on other plugins

by dawn alderson - Thursday, 10 July 2014, 3:52 AM

Right-supposed to be on me bleedin hols! ;)

First: Like this-tis funny...all will agree, am sure:

I think the problem is that you are living in an ideal world (lucky you )

Secondly,

'even if I cannot produce perfect understanding, design and code today'.

You surprise me! You are wrong!

Thirdly:

So, to summarise what I am advocating:

Where we can define a good generic core API, this is the preferred approach instead of plugin-to-plugin communication.
In other cases we might acutally be looking at a new plugin type. (Another recent example is question bank columns MDL-40457 - should that have been a new plugin type?)
But, in other cases where direct plugin-to-plugin communication lets us implement some useful functionality today, than a core API that would be too hard or prohibitively expensive to implement now, then it should be allowed (with an understanding that we might want to re-design to a core API in future, when we work out how).

RE:

Point 1: This is Dev world-confuses me....shall leave those matters to those who know more.

Point 2: I think this is your world Tim: Expert -ADVISE Eh? This is your bag.

POINT 3: We have some synthesis of thought here, e.g. As I said earlier:

it may be that time is a key factor in applying a one-rule to all features and what-not....identifying a timeline for those features concerned with regrd to API changes....might enable the more succesful/used features to be affected after seeing how things work with less-used features....I guess I am suggesting a roll-out for change as opposed to a fast-blanket change.

Average of ratings: -

Re: Plugins depending on other plugins

by Dan Poltawski - Thursday, 10 July 2014, 7:10 AM

I think the problem is that you are living in an ideal world (lucky you smile)

Perhaps, but I believe in Moodle core being this 'ideal world' you speak of - thats our job in maintaing the platform. Its what distinguishes things which land in core, we do them properly so they support all users - we provide edit_dates functionality which can be used by all modules or to use another real life example, when Davo moved his dndupload stuff into core, he removed the hardcoded module links and added API hooks, shortly afterwards - someone else added SCORM support (MDL-32937).

It is interesting that you mention global search, because I feel that it proves my point.

Heh and I think the opposite! It seems to me that it would be quite easy if we were using ousearch - local_ousearch is the search API for core, wrapped up in a local plugin. If we were integrating ousearch that distinction is already made for us, so it seems quite easy to me. The API design and plugin communication is not what has caused global search to stall.

But, in other cases where direct plugin-to-plugin communication lets us implement some useful functionality today, than a core API that would be too hard or prohibitively expensive to implement now, then it should be allowed

Or, we make a limited core API and evolve that. To me that is a better situation than p2p problems I mentioned above.

Average of ratings: -

Re: Plugins depending on other plugins

by Damyon Wiese - Thursday, 10 July 2014, 9:43 AM

"It is interesting that you mention global search, because I feel that it proves my point. "

I feel it proves the opposite. The current GSOC project uses solr which sounds like a PIA for general use. Databases have their own built in text search functions which I feel would be a better approach, so once we get a global search API - I would be interested in replacing any solr backend with a native DB end. I don't want to have to add code to every plugin to do this.

"we have had local_ousearch" for years - great that is exactly what should be in addon land.

"Do you not think that ..., it will be much easier to work out how that API should work by having an number of specific working examples to refer to?" Not really - I think that recent activity could have benefited from a real API from day 1. The data would be used in more places in Moodle already and could be cached etc...

"Where we can define a good generic core API, this is the preferred approach instead of plugin-to-plugin communication." - We all agree there.
"In other cases we might acutally be looking at a new plugin type." - That can be decided case by case.
"But, in other cases where direct plugin-to-plugin communication lets us implement some useful functionality today, than a core API that would be too hard or prohibitively expensive to implement now, then it should be allowed (with an understanding that we might want to re-design to a core API in future, when we work out how)." In core code I strongly object. If the API is hard to define - it needs to be considered whether the feature needs to sit in add-on land to mature for a while.

Note that in your qtype_combined case you mentioned - this really seems like a generic feature for questions - not a custom question type. If you hard code support for the "qtype_combined" into all the other question types - there is no benefit to having qtype_combined as a plugin - you can't replace it.

Average of ratings: -

Re: Plugins depending on other plugins

by dawn alderson - Saturday, 12 July 2014, 5:06 AM

Hey hey,

my concluding thoughts follow:

So, to summarise what I am advocating:

Where we can define a good generic core API, this is the preferred approach instead of plugin-to-plugin communication.
In other cases we might acutally be looking at a new plugin type. (Another recent example is question bank columns MDL-40457 - should that have been a new plugin type?)
But, in other cases where direct plugin-to-plugin communication lets us implement some useful functionality today, than a core API that would be too hard or prohibitively expensive to implement now, then it should be allowed (with an understanding that we might want to re-design to a core API in future, when we work out how).

RE:

Point 1: This is Dev world-confuses me....shall leave those matters to those who know more. (I think the last two posts have squared a good argument for this point, here).

Point 2: I think this is your world Tim: Expert -ADVISE Eh? This is your bag.

POINT 3: We have some synthesis of thought here, e.g. As I said earlier:

Average of ratings: -