Moodle in English: Moodle's future with YUI

It looks like it is time to start planning an orderly transition to jQuery.

You replied too quick

Expanding on what I wrote below - I don't think that is a useful direction. jquery might be around for a long time due to its popularity - but I think its use is becoming less and valuable as javascript improves. See: http://youmightnotneedjquery.com

Here's a simple reason why Moodle should adopt jQuery. According to the linked report, 60.4% of web sites use jQuery and the next highest total is none at 35.9%. Of the other proposed libraries that I remember being suggested, AngularJS is at 0.1% and EmberJS doesn't even rate.

Now, I'm not saying popularity is everything, but we should consider that it might be in Moodle's best interests to embrace the overwhelmingly most popular Javascript library simply because it is overwhelmingly popular. There are people who want to work on Moodle, and chances are that if they know any Javascript library, then they know and use jQuery. Thus, if Moodle uses jQuery, it is easier for those people to make new things for Moodle and I think that's something that we all want. That, in and of itself, is something that Moodle needs. Moodle needs happy and productive developers, not just for the PHP core, but also for the plugin ecosystem, theme development, and UI work.

Sure, there may come a day when jQuery development stops or jQuery is relegated to the 0.7% market share of YUI, and we'll have to consider what to do next. However, that's life. Changes come and go and we adapt to them. For now, by adopting jQuery we would be making a lot of people who already work on the fringes of Moodle happy, while getting the advantage of a Javascript framework that a lot of people actually like to work with.

Personally, I think there'd have to be some very persuasive arguments to convince me that going with jQuery is not the best short and long term decision to make right now. I'm prepared to listen to them, but I haven't seen anything that convinces me that jQuery would be a bad choice in this thread, yet. Furthermore, to me the suggestion that we should develop a Moodle specific javscript library instead of using jQuery (or something else) is a non-starter. Even though I've re-invented the wheel many times before, I am generally of the opinion that it is not a productive use of time and energy. That's particularly true if we're going to make a wheel that only fits on our own car, because more often than not when we're done we find that our new square shaped wheels don't work quite as well as we expected them to, and nobody has the the time to sand all the corners down.

Note: Some people have mentioned that there are some bad jQuery plugins among the 2576 plugins in jQuery's official repository and I'm sure they're correct, but I also know there are also some bad plugins among the 899 plugins in Moodle's official registery.

Actually there are NO bad plugins in the Moodle registry (big grins)

Interestingly, this is basically the reason why Martin chose PHP for Moodle. Everyone knows it and it has a low barrier to entry, so let's not care if it sucks in all sorts of technical ways.

Of course, since then, the technical requiremenst for contribing to Moodle have gone up (even though we still use PHP) and as a result the Moodle codebase is more robust, but it is harder for new developers to make their first contribution. Parts of that are good, and parts of that are bad. We would like both a robust system, but also ease of contribution.

No. This is not about YUI vs jQuery. This is about JS framework or no JS framework.

You're right, it's not about YUI vs. jQuery, but it's also not about JS framework vs. no JS framework.

It's about what we need overall and whether we need a single framework, or to look at a selection of modules instead.

Andrew

jQuery is for very simple applications.. I think Moodle need something like Backbone as JS-framework, WordPress use the one at own admin)

I think the choice is about

- Backbone. Uses by WordPress team, but difficult

- ReactJS. Great JS-Framweork from Facebook, not so difficult and very fast

- AngularJS. Only from 2.x.x release with ES6 integration, at this moment its slow and will be "killed" in some time (1.3.x)

My vote is ReactJS.

"jQuery is for very simple applications.."

Are there no complex applications that use jQuery?

Hi Igor,

Those frameworks you suggest are great for single page applications, but not for PHP applications (Moodle is currently a PHP application). Most of the JavaScript that is used in Moodle is quite simple (so according to you, is quite a good fit for jQuery).

Kind regards,

Brian Barnes

JQuery on it's own does not do very much I agree. But it is only one small piece of what I have discussed here.

I vote for XX framework because I used it for my last project and I liked it (but I will probably move on to trendy framework YY in one month) is not super helpful to this discussion.

I tweeted a quora answer from Ryan Grove on this topic a few weeks ago which I think gives good advice in what the direction should be:

I recommend using ES6 or CommonJS modules and seeking out small, focused libraries that do the things you don't want to do yourself. Embrace ECMAScript 5 features (which all modern browsers support), and consider using polyfills or transpilers that allow you to use ECMAScript 6 features.

Like Ryan alludes to in his answer and Yahoo say in their announcement the usefulness of the big JS libraries like YUI is reducing as the language gets better - so I don't think our direction should be to look at something like jquery (argh, i said the J word! ).

In Tims response to my tweet the question comes up.. Do we do anything until ECMAScript6 modules are better supported?

I think we should - it will be a long road and we should start thinking about it now - I'm sure we could change our approach to new development to reducing our use of the monolithic YUI components and instead focus on use of the necessary needs (YUI as the polyfill..)

If we decide to use strait js (I assume that we would developed a small library in Moodle for common functions/UI consistency), we may have to take a serious look at what Web browser versions are officially supported.

Whilst a custom js library would offer more control, it would be a lot of work considering pretty much everything we could need is available in jquery in some way or another, and there is a massive community of jquery developers, with tonnes of plugins available.

I've never bothered with the Yui myself anyway, all the blocks and plugins I've written for Moodle have used jQuery, which in my opinion is the way the project should be going.

I think the key question is, to what extent do we need a module system?

We might need it need it to:

Prevent different bits of the JS on one page from breaking each other. (E.g. on a page of the quiz, you might have JS from moodle core, including big modules like the filepicker, the theme, the quiz module, various blocks, various question types and various filters.
Allow different bits of JS on the page to communicate in controlled ways. (E.g. lots of JS needs to call into Moodle core, and vice versa.) E.g. the quiz and question type JS code needs to interact with form-change-checker, if I recall correctly.
Handle rolling-up all the JS required by the page in to a few HTTP requests, while only loading the JS that is needed by the page, not irrelevant JS.

Eventually there is going to be a standard mechanism in ES6 to do some of this. We should try to do something that will lets us use that without custom hacks once it is available. (I know nothing about this, but i would guess what is in ES6 does 1 & 2, but not 3.)

Generally I would favour plain JS for as much as possible. However, there are many different ways to write plain JS code, and to keep Moodle maintainable, it would be good to establish some strong conventions for how we do things (like we have for PHP) so things are generally done in a constant way.

So, I guess a first step would be to start working on a migration guide from typical Moodle/YUI code, to how we can do equivalent things in plain JS. E.g. what is the equivalent of a Y.delegate call?

Then, there are the things that we commonly need on top of plain JS, like dialogue boxes and simple animations. We need a library of things like that. Building our own would allow us to integrate into Moodle renderers etc.

I think that building everything from scratch in Ecmascript is probably one of the worst ideas. For starters there are the cross browser issues, the edge cases, the need to do your own performance optimization.

furthermore it would be unwise to ignore the fact that many of the plugins and themes already use jQuery, and eliminating the need to load additional frameworks, even if it's custom built, will reduce the necessary JavaScript load and processing.

however there is another option that I have not yet seen discussed, which is whether Moodle wants to go dynamic data first instead of DOM manipulation. To me it would make sense to at least consider angularjs for all the backend JS, so we can finally get rid of much DOM manipulation, and instead consider the data dynamic from the getgo.

As a non-contributor to code, my opinions are probably worth less than a feather in a storm, and I am sure you realise that as systems become more complex the likelihood of error exponentially increases. Before moving away from YUI, and I remember some rather intense discussions around going to it in the first place, try to apply the idea that simple is best. Tim questions the idea of using modules, but if modules are independent and somewhat bulletproof, and can be made more robust why would you not want to use that kind of structure - no matter how it is actually coded.

Tim also mentions that plain JS is likely to be far more beneficial in the longer term than things that are currently under development. JS is well known, is stable, is in continuous development and I suspect, likely to be more backwards compatible than a lot of other languages. From a systems POV, this makes far more sense than rebuilding in an unknown in the hope that it will be alright on the night.

My only question would be is what is likely to get a smaller, simpler, codebase, that is faster to load, easier to run, easy to work with, and less likely to break? Whatever the answer, those are the measures that will largely affect the future success or otherwise of Moodle. Just don't lose sight of that guys.

For the benefit of those of us who don't know what "angularjs"is (and who are too lazy to Google it themselves) would you like the explain in a few paragraphs what it is, and why you think it is so good? Thanks.

Separation of presentation, data and logic.

You use templates that contain html with handlebar style fields inside.

You can establish relationships between elements without having to actually code any js.

Data models map to handlebar fields in templates - easy peasy.

If your data is an array its easy to loop through using an each function.

Routing - you can specify a model for each route so that if you click on a 'user profile' link for example, it would automatically load the 'user profile' model and apply the data to the template.

That is certainly a possible way to build a web application, but not the only way. Is it relevant to Moodle?

Moodle has its own ways to separate data, presentation and logic (e.g. renderers) which work in PHP on the server-side, and have nothing to do with JavaScript.

Whatever future approach we take to JavaScript in Moodle, it must be possible to get from where we are today, to where we want to be, in an evolutionary way. It seems that what you are proposing would require a complete re-write.

What I think about EMBER

NOTE: Not ember expert talking.

Probably I am not a experience ember developer, indeed I have been working with it since 3 weeks. But, I think a transition to ember would be kind of complicated. Trying to introduce ember means a lot of refactoring (personal opinion).

To introduce ember, you should have a Resource API (completely service oriented) to get a great separation between your backend and frontend. If you don't do it, scale your application will get harder and harder until the point it is impossible. Thats why a project like ember-cli appears (http://www.ember-cli.com/).

Thats why I think it will need a lot of refactoring, because as far as I have seen, not all modules inside Moodle offer their functionalities as services. Indeed, the behavior of the applications (flow of execution) built with ember are very different of what I have seen inside Moodle.

BTW, using ember arises other challenges and they are very interesting.

About jQuery inside Moodle

At the other hand, if you analyze the amount of jQuery plugins that have been used inside contributed modules, you will realize that it is increasing a lot. Bootstrap could be the reason of this in themes (another supposition).

Just saying, you can choose the option you consider better for Moodle. But I think jQuery plugins will still there. Probably there are better options, but popularity is important and jQuery will be used longer.

Improving the way jQuery plugins are included inside Moodle will help to avoid the amount of uncompressed jQuery plugins on header. And clarify (for developers) how moodle uses the compressed and uncompressed version of plugins will prevent platform crashes when debug mode is activated.

IN SUMMARY:

Migrating to ember could be complicated.

Improve how Moodle include jQuery is not a waste of time even if it is not going to be the official framework. And it can be done without break backward compatibilty.

About why AngularJS is good, one word: two-way data binding! Oh wait, that's three words, or is it four?

Seriously though, I wouldn't see Angular as an option for Moodle. The real benefit comes when the data is mostly handled as JSON objects, both on the client and on the server.

Angular is designed with node.js and no-sql-databases like MongoDB in mind so this would require quite a bit of dual-logic (client-server) if Angular is not embraced completely.

Bootstrap JS is already tied to jQuery and it has quite a bit of javascript UI elements. I wonder if there is something that Moodle is using from YUI that the Bootstrap JS libraries are not providing?

Disclaimer: haven't touched Moodle code since the last decade, and only worked on one relatively small Angular-PHP-MySQL project.

Javascript MVW frameworks like angularjs are great for separating presentation, logic and data. If you have complex logic and relationships between elements on the page then I would choose emberjs over angularjs. Emberjs enables you to bind functions to data relationships which means that when you update one element on the page (e.g. a form field) then anything that is dependent on that field via a function gets updated. With angularjs it inefficiently re-runs ALL functions per element change which is rubbish.

BTW - to explain what I'm on about, here is an example

You have a DOB (date of birth) field and a age element for displaying a calculated age based on the DOB. The calculation is a function using both DOB and age elements.

You also have a post code field which automatically updates a state/province element. This is again done by a function using each element.

With angularjs the functions for calculating the DOB and state/province are BOTH executed even if you just update one of the fields.

With emberjs the functions are bound to the elements, so only the appropriate functions are executed when an element changes - change the DOB and the age function is executed, change the postcode and the province function is executed.

On the topic of dialogue boxes, the Chrome team is trying to standardise the < dialog > element for this use case, and has a polyfill library for backwards compatability:

http://demo.agektmr.com/dialog/

The dialog element is in the early stages of standardisation but generally, taking advantage of things provided by modern browsers and using polyfill libraries to fill in the gaps for older browsers makes a lot of sense when it is possible.

Particularly when dealing with multiple device formats e.g. small touchscreen phones, a date picker or whatever widget provided by the platform itself is basically always going to beat something built from scratch in JS.

I agree with Tim. And I think we should start making a "rosetta stone" document for moving from YUI to standard JS. (like the YUI <> jQuery rosetta stone document)

My heart beats for the minimalistic option, and Ryan Grove's suggestion sounds good... but looking at CommonJS I see a pile of partial implementations. Not sure any of those is usable as an option for Moodle.

Maybe jQuery is the only reasonable option

-- I dunno, I'm a bit out of touch with the JS world.

I'd like to see Moodle focus on a mix of plain JS and jQuery. Of course it's always possible that YUI development is taken over by a third party and revived into a vibrant community, but I don't know how likely that is.

The current state of things right now - Moodle is one of the last users of YUI, YUI devs (outside of Moodle) are rare, the framework in now unmaintained, and new devs aren't interested in learning something no one else uses.

In any case, I'd hate to see Moodle stick with YUI in the long term (i.e. Freeze at the current version, start maintaining a YUI fork, etc) due to the existing investment that has been made.

EDIT: I think I should clarify "A mix of plain JS and jQuery" - I think plain JS for core, critical parts of Moodle, and jQuery for less critical parts of core and as the officially supported framework available for plugins.

Is it feasible to learn from what YUI has provided us over the years and either fork YUI and retain the basics (modular approach, dependency loading, etc), and allow in that structure for other tools, libraries, accessibility and progressive enhancement techniques?

Extending on my comment...

Alternatively to forking, how about taking the good bits of YUI and whittling it down to a vanilla JS implementation as required. So long as 3rd party components and libraries can be bolted on in a modular fashion, whether using jQuery or other bits and pieces, it would mean good for everyone.

It would mean Core would be modular and tasty - thanks YUI for the guidance there - and the juicy bits that enhance the UI or make rainbows fly out of masthead banners can be achieved using whatever tools are deemed worthy, or are popular.

I think doing the above would also allow for entry level front enders to get their hands dirty quickly, but also offer a deeper level of code organisation and pattern usage that can be learned with time.

I also feel angularJS would be a suitable replacement and agree with David Bezemer that building from scratch is not the correct solution, resources could/would be much better distributed in other area of moodle.

I'd be very surprised if they choose to go Angular. It's a big change to what they have been using for years.

I agree but at the same time still feel it would be a good solution.

It could be, it comes down to how much work things are to implement though. I mean it would be great if Moodle was more structured and followed re-written to follow a proper MVC design framework, then I could see them using Angular JS, but it's not going to happen, the codebase has grown out of control and it would take years to do.

Oh no...

That was the first thing that popped in my head as I read it. Then I read the whole discussion and decided to add my two cents.

First of all, since Moodle 2.7 is here to stay for a few years there's no need for frantic coding because the YUI will still be there.

And as for the talk between pure JS vs. JS library I think the answer is pretty obvious. Looking at the PHP code in Moddle reveals various levels of coding and styles that are unavoidable due to the large amount of contributors, and in my experience many back-end developers aren't quite as "fluent" when it comes to front-end developing and the result would be a multitude of script code that will most likely cause unexpected errors.

So in short, a JS library is a MUST. The best thing you get from using a library is a standard. A library already handles some of the backward capabilities and also cross browser issues (does everybody knows Mozilla uses a different mouse wheel event? with a library no one will find out the hard way). A good library also has good components/plugins to use all across the site, otherwise we'll end up using one calendar plugin in one place and a completely different calendar in the next page.

Now, about YUI, coming from JQuery I admit not being a big fan. This showed me the difference between a code written for other developers and a code written for web designers in mind, but still I'm not calling to run straight to JQuery.

As for creating a new library or like in YUI adapting an existing library for Moodle please consider this!

Personally, while I understand the benefits of using a modified version, it does "lock" us from fully using components. For example, the date picker component is strongly attached to a Moodle form and makes it hard to use in other places.

Using an existing library with minimal adaptations will allow us to update it more frequently and allow the full use of third party components written for the library (there was a good example in Moodle 1.9 where the dragging of elements stopped working in IE9 because YUI 2 didn't support the browser's new version and Moodle 1.9 didn't support YUI 3, so all IE browsers were forced to work in IE8 mode).

So here is my wishlist

A JS library with more extensive UI support
Minimal modifications that won't create a separate version from the rest of the world
Enabiling a modular framework that will allow calling any needed component without special requirements

Hello,

this remembers me of https://moodle.org/mod/forum/discuss.php?d=188627

;)

I am particularly happy with the end of YUI

I will be happy if Moodle just go with JQuery or even AngularJS. Moodle nowadays has more and more webservices, we could really think about decouple interface from functionality so we could have one "moodle core services" and several interfaces, like the mobile interface, angular interfaces, jquery interfaces , zepto.js interfaces, and so...

My vote at this stage would be to tend towards not using a "big" framework like Jquery but looking for "..small, focused libraries.."* that meet our needs.

Frameworks are great - being able to leverage someone else's hard work related to browser compatibility etc but they can introduce a lot of bloat/overhead and browsers have got a lot better at this stuff. If we could implement something that was light/flexible and high performing it would get my vote.

Looking further at Ecmascript seems to make sense - despite the fact that we all struggle with particular browsers they do a much better job of providing similar/consistent support when it comes to JavaScript when compared with css.

But - we should continue to support those users that want to use various elements of JQuery as we do try to make the barrier to new developers low and they all seem to have some level of jquery experience these days.

* quote from Ryan Grove

After much discussion with the RL team, I have to cast my vote for a base jQuery integration, with a limited number of jQuery modules used by core Moodle.

Our team pulled through a transition to jQuery and, after the initial difficulties, they actually find it much easier and saner. All the experienced devs at RL love jQuery now.

As folks have pointed, jQuery's popularity argument is undermined by, uhm, "varied quality" in the jQuery world... let's say that the same argument can be made for PHP . The base jQuery is comparative or smaller size to YUI, and we can pull in modules/libraries very selectively.

Un-sorted array of points in favor:

It is unassuming, it does not want to control everything (many new frameworks do). It can coexist with YUI, as it already does today, and does not require reworking our class/id names. A gradual transition is possible.
Base jQuery is small and fast, and has some excellent modules/libs. It also has less excellent ones; gotta be selective.
jQuery is popular and has a large installed base, giving us a good hint of long life.
It does not have a single large corp sponsor, it is a more organic community. This is effectively what hurt us with YUI -- folks flocked to it because of Yahoo, but that also limited its community-driven development.
It covers a lot of web browser compat; that is a very large surface we do not have to contend to.

As a contrasting point, we have recently moved to build our own with Atto. Some degree of NIH is good and healthy, and Atto is a good example. The HTML Editor is central to our UI, very visible to our end users -- it hurts us deeply even on "superficial" bugs such as imperfect look-and-feel. But it is also a burden. We cannot put every framework on our shoulders...

What is RL?,

Edit: I have just made the connection RL=RemoteLearner, one of the MoodlePartners and major contributors to the Moodle project.

IF we were going to change to a different framework (That's a big IF) - then I agree that JQuery seems to be the right choice for a number of reasons. BUT - a lot of the basic stuff we use YUI for now can be done reliably in plain JS in a cross-browser compatible way and our existing JavaScript guidelines guide us towards using the framework instead of plain JS. With the improved support in browsers I think any changes to the guidelines should allow us to use Ecmascript or plain JavaScript instead of using a framework to achieve the same purpose.

Personally I dislike the idea of having 2 javascript libraries included in core causing unnecessary bloat but realistically we are already in that position.

Editors can be changed and when HQ/Frontend team chose Atto (default editor) they were also discussing (in the side notes) about CKEditor & TinyMCE 3 and 4. The main benefit of Atto was that it was YUI code. But the main weakness of Atto was also that it was YUI code.

There is no need to create a separate jQuery editor anymore because TinyMCE (both versions) support jQuery and CKEditor offers native jQuery integration through its jQuery Adapter (a jQuery plugin basically). Full mobile support will be introduced in CKEditor 5.

So if you need to give some more weight to editor as a main component of moodle's UI you can do all the things in editor with jQuery, core CKEditor or TinyMCE and even more.

hmm - disagree with "The main benefit of Atto was that it was YUI code" - the main benefit is the improved User Interface - UI - not YUI!

I think that when HQ said

"The main benefit of Atto was that it was YUI code."

The meant

"The main benefit of Atto was that it was built using the same JavaScript coding style and development tools as the rest of Moodle JavaScript."

The first version was just a convenient short way of saying it.

Further to this, Atto is better integrated with Moodle APIs at all levels. It's not just about which JS library it's using.

Anyway I think Atto could migrate to JQuery fairly easily if it has to.

And the reason I'm saying jQuery is because it's the framework that any dev who comes to Moodle seems to already know ... for me this popularity outweighs nearly all the technical arguments. But let's see.

I had to teach some Javascript a few years back, and despite being fairly involved with Moodle already I chose JQuery as part of that course because of the wide support and resources. Also I figured if I was going to get a new skill I wanted it to be as widely useful as possible.

The important part of software is not the lines of code but the eyes that read and write that code. If the brains behind those eyes are already invested in a technology then it makes sense to adopt it. All other things being equal (which they rarely are).

It might be easier to create also new features to existing tools if you had some other people interested in the same ideas and using the same scripts. If jQuery had not been seen as a "bad word" in moodleland we could have tried integrating GPL code like http://visualmatheditor.equatheque.net/index.html to an editor plugin... that is using jQuery

Hi All,

Typically this announcement comes when I'm away and having some Internet away-time, so sorry that I'm late to the party.

Firstly, thank you to Dan for starting this conversation. It's good to start it early and to consider the best approach.

Secondly, and more importantly, I think that the important thing to remember is not to panic. YUI is not going away. The source is still open. It is still being worked on by Yahoo! and they will still accept third-party contributions. The only change here is that new development is being reduced and/or entirely removed. Releases will still be made, but no longer on the previous release cycle.

So to start putting my two cents into the mix, here are some points to consider:

YUI is not going away;
AlloyUI is a maintained fork of YUI which has until now closely followed YUI - we can switch tracks to AlloyUI to buy more security support time if required;
we do not need to rush into any change for 2.8, or even potentially 2.9;
ECMAScript 6 is nearing (June 2015) and many frameworks are changing the way in which they work, as others have alluded to; and
the question at this stage should not be "jQuery or Angular or otherPopularFrameworkToday", but "what are our key requirements for now, and in the foreseeable future".

To that end, I think it's worth considering some of the reasons why YUI has been good for the Moodle project, look at what hasn't been so good, and work out what is now important to us. Clearly, something that is familiar, has good standards, is easy to use, etc. is high up there; but what are the requirements beyond that?

For me, one of the biggest features of YUI in comparison to most of the other frameworks of a similar era (and I think it's worth bearing that in mind because the JavaScript landscape is constantly evolving) is its modularity. It has an extremely intelligent loader, and container system which modules are loaded into. This may seem an unusual thing to focus on, but Moodle is also modular and we need to try and ensure that one plugin or part of Moodle does not affect every other.

I think it's also worth pointing out that if we do go down the ES6 module route (and I'm not saying that we necessarily should), the changes we initially need to make may be minimal as YUI3 already supports loading of ES6 modules and has does since YUI 3.15.0. Since we already use YUI 3.15.0 in Moodle 2.7, changes to the loader and wrapper system could feasibly be back-ported, thereby improving backwards compatibility of plugins.

So in summary:

we need to consider our needs first;
we should not just pick the most popular framework today - we've all seen frameworks come and go;
we do not have to change today, or even tomorrow... just soon; and
it's highly unlikely that we will be rewriting in a big-bang approach - YUI2 and YUI3 will be supported for some time yet.

I've started a discussion on the Moodle Developer Documentation to act as a breeding ground for some of the requirements that we foresee.

Cheers,

Andrew

AlloyUI have put up a blog post regarding the original post from Yahoo! Essentially, nothing has changed - business as usual.

I agree with the general notion that there's no immediate need to panic, but "nothing has changed" doesn't seem like a great summary of their blog post.

They say they might try to take over stewardship of YUI. They might migrate off YUI. They might start a radical fork of YUI and dump a lot of stuff *they* don't need. But, they don't actually know what they're going to do yet and it seems like all the options are different from the current situation. At best they'll help to avoid the worst of the withdrawl symptoms while Moodle plans its next move.

Heh - fair enough, and good point. At the moment though, nothing has changed. YUI is still being used by Yahoo! and they will continue to work with it, and patch it, etc. for many years to come. In that sense, nothing has changed. But yes... changes will be afoot.

we need to consider our needs first;

Agreed - but seeing the direction of some of the comments already on that document also remember that just because we had it in YUI doesn't make it a requirement for the future.

we should not just pick the most popular framework today - we've all seen frameworks come and go;

That comment almost suggests like popularity shouldn't be a factor and I disagree strongly.

The extent of ways we've gone against the grain of the community to keep with YUI has been a tremendous time sink.

We slyly added jQuery dependencies in the MyMobile theme
Then later we had to add official jquery support so that we could adequately serve our plugin ecosystem
Multiple times we have spent time on community contributions not adding features or improving it but converting the jquery into YUI.
We have the situation where features have stalled because the best tool for the job has a jQuery dependency
When googling for YUI answers, you often get Moodle in the search results (without answers) or put in a different way - there are 1,707 stack overflow answers for YUI, compared to 524,982 for jQuery.

These are problems and we would do well to avoid going through that again.

(Note though, that when we started with YUI, jQuery didn't exist - if there is a lesson I would take from this - its that we should be prepared to change direction when the tide turns rather than wed ourselves to something for the next 10 years).

Great find on the "when we got started with YUI"

-- 2006 to 2014 is not a bad run! I think YUI and Moodle did great.

Some frameworks have a short-lived time under the sun, so we are right in being careful to switch to another framework that is likely to be well maintained for a long time.

Excellent points, Dan!

One note: development on YUI began in 2005 and YUI was released for public use in February 2006. According to https://jquery.org/history/ (alpha) jQuery did exist already in summer 2006 and the first "stable version" was released in August 26, 2006. So the gap is not long

Many theme designers might actually vote for jQuery on the basis of Bootstrap (and jQuery dependency) alone - or like to test some other new theme tools like http://www.fromdev.com/2014/08/best-jquery-based-ui-frameworks.html

Relevant post: http://www.sencha.com/blog/a-fond-farewell-to-yui

Just when I was thinking of migrating our JavaScript to YUI...

My personal preference is to go more towards jQuery, considering it is currently included in core (although is not always available) and it widely used across the web. There is also quite a large number of plugins available. If lazy loading is required requirejs should also be considered.

I am definitely against creating moodle's own as (previously mentioned) there are a large number of browser inconsistencies (including one that I have just fixed in tinyMCE recently). Using the likes of jQuery means that, not only will we have Moodle developers (myself included) keeping an eye out for these issues, we will have a much wider community looking at it as well.

just my 2 cents.

Brian Barnes

Hi Brian,

Thanks for your 2 cents too - I feel richer already ;)

As I mentioned in my reply above, I don't think that we should decide to go for a specific framework at this stage, but evaluate what we need first. For the immediate future, there is no need to move at all.

Yes, jQuery has many plugins available, and yes through things like requirejs we can leverage a loading system, but we also have to consider the quality of those plugins. Whilst most may be reasonable, we have to consider how people using different versions may fare, and how different plugins achieving similar things may conflict with one another. We should also look at how the drupal, and wordpress cope with these kinds of issue.

Cheers,

Andrew

Not going for a specific framework at this stage is the right thing to do. We don't want to end up with another framework like YUI (an overly verbose niche framework) .

What we should take from YAHOO though, is the reasons for why they have ditched YUI and use that to help us select a more appropriate framework.

Sad news on yui, but...

This is great opportunity - alongside the current work on a pattern/elements/ui library - to identify moodle's core requirements from js and build a better moodle long term.

For most user in their primary workflow through moodle its difficult to find use cases for a js lib (which makes sense given moodle's previous/long term it must work without js policy).

Currently the primary flow for most users we find is :

login - find course - view/engage in resource/activity in course

and it is quite nice when the primary workflow in a system just works without relying on a js lib.

Once you get into the admin/content creation and engagement side the use cases for js get a bit more obvious - lots of interaction with forms where we know the user experience can be enhanced with js.

In picking a js lib to use in any project i find it useful to first identify those core use cases and requirement for the js - where it would have the greatest benefit to the most users. For moodle the first place would seem to be forms.

Moodle's core and mods are more often than not just a bunch of forms - forums, quiz, content creation etc.

If no one is planning any radical webapp style restructure of moodle it might be good to look at improving this stuff moodle is made of - forms.

Updating the forms lib - selecting a js lib designed to provided the best user experience for forms, forms currently in moodle and those we haven't yet conceived that a good js lib may inspire

(footnote - this doesn't scream jquery to me)

One of my thoughts on reading this was that it shows that using a product supported by a multi billion dollar corporation is no guarantee of continued support. I'm not saying that is why it was selected for Moodle, I think the reasons were more about the reasonable choices available at the time.

just to check marcus - is that a reply to my post or the parent post?

I was replying to the original post. In the early days of Moodle I found it annoying that some people assumed that Moodle was a high risk solution because it did not have some established multibillion dollar corporation behind it. Then those corporations bought and sold the alternatives like so many toy collectors and people were expected to convert to some other product or be left high and dry. The apparent wealth of a technology backer is no guarantee of continued support.

ok - nesting looks a bit odd - looks like a reply to my post not the original. apols.

Agreed that use/cases and requirements gathering is a must, but I wanted to plop this down before I forget to mention it.

Since the Moodle UI Elements Library is kicking off and there is a need for a JS library replacement, then this is a great time to work towards a new front end that can support a re-usable element library. Google is pushing towards web components which are re-usable widgets that can be used just like regular old HTML elements that take custom attributes and can emit events. The foundation of this is polymer (http://www.polymer-project.org). Other projects in Google also have web components like Dart. AngularJS has its own version, directives, but it looks like AngularJS 2.0 is going to be compatible with web components.

Basically you can make a web component can look like <mdl-core_reveal animation="rainbows">Content to reveal.</mdl-core_reveal>. The web component is configured by its attributes and we could listen to events by doing something like revealElement.on('reveal', ...). So why is this neat?

The implementation is hidden. We don't care what the developer did under the hood to make this component. Nor do we care if the developer refactors the innards, just as long as the same attributes/events are maintained. Basically we are decoupled from the implementation.
They are re-usable just like any other HTML. Web components can be used by other web components too because they are just HTML.
Easy to document. What does it do? What are the attributes and their values? What are the events?
They are self contained black boxes. One web component will not conflict with another. This also loosely couples the components because they communicate via attributes and events, just like regular old HTML.
Easy to use. Do you know HTML? Great, you know how to use a web component.
Many more reasons, just start researching it, they are really interesting.

To see what goes into creating a web component, checkout the polymer docs. This page gives you a good idea of how they are built: http://www.polymer-project.org/docs/start/creatingelements.html

In the end, the idea here is instead of picking a JS framework or library, we are picking web components and we don't care what tech is used inside of them.

TL;DR

Plan ahead, long term
Evolve | change (with an emphasis on evolve)
Use frameworks (front and back)

I'm also not a core member but am a Web developer. Here are my general thoughts about Moodle, from an "outsiders" perspective looking in. And the YUI discontinuation has started some good discussion.

1. Moodle needs to evolve. Even though there have been recent improvements both visually and with the core framework, Moodle still "feels" old. It needs a drastic change to make it look & feel "new" and moving to an SPA would help make this change in my opinion. Other creative ways to access and use Moodle are also needed.

2. Standards and conventions. When developing something in Moodle, it always feels dirty and hackish. The core PHP framework feels like it was hacked together rather than starting from good conventions. Yes, improvements have been made but there is still a lot left. This point leads to the next..

3. Recycle, reduce, reuse. I've always wondered why Moodle hasn't moved to a solid PHP framework. Why re-invent the wheel when you can start from a framework and quickly build-up from there? Same applies to a JavaScript framework. Don't spend time deciding and setting up / programming the mundane things. Pick good frameworks and build-up from there.

jQuery vs ** The JavaScript framework depends on the type of app being created. If Moodle decides to stick with server side rendering, jQuery will probably make more sense for DOM/visual manipulation. If Moodle goes towards an SPA (which is a bigger change), Angular or Ember makes more sense.

Plan Ahead I highly agree with Andrew's "define the needs first" approach. Everyone likes to jump on their soapbox right away (yes, me included) instead of really thinking it through. There is certainly some forethought needed on planning the end-game. It's probably time to sit down and really think about where Moodle should go in the long term.

Moodle 3.0 Here is what I'd like to see Moodle evolve into for 3.0, here come my framework opinions. It's important to keep the server stuff on PHP as that's what the greater community is comfortable with for installations. Therefore, REST and RPC API's should be build using the Zend Framework (or maybe something more API specific). A full SPA should use Ember.js as it imposes convention (which Moodle desperately needs). Building an SPA will also bring the core developers that much closer to building Apps, as you can use Cordova and such to bundle SPA's into Apps. The one unknown is the CSS framework, I'm a fan of both Bootstrap and Foundation and both have big changes coming (Bootstrap 4.0 and Foundation for Apps).

That's all for now. Take it for what it's worth.

My life consists of programming for my Job, programming as a Hobby, reading about programming and sleeping. What does SPA stand for?

http://en.wikipedia.org/wiki/Single-page_application

mmm I seem to recall discussing the idea of a SPA quite a while ago, but the idea was abandoned because it was felt that it would take too long to initially load the libraries required, no matter how fast the site actually ran. It would be no good to have the first page loading in well over a minute. Unless I miss my guess that was Martin Honen and another bloke, at irt.org, a javascript FAQ site I was involved with...ages and ages ago - before broadband-on dial up then... duh. So, apart from download speeds, faster server-side processing, better graphics handling what else has changed, David?

If I recall, Angular can load things in asynchronously (parts of the app as needed). However, I'm more familiar with Ember so I could be wrong.

There are certainly things that will be better to process on the server side, which RPC's could handle.

I feel like SPA frameworks have really come along way and this is the year you'll start seeing more about them. There is certainly more improvement coming, I'm personally looking forward to seeing at Angular 2.0 comes up with. Ember also has some good things coming to help refine and simplify its usage.

Another random though in this area:

An intentional design decision in YUI was that YUI code did not look like plain JavaScript. This was intentional because it was part of ensuring that YUI code could not possibly interfere with other JavaScript.

So, instead of saying

document.querySelector('#myhiddeninput').value;

you would say

Y.one('#myhiddeninput').get('value');

While the keeping YUI from conflicting with anything else was a good thing, it does mean that there will be a lot of fairly mindless code rewriting involved in moving to something else, if we do. Still it should mostly not be too hard. Just time-consuming.

Another random though:

On the growing requirements page https://docs.moodle.org/dev/Talk:Javascript/Requirements, we have

"The ability to load different content depending on the page requested"

Damyon has already questioned that. The point I wanted to make is that we are not consistent in this respect.

For CSS we combine all the CSS from everywhere into a single file, which means that the CSS is the same everywhere, and browsers can store it in their cache.

For JS, we load separate bits depending on what the page needs.

Does this inconsistency matter? Should be be consistent?

I don't know, but I wanted to raise the question.

This relates to something I'm pondering about.

It might raise some eyebrows, but I personally believe that the bar for JS should be more strict here than the one set for PHP.

Small errors on the back-end will raise error/notice messages but errors on the front-end will probably create a very bad user experience, such as constantly popping error messages or zero functionality in displayed components or even endless loops that will freeze the page. To clarify, PHP errors are more serious but JS errors are more visible and get more attention.

This is one of the reasons I'm against the idea of using pure JS instead of a stable library. One of the best things in CSS is that you get no errors, the most you get is conflicting styles that can be resolved by editing some selectors. Doing the same for JS code will be a recipe for troubles, something that can also happen with a JS library but a lot less because of its structure restriction.

I might be wrong, but I also assume that the overhead of having a large amount of CSS loaded on a page is probably a lot smaller than having a lot of redundant javascript running on a page.

This would particularly be the case if the javascript sets up general event listeners on the page, which ultimately are fired and then do nothing, because they are not relevant to the page they are on.

Hello,

joining the new Elements API hook,and strenghten my idea exposed before about not having a PHP API for interfaces but delegate everything to html + javascript + css (maybe + web components) and interface frameworks (jquery, angular, whatever).

This has various advantages including offloading server from rendering complex html, reduces coupling of code (separating logic from interface) and so on...

Moodle has started in an epoch where javascript was a weird monster in some niches, but now it is dominating markets and is proving its efficiency (if you look outside moodle nutshell you can find it anywhere on the web) in various computer platforms, from browsers to servers to embedded.

We can choose to go on with open web, w3c, mozilla and others, or clig to self-imposed roots.

Kind regards,

Daniel

I have experienced several situations where wrong use of YUI code in various components generated errors that halt JS execution for the entire page. (causing Admin block menu not to function).

We should definitely move into a new single JS framework, at some point and after careful consideration. As everyone already pointed out: "There is no need to panic, YUI can support us a long way into the future, still".

I think we should use more and more of the working common "standardised" JS that is implemented in browsers and only use the JS framework we choose in those cases where there is no standard way to implement what we need.

Nadav,

I have experienced much worse things with the current JS. For instance the hide/show functions are tied into the image NAME which means that even replacing the image will completely break that functionality.

Also, it is trivial with YUI to write code that upon halt will break all code execution on the entire page, without throwing a clear error to where it went wrong, as the library is humongous.

So to add to the shortlist of requirements:

Proper error handling
Async operation
define best practice code standards (like not using image names as trigger!)

So you are saying YUI is to blame for crappy code developer writes? And I assume if Core adopts whatever is current fad miracle framework bugs will just vanish

I'm saying that YUI allowed for it, and more strictly defined/typed frameworks don't.

With the adoption of HTTP2 concatenating CSS for a project like Moodle starts to make less sense.

There's a nice intro here:

http://daniel.haxx.se/blog/2014/04/02/presentation-what-is-http2/

As it says though, there will probably be a whole new bunch of optimisation techniques required to get the best out of HTTP2, which people will discover as it goes into wider use. And what we do now to work around old HTTP deficiencies (spriting, concatenation, sharding) may be actively harmful.

Completely finger in the air response as I've never done any real investigation - but i've always felt like we put too much emphasis on the benefits of comboloading and modularity (beyond the beautiful modularity of including a file).

I frequently see unrelated bits of javascript completely break a page in Moodle - so the isolation doesn't really seem to be very effective to me. Most tellingly - the rest of the internet seems to cope fine without it - bbc.co.uk loads 33 js files, facebook 41. Its likely the browsers are better suited to this environment. And with the direction of SPDY and other http improvements - the 'combo loading' starts to get handled for us lower down the stack.

When I first read the YUI announcement to stop active YUI development I thought this is a tough but wise decision. My second thought was - what will happen to »Atto«?

Reasons to stop YUI development are the much changed JavaScript support in browsers and a much improved JavaScript tool landscape.

I find reading the forum postings interesting under two aspects. One is that many of the contributors against a switch to jQuery in earlier discussions are now proposing to use jQuery with the same enthusiasm. And second is that many contributors in this discussion propose to use their momentary favorite JavaScript framework.

Not only the YUI developers see the signs of the time, many articles written lately propose to use pure JavaScript over frameworks. I find it interesting that in several articles the topic is »not to use jQuery« For example http://alistapart.com/blog/post/choosing-vanilla-javascript (And within the article the link to the »You Might Not Need jQuery

http://youmightnotneedjquery.com/« resource.)

I have the impression that if Moodle may decide to go with jQuery or another framework they are running behind in time several years - at a time when the web community seriously starts to move away from big JavaScript frameworks like YUI and jQuery Moodle may decide to switch from one to another

I have the impression that if Moodle may decide to go with jQuery or another framework they are running behind in time several years - at a time when the web community seriously starts to move away from big JavaScript frameworks like YUI and jQuery Moodle may decide to switch from one to another

I am completely with you on this point. But we need to make this counter-framework argument more substantially.

Actually I think its much easier to move away from these frameworks in short order because we've been quite aggressive about browser requirements in core. But, i've my own skepticism about this approach. If we move away from frameworks in core, but all our partners and contributors still have to support older browsers, we've not really gained anything IMO..

Some examples of things that are not particularly cross browser are touch events, scrolling, positioning, drawing (think editpdf), drag and drop. These things are used where we want to create a level polish/features that are a cut above a simple "form" etc. Redoing cross browser support for stuff like this sounds like a completely inefficient use of time. Also - if we write our own libraries for stuff like this, anyone who goes to use these libraries needs to learn them from scratch because nothing outside of Moodle will use them (so it's inefficient use of other devs time as well).

I agree with the notion that there is no rush to do something about this. I would however be concerned with investing a huge amount of developer time in new features that rely on YUI. I think adding better support for jquery (probably combined with requirejs) would benefit third party devs immediately. I think there are a number of things in JQueries favour:

* web developers know it
* there are millions of plugins, examples, tutorials etc
* it has been around a long time, so it has a huge user base
* it has backing from deque (accessibility)
* it has many big name corporate backers (resilient funding sources)
* it is not so different to YUI
* it works

So my vote is to add better jquery support now, write new things with that, and slowly move YUI stuff over.

I would really have preferred to continue on with YUI - not because I like it - but because I see all this continuous refactoring as a bunch of time working on stuff that does not benefit teachers or students.

I am a fan of JQuery, but last time I looked the support for Mobile was not as good as it could be.

There are two layers here.

There is the low level, where you used to need YUI or jQuery to let you use standard DOM methods and events consistently across browsers. This layer is no longer necessary. E.g. MDL-40006 which I fixed recently. I found some old YUI2 code(!) and was able to remove it completely, rather than convert it to YUI3, or something else. (I did it that way because of this thread.)

Then there are the more complex things, where you want to work at a higher level than the raw DOM methods. E.g. drag-and-drop. A good example of that is in Atto. There, to handle selection, I notice you used a library called Rangey. That use of a specific library for a specific task seems good to me.

This seems like a mode we can adopt. No library needed at the low level. Specific high-level libraries for specific tasks.

Just like Moodle uses PHP, plus specific libraries, some we have done like lib/dml, and other that are third-party, like what we use for PDF.

+1

It should also be *easily* possible for plugins to use what ever specific libraries they want to use.

An observation here, Tim,

"...here is the low level, where you used to need YUI or jQuery to let you use standard DOM methods and events... I found some old YUI2 code(!) and was able to remove it completely,"

and

"Then there are the more complex things, where you want to work at a higher level than the raw DOM methods."

Does this imply that once a standard is set, then it really does not matter how it is reached? That is probably the wrong question, but what I am thinking is a core Moodle, not the one currently released, but an even more basic shell perhaps with higher level functionality, entirely PHP based, and little or no functionality beyond installation and the ability to accept plugins. Is it necessary that JS be used at all?

Download a User plugin to add users, What type of courses you want? What resources? What activities? If someone wants a quiz module, they download the plugin and install it. Need an assignment plugin? Download and install it. Need additional functionality? Select your plugins and install them. Every element is a plugin, so if it is not used, don't download it. All plugins will then need to be PHP, I would think. The standards are clear, the PHP ext library is already part of Moodle, is anything else really needed?

To me the problem being discussed here would be somewhat simplified. Can all levels of functionality be handled by PHP?

Having asked this, I am sure there are perfectly valid reasons to using AJAX and JS, but the underlying question really needs to be, it that low level of technology still really needed? Bearing in mind that browsers are now mostly adapted to HTML5, even hand-helds seem to cope with HTML5, Is JS still required? Or can it too be dropped?

Hi Colin,

I'll bite.

While all of Moodle's functionality could possibly implemented via PHP and HTML5 alone, JavaScript makes these a lot more user friendly. Some of the functionality that I have noticed that uses JavaScript are:

The block menu's (such as Administration and Navigation)
Modal windows.
Courses and Categories management pages
Dock
...

In short, if you remove all JavaScript from Moodle, you are making a giant leap backwards in functionality.

The trend I've noticed is more towards more JavaScript dependant applications (such as Gmail and Google docs - try using those without JavaScript), and I would not be surprised if Moodle goes towards becoming a JavaScript application in a future (with calls to a PHP API).

If I have misunderstood your argument, and you were saying getting ride of JavaScript frameworks (eg jQuery and YUI), it is certainly plausible (and not my choice) as they are largely syntactic sugar.

Thank you Brian, I appreciate the time you spent responding, and today is a good day, I learned something new and it is not even 7:30am here..

I have no real idea of the extent that JS files are used in Moodle, when I look or edit a file they usually seem to have a php extension. I would have thought that menus and docking would have moved on since I last played around with building menus in JS mumbledy years ago. (I pulled out some code I wrote then, thought "Wow, did I do that?" and am wondering how I actually did it...sad and sorry state of affairs that is...)

I understand the usefulness and cross-platform functionality of JS, but it is the word "...script" that I am concerned about. This makes it easy to read, to edit, but it is not pre-compiled which slows it down somewhat. (I know, this is the same argument about whether VB is a "real" programming language or not that was raging years ago.) If the goal is to make things as fast as possible, is a scripting language going to do that? That was the basis of my questions...

PHP is also a scripting language - like javascript in modern browsers, it is often pre-processed to make it run faster, but neither are 'compiled'.

Much of the functionality of what is described as 'HTML5', is actually extra functionality that is available for use by javascript (case in point - drag and drop upload of files from your desktop - that is described as being 'HTML5', but there is no new HTML involved, it works by providing extra events that the website's javascript can use to capture dropped files and then send their content up to the server).

Hi Collin,

if your sole concern is speed, then you'd be better off using assembly (and even that get's "interpreted" by your CPU) :D.

PHP is generally interpreted, although there ways that it's interpretations can be cached (I can't go into more details as I'm not a server admin). Also as a side note, the original intention of PHP was a templating language (using C as a back end language).

While you're right in saying that JavaScript is not compiled, there are a number of ways that you can speed up your web site. Some easy steps to do with JavaScript is minification which reduces the size of the download (and as a result also makes the JavaScript considerably harder to read - just try to read some of Google's JavaScript) - Moodle uses Shifter for this (http://docs.moodle.org/dev/YUI/Shifter#How_do_I_use_shifter.3F). I suggest you read up on Google's pagespeed tool (and the recommendations that it suggests).

I think part of the reason as to why there is a lot of JavaScript applications (as opposed to PHP or other server side language applications) are popping up is that they offload the processing from the server side (PHP etc, but ALWAYS do server side validation) on to the client side (web browser). IMO this allows the server to handle more requests faster, as it primarily facilitates file downloads, as opposed to complex HTML generation.

To tie this back into the original thread, the use of frameworks do slow page loads down, but IMO the advantages in readability and maintainability is well worth it (and as I mentioned previously in this discussion, my vote is jQuery, or Backbone if Moodle is to head down a JavaScript Application route).

Cheers,

Brian Barnes

I think what I am referring to here is the elemental point of equating complexity with time spent on doing simple things. To come back to the main thread, what will make things simpler to do for the user?

As I have said elsewhere:

It is always going to take time to create materials in Moodle, but the less time spent for a good product is a much better option. Writing maths pages in TeXLive and uploading them is a far more productive use of my time than just writing the same thing in Moodle. It does not take me long at all now, which means I can do more or do something else that requires my attention. Write and test sections in TeXLive is a lot faster than in Moodle, and I do not always have to go back and reopen the editing option every time I make a change. In this case, i understand teacher frustration as "time consumption" - things have to be easier - simpler, less time consuming.

That is really the ultimate point I suggest.

Colin,

10 years ago all (mathematics) teachers were happy to use pen and paper or chalkboard. Then came those iPads and iPods and whatever mobile or tablet devices that were using HTML5 AND Javascript. Everybody wanted to use highly responsive web apps for touch-enabled devices (if somebody just had created such for maths then)

TexLive is an excellent tool for writing LaTeX with your PC - you can install a huge package either to your local computer or to your web server if you have access to root (not just web root). You can create pdf documents from TexLive and upload them to moodle or use tex filter to create images on server side. So why would you bother using javascripts like MathJax - or editor (it is using javascript) - or editor plugins (all made with javascript).

But you cannot install TexLive to mobile devices or tablets. PDF files aren't integrated into the web experience very well. Images also aren't ideal:

They don't print well
They don't scale well
They require pre-processing
They don't align easily
They aren't accessible
They aren't interactive

and so on. The main problem in using maths on web pages is not javascript - the main problem is that there are not yet good enough (crossbrowser compatible javascript) tools for end users to help them in using maths on web pages.

Mauno... as usual you have presented a precise picture of the reality of using TeXLive and PDFs... and the issues surrounding the production and delivery of Maths online. And I am now right off topic, and this is not a point to discuss here... sorry guys...

Hello,

i like the idea of "Moodle goes towards becoming a JavaScript application in a future (with calls to a PHP API).".

Moodle have started with PHP when it was the "de facto" or "best" language for web sites.

Now things are changing and javascript are getting traction (see githut.info for an example of language statistics of github).

For those concerned about performance, javascript is getting better everyday, in browsers and in mobile (https://hacks.mozilla.org/2014/05/asm-js-performance-improvements-in-the-latest-version-of-firefox-make-games-fly/ AND http://arstechnica.com/information-technology/2013/05/native-level-performance-on-the-web-a-brief-examination-of-asm-js/ AND http://arewefastyet.com)

I would like very much if we start to separate logic from interface in moodle and expand the web services (webservices API is a really good part of moodle ;) . It also makes me think about tools like Sandman (https://github.com/jeffknupp/sandman ) that "makes things REST"

Kind regards,

Daniel

That sounds scarily much like death by thousand cuts. All different libraries have different styles, different types of support etc.
I think the big advantage of one centralized framework with plugins is that all code can and should be written in one clear style.

That way code lives beyond it's developer and functions can be re-used easily.

ps. I'm quite sure that the library is Rangy, and with 74 open issues on github, and last releases all being alpha with latest stable from 2012 I am not sure this correctly illustrates your argument of using these various small libs, but rather indicate the liability.

As I say, have a look at the PHP code. Specifically in lib/ https://github.com/moodle/moodle/tree/master/lib/. There are many different third-partly libraries there, mostly used to provide quite specific functionality like tcpds and validateurlsyntax.php. Several of them using different coding styles to Moodle.

Is that death by 1000 cuts? No. It seems to work. Why would the situation in JavaScript be any different?

Hi Tim - on the "frameworks as a compatibility shim" front, I think it is actually a sliding window.

"Old", well understood parts of the "web browser runtime API" get consolidated and no longer need special handling... unless you want to support old webbrowsers.

At the same time, we want to take advantage of the new stuff on current web browsers... which is always a bit wild because they chose different APIs or implemented the same API with slightly different quirks...

Moodle has a gigantic userbase. Our users span a wide range of web browsers, so we'll always be stretching back with browser support, and forward for new functionality (or compat with new platforms such as mobile). What is "old and standardized" shifts under us, but the underlying dynamic remains.

I will note here that Moodle is formally not supporting old web browsers but the degradation of functionality is generally moderate (i.e.: old IE versions breaks when using M2.6 mod/scorm). But we generally don't break "hard" with the past; considering that reality in core framework choices is important, I think.

We are far better off using a popular framework because it's compat code will be well maintained; chances are there will be other users pushing the envelope in both directions (new and old).

Regarding supporting older browsers while using newer stuff, I saw an interesting thing today: http://labs.ft.com/2014/09/polyfills-as-a-service/

There is a choice:

1. Use an overarching framework like jQuery or YUI, that provides all the functionality you want, but using a new syntax.

2. Use polyfills, which add the missing functionality to older browser using the syntax that is now standard in newer browsers.

Probably worth debating which of these two options we would prefer.

Given the trend for new features to (eventually) standardise as built-in functions, I think that polyfill service is a great idea, but only if its blocking/non-caching behaviour doesn't get in the way too much.

If so, it should allow us to use new features before they're available in all our supported browsers, at the cost of a slight size/performance hit. I'd expect that cost to be relatively small and more granular than than a full-on framework.

However, I suppose this only really covers the low-level DOM consistency, and the more high-level things behaviours like drag-and-drop would still need additional libraries.

That's somewhat crazy but interesting approach. Learned something there

However, that's only one of the "backwards compat" aspects that frameworks provide. They also paper over subtle things such as DOM discrepancies and arbitrarily differing behaviors. Are the polyfills so magic that they cover such things?

Read up on polyfills; so the definition of polyfills is explicitly to add new APIs to browsers that don't have them yet. No ambiguity there -- they won't cover up for DOM and general API discrepancies.

Are those gone for real from the modern browsers? I haven't done any interesting JS code in a long time... it is hard for me to imagine that the web browsers supported by M2.7 today do not have any discrepancies that need papering over.

Maybe the Trident, Gecko and WebKit teams have all secretly converged, and just cash the cheques as if they were legion, when it's just one gang...

Yep Tim,

and for special needs like the here often cited "drag&drop" support use small specialized and well maintained libraries - as the YUI people described the JavaScript landscape which lead to their decision.

Nobody seriously would suggest to reinvent the wheel for Moodle and write all JavaScript new.

The actual situation may be a big chance for Moodle to start over and eventually try the vanilla JavaScript approach. There is no need to hurry. What about taking »Atto« and refactor it with pure JavaScript. Check with all relevant browsers and find out where the problems are. If that pilot project may go well the other areas in Moodle should be ok to handle the same way.

I learn more and more that a crucial factor for a programming language/tool adoption is how easy it is to start being productive with the language/tool. I see two key factors »documentation« and »real world examples« which can be taken and modified for the own projects. If Moodle would decide to take the »no framework« road with JavaScript it would be of great benefit if a certain amount of effort is put in writing good and easy to understand documentation and collecting and carefully commenting many »real world examples«. I guess development time and documenting time may be 1:1. With such a good documentation and »real world examples« it should be »easy« to start being productive writing Moodle JavaScript. The Moodle element library may be a marvelous place.

do you realize how ridiculous "Check with all relevant browsers and find out where the problems are." is?
The shortlist is probably 20 browsers, the long list, well where do I start....

Also I think Atto should be considered to be eliminated altogether as I doubt that anyone who actually builds stuff in Moodle considers Atto to be any type of editor at all. Even basic stuff is missing (image sizing, table editing to name a few). Switching to any existing RTE will improve everyones life. Who cares if there is tight code/API integration if the result is barely useful to end users?

David, testing my web design and web development projects in my small device lab on about 10 devices and with about 20 browsers is my daily business. What's ridiculous about that? I see it as serious and responsible work honoring the nowadays user needs.

Urs, the practice itself is definitely not ridiculous, but unless your personal projects are the scale of Moodle then I do not think they are comparable. The overhead of necessary testing when going the vanilla JS road is so massive that the majority of development time would be to sort out edge cases for specific browsers. All of which can be avoided by using a lib/framework like jQuery etc.

David, you seam to have a lot of experience with pure JavaScript work. Does your experience prove the arguments of the YUI people wrong to stop developing a JavaScript framework because they see no need for one nowadays anymore?

The YUI people did not say there is no need for a framework anymore, but they declared YUI dead:

Yahoo announced that they will cease new development on their javascript framework YUI, bowing to industry trends towards Node.js, Angular, and others. The announcement reads in part: "The consequence of this evolution in web technologies is that large JavaScript libraries, such as YUI, have been receiving less attention from the community. Many developers today look at large JavaScript libraries as walled gardens they don't want to be locked into. As a result, the number of YUI issues and pull requests we've received in the past couple of years has slowly reduced to a trickle. Most core YUI modules do not have active maintainers, relying instead on a slow stream of occasional patches from external contributors. Few reviewers still have the time to ensure that the patches submitted are reviewed quickly and thoroughly.

Also many of the former YUI team members still make contributions to other JS frameworks such as Angular, Embed and Node (and yes even jQuery). There is a world of difference between saying YUI is no longer relevantand JS frameworks are no longer relevant.

Yep David I was not precise enough when I wrote JavaScript framework.

There are many JS frameworks with different approaches. YUI and jQuery belong into the same category. Node.js is a completely different area. And Angular and Ember belong into another category - they follow a simmilar development approach.

So I see value in supporting Node.js or Angular/Ember but not jQuery when one decides that the YUI kind of framework starts to be obsolete.

Angular and Embed both tie in with jQuery for UI/DOM manipulation support.
If all DOM related JS would have to be written by Moodle itself the development effort is simply huge, and there is no obvious benefit.

Also using jQuery lowers the entry bar for the slightly less skilled developers to contribute plugins that can use these functions.
Lastly, adopting a framework is not just about the cross-platform, ease of use, and popularity. It is also about having an ecosystem of experiences developers, mountains of plugins, and a vast library of resources such as documentation that make it valuable to pick an existing popular framework over writing your own.

"What's ridiculous about that? I see it as serious and responsible work honoring the nowadays user needs."

It's not ridiculous that someone does it, but it seems a waste of time that everyone doing Javascript development does it, when one of the benefits of using a Framework is someone else has done the testing for cross browser compatibility so they can get on with creating new functionality.

What others are working on lately »Improving Smashing Magazine's Performance: A Case Study« http://www.smashingmagazine.com/2014/09/08/improving-smashing-magazine-performance-case-study/

I need to decide now how to implement new JavaScript based functionality in Moodle. What do you think is the best future prove way?

Analyzing the discussion contributions and taking Martin Dougiamas' contribution here and in the related tracker task and Moodle HQ member contributions I guess it is not recommended to use YUI. I guess the best would be to implement the functionality in pure JavaScript and if there might be issues with browser compatibility or functionality not offered by HTML5/Javascript to use jQuery.

What do you propose?

PS:

Pure JavaScript encapsulated in an anonymous function to avoid conflicts with other scripts in the page will be independent of any future library/framework decisions so it might be the best solution.

The way I see it, Moodle's need for JS script is pretty basic and probably can be answered by most if not all existing libraries. Those libraries also contain commonly popular/needed plugins as well as the basic and above styles/transitions effects.

So what would I like to see in the new library? a more powerful approach to visualization and data handling. As an example I can think of something in the way of D3, where the data is processed and rendered on the client side and reduce the need for server calling on every change. Just imagine how the workflow can be affected by a more dynamic interface!

Is nobody able to answer my question?

Just read John Resig's tweet, may be interesting for the opinion building.

@jeresig: All the talks from last week's jQuery Conference (@jqcon) are all online! Dig in and enjoy!

Just watched it, thanks Urs. I think the complexity of things like pointer/touch event models in the future really make a strong argument for us to stick to a framework that can help us with as much of the work as possible.

Martin, great that you found the information valuable.

For the specialities like the diverging pointer events you mention and Google's dance around them - first they tell us that they implement them, now they tell us, that they don' t implement them - specialized small libraries and poly fills will be the solution in the long run. The actual discussion about JavaScript support of websites/web apps seams to go in this direction.

Probably a strategy taking the actual directions into account may be to use jQuery now but try to implement as much as possible in pure JavaScript - well knowing that also jQuery will not last forever. And knowing that for mobile users less data to be transferd and libraries to be initialized is more.

I think that is the right approach: Plain JavaScript for stuff that is now basic, like simple mouse and key event handlers, and most DOM manipulation. Then some library for the parts that are still to hard, like drag-and-drop.

However, that is not a complete answer. There are at least two other components.

First there is the element library (which I guess is missing yet another release). That should provide most some useful widgets like JavaScript dialogues. It should be possible to use those bits of JavaScript without writing any JS code at all.

Finally, there is the question of how it is all put together. Where to the .js files sit in the code-base? How do you say which bits of JS are required on this page? How are we going to minify JavaScript? Is Shifter going away with YUI, or is that something we keep? It is this final area where I still don't see a clear picture.

I think the complexity of things like pointer/touch event models in the future really make a strong argument for us to stick to a framework that can help us with as much of the work as possible.

Think its also an argument in favour of going for something with the popularity of jquery which is has such wide deployment that it hopefully has a better chance of iron out the weird edge case issuese across devices earlier, rather than us. (Ref: MDL-45226 MDL-38371 MDL-37528 MDL-42409 etc) - Dave's description of the mouse/vs touch thing reminded me specifically of this.

JS Parse and Execution Time

An interesting article and worth reading.

In the last paragraph Tim Kadlec writes:

"When you consider the combination of weight, parse time and execution time, it becomes pretty clear that optimizing your JS and reducing your site's reliance on it is one of the most impactful optimizations you can make."

http://timkadlec.com/2014/09/js-parse-and-execution-time/

Optimizing javascripts is one of those things that can improve loading times - my mobile takes typically 1 to 3 seconds to load pages of moodle. Most parts of the content are loaded within 1 second and the rest follows when critical parts of page are visible.

100 milliseconds is about the same time as it takes to move your finger up from a touch screen.

Luckily we have lots of tools for testing mobile or our web site performance, for example

https://developers.google.com/speed/docs/insights/mobile

and

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fmoodle.org

I say we should take a pragmatic approach here; when faced with a choice between competing technologies, the best choice is often not about technical excellence, but about practicalities - e.g. how available are skills to work with A, B, or C?

In this case, I'd pick jQuery, simply because there are a lot of experienced jQuery developers out there, and more experience generally means fewer bugs.

FWIW, I think I can speak for most of the NetSpot crew in saying we would much prefer jquery as the direction forward.

If Moodle ends up without a framework, or builds it's own (the former leads to the latter, inevitably) then we have a higher barrier to entry for new developers, almost certainly already experienced in jquery.

There is also an established culture around this lib already, so there are many examples (yes, some not so good) to follow, and plenty of documentation, books, etc.

Building on that, I think also the other effect will be to encourage proper style and technique; if Moodle does its own thing this is less likely to be consistently enforced. While core might not be a major problem (codestyle check fixes this), it is not as certain for 3rd party code, and we really want to see the best practices there too - if only so that more 3rd party plugins etc are of a high quality, suitable to consider for integration to core and so on...

All of that is invariant whether or not a lib is really strictly necessary from a technical or code size POV.

Great points.

I think the culture is a great word to use in the context of this discussion - I've changed my viewpoint throughout this thread but thats the key thing which actually brings us a big benefit rather than necessary pain.

How do we transition?

Same here. Thanks everybody for keeping very good and objective discussion here.

Have decisions been made which way to go?

For my Moodle JavaScript work I decided not to use YUI any more and to go with jQuery in new projects. May this decision correlate with the future Moodle development line?

Thanks for your answers.

Hi Urs. No, it was not decided yet. You may want to watch MDL-47036 to keep up-to-date with the discussion and the process. Good luck with your projects.

Thank you very much David for the reference to the tracker discussion. From what I read there to work with jQuery does not sound like a wrong approach.

Just posting an update here - the policy decision MDL-47036 was discussed in the Tuesday meeting and we agreed to create a "first prototype/example as described by Dan above with JQuery, RequireJS and grunt".

I'm writing the specs for this now here: https://docs.moodle.org/dev/JS_Framework_Specification

and I created an Epic (MDL-48392) to track tasks associated with this.

Please continue the discussion here and I'll keep updating the spec accordingly.

Also - thanks everyone for this useful discussion - there is lots of good stuff in here and we have remained on-track.

Cheers - Damyon

I just wanted to sort of comment some of my thoughts on this:

I think the crucial part is actually trying to make the platform which is relatively agnostic to the framework - i.e. can support in the inclusion of jquery now, but also can make it easier to use a trendier new library in a few years time. Right now we have a good model for loading and declaring dependencies on yui modules - i'd love to see it so that what we end up with is a way to declare dependencies on any different type of library - maybe a jquery plugin, but equally might be a small focused library which were discussed earlier in this thread.
My underlying thought is that jquery is the pragmatic choice now as a bridge - especially as we already bundle it, its small on its own, add-ons already use it, partners already use it, there is a huge community. Its just us stuck in the mud in core without it. But I see this landscape evolving rapidly and so I like the idea of a small step towards jquery where it makes sense, not a jump all in conversion project.
... but.. Its such a shame that YUI is so huge because that to me is the main motivation we might want to move towards a faster conversion away from it, reducing our client side bloat (1.4MB of YUI JS when logged out on the front page of my master theme_clean install)

Hi Dan,

I agree - I would like to see an agnostic approach taken. An ES6 harmony loader would enable us to do this obviously. I also agree that jQuery is a good bridge, though I think we need to improve the way in which we include it, and try to write good guidelines on how things are used (e.g. use of data attributes, etc.) to allow for a co-existing ecosystem in the future.

All that said, I'm not quite sure where you get 1.4MB of YUI JS from:

Chrome Developer network tab output when logged out on front page with developer mode disabled

That shows 113KB when developer mode is disabled (e.g. on a production site). The total page content transferred on the front page of a master installation running the clean theme is only 237KB including HTML, CSS, JS, images, etc.

With developer modes enabled it is higher:

Chrome Developer network tab output when logged out on front page with developer mode enabled

But even then, it's only 461KB of JS and a total of 1.1MB for all page content.

I suspect you're looking at the total page content, and not just JS.

Andrew

I suspect you're looking at the total page content

No, I was looking at JS files individually reported by safari:

yui size

Looks like despite my best efforts I didn't have debug mode off enough to get minification working there!

Damyon is starting to summarise thigns at https://docs.moodle.org/dev/JS_Framework_Specification. I am a bit confused by one thing there. It says

"Attributes we want in a JS framework ... dom manipulation ... positioning ... events"

Surely those basic things are now all handled well by all browsers we might support. OK, so the raw DOM methods are not quite as slick as Y.append("..."), but they work in a standard way.

Surely those basic things are now all handled well by all browsers we might support. [...] but they work in a standard way.

See the jquery talk mentioned by Urs commenting on touch handling earlier up this thread.

Looks like a good summary.

Just throwing in a personal view, but I think if we are moving away from YUI, we really really should aim to get the benefit listed regarding build tools under 'use a PHP library - maybe jshrink'. Not necessarily suggesting that option itself, but that we should if at all possible remove build products from the source tree and as a more important result, make it easy to edit javascript in moodle without having to install or run any other tools.

Although I know how to use shifter and have done it, this is a really big barrier in front of development - we have developers of varying experience and with different setups here and getting people to use shifter correctly is a lot of hassle. Particularly when JS is going to be more and more necessary as part of other developments, it's really a setback that there's a big step between 'I am completely set up for Moodle PHP development' (which is definitely far from trivial in itself) and being able to do JS development as well.

I know to some experienced developers it can seem completely trivial to get shifter set up and working, but even in a large development setup like ours, this is a barrier. (And you have to bear in mind, people are often writing something that only uses JavaScript for like one small detail - they don't want to have to go through a huge learning curve to write one short function.) I imagine for individuals who may be doing development as a side project rather than their job, it's going to be an even worse problem.

So anyhow, in general I'd like to suggest that whatever framework we use, we consider the ease of development overall, including building/minifying and other aspects. Some specific goals I'd suggest:

There is only a single copy of each JS source file in the tree.
If you turn on a suitable debugging option like 'JavaScript editor mode' (cf. theme designer mode), you can then edit any JavaScript code using the new system by simply editing the single .js file and have the result reflected immediately if you reload the page.
If you need to add a JavaScript file to a Moodle plugin that didn't have one before, ideally you should need to add only a single .js file and at most one folder (example: create a folder inside your module called 'js' and add a file called 'editform.js' in there), and add a PHP instruction to require the JS file on the page that uses it simply by passing in your plugin's name and 'editform.js'.
- If metadata files are needed these should be optional, i.e. whatever base stuff should be available on every page and you only need metadata to introduce unusual dependencies. If metadata files aren't needed because we can use something like ES6 module loading to handle dependencies, that would obviously be preferable.
There should be a single recommended structure for Moodle plugin JS files regarding namespacing and basic structure. This should be very simple and not involve complex or weird JavaScript constructs. For example this could be like the current M.mod_mymodule = { }; convention, or something else. Basically just so developers know the pattern 'here is how I make something with an init method and some other functions' and it's very easy. (Might be quite minimal if using ES6 modules for instance.)

Obviously this is only part of the requirement as I'm purely talking about stuff that makes it easy to develop, but I thought I'd write this down as my response, anyhow. I've done a fair amount of JS development in Moodle, but I've also (as you can tell from the above) had to introduce others in my development team to doing it, so I have a certain amount of experience as to (a) what is icky for me as a developer, and (b) what is difficult to get across to new developers. Basically, since there's going to be a big change anyway, it would be nice to get the developer experience right.

--sam

These are very good points and suggestions Sam. I can only second them. Thanks.

Hi Sam,

Although I know how to use shifter and have done it, this is a really big barrier in front of development - we have developers of varying experience and with different setups here and getting people to use shifter correctly is a lot of hassle.

I agree that it is a barrier and it was my main concern when we were talking about integrating shifter into the core process. However, I think the ship has now sailed on this requirement for build tools - we are going to have a CSS preprocessor for the foreseeable future.

So I feel we should do what the best we can in this situation. What I proposed in my comment on the policy issue is that look at using grunt as a single build tool. People were talking about it when we started using shifter, but now seems the world and dog has settled on this as the build tool in web development. Our themers are already using it with bootsrap 3 and it can be used to run less, phpunit, behat etc - its just like the Web 3.0 'Make'. I think that can bring a benefit, even if it doesn't eliminate post-processing entirely.

I am undecided on this ATM too. I think that being able to just edit raw js files and have all the build done automatically is very developer friendly. But there are downsides too.

If we want to do anything but minification we will need to build support for that in php.

The node.js minification (uglifier) does other neat things like source maps - (do we need these?).

Another option I want tested is writing JS as ES6 modules and using traceur compiler to convert them to AMD modules for loading with RequireJS until all the browsers support ES6 natively. There is an existing grunt task for this.

Fred also mentioned today about optionally not serving the JS through a php script (less load on the webserver - static file serving can be accelerated etc). We might need some other tricks to support this but not without a build process.

I am in the process of creating an angularjs front page for our moodle site and I came across this post and I wanted to comment on the plain old js vs jquery vs some framework. I am going to try to keep it short..

For a project of this complexity you need something more than plain old javascript, but I think most of the developers know that.

jQuery is not a framework. << read the responses jQuery is really cool and it has a lot of nifty plugins, but it is not an framework. It is not very opinionated about how you build/load/invoke it. It just does what you ask. That can be good and bad. It makes it easy to just hack in some functionality and make your thing work. It does not prevent your thing from breaking other things though.

AngularJs is cool. I use it. I won't say you should use it, but you should look at some of the reasoning behind it, try it out and try to understand why it does things the way it does. It has a steep learning curve, but I wouldn't say it's more difficult than jquery plugin development. I have a feeling "the angular way" is going to start weaving it's way into other frameworks much like jquery did. Some of the good things. .it uses plain old javascript for functionality. It was created with testability in mind. Invoking angular is done in using special markup in your html. It is growing in popularity.

It's not all great though. First they are completely rewriting it for version 2(porting code will not be easy). They dropped support for IE8 starting with version 1.3. The upgrade an support path are uncertain at this point. I still start projects on 1.2 and there is still development going on in 1.2 because a lot of users still need IE8(most of mine). You also have to do some special tricks to minify your code because of it's dependency injection scheme.

Ok that's more than I wanted to say....

It a post to the Moodle accessibility email list, Damyon wrote: "We also have a mandate for jquery (and jquery compatible) libraries."

We do? Who gave that mandate, and why? Or was it just an unfortunate choice of words?

Presumably those of us who make it to the developer meeting next week might be told, but it would be good to also have the thought processes recorded in this thread.

I talked about this (for too long) at the general developers meeting last night. Here is an executive summary of where we are up to:

DOM Manipulation/cross browser normalization: JQuery
Accessible widget library: JQuery UI but we need to wrap it to make it work with our theme system and improve the accessibility
Javascript Module format: AMD
Javascript Loader: RequireJS
Javascript build tools: 2 options - "None" or "Grunt" - we need to decide this.

"None"
You edit the JS files directly, and they are minified, concatenated and cached the first time they are requested. If in developer mode or after upgrades the cache will be invalidated. No linting step and we still need "shifter" for yui modules and "recess" for less.

"Grunt"
We can unify all our build tools so you just run grunt and it will compile less, run shifter, or minify AMD modules as required. Downsides are more files in the moodle root folder (Gruntfile.js, packages.json), a need to install node_modules locally in each moodle root folder, need to remember to run grunt after modifying JS, need to remember to commit the built files, lots of conflicts in the built files. Other things mentioned where running behat/phpunit with grunt (but we already have mdk for that).

Please reply with comments about the 2 options above.

With all these libraries in place you can create a JS module simply by creating a modulename.js file in the "amd" folder for your plugin.

mod/assign/amd/test.js

 


// Define a module with a dependency on jquery. 

// Note - no global variables are required here - even for JQuery.

define(["jquery"], function($) {

    // Do what I want with jquery here.

    $('h1').hide();


    // Define private variables and methods.


    // Return a list of public methods and variables.

    return {

        get: function() {

           return '5';

        }

    };

});

Then to call this module from anywhere we use:

 

// All modules are named "componentname/modulename" automatically.

require("mod_assign/test", function(test) {

    // Print 5 to the console.

    console.log(test.get());

});

And my opinion on "None" vs "Grunt" is that I think "None" is my preferred choice because:

* There is less to learn for new developers
* No build products stored in git
* I don't like polluting the root moodle folder with dev tools
* We have mdk for dev tools already (behat, phpunit)

I'd vote for Grunt too. I'm also not keen on having loads of dev stuff in the root, but it doesn't seem a good enough reason to avoid using it. I guess the most important thing would be to make sure that Grunt was treated as the Moodle build tool, rather than the "Moodle Javascript build tool", so we don't end up with other parts of Moodle using other tools that effectively do the same thing. It's already pretty much essential for creating new themes based on bootstrapbase, so there's some crossover there.

I don't see that it's a big burden for new developers to learn, particularly given that they probably wouldn't be writing their own gruntfile. Even if they were, it's a pretty low entry level - you can install Node and Grunt in about 5 minutes and learn how to write a gruntfile in about 10 (assuming you're already familiar with Javascript).

(Also, I completely agree with Tim: I don't think the presentation was too long at all - you've clearly done a lot of work on this and it was really useful to hear the results in detail.)

You did not talk for too long. I thought it was an excellent introduction to everything you have learned over the last few months. It was just a lot for us to take in at one time.

My vote is "Grunt". Unless you can also move recess & shifter to a "None" model of operation, we don't gain much. Moving all building to Grunt is a win.

I don't like build products in git either, but we have lived with that for a long time and can handle it.

If we have Grunt then in could consider using it for some other 'build' actions - namely as a way to let a developer run all the CiBoT checks on their local machine.

Also, even if we use Grunt, we could implement one part of the None model. Namely, we could get Moodle (in developer debug mode) to do the dependency checking. So, if you change a JS or Less file, and forget to re-run grunt, it could print a developer debug warning telling you what is out-of-date, with a nice link to the docs to explain to you what to do about this.

So, it summary, I don't think the choice is completely one-sided, but I think on balance, the extra power "Grunt" gives us is worth the small extra pain for developers.

+1 to running CiBoT checks locally.

I'm sure I don't need to tell you (but I will anyway ), that It's a real drag to have to go through the cycle of fix, backport, push, test via ticket - then having to go through it all again if the fix isn't quite right, the expectations change, or CiBot learns some new tricks.

Minor question about Mustache, since I know nothing about it and would like to start learning, and I just happened to think of this case:

In html_writer we have a helper

html_writer::non_empty_tag('div', $somecontent, array('class' => 'mywrapper');

Which means if $somecontent is not empty, output

<div>{$somecontent}</div>

but if $somecontent is empty, output nothing.

This is a small thing, but a bit more elegant than writing out many explicit if statements, or polluting the HTML with empty tags. Is there an nice equivalent of this idiom in Mustache templates?

The equivalent in mustache looks like this:

 


{{ #somecontent }}

<div class="mywrapper">{{ . }}</div>

{{ /somecontent }}

It's slightly verbose because of the surrounding div tags. If we didn't need those it would just be:

 

{{ somecontent }}

Oh, that's pretty nice

What about building complex lists of attributes? I guess the example I am thinking about is outputting the radio buttons for a multiple choice question, where the PHP is pretty hairy: https://github.com/moodle/moodle/blob/da0ef2e4cf9c02cfa0444814b4e6e9b2cb000cd6/question/type/multichoice/renderer.php#L62

(I realise that the answer may be to do even more of the work in PHP before calling the template. Also, that code is more complex than might seem necessary since it is trying to do the radio button and checkbox cases without duplicating too much code.) Still, although it is a big ask in a forum post, I think it is a valid example of a more complex situation that the templating system will have to handle.

So - yes - you probably need to do a little more "flattening" in php before calling the template for this case. Once that's done the template would probably look something like this:

                                                                                                                               


<div class="qtext">

    {{questiontext}}

</div>

<div class="ablock">

    <div class="aprompt">

        {{prompt}}

    </div>

    <div class="answer">

        {{ #options }}

        <div class="{{ #showcorrectness }}

            {{ #correct }}{{ correctclass }}{{ /correct }}

            {{ ^correct }}{{ incorrectclass }}{{ /correct }}

        {{ /showcorrectness}">

            {{^readonly}}

                {{ischeckbox}}

                    {{! Force php form handling to handle unchecked as "false" }}

                    <input type="hidden" name="{{name}}" value="0"/>

                {{/ischeckbox}}

            {{/readonly}}

            <input type="radio" name="{{name}}" value="{{value}}" id="{{id}}" {{ #isselected }}checked="checked"{{/isselected}}></input>

            <label for="{{id}}">

                {{answernumber}}

                {{answer}}

            </label>

            {{ #showfeedback }}

                <div class="specificfeedback">

                    {{ feedback }}

                </div>

            {{ /showfeedback }}

        </div>

        {{ #showcorrectness }}

            {{ #correct }}

                {{ correctimage }}

            {{ /correct }}

            {{ ^correct }}

                {{ incorrectimage }}

            {{ /correct }}

        {{ /showcorrectness}

        {{ /options }}

    </div>

    {{ ^valid }}

        <div class="validationerror">{{ validationerror }}</div>

    {{ /valid }}

</div>

Yes, suddenly it becomes a lot less pretty. This is the kind of code where html_writer really comes into its own.

Templates for optional arguments are ugly (and I don't think there is any way around that). But, livable-with.

One further thought in this area. Althought you want to do preparation work before calling the template, you don't always know what you will or won't need. For example here, you may not need {{ correctimage }} or {{ incorrectimage }}, but computing them is moderately expensive (calls to pix_url which is not entirely trivial).

A more singificant case is likely to be where you have to call format_text to prepare something, but you are not sure if you will need it.

In this case, to get performance, we probably need more classes like lang_string, that store all the things required to compute something if necessary (probably with a __toString method). I think this is what a LISP programmer would call a thunk.

In html_writer we have a helper [...] html_writer::non_empty_tag

I learn something new every day..

Has anyone done performance testing on the "None" option yet? I like the idea of not having to deal with a build tool for JS (other than Shifter for YUI modules, which should presumably only be legacy code) - but if the minification/concatenation is too slow, even though the products are cached, it may be worth the extra pain of using Grunt to avoid performance issues when the cache is empty.

I was not able to view your presentation but luckily all was recorded. It was definitely no long, these are big changes and it's great to be able to view your presentation on your research and results.

On the question of Grunt vs None I would vote for Grunt too. You are right about it adding more clutter to Moodle with its node_modules folder but using the right .gitignore config only the developer will see this folder. I have been using shifter a lot too and really like they way it works, specifically the linting and watch options. This allows me to write my YUI modules and see if I made any mistakes each time I save my .js file using the "shifter --watch" option.

What I would like to add to this discussion is that Grunt provides watch options too and could offer the same linting and watch options for AMD modules and more.

I have been using Grunt for theme development a lot. It automates these steps:

start grunt in my terminal and have it watch my /less folder
have it create rtl-css / css cleaning and lots of css specific node modules
clear the moodle theme cache
signal the LiveReload plugin in my browser to reload the open Moodle page.

So all I need to do is save changes and all other actions are automated.

Would allowing the developer to choose using Grunt vs None be an option too?

Thanks Damyon - I agree with others here that it was great to see the outcome of all that research work and I was in agreement with almost every decision you reached.

Accessible widget library: JQuery UI but we need to wrap it to make it work with our theme system and improve the accessibility

Hopefully we can also influence upstream work in JQuery UI and it would be great to see that happen - I have seen evidence that they are keen to improve in that area.

Javascript build tools: 2 options - "None" or "Grunt" - we need to decide this.

I think its a completely false economy to talk about a 'None' option while we still have a CSS build tool. If it was a real choice between 'none' and 'something' then I would be in favour of the 'none' option. But its not.

By harmonising to a single build tool I think we will make things simpler, provide the option to add additional useful commands and also have the possibility of switching out without affecting developer workflows in the future (e.g. switching css preprocessor). As a final point - a lot of our developer community are already using grunt (which is stark contrast to when we started with shifter, for example).

With all these libraries in place you can create a JS module simply by creating a modulename.js file in the "amd" folder for your plugin.

At the risk of bikeshedding I am not very keen on the 'amd' folder naming - would prefer some reference to JS - is there a reason to be technology specific there (referring to AMD)?

Regarding naming, we have been terribly inconsistent in the past:

We used to do path/to/plugin/simpletest
but then when we moved to PHP unit we changed that to path/to/plugin/tests
YUI modules go in path/to/plugin/yui
CSS goes in path/to/plugin/styles.css
Less goes in theme/name/less
then when we introduced Behat we added path/to/plugin/tests/behat

So, if you want to maintain the strictly inconsistent alternation, then you are right, it is time for path/to/plugin/js, not path/to/plugin/amd. However, we might want to take an overarching decision at some point and start being consistent!

I think this is technology-specific, and at some point we may want to move to es6 JS modules. Therefore, we should admit it, and call the folder amd, rather than using the js name now, and regretting it later.

I agree we should have the module format in the name. In a few years we may support both styles of modules etc.

We already have a less compiler in php that is used by the more theme. So "None" is still real choice.

So "None" is still real choice

No, I disagree. The performance of that is unacceptable. Have you tried using Moodle with the more theme and theme designer mode enabled? Its definitely not a solution.

Another thought / question:

Is it easy to call YUI modules from AMD, and vice-versa.

E.g. suppose a new bit of JS I am writing (in and AMD module because I am keen) needs to call some existing JS like the file-picker, or form-change-checker. Can I do that?

Or, conversely, suppose we want to re-write something existing from YUI to be an AMD module, or we make a useful new widget in AMD, will existing YUI code be able to use it?

While it is easy to call amd modules from YUI - and YUI from amd code - we should not encourage the second case because it creates more work to transition from YUI. While it is not critical to rewrite all the YUI code ASAP - once we can stop loading YUI on every page we will see a performance increase.

You can say:

require('mod_assign/somethingnice', function(module) {
// do something with module
});

Anywhere (even from YUI code).

That is good, but I think you misinterpreted me on one point, where you say

"... it creates more work to transition from YUI."

I think it is the other way around, this is necessary to make the transition from YUI possible. We need to be able to re-write core YUI modules like formchange-checker, or filepicker, or atto, to AMD at some point. If there was no way to update YUI code that uses those modules, the you would not be able to do it.

Well, in core, you might be able to analyse the dependency graph of what calls what to find an order to do the conversion where you don't need this, but you can never know all the third-party modules that call a core JS module, and you need to be able to say to them "We have changed this module from YUI to AMD, so you will need to update your code. However, don't panic, you don't need to rewrite it all. Just make this small change."

I am not sure exactly what the concern is here.

E.g. to migrate formchangechecker we need to:

1) Write a new formchangechecker module in AMD format using JQuery.
2) Change the existing YUI formchangechecker to be a wrapper that calls the new one.
3) Wait 5 years

4) Deprecate the YUI wrapper
5) Wait 1 year
6) Remove the YUI wrapper

I don't have a concern, as such. I guess I am just asking if an alternative path would be feasible:

1) Write a new formchangechecker module in AMD format using JQuery.
2) Put a note in upgrade.txt explaining how to convert your existing JS that directly calls formchangechecker to call the new module.
3) Delete the old YUI formchangechecker (without waiting 6 years).

Specifically, I was asking how hard 2) would be to explain.

Obviously, this way makes it a breaking change, not backwards compatible, but it gets rid of our technical debt much faster.

If you are prepared to do the hard work, obviously your way is better for plugin developers.

Question about grunt - does it support distributed gruntfiles (for plugins)?

An interesting use case from Sam made me think of this. I would envisage that a plugin could add some build steps and then we would be able to verify in integration by running the top level grunt build (or whatever) that built files have not been changed without source file modifications.

This is a good question, but I would like to think that in general, we take the approach of all plugins doing things the same way (Less in plugin/less/*.less, JavaScript in plugin/amd, ...) so that the single top-level gruntfile will process every plugin the same way.

While there will be exeptional circumstances whenere one plugin will need a completely custom build step, we should try to make that as rare as possible.

Hi,

got drawn to this.

Lengthy, lots of roots and shoots. I can bring it all together and set up a table...pros/cons-type thing...will do it soon.....today/weekend.

Must be honest Damyon is standing by something he sees....so having the strands together might be helpful to aid a bit more clarity for all.

back again with word doc....

hope helpful

D

cool... have that insert too

Can get job done prob today...this eve...anymore inserts and it will take me back-time wise

just sayin you lovely lot of peeps!

D

Hi,

OK.

I have attached two tables:

1. tracked info about JS lib for the decision-making process that took place. So, the rationale is clear and the doc should be useful for the future if/when further considerations are made about JQuery-long term..that is my understanding having read the 166 posts.

2. None/Grunt, again pros/cons/implications as per the thread posts. Again, hope this synthesized, tracked communication will be useful for now as well as the future.

Apols, if I have missed anything in my analysis-I am sure extra stuff can be added if need be.

Hope helpful

Dawn

Thanks Dawn - thats a very comprehensive summary. Great job.

I am not really against the "grunt" option - I just prefer the "None" option mainly because of (a) getting build products out of git, and (b) simpler workflow for new devs.

But - I have heard from a few senior devs that they would prefer the "Grunt" option and I have not heard strong support from anyone else for the "None" option so - I think we shall spend some more time fleshing out the "Grunt" code.

One outstanding question about "Grunt" is whether plugins will be able to use their own "Gruntfile.js" to override the default behaviour - and I think it's possible but we will need to use a grunt plugin and add some special code for it to work.

No worries, glad the doc is helpful.

My view from the analysis....and this can be taken with a pinch of salt....am not the one implementing the changes/enhancements...but I thought the outcomes of my reading in table 2....suggested Grunt entailed more hassley work-compared with None....

that is all from me.

All best

Dawn

i am trying to figure out why shifter / grunt is needed in first place?

c / c++ uses a compiler to crunch all the coding into minimized code, and then the code is actually ran.

php / javascript / html, is being complied at run time. and there is a time limit of how long the compiled version (minimized version) of the code is stored in cache.

the only time code changes happen is during development, installation, upgrading, un-installing. with emphasis at development.

give me....

current moodle (php, javascript, html, and other files with full comments)
give me an option to automatically...
1. remove all comment lines, in all files
2. remove all extra misc readme.txt like files
3. remove return / new line types from files (excluding language files)
4. convert all tabs to 1 space
5. convert 2 spaces to 1 spaces ((looped say 20 times))
  1. single space needs to be left for some php / javascript / html / css pending on type of coding styled used, html only renders 1 single space. so language files are safe converting 2 spaces to 1 space.
6. allow me to select specific variables names (tagged for database names) all other names / functions get reduced down to 2 to 4 letters
save above changes to a new folder. and auto adjust config.php to point to new folder for "production / compiled run" errr minimized version of moodle that includes complete minmization of php, html, javascript, css, heck even SQL statements get minimized to an extent
1. take the load off the server cpu of having to read in extra longer size files.
2. take the load off the client browser, by delivering a fully compacted minify html/javascript ((no tabs, 1 space between things if that, min amount of return lines if any))
  1. firefox and chrome dev tools auto "un-minify" code to a readable format. for development.

doing above can produce a large "increase" in performance. and it gets away from all the "run time checks / cache of information" having to load from hard drive and reading over a few 100,000 spaces of code, reducing variable names reduces code even further, it gets more near point of c/c++ compiling to machine code. not completely but.

==========

why do i need command line doings for javascript? for moodle development?

YUI even jquery, already has base ability to "auto load" stuff i thought? and would load dependencies as needed when needed? yes with extra calls to the server.
what is difference between, sending a "full packaged up minify" YUI and/or jquery package to the client. so it is cached in the browser. vs trying to send only what is needed to the browser in smaller chunks?
is there an issue with browsers not caching javascript long enough?
issues when folks are roaming through different wifi connections? and javascript not caching correctly?
http vs https? not caching info. and need to be re-loaded on each page request?

at moment... it seems like i need to work with 3 or more versions of javascript. i understand need to build up a dependency list. but i thought that is what the browser is already doing? along with the YUI and jquery auto loaders at the client side? is this being done now at server vs at the client? stripping away all javascript that might deal with IE and chrome, and mobile devices, and just sending me firefox for desktop computer javascript code? and if on a mobile devices, only sending me just moblie javascript (stripping away desktop computer javascript and other browser javascript checks?)

is it going through entire www.yoursite.com/moodle/* and all subfolders looking for dependencies. and creating a single "javascript cached file" vs sending the entire full YUI and/or jquery info to my browser?

================

what am i achieving using shiftier and/or grunt? now vs what is wanted in the future? not talking about YUI to jquery conversion. but after jquery conversion...

There is no point minifying php - it is not send to the browser so it doesn't waste bandwidth etc.

We already minify CSS and JS. The open question on this thread was about whether you want Moodle to do the minification for you automatically - or if you would prefer to run "grunt" manually after making a change to a javascript /CSS file. There are pros / cons to both methods listed earlier in this thread.

As to what can grunt achieve? - if we choose grunt, the goal would be to wrap all existing build tools in grunt (shifter, recess) so there is only ever one command you need to run "grunt" and everything will be minified, packaged, compiled etc.

On distributed plugins. You can put a gruntfile in a component, but it will only be "seen" when grunt is run from that component. If we run grunt at the top level of moodle it will will only run the global config tasks - it just wont see anything in any other gruntfile.

Cross post (sorry there are 2 very related threads going on).

Progress report!

I have a chain of issues in an Epic for this new template work (and the JS Framework work) - and they are ready for more eyes.

The JS Framework issue (MDL-49046) has had a good peer review from Andrew and is up for integration already - the rest are up for peer review - but I also want external feedback, particularly for the templates.

There are draft developer docs explaining the new systems.

Epic: MDL-49045

New Javascript module support (including jquery).
Issue: MDL-49046
Dev Docs:https://docs.moodle.org/dev/User:Damyon_Wiese/Javascript_Modules

Templates
Issue: MDL-49152
Dev Docs:https://docs.moodle.org/dev/User:Damyon_Wiese/Templates

Another thing to mention is that - right now JQuery UI is not a done deal.

It has incompatibilities with bootstrap. It's dom structure and theming system is alien to what we have used before. It's accessibility features are only "good" not great.

In the meantime, what I am proposing is to use AMD "wrappers" that still call our old YUI implementations. That way we can write new code in AMD without using YUI directly, and later when we have a good alternative we can replace those wrappers.

It would be terrible to have different looking/behaving UI widgets all over this place - so this approach at least keeps things consistent while we start transitioning.

By the way, if you have not already seen http://html9responsiveboilerstrapjs.com, it might be worth a look

I have a high appreciation for your average posts Tim, so I did have a look. Silly me.

It is a spoof, right?

Probably shows the shallowness of my appreciation here in this thread.

-Derek

Thanks for the link Tim, we will try and add support for "cross-universe compatible" real soon

i am getting a tad over whelmed by use of extra folders for any plugin. they are getting so deep, for only a couple files. KISS (keep it simply stupid), example amd_filename.js? amd_src_filename.js? vs amd/src/something.js amd/mini/src/something.js

i realize you are using names now to explain what is what, and a folder structure to mirror the explanation. but could things be dropped into moodle/mymod/ or moodle/mymod/js/ scrolling a list of files is so much easier, than it is to click back and forth between different folder depths.

js files are different from php files, etc... only a few uses of "form_" and a couple other uses of underscores _ that i know of that is used currently within moodle. perhaps issue with cache and how they handle things.

==============

explanation of templates i understand, and loving it.

though i am tad confused use with renderer.php and javascript template usage. not seeing the 2 coming together. i see javascript template good use of ajax ability, but then renderer.php i am not fully understanding the split or reason for the split.

===============

is node.js going to be required to be installed on production servers? *would be nice* but can your setup handle different node.js version on given server environments?

==============

i would like to offer better feedback, but you project is so intertwined into everything else. that it goes above my head.

Hi Ryan,

I'm not convinced that just putting things into the same folder is any better. We'd just end up with hundreds of files in the same folder, rather than something you can clearly see the structure of with tools like `tree`.

Not quite sure what your third paragraphs means.

Templates: We render the templates both in php, and in JavaScript. This means that we only have one template system, and it is reusable between the two. You can write code which is usable in php, and which you can also update using JavaScript. Awesome sauce news for all of us IMO.

Re Node.js: Nope. Not on production. It's not realistic to ask people to install node.js on production servers for a php application. It's architecture dependant so we wouldn't be able to ship it with Moodle anyway. This has never been something that we've expected to happen and has never been an expectation.

Hope this helps,

Andrew

when i goto moodle/mod/any core plugin.

lib.php
locallib.php
mod_form.php
module.js
settings.php
version.php
styles.css
view.php

above are in essence core files for any mod (activity/resource)

===========

yes i agree with you some Andrew. there are some mod's that have 30 to 50 files in the root folder of the mod. most of these extra files = extra pages for the given mod. quick click through, and i am seeing about 10 mod (activity/resources) out of 24 that that have between 10 to 35 files in the root folder. everything else = less than 9 files in root folder.

quick click through rest of folders in moodle. there is not that many overall files in most folders. including there subfolders. across everything within moodle.

do not get me wrong with above. if i goto a friend / family member computer and having to retrieve backup stuff of theirs and it is a never ending list of files. and first thing i attempt to do is begin tossing stuff into organized folders. i like stuff in folders, that are neat and organized. but i am questioning if to much.

===========

when initially getting back in track with moodle. it was problematic... going into www.yoursite.com/moodle/mod/someplugin/db the upgrade.php, install.php, access.php, etc.. these had very specific file names and were common for nearly everything. and to this day i am scratching my head. why put these files into another folder. if they have a very defined file name / meaning to moodle.

and when i am looking at... amd/src/pluginname.js (a few folders deep). i am scratching my head. why not just call it amdsrc.js the file name is declared in moodle like many other specific file names. and drop it into the root folder.

**maybe issue. i am not expecting to see a whole bunch of files per plugin type in amd/src and only expecting 1 or 2 files if that. per plugin (any type of plugin within moodle for that matter).

============

another issue i have had. is using the plugin_name as the file_name for some specific folders/files. vs just declaring plugintype_as_file_name.php it got rather old fast. when i go around looking for miss-spelled filename. due to i forgot to rename the filename to = the plugin_name. in code no problems. but folder/file structure?

Hi Ryan,

I think that this is branching into the off-topic realms. I will quickly reply, but I don't think that this kind of conversation is going to benefit the rest of the topic.

Moodle uses a range of directories and is unlikely to just stop for no real reason. There are only two types of logical groupings available to us when it comes to the filesystem: filename, and directory; we are just making use of them to organise our content. Yes, technically we could just flatten everything, but how does that benefit us? If anything it makes our lives far more complicated. If we want to find files, we have to do a deep scan - we can't just look for the contents of known directories like we do frequently now.

With regards the amd directory, it makes sense for us to keep things in directories. At a glance I can see what amd modules we have. We allow for multiple amd modules per moodle plugin, so amdsrc.js is not going to cut the mustard when there are five modules. Furthermore, we can more easily apply rules to different types of code (such as those that we apply in the codechecker), and this helps to automate the grunt build process.

Andrew

> Downsides are more files in the moodle root folder (Gruntfile.js, packages.json), a need to install node_modules locally in each moodle root folder, need to remember to run grunt after modifying JS, need to remember to commit the built files, lots of conflicts in the built files.

I just thought about if it's possible to wrap the grunt deploy process into a php script (and fire it just when a new moodle version or plugin is installed)? If we could automate the grunt step we get rid of build products in moodle and save the additional step during development and lowering the entry barrier for new developers like with the 'None' model.

And anyway, it's also possible to not store the deployed files in git but run the deploy process manually on every install. I don't prefer this because it is the worth of both models from perspective of usability. Just want to mention it.

Hi Johannes,

Yes... technically it is possible to wrap the grunt deploy process in a php script, but there is no benefit as such:

to call it during development, you'd have to run a php script instead of `grunt` - you'd just be abstracting the process unnecessarily;
to run it on new moodle version or plugin install, your moodle directory would have to be writable by your web server - although many people do this, a majority of service providers do not (e.g. Partners hosting a large number of sites); and
to run it on your web server, you would have to have nodejs and npm installed on your server - this is not feasible for many people on entry-level hosting, etc. and undesirable for many others.

If we were to go down this route for those that could support it, and provide the existing methods for all others, then we would be doubling up on our processes for no perceivable gain.

Yes, we could not store the deployment files in git, but one of the benefits of storing the files in git is that we know that each release and install has tried-and-tested compiled JS files. These are in the same consistent state and have been linted, and minified.

Best wishes,

Andrew

Any thought/consideration for MooTools or or Prototype?

https://en.wikipedia.org/wiki/MooTools

https://en.wikipedia.org/wiki/Prototype_JavaScript_Framework

General developer forum

Moodle's future with YUI

What I think about EMBER

About jQuery inside Moodle

IN SUMMARY:

JS Parse and Execution Time

Have decisions been made which way to go?