multilang done 1,000 times more efficient, more comprehensive, more flexible

multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -
Number of replies: 22

The multilang filter is nice in that it allows users to enter text in multiple languages at once.  However, it's slow and rather limited. I just tested a VERY simple solution that makes multilang:

  • Much faster
  • More comprehensive (it works _anywhere_ on the page)
  • More flexible (even use images, such as <img class="multilang" lang="en" src="stop_sign.en.jpg">)


The documentation warns that enabling the multilang filter for headers makes your site slower. 

In fact, enabling it for content requires the server to do a lot of work, slowing things down.  As some of us know, it also doesn't work everywhere- there are plenty of tracker issues about "multilang doesn't work here, and there, and it was forgotten over here".  I'm sure there are plenty more remaining.

It occurs to me that since IE7 (in 2006), browsers have been able to handle multilang tags in the same form Moodle uses, without Moodle needing to process every bit of text.  Moodle now requires IE9 or above, so we can use the functionality of "modern" browsers to make this far more efficient and flexible. Users don't really need to do anything different.  By changing the PHP code a bit we could just make it much, much faster by letting the browser do it's job - and the browser won't forget to handle it in names of things or the other dozens of places where multilang support is missing in Moodle.


Leveraging the fact that it's 2015 and we don't have to support browsers from 1997 anymore, just those made in the last ten years or so, multilang doesn't need to be limited to just span, either. The browser can easily handle div class="multilang" or img class="multilang", to show images of signs or other images that include language.  This could be achieved by simply having Moodle emit small bit of CSS:


.multilang[lang]{visibility: hidden; height: 0px; width: 0px; }
.multilang[lang~="en-us"],.multilang[lang~="en"]{visibility: visible; height: auto; width: auto; }


That's it, to completely replace almost all of the multilang filter. Just tell the browser to not display any multilang chunks other than the current language. All that parsing and processing replaced by one CSS statement. 

To replace the bit of multilang that allows for either a specific language dialect like "en-us" OR the parent type "en", we place the most specific first and add this line to hide the more general version if the specific version is already present:

.multilang[lang~="en-us"] ~ .multilang{visibilityhiddenheight0pxwidth0px}


What are your thoughts on doing this in this much faster, more efficient, and more flexible way eventually?  The one tradeoff I can think of is that as the old multilang which parses all of the html is deprecated, we'd add an instruction to put the most specific language first (the en-us version would come before the generic en version, unless they are one and the same).

I started to post this in the language support forum, but it occurred to me that wouldn't give a representative sample of opinions.  Those users and devs focused on language support may well be more willing to sacrifice much performance in order to avoid the slight change of having the most specific listed first.  Moodle users and devs in general may be more supportive of the increased performance and flexibility, and less concerned about the "cost" of needing to put the en-uk version before the generic en version.




Average of ratings: -
In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Nice solution from a technical point of view.

The obvious down-side is that all users end up downloading all the text for all translations, even thought they will only see one version of the text. That won't be fun on a mobile phone contract.

Also, it is more work on the client-side for browsers, even though it is less work on the server-side.

I wonder if any of this is measureable, e.g. with browsers' developer tools?

In reply to Tim Hunt

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -

> I wonder if any of this is measureable, e.g. with browsers' developer tools?


1%. That's one key measurement.  The html for a quiz page, for example, is about 85KB.  Of those, about 1KB is the question and answers.  So the difference for including it in two languages is just over 1%.  Suppose the user typed something much longer, maybe 8 paragraphs like my initial post in this thread.  That's 3 KB, or about 3%-4% of the html on the page.


Also, it is more work on the client-side for browsers, even though it is less work on the server-side.


0.005%.  The browser is already applying the ~ 5,5000 lines of CSS Moodle gives it for the page.   This adds three lines, or 0.005%. The browser ALREADY has to parse the HTML and consider the classes, etc, in order to apply the existing CSS.   We're just adding 0.005% more CSS.


The server, on the other hand, doesn't have to parse all of the html, except to apply this filter where it finds span class="multilang".  (Even if the server did separately parse it for some other reason, the multilang filter has it's own, separate parsing step which could be removed).


In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Performance is not simply related to the volume of CSS.

A rule like #some-id is massively faster to match than one like .multilang[lang~="en-us"],.multilang[lang~="en"]. Hence my question about is client-side performance of this measurable.

In reply to Tim Hunt

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -
I'm fairly certain it isn't going to be measurable.  You are of course free to try to measure it, I just don't think you are likely to have any success. Yes, a class selector is slower than an id selector. Right now, there are about 10,097 rules class selectors in the CSS.  Some are fast, some are slower. We're talking about one more class, with three rules. I don't think it'll be measurable whether it's three faster rules or three slower rules.


Suppose you have 11,000 people in a stadium.  You want to weight the stadium, then add one more person and weigh the stadium again and try to detect the difference.  I think it doesn't matter whether you add a skinny person or a fat person, you're not going to notice any difference by adding one person to a stadium that already has 11,000 people in it.  Not even if you add a REALLY fat, 160 kg person will you be able to notice by weighing stadium with a million kg of people in it. Feel free to try to come up with a way to measure that, I just don't think you're going to have any success.  Let me know if I'm wrong and you're actually able to get consistent measurements.

In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

The one issue I can see is that we have users on one campus from different language backgrounds (e.g. Welsh and English/English and Arabic - or even multiple languages). At the moment they can sit at any university PC and change the language on the front end of Moodle and the multilang filter can do its work to present different content (We've already altered it to divs as well as spans).

How would this impact those users - ie shared PCs with no access to browser settings?

Also with Tim's point about downloading all content and then just using css to hide it - some of our multilanguage content includes videos and graphics in that language, all embedded within a single div for each language. Depending on the nature of the content, serving it all up and then just using css to hide it may be more detrimental to performance than using the filter itself.

- Quick to point out I'm not an expert in any of these areas, but I have done a lot of playing with the multilang content over the last 6 months for some Welsh/English bilingual content, so responding with my initial thoughts as part of your discussion.

In reply to Richard Oelmann

No change re settings and shared computers

by Ray Morris -

> How would this impact those users - ie shared PCs with no access to browser settings?

No change.  It continues to work in exactly the same way it currently does. Rather than Moodle parsing all of the HTML and changing the HTML, it just changes the CSS.  This can still be done via the drop-down selector, auto-detect if that's turned on, etc.


Also with Tim's point about downloading all content and then just using css to hide it - some of our multilanguage content includes videos


I believe the browser generally won't play, and therefore won't download, an invisible video.  That's something to test, though.

In reply to Ray Morris

Re: No change re settings and shared computers

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

Sounds good, Ray - have you tried it out? Sounds like something that could be mocked up through a theme in the first instance as the content would all be through html editor itself, the css could be added to the theme (although eventually automated by detecting the language menu?)

Would we still need some kind of filter to add the multilang to the content though? (I know it could be added direct to the html, but many users will not want to do that)

In reply to Richard Oelmann

Re: No change re settings and shared computers

by Ray Morris -

I've tested it out in a bare proof-of-concept way outside of Moodle. I may very well be going all-out with it over the next few weeks, depending on the needs of the department we're assisting in this instance.


Would we still need some kind of filter to add the multilang to the content though? (I know it could be added direct to the html, but many users will not want to do that)

That might be a nice, separate and orthogonal improvement.  The multilang spans already need "class=multilang" added to them with the existing implementation, no change to that is required or implied by this current suggestion.  We've only changed the display rules to be handled on the client (which is already handling 5,000 other display rules aka css) rather than the server. 



In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by ryan sanders -
it is still way much "code"  for text input boxes. with a limit of 255 characters at the database. without funky core edits. both at database and in code for checking length before saving to database


all i care about is.....

lang="en"  or lang="en_us" 

add some brackets around it.

<lang="en">  or <lang="en_us">

or

<div lang="en">  or <div lang="en_us">

or

{lang="en"}  or {lang="en_us"} <===== personally prefer this one.  quick and fast.  and let a moodle -> filter deal with adding css / html markup to the entity.   14 characters vs 20 to 30 characters just to change to different language. in a 255 character limit text box....


In reply to ryan sanders

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -

Ryan Sanders wrote:

or

<div lang="en">  or <div lang="en_us">


That happens to be the HTML standard decided upon by the World Wide Web Consortium six years ago, and almost what my suggestion implements.  It's similar to / compatible with what Moodle has been using, but not exactly the same.  Maybe someday will choose to do that.  To do that, the CSS would be something like:

*[lang]{visibilityhiddenheight0pxwidth0px}

*[lang~="en-us"],*[lang~="en"]{visibilityvisibleheightautowidthauto}

*[lang~="en-us"]*[lang] {visibility: hidden; height: 0px; width: 0px; }



In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Damyon Wiese -
Note that we couldn't use this syntax because it would prevent you writing mixed language content (you would only see one of the bits of content at a time depending on your language). This is supported by standard HTML (and is useful) and we would be breaking it.
In reply to Damyon Wiese

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -


Note that we couldn't use this syntax because it would prevent you writing mixed language content (you would only see one of the bits of content at a time depending on your language). This is supported by standard HTML (and is useful) and we would be breaking it.


Could you please elaborate on which syntax you're referring to exactly, and what would break?  My original suggestion is to keep the syntax exactly the same as it already is in Moodle. We'd just be moving the display processing from the server to the client. Is Moodle doing something I missed in my CSS re-implementation?  The only "syntax" change should be that <div class=multilang ... can be used just like <span class=multilang ...    Well, there is the secret "secret" bit is that it actually allows more flexibility because you could do things like have the same text for two languages:


<span lang="en af">Jesus</span>
<span lang="es">Jesús</span>



In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Damyon Wiese -
I was referring to this syntax:

<div lang="en"> or <div lang="en_us">
In reply to Damyon Wiese

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -

Damyon, I'm not sure which of three things you're wanting.  Two are included in the CSS I suggested, the other I've never seen implemented anywhere.  I think you're wanting the first case, which is included.   This is easy:

<div lang="en-us">Flashlight</div>

<div lang="en">Torch</div>


That's the last bit I mentioned in my initial post "the most specific language first (the en-us version would come before the generic en version, unless they are one and the same)."


You can also do this, where multiple languages have the same text:

<div lang="en es">soda</div>

<div lang="fr">soudeo</div>



Are you saying you'd like to do this?:

Write text in Spanish and in English.

Some users see the Spanish text.

Some users see the English text.

Some users see BOTH the Spanish and the English.


Indeed I don't think that's supported in Moodle, and if it's supported in HTML I'd like to see a reference for it.  The specs I've read gives a list of languages by PRIORITY, and it uses the ONE language with the highest priority.


In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Damyon Wiese -
No - I specifically DO NOT want any hijacking of the HTML5
syntax. It exists to mark sections of a document as being in a different language to the rest of the text and the browser uses this information to e.g. show the text in a different font.

There are valid cases where you DO want text in Spanish and English and you always want all users to see all the text (Like a spanish course for english students).

We must not try use the lang attribute to hide any text - it must be a different syntax like the class or {{multilang}} or something else.

(I am trying to be very clear).
Average of ratings: Useful (1)
In reply to Damyon Wiese

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Michael Milette -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

I agree Damyon. Here is an example that we often come across.

From time to time, we need to add to a page the name of a business, product or publication which is officially only French or in English. It would be inappropriate to translate these name so we need to use the HTML5 lang="xx" syntax to ensure that these words are pronounced correctly by screen readers.

If we were to hijack the HTML5 syntax, we would no longer be able to meet this accessibility requirement.

Best regards,

Michael

In reply to Ray Morris

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Michael Milette -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Ray,

Sounds like a pretty good idea. Would it not make more sense to use display:none instead of visibility:hidden? I also agree with those who mentioned that delivering multi-language content to the client browser and letting it sort things out is potentially problematic because you are delivering content multiple times over a relatively slow connection (though enabling gzip compression can help) and then hoping that the client end has the CPU power to make things happen fast enough to ensure that students don't get board waiting. I also have to wonder if screen readers would all handle this properly. Accessibility is becoming a major concern for more organizations every day. As an example,  WCAG 2.0 compliance and has been a major priority for the Government of Canada for several years now. In the US, I believe it is Section 508.

All that said, we've since moved away from filter_multilang. Our content editors find it too difficult to work with for the following reasons:

  • They have to edit in HTML which can be challenging and confusing because the editor doesn't provide a way to tag text in a particular language.
  • Once entered, there is no way to be able to identify which text is tagged in which language.
  • Because you have to use <span> tags, you can't just tag who blocks of content. You have to almost tag each line of text which makes for a lot of tagging.
  • Finally, as someone already mentioned, you are limited to 255 characters (or less) in many places.

Instead, we have been recommending implementing filer_multilangsecond (Filter: Multi-lingual Content) for a while now. We've had great success in using it and our clients love using it because it is so much easier. It addresses all of the above mentioned issues and more including:

  • It uses plain text {mlang} markup tags - no complicated HTML editing or syntax required. And if you make a mistake, it's just plain text. No surrounding HTML code to mess up accidently.
  • Since {mlang} tags are in plain text, you can easily see what text has been tagged and in what language. Example {mlang en}English text{mlang}.
  • {mlang} tags can be used inline just like the usual filter_multilang <span> tags but it can be used to contain large blocks of content. So for example, we oftenr put all of our English content for a page between {mlang en} and {mlang} and then put all of the French content between {mlang fr} and {mlang}. As a result, the server literally need only filter two blocks of content (or 3 if you have 3 languages) instead of dozens or even hundreds of <span>.
  • The tag syntax is really short by design which means you can actually fit a lot more text or translations in a 255 character field.
  • I don't think it would not be too difficult to create an Atto plugin that would allow you to tag selected text with {mlang} tags.
  •  a regular Moodle filter so it will work everywhere that the filter_multilang currently works and even in some places it doesn't.

I really hope that filter_multilang second eventually makes its way into Moodle core.

Best regards,

Michael

In reply to Michael Milette

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Ray Morris -

Because you have to use <span> tags, you can't just tag who blocks of content. You have to almost tag each line of text which makes for a lot of tagging


That's an important improvement implemented by both filer_multilangsecond and by my suggestion.


Would it not make more sense to use display:none instead of visibility:hidden?

That's what I tried at first, but it doesn't work because the logic is:

Don't display multilang content.

Unless it's the language we want (en, en-us, or default).


So we'd have a rule like:

.multilang[lang~="en-us"] { display: block; }

or is that:

.multilang[lang~="en-us"] { display: inline; } ?

Maybe:

.multilang[lang~="en-us"] {display: inline-block; }

or:
.multilang[lang~="en-us"] {display: table; }


Since "display" isn't a boolean on/off value, we can't use it - we don't know what the correct "yes, display this" value is.  Let me know if you think of any way to invert the logic and have it work in all major browsers.


In reply to Michael Milette

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
> I don't think it would not be too difficult to create an Atto plugin that would allow you to tag selected text with {mlang} tags.

We did just that for TinyMCE (because we started working on this a loooong time ago, before Atto was available, but stopped working on it for nearly two years and only restarted work on it 2-3 months ago).

We use a very similar markup to filter_multilang, we started using a completely different syntax, but when we restarted work on it and found filter_multilang, we switched to their syntax (but used a longer name).

You can see demo of it in the following video (sorry for my English accent, I'm not a native speaker smile):



I plan to publish the git branch with the development in my Moodle github repo in a few days (I'm swamped with other work at the moment) and if there's interest in them, to upload the two plugins (the content filter itself and the TinyMCE editor plugin) to the Moodle plugins database.

Saludos.
Iñaki.
Average of ratings: Useful (1)
In reply to Iñaki Arenaza

Re: multilang done 1,000 times more efficient, more comprehensive, more flexible

by Michael Milette -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Iñaki,

That looks great! I love the integration you've done and definitely see the obvious usefulness, especially to non-technical users. A few suggestions:

  1. Is there a reason you are using {multilang} instead of {mlang}? The reason I ask is because I would be interested in using your improvements however I already have several established production sites with tons of {mlang} tags in them. Alternatively, I don't imagine it would be too difficult to modify your code to use mlang instead of multilang.
  2. Regarding the language dropdown. I noticed you have many more languages in the list than those available on the site. Any chance you could only list languages available on the site?
  3. I noticed one difference between the mlang plugin and your plugin is that you allow standalone {multilang} tags that don't require adjoining alternative language tags. For example, in your video, you had Spanish on one line and French on another in your content. Have you taken into consideration the fact that you might end up with empty tags when doing this? For example:
    <p>{multilang en}English text.{multilang}</p>
    <p>{multilang es}Spanish text.{multilang}</p>
    When filtered, would I end up with an empty set of <p></p> tags after everything has been processed through the filter?
  4. If you haven't thought of it already, you might want to have an option to only have the language dropdown to non-students.

Those are all my suggestions for now. Looking forward to seeing your final set of plugins, especially an Atto version (since Atto is supposed to be more accessible).

By the way, I had no trouble at all understanding what you were saying. Your accent is not an issue at all.

Best regards,

Michael Milette