## General developer forum

### Implementing pluggable video/audio players in Moodle

Implementing pluggable video/audio players in Moodle

There is some interesting discussion going on MDL-38158 at the moment and I would like to bring more participants/opinions into it.

Initially this issue was about introducing Flowplayer HTML5 (see another forum discussion) but since nobody can actually agree on the choice of player, the issue was renamed into "Introduce pluggable media players in Moodle".

Moodle will ship one of the nice JS players with standard package and individual users will be able to substitute it with a different one.

First of all, let me outline the necessary work to be done that I hope everybody agrees with:

• Allow including <video> and <audio> tags in the content even for non-trusted users (students): see MDL-54847 and MDL-55581
• Create atto plugin (most likely update existing atto_media) to use <video> and <audio> tags for including media: see MDL-55324 - we should be able to add more than one source, subtitles tracks, specify dimensions, etc.
• .... more small issues, see the new component Media in tracker (feel free to add it to other existing issues if you find appropriate).

Now the contradictory part - how to "plug" the media players?

Several years ago media settings were separated from filter_mediaplugin and formed their own core_media API. It can be customised by themes. This is not the best solution because we often want to have one player that can be used by many themes.

By the way, I have not found any themes in Plugins directory that overwrite core_media_renderer, if you know any - please let me know and ask the developers to join this discussion because we are most likely going to break this functionality for them.

The reason for separating was that media players can be used for displaying video/audio files when these files are not part of the text. Examples of current usage: mod_resource ("File"), mod_url ("Url") and mod_lesson ("Lesson" - see well hidden media file in the module settings). However it is not difficult to use something like format_text(html_writer::link($fileurl, 'file name')) to display the media file in these modules and i have written POC. But there still can be use cases where it is beneficial to use media players without calling format_text(). So currently we have the following options: 1. leave core_media API that is used by both filter_mediaplugin and mod_xyz (resource, url, lesson, etc) and add a new plugin type for media players 2. remove core_media API (convert to several filter plugins), force mod_xyz to create text fragments for their embedded content and pass it through format_text() 3. [a bit of both] create new plugin type for media players, refactor filter_mediaplugin to embed players using Media API (process <video>, <audio>, old <a> media) and force mod_xyz to create text fragments for their embedded content and pass it through format_text(). Advantage of using filters is that we already have API and UI to configure filters in each context and it allows better customization. Advantage of using new plugin type for media is that media players can be used without calling format_text(). All opinions are welcome. Please think first from student/teacher point of view, then from administrator and only in the end as a developer. Let's aim for user friendliness. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle I agree with changing the syntax to html5 audio video tags - mainly because these allow us to support captions / text alternatives which is not really possible with the current media filters. But in the interest of usability we should not deprecate or prevent the existing "link based" methods from working - many existing teachers will be trained to add videos this way and changing it will frustrate them. It would be great to build subtitle support directly into the Atto plugin which could store the text track as an additional file in the file area associated with the current text field (if it supports files). Average of ratings: - Re: Implementing pluggable video/audio players in Moodle Hi Damyon, there are no plans to deprecate link based method. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle core_media API was added by the OU because we wanted to use our own media player, and we are still using it. At the time linking in with theme renderer overriding was a simple way to achieve what we wanted to do. It was sufficient for our needs, and I guess the integrators were happy with it. I have to looked at the detail of the existing discussions, but off the top of my head I would imagine a solution like this: 1. New plugin type for media players seems sensible. 2. Keep the core media API as-is, or try to only extend it in backwards-compatible ways. (For example, add new options to some methods so information about things like transcripts can be passed.) 3. Change how the API does its output, so that instead of (or on top of) the existing renderer code, so that it uses a new plugin type to output the appropriate player. That is, the body of each renderer method (only kept for backwards compatibility) would become get_player_plugin_for_content(...)->render_player(...); I will try to point sam (who originally implemented core_media) at this thread. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle Hello Tim, thanks for the reply, i have already pinged Sam on MDL-38158 and he replied there. There are several problems I see with new media plugin type: 1. We loose per-context customisation that is available with filters. This is currently the biggest downside from my point of view. Re-implementing the same thing on the level of media plugins will triple the work (and triple the potential regressions) and make UI more complicated for teachers 2. We need to very carefully think about API and consider all possible parameters, such as alternative sources, subtitles/descriptions tracks, dimensions, autoplay, etc. It has to be at least all attributes that <video>, <audio>, <source> and <track> have. If we forget something or want to extend it later we will need to change API, wait for the next major version, etc. 3. html5 media players basically add their css class to the <video> tag and load their JS/CSS files. Creating a text filter for that is an easy job. Creating a media plugin type will require parsing and re-creating element, it's a new API that developers need to learn. Believe me, they are not going to bother. They will just quickly make a filter plugin, look at existing "filters" category, basically every second plugin there converts links with some particular filetype to embedded code. In the end we will still have both media player plugins and filter plugins. Those who want to use media players directly will suffer because not all players will be available for them. We already suffer in mod_resource and mod_url because all these nice filters don't apply to them. We can't run around and shout "guys, you did not do it right, change your filter plugin to media plugin" 4. This is similar to #2 above. At Moodle HQ we are going to allocate time now to develop atto plugin for inserting <video>/<audio> tags and xxx plugin for displaying it with an accessible player. If xxx=media - nobody will be able to use it before 3.2, if xxx=filter - everybody will be able to download and use it with 3.0 and 3.1 So here are my arguments for leaving it as filter plugin. I will be happy if you can point me to any of the existing plugins that use core_media directly. I really need to see the real examples of the code that can not be converted to use text filters and make sure that it is worth all sacrifices mentioned above. Average of ratings: Useful (2) Re: Implementing pluggable video/audio players in Moodle Please also don't forget that we can extend Filter API in any backward-compatible way. For example: • allow filters to display more information on "Manage filters" page • allow filters to return information about supported extensions (to be used in editor plugin) and so on Average of ratings: - Re: Implementing pluggable video/audio players in Moodle 1) I'm probably missing something, but I cannot see the requirement for different media players in different contexts, either from a teacher of an institutional point of view. What am I missing? Note that, media players are not only used for rendering media in user input (filter_mediaplugin). They are also used • mod_resource and mod_url, to display media files inline. • old TinyMCE moodlemedia plugin preview script. (Does the equivalent atto plugin not offer a preview?) • (In mod_lesson, but that does not look like a good thing to me. I don't see why that is treading media any differently from any other user content.) To me, it seems clear that there are two completely separate bits of functionality: A) Given a media file, and related options, display a player for that content. That is an output thing, the responsibility of the proposed media player plugin type. B) Parse user input, and determine that they want a media file to be played here. That is the responsibility fo the media filter. Of course from a user point of view, they need to work together seamelessly. But, from a code point of view, if we mix them up, we will get a mess. Average of ratings: Useful (1) Re: Implementing pluggable video/audio players in Moodle To answer Tim's point about "why the need for different players in different contexts" I will just give some examples from my experience. Example 1 context 1: You have 3 minute audio clip in a Moodle quiz. Students listen and answer based on what they hear. context 2: You have a page of new words for some students to learn. Beside each word you put an audio player so they can hear the pronunciation In this example both contexts require an audio player, but the length of the recordings and number of recorders on the page are different. In context 1 a progress bar and the ability to scroll to different parts of the audio mean that a standard audio player is appropriate. But in context 2 a single button audio player is more appropriate Example 2 context 1: You have a 5 minute how-to video on an page resource context 2: You have a 10 step tutorial in a lesson with screenshots and text explanations. In context 1 a regular video player is appropriate. In context 2 real estate is limited so you want video thumbnails as placeholders which when clicked open the video player in a lightbox Average of ratings: - Re: Implementing pluggable video/audio players in Moodle Those examples don't convince me that configuration by context makes any sense. Certainly, the media filter needs different options so that video and audio can be displayed in different ways, but using Moodle context to control that does not solve all use cases (what if two questions in the same quiz contain very different types of audio clip?) and it not very usable. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle Sorry I didn't communicate that "context" thing very well. I was really referring to situation contexts, not Moodle contexts. And you are right, Moodle contexts are not a complete solution to this. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle Basically I am advocating option 3 from Marina's post above. I think we need just one filter_media that will act as abstraction layer between media html content (<video>, <audio>, legacy <a>) and Media API, which in turn: • implements a new plugin type for media players; • makes possible to clearly describe players-specific capabilities in terms of features and supported formats; • provides unified process of embedding players from other areas of Moodle; • informing other areas of Moodle of players features and current configuration; I am re-posting my tracker response here, to keep discussion in one place. Below is the list of statements I made to support using new plugin type for media players, rather than using filters for that purpose: It is possible to use players through filter, but I think we may hit some limitations soon. It is not only about embedding. 1. What if I need player A for wav, player B for mp4 and player C for mp3 - does that need to be configured in the particular player filter and then filters needs to be shuffled in the right order? How to look up the current configuration apart of opening every "player" filter settings and checking each of them? With player plugin this task it may be easier - the plugin type management interface may have a column showing which file type each player supports and has enabled. 2. What if player need to inform editor about its capabilities (e.g. that certain format is supported or not supported) so that editor will behave taking into account what is possible and what it not in the current configuration (e.g. inform user that none of the players support .ogv when user wants to add this file). Some API would be helpful here and I do not want to mix it with filters. 3. Finally, what if player supports recording and someone wants to implement it (again, this is not a hypothetical example) - still need to be a filter plugin with hacks or rather a clear methods for that functionality in core\media\recorder and corresponding plugin? :/ In any case, whatever option we will choose (with filters for individual player or without), it is a good start! I understand that there is less work if we do atto video/audio support first and keep players in filters for now, and it is fine, it will already be a good improvement. Once that is done, switching to media plugin type will be easier than now if we get back to this route. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle I think we need just one filter_media that will act as abstraction layer between media html content (<video>, <audio>, legacy <a>) and Media API</a></audio></video> Completely agree. I think this is the most sustainable solution. Average of ratings: - Re: Implementing pluggable video/audio players in Moodle I haven't spent enough time in the code lately to be as on-point as I should, apologies, but I would like to relate the issues I had when we needed a simple extension to the standard Moodle Flowplayer in order to play RTMP streams (campus server running Adobe FMS, now Wowza). Having everything seemingly controlled by javascript-static.js (whether or not to load Flowplayer based on array element counts, loading one and only one player, etc.) meant I had to create a different filter, with its own YUI module. So, yeah, providing for a render-time selectable player (to override a default) would be welcome. Supposedly the selection of the player would be made by the the context laid out in the data- tags in the markup. Choosing between how the markup is generated and placed into the page, mod versus filter? Can't say right now. But from the user's perspective, when they select a specialized media plugin from a list of (activity) resources, the context is laid out and it's a simpler to generate and render the needed markup: e.g. choose an .mp4 movie (repo hosted), we know what and where from the outset. With a filter, there's the pattern matching, then determining, what and where, and so on. Filters are problematic if the user doesn't enter the URL or other sought-out pattern in the correct way--so for filter_rtmp, we chose to create a repository plugin to take care of that for them. So: media player plugin, yes; specialized media resource plugins, open to debate, but preferred over filters; on the browser, provision for using a JavaScript module supplied by the specialized media resource mod (via$PAGE->requires->js_init_call) that would select/extend the media player.

Marina, while our filter is obviously not a theme, we did have to override the core_media_renderer, so thanks for that heads-up.

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

Having made several filters that produce media players from links, there are two observations that I would make. The first is that virtually every media player that I am aware of does not require a jot of backend code (ie PHP). The second is that media players are an area that users want flexibility.

backend / frontend

In the VideoEasy and PoodLL (ver3) filters most players occupy between 2 and 30 lines of JS, and load a CSS and a JS file. What happens in php is the parsing of html elements on the page, version handling and the settings page. Actually the parsing function is so generic that it should be centralised into a core api.

Last year or earlier I proposed a different solution and went as far implementing a prototype which I just kept on at. That is VideoEasy and it uses a template of CSS/JS/HTML per player. The template is editable in the admin settings interface of the plugin.

It works great.  There are templates for VideoJS, Flowplayer, MediaElement.js, JW Player as well as some original players that users write themselves to solve their own needs. I have yet to see a player that a template couldn't be made for.

So sure we could make a new mediaplayer plugin type, but it would be almost exclusively front end code. VideoEasy templates are around 2-5kb each.

Flexibility

The ability for admins/teachers to edit, copy and share templates is a factor I think that has not really been considered in this discussion. One user (Adam Murray) wrote and submitted a 3 speed audio player (slow normal fast) written entirely in the admin settings interface. And how about subtitles? Some sites like their subtitles on the screen, just like the movies. Other sites use interactive subtitles below or beside the player that highlight and can be used to navigate through the video.

I think we should get away from thinking about decisions like "flowplayer or jwplayer" and more along the lines of allowing users to create "players" that match their needs. That might be the outright creation of new players. Or it might be 3 different flowplayer templates with sets of settings appropriate to the way they will be used.

The major weakness of the template system in my experience is that there is no nice settings interface per template. The admin has to get into what looks like code. But if we could solve that, we could give the Moodle developer and user community a much better ride. (And ...  I might add .. VideoEasy is already most of the way there, so that saves someone a lot of work ...)

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

I've thought about this a bit more now and I see a few issues with the 'ditch core_media and just use filters' approach, but they are not necessarily insurmountable. Anyhow here is a rather tl;dr brain dump about it

1. If we use filters then that means every filter has to reimplement the necessary regular expression matching. (OK this might not be a big deal for players that just annotate the video/audio tag but for anything that works similar to current media plugin, this is complex.)

This is important because what you're really saying is that we will have a defined input format (as we kind of sort of halfway do at the moment) and then a whole stack of independent filters will deal with it. So we should have a defined shared way to match it.

Proposal: Go ahead and use filters but put code for detecting and replacing all the media in a text (along with all options, subtitles, etc.) into a separate class in core. So filter_media or its replacement becomes basically a thin wrapper over this class with a callback to slot in the actual media player (or just stick a class in and load some javascript). And you can easily copy this for your custom filter.

2. Including media from a plugin (not in user-entered text) by doing html_writer::tag('video') is hacky and means all the plugins have to do the same hard work to figure out how to correctly include subtitles etc.

Proposal: Create a standard API for writing the HTML code for video and audio, which supports all the alternative formats, dimensions, subtitles, etc. as standard PHP parameters or class members or whatever it is, so that modules (resource/url/lesson/third-party) have a nice standard way to output media. I'd suggest this could maybe be in the same new class. Anyway, this can then even (optionally) wrap the format_text part so that you can just do echo \whatever\class::video(\$params) similar to the current API, and it will internally use filters, but who cares.

3. The filter mechanism doesn't provide enough information to configure media players. However, this is actually true of the current player mechanism too...

Proposal: Define a standardised way to include parameters for the media plugin, perhaps (just for example) as a specific type of HTML comment within the video/audio tag:

<!--filterparams:name=value,name2=value2-->

Alternatively it could be a data attribute which might be nicer. Anyway, this would then be interpreted by the input parsing API described above so that filters can easily access it. We could have a convention like if a parameter is some kind of generic one (for example to make a 'compact' version of a player) then it will be defined by Moodle, whereas specific parameters should begin with the plugin name.

4. The filter mechanism is a bit rubbish at supporting multiple players. For example, let's suppose you want to use two separate media player plugins - a regular one and a compact one; or you want to use one player for FLV and another player for everything else. If you just enable both filters it will probably screw up and try to filter things twice.

Proposal: Make a standard parameter (as per 3 above) called useplugin where you can specify a particular plugin name, and other media player filters will use this parameter to intentionally not filter that particular video. Also define make some standard for marking the output when a video HAS been processed so we can be sure it won't be re-processed, maybe as another data attribute such as data-filterplugin='myplayer' (obviously this is only needed in cases where the processing results in a <video> tag again).

So in summary I think Marina's preferred filter option is workable but I think we should clearly define the required input format (and where applicable some parts of the output format) for all filters that support media.

Regarding a defined input format, preferably using <video> and <audio> tags and  supporting the current link mechanism only for backward compatibility, I have a few comments based on our experience of this:

• With my accessibility hat on, this should include support for both subtitles and, ideally, audio description MP3 files. And probably transcripts (in various arbitrary formats) too.
• This should also support a way to specify a 'poster' image (the image, usually a jpg, that appears before it loads the video rather than a black box).
• Whatever element is used for giving subtitle files, it would be nice if there is some mechanism for supporting alternative formats (i.e. if Moodle only supports one format, like whatever the W3C one is) - here at the OU, we have a lot of material using a 'legacy' subtitle format so our media player supports this. It would be good if we could use this in a niceish way in the new system. (One option would be to use the 'parameters' thing I described above with a custom parameter for our filter.)
• When specifying dimensions this should be clearly defined as the video dimension and not the player dimension (often these are the same but not always) so that it works OK when changing players. In other words if the player (filter) knows it has a 30 pixel control bar then it's up to the player to allow extra room for that.

I'd suggest that the input format should be a clearly defined subset of the video/audio tags (subset because it makes it easier for filters to handle) plus some extra stuff, as comments or data attributes or whatever, as needed to control the filters.

--sam

Average of ratings: Useful (5)
Re: Implementing pluggable video/audio players in Moodle

Thank you very much Sam for a long and descriptive post, you outlined very well how existing filter API could work with media embedding.

One more argument for improved filter API: Not all url-to-embed converters are about media (video or audio). For example embedding websites previews, display equations, chemical formulas, flowcharts and many others. There are sites that provide you with embed code to use on your website and there are filters that convert URLs to such websites into their embed code (remember that students can not use <embed> or <iframe> tags by default). It acts exactly like youtube converter in core but it is neither video nor audio.

I think we need just one filter_media that will act as abstraction layer between media html content (<video>, <audio>, legacy <a>) and Media API

If we do so how do we decide what is media and what is not, what should be implemented as media plugin and what should remain as filter? Or should we just rename "Multimedia plugin" into "Multimedia / embed plugin" ?

By the way, Sam, you said: "Marina's preferred filter option". This is not exactly true I personally initially supported new plugin type and later heard other opinion from Justin Hunt and started doubting. Then I searched plugins directory and found many existing filters that convert link to different file types into specific embedded code.  So I thought that enhancing filter API will be easier for developers and their existing plugins. I also think it will be easier for admins to configure everything in one place. I still do not see anything that the new plugin type can support that is not possible in filter API.

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

This is how videoeasy and poodll handle the different file extensions.

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

In case we go with new media plugin type, help me understand the distinction:

• youtube - clearly media plugin
• multilang - clearly text filter plugin
• filter_oembed - what is it? The functionality is very similar to youtube (convert a link to embedded code) but you can hardly call it "media"

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle
• youtube - clearly media plugin
• multilang - clearly text filter plugin
• filter_oembed - what is it? The functionality is very similar to youtube (convert a link to embedded code) but you can hardly call it "media"

Not at all!

Youtube and Oembed may play media, but those are externally hosted media rather than internally hosted content. You can't just put any old video player round a youtube video or vimeo content, they are about embedding the externally hosted content which means we have to use their players and don't have a choice in the matter.

Average of ratings: Useful (1)
Re: Implementing pluggable video/audio players in Moodle
Dan, I asked "what plugin type is it?" And you answered "not at all". Can you please explain?

Currently core_media_player_youtube is part of media api and so far there were no suggestions to remove it or vimeo from there so I assumed they were going to be converted to the plugins of the new media plugin type.

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

As someone working on the Oembed filter (MDL-55332), my immediate answer is the same as Dan's. Oembed provides an externally hosted media resource with its player, so we have to honour that player.

But, there is some discussion above about coming up with a standard to represent media so that players can be determined. Perhaps the Oembed format could be something we adopt in that context? It already provides a mechanism to describe embedded media.

mike

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle
Regarding how to prevent multiple filters applying on top of each other:
after processing the link/video tag filter A must add class="nofilter" and all consecutive filters will skip processing it. There is already class="nomediaplugin" that is respected by filter_mediaplugin

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

Take a look at the implementation of the media player in the WET-BOEW framework, developed by the Government of Canada i collaboration with the WET community, which places an emphasis on accessibility and multiple language support.

You can start learning about and viewing the documentation and source code by visiting the WET- BOEW Multimedia Player Working Examples page for the media player at:
https://wet-boew.github.io/v4.0-ci/demos/multimedia/multimedia-en.html

The benefits of this approach includes:

• WCAG 2.0 Level AA accessibility compliance
• Support for multiple languages
• One plugin to support both video and audio
• Support for 2 video formats: MPEG4 (H264+AAC), WebM (VP8).
• Support for 2 audio file formats: MP3, Ogg Vorbis.
• Supported closed captions sources: Timed Text Markup Language (TTML), HTML5 Data file and HTML5 Inline Data.
• Support for YouTube: Although you can embed a YouTube video in a page, using this player enables your YouTube videos to match the common look and feel of your site for a consistent UX.

"the multimedia player leverages the native HTML5 video tag and relies on Flash as a fallback mechanism when the HTML5 video tag is not implemented. The MPEG 4 format (H264 codec) is the minimum requirement for video because the Flash fallback relies on it and H264 is the standard video format on many mobile devices. An optional but highly recommended Webm (VP8 codec) should be added as well to allow some browsers such as Firefox that do not include native support for H264 to leverage the native HTML5 performance gains. The multimedia player's timeline relies on the HTML5 progress element and uses a polyfill when the element is not supported."

Open source code is available on GitHub under the MIT license.

I believe it should be possible to do something similar as a Moodle Media plugin.

Hope this helps.

Best regards,

Michael

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

Hi Michael,

Thanks for the link. I did not want to talk about the player choice in this forum post but rather discuss media API.

But speaking about player accessibility, I was recommended able player which works good from accessibility point of view but does not look that great to be honest. Player you linked to has the same problem - it adds a huge controls block under the video which is good for accessibility but unnecessary clutters the interface for majority of users. Videojs looks so much nicer and actually has lots of accessibility points addressed too.

This only proves that point that there is no universal answer which player is the best and we need this media API.

However at the same time I realised that we missed very important point in all previous discussions - should media players be user preference?

We are really close to the code freeze now and I'm afraid I won't have time to add user preferences to the media players before 3.2 release if we want to have media api then.

Also, http://prototype.moodle.net/media/ was updated with the new players (still work in progress).

Average of ratings: -
Re: Implementing pluggable video/audio players in Moodle

Hi Guys