How are we supposed to use the AI Subsystem ?

How are we supposed to use the AI Subsystem ?

autor Justin Hunt -
Počet odpovědí: 5
Obrázek: Core developers Obrázek: Particularly helpful Moodlers Obrázek: Plugin developers

I have some questions about the AI subsystem. I use AI in a lot of places in my plugins. I route almost all requests to my own servers and completely bypass the AI subsystem. Partly because this process pre-dates the AI subsystem. And partly because I don't think the AI subsystem does what I need (thats what my questions will be about). But I do want to use it, and ideally I could begin the process of allowing users to route AI requests via the subsystem if they preferred to. 

As a representative example of a use case - in Poodll Wordcards we need pictures for the WordCards. So I wanted to use the subsystem, and I made a placement plugin so we could generate images. That worked. But then I went back to the old way.  

Question 1) Do we really need a placement plugin for each place we want to generate a image?

(clarification: This is not about using the HTML editor.) I mean, do we need one for Poodll WordCards and one for Poodll MiniLesson and one the Moodle repository etc.

Is that really the case? If not, cool. Please tell me what to do instead. 

If however it is the case. I don't mind coding a few lightweight plugins. I kind of mind having to maintain them on the plugins directory and through each Moodle version update. I do mind having to get my users to install new plugins for what are basically features.  

Perhaps my needs are some sort of edge case and the html editor is fine for most people. But how about:

  • The file manager - of course we need image generation there.
  • Automated generation content - Poodll has an AI lesson builder it takes a few prompts and goes off builds incredible lessons with loads of images and audio. Most mods will have something like that soon. 
  • Bulk image generation - making 10 images in series is slow. Making 10 images in parallel is fast. 

And that is just images. In fact text generation is a bigger issue. Basically most of my plugins will need a companion AI placement plugin. It seems too much. A hook or callback feature seems more appropriate. 

Question 2) How about all the other AI things we need to do?

A few examples:

  • give students feedback on their submission.
  • give students a score on their submission
  • modify previously generated images
  • transcribe speech.


Most of this is programmatic use of AI. Trying to know in advance what kind of AI action is required and make it available as a placement, seems like forever playing catch up. Thats partly because we don't know what we don't know. And partly because AI is advancing at such a rate new functionality is arriving regularly. The list could just get really really long.

How do we get new actions into the system? I need all of those things now. 

Is it possible to have a catch all that allows us to effectively use the engines creds and supply our own implementation of the request and response handling? I mean, I am sure we can do that now, but I do want to play by the rules. Is that against the rules?

Průměr hodnocení: Useful (5)
V odpovědi na Justin Hunt

How are we supposed to use the AI Subsystem ?

autor Michael Hughes -
Obrázek: Core developers Obrázek: Plugin developers
This is a *really* interesting question Justin, and one that think many developers are struggling with. I feel that that "placements" function is a complex / complicated route to govern where the end-user interaction or initiation of an AI action takes place. This is probably fine in the case of *some* interactions.

But I think it overall takes away from the Developer's to make good decisions (or and potentially *bad* decisions) about where and what that initiation looks like. So personally "I" don't like this approach...

The other aspect is really the back-end consumption of AI "services" by a Moodle plugin. This in my mind should not need any sort of "placement", it really needs to be a mechanism for the developer to get back some sort of interface that will route their request to a "relevant" / appropriate LLM provider instance. This is of itself a complicated question if we're to allow individual system administrators to have the agency to *decide* which LLM provider they want to use, and this could conflict with some developers who don't want to afford sys-admins that flexiblity (say for instance your AI experience relies on a *very* specific model that *you're* hosting).

On this front I prefer the approach the By-cs's local_ai_manager plugin has taken, in offering "purposes" and "providers" so that the systems admin can assert "this LLM is for this purpose", and *every* complying AI plugin that wants to "generate an image" will use it. It doesn't preclude a developer writing an "I need a very specific image generation purpose" and then requiring the admin to wire that purpose up to a very specific LLM instance...

(Full disclosure, we've been playing in that very area to look at doing "RAG" and "Text Embedding" purpose for this plugin, so we sort of know that adding newer more specific purposes can be done here).

But in both the Moodle system and the By-CS one there is definitely an "arms-length-ness" to calling an LLM, it's by no means "direct" (in Moodle you have go via the placement, and in the local_ai_manager you need to go via a purpose)
Průměr hodnocení: Useful (3)
V odpovědi na Michael Hughes

How are we supposed to use the AI Subsystem ?

autor Justin Hunt -
Obrázek: Core developers Obrázek: Particularly helpful Moodlers Obrázek: Plugin developers
Hi Michael. Since writing that post, I realised that it was not getting any replies (till yours, thanks) because there were not really any good answers to the questions.

The major issue I have is that schools care about where their data is going and want to have options. But I am unable to give them those options. The AI subsystem is not working for us plugin developers. I have heard about the local_ai_manager from qtype_aitext and discussions with Marcus Green. Until there is a better official Moodle solution, that might be the solution.

Do you know how new purposes get into the local_ai_manager system?
e.g "edit_image_from_text_prompt" or "generate_audio_file"
 
Actually ... I found the docs .. that explain that and other things ... thanks.
https://github.com/bycs-lp/moodle-local_ai_manager/blob/main/docs/plugindev/index_plugindev.md
Průměr hodnocení: Useful (2)
V odpovědi na Justin Hunt

How are we supposed to use the AI Subsystem ?

autor Philipp Memmel -
Obrázek: Core developers Obrázek: Particularly helpful Moodlers Obrázek: Peer reviewers Obrázek: Plugin developers

Hi Justin,

the documentation of local_ai_manager is probably only rudimentary, so if you have questions or requirements, reach out to me/us.

Also, while we tried to make it behave to cover the typical use cases, it primarily addresses our requirements for Bavarian schools. So there is a chance that other "typical" use cases are not implemented as the default behavior or there is documentation missing on how to achieve certain things. If you encounter something in this regard, just contact us.

Philipp 

Průměr hodnocení: Useful (4)
V odpovědi na Philipp Memmel

How are we supposed to use the AI Subsystem ?

autor Marcus Green -
Obrázek: Core developers Obrázek: Particularly helpful Moodlers Obrázek: Plugin developers Obrázek: Testers
"If you encounter something in this regard, just contact us."
I have found Philipp and the team he works with to be highly responsive in the two years or so I have been collaborating with them.
Průměr hodnocení: Useful (2)
V odpovědi na Justin Hunt

How are we supposed to use the AI Subsystem ?

autor Justin Hunt -
Obrázek: Core developers Obrázek: Particularly helpful Moodlers Obrázek: Plugin developers

Well after talking this over with a few very clever people at Moodle and in the dev community .. this is what I think 

AI Placements

I think these are probably misguided. I cant see much use for them because they are really re-usable UI components that are locked to a single location.

Actions and Providers

There are so many services already that are based on AI, but they are not really raw AI. AI is getting merged into every web service. Try to integrate all of that and we will end up with as many AI actions as web services. We need to be clear about what can be supported as actions, and how that relates to AI providers. (Maybe I am the one that needed "clearing up" ..)

For example I was researching 2 things that I know users want in Poodll. 

1. jogg.ai - AI video generation - specifically Avatar videos
2. eleven labs v3 - Text to speech voices  -  specifically we can script 2 person conversations with emotions.

They would be "generate_video" and "generate_audio" actions, but they don't do anything else. And it's likely that they would require non-generic inputs. In which case supporting sending the same generate_video request to either Sora or jogg.ai would not really be practical. At the point that you could do it, it would be too basic/limited to satisfy what users expect or need anyway.

 
Actions - Generic / Non - generic 

So I think there should be two classes of actions: Generic and Non-Generic 

- Generic
We have to decide which actions are something we can provide a generic API interface for and which are (or will be) supported by at least 2 providers. Generic actions are something Moodle decides and creates an API interface for. I think it would be ok, and probably good to create a range of Generic actions soon even if they are only supported by one provider as a reference example. 

- Non-Generic 
IMO we need to allow AI providers to implement non-generic-actions that can be called by Moodle/plugins, eg elevenlabs::create_audio_dialog(). This is also where we would send the squeaky wheels like Poodll who complain that there is no generic-action for what they need. And it provides a path to elevating some actions to generic if other providers implement them.

Actions - Purposes

Some actions like generate_image (maybe) are fairly good just like that. But others, specifically "generate_text" need to be married to the purpose in which they are used. Where generate_text is used to get some course content like "what caused the French Revolution" it is one thing. But where it is "Assess this students essay and tell me what their score should be" it's very different. Because the type of data we are passing around will affect which service we want to use. So either we add usage attributes to actions, that allow us to decide which AI provider we use. Or we have different actions altogether, such as  "evaluate_submission."


What can be an AI Provider?

I don't know if we really want the full range of web services represented as single-action AI providers. But perhaps it is ok, if someone wants to write a plugin for it. Deep down I think an AI provider must provide at least one Generic action.

- Jogg.ai: IMO - no
- Grammarly: IMO - maybe. Grammar correction is fairly generic. Not sure if it's multi-lingual enough.
- elevenlabs: IMO - yes. It's the go-to place for text-to-speech voices. And it's a good example of where we might want a non-generic action.

Průměr hodnocení: Useful (3)