Moodle AI Provider Benchmarking

Moodle AI Provider Benchmarking

by Andrew Mair -
Number of replies: 1

Has any benchmark testing been done by Moodle or anyone else to determine:

a) the best model to use and the trade offs between price and performance (on both OpenAI and Azure AI you get a list of different models that you can choose from that have different pricing) and what additional safety services are required with the AI service for AI placements to be made available to students?

b) the impact of enabling Moodle AI placements on Moodle infrastructure - server & DB performance, UX latency, additional logging and monitoring required

c) the risk exposure against AI specific security exploits e.g. against MITRE ATLAS

d) what's the best way to purchase these AI services to provide an acceptable baseload performance and the ability to scale them up for peak load?

e) a method for projecting the additional budget required to add AI to Moodle e.g. for every 1,000 students add X% to your Moodle hosting and X% to cover the AI costs

I haven't found anything published yet that addresses any one of these issues which are all important considerations to scaling out Moodle AI features in production.

Keen to hear people thoughts and lessons/war stories smile

Average of ratings: Useful (1)
In reply to Andrew Mair

Re: Moodle AI Provider Benchmarking

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
There is no single bench mark to address these questions.
a) Best Model:It depends. For example if you want to teach languages it might be best to use a model optimised for that. If you want to do Maths you might decide not to use an LLM (inherently unreliable)

b) Performance: Probably minimal, the interesting stuff happens outside Moodle

c) Risk: There is little that Moodle can do to mitigate risky content but generally it should not compromise security as "traditionally" concerning with a system like Moodle. So for example AI use is probably not going to facilitate getting admin access. I have been working on using regular expressions to identify attempted prompt injection attacks but that is more about general warnings than any actual fixes.

d) LLM Performance: That is a good question. One approach is to go for big suppliers and have a large amount of contingeny money. I don't mean to be flippant but if AI is really important to you that is a solution. However as someone who is very engaged with Moodle and AI I advise against making AI reliability (in any interpretation) central to your plans at the moment.

e) Costs: I have been vigoriously experimenting with AI with Moodle for over 12 months and my total spend has been less than £US50 (all with OpenAI). During that time Open AI has halved its rate per million tokens (someone confirm that for me). I believe the cost of inference will "fall off a cliff", due to both hardware and software developments. Inference will always have a financial cost, but possibly in the same way that broadband and internet access will always have a cost.

Caveat on my comments, I have no direct experience with managing AI and real world education. However I know people who do and I will use your question as a good reason to speak with them.
Average of ratings: -