Yesterday I gave my keynote presentation at MoodleMoot Japan 24 titled “Automated feedback for language teaching” and I introduced my AIText question type.
This is a fork of the Moodle core essay question type but with first pass grading and feedback done by a Large Language Model, ChatGPT in this case, as soon as the student presses the submit button on each question.
Sure, all of this is great, but only if the students don't have access to an AI language model themselves.
Google Gemini:
I played a little bit with German text.
The feedback identified wrong words. But it identified also correct and suggested they are incorrect
@Marcus: its a great tool and I will present it at Learntec fair next week. Thanks for your work.
Hi Marcus
How does one connect Ollama with your AI Connector plugin? I can would like to test your question type using Ollama to compare responses.
It can take some trial and error before it works.
I get good results from the models that occupy around 4GB of disk space. Anything much more than that and you need a monster machine with a lot GPU VRam.
Thank you Marcus.
https://poodll.com/essay-auto-grade/?utm_source=rss&utm_medium=rss&utm_campaign=essay-auto-grade
Hello Marcus,
unaccountable corporations
Well done for dismissing with a wave of the hand the missions and values of companies, their sources of funding, their transparency, the work of their board of directors, and the various regulations.
the best model
Well done again for choosing the best model. Large language models are trained on different datasets, they use different underlying architectures, different techniques, and they have different focuses and functionalities. They are all different. Therefore, your best model will certainly be the best model for all Moodle users.
You could ask an AI system, such as ChatGPT, simple math questions to get an idea. The following are unedited screen captures. ChatGPT's answers are perfectly correct.
Etc.
Hello Marcus,
I simply wanted to mention that ChatGPT responds very well to relatively simple mathematics questions like the ones I gave as examples. It's quite fascinating.
Here's a simpler question. In words, the question is "what is the square root of 2 + 3?" and a correct answer is: "the square root of 5".
Hello Marcus,
In materials science, common to several engineering disciplines, the 'von Mises yield criterion' is quite a fundamental concept in the plasticity theory (mostly used for metals). Many engineers have come across this concept at least during their studies.
I have tested both ChatGPT and Google Gemini on their knowledge about it. Both AI systems gave me good explanations but the wrong equation, even if it isn't a very complicated equation. I had to tell them twice that they had the wrong equation before they finally gave me the correct one.
This situation may be quite dangerous, as both AI systems seem like they know what they are saying, but in fact they don't. They give an elaborate answer that looks correct but with an equation that has the wrong factors in it. This can be quite misleading.
I would advise anyone who intend to use these AI systems to first thoroughly check any answer that they may give (if that's possible!). As far as I am concerned, today these systems are of no use for engineering because they are still too unreliable.
I have documented my conversations with ChatGPT and Google Gemini. I find them instructive:
https://dynamiccourseware.org/course/view.php?id=163§ion=1
https://dynamiccourseware.org/course/view.php?id=163§ion=2
Hello Marcus,
First of all, thank you for this amazing fork. It's very close to what I was looking for. I have a question: Do you think it’s possible to display the parsed JSON strings received from the LLM within HTML tags, such as <p>? I believe it would only require a few lines of code, but my coding skills are insufficient.
The reason I'm asking is that I want to use the Moodle essay question type to provide my students with immediate feedback from an LLM. Specifically, I want the LLM to bold the parts it changes in student essays. From what I understand, this fork currently parses JSON from the LLM and displays it in the feedback column in raw format. I want to use HTML tags because:
Thank you again for your excellent work and in advance for your time and help.
Best regards,
Ahmet
After some research, I realized that what is missing is something called "Markdown to HTML conversion." From what I understand, the JSON file is parsed and appended to the feedback box raw, which is why the words are not bolded, and paragraphs do not start on a new line.
I added the following to the marks part of the prompt "Do html formatting of the content. Emphasise the marks part." And the result came back as follows.
https://github.com/marcusgreen/moodle-qtype_aitext
Post any questions here and it might help improve the documentation.
At first glance I cannot see why it would throw that error.
The code is here (in case someone else can have a look to confirm)
https://github.com/marcusgreen/moodle-tool_aiconnect/blob/main/version.php
You have given your version as 20230714 which seems to be missing some digits so to compare the two
20230714
2022112800
But I am not convinced that 20230714 is the whole version number in the version.php file of your Moodle.
Any other opinions till I get home?
Yesterday, I installed the plugins on a local "sandbox" setup according to the instructions and the auto-correction of text answers works like a charm - amazing. Thanks for providing this! All I needed was some prepaid credit for tokens at openAI. Not prohibitive at all, I think this will boil down to 1 or 2 cents for each auto-corrected answer aka less than 15 Euors per year for my needs.
So here is my question:
Currently the AI-connector plugin accepts onyl one API-link that is used for the entire Moodle instance. This would not work for our production setup because there is no central budget for these new activities and the costs cannot be traced back to individual courses. Is it conceviable to generate a connector tool that accepts a different API-link for each course? This would allow me to solve the budget issue as a "small world" problem (e.g. just paying a few Euros per year myself and be done with it).
Thanks a lot,
Klaus
https://github.com/marcusgreen/moodle-qtype_aitext/wiki/FAQ
Here is what I wrote (at the end)
"I intend to submit it to the Moodle.org plugins database once I consider it sufficiently mature. The need for an external LLM adds a layer of complexity on top of a standard Moodle plugin. I want it to work with a variety of LLM systems and that means getting feedback from “real world use”. It is also necessary to manage expectations as LLM/AI systems are useful but are not “magic”."
I have recently integrated an excellent contribution of significant code from some very experienced people.
https://fosstodon.org/@marcusgreen/113737838087924231
Thank you Marcus, I'll get on to my computer man and see if he can add the code for me. I'll let you know how it goes.
Thanks & all the best,
James
Now the good/better news I have received a chunk of additional code that is due to be integrated. I will be making a test site publicly available for testing in the near future with some sample questions.
I recently reviewed a chapter of an upcoming academic textbook written by some people I have collaborated with and the overall feedback is positive.
One of the peculiarities of AI systems (or possibly my prompts), is that it can give good analysis and comments on an incorrect answer with appropriate marks, but if you type in absolute nonsense, e.g. just reply with
wigglepop
or
1lkj0j [j333
It will respond that it is a correct English sentence and give full marks. This can be remediated by appropriate prompts but it is another reminder that AI is not magic in anyway and good prompts are essential.
No need to do any additional work on this - maybe I did not phrase this quite correctly. The tool works fine and will perform according to the language selection of the course participant. This is totally OK once you know it, as it simply means that we don't need to worry about language when we set up the quiz. That is not bad at all...
The only odd thing was that if the response is first "concevied" in English by ChatGPT as requested by my quiz-question prompt, then to be translated to German this must (?) invoke a secod step where ChatGPT is prompted to translate its own response into German. Yet, this is not visible when configuring the plugin or the quiz question. I was surprised by this behavior and simply wanted to know at what stage the language setting of the user is fed into the prompts, and whether that is indeed intended to overrule any language cues placed into the quiz-prompt by the course teacher.
Once again, this is a detail and nothing to really worry about!
That entirely explains the observed behaviour, thank you. Fory my purposes, it is fully sufficient to be aware of it.
Best regards,
Klaus
Hi Marcus,
I think an important aspect of prompting GPT LLMs is context, i.e. the more specific we can be & the more background information we can provide, the more appropriate & useful the output will be.
Perhaps as part of the essay question instance configuration, the instructional designer should specify the LLM prompt, which should be specific to the essay task, i.e. the Field (topic & "specialisedness" of vocabulary & language forms), Tenor (degrees of formality, personal distance, certainty/probability, etc..), & other genre features such as whether it's an expository (simply telling/giving information) or discursive (arguing pros vs. cons, for or against, etc.) text. For example, if the task is to "write an email to a friend asking for a favour" but the submitted text is formal & distant, the LLM should be prompted to focus on the linguistic features that are (in)appropriate for that genre of writing. Without that kind of specificity, the resulting feedback is so generic that it's pretty much useless to learners.
How does that sound? What would be a good way to achieve that?
I have made very good experience with the inclusion of that kind of information in the question-specific prompt. There is a possibility to include some general aspects in the plugin-settings (webiste admin -> plugins), so that is a "site-wide" option. However, this pre-configured text appears in the corresponding sections when you insert a new AI-question in the Quiz ("AI prompt" and "Mark Scheme"). Thus, the course teacher can "override" these site-wide pre-sets.
I have included the expected response in the question prompt - GPT is a really good chemist, but did not follow my lecture and hence does not know exactly what I consider to be the most important aspect. For example:
Test question:
Please briefly describe which properties make glucose a very suitable molecule as a central energy carrier for the cell.
AI prompt:
Verify if the text correctly answers the question and explain the mistakes or omitted things if necessary. You are polite and supportive.
The question is: Please briefly describe which properties make glucose a very suitable molecule as a central energy carrier for the cell.
It should be addressed that all OH groups are in an axial position, making the ring form of glucose the most stable. In all other hexoses, there is a higher proportion of the open-chain form, which is very reactive and can lead to spontaneous glycation of proteins.
Mark scheme:
Deduct half a point from the total score for each minor scientific mistake or omission and a full point for each major mistake or omission.
Without the "It should be addressed..." statment in the AI prompt, ChatGPT will comment on other properties of glucose (rather, how one can deduce that glucose must be a metabolite of central importance). But with the desired aspects in the prompt, correction + marking works really well. I even tried things that GPT cannot know about (e.g. what I did yesterday), but if this info is provided via the prompt, then the commenting / marking will be very appropriate.
I am really excited about this possibility, mostly because many students will use the quiz at a very high load during the last 48h before the exam. This makes human commenting / grading essentially impossible, but with the AI engine we can offer it.
