Evauative AI (MoodleMoot Estonia 2026)

So my name is Marcus Green and my presentation is called evaluative AI which is AI feedback on low stakes assessment. Um, so to introduce myself, I've been a quiz developer since 1999, which is before the dawn of Moodle time. and I was a teacher uh in further education which is for students between the ages of 16 and 19 from 2003 to 204 which uh covers the time when uh Moodle and virtual learning environments were first starting to be used and I discovered it myself in 2003 and I saw the future and um I had my first plugin uh added to the Moodle plugins database in 2012. And you can see here that's a screenshot of that plugin and a screenshot to demonstrate that I was quite young once. So I must give a warning today here now and that is the technologies I'm going to show you will not under any circumstances revolutionize education. I hear that a lot. Product X is going to revolutionize it. It won't. But there is a transformatory aspect of these plugins because they will allow ordinary teachers, non-experts to apply automated analysis of free text which until a few years ago is effectively impossible and also the things I'm going to show you certainly the first plug-in is widely used now. Now to go back to the title of this I would say um low stakes assessment because I believe that high stakes assessment uh a quiz or an assignment where there is consequences such as whether a student's going to get on another course should always be processed by a human a teacher. That's my view. Now luckily the EU quite wisely agrees with me. Um and as you can see here from the EU act, uh high risk systems are those that determine access or admission to uh education and training to evaluate learning outcomes which are used to steer the learning process, assessments which assess the appropriate level of education that an individual receive and also the monitoring and detecting of prohibited behavior. Okay, so there's a legal precedent for all of this. So to step back from all this technology and talk about teachers and why do I became a teacher I was quite late in life. I was 43 years old. Um what are the main reasons that people want to become teachers? Now it may not be the money. It may not be the money. I thought about this subsequently. This this rule may not be true in Finland from what I hear. Let's assume that for most teachers, they don't go into the profession because they expect to become enormously wealthy. And the actual reasons that teachers tend to give, as you can see from here, are things such as a passion to help other people uh learn and grow, a desire to make an impact, and also a passion for your subject. And you'll all recognize these sort of list of reasons that people give. And according to my extensive research, no teacher has ever in all history said, "I became a teacher because I want to do marking. My passion is marking." And then once they become a teacher, they never ever say, "Now I'm a teacher. I am qualified. I really enjoy writing feedback and it's the best part of my job." And I while I'm at it, I go around my colleagues and say, "Please can I mark your students submissions?" It doesn't happen, does it? It as a teach when I was a teacher, I put a huge amount of effort, probably too much into my marking and I would give up evenings and parts of Sundays to give excellent feedback which I strongly suspected frequently the students were what's my grade? I didn't mind doing it because it gave me an idea of what they were and were not learning. So many years after I stopped being a teacher, this led me to create the AI text question type, which is really the first half of my presentation here. And this will give large language model feedback on free text responses. And you may hear me use the term LLM or large language model because in my view, these tools don't shouldn't be called artificial intelligence. They are large language models, but I will use the terms interchangeably. And this question type allows customizable prompts for flexible feedback. In other words, the remote system, the LLM is can be given instructions to accommodate the specific learning that you are hoping the student experiences uh when they respond to this question. And remember, I was saying this is about learning. This is not for me. It's not about assessment. It's not a catch you out thing. It is to try to make sure that students get good feedback so they learn the concepts. And the nice thing about this is that this is available right now from the Moodle plugins database and it is quite widely used. You can see here that at the time of the screenshot it was used uh on at least 361 Moodle sites globally. But as always with uh Moodle HQ statistics, this is almost certainly a very low estimate because most Moodle sites don't register. Um I announced this while other people were eating their turkey or local celebrations on Christmas Day 203. I announced it on Masttodon, my social media site of choice. Uh, and this was working. So, I've been working on it for several months during the end of 2023. And I want to make it clear to wind back a little bit because when we you hear people talk about uh AI plugins, there is a whole world of plugins whereby you can download the code and the code is yours but to make it work you need to buy a token to access that particular service which is a perfectly good way of doing it. But that is not what I'm talking about here. What I'm talking about is you download the code. You don't have to have anything more to do with me. I recommend that you do. Um and you select your inference provider. Classic one would be uh OpenAI's chat GPT but there are a lot lot of others. Plus you can have the inference running on a machine under your desk if you need if you if you want to do that if you're particularly focused and people should be on data sovereignty and privacy. This will run on any uh provider. Just want to make that clear about both of these plugins. So also the software I'm talking about as I say it's free Libra GPL but I want to give a big credit and I will come back to this again. This is not one person working furiously uh with no social life. Well I did work furiously anyway. Um I have a huge contributions from BYCS who are based they look after 1.6 6 million school children account in Bavaria and I mentioned that uh so that you know that this is used very very seriously. So there's an animation now. Uh this is very early animation. It's moved on a little bit from here. And I believe that students should be made absolutely clear that uh if they're going to get some feedback from an AI, they know it's from an AI. And you can see here this is uh an English exam. And you can see that the response came immediately. It says yes, the sentence they kept the seat for you. And also we've got the acknowledgement that this was and this is flexible. uh the the the disclaimer um tells the student that it was done by chat GPT in this case. What is wrong with the following sentence? Tomorrow I am go town. Okay. So you can see here it's done some analysis of that. And by the way I use English uh language learning as an example because I happen to speak English and also there's a lot of people who can understand basic examples. it will work for anything that a large language model can process. Uh I created a few simple maths examples since I've arrived in Estonia. Uh and but if if an large language model can respond to it, you can create questions for it. So you can see here this is a standard uh uh quiz event. But one of the features that was almost accidental and uh it first got its widespread use for learning English in Japan was that if you set your the students preferred language and you can see here I've switched to French. I will try not to pronounce any of these words uh French words. And you go back to the same you can see that the menu options all been translated into French. And now the question is still in English. The menus are in French. I'm giving a response. I'm clicking the button and the feedback is in the preferred language of the user. And the um the teachers in Japan found this very very exciting because it's no good giving feedback to a student in a language that they are learning because they're still learning it. whereas this can give it their native language. But it turns out some of the students set their default language to English and it turns out you can actually force uh the language in a Moodle course. So um that's another option. It's very flexible because it is a standard Moodle question type. There you go. The you can see that the feedback has been translated into French there. So that's what it looks like for students. How hard is it for teachers to use? Well, it's a standard Moodle question type. So, it comes up and we've got this nice little brain icon and there are three fields. I'm going to show you of the very there are some slightly um advanced features in this, but my concept always with Moodle is that um you should be able to learn the basics in one slide. And that is effectively that one slide. You write the question stem. That's what the student sees. you what you write the AI prompt which is explain if there's anything wrong with the grammar in this sentence and then you write a mark scheme. Now I'm bit dubious about a computer giving marks because I kind of think that really ought to be a human but it is a nice quick uh summary for a teacher to see if a student has generally understood the question. So you don't know what how students are going to respond. So, it's good to test a question out and it has the ability to set up multiple sample responses. So, a standard uh response to write uh a a English sentence in the past tense is yesterday I went to the park. Lovely. Uh and you click that evaluate sample response button and it comes back and it gives you some response and says yeah, that's reasonable. You can have as many of these as you like. And I recommend that people put in a correct response, an incorrect response, and nonsense. And I didn't understand the importance of nonsense in AI when I first started working with this. And the important of a nonsensical answer is that large language models generally are tuned to flatter. In fact, I was deeply skeptical about the use of large language model until a friend of mine sent me the results of asking chat GPT, "What is Marcus Green's role in the Moodle community?" And let me tell you, Chat GPT thought I was awesome. And I thought, maybe there's some maybe there's something in this. Uh, and that that's true. I've still got the the text of it. So you have to be aware that the purpose of these tools is not education. Uh it is to basically keep you using them. So the reason I've included that is that is a slightly more sophisticated example. And I find it fascinating that sometimes large language models get familiar with common misspellings and it has made the guess that in that context the word PRK is misspelled and the correct spelling is park. might be, might not be. That's not a bad response. So, I considered removing this uh slide because it's a little bit technical, but then again, maybe it's quite interesting as well in that you can show the exact prompt that got sent off to your large language model. And you can see here, this is not the kind of thing that you would want to teachers to have to know how to create. It's complex. In fact, the sophistication of that prompt where it's mixed up the student response with some other information. This is a more sophisticated version than the one I created. And I credit the people at BYCS in Bavaria for coming up with a better prompt, which means now by default you get better responses from this question type. And that's a screenshot of in Japanese. And I am assured by Japanese speakers. Looks for one person here I know that does speak Japanese. Is that reasonable response? Yeah. Okay. So, that is yesterday I go to the park. And you can see mixed up there of the Japanese characters. You can see the word went. Um and this is the one I made up earlier this week because I thought I shouldn't constantly be showing people only English. You can see here this is a question about um different sorts of averages and the student has responded mostly correctly but they've made a a statement that is not necessarily true and it's corrected. So um this is satisfied customer territory here. Uh Jason I've got his permission to use his photo and he said I hate this term but your plug plugin actually is a real gamecher. Great for students cramming for externals. Great for early career assessors who need to get some pregrading insights and much more. Great for schools marked in red who don't want to pay for expensive subscriptions to AI grading platforms which kind of raises the question what is it likely to cost you to use the AI uh you know chat GPT or what have you and uh experienced people with this laugh slightly at this point because it tends to measured in a mixture of somewhere between pennies and fractions of pennies. It's a very very small cost. Now the future uh Marie mentioned about AI is you don't want to make it obligatory because there are places that don't have this kind of access and what you can see here this is an improvement on it where it does the feedback asynchronously meaning that if you've got a very slow platform and I when I'm say slow I'm talking about Raspberry Pi here. Imagine you could get a Raspberry Pi to run a large language model. You can in fact I can run one on my phone. But because it's a low power computer, it might take a minute, two minutes, three minutes to give a response. Is that useful? Yes, because typically a teacher will take hour or so. So that is the future. And it's also useful generally because it means that you never get hung up waiting on a slow response from your AI. So I'm very excited by that that's coming up. Um before I move on to the second um plug-in I'm going to talk about the perhaps the most important development recently with AI text happened only last week where someone kindly translated it the strings into the Estonian language. That's that's the Okay. So I'm now going to talk about a plug-in called I call AIF which is assigned feedback AIF and that's a picture of tall Tom Rosson who now works for uh Moodle HQ. This is a similar functionality but for uh assignment submissions and the really neat part it can use rubrics to create the prompt and also as is about to be delivered for AI text it can run asynchronously in the background. So the difference between these two plugins is AIEX, the one that's uh uh the first one is great for short text, which is quite common with uh question types, whereas AIF is uh for longer pieces of text. And of course, the the quiz question type is great for instant student gratification. So from rubric to prompt, teachers shouldn't be prompt engineers. Uh they should be teachers. uh students uh do get to see what causes the uh prompt to be sent off. In other words, they can see the rubric and that explains how they're being evaluated. And that means that the prompt that you don't have to see uh stays in synchronization with the criteria that is visible. So you've installed it, it's configured. Thank you very much to your admin. What is different? The thing that's different now is that when you create a new assignment, there's a check box that says AI assisted feedback. You can offer uh instant feedback. So, in other words, if it's just something where you want to go around in a circle, perhaps students can be presented with a feedback immediately or you can arrange for it to be triggered by the teacher. So, I've set up this is just a rubric. You know what a rubrics look like. So you imagine typically you'd have a lot more items in there. And this screenshot shows how it would appear where the student gets instant feedback. You've got a progress bar. It's asynchronous. And typically on a external LLM, it's just a matter of a few seconds. They will see some feedback. This is the scenario where the teacher triggers the feedback. They click that button. They get this warning. And you can see here the the topic here. Here the students uploaded a file called the case for boredom. There won't be one in the next two days here. And then you see at the bottom there, there is no grade from the AI. Okay, I kind of like that. Also, it doesn't give you automatic grading per criteria, but you get overall grading and a teacher can then go in and look at the overall grading, delete it, ignore it, edit it, change it, copy it, paste it. And at the bottom of that there is a predefined disclaimer uh to let the students know that this feedback was generated by an AI system and reviewed by your teacher. Okay. So that is what I call AIF and I have uh I've become very invested in the use of AI in education both for students and for teachers and for some people are quite cautious about the use of presenting uh responses from AI to students for very understandable reasons but Moodle is a complex beast and anything that can help teachers I see as being a good thing and linked with these two uh plugins I've just been talking about um I started creating some code that complements this that to create an advanced grading method basically a clone of the existing rubric that also has uh AI assistance with creating uh the the rubric details the criteria And I've found that if you go into a chat engine and ask for a rubric for a given topic, uh it's quite good at that. But my idea is is that it can reach into the course and use your course materials as part of the prompt to suggest and I emphasize suggest suggest the rubric. So that's one idea I'm working on. Uh I have another idea. One of the things that frustrates me deeply in the talk all the talk of the use of AI and education is their kind of nightmare scenario the end of the world scenario so much so that I've got a loop of playing this is the end of the world as we know it and I feel fine and often I will play that because people say that this is a whole new world we'll have teachers creating assignments with AI giving it to the students who will sub create their submission with AI which will be fed back to the teachers who will grade it with AI. You probably all heard this. Okay. Now, before the arrival of AI or large language models, students could never ever get somebody or something to help them out. No student ever got a clever older brother to write their essay for them. We know that sometimes they got some less clever older brothers and no parent has ever helped out their offspring. It's not new. Cheating is not new or being or or taking shortcuts. And what I used to do as a teacher when my students submitted me a really really convincing piece of work with the words at the bottom Wikipedia. I would say to my students, I didn't get angry. I'd say uh could you explain that paragraph there? And they'd say no. And then I'd say, "Did you write it?" And they'd go, "Yes." Could you change it in such a way that you can explain it to me? Let me explain about what I'm about to talk about. So my second future idea is question generation from student submission. So in other words, they submit their assignment, maybe they use that uh AI feedback tool, they get the feedback, they get a great grade, and then you have another process that looks at what they've submitted, creates a small sequence of questions, maybe AI text questions, and under controlled environments, they take a short quiz based on their own submissions. Okay. So, knowing that this would happen after their main submission, they'd know they'd have to account for themselves. A little bit like doing a viva traditionally, a little bit like me back in the mid naughties saying to my students, could you just explain that paragraph there? Not entirely. So, that's another idea I've got. And if anybody like to approach approach me about uh arranging for that to be created. And then finally, my other idea is, and this has come out of my work with AI text in that if you have a whole bunch of feedback generated by AI, I've actually got some early code for this. You run a report that looks at the feedback and works out what the students have learned and where are the gaps in their understanding. So, a teacher can click a button and very quickly get an understanding of where the students are, what they have and what they have not learned. So before I finish today uh any project and Moodle the spirit of Moodle is collaboration and cooperation. This is Dr. Peter Mayer who uh heads up the team in Bavaria for uh BYCS formerly MIBIS uh and he has been a great inspiration and help and him and the team and particularly uh his developer Philippe Meml and this is a list of some of the people who've worked on it. I say Philip Meml is one of the main developers. Um many of these people are doctor or professor and I didn't want to go through this to find out exactly how they preferred to be addressed but this what I've shown you today is a very collaborative uh project and this is what I love. I used to work in various areas of finance when I was younger and it's so very very different and this gives me pleasure that we can very slightly change the universe. I'm Marcus Green. Thank you very much for your time. >> [applause] >> Okay, thank you so much, Marcus. Uh, maybe we have some time for one question. Uhhuh. Thank you for the presentation. Um, local local models and running those. How has your experience been with your plugins, both of them? And in what case did you choose the hardware and the models running? >> Sorry, I didn't I didn't quite get that. >> Uh, what models and hardware did you choose for testing it locally and how was the experience? >> Excellent question. Which models and how about using it locally? Um, I started off using uh OpenAI's chat GPT. I moved to using Grock spelled with a Q not with a K. Uh tried all the Quen models. Uh I have used some success with the open weights version of chat GPT. But the main local solution and Adam who's here has experimented extensively with that is uh to use Olama. And when I went to Japan and I showed this off, I was sitting on an airplane at 30,000 feet over the Pacific with a modest laptop with Alama running on my machine with some small models, but small models can give you very good results. Definitely didn't have an internet connection at all. Um, so Alarm is the answer to that and now core Moodle directly supports Alarma. Previously, there had to be some workarounds. So uh but one one thing somebody mentioned to me yesterday was one of the great advantages of running your own uh alarm models is you know exactly what model you are running if you subscribe to one of the major uh LLM uh LLM yeah vendors sometimes you know one day you click the button and it's talking to one version of the model and the following day it's a slightly different version of the model so there is a lot of value to um having control over your own model. Also, the cost of inference has dropped like a stone and I believe that the cost and ease of running your own will become very very affordable uh over the next year or two and it really will not be necessary to uh pay for those uh external services. Does that answer your question a little? >> Yes, absolutely. Uh so like what 9b 24b model? >> Uh yeah, I run uh 12 12 B models, sometimes 24 at home and smaller four. So yeah, that that answer my question perfectly. >> Bigger is better, so they tell me.