AmplifyFE - Assessing in the AI era: an AI usage scale in practice

Thanks much, Emma. Um, hopefully you can all hear me fine. Um, Emma could hear me earlier when I was talking, so I'm just going to trust that everything is working the way that it should do. Um, thank you ever so much for joining. It's it's really nice uh to be asked to do a webinar, but also that people are interested in hearing what I've got to say there. Um, so I want to talk just just for a few minutes really uh about how we are making use here at Bridgwater College of uh an AI usage scale. Uh and in particular, kind of what some of the student views of it are, what seems to be working, and a little bit uh about some of the evolution of it. Um, I deliberately said I want to keep this short because we're all busy people. Uh and nobody really wants a lecture that goes on and on and on. Um, so inevitably there will be questions. Please feel free to put questions in the chat as we go along. I've got the chat pane open. Uh so I will try and respond to them as I see them. And uh potentially I might say I'll I'll deal with that later. Uh but you're welcome to ask. Uh I am a uh lecturer in Bridgwater College. If I move along here, and she's who I actually am. Um, so I'm used to dealing with questions being thrown at me when I'm in the middle of sentences. Uh so that's absolutely absolutely fine to do. Um, for myself, I've been teaching here at the college actually for uh 20 years. I joined in December 2006 uh at this particular college. So um I'm part of the furniture here and have been very much a part of helping to build the animal care team, and particularly the HE uh side of it. I'm the program leader for the HE uh animal studies course here. Uh but I teach on level two, I teach on level three. Uh so as far as FE goes, I've I'm kind of fully immersed in there. Uh a lot of what I'm going to talk about today, it has been with my higher education group um because I've got a little bit more freedom as to what I can do on that particular course than we've got up some of the FE uh courses. Um but absolutely what we're learning there and uh we're experiencing it here at the college. We are trying to kind of pull down into the FE where we can and look at how we can influence the decisions that are being made there uh to make them appropriate. So hopefully what I share will be relevant to you uh and interesting. I love feedback. Please feel free to uh email me if you've got any feedback how I could improve um did well. So it's nice to hear as well. Um or connect with me on LinkedIn. Um I do quite like having to collaborate with people where I can. Okay. So um assessing in the AI era. Uh if you're here, chances are that there is nothing new that's going to come up on this slide uh for you. But it's interesting how many people uh around the college and in the sector I think are kind of in that ostrich syndrome heads buried in the sand a little bit um and don't necessarily like the fact that things are shifting. Um but we can't run from it. We can't hide from it particularly. Uh we are very much in the AI era and what that really means um is this that AI is freely available uh to most. Now, I did once upon a time when I've delivered these sorts of slides before, I had this available to all because it kind of is. But we mustn't forget that actually there is a subset of students who do not have the same uh digital access that we might assume that they might do. Um and so we've got a bit of a risk here as well that when we're talking about AI use, um they may not have as easy an access as other people do. Um AI can be used to complete written tasks easily. It's getting better all the time. We we see that if you're on LinkedIn, um certainly the circles that I'm in, it seems that every single week there's a new announcement what it can do now, how it can be improved. Um it is absolutely uh exponential growth that we're seeing here. And so what we're trying to deal with will be different this time next year. But there are certain values, certain approaches that we can take that should hopefully stand us in good stead to go, "Okay, what we have is resilient for the future." Even if we do need to tweak it and we always need to review and tweak things. But what I want to go through is a couple of principles, I guess, that we can build on and hopefully see us good. Um This one I I will I will stand and die by this one. AI detectors are dreadful. Um personally, I cannot stand them at all. I'm very pleased that as a college we've got a rule that you do not use AI detectors. They are not reliable. Uh the false positive rate is far too high and the risk there of penalizing students who don't deserve it and utterly demotivating and demoralizing them is just far too high. Uh so we stand by this idea that, "Okay, we cannot use AI detectors. We can't just say with our assessments, it's okay. We'll check it out. We'll we'll put it through an AI detector and decide whether the student's cheating or not." That option is not open to us. Uh and I think that's a really good thing uh to have as a policy. Don't just try and use technology to beat it. It becomes an arms race and it's not a good arms race uh at that. But we do still need to be able to mark our students' work fairly. We need to have confidence that what they've produced is acceptable and suitable for assessment. So if we're doing that, we've got a little bit of a problem, if you like, because if they've got access to AI and we're saying you can't use AI, how do you police that? How do you deal with that? And that's where this usage scale kind of comes in as a starting point. Um it came out of a little bit of action research that I was doing in the college. And we we kind of built it through. We've talked with the staff. We've talked to the students. Um and kind of what I'm going to present today is where we are up to. But certainly I have heard a lot of this uh over the past couple of years. Uh okay, I hear what you're saying, Pete. You know, AI's available to everybody on a computer Uh and so actually, you know what? Let's remove the computer. Okay, everybody will just go back to exams. That'll be lovely. Easier to mark than all this fatty assessment stuff anyway. Um it is not the right approach. Please, let's not just go for the let's just remove the AI. As much as anything, apart from just kind of going back about 50 years in education if we do that, the biggest problem is the students are going into the big wide world after they've been with us. They are going into careers where AI is available. And if we're not teaching them how they can use it appropriately, they will use it inappropriately. It's as simple as that. So, we really need to be building into our courses the ability to use AI in an appropriate way so we are guiding them, we are giving them those baselines that they need to take forwards. And really, we need to be assessing that as well as the rest of the content. If we're going to have future-proof courses, we need to know that actually what we're doing is relevant and not just resilient. So, going to pen and paper, I it's it's not appropriate for everything. That said, it is still appropriate for some things. Okay? Please don't take me saying this as everything has to be high-tech, let AI run everything. No, no, no, no, no. Okay? Appropriate technology. And if what's appropriate is the ability to do it on pen and paper, absolutely assess that. It it works. Okay? But we can't do it for everything. Okay. So, what approach have we taken? Um yeah, I have seen that as well. They'll just rewrite what AI told them. It pen and paper does not actually solve the problem. Okay, so our approach, what we've done on the animal studies course as our little test bed for how we can take this forward, we've we've taken one overriding principle to begin with and it's this. We do not ban our students from using AI. We also do not force them to use AI. Now, that said, in the lessons, I do teach them how to use it appropriately. I show them different tools. I show them how they can use it. I encourage them to play around with it. Um and then the students for themselves can decide how much they want to engage with AI or not. But I know, for my part, I have taught them what they need to know to get started. So, they all have this grounding. And if they then want to make use of it for certain things, they can. If they choose not to, that's up to them. That's absolutely fine. And I'm not never going to penalize them for not using AI. We decided very early on we have to be realistic and we have to be clear with the students what is acceptable for every assessment they are doing. Okay, if they don't have clarity, we cannot penalize them. We cannot say, "You shouldn't have done this." if we've never told them what they can and can't do. So, we had to have a clear way of signaling on every single assessment exactly what they need to do. Okay? And I think that needs to be across the board in education. There's far too much running and hiding and kind of oh oh yeah, somebody will decide for you. No, we need to be clear. Okay? It's not fair to students to let them guess and then us penalize them. Um and at the same time, we are using AI, some of us more than others. Um and so, we decided we need to be transparent with them. We need to be candid about it. If we're producing worksheets with AI, we we talk about it. If I've done an AI image that I'm making use of, I talk about it. I we laugh about the bits where it goes wrong. Um but we are signaling in the lessons that actually AI use is not evil. It is acceptable in the right context, in the right ways, but transparency is really important so that we can build this level of trust. Um and the students then feel much more comfortable coming to me when they say, "I've done this, you know, I don't quite get it. Is this right? What's the AI?" And we can have those conversations. And I think those conversations are golden. They're really, really important for the whole learning process. So, that's our approach across the across the course. So, this clarity thing, that's where the scale really came in. We thought, "Right, okay, we need to have some way of being clear with the students what they can and can't do." Now, um this is the scale. Shout out to Leon Furze and the team that he was working with. Um they came up with a really nice scale. I looked at it and thought, "Okay, I I can see how it works." Um I've tweaked it. I've changed it. I've It's gone through a couple of iterations over the last couple of years. I've not sadly got time to go through the whole process, but this is the current one that we've been working on this year. And it's a scale that runs between no AI and AI as a collaborator. So, you'll notice each of the different levels we have named what the AI is doing, what its role is. Um and that's something that's come from the students trying to help them understand what each of the levels allows. By naming the AI role, they were able to get that. They could picture what was happening. So, at the top on this particular thing, it's not that it's better, it's just where it's positioned. We've got collaborator, and at the bottom of the scale there is absolutely no AI. So, going up from that, we said, "Okay, well, we've got a planner level where the AI can help you plan an assessment. It can't do any of it for you, but if you're wanting a bit of a conversation about what sort of headings should I use here, how could I structure this?" Then planner could be a so uh appropriate. Or if you wanted a spotter where it could just kind of look through your work and identify where grammar mistakes are or where spelling mistakes are, that's acceptable. Um editor then it could perhaps tweak those things, tell you what the correct spellings, possible other grammar uh that you could use uh would be allowed. And the next level above that then is drafting where it's starting to create some of the work, but you are having to produce the final edit. So, the initial edit might well be from the AI. Uh that works really well with the reflections. Um but then you tweak it to to kind of get it to make sure that it is a absolutely true and uh an honest reflection of what you are wanting to say. And then collaborator is that full on you're working alongside the AI and producing something together, uh however much you want to. Now, to try and make this clear, I've presented it in various different ways. You'll see a QR code um on the screen. And I should say at the end of the um slides here, there's another QR where you can grab the entire slide deck if you want so you can flick through for yourself. There are some hidden slides in here that go into a little bit more detail if you want to have a look through for yourself. Um but that AI prompts for students document is one I actually share with my students. It's got the usage scale written out there in full and they can actually um then keep referring to that. Some students make lots of use of it, some kind of glance at it once when I show it and then it's gone. Um it is a Creative Commons license. Feel free to take it and use it and I update the the one that's shareable. So, and you'll you'll see it's got a kind of when it was last updated some date on there. Um but it is essentially the guide that I use, so it will tweak as I change it uh and go through. I'll let you explore that for yourself. But one of the things that's in there that students find most helpful is this kind of tick um tick sheet. Uh so, essentially they've got questions. What can I do? What can't I do? Um and so, what usually comes up is they get a question, they can look through this chart and go, "Okay, I've got an assignment. It's been given to me at planner level. It's either yes or it's a no." Uh so, if it's a tick, yes, they can do it on that level. If it's a no, they can't. And so, hopefully you can see from this tick sheet it's a cumulative thing. So, if you're saying that the AI is acting as an editor, it is able to do all the things under the planner or the spotter level as well. Uh I do sometimes get people saying, "I don't get it. There's a tick under no AI. What's going on there?" Uh well, the reality is we cannot stop the students from using AI to learn. And nor should we, to be honest. If they want to use the tools to understand a subject better, and it's not part of the assessment, yes, they can do it. We we don't have the right to tell them they can't do that in their life. Uh so, it's making that point that actually before the assessment, they can even on a no AI assessment, they can use AI to help them learn the subject before they get to it. Uh and an exam is a great example of that. Exam is obviously no AI, uh but if they wanted to use AI to help their revision, that's not a problem. Um again, you can kind of look through this for yourself a little bit. Uh the draft and the collaborator tend to be the ones people are most interested in knowing, where does that line uh draw? And essentially, the line is drawn between uh being able to copy uh things that the AI has created into your work uh without worrying about, do I have to tweak this? Do I have to change it? No, if the AI has created it, you have to acknowledge it, always acknowledge, uh but it's allowed. Um and I do stress to them all the way through, they are the human. They are the one being assessed, and ultimately, they are responsible for the quality, the content of the work that comes in. There is no excuse of saying, "Ah, but it was a collaborator, and the AI got that wrong. It's not my fault." Yeah, it is. You submitted it. Um because that's how the real world works. You know, if I want to create something with AI, I have to proofread it. I have to make sure it's appropriate uh to be used. So, um I can see a longer question, so I'm going to come back to that one in a minute when I've had a chance to read through that. Uh but I'm really really um say, you know, that that responsibility element, really important. And the students get that. That is really nice to uh to see. Okay. So, that's fine. We we've got a scale. That's Um I'm going to take 2 seconds just to read this. So, how do I distinguish between those different points in scale with a piece of written work in front of you? How can a marker determine if students used AI to suggest headings and a framework versus writing the sentences between the headings? Given AI detectors don't work and are uh awful. Okay, brilliant question, wonderfully timed. Um because the reality is you can't. And this is why I needed my team to understand when we first started introducing the scale. There's no way that we can actually just plug this AI scale on top of existing assessments and say, "There we go. That's nice. Aren't we amazing?" It won't work. It is not a sticking plaster. Okay? It we have to come to it from the viewpoint of we need to redesign the assessments. We need them to be written with this problem in mind. We cannot police it. So, we need some way of feeling confident that what the students are submitting is appropriate. And actually, if you cannot if you cannot tell the difference in a a sensible way, then you shouldn't be using those lower levels. Okay? We need to think, "Okay, how can we assess what we need to assess at a a higher collaborative level perhaps or as an editor or draft a sort of level?" So, I've got some examples that I want to show you. But the most important thing really, if you take nothing else away from today, is the need to redesign the assessments. Okay? We got together in a room. I sat them down. They weren't overly happy, I'll be honest. Um so, what we uh did is we reviewed every assessment. I said, "You need to go to your module. You need to look at the assessments. We need to look at them with an AI eye on it now. Okay? What's the impact here?" And then, once we'd reviewed it, they were rewritten. Okay? Rewritten to improve the resilience so that we could determine whether we think they are being produced with AI when they shouldn't be and what the acceptable use would be. And as I say, I'll come to that in a second. I've got a couple of examples that hopefully will demonstrate some of this point cuz there are a few tricks you can use that are pretty effective from what we have seen. Not foolproof, nothing is ever foolproof, but again, it's trying to build in that resilience so that foolproof shouldn't really need to come into it too much. So, I see a question there. What guidance do we give students on critically engaging with the AI outputs? That's during the lessons. As I say, I I get them to play around with some of the tools. We look at them in examples and we just keep making that point that actually you need to be double-checking what it's saying. Don't ever trust. Always think through what sort of bias there might be in there. We have those discussions. They tend to be more organic than strict I'm doing a lesson on bias. We bring it out where we can and we talk about human bias at the same time because that's a great opportunity then to also introduce the fact that when you're producing your own assignments, you come to it from a viewpoint and HE in particular, you know, it's good to try and take a a step back and consider what your personal bias might be there too. So, in terms of redesigning the assessments, three steps to it. I get the get the team to reflect on their past assessments, do a bit of recon, play around with the AI tools, try and complete their assessments using the AI tools and then try and reimagine what the assessment could look like. So, if I break that down for you a little bit more, by reflect what I mean? Well, actually if we are honest, AI isn't often the problem with the assessments. They've always had a problem with copying and pasting. Okay, I I see it a lot with students who have certainly have done in the past, but they're not much better these days. That they go on to websites, they copy and paste information. That has been there, you know, I've been teaching 20 years. It was there 20 years ago. People would copy and paste. When I was at university, people would get a book and they would copy out the book. That is always kind of there. There's there's always been that flaw in how assessments have particularly some of the traditional um report writing and essay styles have been written. Um This is quite painful one for some people. I actually challenge them, has it been assessing the correct things? Have they been looking at the learning outcomes? And actually their assessment proves that the student has learned that. Or is it just a case of a kind of it was a nice assignment, quite enjoyed doing it, but proves nothing. Um so I do challenge them to really think through what should they be assessing? What do we actually need to know the students can do? And have they been uh awarding marks appropriately? Do we need to adjust how the marks are awarded? What was given slightly more weight compared to other parts of the assessment? Um The recon part then, as I say, I get them to try and complete the assessment themselves. Go on to Gemini, go on to chat GPT, copy and paste the assignment in there, try rewording it. Can they generate this assignment? And what sort of mark would it get if they did do that? Um that can be quite time consuming, but it's certainly quite eye-opening for some people. Um how easy is it? Because that's one of the things that those of you who've used AI, probably like me, there's an element of actually [clears throat] when you use AI a lot, you recognize it's not always easy to get what you want out of it. You need to have certain background knowledge about the subject to understand what it's saying or get it to work things appropriately for the level you're wanting to work at and so on. There is actually a skill in getting AI to produce content that is relevant and answers a specific question. Um And then the reimagine part of it. We look at okay, how else could we assess those outcomes? What would happen if we we said all those different um scale stages, what would a no AI assessment look like for this? What would a spotter assessment look like this? What would a planner assessment look like for this? And I'd get them to try and reimagine it in different ways and decide what they like. Playing around with lots of different things and trying to get them to do a bit of variety means that actually those creative juices get going and some of what we've had produced has been fantastic and the students have loved the new assessments, which is really nice. And then that question and I was asked earlier, how do we actually ensure the students are sticking to the given level? And if you just have the answer of I can't, then it's not an appropriate level. Okay, that's the bottom line. [clears throat] If you cannot ensure that you are confident the students are going to stick to that level, we need to look at the leveling again and how it's going to be done. So, let me give you an example. As I say, I'll give you these slides later because I'm really conscious of time. But this is an editor one here. It came from a conservation module. And essentially when I sat down with this module uh module leader, uh we decided that actually getting them to do a proper conservation survey, so they're doing the practical work and then they're writing it up. If they write this up as a report of the survey they have carried out, then actually that instantly ties it into the real world. It cannot be generated by AI sufficiently to be realistic because we know the local area. We know where these surveys are taking place. We're asking for maps of where they've gone, the images of what's happening when they're there. Yes, there are elements here where the AI can help. That's why it's editor because actually there could be some editing done by the AI and it wouldn't matter. But essentially that report has to be really human led to be able to pass because it's just not going to meet the criteria. Going up a level, drafting, this is one of mine. Reflections. Again, it's it's embedded in the real world. They've done presentations. Now, reflect on those presentations. Tell me about them. Tell me what worked, what didn't work, what goals you want to do. Yeah, if you want to get AI to draft that reflection, that's fine. But, when I'm marking it, I know whether it's talking about your actual presentation, what sort of feedback you had. I watched those presentations. I know whether they were appropriate or not. So, again, it's grounded in reality. Those that grounding is a really good way of keeping track as to whether the AI is is tinkering too much. It starts making things up. Collaborator one, again, this is actually one of mine. And I I've given this a more kind of real-world example of right, okay, let's see are you ready for industry? Okay, and that's that's the baseline we use. If you're doing a collaboration assignment, it's okay, could you produce something that is industry standard? You have access to all these tools. Produce something that could actually be in a magazine. Don't just produce something that looks a bit scruffy. Okay, get the AI to work for you. And we make it really clear, do do not get marks for using AI. You don't get marks for not using AI. They can do it however they want. Okay, so they can do it entirely without AI and still get a fantastic mark. I've seen people do that. But, those people who perhaps lack some of the creativity or struggle with how do I word it for a 16-17 year old? Fine, get the AI to help you. Get that rewording in there. And they've got that freedom to be able to do it. And really interesting, I marked this one quite recently. We got a really amazing spread of marks right from fail all the way up to almost 100%. So, having that ability to use AI has not shifted the marks all the way up into distinction level. It is very much spread. I was asked recently, what happens if you got two levels you want to use, two different parts of this? So, this assessment was actually a professional professional discussion and we said, "Okay, you can prepare notes using AI, but during the discussion, it's just you and the assessor and those notes. You can't read off them as a script, but they're there to guide you. But, if you want to use AI to help you prepare them, as long as you can talk about it for yourself and you can demonstrate your understanding verbally, that's fine. That is what we're after." So, there's a >> [laughter] >> a couple of examples there. Um really quickly just to finish cuz I I clearly waffle. Um does it work? Well, when I've asked the students, there's about 13 students that replied to me. It's quite a small cohort. Every single student loved this approach. They really like the fact that we don't ban them, we don't force them. Some of them don't like AI, they don't want to use it. Um it feels really natural to them. They're they're they're happy with it. They like to be shown. They like to have the opportunity, but they like that they could choose. Um how clear have they found those usage levels? Uh either clear or very clear. Okay, nobody thought it was completely Yeah, some students definitely do refuse to use it. Um and that's fine. You know, that we don't have that problem on the course because it's their choice. Um how confident have they been at understanding the level of AI use? Now, here, I would say we've got a little bit of work to do still. Okay, because I would love to see everybody be very confident. Somebody was slightly unconfident. Okay, we got work there. Um but overall, I'm happy that most of them were there on the confident. Uh so, does it work? Well, no element of student work was referred to me as the program lead to investigate a bit further for academic misconduct. Okay? Having this variety of usage, the students lent into it. They knew they could use it in certain places, not in other places, and where they couldn't use it actually, if they tried, it wouldn't work for them anyway. So, they didn't submit it. Um so, on that, uh it's not a huge cohort. As I say, we had 14 students uh that I surveyed. Only 13 answers actually came in. Um But, you know, I would say we had it on the second years as well. So, it's about 20 overall for the full cohort there. But, I'm happy. It's the first year that we've never had anybody referred to me for further investigation. So, that's good. So, does it work? In my opinion, yes. You're welcome to have your own opinion. Is it perfect? Absolutely not. Okay, it's something that we will find each year. Every year, I have that open dialogue with the students about what they liked about it, what worked, what didn't work, which assignments perhaps need a little bit more tweaking cuz it wasn't clear. And we're working together, and the students appreciate that collaborative approach as well. So, there we go. I am out of time, but if you have any more questions, please put them in the chat. Thank you ever so much for joining. I hope that's been helpful giving you something to think about. Feel free to email me. Feel free to scan that QR code and grab those slides. Have a little look. As I say, there are some hidden slides in there just going into a little bit more detail. And yeah, connect with me on LinkedIn. Thank you very much. >> Thanks, Pete. That was brilliant. Really appreciate you delivering it. Um if anyone does want to ask any questions, you are welcome to. I but I appreciate we are out of time pretty much. So, if you do need to go, let's just say thank many thanks to Pete for such an amazing presentation. >> Thank you. And I will say I'm happy to stay behind for a few minutes. I'm not in a rush to go. So, if anybody does want to stay and have a chat, we'll kill the recording, chop it off at the end, but you're welcome to do so.