ALT Digital Assessment SIG: AI in Summative Assessments

There we go. Okay. Right. We are now recording. Um so, uh I'd like to welcome Ty who is going to present on something we've heard like little snippets of um for quite a while now which is this um project on AI in summitative assessment. So I will pass over. Well, hello my friends and a very good afternoon to you. This is Todd here coming at you live and direct from North Thumbre University in Newcastle and I'm representing a group of people this evening. Some of whom were able to attend and some who were not able to attend to take it out of the garage for the first time. some of the really original and interesting findings that we have to quite a largecale research project and the first collaborative research effort from a subgroup within ALT which we're calling AI for marketing research group but within ALT. So I'm very excited to share some preliminary findings with you today. We hope that this will all be published in highquality and reputable academic journals in due course and but for now we'd like to share some initial stuff with you. So, as I said, I can't take credit for any of this, though I am taking the lead with the presentation today. You can see the names on this slide of everyone who's involved. And you might be saying, "Wow, that's an impressive team of people." Yes, you're exactly right. We've got educators, education technologists, support staff, and people working outside of universities both in the UK and internationally involved in this group. If you're thinking, hey, I think I could contribute to this work, please get in touch with me. Happy to consider your request or even if you'd like to join some of our conversations that we're having about AI for marketing. We'd love to have additional input and insights based on your experience. So, as you can see, range of universities involved in this research. I'm purposely doing quite a lot of talking and waffling over it and to give you a chance to scan some of the names there. Maybe you see someone you know They're doing amazing work. If you like the look of this group, they see there's a new web page for summitive assessment subgroup at alt. Got the link on the bottom of the slide which will be shared with you afterwards. Okay. We care about AI for marking. We care about what educators think. We care about what students think. And we care about what education technologists think. and we've completed twothirds of the analysis sort of on the data set that we have. What do we know about what educators think? Well, not really very much to be honest. There's a limited number of studies on this topic. If you know more about how AI works, if you got the technical skills and proficiency, do you distrust AI for marking more? Are you more skeptical of it? Unknown. What about complex submissions made by students? So, long form writing, can AI really grade those and provide an accurate mark and high quality feedback? What do educators think? Unknown. And well, when it comes to ethics, professional judgment, and job security, what do educators think? And you see we've got a range of sources there including that offqual report from earlier this year. They failed to answer some of the questions that this group hopes to in the context of educators. What about students? Well, we know even less to be honest when it comes to topics like ethics, trust, we don't know very much. A recentish piece of research from University College London described students views of AI being used for marketing as uncharted waters. So yeah, pretty much true in 2025 and six. Uncharted waters. So we hope to contribute to discourse on this important topic. Well, here's a little preview of what students think. They think it's inevitable that AI is going to be some part of marketing in the future. So we want to make sure that we do it right and in a way that listens to students views on this topic. But more coming on that as we proceed with this presentation. So I said that we've got a big data set. Yes, it's true. As a group we designed and ran a survey questionnaire. It opened on the 12th of May 2025. We closed it on the 30th of November 2025. And we got responses from students. 577 student responses in fact of which we care about only 549 because those responses are from the UK. We got 532 educator responses in total. I can't remember the total number of educator responses in the UK, but we're not considering those as part of the presentation today. We only care about the first 330. The reason that I'm talking about those is because we've conducted extensive analysis on the 330 UK educator responses and have submitted and it's under review a manuscript at research in learning technology. We expect that's going to be published in the coming months. We got some minor revisions for the original submission that we made. So if you think I'd like to learn more about this, feel free to get in touch. I'll update you when that's published. So, wow, what an impressive data set you got there, everyone. Yes, it's true. H you can see from this JISK screen grab, 1,183 responses in total. And this involved calling in favors, asking people that we know in our networks as a research team to complete the questionnaire, contacting students that are studying our modules or across the universities that we work at or the institutions that we work at to take the under five minutes that it would take to share their views on AI for marketing. So here's the way we're going to run this presentation today. I'm going to summarize what we learned from our educator data. I'm going to summarize what we learned for our student data and well, we'll make some preliminary recommendations and conclusions based on this. We hope that this is all going to be published in academic journals in the future, but for now, here's where we're at. Okay, fantastic. So, the 330 educator data set, you get an idea of who the participants are. Maybe you think, "Hey, that's me." Uh you can see the majority of respondents were in the 40 to 49 year age range. That's not me. Uh geographic location mostly from England. Gender mostly female. And that's something that we've noticed across this research. The ladies love it. Yes, it's true. They want to complete survey questionnaires on AI for working. Most of the participants were experienced educators. We've given these a definition just breaking up between new to the profession to much further advanced in their careers. And what do you think of this? The educators preferred a hybrid approach for marking. 62.4% wanted a hybrid model where it's humans marking with AI assistance. Maybe this makes sense with what you were thinking. Marking's hard work. Marking's taking a lot of time. I'm marking as we speak for my module and dang I'm under serious pressure to get this done before the deadline. Would I be with the 62.4% who want the hybrid model of marking though? Not entirely sure about that. What we learned from the educator data is while our respondents reported having high levels of digital literacy. So for our questions, there were single item measures. They ran from one to five with five being the highest of course and one being the lowest. When it came to digital literacy, they were rating themselves on average 4.12 out of five. So the mean is three, indicating that they think they've got high digital literacy, high familiarity with tools like chat GBT, Claude and other chatbots and the like. The confidence with which they think AI systems could be used for marking was very low 2.28. So if you remember the mean is three. We're under the mean. So they're leaning no AI can't produce a fair result. We noticed in our data that male respondents reported having higher technical proficiency, but the lack of trust in AI's fairness was the same across all respondents. We've got some interesting stuff coming out in the future about how men and women respondents to our survey questionnaire report on their technical proficiency and their use of AI outside of work. It's really interesting stuff. I promise you that. From the educator data set, we had open text questions. And well, what I say to students when they're doing their undergraduate and post-graduate research projects is don't include open text questions in a survey questionnaire. If you want open text answers, just do interviews. But we couldn't resist throwing in a few open text questions as part of this survey questionnaire. And people didn't cut corners. People were writing long essay type responses on how they were feeling with regard to different questions. We got some interesting data from that. So it was worth breaking my own rules for once. The major themes relating to drivers of resistance was resistance to AI marking that we identified although you can see them on the screen. Is there anything more boring than somebody who's reading while on a slide? No, there isn't. The first one we call dehumanization of education. So removing the person from this expert relationship between educators and students. The idea was that students would use AI produce work for university that educators use AI to market and what's the point of the whole thing after that. The second theme was about the erosion of trust and accountability and this focused on data privacy and the worry that student submissions would be used to train models without consent. There was also some focus on we don't fully understand the technical mechanics of how AI works and this takes away some transparency and accountability from there. So if a student comes along and says I got 41% in this can you explain my mark to me that that transparency would disappear versus if a human was marking a student submission. There was a focus on staff procarity and deskkilling. So this was to do with the commodification of higher education, job displacement and reduced staff bars. These are the themes that came through and educators were worried that their role would become redundant and it would make them worse writers and thinkers. Some respondents to the survey questionnaire said, "Marking students work improves my abilities as a researcher and a writer. My intellectual skills essentially got some implications for policy. We based on our data believe that technical training won't help with this trust gap. It's based on deep professional and ethical concerns. Over now to the student data. So, the student data, we haven't submitted this to a journal yet. We're hoping to submit it before the end of this month. In the data set, we got 403 traditional students, 146 mature students, and yes, there are some interesting differences between these groups that we're not reporting on today. Again, the ladies love the questionnaire. The majority of the respondents to our survey questionnaire are female, mostly undergraduates. And you can see there are 388 versus 103 post-graduate students, mostly from England, the same as the educator data set. And now you get to see some of our questions and some of the mean values. So as I said a little bit earlier all the questions are on single measure items and they go from one being low, three is the midpoint and five is high for each of the questions. So for instance with regard to do these students use AI in their daily life, professional settings or hobbies, you'll see this is below the mean. So the students aren't using AI outside of their studies which is very interesting. They don't think their cultural or educational background plays an influence on their willingness to use AI. They're saying they've got good familiarity with tools like chat GBT and other prominent uh chat bots. They're quite confident in their digital literacy. So, this is self-reported. Maybe you disagree with this. I think I would as well with a mean of 3.95. They don't think the peer pressure or dorms influence how they use AI. They don't think that AI can accurately mark end of module assessments. And they do think that AI is going to impact their future employability and professional credibility. Strong answer there. I would say they think that AI is less reliable than a human when it comes to marking. They don't think it's going to give a fair mark for essays or report style work, so long form writing assessments. And they think that AI is more or less going to be about the same amount of bias as a human marker. You can think what you want on that one. For qualitative comments, so more to do with the feedback than the marking. They think that this is below the mean, so not useful. and they got very low confidence in the privacy and security of data even when assignments are being uploaded for marking to a university approved AI system. So very interesting indeed the way they shake down with preferences for blended marking wholly human marked or wholly AI marked the students want wholly human marked. You can see here 314 of the 549 students want h wholly human marking. So it's interesting to see there's a tension between what educators are looking for. Educators want that blended marking. Students want holy human marking. I wonder if this could be an issue in the future. From our quant quantitative findings, we can see that students have high confidence in their digital literacy but low trust in AI marking and are rejecting AI marking. They want a person to mark their work. I'm going to pass over to uh Claudia now who's going to talk about one of the questions that we included which was do you have any other comments? >> Yep. Thank you very much. Um we looked at this data with Nurun who's also on the call. Um but I'm going to discuss just one aspect of uh what we saw in the any other comments section. So as you can see first of all only about a fifth of the students took advantage of say of being able to say something uh in this additional question this last question. Um and most of the respondents who chose to answer to add some of their comments were um the younger um the younger lot and more or less equal uh gender split. Uh as you can see from the color to read all the comments. uh there were some very very strong there was some very strong language used and we think that this is significant because again in the space where the students were free to add their thoughts without any any structure uh or any specific uh guidance uh quite a few of the students had really really strong language to um to use so and very negative. So we had some positive attitudes but they were few and not quite as strong but we had as you can see also from some words that I couldn't actually put in the slide. Um they are um a lot of the students quite a large number considering of the students had very very strong emotional absolute and charge uh language that they wanted to communicate. Um and I think this is u very relevant because first of all again it shows how uh emotionally involved uh a good number of the students um are in this debate and also very strong emotions I don't think are always um useful when discussing education there might be sometimes but um so there's definitely this echoes the the idea of distrust and and rejection whether this is due to other factors I there's more research to to be done. Uh but yeah, just just wanted to highlight that um when given the opportunity quite a good number of the students wanted to use very strong language and wanted to make sure that uh in the survey was very clear that they had an opinion and it was not they didn't mean words uh let's say so I think it would be really important to to incorporate this point of view in the overall research. Thanks. Yes, thank you very much. And fascinating, as I was leaving North Thmber University the other day, I saw a sticker on a lamp post that said, "Kill AI before it kills your brain." Also, this year, I encountered my first student of the business school who refuses to use AI. And for my assessment, students use AI to write a research proposal that they mark and critique. So, we had to come up with a creative way around that. Um, so yes, well said, Claudia. There's definitely some student resistance to AI and especially when it's high stakes stuff like how their degree will be classified and the marks that they're going to earn. So I mentioned that we had some open text questions in a survey questionnaire sort of breaking best practice rules for research but it's okay and we got some great data from that from the student data set and these open text questions. I'm going to report on some of the qualitative findings. So students thought that AI integration was inevitable. I did hint at this one earlier in the presentation. They see universities as businesses and as businesses they want to cut costs. They believe that AI marking is going to be brought in as it's cheaper than using lecturers to mark. They didn't like the idea of that. They believe that human oversight is indispensable. So in cases where AI is being used for marking, there must be some human oversight as part of that. So yes, AI can help speed up the marking process, but it must be educators who are making the decisions on students marks. Students wanted full transparency. I say within reason, but they wanted full transparency with regard to how AI is going to mark students work. So thinking about where it's going to be uploaded, thinking about the ethics of where the tool is based and whether it's marking the whole thing or part of a submission. They wanted to know what was happening. I said within reason because that links up with another theme that we'll be talking about in just a moment's time. students wanted the right to appeal. So, say for instance, a student gets a result. They say, "I don't like this result. I think that the AI was too harsh on me for whatever reason." They want to be able to say, "No, I want a human to mark my work." And other students were asking for opt out. So if for a module work was going to be marked by an AI, students wanted to be able to appeal that or opt out of that. So say I don't want my work to be marked by a bot. I want my work to be marked by flesh and blood human being. Students also highlighted academic hypocrisy. So they wanted everything to be disclosed about how AI was going to mark their work. They felt that if they're being transparent about how they're using AI in the development of their submissions for modules that universities should be transparent with them about how AI is going to be used for marketing. And they like this idea of a spirit of mutual disclosure between both educators and students. I'm just flashing these slides up for a moment, but I hope you can see at the bottom we've got an illustrative quotation from this thematic analysis. This student says, "We cannot expect students to tell us about their AI use if educators do not declare theirs." So, it's very important to be totally transparent, open, and honest about how AI is being used. Students felt, why am I paying so much for university when I'm not getting that human touch? So they think that if their work is not even being marked by a real person, what's the point of spending so much for an expert to have a look at it? And I mentioned earlier, students wanted to know everything, but not necessarily everything about how AI would be used to mark their work. Students also recognized that using AI for marking might open up a way for students to game the system. So putting in certain signifiers or markers in submissions to an AI marking tool could mean that some students are getting higher marks than they deserve. So looking for certain characteristics of the work. Knowing exactly all the technical details of how the AI marking works could allow students to find a way around that. Students didn't like the idea of that either. So they wanted to know everything but not everything. They felt it was fair to know almost everything with regard to how AI would provide a mark for their work. Students really cared about socioeconomic and environmental risks. There was lots of talk about the environmental impact of AI, the water use, the power use, the disposal of hardware and things like that. There was talk about AI damaging the planet even more and that jobs would be lost and displaced. That was so nice to see students care so much about jobs of educators more than you would think. Thank you so much for caring you guys. So, we're coming to the final part here, our preliminary conclusions and recommendations. I hope these were obvious from the previous slides that we went through. Students want full transparency within reason about how when AI is being used in marking and there's a suggestion of putting it in the assessment brief. AI will be used to mark section B of blah blah blah perhaps. uh transparency is balanced so non-technical information to provide trust to students about how AI is used but the exact technical criteria that opens up the system for being gamed by students is not provided. Students want oversight so if blended marking occurs that they're able to appeal or maybe opt out of marking in relation to appeals. There needs to be clear procedures about how students can challenge AI generated marks or feedback and these should have consideration to students intellectual property. Maybe not so much of an issue in the business school where I am, unless a student is very protective of a student enterprise they're developing and it's not an assignment on that. But if a student requests that their marking be done entirely by a human, that option would be available. So the recommendations, mutual disclosure, everybody be honest about what you're doing and how you're using AI. Acknowledge the ethical impacts of AI for marking, including data privacy and being open and honest about where, how data are stored, they're uploaded to AI marking tools that are being used by universities and involving students in decision making. So thinking about how can we incorporate students as much as possible into decisions to use AI for marketing tools in ways that show respect for them and hopefully challenge that image of universities as big corporations introducing cost cutting measures that are just going to negatively impact students and the people working for universities. So we're going to go over to some questions in a moment. I hope you enjoyed this speedrun of our preliminary findings. Um, hope I've given full credit to everyone who is involved in this research and we hope to see this all in print in the not tooistant future. So, thank you for your time. Thank you for your attention. That's the end of me continuously speaking. I think >> that's fine. Hopefully, people can hear and see me again. That's brilliant. Um, so I'm wondering if what we should maybe do is have 10 minutes of questions kind of for you, Ty, and for for everybody in that in that group because I think that's probably raised lots and lots of of um questions and discussion. Um, I will also put in the chat um I've got a Padlet board which is doing a kind of lean coffee style discussion. Um, so if people want to leave questions in there or um, uh, broaden things out a little bit and ask more general questions, um, uh, that's a good place for doing that. Um, so I I'll tell you what, I will share my screen just to show that, um, uh, on my screen. There we go. I also put a little um section in there just in case people wanted to introduce themselves. Um but yeah, maybe maybe while people are leaving responses in there, perhaps Ty, you would like to um answer any questions. Oh, David, you've got your hand up. >> I got my hand up because there are some questions in the chat that Ty might have missed. >> Uh I've tried to respond to some of them. Um, but I don't know if if anybody else in the group wants to speak up. Uh, I know Salani is here. I can see Nurun. Anybody else? >> Thank you very much. Okay, time for me to scroll the chat. Here we go. Uh, Emma Herren says, "Strong negative language." Yes, exactly right. There's a lot of negativity out there with regard to AI marking. Yes. And here's an interesting one from Steve Bentley. Yes, we asked about AI for marking mostly very generically in the intentional design of our survey questionnaire. We were hoping that someone would bring up tools like feedback fruits, teacher matic, grade, and the like. As part of this, the awareness of tools of this nature seems to be very low indeed. So mostly when our respondents were talking about AI for marking, they're talking about the idea of loading a rubric to a chatbot and getting an output that's a mark and some feedback on that submission. But you're right about that. As part of our future work, we're going to be exploring stuff to do with AI marking tools, tools designed for that purpose. But you're exactly right about that. We've got a comment in the um in the padlet uh that's means don't show distribution as it could be biodal. I don't know if the person who wrote that wants to um expand on that. >> Mike Wald does Mike will publish our full data as part of the publications that are coming out from this. So you'll find it all in academic journals and we hope to share the data set as well for future researchers to have a look at too. It's very much under development and just preliminary analysis being reported on today. Plenty of opportunity to interrogate the data set further. Okay. Did you look at different implementations of AI beyond generative AI when assessing trust and confidence? We deliberately left definitions of AI open to our participants and gave them some options to open text responses to comment on what sorts of AI that they're talking about. Most of the respondents were talking about chatbot type tools and using those for marking. Phil Marson says, "Curious, if you set an assignment for a student to use a wiki for some reason and they refuse to see the assignment on some sort of principle, do you try and come up with something they're willing to do? It seems like a bit of a dangerous precedent to set to let students refuse to assessment tasks." Yes, thank you very much indeed because my module does two things. Number one, prepares students for their final year research module and two teaches them effective, critical, ethical and responsible use of AI. I am sort of playing a bit of a dangerous game. I don't say to students, if you don't use AI, let me know and I'll figure something out for you. And this is the first time I've ever had a student who refuses to use AI. They said it to me in class and I said, "That's okay, but I'm going to design you an assessment where you mark and critique a research proposal. And what you should do is think, did AI write this or did Ty write