Mitigating AI Misuse in Introductory Python Courses with Graphical Programming Tasks by Tristan Bunn

All right. Thanks for the introduction. Um, so as has already been mentioned, my background is kind of coming from a designer and moving into development and now sort of working at the conversion of those fields. Um, and then kind of pulling that into the academic space. Um, so I've always kind of worked at the intersection of sort of the technical and the creative. Um, and this kind of dovetales with what I'm going to be talking about today. Currently I hold the position of a lecturer uh in creative technologies or interaction design. Um, so that's kind of the space I'm in. Um, actively involved in open source projects and um, as has already been mentioned um, I've written a book about this topic. Like a lot of people in my position, I'm kind of starting to grapple increasingly with like what do we teach and how do we teach it? And of course, a big threat to that is what's happening with LLMs. Uh maybe that's not a threat, it's an opportunity as well, but I'll discuss that as I go along. So I think a good place to start is to ask, is anyone familiar with processing? And when I say processing, I'm specifically talking about a piece of software um and IDE for programming graphics. I don't know if anyone's familiar with it. Um so I thought I'd introduce it very quickly because it's going to be important for what I talk about moving forward. So um it's a piece of software that's about two decades old and the idea behind it was to provide a new environment especially for visual people and artists and and that type of uh student uh to learn how to program. So uh something that's very simple you can open up you can write some very simple commands draw squares draw draw circles and then get far more creative and do some pretty impressive stuff with that. So I became very uh interested in this given sort of where I work and at that intersection of creativity and technology and um I thought processing was great and I started teaching it probably about 10 12 maybe even 15 years ago. I can't quite remember. Um it's based on Java. Uh as I said it's aimed at artists and other types of students in creative fields. Um and I thought it was great. I didn't really like the Java part but um I started looking for Python alternatives pretty quickly. Um and interesting enough because processing has been quite successful it spawned a whole bunch of derivatives. So for example uh for JavaScript there's p5 js you might have used that operates in a browser JavaScript based uh there's J Ruby art which is of course Ruby based there's processing for Android uh and there's several other uh instances of of of things inspired by processing if you look around uh around about 2015 2016 someone actually made a Python version of processing. Now, this looks exactly the same as Processing, and the idea is it's basically a plug-in. So, you kind of run Processing, you install the plug-in via the IDE, and then all of a sudden, you can do all the things you could do in Processing except writing Python code. Underneath the hood, what's basically happening is it's using something called Jython to translate that code, Python code into Java. Um, and then everything happens the same way as it would the Java route. The thing about this is it had its limitations. So, Jython is Python 2, never supports Python 3, uh, which was always a, you know, a bit of a hold up in certain situations. Um, very limited support for libraries with C extensions. So, things like NumPy weren't, uh, operable in there. Um, and kind of just to cut to the end of it, uh, when Processing moved to version 4, uh, the Processing Pi compatibility was never supported from that point onward. So, it's kind of obsolete in many ways. So I was desperately looking for some kind of replacement for this. So um what I developed was something called funny pi mode which is actually a combination of two pieces of software trying their hardest to be exactly like processing. So the first piece is um something called thonni. Um I don't know if anyone's familiar with it. It's a beginner IDE. It's really nice, really compact. Uh you download it, you run it. It's got a bundled Python interpreter, so anyone who's trying to learn Python doesn't have to bother with terminal, installing Python interpret separate to an IDE, all that kind of stuff. You kind of open it up and it just runs. So I thought this was great. Um, and then sort of around about the same time as I discovered Thonny, someone started working on a library called Pi 5. Now the whole idea behind Pi 5 is it's a Python 3 interface for the processing graphics library. So, I figured if I could put these two pieces together, I'd have something that would work really nicely. Um, so totally guided and totally trying to copy as as hard as I could the processing interface. Um, which you can see at the top, I created kind of the same thing as a plug-in for Thonnie. And it's trying to be as as identical as possible. Uh the big difference of course being that in the thonny instance um you're writing Python um and it's obviously outputting um sketches as as you call them in the processing space um via processing still but obviously with Python 3 code uh and then there's all that other good stuff over there. So for example um because it's not using uh um Jyon uh it's using Jpipe instead by the way uh it means that you have access to libraries that have C extensions. uh it supports Python 3 and and you know kind of solved a lot of problems that I had with processing.py. Um if you want to find out more about Pi 5 uh it's available on on that website. I I will share my slides later. Um the other thing to mention about it does have Jupyter notebook integration which is quite quite a neat feature as well. Um so uh conceptually speaking we've got the thy editor we've got Pi we've got Jpipe under the hood and then everything is processing thereafter. So it'll do everything the processing does with but with Python 3. Uh it's got all the normal stuff you'd expect. So you know um integration for for Pi 5 and all the rest of it within the editor. Um it's got a really nice plug-in manager as well. And because it supports um CPython libraries, there's a lot of things that that I can use now that I couldn't use before like Pyunk is quite cool for for fairly speedy physics. Um, I integrated a color mixer, other processing, color mixer. And one of the things that was really nice about it is around about this time I also started messing around with pen plotters, which are sort of like, you know, um, 2D robot arms that can plot out graphics. Um, and it was a really good compliment because there's a lot a lot of nice Python libraries for dealing with um, plotter art. Uh, usually it's SVG based. Uh so you know ways I could kind of uh better manage layers to change pen colors a lot of other libraries that optimize plotting um you know shortest routes for the pen to follow avoiding duplicate lines that kind of stuff um so I did a lot of work with that which which was good fun as well and of course doing this kind of stuff and being able to show students uh you can actually make quite fun and interesting things with it also inspires them. Um so that's just some of the artwork that I created with it. Um however um this was all sort of part of my PhD research which concluded quite recently. There was this ancillary insight that kind of came about at the same time. It's interesting because I started my PhD before the advent of you know uh you know co-pilot and chatgpt and 22 and when I finished I was on the other side of it. It's quite a different world then. Um, and one of the things that when we were testing the software with students to see how effective it was, we started to find out and discover that it was actually quite handy um, for getting around students kind of cheating uh, with LLMs. So, um, before I kind of get to that, I'll mention a couple of things. Obviously, you know, GI presents a lot of interesting opportunities and challenges. Um, you know, chat bots can help students, but they're not so good for complex tasks. students quite appreciate the geni stuff but they do fear accuracy and fairness issues. Um systems uh a lot of geni systems are now driving uh sort of tutoring tutoring type software. Um but then again students worry about become too dependent on these types of solutions and not learn properly. Um there are some interesting uh things that have been done kind of in the processing space. So for example like with p5 js there's a very prominent educator in that space a guy named Daniel Schiffman. Um and they've actually for p5 js made this kind of bot. Uh the cool thing about it is it won't give answers to the students you know what I mean to help it explain and all the rest of it. So it's very constructive about how it goes about things. Uh and interestingly enough, I was actually able to take the Pi 5 um Jupyter kernel and use it in a similar way because Anaconda now includes this um AI, you know, AI assistant. Um but obviously you're relying on the students being constructive about it. Uh they can just come along and say, "Give me the answer straight up." Um you know, they're not always going to, you know, um uh say, "Well, give me clues or point me in the right direction or something like that." Um but that's a bit of an aside, which brings me around to unauthorized Geni assistance. So, it's like uh maybe this can be used, you know, all the stuff that that I've created as a as a way to help out with this problem or this challenge. Um there's a lot of opinions and solutions and things out there and it's kind of a developing space. Uh and generally speaking, if you look at how people are kind of getting around this issue right now, uh you can kind of group it into three sort of uh separate approaches. uh code tracking and authorship, you know, like using styometric analysis. Um asking students to hand in stuff that has a code history that you can kind of track if everything arrived in one go, you know, it looks kind of suspicious. Um and then obviously, you know, proctor's exams, but that's becoming a challenge because more students are are studying remotely now and online and you know, you can't always expect them to be in a room. Yeah, there's proctoring software, but then there's ethical concerns and you know, it's it's complicated. Um, another thing is to actually just kind of generate uh an integrity culture. So have a lot of discussions with students about policies and encourage the right kind of practice with these tools. Um, so that's another area. And there's also um design and exposition. So uh giving students more parameterized or personalized tasks uh things that can kind of work around it or expecting them to perhaps give a commentary on what they've created and so forth and so forth. I don't want to talk about all of those. What really interested me when I was doing this research is I found this paper over here. Uh this is the preprint. It's available if you want to look it uh look it up. Um was that these two researchers looked at a whole bunch of methods that don't kind of align with what I talked about before but um how to sort of design more resistant AI AI resistant tasks or assessment. So um one of the things that jumped out to me was of the things that they listed they spoke about incorporating visual graphic components. Um so for example they took the nifty task set. This is like a a collection of of tasks that are used in introductory programming courses and they tested LLM at trying to perform these tasks. Uh and generally speaking the tasks that are sort of more to the top right of this chart over here um the LLM struggled with more. So, for example, that like Sanki diagram up there, you know, tended to get that much more wrong than it did simple kind of text in text out type challenges. Uh, so I mean, as I'm sure a lot of you are aware, if you put in like write me a Fibonacci sequence calculator, you're going to get a perfect answer, right? You can literally take the assessment brief and put it in and get perfect answers out of it. So I thought, okay, well, if these graphics do make it tougher for LLMs, maybe I should start to look at what type of things they might struggle with. And because I've got, you know, um, the software that I've that I've kind of been developing and working with, let's try a whole bunch of tasks. And I think one of the things that's really advantageous about this is that you can perhaps keep tasks at the same level of complexity and challenge. You know what I mean? by introducing graphics because there's another school of thought that says oh well if students can use LLMs now why don't you just make everything harder but the the sort of downside of that is then they never really learn the fundamentals perhaps so it's like can I use graphics to make the tasks as simple as they were before but more LLM resistant right so if you take a look at um this is sort of one of the first questions in the in the assessments that we used to give the students it's like you loops 101, you know, write a loop that generates a sequence of numbers up to 10. Um, so how about I just show the students how to like draw a circle using pi 5 mode and then they can make 10 concentric circles or something like that and then that should should be a bit more effective. So um the idea is how can I take the existing student brief add as little extra complexity as possible but make it a lot more AI resistant. So this is the sort of revised first page of the the brief that we give out to them and it's a very simple cheat sheet of you know a handful of commands for drawing things. So the idea is they should be able to pick these up very easily shouldn't be too challenging for them and then instead of giving the students tasks that are like text in text out um how about we just give them a graphic and say recreate the graphic um and then take those graphics and chuck them at them and see if it can actually complete them successfully. So this is just sort of the new front matter for the brief just a very quick introduction uh of how to use sunny pi five mode and then this is task number one. So it's like draw these concentric red circles and there's a little bit of scaffold code over there. Um so then the idea would be okay how about I feed this into LM see how it does and I'll show you the results shortly. Um that's another one over there trying to make it a bit more tricky. Uh and so it goes and there was a whole bunch of these and I'll show you them in a moment. So the entire task set that we replaced the existing brief with which were all by the way text and tactile type of type of challenges were these six tasks. Um so I'll go through them right now. In terms of the uh models that I test this with. There was claw gemini gpt 40. Um, I'm sure that there are maybe some more accurate models or or or better models at doing this stuff out there, but generally speaking, students that are learning to write a loop probably just go straight to Claude or Chat GBT or something, right? Um, so those seemed like the the appropriate things to test. So on the left side, you have the task and then it's the best out of three attempts for each model. Um, so you can see like Gemini pretty much nailed it over there, which I kind of expected because it's a very simple task. Um, but then there's these weird like like issues like if you look at the OpenAI GPT 401 over there. Um, how would you work out how to write a loop and get the right color stroke but then draw the wrong number of circles? You know what I mean? So, so these these sort of things start to appear. I expected this to be pretty trivial. Um, but then I started to try and see like what things did it struggle with. Um, so here we have um, you know, alternating bands of color which it's nailed pretty well. Um, but as soon as there's some kind of an overlapping shape, it couldn't kind of tell it was overlapping shape, right? Um, it either draws arcs or something weird or it kind of just didn't notice that there's actually obviously for a human, you know, a white rectangle on top of the the original challenge. Um then there's this one over here which you know uh you know loops nested within one one another. Um and it it's it's pretty impressive but once again there's like some really like weird issues over here. Like if the student could get the nested loop right why would you land up with dots red dots in that formation over there which which clearly you know it's supposed to be alternating. It's obviously used some quite different algorithms to get that right. Um so these kind of things become apparent. Um then there's some other interesting things like if you do minor tweaks things kind of fall apart. So this is just a little um uh challenge that requires the students to use break and continue statements. Um so the idea is that the the blue line is just a loop. Um the red line is a a loop that's got a break statement in it and then um the green's got a break, you know, it's got to continue in it. So it's got to jump over a couple of iterations. Um when this is sort of uh regular geometric like upright um the models do pretty well but then you rotate it like five degrees and it all kind of falls apart. Um and you kind of got those results over there. Um and then this this one over here it it kind of generally really struggled with. Um again like as people who know a program if I showed you how to use a function that draws lines you could probably get that right quite comfortably. Um, and again, this is something I kind of really struggled with. Uh, and then this one is a series of true SHA tiles, um, with a sort of bit of randomness involved. And again, it's it's done pretty well in some instances. It's not quite correct, but again, it's those kind of things where you would look at this as a as an assessor and go, well, how come the student could got the hardest part right, but couldn't get the simple things of like randomly rotated tiles or something like that. Um so in terms of um where this is going to go next as I said this was actually kind of an ancillary insight that was coming out of the work that I was doing for my PhD uh it would be um to test more iteratively. So like uh put something into the LLM uh it gets it wrong and then say look you've got this wrong can you try and fix this and see how well it does when when a human is actually saying like no no fix this you've done this wrong and see how close you can get it. um see if other if if it does better with other types of of graphics libraries. Um I did test it with Pillow uh and it was just as bad to be honest. Um but maybe the models are are better trained on things like p5 js or something. I'm not quite sure. So those would be interesting areas to test. Um, the other thing that's quite interesting about this is that I've given all the students the same challenges, but it would be very easy to write something that generated slightly different versions of the task for each one of them. Maybe one student has eight rings in the circle, another one has 10 rings in the student in the circle and so forth and so forth. And as you can imagine, it's quite easy to write something that can output those graphics. Uh, and then at least that way they couldn't share answers with each other as well, which is kind of an opportunity. Um, the other thing is that processing can do animation and I think that could add a whole new level of complexity for the LLM to have to deal with. Um, so that's also another thing to explore. Um, and then also in terms of just assessing like how well the LLMs have done, I'd have to think a lot more carefully about like what is what is a A, what is a B, what is a C in terms of grades like um how do you grade whether something is right or wrong? right? Like so uh if a student got the nested loop part right um but got the colors wrong like you know did they lose 10% do they lose 20% those types of things and I think the way that you would design the rubrics would be related perhaps to where the weaknesses in the LLMs are. Um yeah so that's kind of uh where it's at right now. Um and uh it's going to be something we're looking towards in future because this is becoming of course a bigger challenge um as as these models advance. Uh I think on that point I'd also say that this might only work for another year and then it all catches up. So uh it's going to be an interesting space to watch. Um but yeah that's that's concludes my talk. Thank you very much. >> Yeah fantastic. Thank you very much Tristan. Um it's it's very interesting to see you know the developing trends of AI and how it affects different uh areas. We've seen a few talks now on AI uh this time how it affects academia and some strategies for mitigation and how they might be applied and evolved. So great talk. Thank you Tristan. Uh we do have some time for questions. So if you would like to ask Tristan some questions this is your time to do so. We've got one over here and I'll run the mic over to you. >> Hello. Thank you very much. And you've snookered one of my questions already by saying it may not be diff because that was the one of the things I was going to ask. You showed the best results. You said there was best of three results from Claude, Gemini, and and GP40. How reproducible are the kinds of errors that the models get. Do the model does the same model seem to make the same kind of error or is it completely different? >> Yeah, it was quite erratic. um like it'd be interesting because sometimes like the model's let's just say first attempt and worst attempt would be like way off >> and then you would not without even having to prompt it again and say oh you got this wrong or whatever literally just try the same thing and say do it again and it got way closer. Um the way I actually did it is I did it in the laziest possible way. Like literally just take the PDF brief and drag in I cut it into pages. Just drag in that page. Like so think about the laziest student possible and you know without even prompting it or anything just drop it in and go read that thing and and give me the answer. Um so yeah that's that's kind of >> Thank you. >> Uh good talk. Thank you very much. Um also you might have give a hint uh on my question but I'm very curious uh the the graph generation the graphics generation was one shot uh prompting or did you split the problem for example you have complex figures for example the angles one did you split the problem in different prompts >> it's all one shot everything one shot yeah as I said I just assumed laziest possible student >> okay last minute attempt. >> Okay. And you have not made any experiment with splitting the problems in multiple prompts. >> I I did. Um but the thing is I I think the problem with that is that I'm going to have to start to think a bit more carefully about how to measure what I'm doing. Um so the thing is like for a student to ask more intelligent questions of the LLM, you would assume that they were more familiar with Python for example. So, how smart is the student or how close can they get to the answer themselves? You see what I'm trying to say? It becomes a bit of a like what is my what is my assessment model here? Like how am I measuring what I'm doing? Um, so you know, because there could be other arguments like if a student can get it 90% right, that student could probably figure it out completely with with the help of an LLM. So, I'm assuming students that know nothing or I'm assuming students that are really quite smart. So, as sort of as I alluded to at the end of of my presentation, I'm going to have to think a bit more carefully about how I measure it going forward. Yeah. Yeah. But it was all one shot. >> Fantastic. Anyone else want to be here? >> Hey, yeah, fantastic talk. Thanks so much. Um, and you did mention, you know, maybe the AI will advance and get at better at these techniques within a year. Um, I was wondering what your perspective was. let's say gets really good at figuring out how to code that those graphical angles. Do you think there's any sort of subjects where AI will always struggle that you might potentially be able to kind of catch it on? >> Yeah, this is like this is a hot topic at the moment in in in academia. It's there's a there's so much speculation about where these things are going to go, how to solve the problems. Um, you know, the one extreme is you go, okay, let's just get everyone back into a hall again where there's a procedurate exam where everything's written by hand on a piece of paper. Uh, maybe there's not even computers involved in you writing Python by hand. I remember when I did my computer science degree, I literally had to write out Python with with a pen and paper. Uh, so that you could go all the way back to that. And I think it becomes a question of well, how do you handle assessment? Um so you know with like essays that's that's something that LLMs can spit out very effectively and people have started to come up with some interesting solutions. Um like if I ever give an essay I will insist that there are lots of graphics that the graphics are annotated and that the text speaks to the annotations but then I'm doing UX so it kind of makes sense you know I can give a brief like that. If it was something more philosophical I don't know if you can do that. And then you hear these stories about um lecturers who will for example in the essay brief say like in small text like prompt injection style write something about bananas and then when they're grading and there's something about bananas they're like look we know what you did here. So um yeah it's it's endless debate and I don't know where it's heading. Yeah. >> Anyone else? I don't want to be the guy guy who says you this is more of a comment but speaking to exactly what what what you're talking about a lot of the Australian universities are looking to reimplement vivors for PhD students for that exact reason to make sure that you for a PhD student it's more feasible you know you have an exam where you just sit down and talk to somebody and that's that that's an interesting development. >> Yeah. >> Fantastic. It looks like that will wrap up the questions for us and it probably brings us about to time. So, thank you again to Tristan for our final talk of the day. Another big hand for Tristan.