From Panic to Pedagogy: Using GenAI to Build Cases in Health Professions Education

Thank you for joining us for this seminar today from Panic Pedagogy using GenAI to build cases in health professions education. Just like to acknowledge that we are just off and work on the ancestral and unceded territories of the Musqueam peoples. I am John Cheng. I use he/him pronouns. I'm a learning designer with the Center for Teaching Learning and Technology and support courses and learning designs across faculties and programs like the UBC Health Integrated Curriculum and UBC 2324's Indigenous Cultural Safety Programs. I'm excited to be delivering this seminar in collaboration with Dr. Jason Min, Associate Professor of Teaching from the Faculty of Pharmaceutical Sciences, Carrie Kuczkowski, Assistant Director for UBC Health, Assistant Director Education for UBC Health, and Michelle Hamilton, Education Program Manager at UBC Health as well. Dr. Jason Min has three primary teaching roles at UBC, which are course coordinator for Pharm 161, Pharm 261, and Pharm 323. He's also Medication Management Co-Lead in Pharm 311, the psychiatry module, and the third role is as a curriculum co-lead for professional education. Jason's educational scholarship is rooted in the PharmD curriculum in the domains of decolonizing and indigenizing curriculum, interprofessional education, and in health informatics. His research and educational collaborations have formed over the past decade grounded in respectful, ethical, and community-based methodologies. Jason's current focus of scholarly research is on the delivery, impact, and assessment of pharmaceutical care with BC First Nations. Carrie Krickoski is um uh leads the planning and implementation of integrated interprofessional classroom and practice education initiatives in collaboration with all faculty and community partners. She also serves as clinical assistant professor in the UBC Faculty of Dentistry teaching professionalism, communication, behavioral change, and patient-centered care. Carrie brings expertise in healthcare regulation and team-based practice supported by advanced studies in educational leadership and policy. Michelle Hamilton leads the delivery and quality improvement of of collaborative health education learning opportunities for students across the health disciplines. With over a decade of experience at UBC, she has played a central role in supporting and shaping a number of complex undergraduate and postgraduate health education programs. Through this TLEF collaboration, we're exploring how generative AI can extend the academic electronic health record or AEHR in two key ways. First, by developing a case generator that allows instructors to quickly create patient cases based on parameters like complexity or topic. And second, a patient chatbot that enables more interactive realistic information gathering experiences for students. You may already be familiar with the AEHR as a tool for simulating practice-based charting and interprofessional communication using fictional patient cases. So, by the end of uh this seminar, uh you'll be able to uh describe uh LLMs and how they can uh you be used to generate patient cases in health professions education. Uh we'll discuss some key challenges associated with the adoption including model proliferation, costs, hallucinations, reasoning limitations, and institutional sustainability. We'll review a couple of practical examples of AI-enabled case developments including a UBC health pilot model and then AI-enabled EHR tools. And we'll reflect on some strategies for integrating multiple AI tools, institutional resources, and sustainable health education and innovation. Uh we're recording today uh uh today's uh seminar, so feel free to turn off your cameras. Uh also be saving the chats if you have any questions uh and and resources that you are sharing today and we are sharing. Uh there will be an opportunity for a question-and-answer at the end of our session. So let's you know, when we consider why uh we might be using AI in the first place uh uh we're thinking about AI in terms because it's not just new, but because it can meaningfully support teaching and learning. We know that there's a value uh of artificial intelligence. It can help us design learning experiences that go beyond basic uh knowledge recall. It can support higher-order thinking uh like analysis, decision-making, clinical reasoning. It uh could allow us to align more closely with learning outcomes and professional competencies, especially in complex practice-based uh fields. And then another key area is is uh new forms of expression, for example, generating scenarios, simulating patient narratives, or creating multimodal materials more efficiently. That said, using AI requires intentional design. Um [snorts] And so, thinking about strategies that guide that work, we can think of using AI to help something to help generate something realistic or or to create context-rich case cases or scenarios. This is especially valuable in health education where authenticity and complexity matter. We can think of AI using it to promote higher-order thinking, creativity, and integrity. Where the goal is not to let AI do the thinking for students, but to design tasks that require interpretation, critique, and adaptation. This also means thinking carefully about academic integrity and how AI fits into that as well. AI can also be used as a tool for reflection. Where students can compare outputs, critique them, and reflect them reflect on their own reasoning. And finally, we want our learners and ourselves to develop a critical AI literacy that includes understanding limitations like bias, hallucinations, inaccuracies, and maybe when not to rely on AI. So, how ready are you in using AI to develop case-based teaching materials or teaching materials in general? We'd like to get a sense from you about how you are using or might use AI to develop case-based teaching materials. And so, using Zoom's annotation tool, which is that green pen that shows up >> [snorts] >> on your Zoom screen, we'd like you to indicate where you fit on this value line from not at all ready to being ready. Maybe you're already doing it. You can show this with a stamp like a star some of you already doing it a check mark and X or other ways. Wow. So it looks like some of you maybe ready and [clears throat] some of you maybe already doing things. Maybe maybe you don't know where to start. Amazing. And from here I I I'm going to share I'm going to pass this to Jason, Carrie and Michelle to see what insights they can offer. >> That's great. Thanks John. Um, Oh, you have to do you mind clearing the annotations? >> Yes, there you go. >> Thanks. Uh, so thanks thanks very much John. Um, and thanks everyone for joining over the lunch hour. I have a little bit of time to talk just a very brief overview of the AI landscape. I I think we're we've all probably heard quite a lot about it. So I just going to be a brief overview of some key key points and then we're going to get to the pedagogy piece and so we have two case examples and I'm happy to share the first one and then our colleagues from UBC Health can share about the second case example. Uh, so if you don't mind the next slide, please. So one of the challenges I think we have seen and this is not an AI thing. This is a health education thing and happy happy to let people stew on this a little bit but I think there are examples that we can all agree where practice has led education in some parts and in other times education has led practice and this is an interesting concept and I'm using the tortoise and hare metaphor here where you know the the race is not always to the swift is the quote that's part of the learning of this story, right? And AI feels almost like that. It feels like we're all racing to a finish line that we don't know exists. We don't know where that finish line is. And so it's kind of kind of an odd thing where we are all pushing towards this outcome that is quite vague at the moment and and I would argue both practice and education are both rushing into this and I don't know where we're going. So we'll talk a little bit more about this in the next couple slides John if you don't mind. And there's the quote. Next slide, please. So one of the things that I wanted to start by quickly addressing is this model proliferation and cost. And so this is maybe more of a practical thing that we've all probably considered. You know, the adoption and the use of AI is extremely difficult to standardize. There are so many different language models and AI models out there. And I would argue our typical way of going through a menu of options is to science our way through it and we cannot science our way through AI. There are language model comparators out there, but the language models change and evolve so rapidly. It it makes it really hard and so how can we respond in an environment where everything is moving faster than we can seemingly keep up with, right? And so multimodal AI integration is really quite common now. It requires you to know more than just your favorite model. And a lot of things are pushing multimodal use of AI languages. Um cost savings is one, reasoning strengths uh is another, and the ecosystems that they're built for. So you've probably heard of, you know, there are AI apps that are better for generating images or better for coding and things like that. And so we have all of these different tools that are trying to differentiate themselves. And And so it makes it hard for us to science through that and justify it. And so as an example, you know, if you send a query in an AI tool right now, that could include multiple models working in parallel. So that could be where in a tool such as something for case building, you could say, "Create me a case uh for, you know, senior-level health professional learners in Canada related to these disease these topics." And that that one app may actually have multiple AI language models in the background. Maybe one generating images and one generating text. You could also have cascading models where you have them working in sequence where you have perhaps a cheaper language model um doing a transcription, like an audio transcription that we see in AI scribes in practice. And that could be followed by a more premium cloud-based model that analyzes your transcription and then creates the output that you're wanting. And then the last example that you can look at, and we see this lots now, are multi-agent systems in one organization. And so you you can see now you have one agent one AI agent using multiple tools at the same time. So it's it's this proliferation where you have one agent using 10 tools, 10 tools are cascading and actually each of them use two or three or four language models all at once. And so you start to get this giant web of proliferation. Uh next slide, please, gentlemen. Hallucination and reasoning limits is something that's I would say most commonly talked about. There's some great headlines out there. I grabbed some of my favorites. Um you know, the one most recently, I think in Canada that's made some news. This is just not too long ago was that bottom left one um about how the auditor general found a significant number of hallucinations in the physician AI scribe domain. Um that's not a surprise, I would say. And you know, there's lots of other fun headlines you can grab and why AI is a terrible doctor. And you know, it wasn't that long ago where we were saying AI and Dr. Google is almost as good as everyone else. Uh next slide, please. So when we talk about reasoning limits and hallucination, uh you know, reasoning and and and actually I'll say, you know, from an education standpoint, I think this is always a critical piece because the quality has to be there, right? So we have to trust in the quality of the output that we're doing. And so how much is it really helping if it's hallucinating here and there? Uh so reasoning capabilities is absolutely evolving. Um it's it's moving from pattern recognition to, you know, breaking down complex tasks uh using different logic. And you know, we we're seeing more and more AI agents and that's probably coming very soon for um us and and other institutions. Um but of course, you know, the problem that that we have is AI outputs can be wrong, but they can still look fantastic. Like they can look plausible, they sound authoritative, they have references. Um, and you know, we have some mitigating options for that. We you can train agents, you can use different data structures like rag, um, or other sort of data layers to help uh, ensure that there's um, uh, uh, provenance in the data that you're using. Um, and ultimately, you know, AI does require strong data provenance. It requires oversight, and it requires management. And one of the challenges, and this is something we could talk more about later if there's time, um, but one of the challenges is this rush to get into the AI sphere has led us to not really doing a good job anchoring our data in authoritative uh, sources of information. Uh, and so that that's the provenance uh, issue that we're experiencing. And so I I think there are some really interesting opportunities as researchers in health to look at that. Um, Fraser Health is one example that has been working with a local AI firm to build extremely robust and comprehensive uh, rag data sets uh, to try to achieve better accuracy. Um, but also AI agents are really smart. So that bottom left uh, uh, screen grab that I have there is AI agents are smart enough to click through the verification test that they are not a robot, right? And so it's uh, it's it's almost this sort of um, again race to a finish line that we don't know. Uh, next slide, please, John. Uh, so the last thing I want to share regarding AI landscape, and again, I know I'm going through some of these key points really quickly, um but is around institutional sustainability. And I I think we can maybe we we don't agree, but I I would like to think that we can all agree that the AI adoption for us on mass requires system-level infrastructure. I think it has been really encouraging to hear uh you know, about LTIC and all the um uh commitments made from the provost office that that you've probably heard about. And so, all of that is really encouraging and there's really good direction happening. And you know, we need to have the right conditions that will bring confidence for us as end users, as instructors, to ensure that there is a governance, there is oversight, that there is costs, boundaries, and mitigation, and ways that we can align what we do with AI and AI agents with the academic policies that already exist. And so, I think there are a few things that make this hard for us. Um I think like I said, rapid evolution of models makes it really challenging to standardize things. I think anybody who's doing AI pedagogy and scholarship is really swimming in a pool of extreme heterogeneity. It It's I don't know how we generalize findings when we look at scholarship using AI. I think it it's extremely difficult. Um and what we hope to do again using a sort of science framework, and we want to standardize approved models and standardized ways we can do this. The problem is those standardized models become quickly outdated, and that undermines the standardization itself because you use an outdated model, it's not as good as this one that you can use. And so, you jump to using that one. So, there's lots of quite interesting things that we can look at at the institutional level. So, that's the quick flyby of the AI landscape. And I know I've dropped a on a bunch of these topics, but I I want you to think about these as we go into some of the examples we have and take note of maybe how we've addressed some of these. So John, if you can go to the next slide, please. So the first case example that I'm pleased to present on is the AI in the AEHR project. I've got the QR code there. There's a link to our Moodle demo site if you're interested. I will show you a quick demo anyways for those that haven't seen it before. But I'll start with a quick background on the AEHR if you haven't seen it before. Next slide, please John. So the AEHR is a vendor agnostic teaching tool. It simulates real world real world electronic health records or electronic medical records. Obviously for those of us in health, we know that there's you know near 100% adoption of digital charts at least on the primary care side in the institutional side. You know, there's there's lots of lots of progress being made over the past decade, but we know that commercial products are also very ill-suited for teaching. You know, vendors will provide sandboxes for us to use like a Cerner sandbox or a Meditech 2 sandbox and they're not they're not great. They're not designed for us as teachers. And it makes it really difficult. Plus they're also inconsistently available. And so the AEHR really came out of that challenge and it was a pilot with a number of UBC representatives here that were part of that pilot. We got some funding from the ministry about 10 years ago. When that pilot ended and the funding ran out, it was adopted by the Association of Faculties of Pharmacy of Canada or AFPC. That was back in 2020 and since then AFPC has invested a significant amount of money and also been successful receiving some federal funding for it as well. Uh but essentially the AEHR is kind of what you'd expect. It's a tool for any clinical or patient-oriented activity. Very commonly used in labs, in tutorials, in case-based learning. We have almost 5,000 users across Canada. Uh pretty much every pharmacy program in the country is using it. Also several other medical and nursing programs, especially in IP environments, are using it. Uh so there's lots of really um neat things about it and I'm going to show you in a second, but it is locally hosted at UBC and it is embedded into Canvas as an external tool. Uh next slide, please, John. Uh so our project was called building deep AI integrated pedagogy using the AEHR and we really wanted to build AI tools in the AEHR to help uh us teach some of this stuff. And so this project, uh as John had mentioned earlier, we're building uh three tools. We're building an AI case builder and transcriber for instructors. So think about the immense amount of time you spend building cases for students, especially multi-disciplinary cases, uh can be very time-consuming. Number two, we're building a patient simulation chatbot. Everybody has chatbots, chatbots for everyone, so we're we we want one, too. Uh and then the third item is we're building an AI scribe for students and we have a series of evaluation planned um supported uh by CTL. So I'm going to show you a quick uh very quick demo. I'll just click through it and so maybe John I will uh share my screen if that's okay. I'm just going to show you the student side of things. So this is a very familiar site for most of us. This is a Canvas assignment and I'm in student mode. And so if this was an assignment that you were launching for your students, they would read your instructions here. It tells me what to do, and the students would simply click on it, and it's single sign-on. They're directly in the app. You can see that the navigation is on the left-hand side, and you can see I'm on the demographics, but there's all the usual things that you would expect in a medical chart. Um There's med orders. So, I know I'm going through this very quickly. The The purpose of this is just to provide a little bit of context so you can see. But, you can see that you can add, um you know, consults and things like that, and these are PDFs that you can upload. And so, you can really imagine designing a case or case-based learning in simulated chart rather than using a PDF document. And again, there's There's a demo there, and let me know if you want access to it. Uh so, when we were building the case builder, and my animations are not working, um you can see I've cascaded several screenshots. I was intending for them to show up one at a time, but nevertheless, that's okay. So, you can kind of see from the uh screenshot in the left that's furthest at the back, uh there are definitely tools out there. If you haven't already looked, you can Google them. There are case-building AI tools already available. They're not cheap, um and in my opinion, none of them really fit uh the context for us. Um So, the one screen grab I have there, it's uh I really like the user interface of it. It's a nursing-focused one, and um you know, it it generally walks you through the steps that I think us as educators normally walk through when we create a case. That led us to create a napkin sketch of what we wanted, which thankfully is partially covered because that's my horrendous writing. And what we have in the app so far uh is what you see prominently on the screen. And so, it it goes, scrolls much lower down, but the idea is you can see at the top, uh, you know, we have um, our mindset towards cost. So, we have total AI usage. Uh, it's five requests has cost us about 60 cents. You can see that there's different modes that we're building in. So, whether you want something quality, so a frontier high-end language model versus something that's fast where you've got mixed models or a cascading model. There's lots of different options that we're looking at and we'll be evaluating for effectiveness. Um, but then you can see this is where you can start to build your case. And you can see on the right-hand side there's prompts and those will eventually be hidden from users, but that's part of our evaluation work to see what kind of prompting gives us the best outcome. Uh, next slide, please, John. So, this is just more information about the case builder we have. Um, we are using Opus 4.6. That is a very expensive model. Um, you get what you pay for is what I'll say. And so, we tested various different models and they were not that great. So, who's going to use a case builder that's not that great? And uh, we wanted something that people can use that we can talk about cost later was the idea. Uh, we do, as I showed you, we do have um, a few different options for model setup. We have cascading and multimodal models built in. Uh, we originally wanted to develop a rag as part of that uh, system setup. We haven't done that yet. Um, part of our reasoning is Claude or Opus 4.6 does a very good job of retaining context and reasoning. Um, and you know, from a scholarly perspective, from a faculty perspective, we finally have a tool, the AEHR, that can map and build and proliferate versions and iterate quickly and track progress and scale longitudinal patient cases very quickly and rapidly that I would argue historically with cases we haven't been able to do. Okay, I'm going to quickly go through the last two items. We have a patient chatbot that I said we're building. We are currently waiting for everyone else to build amazing chatbots before we build ours. So, if you have an amazing chatbot, we would love to learn from you. So, we haven't started that development at all. And then the last tool that I have here, John, if you can go to the next slide, please, is our AI scribe. AI scribes have, you know, you saw the news headline that I grabbed, you know, it's it's come under fire a little bit recently, but AI scribes essentially, you know, poses a new frontier, I would say, of pedagogy for us. And I know things like Dragon Dictation and other dictation services have existed before in the medical world, but nothing quite like what we have with AI scribes. And so, we are just building out the model setup now. These are our sort of inspirational user design that we're going to be drawing on to create our user design. And it will likely include multimodal as well for for processing and inferencing. I will pass it over to John. Sorry, just in the interest of time, I'll pass it over to John. >> All right. Um, so, this was at the end of your section, but maybe we can share it at the end of everyone's section, Jason. Is that this is something you wanted to go over now, the value line? >> Yeah, so I guess I went through quite quickly. Um hopefully it's been able to push maybe some of that needle towards your readiness um and hopefully draw some excitement towards using AI tools to develop uh teaching and case-based materials for you. So I I don't know if we need to do the uh screen annotation, but I think uh it'd be exciting to move on to our second case example. >> Okay. Um so UBC Health has been building an original activity which we piloted last spring and we're scaling up this fall. So it features the uh a chart tool that we just saw and also a patient case, Marie Parker. And we've learned that building high-quality interprofessional cases is really complex and time-intensive work, especially in interprofessional education, when multiple professions need to align on our roles, perspectives, and patient care decisions. Uh so that in this pilot, we explored whether AI could support that process by helping us to create a more dynamic and interactive learning experience. So using a conversational AI patient integrated with the academic electronic health record, students engaged with evolving patient information and they did this while working collaboratively as an interprofessional team. So we can move to the next slide, please, John. So building on that idea, our pilot examines whether AI can support more authentic interprofessional learning by simulating dynamic patient context, so not just delivering a static case. So traditional paper cases are limited in realism and they often can't reflect the uncertainty, the evolving narratives, and perspective taking that's required in collaborative care. At the same time, we really want students to have a psychologically safe environment where they can practice communication, decision-making, and collaborative problem-solving uh before they're entering into real clinical settings. So, what makes our approach with this activity different is the integration of three components together. The conversational AI patient simulation, having structured clinical information available in the academic uh electronic health record tool, and then putting it together with shared interprofessional decision-making in the activity. So, our goal is really to explore whether these tools can create a more engaging, realistic, and safe learning experience for our students to practice in uh in their interprofessional education teams. >> Michelle, I'll jump in for a second here. Um yeah, just to talk a little bit about some of our drivers for pursuing this sort of line is um as uh Jason mentioned that creating the cases there are our opportunities to create cases with AI out there. Um but we found there's, you know, some hallucinations and in generating cases, we've traditionally taken a very long time to do that, particularly interprofessional cases where it requires bringing together groups of experts over time to kind of ratify a case and make sure the authenticity is there. Um so, this is a real opportunity to kind of speed up that process. Plus, we have uh regional sites where learners are situated. They're not always co-located. And so, this gives us an opportunity to bring in um groups of learners from different disciplines across regional sites. So, that uh and asynchronously and synchronously. So, I think that gives a little bit of the background context. Over to you, Michelle. >> Okay. Um so, to our next slide. Um so, we're going to show you a little bit about the patient chatbot uh part of the activity. So, here our learners can interact with the patient, Marie, through a web-based chatbot interface uh that simulates having a conversation with her. So, students can ask questions, explore her lived experience, and better understand the context affecting her health and daily life. So, the AI chatbot doesn't just generate symptoms, but it also has evolving responses that are connected to the social determinants of health. So, things like housing, caregiving responsibilities, access to support, and returning to work. Students can have conversations about that with Marie. And a key part of our design with the patient chatbot is that it's not unrestricted AI exploration. So, we intentionally built in some prompt constraints and embedded cues to guide the learning better. And we wanted to um have this activity reinforce interprofessional perspectives and also align with our learning objectives for the activity as well. So, the activity's integration with the academic electronic health record also helps students to translate the patient narratives into clinical action. So, they can um use that health record to help them to think of the questions and have conversations, and then develop that more. Um so, what we found is really noteworthy here is that the educational design really matters more than the AI model itself. It's really about how we've structured it and structured the improvisation um to have some clear boundaries with the learning activity as well. Uh so, you can move to the next slide. Um so, we want to show you a brief demonstration of the Marie Parker chatbot from a student perspective. So, we had uh one of our students record a short video. Um and just to let you know, to encourage um meaningful interprofessional collaboration in our activity, the students were intentionally limited to only ask three questions of Marie. And the design there is that they could then take their insights back and um converse about them with their team members and really build a more complete understanding from having um multiple different disciplinary perspectives. Um so, I'll turn it over you, John, to show that demonstration and talk through it. >> All right. >> Hi everyone. My name is Iman. I'm a kinesiology student and today I'll be interacting with Marie Parker using AI and gathering a little bit more information about her and her back pain. >> So, Iman typed, "Hi Marie, in regards to your back pain, in which activities is it particularly aggravated?" It's a little small on this video. Marie responds, "It's definitely worse when I'm lifting, bending, or twisting, especially when with Josh's car seat. Sitting for more than an hour at work also makes it really achy by the end of the day. Carrying Josh has gotten harder too, now that he's bigger." Then Iman types, "And in terms of work, if you're mostly sitting, are there opportunities to take movement breaks or schedule enjoyable movement that does not activate your back pain?" Marie Marie responds, "Not really. My desk setup is pretty bad. The desk is too high and my chair doesn't have much support or even adjust properly. I probably could try getting up more often or walking a bit during breaks, but by the end of the day, I'm usually exhausted and sore already." Iman then types, "And are there activities that you enjoy or you used to enjoy that you think you could do with your son. For example, a new mom social group with some physical activity like walking in the park. Vancouver is quite good for that. And then Marie responds, Yeah, I used to really enjoy hiking and going on walks. >> overall, I got some information about Marie and something that she might enjoy uh going forward. Uh so, yeah. Thank you. >> And I'm not able to Oh. Uh yeah, I used to go really enjoy uh hiking and going on walks with Ryan before the pregnancy. I think walking with Josh in the stroller could be manageable and my back if my back isn't too bad at that day. Uh and my mom's a mom's group actually sounds kind of nice. I don't really know anyone here yet, so it might be good to meet other new moms. >> So, that gave you uh a little bit of a demonstration of that piece of the activity in the AHRQ. Did you have anything to add to that? >> I think the um the consistency is notable. Uh John is very I guess uh understating his skills. But, the I think the AI bot has been reproducing very few kind of hallucinations, so good work, John, on creating that framework. But, we of course we always have more to learn. So. >> So, we'll move on now to share some of the key insights and lessons learned from our pilot evaluation of the activity last spring, which had 21 participants. So, our student feedback was gathered in late June and early July through two small focus groups, uh one interview, as well as a student survey. So, help to help us to organize these findings for you, we used a SWOT framework to reflect on the strengths, limitations, opportunities, and challenges that we had with integrating AI into interprofessional education. So, we can move to the strengths. Uh so, one of the strengths of the approach is that the learning experience is grounded in authentic clinical workflow. So, the academic electronic health record and the Canvas course environment acted as anchors for this activity, helping students to move beyond having conversations into documentation, care planning, team communication, and decision-making activities. The student The students worked through uh milestone due dates. They contributed discipline-specific perspectives and responded to evolving information in ways that could help them to more closely mirror real collaborative practice. And the variability in the AI responses also introduced a level of uncertainty that reflected real clinical teamwork, where different professionals might interpret uh interpret information differently and then need to negotiate shared understanding together. So, it gave them opportunities to do that a little better. Uh one of the key lessons from the pilot that we learned is that realism alone uh doesn't guarantee better learning. In many ways, it actually raised the critical question for us, does AI enhance learning? Could it hinder learning? Could it do both? So, we found that AI can absolutely increase engagement and realism and patient context, but not without having thoughtful design uh included. Uh it could potentially introduce cognitive overload and unnecessary ambiguity. Uh so, what we needed to do was really offer enough guidance and scaffolding to support the learning effectively in the activity, and we're still working on that as we're rolling it out for this fall. So, we identified some practical considerations and limitations, as well, with using AI. Um that included the platform access requirements, logging into the AI, accessibility needs for students with the activity, and then the the necessity of offering alternate learning options due to institutional policies. So, although the chatbot did give us a stronger sense of patient voice and context, we also know it can't fully replicate the complexity and nuance of real patient interactions. So, it's it's still a tool. So, it really reinforced to us that AI works best as a thoughtfully designed support tool that can complement facilitation, reflection, and other collaborative learning design. Uh then we thought about as a team discuss some of the opportunities for the future based on our feedback from our activity. Uh so, we Uh for example, we'd like to see how AI could gradually release new information over time to increase case complexity and better reflect real clinical practice. It could also provide more relational patient feedback to the students responding to how the students are communicating with the patients, how they are aren't demonstrating empathy, or how they're working as a team. And because cases are dynamic, AI could also give us the opportunity to explore different what-if scenarios, and perhaps consider how different team decisions could influence patient outcomes in different ways. And then moving on to the threats, our discussions highlighted several important considerations as we continue to use AI tools as they continue to evolve. So, this includes protecting student student privacy, recognizing limitations and occasional inconsistencies with the large language models, and also navigating the rapidly changing AI landscape in education. So, I'm going to pass it over to John to speak a little bit more about these and his experience with the chatbot development. >> Thanks, Michelle. Yeah, I was thinking in terms of threats and challenges, a few to consider are considerations over definitely over hallucinations and biases. This is This was built maybe in late 2024. Um and so in testing this particular example, the chatbot was I When I was developing the chatbot, it was making up information that wasn't guided by the prompt or the information defined in the scope or the the context documents that we were uh providing the the agent. For instance, it started speaking to me in Spanish and creating a whole backstory for Marie until I provided prompts that limited its ability to interact based on the scope that I provided. So it was making up information that I we had not inputted. Um another consideration is like student access and and particularly uh with the choice of using particular plat platforms and maybe thinking about noting concerns over using maybe um ChatGPT or their access to ChatGPT. Uh the requirements for students to be setting up accounts uh using particular agents. Um and so uh providing students with alternatives uh if they they chose not to use uh ChatGPT for instance was really uh key consideration um in building uh a chatbot activity alternative in this in this in the AHR um curriculum. Was there anything else you wanted to add Michelle or Carrie about thinking about some of the threats and opportunities and things as we were building this? >> I was just um typing something in the chat actually. So, I think it's worth saying that an AI bot doesn't replace simulation with real or standardized patients. However, we all use case-based scenarios in health professions education and it does provide that additional dimension as well as the engagement piece. So, I think that's just wanted to say that because I'm sure there's some folks out in the in the audience that are thinking, "What about the real folks?" But, back to you, John. >> Sure. >> Sure. So, we can just summarize a few of the key lessons that we learned from our pilot for you. First, we found that having the structured asynchronous learning works really well when the students were given clear expectations and guidance and scaffolding to support their self-facilitation of these types of activities. Second, we learned that offering multiple ways for students to engage through discussion, reflection, visuals, and interaction helps support diverse learners and it lined really well with the universal design for learning principles. And AI definitely supported that in this case. We also saw the value of creating safe collaborative learning spaces where structured teamwork and reflection could be used to foster team accountability and communication and deeper interprofessional learning. And finally, we really feel that this work opens the doors to future innovation showing us how we can layer technologies like AI chatbots, the academic electronic health record, and our learning management systems to combine these together to create more adaptable and customizable learning experiences for students. So, we're excited to see where that can go in the future. Anything to add, Carrie? >> Nope, great job, Michelle. Thanks. So, back to you, John. >> It's the show. So, we have about 10 minutes left and a few questions for you to reflect on. We I like to offer you an opportunity to share your curiosities about AI, the HR, building teaching materials with AI, such as cases. Um what has inspired you to do to do differently in building cases for teaching and and learning contexts in your own teaching and learning contexts. What possibilities do you see? What have we not thought about yet in in this TLAF project? Um what may be some of your barriers to implementation? Um and if this is, you know, part one of our TLAF initiative, what would part two look like for you if you could envision that? And as we are closing up, um thinking about that that value line of readiness as we shared it throughout today's seminar, um you don't necessarily need to annotate this um if you don't want to, but um we're just curious and for your own curiosity and your your own reflection, um how ready do you feel how excited are you to use AI to develop case-based teaching materials? And, you know, I I see I see a you know, generally the same amount of readiness, maybe a little bit more ready. Um which is exciting. And I wanted to and today share our contacts, some stars. Um, feel free to reach out to any of us today. Um, this could be part one of uh maybe a second seminar sometime in the future when materiality um begins to unfold and evolve. Um, so we're really looking forward to following up with all of you um at some point in the future. And I'd like to thank Jason, uh Michelle, and Carrie for this collaboration today. It's really exciting to work with you.