AI for You Podcast | Episode 14: The Syllabus Broke, the Students Didn't

[music] >> All right, welcome back to another episode [music] of AI for you. I'm Philip Mayman and joined by Dr. Jay Tao. We're both professors of analytics here at Fairfield Dolan and Jay's also the director of the AI and tech Institute. AI for you, we've had a lot of guests over our podcasts from industry and academia. We've had students, we've had faculty, we've had administrators. And you're probably wondering behind the scenes there's a lot of interesting stuff we've talked about, but we what we have not had is two brand new fresh recent graduates of our master's program and now what's now called the MS in business analytics and AI. Uh so it's a very exciting opportunity to talk to them. Especially you know, the program itself has a lot of cutting-edge stuff. We've talked about that and what AI is and what's important and what the future is holding. But there's also uh what what AI brings is flexibility. And one of the beautiful aspects of flexibility is Jay's new course that he offered as an experiment. Uh and that's kind of the whole point of this entire podcast is we AI is like this new genie that comes along and grants wishes, right? So how do we know how to deal with it? We don't. We have to experiment. And one of the magnificent experiments that Jay will talk about here today uh is a course he ran uh in the spring semester just finished um with some pretty incredible results. Jay, tell us about the the course and the idea behind it. >> Thank you, Phil. Um I think, you know, when when I look at AI we can I think we have, you know, two ways to look at this. One way is the very technical way to, you know, basically decode the the algorithms and and figure out, you know, how to set up the models. The other way is a very application-heavy way of, you know, how they use um um GPT image to generate image, that that kind of stuff. And and that's typically what you get from these online resources, right? Either somebody going to lecture you 2 hours about how to set up this model or somebody going to show you in 5 minutes how to generate, you know, um headshots for LinkedIn. And I don't I think both are great, but the I think our students need a little bit more um for the time they invest in the classroom. So, I keep asking myself, what is the thing that everybody still need um with in in the age of AI? And and my conclusion might not be the correct one is basically we're still doing whatever we do. And uh how do we actually infuse AI into whatever we do to me is very important. So, I think it's guided by two principles. One is I don't really I mean, I for one don't really want to change whatever I do because of AI. >> Wait, wait, what do you mean by that? >> Meaning I'm not learning all all scratching off my, you know, how I do things because of a new tool. I'm still doing things I do my way. AI is here to help. It's not the other way around. >> Okay, okay. >> And um the other thing I think I I really want to achieve throughout this course is basically um I had some experiences or lessons learned in how to actually automate my own workflows, my own way of doing things. And I want to pass those along to our students, actually to to a broader audience if we can. That, you know, this thing at least these things worked for me. Right? And and I think, you know, other people should try. The >> Awesome. Well, introduce our wonderful students. >> Sure. Um, well, we have Magda and Sheila here. >> Um >> So, in the course we want to make it as real as possible. So, we divide students into groups and for every group we actually have a project manager to basically manage the the >> [snorts] >> the the the project we're going to deliver. Actually, the other part of making the project real is we have a real client. I'll let the students talk about the client a little bit. Um, so basically they are like real um product heroes. They are um in charge of, you know, managing the projects. And, you know, coordinated with their colleagues and also coordinate with me and the client. So, they did a lot of awesome work. >> So, would you think of it more as like a research lab or a hands-on mini consulting environment? >> I would have think it's a it's actually a student body consultancy. And that's, you know, because even with research, you know, um my own academic research, I think, you know, you have to tie it to practice. And I think um not not to mention, you know, the courses we're teaching, we are tied to practice to reality as much as we can. And you know, like I said, having a real client and everything. >> But, one question before we dig into the specifics. Um, how did overall the class go from your perspective, from what you had expected ahead of time to what actually transpired? Because it is an experiment, right? We never know what's going to happen. >> I would say um it it way beyond my expectation. For for for two reasons. For one, you know, these students are awesome. You'll you'll hear from them, you know, what they did is actually truly impressive and not only I say that, the client said that, you know, that So, that's that's really good. And and I don't know how much client-facing experience they have, but clearly they showed they can handle that. And that's what we want, you know, to train them to ready for for the real world. And the other thing I think is impressive is in my not so short, but not so long teaching tenure, this is the first time after change your course this master. >> Ah. >> That's I don't know if you have done that before. It is actually a lot of work much more than I thought. And we have to work together because, you know, we have to um adjust to what the the client wants. So, um I guess, you know, I hope I didn't mess up with their schedule, but uh I would have say, you know, it it all turned out to be a good a successful experiment. >> Nice. >> Yeah. >> So, Sheila, Margarita, tell us from your perspective, what what were you expecting going into the course and how did it end up? >> I think uh I was expecting to learn a little bit about AI automation workflows, but we didn't have a very clear picture ahead of it because there was no previous edition of the course, so there wasn't a previous syllabus or anything that we could like look at like there is for other courses. Um and then it sort of started out like that in we were discussing personal projects, like you were saying, you do things still your own way just supported by AI. So, we started out by picking an own personal project that would benefit from automation. >> What was yours? >> In our case, it was um automating LinkedIn reaching out to people on LinkedIn. >> Awesome. >> If you're looking for maybe um networking uh with someone at a company that you're interested in, finding things in common, like looking for maybe past similar experiences or the same university, and then it would research that person's profile and draft a message for you to send. >> Nice, but it wouldn't send automatically? >> No, it wouldn't. We We wanted like we learned from the beginning that hallucination is a feature, not a bug. So, we're we're going to look at that message before it goes. Um and then like Dr. Tao was saying, we changed halfway to focus really on the client project. So, my initial thought, just to answer the question, was we would focus on the personal project, and then it was suddenly a really nice twist to have like a consulting project essentially. Yes. Uh I had some experience as a consultant before, so it was nice to relive that whole world and be a project manager for a little bit again. But, it was really good to be in the classroom, but also have that industry contact and have a real project at the end. So, that wasn't what I expected, but it was better than what I was expecting. >> Yeah, I feel like I went into it honestly having no idea what to expect. I mean, I had heard that about you as a professor and how you make us put in the work, but it's worth it. That is exactly what people who had you previously told me. And while that part was true, I definitely was not expecting to have like an actual client. That was not something that was disclo- You probably had no idea when you were even advertising the course, but we definitely had no idea that that was going to be any part of the course. We were told about the personal project. Mine was about cover letters because cuz we were all in like groups of three. >> Mhm. >> In my group we were all like the same age range, which is younger compared to most of our peers. So we were all like working on the job hunt, so we did something relating to that. And you helped us come up with the idea cuz we were like kind of thinking resumes, but then you had the great idea of actually having it like write your cover letter for you by having the AI like do the company research and look at your past writing and look at your resume and pick out those buzzwords. I mean obviously again hallucination's a big part of it. There was a lot of trial and error and a lot of human review needed in that. But it was definitely really cool to put together and definitely not what I was expecting, but totally awesome to do. >> Do you guys still use those tools, the cover letter and the LinkedIn outreach? >> So halfway through the course we had to sort of put it in the back burner because we were so focused on the synchrony project, like the client project. Uh actually I've just revisited it this week to pick it back up and the amount of things I've learned even from when we stopped or when we put it on hold, I should say, to now with the project, it made me look at it completely different. So now I'm going back We might talk about this in more detail uh after, but we're doing spec driven development, so we were specifying all of the requirements ahead of any code or anything that the LLM would help us produce. So I decided to revisit this skill just now with that mindset of spec driven development first that we weren't using at the beginning of the course or not as much I would say, not as formally. And so I just picked it back up this week actually. >> Interesting. So we'll have to dive into that. >> I've thought about it. I haven't actually did dove back into it yet, but I've definitely it's been on my mind like oh I should really pick this back up. Like I'm in the middle of that process and it would be really helpful to me. Though you might be inspiring me to actually bite the bullet and pick it up. >> And I think you know what I heard and and thank you for the kind words and what I heard is actually unexpected. I I think you know I want to say a few things about this. The one is this is the last thing a professor want is basically that your students tell you you know you basically give us something unexpected. Um that because the expectation is basically we have a contract. This is what we do in a semester. You don't want to change the contract changed you know. But I and and I think I also want to apologize not only to you two but also to all my students you know the added work because of the kind project. Um but like I said that's also that that's not really part of the design. I was actually surprised by that as well. But that's how we work in the real world. Things happen. And and I want to say at least you know it doesn't matter um um how much you hated that you know in flight I think the result turned out to be good. And uh I and I think you know this would help you um say if you're looking for a job and everything you know you actually have a Fortune 100 company on your resume with a real project and you can actually ask them for reference. Um I think from my standing point I I think it's worth it. Of course, you know, um it's it's for you guys to decide, you know, if it's worth it or not. >> So, Jay, tell us more about this client. So, it's Synchrony Financial, which one of the top uh firms in the world. We've had a great partnership with them with our department and our school for 8 to 10 years. Right? They do a co-op program, which is unique in the world. Um tell us how it transpired with you. >> Okay, so it's actually the uh uh interesting story because um um my um contact at Synchrony, um um Abby, we're going to interview him in in a later episode. And you know, we got to know each other because he's very into AI. And and as a marketing professional, he's very into AI. And so, we started meeting in a in a small coffee shop and start talking about what we can do with AI. And the conversation got into uh how can we help Synchrony with with AI? I'm sure in the in the conversation with him, we'll talk more about um basically how to adopt AI in a heavily regulated, you know, field because Synchrony Financial, they're in finance, they're heavily regulated. There are a lot of things um they cannot do with AI. So, um the So, the thought process um started from, you know, A um what you know, from all the way from, you know, um how do we build this great AI for Synchrony to how do we actually pick tools that support what Synchrony does, which actually perfectly align with the design of the course is basically using AI to support existing workflows. And since they are in a heavily regulated domain, we cannot really touch their core businesses. And and the other alignments so is actually in that is you know, our students are in the analytics program, the graduate program, and we Luckily, we uh basically connect with their analytics function. So, all three projects, these these two going to tell you, is actually from their analytics function. It's actually support their analytics function. So, um which I think is a great strategy is, you know, um these are important workflows because, you know, they deal with a lot of that. Everybody does. And and this are high value and transferable skills that doesn't matter if you work with a Synchrony or or somebody else, you know, they all have some kind of analytics function and you can reuse these skills. And and and And the last, you know, but but not the least is um you know, um we're we're helping the client to reshape a important but not core workflow, meaning given this is experiment, even if it didn't pan out, it won't hurt their core business. So, I think, you know, everything li- lying out to be well. Okay. So, uh What put put us in the same stage that they were at when the project came in or the client came in? Was the project already set or did they have to decide it? Was it the same across people? Just put us in the same framework that they were when they first started the client side work. So, um That That's That's a very interesting question because when the course started, like they told you, we don't really have a client. >> Right, because >> um Abby and and I were still working on how to actually iron this out to be um projects. So, we are We are actively We were actively searching for actually a business function that would need help. And And that's where we found their the their analytics function. So, and then, you know, um I I was thinking, you know, even though some of our students like Margarita would have, you know, some experience in the field. And but we also have younger students like Sheila who will need guidance. So, I think it the typical way of let a student discover the projects will not work well. So, that was my, you know, judgment call. And um I don't know, you know, maybe we should run experiment again in scientific way to see, you know, if that's true. But anyhow, I made that judgment call. So, um what I did with the client is I reached out to the head of their analytics function and and basically asked, "What do you need help for?" So, the client actually um posted a basically a call for proposal of I think six ideas. And uh um I I think it kind enough. They ac- They were actually very specific about each idea, what they want, what they don't want, what consider as success. Um uh happy to work with them on that. And then I think, you know, what I want to make sure is when these ideas or a or uh project a call for project reach the students, they're not ambiguous. They The students should know what they are expected because they need to make the the best decision for themselves that this project will help them in the long run. >> I'd say that's a fair retelling. Also, you gave us a lot of iterations with you and then you would go back to the client if we had questions, even though they were quite specific from the beginning on what success looked like or what they didn't want what what was out of scope. We still had some questions, of course, cuz it's just a short document. So, when you start actually thinking about a plan to work on it, questions come up and so we would be able to talk to you, ask those questions. At some point we even had meetings with them to make sure we were aligned. And so, it was good that we had not only the document to figure out which project, cuz then we got to choose each group got to choose what project they wanted to do. But also, before choosing we had an option to ask questions and make a more informed decision. >> I think that's that's a very good point and um and that's that part is actually not designed. A lot a lot of the parts are actually designed by me to make it look real, like a real consulting projects. But that part is not designed. That's true. Even though you guys didn't do the client discovery work, you actually did a lot of the project discovery work. Because I I think of for two reasons. One is, you know, um they're when when I discovered the client and you know, um um the they had some ideas, but still their understanding of what you can do or what AI can do is basically limited. Right? So, they don't really know, um you know, these problems they what they supplied are basically problems they have. Right? And and when you are matching problem with solutions, which is I think what analytics is all about is find the right solution for the right problem. Um you will come up with a lot of follow-up questions. That's That's what we do in the real world, right? You'll have constant conversation with the client and figure out, you know, is this what you really want or does this really solve your problem, right? You're constantly matching these two. I think that's, you know, that's that's natural. And and and um >> [snorts] >> what's more natural is is not re-engineered by me, right? Um I I think this this second part is um one of the missions at at the AI Tech Institute here at Dolan is we want to actually educate beyond the classroom. Basically, um how what is the the the way of of using AI. At least our understanding here is um you know, basically find a a few pilot um workflows and automate them. And rather than, you know, the other two ways as I just mentioned, you know, either the deep into the algorithms or some some tutorials. And and then once you have an understanding of what AI can and cannot do, then you can either um expand horizontally into other workflows or actually go deeper into, you know, building your own solution, which, you know, I My understanding is, you know, Synchrony Financial is doing it They're They're doing both, right? They're expanding horizontally. They're also going deeper into this. So, while educating our students, I think our students are also educating the client. Or or at least, you know, uh provide uh you know, valuable information to the client to guide their business decisions. I think that's great. >> You mentioned before in this podcast one of the I think really deep insights from you has been not just uh maybe it's It's to the vertical, but you've talked about breadth, you've talked about depth, we also talked about elevating. >> Yeah. Like maybe you could do something completely different now that you have AI. You don't have to just do what you've been doing. Did that come into play here, or was that sort of outside the scope? >> So, um that's a very interesting um um thought. So, my my vision of that is more like a pyramid. So, typically, if you look at typical, you know, um time engagement. And then everything almost look like a pyramid. And you have the high-value targets at the top, but you have the the broad engagement at the bottom. So, um a lot of these uh project turned out to be either bottom-up, meaning you engage the actual worker um at the bottom, then you move your way up to the strategic level. Or it top-down, which is, you know, basically uh you gain support at the the C-suite, for example, then you basically go down the the the command line. Um I think I, you know, MIT did a study last year and saying neither actually work for AI projects. So, um the the bottom-up way, it information would get lost in translation from the very tactical level to the strategic level. Um and the the the the top-down way, um basically uh sometimes, you know, you don't really know at at at the strategic level, you don't really know how things actually work at at at at the you know, the lower level, the tactical level. So, I think we took the third strategy. We took a middle-out strategy. Right? So, basically, we cutting into the connecting level. We're We're targeting uh department heads and and team leads. And And basically, talk to them about what do you need help for in this case in their analytics function. What do you need help with? So, now we have two options. They can go deeper into the everyday level. Right? That's what I talk about. But what you said elevation is now if we have more functions in the same organization, Synchrony Financial or or some other organization that adopt this methodology, then they can form their own AI strategy, you know, at the level. So, we are middle out. We are going We can go up and down in both directions. Wonderful. Okay, so tell us about what How did You were on different teams, right? So, tell us how you chose your projects and >> Well, uh our process, my team I was working with Shaw and Valerie. Shout out, they did an amazing job. They are both uh current professionals um who are either reskilling or upskilling. And I'm similarly also taking a break in my career. And then learning this new analytics world and everything. So, they weren't as familiar with coding and programming. And a lot of the projects were about either selecting the best LLM for certain coding tasks, uh documenting code, or in our case, what we ended up choosing was cleaning data and feature engineering. And it was a bit daunting for all of us, I would say, but especially for those with less of a technical background to choose a coding project. But our thought process was we're going to have the support of not only the professor, the client, but also we want to learn this new thing even if we're not very comfortable with coding yet or all of us aren't. So, even for say Xiao who was coming from an accounting background, it was a a driving factor, a motivating factor to learn something new. And that's a big part of why we chose it. Let's learn while we're doing this. >> awesome. >> Not only the AI aspect of it, but also a little bit of a how Python code works, what feature engineering is. So, let's make the most of those two opportunities, I'd say. So, that's how we chose it. >> And what and go ahead. >> undersell that because to me in any kind of analytics, particularly machine learning projects, data preprocessing will take a lot of time. >> Yeah. We also knew that was actually also a very big part of it was the value added. We knew all of those statistics. I don't know, it can be anywhere from 50 to 80% of the time that you spend on an analytics project is spent cleaning the data, not working on the algorithms themselves. So, we also thought, yeah, there's a goldmine there. If we can add value here, it's going to make a big difference. >> And it is going to make a big difference because I think, you know, that particular task is not only time consuming, it's mostly manual. And that's where automation will come in as a big play. And then the value actually also means if you actually do a good job in preprocessing, in cleaning up your your data, it will have a much bigger positive effect on the results than selecting the best model or training a model and things like that. So, it is actually very um very very important project. >> Yeah, and like you're saying, it will help to generalize to other projects both in your life and in theirs. >> True, especially if cuz all of us were considering a data science pivot in different degrees, but we all knew we were going to work more and more with data, so it would help our own projects in school or in a future career. >> So, what was the project exactly? >> Essentially, we had two stages, the cleaning and the feature engineering one. You got a messy real-life data set that companies deal with cuz there's human error, there's input >> What like what's roughly what are we talking about? Credit cards >> In our case, yes, we used credit card data, so a a synthetic data set from Kaggle, which is a data a website that has a lot of data sets. And we used credit card default data. We sort of messed it up on purpose to have our skill agentic skill clean that data. So, we made intentional mistakes, say different categories uh spelled in some wrong ways, or maybe we had say we identified someone who's male, female, and then with an M, with an F, with the full word, things like that. We messed [snorts] it up on purpose. We had added some null values, some numbers where there wouldn't be numbers. Stuff like that. And our skill would run through it and clean it and not only just output the clean version, but um log every decision and output a confidence score and a reasoning behind because sometimes you have a lot of missing values and that means some things, and sometimes it it's just noise. You want to differentiate between those two scenarios. So, we didn't really just want it output a clean Excel with no explanation. A major part of where the LLM could be of good use was to justify the decisions and log them. So, that was the first part, and then that clean file would then be used for feature engineering, which is essentially creating new variables. Of course, we didn't go crazy with the types of variables we'd create. It was mostly ratios or differences, deltas. >> You came up with them or the AI? >> The AI would come up with them. So, we would essentially inform through the type of information that there was. Um also have someone log the decisions and discuss. We had like three personas. A more a business one that would ask what would the business like to see from this data? And a more technical one that would think is this correctly computed? Like statistically, does this make sense? And then a third conservative one just to have a final check that everything was done right. And then that would create the new variables. And if we didn't create one that was discussed at some point, we would also output that in a report and say why we didn't create that new variable. >> Very good. >> Yeah. >> That's good. When you when you were doing the data cleaning, did you tend to think of or run the scale like row by row or big chunks at a time? >> It was big chunks at a time. We had a limit because we were using the web interface, so we couldn't really go overboard. >> Mhm. >> At some point, Claude we were using Claude, they decreased their usage limits and stuff, so we had to have some sort of a limit, but it was a big chunk at a time. I forget the exact number, but it was in the thousands. >> Oh, nice. Very cool, Achillea. >> So, I don't know how much this has been mentioned today, but when it came to picking the topics, that was probably one of the things that took the longest. Like while we only worked on it for six, seven weeks or so, picking the project itself was probably like two weeks on its own. Because we started talking about that from the beginning of the semester. And we were given the information I think we were given like four different topics. And we went back and and on asking the questions about them. And that alone was probably a good month. Like, it was a very long process just to even settle on a topic. >> So, tell us more from your experience you haven't dealt with consulting before for clients. What what did that feel like? Was it um complicated or did it feel easy or comfortable? >> Definitely felt complicated and I I feel like all of us like group-wise we're all pretty like similar in our career levels. Like as Margarita said, they were all like taking a break trying to grow. We're all just starting out in my group. We're all the same age. We all just finished undergrad the year before. So, this was the start stepping stone into the real world what rather than like taking a break from it. So, and from our perspective and I hate to compare, but we viewed ourselves as like the least technical because like being younger and like kind of like grow going through college with the AI like our coding experience was a bit less. One of my group members had never coded before and the rest of us only started like in this program. So, we all viewed ourselves as less technical than the other groups which played a part into the topic that we ended up choosing and we heavily considered every single topic to the point where all of us groups kind of collaborated on who was picking what because we didn't want to all pick the same topic and when we first heard all the topics we were all thinking the same one. >> Which one was that? >> I think it was mine. >> What was yours? >> I think so. >> It wasn't mine. I know that for sure. >> [laughter] >> Um but we ended up settling on like compliance documentation and having that get automated partially because we viewed it as like the least Cody technically like we knew the pre-processing and the the third topic ended up being My gosh, what is it? >> Picking the LLM project that we talked LLM and we figured we would kind of stray away from that and go with the documentation because out of all of our years we've done the most with documentation rather than the code itself. So we thought it would be interesting to dive in that aspect more. >> That makes sense. >> And >> And did they fit your expectation? Meaning less technical? >> No. >> [laughter] >> Good. Uh so so so sorry, the project was compliance documentation? >> Yes, so basically because while we haven't done it ourselves, what we researched is that compliance documentation can get really messy and it takes hours and you're constantly trying to go back and forth with people. And we our system kind of cut that time down. We ended up using one of Margarita's group's output. >> Really? That's so cool. >> It's it all chained somewhat chained together. It was fun to see how it worked. >> It's good too. >> And even like the order we presented in like my group presented last because we kind of took from the other groups a little bit. >> Uh-huh. >> Like we went in the right order of the chain which I don't even it wasn't on purpose like when we decided the groups and like it just happened to be what we all picked. >> That's so interesting. So what so what the project what are you given and what is your what was your goal? >> Um so the goal was to have the LLM create the documentation obviously quicker than a human would but also staying organized. And so like we had the JSON output from her group and we kind of put that through a skill that made that into a data dictionary. And it like took the confidence scoring and just logged everything and validated it all. And then that would go to another skill that took like the tests and the artifact and like the context library. And then created narratives and descriptions as well as validation for all of that. And then obviously it ended up like you need human review with AI. It's a tool. It's not to replace us. So then it goes back to the human to review. And it also is one of those things where like if some cuz we were using Claude and the limits. So like you never know if something's going to break. So we kind of had like precautions put in place for that as well. So like if something breaks it doesn't just delete everything. Rather it spits out what it has and like it's just a little more human review. But at the end of the day it >> Very interesting. So question for all three of you. So it seems like the projects connected. Does that mean you had to wait for hers to be more or less done or and how would you organize them? Like that's it's a very real world consulting problem, right? The connection. You don't want to be sitting around for too long. >> Yeah, we were we were worried about that at first because we knew we were told that like we should collaborate with our projects because of how aligned they were. And that was something we were actually really concerned about. So we're like we don't want to wait for them. Like it's already a very limited time frame. We don't want to have to wait longer. No offense. But >> [laughter] >> Um but what ended up happening is we did the building and at first we were also using a data set from Kaggle. But then when they finished we replaced So we didn't have to rebuild anything but rather we just switched the input. >> They sort of scaffolded it in a way with the replacement file. And then we just had the same structure with our own actual output. >> That's awesome. >> It's the good part of being in the sandbox is that you can have some trial and error. >> I think that's the beauty of spec driven development because the specs, the requirements are basically the same. Doesn't matter what exact input data you're dealing with, you're all you you created a a a data dictionary, basically there are steps and and and to to follow. There are items to be generated. And and you can plug in any data set as long as it's not too far off from, you know, um from, you know, what what you design. And and that's I think that's a that's a that's a beautiful thing about about this. And I realized, you know, it's very interesting to talk about spectrum and development. And and both didn't tell us told us, you know, coding is was never part of that. Um I'm sure we'll have agenda engineers watching this and say, you know, don't be ridiculous. Um you know, because because, you know, SDD is always tied to to coding projects. And and but I I do I think um all these things what we're talking about here, we're going to spend some more time time talk about what skills are. And and and then I I think, you know, I I want to say engineering is more than coding. What we're talking about agenda engineering here, and I think all of them did decent agenda engineering work, and that's more than coding. And um I think limiting um what um spectrum and means to specific coding, to me spectrum and development actually means you it's it's the same thing as as I said, matching the problem to the solution. And now this time you're basically working out the problem to be more specific, to lay out the the the the requirements, the constraints, and everything, and you develop, right? And it never said spec driven coding, development is development. And uh um so I guess my my point being, you know, well, there is clearly an element of luck that, you know, everything line out to be, you know, to be connected. But there is also, uh you know, the element of of design where we we choose um spec driven as the more um abstraction heavy way of of design. Rather than, you know, we start from the the concrete code, then trying to, you know, abstract and line them up later. I think at the design level, um it it it did help. >> I think maybe this is a great time to pause. Uh we've teased spec driven development, and we haven't talked yet about your results and and how you accomplished it, but I think we learned a lot about the uh the power of pivoting and experiments and and having an open mind and working together and how that can all um collaborate to to help a client [music] in in the real world. So, thank you for joining us. We'll see you on the next episode. >> Yeah.