AI for You Podcast | Episode 15: Seventy Percent of the Work Was the Spec

[music] Welcome back to the thrilling conclusion of this two-part series [music] looking at experimental classroom design and consulting in the real world with AI at Fairfield Dolan and our MS in business analytics and AI. Uh I'm Philip Mman. I'm joined as always by Dr. Jay Tao, the director of the AI and Tech Institute and the professor who if you saw the previous episode, but I'll catch you up to speed, created a new experimental class this past semester that used AI uh agentic workflows um to solve real world problems for one of the biggest best companies in the world, Synchrony Financial, who brought on as a as a real client. Um in the last episode we walked through uh the beginnings, how it uh was formulated, what were the problems um the the challenges of pivoting mid-semester, the uh the amazing students we have and let me introduce them again. We have Margarita Sakudo and we have Sheila Green. Uh they're recent graduates. Congratulations of the MSBA. Uh they did very well in the course obviously they delivered for the client. What we're going to talk about today is the next step. So we had the problem set up. problems. You'll repeat what your problems were, how you chose them. Um, and then we'd like to go on the journey with you. Uh, what did we learn? How did we grow? Uh, and focus on we ended last time um with the discussion of spec driven development. So, maybe start with talking about that and skills and all of the rest of the technical stuff. So, let's go tell us about your journeys. >> All right. Um, well, we started out, or at least I started out knowing, of course, about LLMs and how some types of prompting were more effective than others, but that was pretty much it. I would use it as a tool to help maybe debug some code and it would be very adoc interactions. Not very even though the prompts tended to be more structured as I improved my usage of it, it wasn't something that where the overall approach to the conversation itself was very structured. [snorts] And so I think from the beginning of of the course, even though we weren't focusing yet on the final project, we learned so much from an actual good prompting structure, how to handoff between conversations so we don't explode with context drift and the LLM starts struggling with keeping up with everything we've discussed. So tiny things, incremental things like that that you don't even realize you're learning so much while you're in the middle of it. I feel like only looking back I was able to see where we started and where we ended up. So I say we started out very inexperienced and then improved I would say by iterating and by learning incrementally the better prompt structure, the conversation, how to handle it better. And then finally arriving at spec driven development which is a way to approach a project and for it can be for AI assisted coding but it can also be for general coding. I think it was it existed before LLMs but the way we used it for this course was trying to avoid just pure VIP coding where you lose track of what you've built and how the code is organized and everything. you're coding bit by bit and you're kind of improvising as you go >> and you generate what is called technical debt. So you sort of don't know when you have a bug you lose a lot of time trying to figure out what it coded for you and how it interacts with the remaining parts of the code. Specdriven development allows you to figure out all of your functional requirements. So for someone who is not very technical like me or my colleagues in my project, it was an amazing way to make sure that at the end whatever code it produced, whatever automation was generated, if it was failing at some point, it was very easy for us to go to a functional document where we had specified what we wanted our requirements but in plain English >> and we could edit and improve and we knew exactly what was changing at the end when the actual LLM skill would run. >> Okay. Um, let's talk a bit more about that. Well, we'll get to your journey in a second too, Sheila, but I want to dig into spec driven development and try to understand it. So, vibe coding, I understand you're just constantly chatting. You have a new idea, you tell it a new idea, you drift, you're surfing, you're doing whatever you want. Um, is it fair to say that spec driven that the the spec was it written by you by hand or was it part of the collaborative? >> It was a collaborative, >> right? So it's essentially you're are you vibe specking >> somewhat? I would say we we used the interview structure a lot to generate each output. So the way we did spec driven development and correct me if I'm wrong here but I think there's several um tools available or kits available. We use the GitHub spec kit. So it's very straightforward what steps you have to go through. But essentially we used this interview approach where we would say okay this is an idea that I have in our case our project data cleaning data feature engineering this is essentially what I want to accomplish now interview me to so that we can arrive at a spec with this structure and we would feed it the GitHub spec kit structure so it's very easy to make it more contained and have less drift and the interview process itself we would challenge even after answering ing some questions, we would say sort of the Socratic method. We would ask it to challenge some of the assumptions that we made. And [snorts] that was a really good way to refine each idea and make the most of the collaboration instead of us having to think ahead of every hurdle that we could face. Especially because we're not very technical people. a lot of the things that we would face later on about coding and libraries and stuff, it would raise those issues for us in advance during those interviews. So using that interview to arrive at the final spec was I think the best way to collaborate with the LLM. >> I heard a quote, I think it was Einstein, he said uh if he had to solve a problem, he'd spend in an hour he'd spend 59 minutes coming up with the right question. So maybe that's the same thing as with spec. How what percentage of the time of the client work would you say was building the spec itself? >> I would say it was maybe 60 to 70% actually. Yeah. Because we really before you arrive at the final final spec, we had a pre-document that was more generic before we settled on any specific decision on the coding language, the actual LLM model that we would use. That was called the constitution and that's sort of the overarching document that you want to keep somewhat generic so you don't have to revisit a lot, but that guides then the spec. So we spent some time on that and only then did we go through the actual spec with the more detail and specific requirements. >> Okay. I think I'm starting to understand. So it's a hierarchical approach to building. Yes. >> First you have a con. First you have your general prompt, right? Then you have your constitution, then you work on the spec, then you work on the code and the planning and whatever. >> Yes. You do planning, you separate it by tasks which are smaller requirements and that would have maybe some of the tests that you would want to make. Um and only then you implement even then after implementing you revisit anything because >> and how often do you go back? I think we went back so for the final weeks of the project because we were gaining clarity also as we were going we were learning uh we went back uh for the final two weeks I think perhaps maybe five times that >> back to the spec or back to the constitution >> back to the spec >> uh the constitution was fixed >> we changed maybe once or twice we updated the constitution with a little bit more detail but because we wanted to keep it somewhat stable and just be the guiding principles for the project. We didn't have to revisit the constitution that much. >> That makes sense. >> The spec we did a few more often. >> When you revisit the spec, you've already done some work after the spec. Do you continue that work or do you start from scratch? >> No, we continued. And >> so you would tell it like, "By the way, there's a new spec." >> We would update the versions. So we had like version versioning uh for each spec and we kept logs of all the conversations with the LLM and so it would be easy to just say then okay we're updating the spec because we figured this out this step that we were maybe we didn't do the three personas review at the beginning we realized we wanted it in the final report or at some point we realized we didn't want a data dictionary because they would have a data dictionary things like that um we would just update the spec and it knew the versioning process because the versioning process was actually specified in the constitution. >> Makes sense. >> So >> very interesting and your journey was similar. >> Yes, there's I feel like all of us had the same structure because it was like a classroom setting instead of just doing it on our own in a job. We all used the same structure like we were all supposed to have our spec one week and our constitution done another week. And obviously, as I said, we kept going back to those like we definitely had at least seven versions of the spec. Like it was just something >> and same as Margaret in the last couple weeks, you'd still be updating a bit. >> Yes. Like >> and I think it's particularly challenging for your group because you know, you you were trying to basically wrap your mind around the whole plan to spare kind of thing. >> Yeah. >> You know, if you could talk a little bit more about that, that would be I think that would be great. Yeah. I feel like it was something so new to all of us. Like even though we spent so much time picking a topic, I can't speak for my group members, but I definitely didn't fully understand it by the time we were even starting it. I It's just something that was so new to everybody in the room that for a while I feel like I would talk to people even like outside of us and our groups like we would talk to other peers and they would be like, "Yeah, I don't fully understand what's going on, but you know, we're figuring it out as we go. And as we kept putting the hours into this, we kept getting more and more clarity. So we would go back to the spec and be like, "Oh, wait. I actually understand what I was trying to do here and I actually did something different. So I'm going to go ahead and update this." And even though like it was like collaborative with the AI to make the spec, it was still something as I've emphasized like human review is so important and it's something that we had to keep working with. And I feel like talking to people about AI, like I was talking to somebody two days ago, I think, and they were like, "Well, how do you trust it?" Because AI is so known for making things up. Like even with something like this, like it could make something up because it tells you what you want to hear. And I feel like a big part of what we did in the class was auditing and red teaming. Like along with the interview process to help us come up with it, we had to make sure like this isn't just telling me what I want to hear. Like this actually is plausible and this actually works. So, it's a lot of going back and forth and I don't want to say arguing with it because it's not like it's a person, not to personify it, but it's just a lot of going back and forth and working with it and making sure that rather than just telling me what I would like to hear that it's actually something that will help me and my project grow. >> You got a lot of uh you were right to push back. >> Yes. Yes. And I think that's um those are the answers to your question is not really vibe specking because I think with vibe specking I I think the word vibe here means I don't need to look what's under the hood. >> Um >> good definition. >> That's my understanding and I think you know all they did was actually trying to pry under the hood to see what's going on. Maybe they're not looking at the code, but they are looking at the internal machinery and why >> they're looking at the spec document for sure, >> but also they're looking at how how the the model function and respond because they auditing the red teaming they're persona challenging the the the results. So they are prying in under the hood to understand you know what is happening why they are they getting a response like you know they got so I don't I don't think it's vibe backing um and I think you know the the whole um the whole idea of of spec driven I think it you know writing specs is actually I hope you know it's a useful skill beyond you know these project and even beyond dealing with LM or coding because um I I now I realize I can do a lot of things with specs because I I get a task now I can basically the idea is break the task into little functional units and solve one functional unit at a time. So I I think it's a it could be a life skill that's really interesting. And so you you presented this way this scaffolding right this framework for the students but you it came from somewhere for you too right from your own consulting experience what other what other frameworks or structures did you think about other than spec driven that you threw away because this is better >> um so before using spec driven I was using test driven so basically test driven development is you design a test first then you um basically build the product to uh pass those tests. >> That was particularly good with uh more human hands-on and AI assisted coding. >> Meaning I'm I'm doing a series of prompts to to to write the single functional unit. That that's good. you know, when I work with AI more in a more hands-on way, but then I I have shifted from a more hands-on way, tell the AI to write this code to that code to more hands-off way to give the AI more autonomy to do this. And I feel like uh test-driven fall short because uh the model now is both the the the judge and the player, >> the basically the referee and the player, right? So it can actually first it can design um weaker tests and it can also fake through the tests. So then I I I I feel like you know even though I want to be hand hands off I still want to maintain full control of the whole development process and I feel like you know test driven is giving me less or weaker control over this because I don't even though I can say make the test harder it can still find a way to go around and and it I have experiment proving that you know it can't go around that. So that's why I I move it more to spec. Now I don't you know it's it's still test is a part of the spec but now I care more about what you have to deliver. What are the passing gates that you have to reach before I can call this? Um the other one I think you know um I think I I just talked about is the more hands-on the more interactive way of of doing things. And I think you know you guys may have something to say about this as well is because what you went through in the in the course is basically a very interactive way that you you basically you are working you are collaborating with the model to to get this done and I think you know I want to say it's basically um you you are it's it's almost like a ladder right you start from at the at the the least experience level. You need that experience. You need that handholding to understand on one side to understand what that models can do. On the other side is basically to to work your own skills to build your own skills and even to boost your confidence that you can get this done. And and I think once you are more familiar with this, you will move to the more hands-off way to do this where maybe you set up certain you know checkpoints and say you know I need to make sure this spec works and I need to make sure these tests are included in the specs and then you can basically um let go let model you know do the heavy lifting. I think that's where we we talk about you know LMS make us more efficient. I think that's the way to go. >> Mhm. >> So, it's not that uh you still use specri development when you're consulting clients, right? You're not this isn't like training wheels. Yeah. >> Right. That you're teaching them. This is the real world. >> This is the real world. >> So, and then and that's that's my teaching philosophy right now. I I don't think it's responsible to teach students something that I don't use every day. >> Yeah. >> And or or something that have gave up. Right. >> Yeah. then then you know and and it has to come from the real world. It has to be battle tested to to to actually um to work in the real world. >> Okay. So you've done the specs, you've done the constitution, you're constantly referring back, but you're you're now planning it out. You're building things. What what does the rest of that journey look like? >> So the end output would be the so-called skill that we've been mentioning so much. So we used claude skills. They also have a format I would say a structure in which they're written out. So again it's also helpful that you have a guiding framework. And all of this work that we had done so far with the specs with the plans all the details that we had on what we wanted to accomplish was then structured into this way of claude skill organization I would call it. So essentially it's a document that has yes um that has the main point of the skill itself and then is supported by pieces of code scripts uh some people I think call them hooks as well and all of those things were generated through the specdriven development you condense them into a zip folder you upload them to claude and that's a I think also a big part of how user user friendly it is. You can just use the web interface cloud.ai and say you name your skill. In my case it was data- cleaning. Uh you just backslash data cleaning upload the file and it runs the skill for you like a normal chat. And because the skill and the scripts that support it all have the purpose that you specified in the spec, it will output all of the outputs that you said you wanted. In our case, the clean file, the report with the decisions, the confidence scores in the decisions, and then you would hand it off to the second skill, the feature engineering, and it would generate the new variables, the new Excel file with all of the new variables, the report as well on the features that were created and how confident it was and what the decision process was, and then output a JSON file for my group, following group. >> So, you would do each skill separately. you do slash data cleaning, wait for that output, maybe have some interactive chats. Yes. >> Fix some things. Then when you're comfortable, it's the same thing as like first a constitution, then a spec, then run it. Right. >> It was also critical for us to have that checkpoint before starting the new features. If you're going to create new variables, you want to make sure it's on good data. So, we wanted to be able to inspect it. Besides the human review, we complemented it a lot with AI review, of course, all the personas and everything. But you want at some point a checkpoint to make sure that it's going in the right direction. So we didn't want it to break halfway through data cleaning and then still generate features, buggy features, right? >> So we stopped and we divided into skills. >> That makes sense. So So for those of us new to skills, uh is a skill just a text file? Is it >> more or less? I would say I don't know if you want to compliment this but I would say it's a very rich text file that is arrived at through a well if you want a good skill uh through an iterative process complemented by code >> that needs to run >> and the code is large enough that doesn't fit into the same skill you'd have separate >> yeah we had separate files we had snippets of code within the skill for the critical things and but then we just had the name of the scripts that supported it saying, "Okay, at this point you will call this script and the script was a separate file." >> And would you call that a tool or >> Yeah, I feel like I tended to refer to it as like the orchestrator kind of like telling everything else what to do, but I mean I feel like it could be looked at in so many different ways. I think it's very interesting u and and there there is actually a backstory um here and um and it's title we talk about the challenge of changing course mismaster um is when I designed the course you know and we have run this course you know multiple times and it's focusing on the design part of the course so this is the experiment is mostly about the implementation or the prototyping part of the course and how how does how how to make that work and when I was you know um basically brainstorming about this I was thinking about you know what is the popular thing you know when everybody talk about workflow and of course you know the the the tool everybody is talking about is an addend it's a it's a diagramming tool and that's you know that's when they get into this course that's what's on the syllabus and say you know we're going to use it and in the second part of the course to implement whatever your idea was. And then um we we realize um basically when when when I uh talk more with you know synchrony financials we find out you know the a few things one is um what they want. So the the key difference or the key limitation of ANN is it's basically a drag and drop diagram and so human in that case is orchestrator you know I I I like what you bring up orchestrator human arrange the steps and the splits and and whatever so LM is is a part a small part in that process that's basically how how it line up to be and talking to Synchrony Financials, they want more than that. They that that they don't they're not uh content with just one cutting point of using nonjang model. They actually want to redesign part of their process, the data cleaning process, the documentation process and and let AI drive more. And so to me I think the the one of the difference from you know um um it is a just a text file is you know now you it you basically lay out steps and you're letting AI to carry out these steps or the difference between a skill and a text file is how do you ensure you have u enough quo of you know how AI do things your way. It has I I think it has two features we should talk about. One is the the assets where you can say here is how I do things. Here is the template of a report. Here is you know um basically some recipe you should look at when you do these steps. And then you have scripts you know the code part that's the deterministic part where you you have you can say whatever intermediate results you generate at this step it has to pass those deterministic tests before you can move on. So that's how we to me that's how we ensure the AI is doing things our way because we are providing two sides of assistance. One side is basically crutches saying you know here are some help use it and the other side is basically guard rails and say you have to do it this way if you don't pass those gates you can all move on. I was going to ask you um if you thought there's something beyond spec driven development like what might be the next thing but I think you're kind of discussing it here. It's like checkpoint driven. It's not just a spec. A spec is one type of checkpoint, but work and work and work until you get to a spot that you would might later change again, but is a a fairly solid deliverable, right? Just like you would have in in the real world in a corporate environment. Where's this? Where's this? Right? A road map. Yeah. >> Is that fair to say? >> I I I think, you know, that's a that's a very good idea because spec is a is a living document, but it's still a document. I think what you describe is is a living deliverable. So um I think you know all we want is basically a a a iterative development process that have you know we we can improve our deliverable but uh spec is one way to control that is to describe what a deliverable should look like once it improved you know in the iterative process. tested is the other way to describe this is basically how good it it should be. Um I guess you know to to add on to your comment I think you know we need to figure out is there a more abstract way that's not just a document or as passing gates that we can merit the quality of the the deliverable. I mean, I feel like we're at a point where AI and everything is going to continue growing. Like, we're doing all this now, but in five years, it could be such a different process. >> Oh, you five months? >> Yeah, five months. Five days. I mean, >> yeah, >> who knows? I mean, it's probably grown so much since we even started the course. And I think >> the podcast >> I feel like it's hard to speak on these methods when in a week it could be something the next best thing could be totally different. >> That That's true. So the nice thing though is uh you could be the one who comes up with the next thing. >> It's true. >> So when you're thinking about the next projects that you would be using AI for, would obviously you do some spec driven and some checkpoints and all the rest of it is what other thoughts like you're you're going back to your personal project now. Talk about that. How what what how are you approaching it now that is different than even maybe the way you finished the course? Yes, I would say um I am keeping a lot of what we learned for from spectriven development. I may be simplifying it a little bit because what I've realized it is that for some types of projects it might be overkill and you can just get there faster with a summarized version of specdriven development. I think that's the way I'm changing it the most. So, I'm I guess skipping some of the steps or shortening some of the steps there just because I think they don't add as much value for certain types of projects because in this case, the personal project is very textbased. There's not a lot of coding involved. So, there's not as many scripts that I need or a certain type of asset. I've forgotten about assets, but I just need a message. And LLMs are pretty good at knowing what messages sound like within, I don't know, certain tones or limitations, but I don't need to be as specific or restrictive with my specifications. So, I think that's how I'm mostly adapting it is making it a little bit more flexible depending on the type of project. So in this case, because it's very text based and that's what the LLMs are best at, I'm keeping it more loose and giving it more autonomy, less having less scripts that are more deterministic. I basically wanted to be good at finding the right people. Say I'm interested in a finance job and I want to find people at certain banks and who went to a school or who worked at the same company that I worked at something there. I just need to be a bit more specific. But other than that, I give it leeway to figure out what's interesting about that person or how I should approach them. >> I feel like I don't have very much to say on this one since I haven't picked it back up yet, but I feel like I do agree like with the client project, it was such a big and formal thing that with our own personal projects, not that it would be like cutting corners to condense it, but like she said, it would be kind of overkill. >> Mhm. So, I feel like it's good to keep it simple because if you if we did that whole process for every small project, it would be overwhelming and a waste of time at a point. So, I feel like every project is different and finding the way to have that spec driven, but also not going insane with it is a key point of keeping the balance. I think you know that's this is something I learned from you Phil you're the master of this is um we I think the last thing we want is teach tools we don't want to teach people tools because tools will be outdated then rigid like you guys said if you treat specd driven development as a tool or like you said GitHub specit that that is a tool right if you only learn the tool you it's you will not understand how to use it or when to use it and and and particularly like you said when smaller projects why why would you use something like this a huge tool and I learned from you is you you're using the tool to teach the philosophy so the the philosophy is the same the principles are the same right it doesn't matter you have to go through every single step but the philosophy is you have to lay out um different layers of constraints that an LM has to follow or different kind of checkpoints LM has to reach before it can move on. That's basically the the the philosophy of of doing this and that's I think that you know um a lot of the courses we teach actually follow that that way. We teach you the philosophy through the tool because if we start talking about philosophy at a very abstract level for for for for a student you may get lost. You may say what are you talking about? Because that's you know up there in the air but you want something that's on the ground that you can touch, you can feel, you can build. And then once you get more familiar with what you build, what you experienced, what you learned or even what you changed, then you can see you guys are moving up here, right? And say, you know, here is what I think how I should do this. And I think that's all graduate study is about is basically we give you a tool, we try we we try to help you grasp the philosophy behind the tool. >> That sounds awesome. And I love the idea of layers and and uh checkpoints and and constraints like because you you you search and then you want something to solidify and you can search again as opposed to just go everywhere. go crazy wi with your own experiences with your journey on this thing. There must have come a time so at the beginning of the you know this is an overwhelming project we don't really understand it fine let's spec it out let's great at what point do you remember having an aha moment like by god this might actually work >> I feel like for me it was a lot more towards the end than it probably should have been or ideally would have been like once I was kind of seeing everything together >> I was like wait we're actually pulling this off it's actually happening like even getting to the showcase itself and having to present our findings. I was like, "Have I had my aha moment yet or am I like going to figure it out on stage?" Like, [laughter] it's one of those things where we kind of just rolled with the punches either way. Like even like the aha moment did come, but even if it wasn't going to, it's like, well, doesn't matter. You're doing it anyways. [laughter] >> Yes. >> What about you? I think it also helped at the same time as we were developing everything we had each group had to make a classroom presentation and the first group to present we were still in our constitution spec phase work-wise but the group's presentation was already about what a skill was and that really helped me put in perspective okay this is all going towards that and at the end that's the thing that's going to run and it runs in this way and we'll have the assets and the scripts and that I think was my well early aha moment ahead of seeing it actually worked because I hadn't finished the work itself. The second one was the first trial where we actually tried the skill itself and it worked not perfectly but it did this a decent job. We first tried the cleaning one and that was first seeing it work was the full aha moment I would say but it was good to have that preview of knowing what a skill would look like before I had to deliver it actually. So I think it was a good thing to pair those two. We're working on our project but we're also getting a glimpse of what it's going to look like at the end through each group's presentation. To piggy back off of that, you kind of reminded me that those existed. I did kind of forget that that happened, but that was a big aha moment for me because I will admit like as the project manager, my amazing group members did most of the building itself. So, I can't speak as much on that as maybe Margarita can, but with the with my group's project, it was about sub agents. So, that was having we did it based on like one of our skills and having everything run at the same time. and being able to go back to it and edit it. And I feel like that was a big aha moment for me because it was kind of the first time I was more seeing it in action because being more like observant in the process made it hard for me to fully dive in and really see what was going on. So I think doing that project separately, but looking in on it helped me have that aha moment where I was like, "Oh, this is how this works. This is what it's doing." >> Interesting. You guys probably have a bit of an advantage in being younger than Jay and I. Like we we grew up uh thinking thinking in a programming style and when you're programming you have lots of tiny aha moments, right? Teeny tiny. This function works. Thank god. Let's do another one. Uh so as you're you're you're constantly building up and then you get what you expected, right? But I think with now at least my experience, tell me if it was the same thing with you and you with AI there there's a lot of moments of doing nothing and then it outputs something like wait this worked and that worked and this worked. My god. Mhm. >> Yeah. >> Bigger ones. Is that was that your experience? >> I would say so. I would have to agree. Yeah. It was just like you'd be playing around with something like not really knowing even what you're doing half the time and expect something out. It's like wait like >> that's a that's good. That's a key part. Like that that helps. Or a lot of times you're just hoping that it will work because you're looking at a document. You're just working on text. That's true. interviews and >> that's true. >> And then you just need to wait for the moment where you actually upload it and it's >> or like after the interview it's like wait is are the results going to work or did I do all this and it's going to like give up on me and not give like did I do all this for it to not give something that makes sense or what I'm looking for. >> Yeah. Or not generalizable. Is that been your experience too? I think what you said and first line align with my experience and then I think there is explanation for that is I think you know LMS and and you know basically modern tech in general is actually elevating us from the the actual work like you I don't know about you I look less and less about coding. >> Yeah. >> I'm I'm ashamed to admit I don't even read the skills anymore. [laughter] >> You guys still do, right? Or you had to for the course. You probably won't anymore. Yeah. So I I think you know um your little joy lives in the code the line to line stuff right. So as as we're reading less and less of that we're basically >> that's true >> distancing our >> that's true >> ourselves away from the little joy. And so what I've been trying to do at least in this course and and in my daily projects is I still want to touch back at what's going on but not at the code level. All the the interviews, the specs, the checkpoints, the tests, the the the iterative development is all me touching back to the actual development process. I don't want to build a wall that between me and and see let's let's like like we said you know at the very beginning of this podcast you know prompt and pray [laughter] that that is that is that is what vibe coding sound to me right you prompt build me this make make no mistakes >> right and then you pray it works >> right >> right so um I I I don't think I can live with that so I don't want a war I don't want I do want a separation. I do want an elevation that at a more abstract level I still know what's going on. And I think you know um this is also how we train these students is you know they have the basic skills right so they they can get the work done at a very abstract level but they have the basic skills if they want to dig deeper into this and figure out what's going on and answer the client's question how do you build this they can you compare it to um uh if you're if you just have employees working for you right you're obviously not going to do their work but you have check-ins you see they explain to you what they're doing right they certain road maps. What are the similarities and differences from having a team reporting to you to having agents that you've built? >> Um, I think it's very um it's similar in in a lot of different ways and I think you know again very early in this podcast we say you know AI are overconfident interns. I I still feel that way. I still feel my agents not only overconfident but also overeager to get things done. Right. Once you say, "I want this build." Okay. Boom. Done. Build. But wait a moment. How do I know you build? You didn't even ask me how I want this build. I just give you an idea. So, um, I I feel like, you know, even with the model advancing, um, even with the the the the famous infamous fable, I feel like I'm slowing it down. I'm still slowing it down a lot >> intentionally >> intentionally not not that you know well on the other in the same time I'm becoming the bottleneck I realize that but I'm still slowing it down I I want to say you know um wait a moment don't start building let's talk more about this ask me like the interview technique they they mentioned ask me about you know what I want to build ask me what I don't want to build ask me or challenge me what assumption I'm challenge is good. >> That's these are very important and then at the end result I wanted to create a a document a self-guiding document from the interview from the challenging that throughout the process. So um you know I I I think I would spend a lot of time um you know um in that in that phase rather than just directly get into building. But I always believe right now you know if you look at any kind of development project there are three stages be even without AI there's planning there's building there's reviewing so basically as a developer you spend time on all three of those so now we can agree building part is highly condensed you know we can even ignore it >> right so now it's basically you spend time on planning or reviewing >> my preference is always you spend as much time on planning so you don't have to spend time on reviewing because reviewing is painful. >> Mhm. >> Planning where you're brainstorming, you're talking about your your colorful ideas and designs and things like that. That's always a joy to me. >> And look at look at what you build um and you know um and start challenging how did I do it this way and why it's not working. That's always a pain. So my way is I want to extend the planning phase as much as I can so that I don't have to spend much time interviewing. >> That makes sense. Now, now when you guys were obviously wasn't individual projects, you had a group. So you had both challenges working with AI agents and with people. How was that process? >> I mean my group was amazing. I've mentioned them in the previous episode, but Chiao and Valerie were both very good at communicating at each point. I was also very involved and we had some time during the classes where we would break to work that was also very useful. But essentially them being able to communicate to me each milestone each thing that they were producing or struggling with and asking for help or going to the professor the three of us. So we always knew what was going on also because Shiao was focused on the data cleaning part. Valerie was on feature engineering but they depended especially Valerie depended on Chiao's output. So they had to constantly be communicating on expectations and deliverables. >> Would they communicate through you or just directly? >> Both I would say. So especially in class because it was the time we'd have together the most. they could speak more directly to each other and look at each other's outputs on the same screen and we would meet outside of class of course but sometimes I would be looking at the two things together and I'd have questions and then would have to ask each or either of them um but essentially I think we managed to maintain a very good flow of communication that was the most important thing where if there was a project level decision they would just come to me but they could always speak to each other and we'd have maybe a Slack conversation going on with the three of us where I could see things but I didn't have to intervene. they would just discuss between themselves. And it was just very important that we always had checkpoints each week because each week we had somewhat of a deliverable, an intermediate deliverable so that we knew I would be overseeing everything, but they could talk to themselves between themselves to see if anything was missing on either side or some of the output that Chiao was giving Valerie wasn't matching what the skill was expecting, how to adjust everything. So I would say communication was just the biggest part of it and all of us being okay with learning and not having this expectation of knowing everything >> and that's so powerful. >> Having the availability of the professor who was always we could meet at short notice that was helpful too. >> And I feel like for me I could not emphasize communication enough if I tried to. I mean my group members Whitney and Molly were fantastic with it. We were meeting all the time. I wish I was kidding. I even like I worked retail at the time and I had to change my work schedule around these meetings. like we met every Friday and like a lot of Wednesdays. The class is on Mondays and so we were meeting outside of class multiple times a week just to continue collaborating and working together on everything because while everyone's doing their own individual part like everything has to come together. So it's so important to stay in the loop and stay informed and just keep collaborating. And a big thing that I feel like helped helped us is BOD, which is an AI tool that he showed us that allows multiple people to collaborate in one chat. >> Nice. >> Which is probably the most useful thing out of this entire project. It's something I still use today. having multiple cuz in the beginning of the semester when we were doing the individual projects it was one person's laptop doing everything >> and while that was not my laptop I felt so bad for the person who had that responsibility cuz like that is a huge responsibility to have everything on your end when everyone needs to work with the same thing but only one person has access to it and can work with that it's really hard to make that work, which is why communication is so important, but having that tool to allow all of us to work on it did really help make it easier as well. >> Awesome. Sounds like a really challenging and powerful and enjoyable semester. Let's let's go uh to the end. Uh Jay, set the scene with the final presentations. What was it like? And then they'll tell us how it went from that perspective. >> That I think that's the pinnacle event of of the semester. and um happens to be the the one year birthday of the institute and um so at the institute we promise three things we promise you know we'll do community outreach well we'll do you know basically workplace education and we'll do impactful academic research. So clearly we're doing our through podcast like this but we're also doing our like you know projects with synchrony and and and things like that. So I was toying the idea of you know how do we make sure these projects are again a real beyond the regular classroom projects are actually something that will fly in a real workplace. So I was talking to Abby about this and I say you know how about we actually let the students pray them back to your team to um you know and we'll open the audience you know to to to other people to to real practitioners and and to to basically you know um and and and hear the students out and you know I was so grateful that I'll be actually bringing you know so training senior executives from Synchrony and uh you know we actually um um basically advertised this. So a lot of working professionals showed up at you know um the the showcase and uh you know um and they all did a good job. So I'll let them take over telling you know you guys their experience at the showcase. Uh so for my group it was a little bit different because so while the class was nine people, eight out of nine people were at the showcase and the one person missing was one of my skill owners because she is a college athlete and she was away for the NCAA tournament for women's lacrosse. So obviously that's something that we were aware might happen for most of the semester. Like she warned us pretty early on like it is a possibility that I will not be there but it was not confirmed until the week of >> Oh my god. Nice. >> So while it was something we try to somewhat plan for ahead of time, there's really no planning for it until you're in it. And we thought long and hard about what we would do with like her part of it. And for a while we were like, she's going to zoom in or I'm going to have to say all of it and learn all of it. But we ended up settling on her recording her part. And then during the showcase itself, I think for the most part it it was awesome. I don't it was such an awesome experience and probably the most rewarding thing I've ever done. just the pride and relief after it ended was inexplainable. But while we were on stage, of course, the one group that had te technical difficulties was mine. And it wasn't it wasn't our fault. So, we were using one of my groupmates lap. She borrowed a laptop from a school to avoid any technical difficulties because her laptop was terrible. But we had issues all semester with this girl's laptop, but um so we used the school's laptop and then the slides just would not project properly. Like it was something so minor like that and we're just standing there like well might as well crack a joke about it. Like I got to open up like cuz after the slides finally worked I think there were more applause then than anytime else. And I made a joke about that when we started. I'm like well that's the most applause we're going to get all night. So, it was even like something like that to break the ice kind of helped ease the nerves even because it was a big thing that I think most of us were really nervous about. And then even playing her video like we were really worried how that was going to play out and we were like what do we just stand here as this video plays? Like we didn't really know how it was going to work with having to play a video halfway through the presentation. But in the end, you know, it worked out. We we all did our thing and it was incredibly rewarding at the end of the day. >> Yes, I'd say rewarding is the right word. I had some experience in presenting to clients, but I hadn't done it in a while, so it was scary to me, too. And Valerie and Shiao, they both work already, so we're used to it, but it's always somewhat nerve-wracking. I think what we really tried to do was focus more on the results and the value of the project itself so we could really be more captivating to the entire audience because not everyone was the analytics team. There was more executive level, there was professors from different backgrounds, marketing and it was really fun because like I said in the previous episode, data cleaning, processing, it takes a big chunk of time. So you can always lead with that argument. So people are compelled to listen to your presentation. It's like maybe you're spending 3 days out of a five day work week in the boring part of working with data and you could really reduce it to some hours or minutes even. And then we also tried to frame the results in an interesting way. So we tried different data science models with our data set with the features versus without the features. And that was also compelling that we got better results with the feature engineered data set. I think we tried to focus on the value that this skill could bring. And then it was good at the end that we had a lot of questions from people even outside of Synchrony that were interested because they would need it for their own work and their own projects and they weren't some of them were but some weren't really techsavvy and they could just maybe use our skill to bypass that process and have their work made easier with our project. That was good. >> I feel like a big part of it too was that like the audience we were speaking to is such a wide variety of people. some very technical, some not technical. So, we had to find the right way to share our results to make it sound impactful to people who even have no clue what we're talking about. >> And Jay, what was the uh epilogue? I I I guess you know the app blog would be I'm going to ask you each of you if there's one thing only one thing that you wouldn't continue working on after this course from this course but after this course I know it it it might be your personal project but what would that thing be and and why is that help you why is that helping um in your future just just name one thing the one thing come to your mind? >> The first thing that came to my head was like the auditing and red teaming skills that I talked about in the last episode. I feel like that's something that will be used forever. Even I have family that works in data analytics and after we learned about auditing in this course, I reached out to that to one of my family members and mentioned it and she emphasized like how important it is to know that and how she does it all the time in her job and how it's something that she recommends that I keep working on because of how much it's going to be used. And even like in interviews, it's something I talk about like how cuz as we all know, AI sometimes likes to do its own thing. So, it's really important to have those skills to be able to check the work and really really be confident in what you're putting out. For me, besides the personal project, I'm lucky that the skill itself is useful to the career I'm considering. So as I'm thinking of going into data science and working with data science models or deep learning models, that whole aspect of data preparation is something I can actually leverage my skill for. So I definitely think I'll keep refining it and using it in my own. I've used it for school projects even. We have data science courses and deep learning courses. But for my actual career, I think it might be very useful in the future. I just need to be mindful of a way to use it safely within corporate settings. >> And now you have a a more powerful instrument. You you know how to build a skill, right? Not just a skill yourself. You can actually build more skills to what you need. >> What was the feedback from Synchrony after the end of everything? Well, um I think you'll hear it from Abby in the next episode, but the the general feedback um and and you can attest to this. They want to bring as they done back for further conversation about what they build. I think that that's um is the evidence that how useful um these projects are to to synchrons. >> Well, congratulations and thank you both for coming on the podcast. This was really great. Margarita, Sheila, uh thank you all for joining us. We'll have our next episode as Jay said with with more interesting feedback from from Synchrony. Uh what a fantastic episode this was. We last episode we we set it up. This time we talked about spec driven development constraints layers of hierarchy and how that relates both to AI agents as well as within a group and managing the people that you're working with together. The power of communication. It's the same power as with AI agents, right? You you build hierarchical AI agents. How do they communicate with each other to to bring about checkpoints that can grow ultimately to the wonderful aha moments and the wonderful feedback and the great opportunities and and growth that we all experience. Thanks for joining us.