Why Mindless AI Is the Real Danger / Ovetta Sampson / Episode #259

Hi, my name is Ovetta Samson and you're listening to the service design show. This is episode 259. I've been thinking a lot lately about how the AI models we're all rushing to use are most of the time basically just desperate to please us. We treat these systems like they are incredibly smart or even intelligent as the AI companies want us to believe. But these models don't actually care about being right. Their only goal is the mathematical reward of filling in the blank, even if they have to completely hallucinate to do it. Hi, my name is Mark Fontine and this is the service design show. Our goal here is to figure out together what it actually takes to design services that work for people, for businesses, and for the planet. Our guest today is Oetta Samson. She's the founder of Right AI and the kind of person who used to wake up at 4:30 in the morning to train for the Iron Man triathlons and ocean swims. It won't come as a surprise that Oetta has a ridiculous amount of discipline and she's bringing that energy into how we deal with the current AI hype cycle. As you'll hear, OA is obsessed with keeping humans in the driver seat. In our conversation today, we talk a lot about why treating AI like a person is a massive trap and why letting engineers handle AI deployment on their own usually ignores the messy realworld context of how humans actually behave. The part that really stuck with me was when Ovetta brought up a fatal selfdriving car crash to explain something called automatic deference. It's this terrifying human reflex where we just blindly hand over our judgment to a machine the second we're told it's autonomous. This example definitely flipped my perspective. We spend so much time as service design professionals trying to remove friction. But Ovetta argues that right now we actually need to design friction back in so people don't fall asleep at the wheel. So, let's jump into this conversation and I'll catch you at the end for a few final thoughts. Welcome to the show, Ovetta. >> Hi, Mark. How are you doing? >> Hey, I'm doing well. Uh, we met in Barcelona in 2025 at the Beyond the Map conference, and I was the guy sort of nagging you to write a book. Uh, you haven't written a book yet, but you are finally on the SE design show podcast. Uh so I'm very happy uh to explore some uh important and um yeah no important topics with you and let's just dive in. I was preparing my notes and you've made it so easy for me. This is going to be like a walk in the park. Uh I want to start with the fact that um you shared with me that you're advocating for mindful AI. >> Yeah. >> Okay. Let's not let's let's start with OA. What is mindless AI? Can you take us there? >> I think mindless AI is AI that's designed without the understanding of human nature, behavior, and aspirations in mind. >> Can you give us some examples? >> Yeah. So designing your AI with sick fancy and because sick of fancy makes it easier to prey upon our cognitive bias of anthropomorphosism which also makes it easier to have attachments to inanimate objects and ascribe human values and attributes to which also makes it easier for us to disassociate iate from the embodied world which also makes it much diff much more difficult for people to discern and make decisions based in reality. >> So we are being tricked when we are chatting to chatbot Charlie uh thinking that is impersonating something that is not like is that what you're hinting of? >> Yeah, I think so. And I think that when I talk about mindful AI, I talk about what I'm speaking to is understanding that humans will anthropomorphicize inanimate objects. That is an evolutionary trait that we have, right? And so if you understand that about human behavior and you want to adhere to a really wellressearched and known principle about transparency in AI, then you want to design in nonhuman friction points to remind the human that is engaging with the machine that they are indeed engaging with the machine. You don't want to design the machine to take on human attributes like sick fancy, like praise, like um uh hip-hop vernacular, whatever, like uh taxonomy that forces people to have cognitive load about distinguishing between when they interact with a machine and when they interact with a human. Humans should easily know that they're interacting with the machine and you shouldn't put the burden on the human if you're designing the systems to tell the difference. >> So, uh let me ask a naive question. Um >> Sure. >> Why does it matter? Like what's what what's the problem? >> Yeah, >> it matters currently. So I get an email every day or week and it comes from the AI incident space and it's basically a list of AI incidents that are vetted and listed in this database that allude to human engagements with machines and AI systems that cause harm. And so it matters about the human cognitive bias to anthropomorphicize machines because another bias that tends to come there is auto automatic deference to machines in lie of our own human judgment. And so that could char that could cause significant harm. Like for example, the first death of a self-driving car. I remember this vividly because I was in China doing research on self-driving cars and it came over an alert that a self-driving car that was hired by Uber um had hit a pedestrian in Arizona. A woman was pushing her bicycle across a four-lane highway. There was a person in the car whose job it was she was sitting in a passenger seat or she was sitting in the car. Her job was to monitor this car's behavior. Now, that woman had been in that car. She was hired to do it for hours and hours and the car had not made a mistake. And so, what she decided to do was basically check out. She was scrolling our they we have video of her scrolling on her phone, all that good stuff. And so in the design of that interaction with the human monitor and the self-driving car were a few things that I think brings up this human engagement risks of anthropomorphism and automation bias. And one of those was the cars even moniker of being self-driving. because it it's not it's not a self-driving car. It needs human intervention to be able to not have accidents, right? And so the car got confused because it knew the physics of distance equals VT, right? It knew what the speed of someone walking was. And it knew what the speed of someone riding a bike was, but it didn't understand and it had never been trained on the speed of someone pushing a bike. So, it made a decision in that moment to keep going and it plowed through a person basically and she ended up dying. And so the danger of us not being aware and designing with our cognitive biases in mind when it comes to automation is nothing short of death. But also it interferes and disrupts our relationship with the real world. And so now you have what's commonly known as AI psychosis. But there's a that's the popular term for it. But basically, it's people going down rabbit holes with machines and falling and succumbing to it sick of fancy saying that they're the best mathematician in the world. That's how you get things like the flat earth conference, right? Where hundreds of individuals are getting together because their algorithms show them that the earth is flat and now they believe that. So now you're getting distorted realities. You're not just getting new narratives. You're getting people who look at the world in a very distorted way because they're engaging with these machines as if they're engaging with humans. >> Okay. So um we're engaging with machines as if they were humans or they are um in our design process. We might have a design principle where we try to reduce friction. We try to make things seamless. We try to make it a pleasant experience and therefore we might come up with design principles and uh design choices where we actually say it's better for the end user to feel something when they are interacting with a machine rather than them having sort of a sense of a tool or an object. So >> I would ask why do you does why does someone need to feel something when they're engaging with the machine? >> Maybe because uh it's going to lead to a better experience like it's going to give you u a more pleasant experience, a more enjoyable experience u I don't know a sense of being understood um human. >> But I I understand wanting an experience that you can feel. Uh-huh. >> But I'm just trying to get a distinction here between having that feeling with the machine. >> So, how would you like >> do we have that experience with our laptop? Like are we when we engage with our laptop are we like hey laptop let's have this like >> well some some people might I don't think >> experience right I I'm just asking I'm just saying like >> do we have the same expectations for other machines that we engage with so we engage with with a computer is a machine right we engage with a automatic door we engage with like we have lines of demar ation about um what is a human experience and what we want to feel and objects. And the reason that this gets distorted is because AI systems can speak, right? Communicate or they can communicate, right? They so we we associate the ability their capability to for lack of a better term see, speak, hear all these things with with our human ability because that's the narrative we've been sold. But the true idea is that AI systems do not see, think, hear, or speak, right? They process text, image, language into binary vectors and mathematical models, >> right? And and confusing that leads us to say things like we need these systems to feel right. And my work is being mindful about and very careful about how we design these systems so that we're designing to the limitations of AI systems, not their capabilities. And the reason why I want to design to those limitations is because those that is where the risk to humans come in. When you say design those systems, which systems are you referring to specifically and what is there to design? >> Well, first of all, all models, all AI systems are designed that there there are deliberate choices made to get the outcomes that you get. So if there's anybody who says that AI systems is just about math and binary and zeros and ones, that's just BS. That's not true. People make decisions about what kind of models to put in these systems, how to connect those systems, the architecture, what those systems are trained on. Like there's a whole series of human decisions that go into these automatic machines doing things, right? >> When when you say AI systems, again, let it's I think it's good to be clear. What do you mean with AI systems? >> Well, I mean anything that is programmed mathematically to take data and make decisions, right? So that could be a machine learning model and machine learning is the def and I define this ad nauseium in all of my presentations. Machine learning is taking past data to predict the future. Period. Right? That's a machine learning model. Artificial intelligence is taking contextual data to make decisions and achieve a goal, right? Agentic systems, right? Or an agent is a system is an AI system that is giving task and goals and makes decisions in I would say semiautonomously without human intervention. So those are kind of like I mean I could go at nause on deep network. I think it's good to to set that. Yeah. >> Yeah. I think those are kind of like the layman's terms of what I'm talking about here. And when I talk about AI systems, I'm talking about multiple of those three things coming together to execute on all kinds of design products, right? because no no all of the stuff that we're dealing with today like generative AI and all of that is a combination of models systems agents and that entity to me is an AI system. Mhm. Mhm. >> So it could be multiple agents designed to do multiple things coming together executed in a product like clock design, right? Or it could be one machine learning model that does weather predictions and forecast demands whatever that is an AI system. >> Thank you for sort of helping uh to to to clarify that. Uh I think it's super helpful. Now let's go back to the design part because um there's there's an design element in how these models are actually trained like uh yeah that that is very foundational and fundamental work. Most of us will not be uh involved in that stage although we probably should be involved in that stage. uh you you can comment on that in a second. But the the other thing is um I think a lot of us who are listening will be involved in organizations where >> they are delivering services and the question is being raised by senior management like how can we use AI to do whatever in our service and that there's a design element a design component in there. >> Yeah. So again I have a different definition of design. Nice. >> And a lot of people have used design what I call with a a small D. Design as a noun. So when you say design to a lot of people, and I'm not saying this is everybody, but when you say it to a lot of people, they think of look and feel or they think of the end result of an experience. So how an experience feels, how it looks, how people move through it. I spent most of my career in infrastructure design and designing platforms that create machine learning um outcomes, right? And so when you say when I say design, I'm talking about design as a verb. And that is design that makes deliberate choices to influence and impact that outcome. >> So not taking a model that's made and then shaping an experience from it. Right? Taking a model and reshaping the behavior of that model so that you can get the experience that you need. Now when I started working in machine learning infrastructure for you to do that you had to know Python. Now you actually don't have to know Python, right? Like you can actually shape the behavior of a model in the context window and a lot of designers are doing this in evaluation. So >> hold on hold on hold on because one I think you've come to the right audience because uh design as a verb is what we've been preaching for the last 10 years here. So uh welcome you found your people there. Uh the other the other thing is that um uh you mentioned we can shape the behavior of the model. >> Yes. Okay. >> You can. So prior to there's a line of demarcation that happened in 2022 in the fall of 2022. Prior to that, we had machine learning and deep neural networks and AI. And that was the only AI that anybody really talked about. These were the models that um powered Google search engine metas, uh Facebook algorithms, ad feeds, like all of the Spotify, Netflix recognization, all of that, right? Like that was the world we lived in. And there was a very small group of people who knew how to build those models and execute on those models. And then chat GBT came out in October 2022 and the idea of training models, evaluating models moved from a very heavy mathematical statistical quant need to a qualitative one. one that said it's not hot dog or not hot dog from the famous Silicon Valley show. It's is this hot dog delicious looking? Is this poem good? Is this music sounding correctly, right? So, a very qualitative evaluation of model goodness. Prior to generative AI and LLM, determining a model's behavior, whether it was good or not, was a statistical feat. So, not a lot of designers was into that, right? Not saying they couldn't be, but it was definitely the realm of data scientists, right? And people who could really look at whether the model was giving false, positive, positive, all of that. very statistical based on math. Once generative AI came out, the evaluation of model goodness was a lot more subjective. Is this output Shakespeareian? Right? Which is why in 2018 and 19 Facebook started hiring all these PhDs to evaluate the model their model outcomes because they needed experts to say this is Shakespearean because they needed humans to tell the model to train the model on what Shakespearean was because again AI systems don't speak see think or anything like that so they needed humans to evaluate that. This is called reinforcement learning human. Okay. So now you have humans who are taking pre-trained models. That's what the GPT stands for, right? Like generative pre-trained models. They're taking that out of the shelf, out of the factory made at deep mind and open AI and anthropic. And then they're evaluating the outcomes based upon human understanding of different kind of levels of this is what good looks like. They're shaping model behavior. You can too like you like you don't have to be a PhD. We're not we're not entropic or open AI or so how do we how do we shape them? If you're if you're a designer and you're working at a company that is using claude or using um uh open uh chat GBT, you have to shape its behavior because chat GBT and anthropic know nothing about your business. So now you have to teach this pre-trained model, this factory model, what's important, what does goodness look like? And that's done through the context window, right? So now you understand like why people are talking about tokens and all this kind of stuff and prompting. So people are prompting. When people prompt a model, they're shaping its behavior, right? They're shaping the outcome it gets. What designers need to to me if you're going to do design as a verb in the AI era, you have to shift left. So you can't wait until the model is all massaged by engineers. You have to go to that context window and you got to start shaping its behavior so that because you understand the human better than anybody else what this model is going to um encounter when what humans are going to encounter with this model. So what are the prime examples? Sycopancy >> what? Yeah. Sick of unpack that word for me because it's not on my daily uh vocabulary. So sick so sycapant, right? Like if you you've ever, you know, seen a political show and and one of the if you've ever seen VEP, which is on uh HBO, it's a really good show. It's funny. Julia Drifus is in it and she's uh she's uh uh it's called VEP because she run she's a vice president, but her right-hand chief of staff is a sickopant. He basically sees nothing wrong in what she does. So every time she does something, which is some really horrible stuff, he just cosigns and agrees with it. And that's sick of fancy where it's like, no matter what you say, you're great. And I know and I know if you inter engage with any of these models, that's what they do. What a brilliant idea, Mark. You're so smart, Mark. How did you not How did I not think of that? And blah blah blah. That's sick of fancy. Where does sick of fancy comes from? Well, sick of fancy is an emergent behavior. So these pre-trained models, be it anthropic, be it open AI, Gemini, doesn't really matter. They weren't trained on sick of fancy, right? When they were made, they were made to return positive outcomes. That's how they got rewarded. But because generative AI requires humans to evaluate the model, what the models, these systems really figured out, they're really good at patterns. They figured out that humans rated their outcomes higher when they agreed with them. So these pre-trained models understand that agreeing with a human gets them their reward. That's where sick of fancy comes from. It's not, it wasn't programmed in, it was an emergent behavior. Same with hallucinations. Why models will just fill in the blank and not be accurate. First of all, they don't understand what you're asking them. They have to compute that. And second of all, they're trained through reinforcement learning to fill in the blank. They're not trained to be right. And see, that's the difference here. And that's why it's really what I call unmindful to burden the user with the nuance of the difference between being accurate and right and filling in a blank. >> Yeah, that is that is something we should repeat uh and and really hammer upon. The model is not trying to be right. >> No, it's not. It's not like that is and even even if it tried to like it wouldn't know how because it does like >> right it doesn't know what right is what is true >> you and you saw this at Barcelona I say accuracy is not in its wheelhouse and the reason why I say that is reward is in its wheelhouse getting a reward for completing the task is in its wheelhouse that's what it's trained for but accuracy is relative, right? And so it takes an engagement with a human to determine whether that's right or not. But the model really doesn't care about that. Like that's one of the biggest paradigm shifts that I tell to designers because we do are concerned with things like accuracy, trust, and all of that. And if we're gonna design with this new design partner, we really need to understand how this design partner comes to its decisions, >> right? And one of the reasons it comes to its decisions to hallucinate is because it wants the reward of filling in the blank not to be accurate. So you can redesign that. That's an example. You can design a system not to hallucinate and it's called three tiers of confidence levels. So you're telling the system that you have three different tiers of confidence level. The first tier of confidence is I know the answer is correct. I know this answer because I can go retrieve it. Right. Yep. >> Not because I'm a model BUT BECAUSE I CAN GO TO this document and get it. Right. >> Or or can I add something to that one? I sure nowadays models are also able to actually write code and if they use Python to do a multiply something >> they can quote unquote guarantee that the outcome they got from a piece of Python code is correct >> right so the point there is the model doesn't know it's right the model knows it can retrieve the right answer >> correct yes >> right you see the difference so so my confidence level is 100% that I can go retrieve the I answer from this pathway right >> the second confidence level is I wasn't trained on this so this input that somebody's asking me I wasn't trained on this but it that answer has you have given me a pathway to go get it so either it's Wikipedia or a file or whatever I know I can go get that answer even though I wasn't trained on it Okay. And then the third level is I've never seen this before. I'm not sure or I've never seen this input before. I don't have a pathway to go retrieve the information for this. I am not confident that I can return a good answer to this. So then we design an exchange for that a pathway for the and it's not telling the machine what to say. It's giving the machine a foundation by which to make a decision. >> I'm seeing >> that's what I mean by that's what I mean by design with a verb. Right? So now we're not just designing the outcome, we're shaping how that outcome occurs. And that's the shift that we need to do in an automated age. And the the shift from from what to what >> the shift from the shift from user and their device and it being uh nondamic non nondamic deterministic outcome when I write when I create a website and I write HTML and I write bracket hash000000 bracket it's going to be black right like the color I I think I have enough zeros in THERE, BUT YOU KNOW WHAT I mean. The color will be black. It will be black every day and twice on Sundays. Not so sure with machine learning. You can put the same input into your context window and get different outcomes. So if you want to shape that non-deterministic outcome that you're going to get, then you want to give the model a a clear concrete foundational pathway to make decisions. And that to me requires people who understand human cognition and ability as well as engineers, which I'm talking about researchers and designers. Doesn't this lead us to the to a partial explanation of the entroper morphosation uh as in the way to think about these uh general purpose transformer models the uh is that the way they behave is looks and feels very similar to how humans behave. So you I'm sure you've heard >> they're designed that way. Yeah. >> Yeah. So you hire an intern who doesn't know and you sort of provide them with the same instructions and sort of have the similar expectations as you would from a large language model. I see a lot of parallels there. >> Yeah, of course. Because they're designed that way. It makes it easier for us to engage with them. I mean, if those large language models came came in a octopus form with with arms, you know, instead of like a nice little UI with that we can interact with. It would be really hard for us to adopt it, right? So, it's designed that way to be easy for us to communicate with it. And that's fine. I don't have a it's that's not what I have a problem with. What I have a problem with is overly indexing on accessibility with models knowing their limitations. A the accuracy is not in their wheelhouse. Right? And so one of the things I I say in my I said in Barcelona is why do we expect chat GPT to be right? That was like a common expectation, right? And so I'm like, as a designer, do I design in? And then you started seeing all these disclaimers, right, where even in co-pilot, they changed the term of service. They were just kind of like this is for entertainment purposes, right? Like it really went from like you need this in your workflow to this is for im entertainment purposes. Like whoa. Okay. So we got to find a middle there, right? like we we can't expect people to trust these systems if we're not if we're being surreptitious about what these systems are capable of. Like if we're if we're kind of trying to be deceptive, that's what I'm saying. You don't need to be deceptive with people. You can be straight with them and and then let them decide how they want to deal with it. But but I feel like it's disingenuous to design these systems as if they are human knowing that they're not. >> What have you seen? Is it so that these systems don't even know themselves what their limitations are and therefore can't be transparent about it? >> Well, I think I I think that's a that's an anthropomorphicized question, right? Because the machine doesn't know anything. I mean the a system the system doesn't know anything. It retrieves everything. You see the difference there, right? So I mean we could get into a philosophical convers I I was doing a a AI 101 with with teens and we got into a philosophical discussion about memory because I say LLM don't have memory. Now most people are like yes they do. I read a whole paper on it and I'm like no they're calling it memory but it's really retrieval. All that architecture on rag and all the agentic architecture that is about retrieving because tokconomics says LLMs can only take so many letters and hold it in their context window for to engage with a human. Right? It's a limited amount. That's what tokens are for, right? It's a limited amount. And so when it least reaches that limitation, it can no longer retrieve that information anymore. And it has to be designed in a different way. Am I saying that LLMs won't be able to retrieve the conversation that you had yesterday? They didn't when they first came out. They can now. Why? Because we have the architecture for it. But systems don't have memory, right? We create the pathways by which they retrieve past information. And I know I'm being very didactic about this, but this is the reason because when we start saying things like machines think, machines have memory, machines whatever, we start having expectations of human outcomes. >> So what are the expectations that we should have? If we recalibrate like if you if you you're the marketing department of mindful AI like what are the expectations that uh we should have >> the expectation is that these systems are really good at calculations recognizing patterns. So they're better than humans at it. They can recognize and see patterns at scale with lots and lots of data. They're processing systems. They can process a lot of things which is why they do code well. They can process code because code is more binary than judgment calls. Right? So whenever there's an application where the rule is always the rule and the exception isn't the rule, machines are better. Why? Because they can follow rules at scale. multiple rules. They don't get tired. They do it all the time, right? They don't have fatigue. They can keep doing this. So, mindful AI is really about not not using AI. It's and the number one principle of mindful AI is determine both the desiraability and the applicability of AI use first. So whatever problem you're solving, mindfully think about should AI solve this problem and the and it requires both the desiraability from people and the systems by which this problem will be solved uh the people who engage with that system and the applicability of the the capabilities of AI not and minimizing its limitations to solve this problem. So for example, AI is extremely great at forecasting anomalies or forecasting when your machines are going to go out way before they do, things like that, right? Not so great at forecasting which kids are going to end up in jail. Why? Because you can take all the data from every kid who ended up in jail and still have that outlier guy or girl who grew up in with the same circumstances, the same income, the same whatever, but still doesn't end up in jail, right? Because that's that's something that you can't predict in stats. It has to do with a lot of different things which I call the human X factor, right? So, but still you have predictive policing that are taking stats and stats of arrest records and predicting where crime is going to be occurred. Well, I talk about the problem with that because if you take arrest records, you're assuming that everybody is arrested was rightfully so. Again, judgment calls. So, machine systems are only as good as the data that goes into it, right? And so if you use racial profiling data to decide your police predictor, you're going to get bad outcomes and you're going to be wrong. So maybe we shouldn't use AI there. Should police departments use AI? I'm not saying no, but maybe we shouldn't use AI to decide who goes to jail. That's just an example. >> Yeah. Yeah. And one of the things you just mentioned is that you know what kind of data went into the system to create the outcomes that you're getting. >> Most of us like won't have a clue like based on the system like we're just we're just the end. Yeah. But this is this is what I talk about model shaping behavior because in that instances in that instance where I talked about hallucinations right or I talked about your team taking a GPT taking a general generative pre-trained model >> right and and reshaping the outcome from that you can write you can you can reshape the and I teach this in my class you can reshape where it gets It's data to make decisions from. So, hey, you don't know if that pre-trained model was trained on Reddit or scraped from the internet or whatever. Like, it could be all kinds of stuff. So, there are ways to you could be like, "All right, when you make a decision about this, this is the knowledge base that you start from." >> Mhm. Mhm. >> Forget >> whatever you were pre-trained on. Right. Now you're retraining it, right? To to tell it what good really is. >> Yeah. And Exactly. And and I feel that that is a very interesting area where large language models are heading to where you'll use them for I don't know if reasoning is the better word but you will be responsible for what goes into the stuff that needs to be reasoned across instead of hoping that it will give you the answer to who was the present and yeah >> I distinctly remember at Google when large when we were frantic to get Jim and I out and everybody Everybody was just going crazy. And I distinctly remember having a conversation with one of our researchers researchers going large language models are useless because the reason why I said that is like large language models are like the Library of Congress. But people want Netflix. They don't want to search through every book in the Library of Congress. They want the recommended series that goes along with Stargate or whatever they like, right? And large language models are like a nuclear weapon where we really need a scapel. And so what you're seeing now in enterprise is people taking those models and reshaping them to be more specific. And even I advocate for small language models because you don't even need all the LLMs, right? And you could do it on device to make it more private and close the loop and it doesn't even have to go to the cloud. Yay for on prim. And so I've created a product that actually does that. That says, okay, you know what? You don't need all this from anthropic or claude. Let's create your contain system. I call it the triage room where you give the agent a direct pathway to the actual data that it needs to be it needs to use to make these decisions and then give it the goals and task and you shape its outcome how you want it to make those decisions and then it's off you go right and it protects your IP and you don't have to bring in all these bad behaviors that these pre-trained models have and you can reshape the outcome. home. And that to me is where design with the verb and service design and and a lot of us who care about how these experiences are created will will end up being >> I I've shared this example with you prior to our call and I want to bring it into this conversation as well because maybe this is um uh an example of how you see design working along AI. Hi. So, I've told many times here on the podcast that we do a yearly survey uh inside the service design community. We look at salary data, but we look across 50 60 different parameters and every year we publish a report and that report is something that is publicly available. People can search through it. But like you said, people want Netflix, they want a specific thing. So what I went out building is I call it a newsroom with AI agents, specific roles defined like a chief editor, a data scientist, a graphic designer, uh uh a reviewer. Those are defined roles, models, and I've defined a workflow that tells like you just said, okay, we're going to write an article about the salary development in the Netherlands over the last 5 years. here is your data set and like follow these steps like first use I don't know the journalist then use the critic then use the data analyst and at the end um an article comes out something that is written that is backed up by data and that workflow is designed those roles are designed and uh like my AI model isn't like isn't doing much it's just following a procedure that I've instructed it I' I've co-created the procedure with the model, but it's now following the steps that I've told it to. Like I how I don't know. Does that make sense? Is this how you see things? >> I was talking about I spent the last year developing mindful AI because I wanted to give people agency over the behavior of models. Period. We were always talking about the bad things that models could do and people were feeling disempowered because it was like I just have to accept this BS from the oligarchs about how these models and I'm like no you don't. So how do we make it so that Mark can actually shape the behavior? Not just give it the task and and goal and reward but shape the behavior when the limitations of these systems show up. So, one of the questions that I that I would ask you is like when you decided to when you were designing this process, what could go wrong, right? What could go wrong? Maybe you create an article with inaccurate information. >> I'm just saying >> it could it could happen, right? Maybe there's a link inaccurate information instead of just kind of like accepting the continual dis dis, you know, um, uh, destruction of the truth that that now is the internet. Um, you might just add a system command prompt to your agent that invites your audience to correct it. right now. Does it automatically update the data and publish it? No. It just escalates to you. It's like, "Hey, Mark, I generated this article automatically. Someone commented in the comment box. I'm just making this up. That's not true. Maybe you should look at that. I'm just Yeah. You know, >> because you would do that at a human newspaper, >> right? You would have a budman or uh someone who takes calls who's like, "Hey, th this article in the paper wasn't right. Why wouldn't you do why would you just make an agent for that?" And the agent is separate from the agent who generates the content. Its only job is to evaluate accuracy. Why not design that? And and this is I like let's let's sort of double click and zoom in on this because this is the design process thinking thinking about the flows thinking about the steps thinking about the exceptions uh >> and the limitations and the limitations. Yeah. >> Your model has a limitation around truth. So what does that mean for design? Every model has a limitation around truth. What does that mean for design, right? What could go wrong? I'm just saying. So, by asking these questions in a mindful way prior to creating the agent, right? I'm not saying you're going to get all of the unintended consequences, but I am saying you're going to provide that design that I gave you doesn't design out in accuracy. It just designs in that that's a possibility. >> Yeah. >> And so now you have a workflow for what happens when it's wrong >> and it's going to be wrong. You have to assume that it's going to be wrong. >> You have to assume that it's going to be wrong. But the problem is people are designing these agents without that assumption. And so I as a person who's engaging with them have no feedback mechanism to say this thing is wrong. >> And those are the marketing stories that we've been sold to by the by the Geminis and the open AIS and the anthropics of the world. >> This is what I mean about anthropomorphism. This idea of infallibility, of AGI, of all this is all narrative, yo, this isn't truth. This is the story they're selling us. >> Exactly. Exactly. And >> and if it's a story they're selling us, then we could write another story. >> Yeah. And and the story for example or part of the story is assume that these models are not true and that they are going to make mistakes. And the reason why this is important um is you could make the same argument for collaborating with humans on an article as well. Humans are going to make a mistake as well. the the the problem here is that you're going to make uh mistakes at an accelerated pace and at a larger scale when you start introducing technology. >> Yeah. And as a person who has worked to build these the infrastructure to enable these the execution of machine learning and model platform at scale. I can tell you with 100% certainty it is really hard to undo models. >> What do you mean with that? >> Like meaning once they're trained, once they're executed, it's really hard if you haven't done the data and model lineage to trace back where it went rogue. Why? Because they're making a series of the models makes decisions. They do. Now, I would say they make decisions, right? They they follow rules, but they make decisions. That's our job, right? And so, they use data to do that. And so, trying to trace how those decisions are made, if you haven't laid that out really clearly is really difficult. So, you when you're going to build an autonomous system, one of the things in design is designing how they make the decisions. the pathway to decision making. >> Yeah. And so you tell them or you specify as best as you can uh follow these criteria when when selecting for what is good, what is true, what is important, what is valuable, you you design those criteria, >> right? But you can't design it for every scenario. That's the thing, right? because it's non-deterministic. Because you're not going to know all of the inputs that people are going to give this design. One of the things I tell teenagers, and this is a real revelation to them, >> is that Google isn't the only one training these models. And I give them the example where they have to Google themselves, which they do all the time, but then they switch phones, right? So their friend next to them Googles them on their phone, right? And so then they see the difference in the search results and I'm like the inputs are different. That's why you're getting different search results. That's a revelation to them. And what teenagers took from that was I can control Google. That's how that's how they thought. They were like, "Oh, I could change the outputs from Google by controlling the inputs." Right? adults are like, "This is scary." You know, THEY'RE LIKE, "OH MY GOD, I DIDN'T KNOW THAT." OKAY, two very different, you know, reactions. But what the point of that is is that once a pre-trained model goes off the production assembly line, this is the dirty little secret in the industry. Nobody knows what that model's going to do, right? That's why Facebook and Open AI and all of these people pay so much money to PhDs andmemes to try to evaluate model outcomes before they get into production, but it's a crapshoot, yo, right? They they think the model is going to return the code the way it's supposed to, but they don't really know it until it gets into production. Why? Because machine learning is nondeterministic. you could get different outcomes for the same input. >> And someone someone might say here, okay, this is an engineering challenge, but I'm saying you're saying I'm saying that you're saying that this is a design challenge. >> I'm saying that leaving it to engineers is unfair for a a couple of reasons. This isn't a technical problem. This is a social technical problem. And a social technical problem is when you add technology to human environments. The technical problem becomes social technical because humans form societies, communities. They have rituals, unwritten rules, practices that aren't technical. They have everything to do with how we make decisions and how we relate to each other. And so leaving it towards the engineers is unfair because they're building the technical part but the completion of the experience is once it gets into the environment and start getting inputs from that environment. And that's why design is needed because we're the ones who understands the environment >> better. Like we know the social the we we use research, we we use um user feedback, we use all these things to determine kind of what the ecosystem this system will be put in and that needs to be in design of the model's behavior as well. >> Yeah. >> Like the model and this is what I talk about in my class. I'm just giving the class away for free here, but this is what I'm talking about. The job of the designer is to translate the embodied experience to a disembodied system. That's where design comes in. So the engineer has the Python down and it has the rules and it has the model and the algorithm all that and then the designer has to say all right of this is going to operate in a hospital. Here are some of the social technical things that happens in hospitals that the system needs to understand. >> You know what makes this so difficult and what that the shift why this shift feels so big I think for a lot of people is um we see technology as a static thing. You you bring in a hammer into a workshop and you know like the hammer is not going to change. Uh it's going to do the thing that it's always been done. But this is unpredictable. It's malleable. It's um it's moving all over the place. And thinking about technology in that way is is new is different, right? >> Yes. Yes. Because again, and I talk about this, the difference between designing in an interaction age and designing in an automated age. There are clear differences here. And one of the biggest paradigm shifts is that we're designing for a stoatic non-deterministic outcome. Unpredictable basically is what I'm saying where in the interaction age you knew that the iPhone we even had a word for it called affordance, right? That the experience was the same when you pressed the button or when you spoke into the mic or whatever. like it was deterrent cuz that's how software works. Models don't work like that. >> Mhm. >> Because their output changes upon the variables of their inputs and no one can give you all of the variables of the inputs of the input. Yeah. >> Not in production. >> Yeah. Yeah. And maybe uh uh and I'm well I'm sure that this analogy will fall short very quickly and break down but if if the model is the operating system the general system uh we need to load it with the right software we need to provide it with the right instructions tools that we want the model to produce. >> Yeah. And and people are going to hate this. We're not always going to be right. So that means we need robust feedback loops, right? We need to we need our users are co-designing this experience with us. If we don't enlist their help, we're going to miss the mark a lot, >> right? because we're going to assume a lot of things and we which is why the the non-determined na non-deterministic nature of model outcomes really lends to the nondurability of products because before it was like all right we did research we we know people want this or or don't like that or whatever. Okay, let's shape capabilities to be able to do this. Right now, it's kind of like well it kind of changes with the time or the day or it's like contextual. It's all contextual right and so now it's like the agent can make decisions about how the experience should go based on the context by which the environment is giving that. And so to evaluate whether the agent made right decisions, we need that robust real-time feedback >> and we need that understanding about the context. Yeah. >> Yeah. Yeah. And so I I'm just saying like and I remember saying this a long time ago to a team that I worked on that designing in an AI age will be completely different. And it's so different that it's going to be really hard for you to kind of like internalize that. And now that I'm working with teams and I'm working with companies for AI adoption and bringing in this quote unquote new design partner known as AI systems and tools, right? I'm really seeing that play out where double diamond design think all these linear workflows just do not work and it's kind of like even the titles right like the roles like what does it mean to be a designer today >> what does it mean to be an engineer when designers are pushing to code bases and engineers are having UI done by cloud so We're all seeing a transformative disciplinary. >> It's a mess, Oetta. It's a mess. >> Chaos, RIGHT? LIKE WE'RE ALL going through that. So, we should take comfort in knowing that nobody knows what their job is right now. >> So, if that's the case, what do you say to everybody who's made it so far into their conversation with us? like what's a practical thing they could do today, tomorrow to um I don't know gain more confidence, become better at practicing mindful AI. >> Well, I think a practical thing you should do is to know that you can shape model behavior to understand what the context window really means and that these models come out like a factory car, right? like a with factory default settings that you can reshape. You should you must >> and you should reshape because they come with things like hallucination and sick of fancy and all the limitations that models have right and so my number one thing is to think of your job as a designer if you think of yourself as a human- centered designer which has in the United States it has a definition that's a two-parter one is getting requirements from users about what they want need all that and the second is creating products that don't do them psychological, physical, and social harm. And that second part really is about shaping model behavior and outcomes. How model makes decisions. Like if you're not doing that to me in an AI age, it's really hard to call yourself a human- centered designer because that is where the the special sauce is on derrisking these engagements. I think there's a lot for us to still explore and learn and understand about AI as a design material or what the material properties are that we can influence because >> uh we know what the material properties of wood or plastic or metal are and we know we have the tools to shape them. Um but >> well good good news we still have things like gravity and physics. So those haven't changed. We still have things like math that haven't changed because gravity phys well gravity physic well maybe not well gravity but physics and math definitely play into how models behave right. So I think the good news is even though the earth seems to be shifting right below our feet, there are some immutable facts that we can lean into. And one immutable fact that we can lean into is that models have to compute and have to do calculations to make decisions. And because they have to compute and do calculations to make decisions, we can affect what those what they're computing what they're calculating and how they make those decisions. Now when they start making decisions without that, that's a whole different >> then then they become humans. Yeah. >> Right. Right. THAT'S A WHOLE DIFFERENT THING. BUT NOW WE can control that, right? Like we can control what they compute, what they calculate. We can control input variables. We can control how they do it. Giving them workflow tasks and rules and setting up what I call shaping and dictating model behavior, right? Telling them what good looks like just like we a parent does a toddler, right? And so the toddler comes with blueprints or whatever. But when it goes to reach to the stove, we have some things that we have we can say to the child about that, right? We can do that with models, right? So, what I want people to take away from is it's not hopeless and you do have agency. We just have to shift a little bit about where that agency is and how to execute that. And and uh when somebody wants to gain a better understanding of how to shape model behavior, understand this design material, where would you point them to? >> Um well, of course. >> Yeah. No, feel free to. Yeah. So, what is that? Where where can people find you? So, so, uh, uh, on my website, Ovetta Samson, ovetta-samson.com, uh, or you just Google me, look it up. I am opening up, uh, a weight list and a registration list for my series called Mindful AI uh, master class training session. And it's all about this everything we're talking about here. How do you shape model behavior? How do you upload that to uh um um GitHub, whatever, execute that? Um how do you do that as part of a team? It's about designing with AI and designing for AI and what that means and things you have to learn and paradigm shifts and also practical frameworks that you can incorporate. So you can go to my website um and sign up for that master class. is going to drop this summer. But also other places that you need that that can help you. Um, anything that talks about model context design or uh model evaluation design like these are like anthropic just opened up a new, you know, job for model behaviorists, right? Like these are this is what I'm talking about here. And so these are new thing like new jobs that didn't exist but they they do exist now and they're ripe for designers because they require people to really understand the situated cognition of the world and translate that to models. Right. So um some I I'll have to send you a list of resources I think. >> Yeah. Yeah. We'll add that in the show notes. Yeah. >> Yeah. some books and some some other things, some people where I got a lot of my inspiration from and kind of some papers and things like that because I feel like we I'm done with the era of doom and gloom on AI and I want to enable people who are aligned with protecting human values and and aspirations and goals and all that kind of stuff and want to shape these systems to be able to do that. Um I just in my newsletter I just shared a TED talk by um Dr. Rumma Sheldry who I respect. She's a RAI expert and she just did a TED talk on repairing AI systems. So that is one that I highly recommend because it talks about the ability for humans to repair the wrongness that comes out of these systems. a lot to explore and uh chaos also brings a lot of opportunity. Uh one final question I have for you and that is uh we love to end on uh something to think about a question rather than an answer. So what do you feel is worthwhile for us to ponder upon after this episode? >> I think I think we could come full circle about what it means to be a designer. I think it's really good for us to re-examine that because and I I don't just mean in the AI age, but I I just mean in general, what does that mean? What does that mean in our teams? What does that mean in our companies? What does that mean in the AI age? I think this is a good point to kind of like look at some of the things that we've thought about in design and UX and the limitations and the complaints and all the things and say here's an opportunity for us to reinvent our industry um and reinvent it without this almost crippling need to prove our value. Because to me, we're invaluable in an automated age because we're the ones that still live in an embodied world, right? Because that's where people are. So, so for me, our value is being shown every day. Every time some company is like, I I adopted AI and it went crazy or my agent went rogue or whatever, like that is our value, right? >> And I would like us to reexamine our what it means to be a designer and and leave out this crippling like need to prove ourselves. >> I think that's a great question to end up on. Um, OA, thanks for sharing. Uh this was uh thoughtprovoking, inspiring, optimistic, hopeful uh in many ways. Uh thanks for the work that you're doing. Uh who knows, maybe our conversation will uh be continued because again, this is not the end. This is uh part of our journey. >> Well, uh when the book comes out, I would love for you to have me back on >> for sure. Uh you can be sure of that. Once again, thanks for coming on and um to be continued. >> Thank you. >> Yeah, >> thank you so much, Mark. I appreciate you. >> Once again, a huge thanks to Oetta for coming on. During the conversation, I brought up how we are so used to technology acting like a hammer, a static tool that does exact same thing every time you swing it. But large language models don't work like that. They shift. They hallucinate. And because of that, you can't just toss them over the wall to the engineering team and hope for the best. We as design professionals actually have to get our hands dirty in the context window and shape the rules ourselves. OA said something I wrote down immediately. Models have knowledge, but people bring wisdom. If we want to build services that don't just blindly follow the factory defaults, we have to rely on that wisdom. If you've enjoyed today's conversation, as always, you can do me one big favor. Click that like button on this video and leave a short comment if you haven't done so already. Not to feed the algorithms, but to let me know whether or not we are on the right track by addressing topics like this. Finally, before we part ways, please take a moment to reflect and celebrate that by joining us today, you've directed your attention towards learning and growing as a professional. So, from everyone who you are going to impact through your work, thank you for taking the time and making the commitment. My name is Mark Fontine and I look forward to seeing you with us again for a new conversation on the service design show. Take care and see you