Head of Product at Stripe | Framework for Building AI Products Users Can Trust

Duration: 20:55 2427 transcript words EN summary

Video summary

The video features Stephanie Neil, Head of Product at Stripe, who outlines a framework specifically designed for building AI products that users can trust within enterprise environments. Unlike consumer-facing applications where engagement and novelty are often prioritized, the focus here is on high-stakes scenarios involving financial reporting and auditing, exemplified by a fictional CFO named Sandy. For professionals like Sandy, whose decisions impact millions of dollars and must withstand rigorous audits, an AI product cannot afford to be merely "mostly right"; it must be 100% accurate because any error could have severe consequences. Consequently, the development process begins not with coding, but with deeply understanding the user persona, their specific pain points, and the critical need for certainty, accountability, and risk mitigation in every interaction. To transform a generic chatbot into a reliable tool for Sandy's needs, the presenter demonstrates an iterative building process that integrates Retrieval Augmented Generation (RAG) to ground AI responses in company-specific policies rather than broad internet knowledge. The initial prototype connects to a general language model but quickly proves insufficient because it lacks context about the specific business rules and financial regulations relevant to the CFO. By instructing the system to first read internal documents like "company policy.md" before formulating answers, the product shifts from providing generic information to delivering defensible insights that align with actual organizational constraints. This step is crucial for ensuring that every answer provided by the AI can be traced back to a specific source within the company's knowledge base, thereby addressing the user's need for provability and explainability. As the prototype evolves, additional features are introduced to further enhance trust and safety, such as confidence scoring and human-in-the-loop mechanisms. The system is programmed to output a confidence level between 0% and 100%, accompanied by color-coded indicators—green for high confidence, yellow for moderate uncertainty, and red for low confidence where the AI admits it lacks sufficient context or evidence. When the model's confidence drops below a certain threshold, such as when asked about non-existent Martian taxes, the interface explicitly flags the response as requiring human review instead of hallucinating an answer. This design philosophy ensures that computers handle tasks they excel at by processing vast amounts of data quickly, while reserving complex value judgments and final approvals for humans who possess the necessary context and responsibility to make critical decisions. Ultimately, the video concludes with a strong argument that building trustworthy AI requires a deliberate shift away from generic capabilities toward specialized accuracy tailored to specific enterprise needs. The presenter emphasizes that while consumer products can thrive on curiosity and flashy features, tools like this CFO copilot must prioritize precision over speed or engagement metrics. By combining advanced language models with strict adherence to internal documentation and clear indicators of uncertainty, organizations can scale AI adoption without compromising integrity. This approach not only solves immediate problems for users preparing quarterly reports but also establishes a sustainable model where machines augment human capabilities rather than replacing the essential judgment required in high-risk professional environments.

Read the full video transcript

Hi everyone. I am Stephanie Neil and I'm really excited to build with you today. Um we are I guess maybe actually a quick intro and then we're pretty much just going to jump right into it. Um so I'm Stephanie as I said. I am currently with Stripe as a head of product but previously um VP of monetization product at Amazon's Twitch and before that I was an executive director for the United States Digital Service. If you've never heard of that, you should Google it. It's a really awesome public service outfit. Um that really needs great digital talent. So Um all right. So today we are going to build and we are going to build for a persona that um I don't think everyone thinks about. I think there's a lot of thought about sort of consumer um building consumer products with AI but you know what? There are people within enterprise environments or internal environments that also could really benefit from great AI products. So we're going to focus today or actually I guess I'm going to give a spiel even though the fun is in the actual like typing and prompting and building. I'm going to give a spiel because as product managers AI might be changing our ability to code with plain English. Uh with these awesome tools but it is actually not changing the most important things which is like defining who are you building for um and what are you building for them and why is it important to them? What's the problem that it's really solving? So we are going to still start with that. Um uh cuz that still is you know the PM's the the PM's unique value. >> [clears throat] >> Uh so we are going to build to for Sandy a CFO. We're going to build an enterprise AI product. Um and I'm going to tell you about Sandy. She is a CFO at a $500 million a year software company. She is preparing to close her books for the quarter. Investors are waiting and auditors, they're always around the corner. They're going to arrive next week to audit her work and make sure she she's accurate and can explain her numbers and she can prove her numbers. These are really critical for her. And so every answer that the AI that we're about to build, the AI product we're about to build, gives could show up in the company's financial statements. So, we will start to build for Sarah. So first, who? Sarah, CFO at a $500 million company. And she cares about things like certainty, like accountability. She makes million-dollar decisions. And it has to be defensible and she cares a lot about risk. The way I like to think about this persona is if your product isn't 100% right, it is 100% wrong. So it is very black and white for her. Cool. So, why don't we now get into get into the excitement of actually building? So today we're going to build Sarah a copilot. So I will start by asking Cursor to >> [clears throat] >> build a clean web app called compliant. Actually, let's not call it compliant copilot. It is really for our CFO. This page must have and then we spec out a simple uh uh, basically what we're expecting to see in the product um, just like we would spec out for a cool engineer. Um, so one text box where the user um, can ask a compliance question. Uh, a submit button. An empty results um, area below. Uh, and I'd like you uh, to use a clean modern design. Everything clean. Oh, actually pet peeve. Hate it when I forget. Um, uh, when I hit enter um, it should take the same action as uh, hitting submit. I hate it when I have to actually click a button. Um, cool. So, while this builds, uh, >> [sighs] >> yeah, this should all be super familiar to you guys. Um, that was obviously a very simple sort of spec, but um, this is not different than um, explaining a PRD uh, or explaining what you're building in a PRD. Um, here we go. CFO copilot writing enter. Excelente. We'll wait for this to roll. Um, it's taking its sweet time. Maybe I can talk a bit more about um, about our about our um, our user. So, right now when she uses the product after we build it, it'll be 5:42 on the last day of the quarter. Um, she's just closed a $12 million contract. Uh, CEO wants to announce the revenue the next morning and she just needs to know can I recognize this revenue today? She has 2 minutes, so she oops, needs needs some actual help. Here we go. All right, um, excellent. So here we have our first version. Um, and so we have, let's see, where are we? Um, so we have the brain here. Um, this is connected to the to cursors. I guess it's Grok is the model. Um, so let's ask it a question. Um, what is VAT? This is a fun, oops. Sorry, one sec. Uh, okay. So I have to actually add the arms. Hold on one second. One second. My little notes are giving me trouble. do do do All right, here we go. This is what I forgot to do. Here we go. Getting ahead of myself. Um, okay, so now we need to I'm going to ask cursor to uh, connect this application to open AI uh, using the responses, oops, responses API. Um, and then when the user presses submit or hits enter, Uh send the question to GPT and display the response. Hello. There we go. So we have a nice little uh Why? But that is not the important part. There we go. Hopefully this is good. So zoomed as it goes. All right. Here we go. Uh again It's fun at uh at Stripe we have um a lot of internal tools that are super powerful and uh they don't have sort of like standard um you know, uh thinking or you know, it's it's like undulating or um you know, uh uh dancing along to the music or yeah, makes it kind of fun while you wait. >> [laughter] >> Here we go. All right, excellent. So now let's try a question here. So now what is that? Let's see. It's thinking. And it tells us what that is. Uh which is value-added tax. It's uh any uh form of tax uh consumption tax, sales tax or otherwise known as indirect. Um so while this answer is plenty fine for a person like Sarah, um this actually would be a dangerous product because it is using general chat uh chat GPT as uh as the brain. It is not tailored to the business. It doesn't understand the nuances of Sarah's business um and any sort of answer that it's giving is obviously going to be rather generic. So, what we want to do is add rag, um, which is basically asking it, uh, we're going to ask our tool to get company specific context and then fold that in, um, to all the answers. So, search our documents, take the relevant information, and then give that to the GPT and create a more specific and more accurate, uh, answer. So, now, uh, we're going to build the rag. Give me a moment while I find um, company policy. Oops. So, I've created a little doc called, a markdown file called company policy. It has all the important things, um, about my business, I should say. Should say Sandy a signal. Um, and, uh, it's basically different policies and other critical, uh, critical knowledge. So, now I'm going to ask Cursor to please, um, instead of answering from general knowledge, uh, first read company, oops, uh, policy.md. Include that content, uh, in, um, in the prompt. Um, before asking GPT. Let's do, let's see [sighs] how that works. Do do do >> So now um So now as the CFO you you're really not placing your trust in broad AI. You're trusting that it is capable of understanding your critical context that you have input. Um and this allows efficiency and scale as well as um effectiveness. Perfect. All right, let's see. All right. Now I'm going to ask a question which is um do do do can I recognize revenue before implementation is complete. Submit. Okay, according according to my company's policy, revenue revenue revenue is recognized when the company satisfies its performance obligation. So no, um so we would have to wait till the subscription period ends. Um so that is definitely uh what's not making the model smarter, it's it's just making the answers more accurate because of the context. All right. So now um let's see. I want to uh I actually want the more proof though because as my questions get more complex, uh I'm going to want to understand if the model is drifting um or if it's really leveraging what I have in my context. So after every answer uh include the exact section of company policy.md uh that supports the answer. Please. >> [sighs] >> Um I guess bring in tokens with that, please. But, it's fine. Um okay. >> [clears throat] >> Uh let's see. Um while this works through basically, we're just making sure our instructions aka our prompts just get better and better more precise again around the things that our persona cares about. So, accuracy, provability, explainability. Um cuz Sandy's the person who's going to get called on the carpet um if you get audited or when they get audited. All right. So, um there we go. Let's ask another question. Um let's ask another question. Actually, maybe before we do that, let's go ahead and let's make it I feel like we need something to indicate how confident it is in its response because we may not have given it all the context that it needs to be confident. So, before we ask another another question, >> [clears throat] >> I'm actually going to add another feature um which is uh confidence scale. So, similar to a human being able to say, "Mhm, I'm 90% sure." Um so, ask GPT to return answer confidence from 0 to 100. Um uh with an explanation. What was the other thing we had? With an explanation and the source section. >> [clears throat] >> There we go. Had to remember. All right. So, we'll wait for this to roll through. >> [sighs] >> Do do do. And as I'm thinking, so I asked for a 0 to 100, but I'm also I mean Sandy's paid on time, so maybe color coding is just a nice and helpful feature. So, why don't we go ahead and actually ask one more thing if confidence is above 90, uh show green. If it is between 70 and 90, show yellow. And if below 70, show red. Cool. So, that's going to work through. And probably actually the last thing that I will add for real, I think the last. Um I think that we should have a flag to queue out basically things that need human review. Um so, let's see. Uh um uh if confidence is below 70, so red, instead of giving me an answer, >> [clears throat] >> display human needs review. Or needs human review. >> [laughter] >> I'm one of my own machine. Needs human review. There we Cool. Amazing. There we go. And [clears throat] so now uh let's ask an actual question here. Uh so I'll refresh just in case. And then let's see. Um can we ooh recognize revenue before signing a contract? I think we humans know the answer to this silly question. But confidence 90%. Oh, my context is bad. >> [laughter] >> Okay. Okay. Okay. Why don't Why don't I ask a sillier question? Um uh how do Martian taxes work? Let's see. Oh, I need to even review. Uh all right. No evidence is found. It gave us 0% on this silly question. Um clearly I need to improve my context file because um yeah, candidly, that uh that that would did not work. Um but yeah, so I think I think the critical piece here to sort of lay out is um you know, we went from uh a product that could answer a question by just pulling from this like broad pool of knowledge through ChatGPT. Um but we've now done a simple sort of design of asking the question, pulling from our company knowledge, formulating uh you know, reasoning through with the GPT, the big brain, uh and then giving us confidence, giving us evidence, and then pulling out the things that actually need human. There's a saying ever since I started working on products with machine learning, you know, or just in like actually just building digital product product products in general, you always want to find like what are the things that computers are good at and just have computers do all the computer things. And then what are the things computers are bad at and like like humans are good at and the humans should do. So it really should be this like make the machines do all the things that a machine can do and just reserve the value judgment of a human at the top and that's true for this, you know, CFO copilot is true for product management in general is true for true for products today and as as these things get smarter they should take on more and more for us. And then I want to kind of like point out the obvious which is with certain personas and certain needs like trust is paramount. I I think for a lot of consumer products curiosity and splash and bang is like fun and it can generate a lot of engagement but for products like these it's it's about the accuracy, it's about the provability, it's about the explainability and it's about just building trust and that is much harder. It is it is one in inches and lost in buckets or whatever the saying is. Um So yeah, so I think that that's our little product for today. I hope you guys had some fun and thank you so much for having me product school.