AI Product Lead at Typeform | AI Research to Decide What to Build Next in Hours, Not Weeks

Hello. >> Hi everyone. I'm Sun. I'm a lead product manager at Titeform working on our analytics features and I've spent years building products and what's been the hardest part about being a PM and actually building is having the confidence to know that we're actually building the right thing. >> Exactly. Uh I'm Lee. I'm the senior director of research at Type Forum. So even though I'm the senior director, I'm actually 95% IC. I spend much of my day, the majority of my day executing research, running research, not just overseeing it. I have a PhD in social psychology and for many years I actually studied close relationships like romantic relationships, uh, relationships between family members. Then I moved into studying the relationship between mental health practitioners and their clients. And then finally, I moved into studying the close relationships we have with our closest technology, which is how I transitioned out of academia into tech. I'm also a research methods professor at UC Davis and UC Berkeley. Um, the reason I'm telling you this is because what we're going to share with you today is essentially a new research method, a mixed method AI automated project um product. And I want you to know that as um a researcher and as a as someone who communicates research lessons and best practices to hundreds or thousands of students every year, I need the kind of tools that we use for mixed methods research to actually be reliable to be something that I trust to be something that I feel are rigorous and follow best practices. And it is not easy I would say to like hit that bar for me. I don't want to embarrass myself by using tools that don't work. I don't want to be discredited as a researcher by using tools that don't work. And so what we're going to share with you today is a tool we've integrated into our tech stack that has changed our lives. I know that sounds cheesy, but has changed our ability to do research quickly. And so um I know Sun introduced herself as a PM, lead PM at type forum. She and I have been working with each other for two years, two and a half years at this point. And I think something that I've learned as a researcher, director, and I see is that when product and design and research actually collaborate well together, that's when we're able to create the most value. And that sounds obvious, but in practice, it's actually really hard. I can think back, sorry, son. I can think back to the first couple of >> Yeah. that son and I were collaborating and I was like delivering research to her and then she'd like come up with her next roadmap items and I was like is this based on the research? Is this based on any research? And she was like girl I have priorities like yes it is based in part on the research but I'm managing these other things. I'm managing tons of other tradeoffs. Um, and I'd say it was a positive relationship to start, but that we were really trying to negotiate how and to what extent she could bring in-depth uh, intense quantitative and qualitative research into her process quickly so that it didn't hold her back. And I know that we're on a a webinar with AI product builders, product managers, designers, and we know what those competing priorities are, right? Like you have a road map item that you're working on, and then the leadership team has an idea that they think would be worth investing in. You see a competitor launch a new product that makes you kind of rethink potential strategies, how you're responding to this. you maybe have some customers who are giving you feedback saying that this is what I would want or this is what I would need to upgrade to the next plan etc. And as PMs and designers we're trying to manage all of those competing priorities while trying to stay close to the customer and have really good research and data that validates what you're building so you can build with confidence. >> Exactly. And then researchers who want to support your process and your workflow with that data um can't always move as quickly as you need to move. I'm one of the researchers that does actually believe we slow people down and that that's not always necessary. Sometimes we can move lightning quick, but like the main complaint I will get from stakeholders is like look, I need this in two days and I have really strong intuition uh and experience and data. I've been collecting for the direction we need to move in. Right? But if you want to you bring research into the process formally, I'm going to need it in 48 hours, right? And so there's those moments where my team can either drop everything we're doing based off of the urgency and the risk and make sure we get those data to teams or to be frank like we can't deliver at the speed that they need us to deliver and we end up having to move forward without research or we used to have to end up moving forward without research. Um so what we're going to walk through today is this new product that type form has created called research flow. Before son jumps in, I do just want to get real for a second and say that eight, nine months ago when the teams were initially building this, they brought it to my team, right, to test and to trial and get feedback on because we're buyers of the product, right? We're power users or we would be power users of the product. At a company like Typform, there's 200 plus people at the company, only four researchers, and they were running it by us to get our feedback on whether or not it made our lives easier, whether or not it could replace some of the tasks we were doing. And eight months ago, I was like, hard no. I was like, this is making my life harder. I would never use this tool. And so, we worked in conjunction with the PMs, with the designers, with the engineers. They took our feedback very seriously. We also collected feedback from dozens and dozens of customers who might be interested in using this type of product. And we I'd say we turned the corner really in late January, early February where my team did start using it, replacing competitor tools with it, replacing our own workflows with the tool. And I remember if you had seen my feedback at the end of last year, end of 2025, where I was making people cry with my feedback like hand to God, making people cry, giving them feedback about what the ways the tools letting us down, right? And the ways we needed it to step up to now where I can't imagine, I know this sounds cheesy, but like I can't imagine not having this tool in my tech stack. I can't imagine taking on the work that we take on, especially as a four-person team without this type of tool in our tech stack. Like it is night and day and it's still being developed, right? New features are being released every time I come into it. But it's really the question of like what do we all need? What do researchers need? What do non-ressearchers doing research, doing serious research need to get their jobs done? And so with that, I'll actually uh hand it over to Sun. We're going to jump into the interface and show you what we've been using and how we've been working. >> Yeah. [clears throat] So, before we even go into the user face, let me just give you like a highle overview of what research flow is. Building off of what Lee just said, you know, every PM, every designer, anyone who's working under time constraints knows that you're wanting to both have confidence in what you're building, but you're also making trade-offs every single day. And the trade-off that I oftentimes make with research is surveys are really fast. They're easy. I can maybe launch a survey without even having the research team help me because I can draft multiple choice questions and just send it out into the wild. However, a survey just tells you what happened. It doesn't really give you that color or nuance. And that's where I would typically want to get on customer calls and do moderated interviews. But anyone who's done moderated interviews knows how painful that can be. You have to schedule them. You have to be extremely present and engaged for the 30 to 60 minute session. You have to take notes. You have to look at the transcript. You have to find patterns across all of them. And it's it's painful. So you're oftentimes choosing between speed of a traditional like quantitative survey or death which you get in those moderated interviews. And you >> Yeah. And it's not just painful girl. Sometimes it's straight up impossible. Like if I need to be if you want to talk to 10 people and I'm talking to them for 10 hours each, that's at least 10 hours of work because 10 hours on every call, however many hours it took to find them. And then maybe like two to four hours re-watching the hourong video. So I got to express my pain [laughter] around >> I feel it too. >> Yeah, I know you do, girl. I know we all feel it. And [laughter] so >> please continue. Yes, >> it's great when you have a research team like Lee at Typform, but like we already talked about, there's all these constraints. And so what research flow does at a high level is that it's a single tool that runs both qualitative and quantitative data. You, the researcher or the PM designer, can set the structure of the research that you want to run. And an AI moderator takes the context of your research goals, the questions that you've asked, and follows up with the respondent to learn more. So that's a high level. And now that we're in the product, let's actually talk about how you would start using this tool. Let's show you how the research agent even helps you with the study process. And I'll turn it over to Lee to just walk us through that. >> Yeah. So let's just imagine and this is the study that we're going to walk you through today like the one we ran end to end that you're interested in the extent to which people trust AI tools. So what you're looking at right now is the very first stage in the research flow interface where I can straight up drop a question in something sad and busted, right? I need a one to five question asking how much people trust AI. This isn't me thinking really deeply. This isn't me uploading a 10-page research script. It's me moving as quickly as I can to start generating study questions. So, what you'll see is that this is a chat interface. And so, I'm able to say, "Look, I need this type of question." And it's going to think about what I've uh prompted it with just like any generative AI chatbot you've seen, except for this one is trained specifically on research methods. So, it's like happy to help Lee to make this study useful. What specific aspect of trust in AI do you want to measure? That's actually a very good question. overall trust, trust for work decisions, trust in accuracy, trust in how AI uses data. I want to be like, I want all of those. I want to hear about concrete experiences. So, you have this ability to go back and forth with the agent just like you would with any generative AI tool, except you know you're talking to an agent who's been trained on best practices. They're going to probe you until it knows enough to start generating study questions that are meaningful. So it's saying that helps. This study is about understanding overall trust in AI, understanding, sorry, including trust for work decisions, accuracy, data use, the real experiences, shaping those views, etc. So just wanted to show you really quickly how you can go back and forth. But you can also just upload some questions into this uh interface if you already know what you're gonna say, what you want to ask. Like Sun and I might have different experiences, right? I might already have a full script written out because my job as a researcher is to build these types of questions, but Sun, I know that you have a different experience when you're generating your research questions for studies. And as I've used our tool, one thing that I found really helpful is like I oftentimes have an idea or some question that I'm trying to answer. But what I sometimes forget is to think about how would I actually take this research and present it to a stakeholder that I'm maybe trying to influence. Is it my leadership team? Is it the engineers that I'm working with? Is it the designer that I'm partnering with on this? And so it will really help you narrow your hypothesis and be crisp on what you're actually trying to get out of the research, which is so critical because even though this tool helps so much, having more clarity in what you're actually trying to use the research for is only going to make you better as a product builder, especially with all the AI slop that's out there in the world. Like, you need to be critical in thinking about how do you build meaningful products and that's always been the core of building great features. And that's not changing with AI. It's just now a different landscape that we're working in. >> Exactly. And just to call back to what Sun just said, you can see that the research agent here does ask, "What decision will this research drive?" And that is definitely one of the questions we often find stakeholders aren't thinking about at the very beginning of collecting the data. The best PMS and designers always have it on their mind. But just in case it didn't occur to you while you're building out those study questions, interview questions, survey questions, focus group questions, usability testing questions, whatever types of questions you're building, um the agent is going to prompt you with the same types of questions that I as a researcher might ask you if I was planning to run this study on your behalf. And so you can see it's it's actually prompting here. For example, whether to shape type forms AI product direction, probably. How to position AI features and messaging. Yes, probably. Or which trust concerns to address first. So, it's even giving you thoughtful recommendations of how you could use this uh the research moving forward, which listen, [laughter] it's actually hard to ask these questions and then to build a study that reflects not just what you want to ask respondents, but then what you want to do with that data later. So, you're going to be able to do all of that um in this conversational chatbot. So, now that we've showed you this helpful experience, I want to actually take you into a study that we've launched prior to us joining this webinar because we're all AI product builders and we know that we have [clears throat] a lot of questions here when we're building things. And so, we wanted to run a study not asking people like whether they use AI because basically everyone uses AI now. We want to get into how they use it, how much control they're willing to give AI, in what circumstances would they want to give more trust or less trust. And so let let me walk you through how the study actually works. So like a traditional study, we have a welcome screen. We added some screeners. So this is to make sure that we're getting the right people to take the study. And then we have research questions. So you'll see that these research questions are really around the idea of how do we have people um engage with AI. The first question that you'll see is a multiple choice question very similar to one that you might ask on a traditional survey. How much overall how much do you trust AI? It's on a one to five scale. And if we just use a traditional method, all you would get from this is a distribution and maybe an average number of what the average trust in AI is. But we know that as builders, we really care about in what circumstances does this change? There's so much nuance to this and you don't get that with just a straight multiple choice. So, you'll see that we have up to three follow-up questions added, and these are the the questions that the AI moderator will probe specifically launching off this question to get more detail and color from the respondent. Overall, this study has five questions. You'll see um there four multiplechoice, one open-ended. A couple of them have follow-ups, but it's really about how much do you trust AI? What would make you trust an AI tool more? How do you interact with AI tools? Describe what tool you use the most and what reflects your ideal way of working with AI. And so I want to give you a taste of the study that we actually ran because Lee's going to run through the insights, but we keep talking about like this AI moderation experience and you're probably like, what does this actually mean? So, I'm gonna turn it over to Lee to do a preview of this study so you can actually see how it works in practice. >> Exactly. So, I'm just going to do a preview. I'm going to do it in real time so you can see how this agent interacts with us. Okay, I'm going to start the study. First, we've got to set up the interface. So, it's letting me know I need to turn on my microphone and camera, though. Sometimes we use video, sometimes we use voice, sometimes we use text. It really depends like what is the most appropriate modality for the audience you're going after. Sometimes we know people don't want to be on camera, right? And we are collecting data that's maybe more private or confidential. And so we'll we'll send them to voice or we'll send them to text. But for this study, we're going to do video. So let's see. Get set up here. Can you hear me? It is Lee. Can you see me? It is Lee. You can. We're ready to go. All right. I'm going to make this a little bigger. Start the study. Okay, agent. Okay. Hi, I'll be asking you some questions today. Have you ever used AI like Chat GBT, Gemini Claude or similar tools? Yes. Literally, I think of ChachiBT is the fifth member of the research team. I refer to him as Japetto. >> And these are the screener questions that we set up at the very beginning. >> That's right. Thank you. Can you think of at least one concrete experience you've had where the AI tool dis where where the AI tool disappointed you or let you down? Yes, I can remember an experience where AI has let me down. This is uh like Sun said, we just got through the two screener questions because we want to make sure we're talking to people who have used the AI tools and who have had not just positive experiences but negative experiences with them. All right, thanks for your responses so far. This study explores how people use AI and when it works and doesn't work. Let's continue. Okay, let's continue. All right. Overall, how much do you trust AI? This is that traditional liyker scale question that you're going to see in a basic form or survey. There's a lot of power behind these seemingly simple uh liyker scale questions because they do lead you to have quantitative output um and they're easy for respondents to take. But I'm just going to say how much do I trust AI? Answering honestly like somewhat. It depends what makes it land at somewhat trust for you rather than higher or lower. So I don't know like sometimes it's hard to trust AI. So Lee's response here is a classic example of what a respondent would actually say. We have a pool of respondents who are like, I'm busy. I don't really want to take this study, but I'm just going to do it. And so they're going to just give you fast, quick answers because they just want to get it over with. And that's when you would want to have a true moderator in the conversation to probe more and to get more details. Then you have this other pool of respondents who are like helpful, want to are engaged, and think that that response because it actually just straight up answered the question was all you needed. But it's very shallow and surface level. And in both of those situations, you would want someone to be like, "And tell me more." And so, let's see how the AI moderator takes Lee's response and follows up on it. Because you'll see this question, what makes it land some at somewhat trust for you, wasn't in our original research outline. This is an AI moderator follow-up. >> Exactly. And so, I was just like, I don't know. Sometimes it's hard to trust. So, let's see how it handles that. thinking. When does it feel hardest to trust AI in real life? Tell me about a recent moment that made you hesitate. Okay, this is probing me in a way I would want to be probed. I could still give a shitty answer if I wanted to, but it's trying to be more pointed here. When does it feel hardest to trust AI in real life? It usually feels hard when I feel like it's output something that is not accurate or I can tell that it's output something that's not accurate. I use it a lot for uh data an analysis, coding, etc. And there are times it spits out stuff to me. I feel like it's going to work and then I try to use it and it doesn't work or it doesn't complete my task successfully. So that's a that is a moment where it feels hard to trust AI in real life. Move to the next one. I now feel I'm in conversation with this tool. I did talk for a long time. So, it's going to think through this. That mismatch between confident output and broken code is a rough one. It is. What types of things would make you trust AI tools more? That is, what would make AI feel more like a trusted teammate? Uh, it's like when I'm actually collaborating with a trusted teammate, there's a lot that goes into building that relationship. So, I'd already want them to have verified that they're an expert. I want to already feel like they know what they're doing and that I can hand off tasks to them. But more concretely, like when I'm dealing with broken code or output that's not reliable, I want a way to verify and check that the output you've given me is actually something that's going to be functional, efficient, and worthwhile. And notice that first sentence, that mismatch between confident output and broken code is a rough one. Those kinds of connections between questions that the AI moderator asked really makes this feel like a natural conversation. So, as a respondent, you feel like you're actually in conversation with someone that's listening to what you're saying. It's not going to ask you repetitive questions that you're like, I just answered this. Which is what you sometimes get when you're just relying on like logic and branching or if statements like this moderator is really thinking about what is the researcher's goal here and what has the participant actually said. Mhm. Okay, I'm going to drop out of the preview for now because we just uh wanted you to get a little bit of a flavor of what is actually happening when the agent is interacting with a live participant. And so you'll see this is the study and then after you actually have this study, you need to find people to send this study to or you actually need to start collecting the data. So Lee's going to share with you how we've used this tool to collect data for a whole host of studies that we've run. Yeah. Okay. So, finding participants is one of the most challenging um elements of doing research for most teams, whether you're a researcher or a non-ressearcher. If you have a list of your existing customers, that's amazing. And you're able to build this study out, publish your edits, create a link, and just share it with whomever you want to share it with. But sometimes, and I cannot tell you how often, you really do need larger samples. You can't just talk to three customers or that one customer over and over and over again depending on how large your company is. So one thing that we really make use of quite a bit is this recruit panel. So we're able from inside the product to source respondents or participants from prolific which is a great panel when you have like BTOC questions. You want to talk to the general population. You want to understand the everyday average human or a particular segment of the population. women who have recently had a kid and are breastfeeding and are using um considering using formula. Uh but you also have user interviews built into the platform for those deeper um um conversations you need to have with experts like buyers who are making buying decisions or experts in a particular industry like medical doctors who have very unique experiences that you need to learn about in probe. So you can see that we've already launched this study to one recruitment group. We gathered 25 responses and I had a handful of filters on. What I am actually going to do because we're going to walk through the results for those 25 people is I'm just straight up gonna launch this to another 25 people to show you how quickly the tool can actually locate the right respondents, screen them, administer the mixed method interview survey blend to them and synthesize the results. So I am gonna put one >> one thing uh because of this study we don't need a specialized set of audiences. >> Oh yeah best. >> Good call. Thank you son. I was like about to recruit from user interviews. We can actually go to a more generalized sample as son mentioned who hasn't used AI. So almost anybody um over a certain age has probably had experience with this tool. So I'll go ahead and use the prolific sample. Thank you girl. [laughter] And I'm going to put a primary language on for English just because I want to make sure the results are in my first language so I'm able to consume the responses. But a a note is that if you wrote your study in Serbian, um, Spanish, French, Croatian, right, the agent can deliver the study in that language and it will also generate the responses in that language, which has been pretty important for us because we do have an international market and are trying to reach people across the world. But I'm going to put a filter on for English. I'm just going to ask for another 25 people. And I've already put in a little bit of information about the study. It's a five minute study about trust in AI and I'm going to just publish it and launch it and we'll see if in a few minutes we have actually started to get more responses in addition to the 25 we initially collected. So publish and launch says 24 to 72 hours. I think it can move quicker. TBD. All right. Got But we're doing it live so we can really show you like if you were to use this tool and you needed insights because you had a meeting tomorrow with your stakeholders and you wanted to say here's the data to back up my road map item or my design choice. >> How fast you can actually collect the data. >> Exactly. So as of this moment 7:30 a.m. Pacific time we have zero out of additional of the additional 25 responses. I'm even tempted to just refresh now. And to remind you, we already collected 25 responses earlier, and those are the results that I'm going to walk you through. So, we showed you what it's like to actually build the product out. Sorry, build the study out, build your research questions, add welcome screen, screener. There's other elements you can add in as well. You can customize the design, etc. We showed you what it's like to share it. You can either create a link to send to people or you can uh send it to an internal panel. And now we're going to look at some results. So this is my favorite part. So [laughter] whenever um I'm working with a new teammate or a colleague and onboarding them to using this tool um because my team has really started to push people in the direction of using this research flow tool especially when we can't manage or service every request that comes through to our team. I'm pushing them in this direction. But the thing I want them to get to is the insights because this is where so much of the value lies for me. We've got 25 completed sessions here and this truly would have taken me around 10 to 12 hours to field this research. And so getting to the place where I can learn from what the AI is extracting from the raw data in just a few hours or just a couple of days or just a few minutes in some cases. That's where the timesaver is for me. Like I don't know. It's it's it's it's hard to express how like my conflict in this space where like on one hand I'm like is this just replacing me? And on the other hand I'm like yes just replace me. There's other value that we bring as researchers to companies there's additional research that we really do need to have a human in the loop for. But so much of the work that we do, we have smart, intelligent. I know those are synonyms, but smart, intelligent, sophisticated, thoughtful, creative, serious product partners and design partners and engineer partners and GTM partners, right? Marketing partners, PMM partners that are able to build a study, especially with the support of this kind of tool. So I don't know game changer is also a cliche but it has 100% been a gamecher for me and my team. Okay so let's look at some of the things. I'll just orient you really quickly. You get this executive summary that pulls out the top key insights that are aggregated across all of the respondents and all of the questions. Sun and I went back and forth with whether or not the executive summary was valuable. And at first when they were building the product I'm like no no I'm a researcher. I'm gonna dive deep into every question. I don't need an executive summary. Do you remember this sign? >> Yes, I do. >> I [laughter] was like I was like, "No, if you're taking the data seriously, you're just going to go in and you're going to look at every question. You're going to read every response. I don't need an executive summary." Same with the highlight reels. I was like, "I don't want highlight reels. I need to go in and I need to find the perfect clips." But as you can see, you saw that first clip, the second clip, third clip. One of the things I want to point out here is I'm already seeing gender diversity, ethnic diversity, location diversity. Some people it was light outside, some people it was dark outside. I'm seeing age diversity, right? I'm realizing that when we have this type of tool, it is incredibly useful to be able to have summaries at the top. This gave me a quick view of like who was I talking to? Was I talking to the same person in every single respondent? And I'm already like, no, no, I've got a great cross-section of the population here. And then the executive summary, being able to take all of the data from across all of the questions and all of the participants and synthesize it into a few highlights. I will be honest, like for the first two months of working with this tool, I was fact-checking every single highlight. I was reading every single response, watching every single video, and I finally got to the point where I realized like this is doing a as good of a job, if not a better job, no, a better job than me. It's certainly moving quicker than me. And if anything else, it's a great jumping off point for me to do my deeper dive. So, let's see some of the things it's learned. >> Yeah, let's see. Okay. Accuracy and verifiability are at the core are the core trust blockers. No surprise there. Uh users want AI to do the heavy lifting but not decide alone. That's right. Technical tasks earn trust. Emotional depth remains off limits. Oh, that's interesting. I can read that a little more. Data analysis, code interpretation, and general Q&A generate clear satisfaction, making structured information work AI's strongest zone. Creative tasks spark enthusiasm but also hesitation. Well, anything requiring empathy or nuanced human understanding is where users draw a firm line. Very interesting. Um, okay. [clears throat] So, this is that highlevel set of insights you get. I spend a lot of my time here. I think a lot of stakeholders spend a lot of night their time here and just jump into the highlight reels. you can actually watch them and hear what each person had to say around the main key takeaways that were surfaced. But here's here's where we kind of started this conversation. Sun was talking about how you might have this question overall. How much do you trust AI? Classic liker scale. I can look at this graph. This is what you're going to see in a traditional survey. And I can see easily from my first 25 people trust is falling somewhere around three and a half out of five. Right? Can I really take that back to my stakeholders and expect them to be satisfied and expect them to feel like they know how to build, what to build, what to avoid if they're trying to build products that engender trust in AI. No, I cannot because what happens is I go back and I'm like, uh, trust is about three and a half out of five. So, it's not on the low side, it's not on the high side. I can even give them this um summary that's uh a synthesis of this quick bar graph, right? But the first thing my stakeholders ask me when I'm like a trust is about three and a half out of five is okay great that is interesting but do you know like why >> yeah exactly why >> and I'm like we didn't we didn't we didn't ask that question right you told I showed you this question I told you I was going to go in get an average trust right but at the end of the day we have to be able to answer why if we want anything to be actionable That is where I'm able to just go to the follow-up questions. So, what this is is a synthesis of every follow-up question that was asked to every respondent probing deeper regardless of what they selected here. They're going to ask, "Well, nobody look at that. Straight up, nobody trusts AI completely." But whether they said they trusted AI a lot or they said they trusted it a little bit, right? The AI is adapting its follow-ups to that initial question and diving deeper. And so just going to look at a few of these really quickly. Accuracy doubts define the trust ceiling. Yeah, for sure. We already saw that theme. We're seeing trust is conditional on the task at hand. Yes, makes sense. But ultimately, let's just stop here because I just want you to see a little slice of how I would how I use this in real life. It's like accuracy doubts define the trust ceiling. I'm like, okay, that's intuitive to me. I don't feel like that's wrong, but I need to see the receipts, right? And my stakeholders need to see the receipts. So, I'm able to come into the topic. The agent is going to surface the top quotes for me. So, no hallucinating of quotes. No, no quotes that come out of nowhere, and I can't verify those came from an actual participant. I'm actually going to look at the negative one. Let's see what they actually Let's just watch it. So, this is going to allegedly be in support of this idea that AI is not accurate or reliable. We can jump in. Let's take a look at this person says. Okay. Hi. Hi, friend. Here we go. >> Roster players, it was giving me the wrong information. Coaches, I'd asked a specific question about who held the Florida Panthers rookie goalc scoring record, and he told me it was Alex Oveskin, who's only ever played for the Washington Capitals. Uh, you know, so it just really blatantly wrong information. >> Okay, that was compelling. That would upset me if I was watching this and that were my product. the PM. Like what I love about this is as helpful as it is to see, you know, the written out explanations and the charts, sometimes you just need a really good sound bite to get people to understand and empathize with what our users are going through. So this person who's like a big sports fan who's like it was straight up wrong like is sometimes the most compelling piece of evidence that you can have when trying to influence to support your road map like it's unfortunate that sometimes you can have all the data in the world and one sound bite can be more >> influencable that's the reality of the situation and research flow you know Lee talked about how you can do texton audio Audio only, audio, video. We chose audio and video because in these situations, like seeing him roll his eyes, seeing him engage, like take a pause really shows like how authentic people are in these responses. >> Exactly. He's like his tone of voice is so critical here. And yes, girl, I do agree that. And whether, you know, whether or not I like it, a really compelling personal clip of a real person talking about their experiences is usually the thing that moves the needle. But for people like me who are like, "Yes, thank you for surfacing the top quotes. I find these helpful. I actually need to go and have an easy place where I can look at everything." And so this is where you can come in and you can actually break down. It seems that the 24 respondents who mentioned this topic, 24 out of 25, they mentioned it 90 different times. So, we can see one person had a lot to say about accuracy and reliability. H such it's such a good range. Sorry. Such a good range of participants when I'm just seeing who they are and where they are. And I'm like, I'm going to hear lots of interesting ways people don't trust AI. I can split this up by if I actually want to go to stakeholders and communicate, look, this is how we're feeling, then I'm going to zoom in on the negative feedback. If I want to talk about where we're succeeding and what we're doing right and how we want to just push that further, optimize it, I'm probably going to focus in on the positive feedback. And so fundamentally this gets me to a place where I can just answer almost every stakeholder's questions and push uh push data in their direction for whatever their need is. Whether it's improving something uh because we know it's broken or not effective, whether it's discovering something new because they uh want to develop a new set of features that aren't in the product yet or a new way of conceptualizing the design in a way we haven't thought of. I can also give evidence to the board, right, or the executive LT around the places we're doing really well and pull up those highlights in those clips. So, I know we're coming on time, right? I know. So, >> we might since we ran the panel earlier. >> I'm gonna check. I'm gonna check. >> We should refresh the analysis. >> All right. >> Refresh the analysis on the insights page. Lee, >> first we got into 37 complete from 25 just in the time that Sun and I were talking. And so I am going to refresh the analysis. You can see we've only got 25 here. So what in like nine minutes? And >> and then let's go to the sessions table while we have a couple of minutes and just show people how you use that. >> Perfect. So it's like I see more people are coming in. But while that refreshes, one of the features that I also really love is while the insights aggregates data across all participants and all questions, sometimes you really need to do a deep dive on a single person, one respondent. So what you're seeing in this table is um each response has its own row and you've got their responses to every question, but you also get this brilliant little summary of this person. So I can see this respondent uses AI for both personal and work tasks. They use it mostly for content creation. They identify as a creator. And then it gives me the main findings for this one person, their motivations, their concerns. And so I can actually zoom in on them without having to read, "Yes, I love this person. Look at their sunglasses." I can zoom in on this person without having to read the entire transcript and have a quick overview of who they are because maybe they're a particular customer that I want to highlight in a deck. And so, okay, let's come back and see. All right, we're refreshed. We've now got 37 responses. Still, nobody trusts AI completely. But what you're going to see is that the insights themselves here. I know we didn't go through all of them, but they've already updated. So, we've brought in an additional 12 people and it's synthesizing that information immediately, updating the topics, updating how much of the sentiment was positive versus negative in real time. um updating some of the quotes previously um and updating the mentions. So there we Okay, I feel like we're close to time, but we wanted to show you how we use this product. Yeah. >> And if you're interested in bringing research flow to your team, we're offering a special product school attendees discount of $500 off your new research flow plan and we'll share a link that you can connect with our team to learn more and take advantage of the offer. We also didn't share with you this actual study, but you can actually share this link publicly if you want to show that really quickly. >> We'd love to send this. Um, you have to turn the hidden on to visible. >> Thanks, girl. >> That's for privacy reasons. Um, so you can then see and access this yourself and see the different responses and see like as responses come in how the nuance and depth changes. And there's a lot of really interesting things here that we didn't even get to cover with just the study with five questions. >> So, I think we're at time, but thanks so much for hanging out with us for a little while. We can stop the demo. um hanging out with us for a little while. Um we're also always available if you're thinking through like how do I um use this kind of tool or even this kind of method, right? Whatever tool you use to get at quantitative and qualitative data in one study. I can't think of a more powerful method to use from going from like your output being three and a half out of five to te paragraphs and paragraphs of text video clips that have been organized um organized by topic and sentiment for you. This is just a massive jump and I'm at the point where I don't even run traditional surveys anymore unless there's a really strong reason to not bring in the qualitative data. like why would I not bring in that additional color, the tone of voice, the body language, the eye rolls, the smiles, the frustration. And so I think it just gets us closer to real human experience instead of just clicking buttons on a survey and not really paying attention. That's what we got [laughter] >> much everyone. Right. Thank y'all. Thanks y'all.