AI is Transforming UX Research

So, my talk is about AI with research. And um so, I figured I'd give you a quote cuz you got to have a quote. So, we're in a revolution. We're talking about fundamental shifts. You got to embrace AI. So, let's give it a try. Let's see what it does. And uh I would argue that yes, you need to keep trying these things. So, um just to give you an example, I was like, "How should I introduce myself?" So, of course I asked ChatGPT. And so, so here you go. So, I like ice cream. It didn't get that. And gelato I actually prefer. And uh let's see, like walking my dog. Didn't mention that, but it was did pretty well. Um All right. Well, let's just keep on going. Cool. So, here's what I want to do. I want to tell you about how our research at um Brilliant Experience, my company, um has changed. And how sort of the world of the researcher, the roles are changing. And a little bit of predictions of where things might be headed as well. Okay, cool. So, I have done a little bit of I've interviewed some of the founders of these tools. I've tried a lot of them and sort of tested them formally. We also did a uh study where we had um I did research the good old 2022 way of humans interviewing humans. And then uh we did humans with AI assistance just on like, you know, getting the interview guide stuff and going and things like that. We had another condition where we had AI interviewing the humans. So, I'll try and keep this all straight. And then lastly had an entire synthetic one as well. So, that's on our blog page, but the key point is we've, you know, I had a fellow PhD in psychology who should be a good researcher, too. We got, you know, or the I should say the AI moderator got about 80% of what we did, and we were kind of shocked that we were all set to hate it. And And we were we had to take it seriously. So, that got us diving in and starting to interview some of these founders and things. Okay. So, let me tell you about how our research probably just like your research, but just in case, I'll let you know what's happening. Okay. So, for us, I looked it up over many many many studies. It took us on average about 7 weeks to do a qualitative study where it was a big study, you know, international and things like that. And we've gone from 7 uh weeks to do things to compress that all down 85% to 7 days and still have high-quality research. And so, now our researchers, instead of doing a lot of the interviewing and um other things like scheduling, etc., almost all of their time is spent in analysis, uh synthesis, um storytelling, working with stakeholders, and so on. Cut. So, I'm not sure if this looks like your research or not, but I wanted to give you a feel for what this is like. So, we start by um our You know, you want your stakeholders, you want to know what they are asking, what they're interested in. So, of course, what we do is pull out our synthetic user. And they're like, "Yeah, I think I'd like this kind of person to ask this kind of question." And so, great. Let's go make that right here in the workshop and ask the question of the synthetic user. So, of course, we now have our data, we can go home, everyone's fine. No. But, the person often says something like, "Oh, that's not quite what I was getting at, and that's not quite the ideal person because of the way they answered this way. So, I just learned two things about what we really need to do for our research. So, it's a tool. And so, we do all kinds of things that either are dramatically increasing the speed or helping us do research globally at scale multilingually. And also just it's really interesting to see that when we um are giving a final research report that we often are being asked to do that not in a deck, not in a PowerPoint, whatever, but as a synthetic user that they can talk to. Okay, cool. So, um hands up, how many other people do it just like this? Nice. We got to talk. Okay, cool. Maybe he's hiring our company, I don't know. Um and so, uh the things that I really want you all, I know Actually, how many of you are not researchers? Cool. Okay, well, you're going to be soon. So, I want you to know that um it's really important for you to learn a little bit of prompt engineering. I definitely poo-pooed it before I started and I found out just how valuable it is. But I'm going to show you a little bit of what it means to have an AI moderate an interview with a human, what it's like to have AI analyze your data, and whether we should trust that or not. And I also want to give you just a feel for this notion of this synthetic user. So, okay, so first of all, what is John talking about? You know about prompting, so let's go right on and talk about um AI-moderated interviews. Okay, so QR code, that's for the workshop thing that we did that allows you to try being a participant. Um so, don't you start right now and start talking in the talk. But you can use it at your leisure. And so, remember to ask answer questions appropriately. And also try answering it with things like I like ice cream sandwiches, which has nothing to do with the questions of the interview. You see how it does. So, you can see how just how competent it is, how you feel about it asking you dynamic questions and so on. And so, the interesting thing is that it adapts to what you are saying. So, let's put that to the test. All right. Okay, there I am. Can you see me? Yeah, okay, good. How often do I use AI for personal tasks? Okay, this is embarrassing. Okay. Last time I used AI. Um I used AI to help me design my garden, and I got it to help me with what kind of plants are good in shade versus full sun. I also wanted to know how far to space them apart and what the watering and and other things uh I needed in the fertilizer and stuff were. And it helped me with that. Who Who, sorry. Oh, sorry. I have to adjust. I don't know how this stuff works, but you know, somebody figured it out. Um Okay, tell me the last time I used it. Okay. I uh So, here, I used it for uh designing a garden and uh helping me pick the plants for the garden. Okay. I used um ChatGPT deep research, and I also used it uh with the some of the drawing programs to sketch out what the garden would be like. The garden rocks, man. Everyone here has to come and visit. Um Yeah, so I was reasonably satisfied with AI's suggestions, though I didn't like the picture as much. Okay, so you got the idea, right? So, it was adapting to what I was saying and then asking me follow-up questions. They weren't wildly off base, right? So, there's something to this. I'm not saying it's perfect yet, but it's something we should take seriously. Okay, cool. Okay, so that's AI moderated interviews. Um So, when now we're talking about AI enhanced analysis. Okay, so remember there are times when we have like 90 in 1-hour interviews we've got or more, and we want to know what happened in those, and also inevitably there's some stakeholder that says, "Did anyone mention this?" And you're like, "Oh god, I've got to go through those 90 hours of interviews again." Um well, this kind of tool can really, really help you. And I'm just going to show you one example. This is to let you um get started to do a free um trial over at um Code Loop. It's just the standard going to the website, really. Okay. So, let me show you. So, this is actually slightly different. This is Listen Labs. It's the tool we used in our workshop. And so, um the one of the questions in our um survey was uh imagine that you had a perfect AI assistant, what would be the most important things for it to do, and why would it be better than something else? Okay, cool. So, by the time that we had So, we did this interviewing, I think right before lunch, right? Yeah, okay, good. And so, by the time lunch had happened, we had all these answers. And so, it gave us some "Hey, you know, it'd be great if it was personalized and it had contextual understanding. Great if it had memory retention. Great if it had uh simplified schedule." And then you see there's like a little tiny number six here? So, that gets me to the quotes from people. So, just to give you an example as well, so I can I can also just chat with this data, so I can ask questions about what people were were telling, and it refers specifically to the data in this. And I can also look at clips. So, So, just So, thank you all for um participating in this study. I don't know if you know, but you're now famous. So, um we did not sign a consent form. Don't tell the GDPR people. Okay, um Okay, let's see if we can get sound going. Okay, here we go. Like it sometimes I talk to a GPT about like planning my days or weeks or whatever or stuff, and it gives me some nice suggestion of like you could do this, you could do that, but then it doesn't you know, follow up or proactively bring that up. So, It's a lot of questions giving me a sense that it understands. Trying to get Okay, people were doing a great job of answering these. They were all trying to whisper in a conference room. You could heard the background there, and yet I was able to pick up the language here, and it's done the uh you know, like um CapCut and those kind of tools. Um the They'd be understanding what I'm looking for, but uh still being efficient and not wasting my time. I think it could help me with the timelines or how long the task will take and when it's due, and how important it is. That would be helpful. They can actually, for example, write I don't know. Uh a story or a character or a prompt uh that actually convinces me that it's creative and unique. >> So, my perfect personal AI assistant would be able to understand perfectly European Portuguese >> Java has a strong and be able to answer to me also for in perfect European Portuguese. to make good decisions. >> in a way if AI could hack in to my everyday life and it was analyzing everything that I did and maybe one day it noticed that like maybe I'm not getting enough sleep or my performance in the gym was like this much. And uh that could offer me the answers in many different formats uh as um documents, video, images, text with Okay, so you get the idea. It has pulled out what it thinks are relevant clips here, right? So, I didn't do any of that. And so, what I'm telling you is that within the time of lunch happening, we had a set of video clips, we had summaries, we had highlights, we had the top findings, we had a thing that we can ask questions of that answers specifically about the data. And I'll see you in a minute cuz you're like, is it just making stuff up? I don't know this hallucination stuff. So, so this tool here is called Coloop. The last one was Listen Labs. And with this one, you can see that everything is underlined. So, let's just try this one right here. Okay, so this thing about participants prioritize AI functionalities that integrate with calendars and appointments. Okay, perhaps AI could be used with calendars and appointments. Okay, so it's highlighting the things that it thinks are most relevant to this point. And it gets me specifically right to the points in the transcript. If we had videos, Listen Labs exported a transcript to this tool. Otherwise, it'd have the video to play. And you can get right to the right spot in the video. So, you you can trust, but you can also trust and verify. You can go look at all these links and make sure is it really summarizing in a way that makes sense to me. And so, we found that though that these things make an incredible time savings for us. They may not be the very best. They may not be perfect, but they're getting us further faster than ever before. Cool. How many of you are using some sort of AI analysis? All right, I'm going to beat you to the finish then. Okay. Um, let's keep going. Okay, so there's this note he keeps going on about these synthetic user things. What is this synthetic I don't know. Um, so synthetic users are really, you know, designed to represent in this case we're talking about a target audience. So, it could be some specialty. Know that there's no reason in principle why this can't be, for example, a fellow expert researcher and then I use them as a fellow expert in addition to say a consumer or a custom B2B customer and so on. Okay, this one lets you get to a tool called Verbs from Verve. And let's see um Yeah, I think I'm going to do it this way with you all. Okay. So, what I can do and so there's actually a tool um Sorry, an app that you can get and download on your phone and it's got they call so it's Verve video survey. So, they have the V thing. So, they've got Verbs in English it would be V E R V E R V S. In this case it's V U R V S. And um but what I can do is I can in Verve here I can say yes, I want to bring my data in from my real life humans that had family international travel and I'd like to bring in my synthetic user Fiona who represents them. Okay, and now I'm going to say um Tell me about all the things uh so how many of you have or I guess you don't have you have had at least at some period my kids in college um young children that you've traveled with and to like a hotel. Okay, so you all know what it is that the hotels never tell you that you wish they would have told you to help you or let you pick a room that's more suitable to you with small kids. So, um you wish that hotels told you about rooms. Like manic I type so okay. Okay, so now Fiona the family vacay planner is thinking way and using the data we had from our interviews as well as her representation of her persona. And let's see what she says. And by the way with the synthetic user you don't just get answers to questions like this. You can ask the question, "By the way, in the way that you find hotel rooms, tell me how you do that in jobs to be done format." And it will give you a lovely version of jobs to be done format. So I just want to say you can ask sort of impossible questions of synthetic users. You can ask it to you can work through one of your interview guides before you go and test real humans to see if there's anything surprising. So here let's see. Okay, so we want to know about baby items, if they I can have a fridge. Let's see laundry services, entertainment in the room, mini fridges a lot of people ask for that. So these are all really appropriate answers actually for what this is. And anyone with small kids and is anything really out of whack here? Oh someone wants a bathtub. What did they mention a bathtub? Yeah, okay does it have a bathtub? Where I put Oh oh oh whether it has bathtubs. Woo! Okay. Nice. Okay, so there you go. So again what I'm showing you is that maybe these might not be perfect today, but remember what chat GPT one or two was two years ago and what it is today. And imagine harnessing that kind of power with these kind of things. How might you be able to use these in the future in your research? Okay. Cool. So let's keep rocking and rolling. Okay. So, in general, I just want to give you a sense of how things are changing for How many of you again are UX researchers? I'm sorry, me too. I'm going to give you the news. Okay, here we go. Okay. So, I you know, I used to like Wired magazine back in the day, so they did this. Um so, um right, I've got taking weeks to do research only in American. Um versus really taking, you know, really it's a period of hours or days to provide global and scaled insights, which is possible with these tools because remember my AI um moderator can speak many languages. It loves doing testing in New Zealand when I'm sleeping. And um so, I can really do 300 interviews over a weekend easily, right? So, so it's very different. So, I can be more inclusive than I ever was. Okay. So, I personally love doing the interviews myself, just like many of you probably do. But, um you know, we really will be doing more oversight of having these tools allow us to move fast and do scale. And really be the the person that's designing the system and validating. So, you go from being a doer to an orchestrator. Okay? And I I you know, I used to talk, you know, I am a mixed methods researcher. I can do both of qualitative and quantitative things. And now I just feel like we have to adapt. We have to keep learning these new technologies and see what's possible, what fits in our organization, what's appropriate for now. But, I want you to be ready for when these get better and better. Okay. I We use them today, but I want you to start to um think with study these things and learn them. Okay, cool. Um so, let's talk about how will this change our world? If we have this AI um ability to do this AI moderation, these synthetic users, these um, analysis tools that let us scale with thousands of interviews or other things. Well, I think one thing that I know is happening, and this isn't really a prediction, it just is. And by the way, if you are UXR, this doesn't mean your salary is dropping. Just keep calm. Um, uh, so first of all, the cost of doing research is getting cheaper because I don't have to spend as many hours doing what I do. So, it's just more efficient. So, the participants, we still have to pay, but um, actually doing the recruiting, you don't have to have a custom recruiter anymore, and the the UX researcher can do many studies over the period they would have done, say, two. Okay, cool. By the way, are these all factual, you know, carefully researched things? No, these are in John's head. Okay? So, you're welcome to challenge me, but I but as I've been learning and sort of doing thought experiments, um, ChatGPT and I have been thinking about this, and so an- another thing that um, I really think is interesting is we don't really have the same limits we used to have on doing research. So, we can do kind of as much as we want because we can have these AI moderated things that give us scale. We can do things globally. We can make prototypes so much more easily than we used to, right? So, we don't have the same constraints. So, what are you going to do differently without these constraints? Another thing is that um, maybe we're not going to do, relatively speaking, as much research. Maybe it's the case that you, because you're going to have that synthetic user you can ask questions to. If you're a designer, you're a uh, programmer, really a PM, why not just ask that first and see what happens? See, you know, how that um, expands your thinking, how that, you know, gives you different perspectives. So, I think people are going to be banging away on these synthetic users all day, and that's why that's what are agentric studies. Um and I also think that because of these AI analysis tools, all the things you've done in the past, now you can start to harness valuable. And also social listening, and also customer service calls, and also um like sales things. And basically every piece that you've got of the puzzle can start to be harnessed for research. So maybe it's the case that you don't have to do so much one-off research and the strategic research is more focused than it used to be. Cuz you may not need to do research every single time. Cool. Uh another thing is that um I would argue that a lot So the reason why I said most of or you're all going to be doing research someday is because I would argue that the research teams are going to be laser-focused on the highest value, highest-impact things that you want, like desperately want right. This is the next $300 million venture. Let's do this right. And the other things, like should we put the button here or there? Great. Let's go ahead and let other and there's plenty of substantive things that we wish we had research for, but we don't. Well, now we can. So you all can do research. So the people in our class know this, but with um Listen Labs is one example where you say, "Here's the question I want to ask. Here's the kind of people I'm interested in." It builds a interview guide. It can build a um a way to recruit the participants. And then it can go and do all the analysis for you. So we may not need those those researchers quite as much. Um so so I'm saying that there are a lot of questions that went unanswered that maybe can be answered by humans as well. Cool. Okay. So um Oh, that's supposed to say gosh golly. I'm sorry. I forgot to change. Okay. Um Okay. So I I I want to tell you this. I want to tell you that um First of all, here's another QR code for you. Um so these are uh some of the people in our um workshop got these guides on sort of getting started with this stuff. They're all free. Um so you can go ahead and grab them. And um I think that's one way you can get started is to say, "Here's a little, you know, what are the things I should be thinking about?" Another thing I've got for you, same QR code, so just um remain calm. Um is that I I also do sort of free half-hour things as well. So if you want to just see these things in action, ask questions, you can do that, too. Um and I actually sort of forgot to say that I had interviewed all these founders. And so if you want to see these tools in action by the person who built them, then and you basically I asked them, "Who is this really for?" They do a demo and they say where they think the future of research is going and why this matters. So all those are on YouTube for you as well. Cool. So I think that's what I wanted to say. Hopefully that's enough new stuff for one half-hour after lunch. But um I very much appreciate your time and I wanted to make sure that you heard these new possibilities. Cool.