Design for Impact - Rally Your Team Around a Process that Drives Growth

[Music] Thanks so much. I'm going to start off with a little bit of a story. Um, back in 1919, a guy by the name of Sir Ronald Fischer got a new job and he was hired at the Rothamstead research station in London, England. And one of the things that he was tasked with doing was to look at decades worth of serial experimentation data. So they were, I don't know, planting plants, changing fertilizers, um seeing how well things grew next to other things, and they were collecting data year after year after year after year. And it was his job to go back to his desk, crunch all of the numbers that he could, and then figure out what kind of conclusions can we draw from all of this data. And you will never imagine what it is that he learned after looking at all of that data. It shocked everybody. And what he took away from all of that is that all the data was garbage and he needed to throw all of it out. was just completely useless. Right? So why did he say throw away decades worth of experimental data? There were a number of reasons and the reason is poor experimental design. And there were a few reasons why the experiment design was bad. First of all, more than like a hundred years ago, the concept of a control group, a base version to compare a variant against didn't really exist. So you may or may not have anything to compare the data against. Um also if you manage to have something to compare against, chances are the sample size was so low you couldn't draw reliable conclusions from the data. And if you managed to have a high sample and a control and a b a base and a variant, chances are the randomization was done in a really ad hoc crappy way. So you wouldn't actually know that the change was isolated and you couldn't trust the cause and effect relationship. And finally, if you had all those other wonderful things, chances are people were just writing [ __ ] hypothesis, which still happens to this day. I'm sure you've all experienced it. So what he did, because what was interesting about Sir Ronald Fischer is that he was not just a biologist and a researcher. He was a statistician. So he took all of these different viewpoints and was able to push an entire uh industry forward by combining different ways of thinking uh and different crafts together. And what he ended up writing was the design of experiments. And this text has fundamentally shifted how we as a species run experiments. And it's what modern-day experimental design is based on uh for medicine, for science. So, if you've ever run an experiment, you can thank Sir Ronald Fischer for those innovations. At this point, you're probably thinking to yourself, am I in the wrong place? Like, why is she talking to me about a scientist? Like, I thought I was at UXLX. And you're right. I am telling you about sciency things. But, uh, I am a designer and I'm here to talk to you all today about a very specific type of design. And that type of design is called conversion design. Um so today what we're going to cover is what in the world is conversion design. Second of all uh we're going to take a look at the conversion design process and how it's different from other product design and development processes and then how you can use conversion design within your organizations to drive growth at your businesses. All right, are you ready? All right, I hope you enjoy dog memes because I love dogs. All right, here we go. So, what exactly is conversion design? We'll start there. When most people think of the word conversion, they often think of sales or money, something related to profits. But if you're a nerd like I am, you will have dug into the origin of the word conversion to find out that the Latin root word stems from converter. And converter actually means to twist, to bend, to transform, but ultimately it means to change. So conversion at its core means to change something. And design, a lot of people don't know what design actually is. And when I talk about design, I'm not talking about what something looks like. I'm talking about capital D design. And the best definition about capital D design that I have found is design is the rendering of intent. Right? And this was said by Jared Spool um a prominent design thinker. So if you think about it, design is having an idea and making sure and giving a shape and form to those ideas to drive the outcomes that you're aiming for. So at its core, it doesn't necessarily mean making something look pretty. It means to drive the outcomes that you're aiming for. So when you put the two words together, basically conversion design means to craft intentional change, but not just any change because change is easy. I can change anything willy-nilly and make things a disaster. I have done that in fact. But the kind of change we should be aiming for is improvements. And ultimately the heart of conversion design and how you approach your work on a day-to-day basis is to make things better and not just different. Because when you start experimenting on the changes that you're making and understand the cause and effect relationship between what you do and the outcome it drives, we're [ __ ] things up a lot more than you think we are. Sorry, I'm just reporting the news, right? Okay. So, conversion design is a little bit different than a typical design process, and it's because it combines three concepts. The first concept is design by way of human- centered problem solving. Uh, science by way of Ronald Fischer's hypothesis testing and the modern scientific uh process. and then business by way of value creation and transfer. And that sweet little spot there in the middle, it's like a dark like void, you know, into the ether. That is the conversion design process where you can get lost and that's how it feels in your soul if you do this long enough. Um, anyways, so how does this actually look different, right? Now that we know all the ideas and how it comes together, um, how does this look different from how we might be working today? Well, your typical product design and development process looks something like this. Discover, design, develop, deliver. Very linear, right? And then you also have the conversion rate optimization process. Who who's familiar with conversion rate optimization? Okay, cool. Decent amount of you. Nice. So, these people pretty smart. They took the line, turned it into a loop. That's a nice little evolution, I think. Right. So pretty clever because they realized that when you do these things you have to iterate. So they captured the iterative nature of the process by turning it into a loop. But both of these I would argue are primarily based on linear thinking which is one way of looking at the world. And you'll know if something is based on linear thinking because you can visualize it with a stepbystep kind of diagram that has very simple cause and effect relationships. And it also gives you a sense of clear progress. You're always moving forward, which is typically [ __ ] but it makes you think that, right? So, there is another way of looking at and thinking about the world, and that is through the lens of systems thinking. And you'll know something is based on systems thinking because it typically captures the dynamic nature of how things actually are in the world. It also captures more complex and interconnected relationships, the knock-on effects that we typically lose in linear thinking. And then finally, it gives you a more realistic sense of progress, which I think we've all experienced. Sometimes you have to go backwards before going forward again. And sometimes you go back and forth a whole bunch before you're actually able to really make any kind of progress. So with conversion design, we have to embrace the complexity, the messiness and the chaos of the real world through our systems thinking. And we also can benefit from linear thinking by gunking through like going through all the gunk and the mess and then finding those little nuggets of wisdom where you can really trust a cause and effect relationship. So we have to actually use both ways of thinking or as I like to call your whole ass brain. Not just your halfass brain, your whole ass brain. Because when you use your whole brain and you think through things in me multiple ways, that's when the magic really happens. You're actually able to learn something that can change things for the better. So if you are interested in diving more into the concept of systems thinking, these are two really great books on the topic. One is closing the loop by Cheryl Kaba and the other one is called Thinking in Systems by Danila Meadows. So I encourage you to check out those books. Alrighty then. So now you're all probably wondering, okay, you've just talked about this thing for I don't know maybe eight minutes or so. Uh, but what is the conversion design process, Aaron? You've been talking about it. What is it? So, here we go. It's the time for the big reveal. Are you ready for this? >> That is so unconvincing. I will leave now. Unless I hear somebody that really wants Do you want to see it or no? Should >> Oh my god. Yes. Good. Cuz I flew here and you know stayed in a hotel. So, I hope you want to see it. Okay. Um, so this is the conversion design process and it is a sevenst step interconnected system diagram. And as I go through each of these steps, you'll probably be like, well, this kind of looks like something already familiar with. And yes, you would be correct because oftent times as things evolve, there's never like except for AI, there's huge fundamental shifts, but oftentimes progress looks like just an iteration on the thing that came before it. So the first step in the process is called the understand phase which is basically the research aspect which we all know and love and can do faster now thanks to what Don shared with us today. So now then we have the hypothesis phase which is again where you think about the change that you want to see in the product that you're maybe designing for. Um, and then you have to think of all the hypotheses and then prioritize them, which regardless of what your product manager says, it really is just fancy guessing. Who's a product manager? Okay, just a handful of people. Yeah, priority. I can say it's pretty safe in here. I think we could take them if they tried to like jump at me, but prioritization is just fancy guessing and most of the time we guess wrong. Sorry. Uh and then you have the create phase which is where content designers uh regular design like UI designers, UX designers, developers all work together to bring shape and form to the ideas and concepts that you come up with in the first phase. Then you have the test phase which is where you run an AB test or some kind of randomized controlled experiment to extract a cause and effect relationship. the analyze phase where you sift through all of the data you've collected up to this point. And then finally, you have the decision phase where you take a look at everything that you've learned throughout the whole process and make a well-informed optimal decision. And when you take a look at this and you breathe a little bit of life into it, you realize that it's actually moving forward. So you it's not a a linear process. It's dynamic. you can go back and forth through the system and like if you realize that something that you're going to develop is going to take too long, you can go back and rep prioritize something else. So it's a living breathing and it operates like kind of like a machine. But like every good system, it also has an output which is collective knowledge. So, by the time you finish cycling through the process, you have hopefully learned something and you were able to take what you've learned and become smarter and create a positive reinforcing loop, making the process better as you go forward, making your entire team smarter and better in the process. But in the conversion design process, there's also another output which seemingly goes into the heavens. And that one is value. And this is where the conversion design process starts to interact with business. And that little ven ven diagram I showed you. So now we're going to talk a little bit about the business value cycle. The way businesses typically work is somebody invests their time, their money, their energy, their knowledge to start a business, right? And then eventually businesses they end up growing to a point where they're able to invest in hiring employees. The value that businesses give to employees are maybe benefits, salary, um sense of purpose, all these different kinds of things. And then employees invest a value that they can create into doing different types of tasks. So they give their labor which is a form of value, their intelligence, their passions, their creativity, their skill sets to transfer that value to a task that they do. And then if you've chosen the right tasks, the task that you complete will create value for your customer, which then would hopefully, let's say, save them time, bring them more joy, make something easier. Uh, but you're trying to always create value for the next inline stakeholder. And then if the customer finds enough value in that they'll invest maybe their time, attention uh or in buy a subscription or buy a product. So it starts getting the value cycle moving forward. But it doesn't move forward itself. And that's where businesses need us because when you work together with your team in the conversion design process, the conversion design process then starts to power the business value cycle to push forward. So your goal is to create as much customer value as possible to reinforce the system to make it stronger to help the business grow over time. All right. Are you impressed with those animations? You you should be I did them with CSS because I don't actually know how to use animation tools. So I did it like 3:00 in the morning maybe a year ago but I will not change it. This is what they are. Okay, cool. Now how does this whole thing drive growth? Because your bosses want you to create growth. That's how we stay employed and in this market just get paid people. Yes. Okay. Cool. So um the way that this process is differentiated from other processes is not just the fact that it's based on systems thinking. One of the key differentiators is in the understand phase, right? And there's a very specific research strategy that you approach your work with when you engage in conversion design. And most times user researchers will say or people will say we got to focus on the customer problems d everything is about empathy which is totally true that's great but we have to pull our vantage point back and say like yes we need to solve problems for people but if we want to drive an impact in our businesses let's look at what problems we can solve through the lens of our important business goals and objectives. Right? So take one step back and look through uh the the lens of business goals. From there, once you understand what business goals you're tasked with achieving, then you take a look at what top customer tasks are your users going through frequently that they experience problems with, right? And then that's when you come up with research questions which then help you choose an appropriate research method. And then finally after you find these problems that you need to solve four people and you go through the research method to answer those questions you'll have come up ideally with some actionable insights of problems that you can solve. And if you do this well what you'll end up with is a direct line drawn back from the actionable insights your teams will be working on to the business goals that you all need to impact to stay employed. Yes. Yes. Stay employed. Okay. All righty then. So, another way that this process is different from other processes is the test phase. And in the test phase, what you want to do is run as many AB tests as possible. And I know that's not in vogue to say anymore, but I'm not very cool, so I'm saying it anyway. So, you want to run as many AB tests as possible. And here's another thing that I stole from science. Um, so this is called the hierarchy of evidence. So when you run an AB test and you con and when you engage in conversion design, what you're actually trying to do is to walk your team up the hierarchy of evidence um to collect the highest form of evidence that you're able, the highest quality evidence you're able to collect to make better decisions with. So when you take a look on the scale of available, right, if it's at the bottom, it's really available, but it's not very reliable. and that's where expert opinion is. So opinions don't really mean anything. Um then you go one step up and you have observational studies that require some effort but again the reliability is a little bit less than other forms of data because they can be biased uh depending on how you design them. If you go and the most product team themes really only exist or only do a lot of uh they only collect evidence in these two rungs. What we want to do to differentiate our businesses from other product teams is to engage in the highest form of evidence collection, which is randomized controlled experiments. This is the gold standard of scientific evidence. Um, and like I said, that's just like an AB test. It's a fancy name for an AB test. But as you can see, there's still I mean, if you're not color blind, maybe if you are color blind, there's a gray triangle up at the top, right? And that part up there is called engaging in a systematic review. What you do when you engage in a systematic review is you take a look at all of the evidence that you have collected by going through the conversion design process. You weigh it based on its availability, its reliability, uh the ethics involved, what your long-term goals are trying to achieve. And then you as human beings, you know, not the data, not the chatbot, whatever. you as human beings make a well-reasoned um decision based on your judgment to maximize the value created for all of your stakeholders. Right? So that is our jobs within organizations. But in order for this to work, in order to collect really high quality evidence, you have to design good experiments because we should learn something from Sir Ronald Fischer. If you go to my website, aarendon does things.com and look in the resources section, I have a bunch of experimental design templates that you can use to structure your thinking to make sure that you end up with reliable, high quality, statistically sound AB tests at the end of it. All right. Now, how does this drive growth? Good question. All right. Well, if you only have a few teams in your organization that run experiments, which is what we often see, right? There's maybe a growth team or a CRO CRO person or somebody over in marketing who runs an experiment. If we do not infuse this way of operating in our culture, you're not going to win, right? It's you're not going to ever um get the positive reinforcing loop of growth off of the ground. Everybody needs to behaving to be behaving and operating in this same way so that we're all making good decisions together. Because like Peter Ducker once said, culture eats strategy for breakfast. You can have the best ideas in the world. You can have the best team ever, but if they're not behaving in a way that reinforces your strategy and achieves your goals, doesn't matter, right? So the culture is what brings the strategy to life. When you work in this way or if you want to challenge your company to operate in a different way that drives growth, have them align around this mental model. Um I used to work at a company called Booking.com which is one of the most highly profitable companies in the world. It is a cashg generating machine because we all operated like this. So this is basically what I learned from working there for almost nine years. And the reason that working like this works is because of something called the compound effect. Right? And this is how the compound effect works. Imagine this is your typical backlog. on anybody's backlog, you have a mix of good ideas, ideas don't won't really do anything and ideas that are just absolutely terrible and are going to lose your company money, right? And most product teams have a successful quarter if they ship everything on their backlog. But if they're not experimenting, if they're not getting high quality evidence, they don't know what's good and what's bad, and they're just shipping things that negate one another. So if you ship some good things, some bad things, you end up with business results that look like this. Just get sad panda results for your executives. Really awful. So, what I have learned after experimenting on literally every change that I made um when I worked at Booking.com, I learned that when you push code, when you when you change something on a product that you're building, there are actually far more ways to fail at this than there are to succeed. At booking, nine out of 10 experiments that we would run would fail. And that would mean that they did absolutely nothing or they actively lost the business money. So that means only 10% of anything that we did even gave a slight signal that it made anything better for our customers and ultimately our business. So sit with that. Most of the stuff that you do doesn't do anything or makes things worse. Again, just report in the news. Anyways, so again now if we operate in a different way than most product teams, you have your classic backlog mix of good and bad things. But when you experiment and you run a randomized controlled test, an AB test, you're able to systematically identify those things in your backlog that you shouldn't be putting in front of your customers or they shouldn't be there for much longer. So you're able to identify the [ __ ] decisions and throw them in the bin. And what you'll end up shipping actually looks more like this. And this is counterintuitive because they'll say, "Well, why did you ship one quarter of the things that you shipped two quarters ago?" Because most of what you were shipping previously was terrible and it was losing. It wasn't doing anything. So, you ship less, but you ship better. You ship the things that matter. And then when you know this information, then you refine your strategy to go after the positively impactful changes, which then starts to give you business results that compound over time, right? Because good builds on top of good, builds on top of good, and then you throw away all the bad decisions so there's nothing holding you back, right? Anyways, so we're going to have a little story time just to give you an example of an exper an experience that I've had relatively recently about a time that I was told not to AB test and I ignored them. So here we go. So I work somewhere. I'm not going to tell you where. I work for a lot of different companies at the moment because I consult and stuff. And what we were doing was we were doing some AB accessibility testing on apps because accessibility is important people. Very important. Yes. Especially now when you live and work in the EU. There is a new uh accessibility legislation coming up. Uh be ready. It comes out in July. So you need to care. So I was working on this brand and this brand has a very bright brand color. You know, brand designers love to have flashy colors, right? And this is what the UI looked like on the mobile applications, super eye-catching, like bright color. Um, and they also love like really light text and thin fonts and that kind of stuff. It was very subtle. You know, visual designers really thought it looked great. But like I said, there is an an accessibility act coming out and I said, you know what, we really cannot do what we think looks good. we need to design better UI, right? So, I said, let's design something better, something that passes um W CAG accessibility guidelines. And basically the experiment that we designed looked like this. We changed the UI of the entire app for both Android and iOS. Um so that everything had a lot better contrast, right? And then what we did was we ran an AB test to understand what is the impact of running this particular test. So after running it for a certain amount of time and what we were actually measuring was conversion. So if people saw the accessible UI would they buy more? Would they finish their orders more? Right? So, and then again, because it's an AB test, there is a direct causal link between the UI color changes that we made and the increase on sales that we would hopefully see. So, what we ended up finding is that it drove an incredible amount of new sales. So, we just changed colors by making the product easier and better and more accessible for people. And it gave the business more money. It also reduced the legal risk. So, it was like a win-winwin for everybody, right? So, you can imagine the product people at the company were like, "Oh my god, that's so what? I didn't know that color was important." I was like, "I could have told you that color was important because this is not my first rodeo, right? I knew this was important because we did it at Booking.com as well, too." So, they're like, "Let's do it again." I was like, "Yeah, we're going to set it up on web now, too, because all of our products need to be accessible." So, we set up the experiment on web. The exact same thing. All the we used design tokens actually Brad. So we used design tokens. That's the technology that we were changing the colors uh in. So we used the they were all pulling from the same backend, right? So everything visually was the same. And when we got everything ready for the web, I went to the product manager and I was like, "Can somebody please set up an experiment tag?" And they're like, "No, man. Just roll it out. Just ship it. We don't I mean, you've already validated the colors are better. Just roll it out. ship it. And then I looked at them and I was like, "No, I think we should test it. I really don't feel comfortable just rolling this out. I know that color is a powerful tool to drive sales and to help C and usability." And then they were just like, "No, Aaron, experimentation is slow and expensive. That's a stupid idea. Don't test it." So, I'm a little bit of a prickly person, so I just ignore things that I don't want to hear. And I fig I found somebody that would help me and I just did it. My I we did it. I ended up testing it. And you know what? So here's the thing that we tested again. Same colors, same everything. So they're like, "If it's the same everything, why would you test it?" And then the reason why we did it is because the results were terrible on profit and customer lifetime value. All of our important business guardrail metrics. what had previously driven an incredible amount of net profit previously tanked it. So basically the experiment for a number of weeks was just like burning money. I lost so much money with that experiment. But if it wasn't in an experiment, I wouldn't have been able to see it, right? If we would have just shipped it, we wouldn't have known that we should revert the problem, right? But because we got the data from the AB test, we were able to stop the experiment, stop the hemorrhaging of cache and then figure out there is a bug somewhere in the implementation, right? Because web is not apps, apps are not web, one developer is not the same developer. So somebody somewhere along the line messed up. And this is why we test people because just because the concept is the same oftent times the context is so different and you can actually screw up the implementation. The executions often fail more than we would like to believe. Right? So always test stuff. So basically what we did was we went back to the drawing board. I had the developers go through and see what what bug did we possibly cause and then ultimately we will iterate on releasing the new accessible color palette for this business until I can actually see solid evidence that we have indeed made the experience better for everybody and not just different because we could have just pushed it because it's accessible must be better but then we would have been exposing a bug to god knows how many people and be losing money for the business but anyway So, what did you learn today? Hopefully, well, conversion design combines design, science, and business um to help us learn more about the product changes that we make. The conversion design process is a dynamic system diagram that helps move our businesses forward to help us drive growth. As we cycle through the conversion design process, we work with our teams to methodically walk our way up the hierarchy of evidence so we can maximize the value creation for all of the stakeholders we're responsible for serving, which requires us to think in both systematically and linear thinking ways or using our whole ass brains. But we need to do it in a way that we follow good experimental design. So we're collecting data that we can depend on and it only works if we all work together in unison because if the if it's not infused in the culture all the ones that you find over here are going to be negated over by Debbie and marketing right so you need to have everybody vigorously testing everything that they're changing because as we've learned looks can be deceiving concepts can be good but executions can be terrible and ultimately at the end of every day we need to be able to say Yes, today I have evidence that I have indeed made things better and not just different. So, thank you so much for listening. [Applause]