Forget Vibe Coding. This Is How Real AI B2B Products Get Built | AI Product Management

AI has transformed majority of the industries and including health care. However, most people do not know how to combine the power of machine learning and joining I could quickly launch your AI product within 90 days. And in this episode, I invited him provide a match AI to show you how you're able to quickly launch your product in the space of health care within 90 days. >> [music] >> Hey guys, this is Dr. Nancy Li at the product feature imports. I've helped thousands of people land a dream PM job offer in fan companies and unicorn startups and continue to get promoted as product leader. In this channel, I talk about tech trends and free product management training. Like and subscribe to our new video every Tuesday. Hi Mariana and Quan and Farah, welcome to product insider podcast. >> Hello. Hi Dr. Nancy. Hello everyone. >> Yeah, so glad to have you guys. >> Yeah, welcome. So glad to have all of you guys. >> Glad to be here. >> Awesome. So so glad to have you uh here today and actually your team is one of the top team who launched the real life version of AI ML product and especially in the B2B space. Why don't we get started regarding introducing yourself to the audience and especially what's your background and how do you upskill yourself in AI and so quickly? And Mariana, why don't you get started first? >> Hi everyone. Um my name is Mariama Jabbaty. I am an AI product manager with over 5 years of experience across the health care data space, life science commercialization, and product strategy. Most recently um I completed the AI PM accelerator program where I led a 17-person cross-functional team to build an AI-powered uh provider matching product which reconciles fragmented health care data into single reliable source of truth in 90 days. And I'm especially interested in building AI products that tackle real healthcare challenges and make complex data more usable and trustworthy. >> This is beautiful, Maryanna. You have been healthcare for a long time and it's always your passion to see how people use AI and use AI to quickly transform the entire industry, especially patients experience as well. I'm so glad to have you. And Farah, you are an AI engineer. Tell us about your background and how you upskill yourself in the AI space. >> Thank you for having me, Dr. Nancy. So, for the past 2 years I've been a self learning programmer for full stack AI engineer. And for Provider Match AI, I was the co-lead engineer I was overseeing the developer and the data science team. >> Gwen, can you quickly introduce yourself? I know you are AI designer. Tell us more about your background, how you grow yourself in the space of AI. >> Um I'm Gwen and I'm a master student in UX design at Wilfrid Laurier University in Ontario, Canada. My background is international business and I also have experience in marketing and graphic design. And I'm currently focused on learning and designing AI powered products. And on Provider Match AI, I was the UX UI designer responsible for the end-to-end MVP interface. So, um when I moved to UX, I noticed that um products are increasingly AI powered nowadays. So, but the problem that designers they don't have much knowledge on how to design like good AI powered product. So, that's really inspired me to to learn a lot. I started um learning a lot. I I feel I I'm lucky to have the chance to be part of um PM Accelerate Accelerator program because I was given so so much training and material for me to learn by myself. And I also use um AI of course cloth to for self-learning. Yeah. >> Beautiful. And like everybody is using the combination skill set regarding leveraging cloth, the AI tool to learn, and also getting more uh advanced coaching, especially building a real team. And now let's talk more regarding your product and specifically ProviderMatch AI. And Mariana, can you tell us more regarding what is ProviderMatch AI, who is the end customer, what it do, and then maybe you guys can give us a demo. >> Sure, we can do that. Thank you so much, Dr. Nancy. Um so basically um you know, all of us have experienced this in one way or one form or the other, you know. You know you get referred to a specialist, um you call the number, and it turns out that the doctor doesn't practice there anymore or isn't is not even in your network. So basically what happens, you you you you kind of come back to square one, and you you keep calling and starting the whole process again. And most people will often think of this like, you know, as a healthcare they're thinking of this like the healthcare doesn't work. And it's not often the case. Um often the case the issue is a data problem. The same provider can show up differently across system, you know, different name formats, different addresses, sometimes even different IDs. And those systems they don't communicate with each other. So someone has to figure out who does that. And um you know, and this is the the um how so how how to how to do that and sorry. Wait. Someone has to figure out how which records is actually correct. And today this has been done manually um every single week. And the people dealing with this are the data analysts in the healthcare, the pharma, the medtech. And when we spoke with them, the same problem keep coming up, you know, they're spending 20 plus hours a week just reconciling provider records. There's no clear audit trail, no explainability, and a lot of these decisions come down to like manual look-ups and judgement calls under time pressure. So, we built Provider Match AI. Instead of doing all of this work by hand, Provider Match AI is a tool, AI-powered tool that ingests data from multiple sources, compares the records across key fields like the name, the NPI, the address, and specialty, and scores potential matches from zero to one. The key difference here is it's explained why for those scores. Like in in plain simple language, the analyst gets to quickly validate the match, make a decision in seconds, and every decision is logged. It takes the guesswork out of the whole process, which is what we had mentioned. And their existing tool, however, these tools are often built for large IT teams, requiring long implementation, heavy infrastructure, and not designed for the analyst who actually has to work through this messy data on a daily basis. This is the gap Provider Match is trying to fill. >> Miranda, I was so surprised to see that people spend 20 hours per week to massage or clean up and matching all the healthcare data. This is like is this like everywhere in the industry? Why people don't use any existing tools to help out? >> I mean, this is a problem that is widely known across the industry, you know? As I mentioned a couple of the fields where it's applied, often time it's a you know, there could be tools as I mentioned, but there's those are enterprise level and they often don't take the analyst into consideration. So, and in in in in our case, actually, this is a problem that was very close to my work where I had to work on a project that required to, you know, it's a provider matching project. A portion of that, as I mentioned I'm a consultant in the consulting space, a portion of that required me to do the analysis of the work and I had to deal with provider data from multiple source and we were expected to spend at least 60 hours per person in a team, which is almost impossible. We have a 40-hour work week, which means people have to rush through. There's a lot of errors. So, it's a problem that is very burning issue and you know, it leads to a lot of downstream problems, patient not having treatment in the right time, organization losing money. It's almost a three bill approximately a three billion industry-wide problem. >> So, this is crazy. >> Yeah, we're trying to work here. >> Three billion industry-wide problem. Now, I am very curious. What does this new solution look like leveraging AI and the machine learning? Farrah, are you able to give us a demo for your product? >> So, the the analyst will upload their data set to be matched in our system with the providers that they have. And then we upload it, then we would standardize the formats, we would go through the matching and the matching script for this as well against our 9 million records in our as our internal data sets. Then these scores, as you can see, their processing has been complete. The 9 mill 900 matches are ready. We can find all these scores now that have been the cases have been triaged. So, you can see that there's high, medium, and low confidence and you can also see the scores there as I was mentioning. Now, if the analyst wants to filter through just to test things around, they can with the high high confidence, medium confidence, or low confidence. Or if they're just testing this out for the first time, they can look at a case. So, let's take for example one case, which is a medium confidence. Now, you can see that they can compare the their fields with their data set they have uploaded and our internal data set. So, for example, the MPI matches, maybe the address is different, there is no organization provided. And if they need a little help, there's an AI explanation explanation sharing why or why not these fields match. Uh and then why given the the score. Now, based on the all this information, the analyst can now decide, "Okay, would I approve this match? Do I reject this match? Or do I flag it for manual review?" Um so, for example, the analyst approves this match that, "Okay, these two records are the same person. They're the same provider." We can then go to the review queue, um where you would see that that now approved match would go to golden the golden records tab. Now, if we go back to all the cases, we if, for example, you would like to um uh filter through, for example, high confidence, and uh the analyst is sure in this case that all of them are true matches, they can now auto accept all these matches to go to the golden records tab. Then, if you go to the golden records tab, you can select all these records that are now our ground source of truth and export them for further review um in the future. >> So, that's a very comprehensive product. Let me ask you this. So, what's the secret developing AI product from just what's in just 2 months and any special lesson learned building it from zero to one so quickly? >> Yeah, I think for us, um and I can I can say and the others can chime from the PM perspective, it was very, very important to have early cross-functional alignment um between the AI, the data, design, and engineering team so that we doing that to kind of reduce any ambiguity and keep the team moving. So, it was very crucial for us to do that and I think that helped a lot. And it also helped us see what what which which tasks are dependent and which which is dependent on what we can get a whole flow of of of the different the different pipelines work streams and then we are able to anticipate where we could potentially see a bottle a bottle neck or a blocker and we are able to avert that. So, another thing for us really is cutting down on the scope. As you see, it's a very it could we could go on the on the whole tangent because there's a lot of cool stuff that we would want to have. But one of the thing that really helped us is really having to define that user flow very early on and seeing what is really important from the user perspective that they want that this tool to do. So, and we focused on building that simplest version of the tool that the user can actually see they can actually see the impact in a short time, you know? And then we can now build on the other functionality post MVP. But that was very important for us to really understand the flow for the user and have that set and across the whole team engineer, designers, the data all understand what is happening across these different work stream. And I think that really helped for us in general. >> That's a great >> Overall trust and team in the team was you know, having to trust each other and you know, having an amazing team team dynamics plays out a lot. So, because there was a lot of challenges and you know, we had to encounter we had encountered and because of the dynamic we had we were able to overcome them. >> Beautiful. I think the most important skills in the age of AI is being able to manage teams, manage real humans. Um to be very frank, based on the speed of how AI develop in the coming few years, many tasks will be replaced by AI. And the person believes that the only skills cannot be replaced by AI is actually human relationship, leadership skills, communication skills, and then including managing conflict. Those important skills can never be replaced by AI. Especially nobody want to be managed by your AI boss. You know, you need a human to manage others. Right? Uh I'm glad you guys are master those um very well in terms of human relationship. Um so, you mentioned challenges. Let's dive deeper a little bit. What challenges did actually face when you deployed and implement the AI strategies and technology? >> I think um the one of our biggest challenge we faced and we really did not anticipate that to some extent was the data. So, because we were dealing with um the the provider record that's like 9 million um records that we had to, you know, train the model, like you know, do actually do the the the labeling for the the training pairs and all of those things. So, it took a lot of time. And that that aspect of the the build was dependent um the rest of the the engineering work was dependent on that. So, um you know, we had to like have the the the team, you know, from the the the PMs, the engineers, everybody chiming. It was not It was less about like, "Oh, this is my specific work per se." but the areas that could be done outside of the data cuz we had the data scientists, of course, like labeling and all of those things. We had uh collaborative efforts to actually move that process much faster. Also, we had the computer uh the computer we were working the data scientist was working on crashed. So, we had one of our team member, Farah, here had to use her mom's computer and >> [clears throat] >> uh it's like a super power key. You can talk about that. But like we she had to like bring in a mom's computer and uh and maybe I can let you explain that a little bit, Farah. >> Um so my mom's computer had 64 GB RAM and it was able to process uh all these scripts a lot faster because we were dealing with a like a laptop crashing because of you know as I Maryam mentioned 9 million records. So luckily you know that it came in handy. Um and anytime in my free time I would uh my our data scientist she would send me, "Hey Farah, can you run this for me?" And I was able to. So that was uh that helped a lot to get our work done faster. >> Yeah. >> Is your mom a gamer? >> What? We lost a data Sorry. >> No, my mom is not a gamer. >> a joke. >> She just used her computer. >> Maryam, what do you got to say? >> No, I said to make you to us we lost a data scientist in the middle of the project. So we had two data scientists and this the data is heavy. The most of the work is dependent on the data and then we lost a data scientist. So it was it was a hard you know a lot of I would say again it comes with having the team really believing in the problem that you are solving. So everybody you know if they have a background in this they can chime in and that was really helpful. We were able to really help uh it was really to help us move our product build in a much within the time frame that we had. >> [snorts] >> That's very insightful especially you guys are very agile and really being resourceful solving problems there. Um so Gwen as a designer are you facing any challenges? Especially lots of people say designers can replaced by AI anyway. Tell me more. >> Um I think that for designer when designing for AI product uh one challenge is about trust like I've mentioned earlier. Users usually be really skeptical about AI results. They tend I have a tendency to to to not yet believe it until like the design kind of give them a heads up or an explanation about why did the AI make that suggestion. Yeah, so we couldn't just show the confidence score. So we needed the AI to explain itself. So in the our product, um we are our product is different from other competitors on the market because we have like we did some research with other competitors. They just show the confidence score. But for us, we have like the the AI explanations like it it's it gives explanation to user that why did it make that decision. So it gives the user the trust and also the confidence. And I think designer in our day I've been reading a lot of news that we are getting replaced soon, but I I don't really believe it. Like I mentioned, I did research on competitors and some of them have like I feel like they used AI to generate UI design. When I try to use use the product, I just don't as a user I don't feel intuitive at all. You guys brought in really good point. What I believe that right now the most important skill set is not get replaced by AI is actually using AI and understand how AI is going to transform how people use AI, trust AI, and how AI being able to process the data without hallucination or with like minimum hallucination. So still able to get the best result. And I also personally believe that the best way to really grow in the AI era is by building hands-on experience, getting the real user because no users, you cannot have data, you cannot see hallucination, everything is in a prototype phase. Prototype nowadays, everyone can just write code, prototype something. So, it's it's no longer common advantage. More important getting hands-on experience. And I have a free download with a top 20 AI product ideas to gain real-life AI experience people can start working on right now. And you guys should go to pmster.io/aiproductideas to download this. I'm also going to link the free cheat sheet in the description of the show note as well. Now, let's talk about something very technical. What tools and models have you leveraged to create the AI product? So, let have someone technical talk about this. Maybe Farah, you talk more regarding the technical elements of tools and models you used. >> So, for AI-assisted coding, we used mainly Claude and ChatGPT. But for the project itself, we used an XGBoost model classifier, we for the backend FastAPI Python, and for our scripts, we had everything on AWS EC2 and our database on RDS. Then for our frontend framework, we used Next.js. >> So, tell us more about why you select certain AI models. You guys had two things, right? You have machine learning model, you also have GenAI element. Tell me more regarding what machine learning model, what GenAI model have you used, and what's it look like? And why? >> Um you can take that. >> Uh regarding the the model the classifier model, we were also considering LightGBM, but uh time did not allow us to test that model as well. Uh for the explainability layer, we stuck we stuck with Grok since it did the job well, but we were also considering using Claude. >> Mhm. >> Yeah. So, we had we had like a Grok um we had Grok, Claude, and then the Gemini stick for back you know, so first since it's a Grok was free for us. So we considered the cost implication. So Claude was like you know, we had to use Grok and then for higher higher cases or more significant ones later, they can you know, that's is not generated by the by not giving us good input from the from from Grok then we can use Claude, but we didn't have to use it anyways for for our MVP. We only had Grok. >> That's very smart and especially cost of all those APIs and tokens for those like AI tools is actually very expensive and our company have two AI product AI interviewer and AI resonator. I see how much the bills I get through those AI product and then what I found out is the most expensive AI credit depends how you use is actually is minus AI. I use minus AI for those AI agent task and actually very expensive. Use my credit within like two days. It's I say crazy and keep on upgrading myself. Now almost paid a thousand dollars for different kind of like credit needs to be used and also very smart you guys are controlling your cost. The basic level use free version like Grok and then the more advanced use more like expensive stuff and which also preventing lots of startup or lots of like early MVP stage of the AI product and start die out just because they couldn't afford all the expensive API cost. That's very very smart regarding how you guys did it. And so now all sounds very exciting and I do want to ask this question on behalf of the audience. How can people test and use your product especially those kind of medical professionals? Where should they go? >> They can [clears throat] go to our website and we will have it in the video, but just go to our website. >> It's provider match AI. Right now it's hosted on Render. So uh that is temporary until we have our own website. So, the URL is providermatch.ai uh dot com. Uh, we can send the link. Um, it will be in the description. >> Awesome. Great. Cool. We're going to link in the description. You guys should totally go check it out, especially if you're one of the analysts and spending 20 hours per day uh per week crunching data, this impossible. We need to leverage your AI to be more productive. Now, let's talk about career growth and advice. Um, what advice would you have to advance their product management career or AI designer career or AI engineer careers? Especially for all the audience today, they are interested growing their career in AI. So, what would I advice would you give them? So, Mariana, you go first. You can represent advice for AI product managers. >> Sure. Um, thanks, Dr. Nancy. I think from the AI product uh manager point of view, I always say it's good [clears throat] to be um very uh be very um ready for pivots. Like, be ready to learn, like open to learning. Um, the space is the AI space is growing very fast. So, you have to be willing to learn new tools, new uh um frameworks that uh it's been used in the space. And um you know, one of the areas for us in terms of like I would say I, you know, having to get the experience was doing the AI PM boot camp. And that was very, very helpful for me in really transforming my understanding from a theory, like what I think, okay, AI PM AI product managers do, to actually getting a hands-on experience from, you know, building a product from ideation all the way to launching and actually having to get users to start testing our product. So, I think just being open to learn, like seek out resources to grow and don't feel the industry is going to change. Use Use AI as a tool. You know, you are the experts, you are the you have a domain knowledge or expertise, then be able to leverage AI to be able to do your work and be able to solve problems and build amazing products that you're proud of and you make a difference in the lives of people. >> That's beautiful. For people who want to check out our AI PM bootcamp by working with team engineers and data scientists, developers, you should go to our website pm.ai to learn more. I'm going to also going to link in the description of the show note for you guys to download the syllabus and also talk to our team to see if you qualify for the program. Now, Farah, let me ask you this question. Lots of people say AI is about to replace developers. What advice would you give to today's AI engineers or developers? >> As many probably know if you've worked with AI and AI system going before it can hallucinate. So, it is great to test yourself even with when you're reviewing the code. You have to make sure, you know, what you want is implemented and you have to you're also the one in charge of making these decisions if this is correct for your product or not or your project that you're working on. And and it it is helpful to get things done faster, but at the end of the day you're responsible for what the AI produces and the code quality. >> Very beautiful said. So, in short, you guys are not going to get replaced by AI. In short, you need to monitor AI. AI is your intern. You guide AI, use AI more efficiently. And very well said. Gwen, as a designer, what advice would you give to other designers out there who want to break into AI? >> I think my biggest advice to adopt humans first thinking and human-centered design thinking. You know, it's it gets really exciting if you like know what AI can do and we have like ideas to design around that models. But the best products I've seen, I think it should always start with the question. So, what are the users struggling to do right now and how can AI make that easier and where which part of the user of the user flow that AI can actually make it better? So, yeah, I think we as a designers it's always um crucial to start with your user first. >> That's very beautiful, very well said. Human-centric design. Um which is true. AI does have so it's going to be yourself giving the best answers, but are you guys aware that recently on Oprah Winfrey's uh show, there is a therapist she just conducted a therapy session of a human with his AI girlfriend. Is is a crazy. I think people are trying to use AI to reduce human and replace human in some extent. However, the human-centered design is the most important part or we're just missing who we are. Just like a the the TV show Her. You guys should watch it. It's I think it's very creepy and we shouldn't go that way. It's more about how can we design it in the right way let AI to serve us as as intern or assistant. So, now let's talk about the the the final question regarding growth. How do you guys grow yourself personally and professionally? Because I am big believer of growth mindset. It's very critical to continue growing. So, how would you guys grow yourself professionally and personally? So, let's wrap it up. What about you, Farah? >> Working in a team professional team environment since I've noticed myself as a person I've changed after working with Prior Match AI. Especially how I think and how orders should go or how planning should go, task delegation, how to communicate with the rest of the team effectively, um giving updates. It's really helped me as a person. >> Beautiful. Um what about you, Quynh? How would you grow yourself professionally and personally? >> So, um I'm technically I'm actively learning how to integrate AI into the design process. As you know, um everything is super fast right now, especially in a UX field. So, I'm learning about how to use cloud, cloud code, cloud design, connect Figma MCP. And you know, lately, Figma just released Figma agent inside a Figma file. So, I'm trying to learn everything as quickly as possible so that I'm ready for the for any kind of job and project. Yeah. >> Awesome. What about you, Mariana? >> Um I would say um what has been said from the design and engineering point still applies, but to add on to that, you know, really as PMs um in this space, I feel like you have to be be be ready to learn. Like have a a thirst for knowledge and growth, you know, so cuz the field is moving fast. And you you have to be open to learning and upskilling yourself as much as possible as these um changes are happening. >> Beautiful. Very well said. And so that we can get ready for the fast AI revolution. And finally, for all the audience, if you any other follow-up questions and or want to check out the product, make sure to go to their LinkedIn profile and website. I'm going to link it in the description of the show notes. And and the most important thing we need to do in the space of AI is actually getting hands-on experience. And make sure to download our top 20 AI product ideas to gain more real-life AI experience. I'm going to link it in the cheat sheet of description of the show notes. Anybody who want to learn more about AI product management bootcamp to gain hands-on experience just like Farah, Mariana, and Quen, and make sure to go to our website and submit application to talk to our firm or to talk to our career advisor by going to pmsor.io to learn more. I'm also going to link it in the description of the show note for the application page. Thank you for joining me today. So glad to have you guys. >> Thank you, Dr. Nancy for having us. We had It was great to be here. >> Thank you, Dr. Nancy.