What is the role of Product Managers in an ML powered product? by Oren Steinberg

Arun, how are you? Great to see you. I'm good. How are you? Thank you for having me. Welcome. Thank you. If you're all set, the stage is yours, Arun. Take it away. Excellent. Thanks. Can you see my screen? Cool. Thank you. All right. So, nice meeting you all. As I was in production, the role of product manager, whether held by a product manager or by an engineer, any other builder that is acting as a product manager is evolving once AI and machine learning is embedded into the product. And this is the topic of the conversation we're going to I'm going to talk with you today. I'm going to start with an overview of what is a mail map. As this is not an educational session on what is a mail is just to make sure that we're all on the same field as far as terminology. Then I'm going to dive into deeper into the use case methodology of finding the use case for a machine learning and AI ML using the PRFAQ of Amazon, which is the methodology of Amazon for working backwards. We're then going to look at how do you qualify for use case as a good fit for machine learning or not. We're going to talk a little bit about the POC builder. How do you decide which element do you want to de-risk in the POC? And can you do uh machine learning in a lean methodology approach? People used to think that not today actually with auto ML and pre-trained model, this is doable. We're going to talk about the team. How do PMs and data scientists collaborate in an empowered team structure from the along the different journey stages of the ML journey of the company, whether you start with first use case all the way through an ML platform that can support an abundance of machine learning models, some in training, some in deployment, and so. Data is a very big piece, and we've heard a lot about that today. Data is a very big piece of any machine learning process, and product managers play a key role in us retaining data quality and preparing the data and labeling the data for any machine learning model. An important thing to remember is that a model is not a product. Taking a model into a product requires earning the user's trust. There are multiple mechanisms and tools we can do in order to you can apply in order to do that. Some of them are called human in the loop, some of them are called explainable AI, and we'll talk about them. And we'll talk about the the ML flywheel in a feedback loop. How do we take the data into a model, change the behavior, and then the model will drift because we are successfully change the behavior, so we maybe need to identify new data, relabel it, retrain it, and so on. And finally, I'll talk a little bit about how AWS can help you guys with a channel with a journey of the ML challenge of ML journey, whether you're starting it or an advanced stage, so that. So, let's dive in. What is ML in a high-level terminology, just so we can all kind of know that we're speaking about the same thing today. There are three terminologies that are often used interchangeably, so artificial intelligence is mimicking human behavior even in an in an rule agen rule engine based method. Machine learning is where we let the machine learn from the data by itself. Deep learning is a subset of that when we have hidden layers. So, we have an input and an output layer, and everything that's happening in the middle is hidden from the human. Though today with explainable AI, we can probably interpret and and understand a lot of the reasoning behind the machine learning. Another way to look at the classification of different machine learning types is supervised learning, where we have a label where we tell the machine what we want it to learn. So, say here's an image of a cat, here's an image of a dog, or a million of each. And then for the 2 million and 1, we want the machine to identify and classify binary classification by itself. Or unsupervised learning, where we do not tell the machine what it is that we want it to detect. We just put the data in front inside the machine, and we expect the machine to identify patterns in the data, whether it's anomaly or clustering that are attributes of the data that can help say this group is more similar than that group. But, what is the meaning of that group is something that we require an SME, a subject matter expert, to later look at the cluster and maybe change them a little bit, name them, and give it a meaning and a context. So, supervised learning is a place you want to start. It's easier to start, but it requires labeling. Unsupervised learning can support any cascading of models. Sometimes, we have a lot of data dimensionalities in the data, and we want to reduce the dimensionalities using unsupervised learning, and then feed those into a supervised learning model. Reinforcement learning is where we have a lot of iterative games or opportunity for machine to learn and design a policy based on a reward. So, we tell the machine, "Hey, this is good, this is bad, and now you can learn from that." A recommendation engine, for example, might learn from how users interact with the recommendations that we give it, and we can decide as product managers, not as data scientists, we can product managers can decide whether we want to to exploit what we learned about the users so far and give them recommendations based on what we know about them, or whether we want to explore new stuff that they might be interested in. So, this balance of exploration versus exploitation, I think it's more of a product manager decision than a data science decision. And we'll talk about those decisions and what is your role in those decisions going forward. In a very high level, and why I think um machine learning is probably the most disruptive technology for product managers, is that traditionally, we would look at traditional programming in the following manner. You have data, we have a program that the computer scientists and and the engineers are building in order to get the outputs that we want. This has usually high price, a lot of effort, and we as product managers often have a desire to do a lot of features, new products, many ideas, but then we hit the wall of hey, this costs a lot to implement, the heuristics are very complicated, the software is very complicated, and so on. In the machine learning world, we actually work the reverse. We say, here's the data, here's the output that we want the machine to do for us, and the machine takes care of designing its own program. This makes a lot faster and easier for product managers to actually test, iterate, and develop new products, provided that you have an ML platform to support this fast iteration of data scientists and product managers ideating together and trying out new stuff. You still need the data, you still need to label the data, but the process of not having to now heuristically code everything actually in the long run makes it easier to design, develop, and roll out new products and new features. So, machine learning has been around for a while. Why is it now so successful? Oh, we got something not working here. Okay, then can you guys still hear me? I'm We can hear you, Arun. Okay. And the stack I'm for whatever reason my deck Your presentation is stuck. I guess you can see that as well. Okay. Let me just start that over. I apologize for this. It's some glitch with the computer. Okay. So, machine learning has been around for a while and the reason we're now seeing machine learning taking up so fast is for three main reasons. Compute is now available for everybody, basically. Data has been accumulated in a fast-growing manner and research showing that machines they can see better than people in computer vision can almost analyze language better as people in NLP natural language processing and can detect patterns in data probably better than people. All of this is available on the cloud today. So, what does that mean for you, the product managers? What is your role and how does that change in a kind of machine learning-powered product management process? First, you can now productize and feature complex situations. Situations are first originally too complicated to condition. When data scales fast, you don't have to redesign your heuristics. Machine learning can support that. When the value is of an individual, hyper-personalized path or journey of a user rather than the segments. Let's say, instead of looking at an age group or an age cohort, you want to look at each user individually along their digital journey, for example. When data changes in an unpredictable manner in real time, machine learning can cope with that much better than traditional heuristics. If you have a lot of data and you're just looking to generate some insights from this data. So, you start with visualizing that. If you want to go deeper and identify stuff that visualization cannot detect easily. When you want to automate stuff, like computer vision, like BPM processes or NLP chatbots, and so on. In traditional, people used to say that, "Hey, we need millions of images or data points in order to train a model." And that doesn't really support the lean approach. Well, actually today that's not true. In the last couple of years, we've seen a significant uptake of auto ML capabilities. We at LBS call this autopilot in AI services. These are pre-trained pre-built models that can be tweaked to a specific use case or specific data set of any specific domain or vertical, but can actually get you started with a POC in a matter of hours or few days. With a POC that has machine learning embedded inside, it might be not the best or optimized model for scaling your own use cases. You can, like I said, tweak it later as going forward, but you can really get fast into a POC with pre-trained and pre-built model. And today you can do that in autopilot with an open box machine learning. So, it's not a closed box. You're actually, if you have data scientists that just want to save their ramp up time, they can start with auto ML and then get the notebooks that the machine has done in the background, all the ETLs that were designed by the machine and all the algorithm that it checked, selected, and and so on. And then you can take it and continue to improve on it going forward. So, what is the role of the PM in a machine learning process look like? First, we have the problem use case KPI definition. We then want to de-risk something with through a POC. The modeling, which usually includes data selection and labeling. Feature engineering which takes the raw data into a features, meaning that longitude and latitude can be a raw data. But saying that this person in that location is near a ball game or near the high road or a shopping mall, that is feature engineering. That is taking two features of location and kind of organizing them in a way that you can see whether it's 50 m apart from each other. Hyperparameter optimization stands for all the elements that you can control inside the black box, meaning that we talked about deep learning being a hidden layers between the input and output layer, you can control how many hidden layers are in there. You can control whether the data moves in one direction or circular direction. If you're looking at an example of recommendation of products which Amazon has been developing for the past 20 years, you can look at what is the data that we want to look at historically and when do we want to sample or give weights to the data? So if you're now wanting to look at data pre-COVID, maybe it's not relevant for this specific use case, so you want to kind of tone down the weights of the older data. Or maybe as we're coming out into vaccination or vaccinated world, maybe the data from COVID is actually the one that you want to weight down into the model itself. So these are decisions that are not science decisions, these are product decisions that are derived from the data that we put into the model and impact how the data scientist will tweak the hyper parameters of each of the model. Metric selection and optimization. Again, this is where you decide what you want to optimize the model for. Let's stay with the example of recommendation engine. Are you is your workflow is the app the way you embed the model into a feed into a product built so that the user sees one button, he needs to see the next action and then click next and approve it? In which case you have to have one recommendation that needs to be either good or bad? Or does your flow of user journey is such that you have three recommended items and anyone that he can choose from is good? In which case you're just looking to have one out of three for the or three out of five. So all of those inputs into deciding what are the metrics we want to optimize for, like you see is a product decision, not a science decision. A model is not a product. In between there is earning user trust. We'll talk about that. We'll talk about human in the loop. We'll talk about explainable uh AI. Measuring feedback and changing behavior. Of course, the beginning and end of this process are led by the product manager. Right, you are the decision maker on what is the problem use case and so on and you're the decision maker on analytics and feedback and what does the product behave in the market. But you're also a shared decision in productization POC and even in the model process. Not unlike traditional engineering sometimes that we write the PRD, MRD and then we hand it over to engineering to build and we wait for the outcome somewhat involved in the Q&A phase and we then we just helped productize and launch it. Actually here you as a product manager need to be involved in the data selection and the labeling. This is just as much a product decision that it is a science decision. I gave some examples on the HBO and feature engineering and and the metrics and optimization selection. So now for the next 20-25 minutes I'm going to talk about each of these seven steps and dive a little deeper and give you some insights and hopefully you can start to ideate and lead these processes internally in your companies. At Amazon we have what we call the working backwards mechanism. We When we design a new product or feature we first write the press release with a customer quote inside and the FAQ, the external FAQ, what are our customers going to ask and the internal FAQs, what are our stakeholders going to ask about this product. Only then we go to design the visuals. In the machine learning phase you'd want to run these discovery workshops with the following three personas, the business owner. Could be you the product manager but it could be the head of marketing, head of sales, head of operations and so on. Someone who knows where the data resides and how to access the data and someone who's going to build it. And then you want to ask them or kind of force them into a process in a workshop manner that answers the following three questions. If you knew X you could do Y in order to get Z. So it's not enough to say I really want to know how many revenue how much revenue we're going to have next quarters. But what is the action that you can drive from that? Well, maybe if you're a publicly traded company you could use that in order to report to regulators. Okay, what does that ROI look like for this? I mean how much do your analysts today spend time or money to design these forecasting models and does that worth a replacement? So keep really asking all these questions. Sometimes we tend to really fall in love with machine learning as far as the insights phase, but we don't really see an actionable thing that we can do with it. And even if we do, we don't always have the KPI in place. Remember that the process of machine learning is can be costly cuz it's an iterative process and because data scientists are relatively scarce and expensive resource. So, make sure you are choosing the use case that has a strong ROI. Here are some examples of how this looks in customers that we've helped with in the past. Which prospects will click on a campaign so we can get the right message to the right person at the right time. Or which customers in the last example here will churn so we can proactively mitigate them and reduce our churn. There are more examples, of course, this is an endless number, but this is just kind of generic BI type of examples. And here are some examples from real-life customer examples from us. Business tools like in the top left corner here, forecasting, churn prediction, support routing, and so on. Consumer application, next best action, sentiment analysis in the bottom left, top right, cybersecurity, anomaly detection in logs, and other examples. Or BI examples, you know, making your analyst use auto ML under the hood in order to generate insights about the business that leverage machine learning without necessarily having to have a data scientist in the team of the BI analyst team. Once you have a use case, try and be critical in the in the thinking process whether it's really in ML or you're just falling in love with idea of doing ML here because sometimes you can say well if you can if you can really solve this in 10 lines of if then, you don't have to ML this. If the data is gradually scaling, you don't have to ML it. We talked about the the hyper personalization example. But if the data is unpredictably changing in real-time like user journeys in in a in an online manner, then that's that's good for machine learning usually. If it's confidence tolerant, if it's life or death decision, well you need some explainable maybe so doctor can make a decision. You want to know why the model recommended the surgery versus not, but it's not it's easier usually when there's some tolerance for whether false positive or false negatives as an explain next few slides. And whether we have access to the data and this is an a critical element. Typical use cases for product managers to dive into are either computer vision, language, or the entire realm of um um patterns in data. Whether that's for ranking, forecasting, personalization, anomaly detection, and so on. So, designing a POC, I like the Marty Cagan approach and inspired that you want to de-risk one of these four elements. Will customers want to use it or willing to pay for it? Will they understand how to use it? Can we build it? Is it feasible from a data science perspective? And is it viable? Should we build it? Is it profitable for us? Ethically right for us to build this product. Now, you want to timebox the POC four to six weeks. You want to make the decision which assumption together with the data scientist and share your insights with the rest of the stakeholders of the company. It's not always obvious that the POC needs to be machine learning. It could be a Wizard of Oz sort of pilot where you have a manual person actually imitating the machine that you'll build later just so you can check whether it's valuable enough to your for your users. Today at AWS, we also have this site where you can actually use pre-trained models. You don't have to go through AWS console. You don't have to use the engineering team. You as product manager can go into this link here and start to test some of our language and vision services. This is an example of one of our customer that use recommendations, which is a computer vision service that we have, a pre-trained one, to match people with the art that they like. And they were able to get up a prototype in 4 hours in production within a week based on machine learning that they didn't have before this process. Now, as you grow, you will see that the product manager, you may first get some help from a data scientist that is a broad resource company resource or maybe even one of our data scientists can come in and help you in a prototyping or ML solutions lab engagement or pro servant so on. But as you grow, you'll see that you want to create an empowered team of product manager working alongside a data scientist. And these people would want to focus on iterating the ideation and the use cases and the math of the things, not the IT and the devops that supports around it. You don't want them to spend a lot of time on getting the right machines up and running or sending the model to multiple GPUs to be tested just for distributed training and so on. All of that stuff you want to put in an ML platform so you start to seeing platform engineers. And as you grow more and more, you have these core teams, multiple empowered teams with a lot of support around it. Now, the modeling side of the work, like I said, it's not just for the data scientist or engineering. This is stuff that product managers can, should, and need to actually own and be a shared decision-maker. Especially around data. Is data available internally or is it public? Do we have the permission, the IP to use it? GDPR, HIPAA if you're in the healthcare space, privacy. Is the data fresh? We talked about the example of using data pre-COVID or pre-vaccination. Sparsity of data. If we're in a time series use case and we're trying to forecast, but we have a lot of missing data in a historic data, that might not be a good support for for a forecasting model. Very often in in anomaly detection, we have very few labels because it's anomaly, right? It doesn't happen a lot. So, we may sometimes want to double index on those few anomalies, kind of over sample them so we can really train a model properly. But we need to remember, data will confess to everything if we force it enough. And the way to overcome this is called overfitting is not just to train on everything that we have. Usually train on 70-80% of the data and we keep a holdout set, 10, 15, 20, 30% of the data kept free from the training, and then we can test validate that the model we train can generalize. And that 15-20% that we kept on the side is a good place to validate as a general example. And then also in deployment, we usually don't deploy machine learning model from from the zero to 100 immediately. We do some canary testing, AB testing, and we deploy gradually in order to make sure that the model that we trained on a certain data behaves the same in real world data. But very often, the problem is that we have missing data. And within that, missing labels is the most common scenario because we need to manually label, manually tell the machine, "This is a cat. This is a dog." Sometimes we need a professional to do this manual labeling. We at Amazon have a tool called Sagemaker Ground Truth, which helps you manage the workforce of the labelers themselves. For example, if you want to send a certain labeling job to three different people, and make sure that it's only consensus vote that this is a label or majority vote. But then, these people might be expensive people that are really experts in their field, and you have a lot of labels to generate. So, we have what's called active learning. While they start to manually label the data, we actually build a machine learning model under the hood that starts to use the data that is already labeled to develop a model and see in the remaining data that has not been labeled yet, can the model already detect clear confidence labeling? And then reduce the amount of effort that's needed for manual labeling. As they advance more, the model gets smarter, and it's better at reducing the amount of work for those labelers. You can also use uh leverage other third parties, or you can start by using an example of recommendation engine. You can recommend the most popular item just to get started. And as the data accumulates under the hood, you can replace the engine when it's seamless. Think of like a hybrid car. It's seamless for the user when it's electric or or gasoline-powered. Same thing here, from a popular um engine to a machine learning engine. You don't have to have a lot of data. Amazon Personalize, which is a our AI pre-trained or pre-built service for recommendation engine, requires thousand interactions between items and users to to actually generate the first model. It can be 25 unique users each having at least two interactions. So, we're not talking about hundreds of thousands of examples and so on. Very relatively much less data is needed and you can start, like I said, in transitioning to machine learning from a popularity based engine. You can also use synthetic data. You can use reinforcement learning, which we talked about before, or use GAN, generative adversarial network, which is two networks training each other and this is how usually fake human beings, for example, are created. One network is trained on all the images and other is creating and trying to trick the first network. When the generator is able to surpass the second network, then that person is considered to be a good enough that it was able to trick a machine for that. Airbnb is an example of a customer that's using Sagemaker Ground Truth to manage their labeling workforce and it's a good example to remind us that some of the labeling can be sensitive and the workforce that you want to manage needs to be internal or within your customers and users. And some can be external because it doesn't have any sensitive data. You can use Amazon Mechanical Turk just for the low cost of labeling generic or generally available data. Now, the outcome of the model comes in the form of a confusion matrix. In this example, for instance, could be healthcare, could be cyber. If you have a true state, which could be malicious or benign, and a prediction. If you're predicting something to be malicious and it is malicious, it's true positive and same for benign. But if you're predicting something is malicious, when in fact it's a benign, that is a false positive, false alert. But there's also a situation of false negatives, when you're actually letting one bypass. You know, you're saying, "Hey, this is this is benign." When in fact it was not a benign action. Now, this is the science. What is in it for the product manager is actually to decide whether you want to optimize for precision, which basically says, "I want to have fewer false alarms, but I can live with missed events." Or you want to optimize for recall, which says, "I don't want to miss any of the true state situations, but I'm okay with living with some false alarms." You can see how this is a product decision, not a science decision. And it depends really on how the product is embedded in the workflow of the user. If you're talking about a real-time CISO, you know, hot room that now needs to make a decision, you need to reduce the amount of false alerts and and alert fatigue and so on, so you want to really focus on precision. But then you got to remember that all the data that was not alerted on is not necessarily clean. You are having some missed events in there. So, when you're doing, for example, manual forensic investigation in offline manner, and the person needs to review 100 percent of the uses, if you can identify 50% that are very likely not to have any uh malicious activities, and that person needs to review only 50% and not 100%, you've just created a lot of value for your company. And that you've done that by optimizing for recall, because you're not looking to have fewer false alerts. The 50%, yeah, I have a lot of false alerts, but you're looking to see that you don't miss any real true state scenarios. Now, it also goes into how you as product managers set the customer, the user expectations. What is the microcopy that you use? Are you saying, "Hey, here is some suggested action decision support system. Here is um probably all of the cases, but you may have some errors." So, there are different ways to position this within the product. Remember that that's why it's important for product managers to understand what is precision, what is recall, and whether you want to change that in the life cycle of the product. Maybe you want to start with one, and as trust is gained through using the system, you want to change to another. So, I said earlier that model is not a product. Something needs to happen in between and that is user trust. There are multiple ways of earning user trust. One of them is adding human in the loop. Human in loop is just like in the labeling example I gave with the Amazon Sagemaker Ground Truth. We call it Amazon Augmented AI. This is where you take the outcome of the model, the inference, let's call it of the model, recommendation or the prediction of the model, and you decide whether you automatically plug it into the application, into the user interface, or you want to first pass it through a human that will take a look at this data. You can do that when the model is not confident enough. You say below 95% I want a person to look at this. You can also say, "Hey, I'm you know, screening for resumes now, and even if the model is very confident, but the the person that is the candidate has a PhD, I want the recruiter to actually look at this personally." And you can also say, "You know what, any 5, 10, 20% of the data, randomly just pass them to the a human in the loop, and with time maybe reduce this number as we earn trust of the product." But there's also a a tool called Amazon Sagemaker Clarify. It helps you detect the bias in the data I mentioned earlier when you have over sampling the data in some areas, or you may have miss weight like unbalanced data in the different classes. So, you're an insurance company and you're trying to cohort your age groups, but in some age groups you have very small amount of people because, you know, you're focused on older drivers and not young drivers, for example. And then this tool will kind of alert you, say, "You know what, this is not balanced data. So, you may want to take attention and maybe either synthesize more data or change the cohort groupings. There are different mechanisms that you can do, but this tool can detect that for you. It can also detect whether you have bias post the training phase and during the the model monitoring. when the model is running in production and it's dependent on data coming in, we're actually creating a baseline of this data when you first train the model. And then once there is a drift from the baseline and you can configure what does it mean a drift for you, which metrics and how much of them is considered drift. You can also design your own metrics into that. Then you can get an alert from that. But sometimes what you need is to actually explain why the model. Why did I not get a loan from this bank based on I just sometimes not good enough to say the model said so. This is what we call individual explainable or local explainable AI. We use a shap s h a p model under the hood. We kind of basically took away the under frustrating heavy lifting for you. So shap is something that's publicly available. It's published on research papers. We just coded it. So you can now use it embedded into the sagemaker platform. You can provide explainable local predictions to your users, but you can also provide that to regulators. Let's say the regulator wants to know whether you use gender or what weight did the gender feature receive in your overall model for underwriting scoring of this group because it's okay to use it, but not too much, not to rely on it in over manner. So this is with global explainable AI. And this is also supported by sagemaker clarify. And now lastly, once you have the model out there, it's very important to remember and especially for us as product managers. If the customer is saying something, we need to listen. How does that behave in real life situation? So we we call this ML flywheel. We start with saying, all right, do we have the data and labels that we can do that you can use for modeling? Great. Can we generate a model meaningful enough in the prediction? Great. Do those prediction and the way they they are embedded in the product with all the earned trust envelopes I mentioned before, the bias detection, the human in the loop, the explainable AI, and all of that. Are those driving the behavioral change that we wanted to achieve with the product? If so, great. But that by definition is going to change the data because we just drove or generated the behavioral change, which needs to be manifested in the data. So, ask yourself, if my model is successful in generating behavioral change, how will I detect that in the data? Will I need to relabel? Will I need to maybe some of the people that have been successfully altered their behavior based on this, they need to be cascaded to another model to provide more granular prediction with a different data set with a new label. They're different decisions. Some of them can be made before you get started. Some of them need to be made on the fly. You just need to get an alert that, "Hey, there is a drift in the model. Now, let's take a look at the behavioral data and rethink and retrain the entire process." Okay, so how can AWS help you along the ML journey that you're at? We have what we consider three-tiered ML stack. And in the top tier is what I'm referred to before as pre-trained, pre-built models. Where if your use case fits one of them, then you can easily and very fast go in from idea into a POC. Like I said, we saw an example before from hours or few days using one of these pre-trained models. Some of them can be great fit for your use case and then you never need to do anything beyond that. And these do not require any data scientists and so on. And some of them could be just a good starting point for your POC. In some of them, you can also tweak or customize the model for your own language. Let's say in the example of transcription in Amazon Transcribe, which is the the speech section in the bottom left in the top tier. I'm still in the top tier here. And you want to transcribe calls to your call to detect sentiment, to detect what's going on, and train your people, and and and manage the entire process. You may have a unique domain. You're working in the legal domain, and you want the training of the model to be tweaked customized to the legal area, or the manufacturing, or the airline industry. Then you can actually create a CLM, a custom language model for transcription. So, you don't have to start from scratch. You do need to give some data from your own, and then you can get the model to be more uh customized for your own needs. But we also have in computer vision, the recognition, chatbot, and Lex, Text uh Comprehend, which is natural language understanding, how to understand what the language that is within the call or within a paper, which is can be extracted using Textract, which identifies columns and graphs and and can extract entities even from radio buttons if it's uh a form that you fill, and so on. But we also have business tools. The example I mentioned before of Amazon Personalize for recommendation engine or Forecast for time series. Fraud Detector as an example for any fraudulent activity transaction that undertakes, and you want to say, "Hey, this IP, email, credit card, how does that impact your score of how likely that is to be fraudulent?" And that can actually enjoy from Amazon 20 years of embedding. So, if Amazon has had some exposure to this data points, that can influence the score that you get from the model. So, you get kind of more than just a model here. But they're also verticalized solutions for healthcare, for industrial, and so on. Then if you do have data scientists, you can start in the middle tier, which is the Sagemaker. You can choose pick and choose. So, we want to take away the undifferentiated heavy lifting. So, if you have something that you think is really differentiating for my business to do my own feature store, fine, then use us for model monitor, use us for distributed training, use other elements that are non-differentiating for you. Um this is a complete uh IDE with CICD that you can manage. And it also has the autopilot, which I I before. If you have tabular data, and you want your data scientist list to get started with some recommendation on ETLs and algorithms and selections, but then have it as an open ML AutoML in the sense that then the data scientist can get the notebooks and see what the recommendation um AutoML did, and then do it on their own. Continue and expand and improve this on their own. It's called Autopilot as well. But, if you are an expert and you don't like uh to use these managed services, you want to build everything from scratch and manage it, we have that bottom layer of the stack as well optimized for TensorFlow, PyTorch, MXNet, and so on. So, kind of recapping, we have about 5 more minutes. I want to leave a couple minutes at least for questions. So, first, we talked a little bit about the product management powered by ML. I recommend running these discovery workshops with internal stakeholders from business, data, and technology, so you can get a long list of potential use cases, qualify whether they're a good fit for ML or not. Then, we can help you go through the prioritizing those use cases and help you choose how to get to the first POC or MVP. We have multiple mechanisms where it's a prototyping engagement where we work together with your team hand in hand, and your team can learn in the process, whether we have a pro server we can build for you, or ML Solutions Lab where you can give us the data, we will build it and hand it over back to you. Different modes of engagement, and and I'll soon show you some slides of the people you can contact if you want to test out different things here in the programs. But, if you already have several use cases, now you want to get to this next level where you want to do an ML platform. Now, you want to create a situation where you have empowered teams, and they don't have to worry about the ML Ops and DevOps. They just want to build the models, iterate on them, improve, and all the DevOps and IT work, let let us worry about that. This is our expertise. Your expertise is your domain, your products, your services. Then, we can help in that phase of the journey as well. If you want to um you build everything on your own and just have us help you train your team and get your certification, we've got the right side of this slide as well, and we will be happy to support and do some training and certification for your team as well. So, with that, I want to thank you. We've got uh I think a few more minutes for questions, and here the people different languages. And we have a very uh wide and very diversified audience today. I encourage you to take a screenshot of this uh screen so you can then know who to contact, and if you have any questions or want any follow-up or or work with AWS on any of your use cases. Arun, thank you so much. What a fantastic talk. Thank you. We enjoyed that very much. You emphasized early on in your talk the importance of developing trust. So, uh maybe you could just expand a little bit on that in particular. What happens when human intuition, that gut instinct, what happens when that comes into conflict with what the machine learning is telling us? That's a great example. I think the question is how do we as product managers communicate that to the user? So, the user may have an intuition. Let's say you're a professional, you're a doctor. And now you have a decision support system in health care that recommends certain action to that patient. But the doctor has an intuition. So, if you position the machine learning recommendation as, "Hey, I know, I'm the machine, you should follow my recommendation." That may can can create some clash with the doctor's um um opinion or you said intuition. But if you're saying, "Hey, this is a decision support. Here's my recommendation, and here's why." Meaning that I am a model that was trained on data of type X. This is a type of patients that I've seen. It was labeled by the community of doctors that have all been certified with this grant and this group and this and this and that. So, now the doctor either identifies that this is not relevant for his patient because the training data was not of the uh patient that he had or the type of illness that the patient had, or that the people who labeled are good strong candidates good strong physicians that he trusts. He says, "You know what? Maybe I will start to trust this because I see that other physicians that I would have called to get some consultancy with, they've actually labeled this as as a tool." So, I think by being able to provide insights into how the model, what data was used, how the process of labeling was conducted, and also in that local explainability. So, I recommended to undergo surgery here because I've seen this size of the tumor was about this big and not that big and because the texture was this and not that, this is why I recommended it. And then doctor can say, "Okay, you know what? I understand. That's a valid point, but you haven't seen or you haven't given enough weight in my opinion to this thing that I've seen in the side." In this way, together, a machine and a human can actually make a better decision cuz either of them can accidentally not notice something or not in not give enough weight to a certain feature, and together, if it's exposed and explainable and open, that decision can be improved. Okay, that that makes sense, but not just uh conflict. Let's consider about possible pushback because humans, we're not all that rational. Uh maybe uh we can accept that ML is providing us um a better solution that we might come up with on our own, but then perhaps we're fearful that we're not needed so much. This is encroaching on our own role, our own job. Is that something you're seeing? Yeah, I think it's always a question with innovation, you know. It depends on how you use the innovation, right? Innovation by itself is not good or bad. It That's why I'm talking to product managers. I am encouraging an ethical approach to AI in the sense that you think, "Should we build it? How do we communicate to the user what to do with it? Do we force? Do we say, 'Hey, this is a machine and this is how it created it's it's a recommendation and these are its limitation. You can decide to overrule it if you want. Hey, "Hey, to go wild. I want to do something different." You know, like Netflix can recommend in the next series for me to binge, doesn't mean I have to take on this recommendation, right? It's nicer if it said, "Hey, you know what? I'm recommending you this series because of da da da da da." Then I know, "Okay, now I understand." And I'm maybe more friendly with that recommendation because I understand where it's coming from. So, I think that's more of the where this this world of machine learning is headed to. A collaborative approach of human and machines openly making decisions better. And yes, the humans can still have the control, but the machine can support that, can detect things that the human may not have the time or capabilities to do. So, it can actually augment what the human is doing and not replace. That's the way I'm I'm seeing things heading more today. Yeah, you're very optimistic. Aaron, I wish I wish we had more time to expand on this. This is absolutely fascinating, but we are sadly out of time. Anyone out there who wishes to find more questions to Aaron, please do so via the via the website, via the forum. Aaron, once again, thank you so much for your time. Best of luck for the future.