Uncertainty in Materials Science Property Prediction: The Good, The Bad, and The Uncalibrated

nanohub.org online simulation and more for nanotechnology. Thank you everybody for joining us today for uncertainty and material science property prediction the good the bad and the unccalibrated presented by Ashley Dale. Ashley Dale is a Schmidt AI in science postdoal fellow at the University of Toronto with Jason Hatrick Simpers. She completed a PhD in physics from Indiana University with a topic in experimental condensed matter physics and a PhD from Purdue University in electrical and computer engineering with a topic in latent space analysis methods for machine learning image processing algorithms. During her graduate studies, Ashley contributed to several projects focused on trustworthy AI for computer vision algorithms and realized the need for these frameworks in the scientific community. Ashley hopes to develop trustworthy AI metrics that add explanability, reliability, and robustness to models used for discovery of novel materials. She would be the happiest if this resulted in some new physics along the way. So, thank you so much for coming today. I'm really excited to get to talk about this. It kind of lives rentree in my brain. So, really quick overview. Um, we're going to do basics of uncertainty 101 just to get everybody on the same page. Um, and then we're going to go through an uncertainty characterization workflow. So, how do we actually calculate this and implement it? And then finally wrap that up with a case study using the nano hub tool um, and using random forest. I'm going to be right up front about the bait and switch compared to my abstract. We're using random force because it's a lot lighter weight, but everything that we uh do will generalize to other models including the align model. So uncertainty 101 within machine learning for material science property prediction, we have this workflow. All right, we start with a chemistry search space, right? And for other problem domains, you could think of this like your original data set or the actual problem you're trying to tackle. Once we identify our search space of you know interest, we want to identify what features we're going to use in our data set. So things like the phase of a material or the band gap or the space group, all of these uh features are going to be embedded in some kind of numerical method. And then that embedding is what's going to go into our modeling. So at every point in this workflow, we have choices to make. And each of those choices can have some uncertainty associated with it. So when we're going through and we're looking at uncertainty for materials property prediction, we have lots and lots of options which kind of gives away this next slide and I would like you all to think about this question is for your particular work case um do your model parameters have uncertainty and do your data the actual things you're putting into the model does each data point have uncertainty? So if your model parameters do have uncertainty then congratulations you might be using frequent statistics. So within the whole umbrella of statistics frequented statistics is what came out of the early 19th century. This comes with an assumption that you have an infinite amount of data that you can just keep sampling from your data distribution and that as you do that your uncertainty will drop to zero. So along with this way of thinking about statistics and thinking about your data comes methods like confidence intervals or significance tests or hypothesis testing and all of those things are coming from infinite data your data is perfect or your model is perfect all of the uncertainty is a property of the data. Now in your use case if you're thinking about your model parameters and your model parameters have uncertainty then this is where you might be a basian statistitian. So in this case you can keep increasing your number of samples but you're going to do it based off of previous knowledge because in this case your model has uncertainty. You're trying to drive down that modeling uncertainty with wellchosen updates to your model parameters. And so basian seems to be really really good at estimating prior knowledge. But what we see here is that in this framework uncertainty becomes a property of the model and not a property of the data. So kind of to drive this home, I'm going to present this controversial comic. Um, and I'm going to explain the joke to y'all cuz I'm that funny. But the idea here is that we have two scientists and they have an experiment. All right? And they're going to see did the sun explode or not. So the way their experiment works is they have a nutrino detector and they say okay it's going to measure nutrinos and if it measures the correct nutrinos for the sun exploding and it gets two dice that both are six then it's going to lie. So the odds of this machine lying are very very low otherwise it's telling the truth. So in the next scene we see that the machine has rolled two sixes um and therefore or excuse the machine has not rolled two sixes but the probability of it rolling two sixes is very very low. Therefore it's going to get a very reliable result that the sun has exploded. Now if the sun had actually exploded everybody on earth would be dead right even though it's dark you know the earth isn't going to survive that kind of event. So the Bay Institution updates his prior based on the fact that they're all alive and says, "Hey, bet you 50 bucks the sun hasn't exploded." And the controversy gets into uh why this is like a bad representation of stats in general. In truth, if you are doing both frequentist statistics and basian statistics correctly and you've done like a beautiful job of it, your results should always agree. [Music] But going back to the big picture here, um, in your particular use case, if neither basian or frequentist modeling seem to match the way you think about things, then congratulations, you might be doing machine learning. So machine learning is borrowing ideas from both of these things. You can have confidence intervals, you can update your priors, you can do hypothesis testing, but you're no longer coming from those same assumptions of perfect data or perfect model. you just kind of have uncertainty everywhere. So when we're doing machine learning and we're thinking about uncertainty, we want to be thinking about these different aspects of it, right? We have uncertainty from the data, right? Is the thing that you're putting into your model actually reliable? We have uncertainty from the model parameters. This is drawing on that basian understanding. But then we also have uncertainty from the algorithms themselves, which is beyond just the number in the model, but how we're using it. Right? So the optimization process, the model you chose from the very beginning, the model hyperparameters, all of those things have some kind of uncertainty with them, right? Which is how we get these amazing hyperparameter searches out there. On top of that, we have uncertainty we can fix, right? So if you have some bad or noisy data, maybe you can clean it up. Maybe you can add a regularization to your model. And then we have uncertainty that we can't fix. All right? And on top of that, we have acceptable uncertainty. Right? So maybe we're okay with some error um or some region where you know we're not quite sure maybe we want lower uncertainty for some kinds of predictions and other predictions and then we also have unacceptable uncertainty right so I think we've seen lots of examples of self-driving cars where you know there's just unacceptable uncertainty in their uh predictions from the models driving those cars. So kind of thinking about these different sources of uncertainty. What is acceptable? What is not acceptable? What we can fix? What we can't fix? Where are these different things coming from? This leads to the idea of different kinds of uncertainty. So I have this lovely plot here. We have our input features. All right, the x and the y. And then we have model predictions which should ideally be like a straight line through these points. Now the different shaded regions are talking about different kinds of uncertainties. Right here between -6 and4 we have a region where we have no data. Right? And our model still wants to make a prediction here. But because there's just no data and no information, we're going to have epistemic uncertainty. So epistemic uncertainty is coming from missing information. All right? It's talking about what things could we give the model that would make it more confident that this blue region here is in fact an almost smooth line. On the other hand, if we look between pos4 and pos6, we see this really noisy data, right? And if our model wants to accommodate this effectively, it's going to have a lot of uncertainty just due to the noise. There's randomness here. And so that's ali uncertainty. We cannot make this uncertainty smaller because it's literally not going to be modeling the data anymore. The randomness is built in. The total uncertainty of this system is the sum of the two. The alitor uncertainty and the epistemic uncertainty. Ideally, our model would be able to handle both kinds, right? It would be able to figure out what this missing value is here and it would be able to not be thrown too far off track by the alitor to. That's part of what we're looking for in the analysis that's coming in the following slides is how do we get to this point where we know what kind of uncertainty we have and how to handle it. So in order to measure it we need to be sensitive to these different kinds of uncertainties. And in general you can break down measurements into four different axes. So with accuracy, what we're looking for is the all of our values are very close to the true values. So I'm going to introduce the parity plot here. We have the actual value on the x-axis and the predicted value on the y ais. If our model is accurate, then these two values should be very close and it should be a straight line. All right, an inaccurate model is going to have things far away from the par line. That leads us to calibration. All right. So, calibration is a very tricky thing to talk about in machine learning context because most of the time we don't know the distributions that we're drawing from. And calibration is saying did your model learn the correct distribution of predictions. So on the x-axis we have the expected cumulative distribution. All right. So thinking about what is the probability of the sample that we expect to measure. Is this a common value in our data set or an uncommon value in our data set? And then on the yaxis we have observed cumulative distribution. So did our model correctly capture that this point is an outlier or not? Now, I'm going to talk about this more later, but for right now, what you need to understand is that if your model is calibrated, you should have error bars for each point that overlap with the parity line. All right? If you have a model that's giving you uncertainty estimates, if it's giving you those error bars and the error bars are not overlapping with the parity line, the model is definitely unccalibrated. The next thing to think about in terms of uncertainty is sharpness. Okay. So on my x-axis here I have predicted standard deviations. So already we're coming into this idea for the first time that the standard deviation for a predicted value is equivalent to the uncertainty. They're proportional. And then we have the normalized frequency. So how many samples we have that have a certain standard deviation of uncertainty. We would like all of our predicted standard deviations to be very small. So we want this distribution of values to be shifted to the left because my zero axis is right here. We want small standard deviations, very sharp distributions. And that again shows up in our error bars. So here you can see the error bars are pretty small. um they're they're not large compared to this plot on the to the right of it where the error bars are very large. We don't want big error bars, we want small error bars. And then the last one is dispersion. So here we've got the same x-axis, the predicted standard deviations, but instead of just looking at the average value of the standard deviations, we're looking at how broad the distribution of standard deviations is. So the narrower uh the narrower of the dispersion is is better. We don't want a wide standard deviation of standard deviations. This is again showing up in our error bars. All right. So here we can see the error bars are not all the same size that corresponds to the blue histogram here which is wide. This plot on the right the concentrated error bars we want them all to be about equivalent. All the error bars are the same size. So from looking at the error bars, we can start getting a good idea of these parody plots and how they relate to the uncertainty of the model. All right, let's get into our characterization workflow. So in general, you can break down your uncertainty characterization into the following steps. The first thing that we've already done is identify potential sources of uncertainty, right? And the more specific you can be about this for your problem, the better your uncertainty analysis will be by the end. Um, so if you're realizing that, you know, you have not one learning rate that works for your model, but perhaps 10 learning rates that work work equally well for your model. You have a lot of uncertainty in your learning rate hyperparameter. Same thing for your data. So if you're realizing that your data is very very noisy rather than very very clean data, um that will be a source of uncertainty that you should consider as you go further down. Once you're identifying these potential sources of uncertainty, the next thing to do is figure out where you actually can measure them. So this is actually a challenging problem. For example, if you have noisy data, is the correct space to measure it on your input, right? Should you be doing your noise analysis there? Should you be looking at the output of any layers in your model to see how much the noise is perturbing the activations there? Um, should you be looking at it at the end, right, and seeing, okay, did that noise survive into the prediction space and is there some kind of mutual information measure between input and output that says what the uncertainty um is. All of these things need to be thought about. Where are you going to measure your uncertainty? Where can you measure your uncertainty? what things are appropriate for your problem statement. As soon as you figure out your measurement space, the next thing you have to do is actually quantify it. And that's where we're going to spend the next part of this talk doing is quantifying the uncertainty. So quantifying uncertainty is still a very active area of research. I'm going to give just the basic overview today, but I'm happy to pass along more papers if you'd like a deeper conversation on this because there's a lot of good work being done. Once you have a quantification of inner uncertainty, you need to figure out what it means. Is it the good? Is it the bad? Um is it going to, you know, ruin your day? Is it acceptable or unacceptable uncertainty? All of these things. And that kind of brings you to your final conclusion. Now, these steps are pretty well laid out for classification models, but within material science, we are really interested in regression. And the interesting thing about regression models is that not all regression models will return a probability distribution with its prediction. So the state-of-the-art for quantifying uncertainty within regression models, although this definitely applies to classification as well, is you can retrain with different hyperparameters, right? And so basically you are getting an ensemble of models that will generate your in your prediction um distribution. And the great thing about this is you have a good understanding of the role the hyperparameters play. Um but the bad thing is that it's very computationally expensive to keep training models. How do you know when you've trained enough models? And then also there's a very large search space associated with this. Your next option is to use a model that has an intrinsic uncertainty estimate associated with it. And basian neural nets are really the gold standard here. Um but there are other things like Gaussian process regression uh which do a very good job as well. Like we're not ruling those out here. Now the problem with this is that there's not a lot of algorithms that will just do this for you automatically. And again for example in the case of basian neural nets you are still kind of end up generating an ensemble of models and that can also be expensive right there's just a cost involved with doing ensembles and then the last way where there's also a lot of good work being done is that you modify the model after it's trained. So the idea here would be you train your model um you get some kind of prediction out of it and then you realize hey this isn't quite enough to get me my uncertainty estimate. So maybe you chop the end of the model off, you stitch a new model on um and that new model will help you train to get a prediction distribution after you've already finished training for the predictions, right? So it's just learning to predict the uncertainty. So the good thing is these methods, you can do them to almost any model. The bad thing is the model may not necessarily perform the way that you want it to after you've retrained it. So, I've got two good references here at the bottom, and I encourage you to look these up, take screenshots, um, but they will help you dig deeper into these different types of methods. So, we're going to go light on the math today. There's a lot of math in this, but the first thing that we should remember is how do we calculate the total variance for a single sample? All right, so let's say I've taken one of the ensemble methods here. I my model is giving me n different predictions for a single input and with those n predictions I'm going to assume a normal distribution. All right. If I do that I can take all of my predicted values and compare them to my target value. So I'm assuming a ground truth in this case and calculate the total variance for that sample. Straight up sixth grade math. [Music] Once we have this total uncertainty, we're going to try to break it down into the epistemic and ali uncertainty just like we talked about a couple slides ago. So here we can reduce the epistemic uncertainty with more knowledge. The aliatoric uncertainty can't be reduced. It's unexplained. And we want to get good estimates for both of these as best as we can so we know what's going on. So the epistemic uncertainty is good because it's explainable. You can actually attribute it to something and it's also fixable. If you have more information present in your model, then you can reduce it. So the way we're going to calculate this is first we're going to get the mean of the predicted values. All right? So you're just summing all up taking the average and then you're going to calculate the variance of the predicted values with respect to the mean prediction. So before we had our target value here, that ground truth. Now we have just the average of all of the predictions. The aliatoric uncertainty because it's coming from these random noise sources where we don't have a good idea of how to sample the distribution is much much harder to deal with. It is the bad kind of uncertainty. So the way we're going to tackle this is we're going to tackle it with a difference. We're going to say that the alioturic uncertainty, the aliotauric variancy is coming from the total uncertainty minus the epistemic and that's how we're going to approach it. Now, all of this has to be taken in context of the model calibration. All right? Because here's the thing. If your model is giving you bad distributions of values, if it's not predicting things properly um within that target distribution, your uncertainty, your variance of the distribution is going to be essentially meaningless. So this comes to the idea that just because your error bars overlap with the model parody line doesn't mean that your model is calibrated. In order to actually do a good job understanding if your model is calibrated or not, you really need to do one of these calibration curves. But doing these is difficult for regression models. So there's actually no clear consensus on best practices within the community right now. What I found um these two tutorials are good references for like how do you actually start tackling a calculation of this um and I found them very helpful for my own work. So I'm going to put them up here. But right now, as far as how do you go about and estimate the uh distribution of what you should have that becomes a basian statistics problem again, right? So we're we're butdding up right against the edge of what do we know, what do we not know, how do we know the models learned it appropriately. Um the good thing is if your models doesn't have error bars that overlap, it's definitely unccalibrated. So you can say for sure I should not trust this uncertainty calibration which brings us to our case study. So setting up the experiment. All right. What we're trying to do is we're going to try and get our model our random forest model to generalize to an out of distribution element. We're going to take our data set and we're going to split it into chemistries which do not contain element X. So, element X can be F like florine. Element X can be iron Fe. Um, element X can be literally anything in the periodic table. But then your test set is going to contain element X. All right, that's going to be what's going on here. We're going to take our training set. We're going to train our model on it. So, the model has never seen element X. And then we're going to ask it to predict the energy of formation for both our training data and our testing data. So we want to know does our model generalize to chemistries containing an element that it's never seen before. Now my collaborator Kunging did a great paper on this and I'm happy to piggyback off of his work where basically he showed that whether your model generalizes or not is going to depend on what element gets left out of the training set. So you can see we have this block of elements right here and flooring in particular does a not great job of generalizing when it's left out of the training data set but everything else looks pretty good right if we're looking at our R squar score so that's coming from those par plots we can see most of them are in the 0.9 territory unlike poor florine who's at 0.15 so the idea here is that whether it's going to generalize or not we already know depends on what element was emitted from the training data. So within a random force prediction regression problem. So now we're asking it to regress those uh energy of formation values. Okay. And we're going to look at how the model does during training. So in this case when I say an untrained model I mean a random forest that does not have the appropriate number of trees for the problem. All right. Um, and as we keep adding trees, we're going to say that the model is training here. When I leave florine out of my training data set, the model does a perfect par. R2 equ= 1 almost right away. This is for two trees and my R squar value is already pretty much perfect. But for my testing data, right, so test data now contains florine. My R squar value is just terrible. It's.2.4. 4.6 and adding more trees is not solving this problem. My model is kind of stuck. Another model which I've retrained using a training set which does not contain iron does a very very good job. So there's a little bit better training right here at the very top. Right? Even though it learns very quickly. Um but we can see that adding more trees actually does help it generalize to iron a little bit better. And looking at the scale of this, this is 888. This is 98 models doing great generalizing. So on the left we're expecting to see some high epistemic uncertainty and on the right we're expecting to see some low epistemic uncertainty because we know that the model on the left does not have enough information to generalize to that out of distribution case. Now when I go through and I do the uncertainty decomposition, right? So now I'm looking at the epistemic and the aliastoric uncertainties calculated using those equations from a couple slides ago. We can see that on the left the aliotauric uncertainty is low and constant. And this is actually what we would like to see for an aliotauric uncertainty, right? We know that it should not be getting smaller. We know it should only be staying the same or getting bigger. The epistemic uncertainty though has a huge gap between the training and the test data. Right? So we can see that here my test epistemic uncertainty is much higher than my train epistemic uncertainty. On the right for the model which did generalize well the situation is not as clear. So if I'm looking at my blue lines and my orange lines for aliotauric uncertainty, they're low, but they're not quite constant. My orange line actually does decrease along this curve here, even while my blue line stays pretty flat. And that's concerning here. We would say that the decomposition did not really do a good job tackling this problem because we would expect to see at a minimum the aliotauric remain constant or get bigger. The epistemic is also kind of noisy, right? Our epistemic uncertainty is very very low. It's on the same scale as the aliotauric, but it's not smooth. It's it's kind of jaggedy. So there's interesting things going on with this model, but the uncertainty situation is not a good candidate for analysis using this technique. As far as the final step, when we're actually analyzing these results, what are they good for? Well, if I'm taking my epistemic and ali variances, those uncertainties, and I just plot them against each other in a scatter plot for each sample. So one point here it's representing one sample in my data set. You can see I actually have this region where I have high epistemic uncertainty and low alluric uncertainty. And that's really really interesting from a materials discovery standpoint because if you start thinking okay in a sparse data regime right if you're in an experimental setting or a very high cost computational setting how do you know where to look next? Well, if your model is telling you, hey, here's a part where it's very not random, right? My randomness is low due to aliatoric uncertainty, but I don't have a lot of knowledge, that can be a good clue of where to start looking for your next target sample, whether you need to calculate it or measure it. So, when you're balancing exploitation and exploration in this kind of context, you know, are you going for the safe bet or a riskier bet, this can help make that decision. So, turning to the notebook for today, um, here's kind of what I'd like to walk you through. First thing we're going to do is we're going to choose an element X based off of this periodic table. Um, I'm going to stick with flooring because it's kind of really an outlier out there, but you're welcome to choose any element that you like. What we're going to do is we're going to track the change in our uncertainties as the random forest model gets larger. So we're going to start with a small number of trees and then get bigger and bigger and bigger and then we're going to plot them for whatever element that you've seen. Do you see good separation between the epistemic and the ali uncertainties? Is it going to be a good candidate for this kind of analysis? And what could that help you work on next? Ashley, really quick, did you want to answer some questions? Sure, let's do it. Okay. Can you combine GAN for new material development? Yes, you can. Um, but I think that, you know, you need to be aware as with any model what you're expecting to benefit from doing that. Great. And then the other question, where does calibration lie in the whole workflow? Should be used an ensemble of models or keep a track of calibration with each new training run? Ah okay let me think about this for a second. So for me when I did this calibration was not something that I could track during training because I was inheriting my collaborator's models. He just gave me the models and I needed to figure out if they were calibrated or not. So in that sense it can be done completely at the end of training. Now, if you are trying to improve calibration as part of your training process, there are methods which do require you to implement special loss functions while training your model that help you improve that calibration. And in that case, you do need to track calibration at every point during your training workflow, but it's not necessary all the time. Great. One more question and then we will continue. There's only one more. Are there any resources for uncertainty in reinforcement algos? So any of the methods that use or any of the uncertainty methods that I'm discussing here and most of the uncertainty methods in general can be used for reinforcement learning. All right, on we go. All right, so walking through this one at a time. I have the basic equation up at the top here. We're just going to start running things. So I did include uh this script to download the data set if you're interested in in running this locally but it should be loaded for you on the tool. All right. So first decision I'm going to keep E for as my enalpy target my energy of formation. And I'm going to change my element right here to foreign goes through and gets the training and test. Right away I'm going to change my number of trees in this forest too. Going to initialize the model and fit. Hopefully it should just take a second here. Looks like it did. I'm getting my ground truths um for my training data and then my predicted values for the testing data. So this right here is going to give me averages, the Y bar that I was talking about. Now that I've gotten my average values for both my training distribution and my test distribution, I'm going to go through and get the predictions from every tree in my random forest. So right now there should only be two trees. And it spits out the two values. Do a little bit of work here. Now these right here, my variance estimate is quite literally applying that same equation in my slides twice. So whether I pass in um my ground truth value or my predicted value is going to determine which variance estimate the epistemic or that or the total that I'm getting from this notebook. Now in this case uh because the way I have set up my problem my variance estimate is actually equivalent to my mean squared error which is why I have labeled my variable this way. So what I'm going to do now is I'm going to pass in the mean of my predictions in order to get the epistemic uncertainty and I'm going to use exactly the same get variance estimate function that I did before to get the total uncertainty. All right. So, I've taken the difference between my target and my uh excuse me, my my total and my epistemic. And what I'm going to do right here is I'm going to actually visualize the distribution of the uncertainties. So if I'm looking at how these things are spread out, you can see that for my training data, my total uncertainty, my epistemic uncertainty, and my aliatoric uncertainty are close, but not exactly the same. My alluric uncertainty is actually a wider distribution, right? It's got more uh meat around here on five than my epistemic uncertainty distribution does. But the opposite is true for my testing case. My testing case has very low epistemic uncertainty and most of my uncertainty is coming out in the aliotauric equation. So in order to keep units on my par plot, I'm going to take the square root of my variance and just get the standard deviation. And oh boy, look at those error bars. All right, so the good thing is that most of them overlap with the parody line, right? Like even this guy right here, he touches it just barely. Um, but they're very large, right? So, I'm not going to say that my model is unccalibrated in this case because the arrow bars do overlap. But I also can't say that it is calibrated because I haven't done that uncertainty calibration. On my left from my training, I can see that I have a wide range of error bar values. So, I have very small error bars down here at -4 and they generally trend larger as my target gets more positive. So here you can see I've I've done a model that does not meet those slides, right? Where we talked about your error bars should be small and they should all be about the same size. Not happening for my training at all. My testing is better in the sense that the error bars are all the same size, right? Worse in the fact that there's fewer small error bars. So we have a lot of uncertainty in the testing values. And also my R squar is super negative. It's negative.2. So we're not super happy with this. And that's why we're going to start retraining as we go through things. If I just plot the standard deviations against each other, it's really easy to see that this distribution of sample uncertainties for my testing data is wider and larger. And then that final plot. So if I'm looking for that region of high epistemic uncertainty, low aliatoric uncertainty, I don't see that for my training data right now, but I could potentially have it for my testing data. Right, we have this lower right region in the log scale where things are looking kind of interesting. So I'm going to create just a list of results for myself here to start tracking things. And now I need to be careful to uh not erase these as I go through. So I'm going to say two trees and I had let's see where did I print them out. I did not. We're going to do the trained and the test. Will it let me do a new code? Yeah. I'm just printing these values for myself so that I can put them in my table that I'm going to build as I add more trees. And I'm going to take the mean of these. So right now what I'm doing is I'm saying that saving the average into my list of results. This is not elegant. I would encourage you to write scripts to do this. That's enough to get us going. Throw this into a list. Now I'm going to run it again with five trees. So coming back up to the top here. Should still go pretty quickly. I'm going to capture these while I'm at the top. adding these to my list of results. It looks the same. Did I not wait for it to finish running? Interesting. So, if I go back and I analyze my plots again, um you can see the situation hasn't changed much at all. So, I'm going to crank it up again to more trees. And basically at this point you're just looking for at what point does your model behavior start to converge. Just like any training problem. So, I'm not sure I made this clear earlier, but one of the nice things about random forest is that it's actually assumed to be uh calibrated. And that's due to the randomness built into the bagging algorithm for random forest. But trying to get a theoretical guarantee of that can actually be pretty hard. All right. So now if we're looking at things, we can see that increasing the number of trees is starting to drive down our training value, but our test is actually getting larger. And going back and looking at these plots again, right? So we can see that the error bars on our random forest parody plot for the train data are decreasing. They are hitting that nice same size, you know, still overlapping pretty well. the testing situation is not improving and in fact our R squar value has gotten worse. It went from0.2 to0.4 and again that's matching the trend that we're looking at. If you look at our uncertainty distributions here, right? So train again is looking more and more left shifted, right? So that's good. We expect to see that the test distribution is actually starting to look biodal with maybe a third peak coming out here. And so what this is telling me is that there's actually going to be groups of samples which are behaving kind of in their own little clusters. Right? So you could think about this data set and as I'm playing with it um I have one group of samples right here where the uncertainty looks like it's becoming left shifted. may be falling into uh the range of good predictions, but I have two other distributions coming over here that aren't behaving nicely at all. And this distribution around three is probably what's driving that really bad R squar value for my testing. And then as I look at my scatter plots here, I can see that I've lost that really small clump of really low epistemic uncertainty, a relatively high aliatoric uncertainty. And that's telling me that those early models, they were kind of overtrained. In a weird sense, random forest works backwards because it's an ensemble. Only having one tree or two trees in your random forest means the model is overtrained. Adding more trees and making your model bigger and having more uh numbers in it. I'm not going to say parameters cuz random forests are not parameterized, but having more numbers in your model actually helps it generalize better. So increasing the size of my random forest, I can see that this clump went away and I'm starting to get this region of epistemic uncertainty, low ali uncertainty. Same thing for my testing data. So if I'm looking at this list of results, what I'd like to do is plot them, right? So I'm going to just do array of results. I'm going to print my array. Make sure everything looks good. Oh, I didn't rerun it. That's why. There we go. So, now I'm going to throw together a plot. The first thing I'm going to do is I'm going to plot my training data, which is my first column. So, it'll be array results. And my first column is the number of trees that I have. So I'm going to do all my rows column zero. And then my second column is going to be my training data. So I'm going to do array results rows column one. I'm going to label this train. And there we go. We can see it doing a nice decrease. Going to add my test. Do we have any more questions while I'm sitting here typing away? what's the version of the Jarvis data that you're using? So the one that I'm using is actually coming from the paper that I cited in the uh PowerPoint slides. It is available on Zenodto as a download link. And the reason I'm using that one is because it already comes prefeaturized using Matt minor features. Um but the actual version if you wanted to redo that featurization itself is um Jarvis 3D DFT and then I think from 2018. All right. So you can you can see here my test stayed flat, my train went up. Um, and all of this is kind of keeping in line with what I saw in the PowerPoint slides, right? I'm able to reduplicate those results. So, yeah, at this point, that is the workflow. Um, and you're welcome to keep playing with it. Feel free to send me pictures and questions. Um, pictures, questions, results, paper requests. I can go ahead and put my contact information on the screen again. Yeah, thank you so much for coming. I really appreciate everyone taking time out of their day to be here. And thank you so much, Ashley, for your presentation. It was wonderful. And we do have a couple more questions. Great. Let's see. If one uses a different model other than random forest, what will change and what needs to be considered? Oh my goodness, that's a huge question. So the workflow can stay the same. You would need to do the same kind of analysis that I showed for the workflow between the two plots. So I'm going to go back here where you can see on the left my assumption that alitorric uncertainty remained constant was a good assumption being true. On the right the decomposition didn't work as well. So at this point is when you make decisions about whether or not to trust your uncertainty analysis or not and that would be the big thing that needs to be considered. So there's a very real possibility of you know how certain are you about your uncertainty? Is the model calibrated? Um are your analysis steps based off of good assumptions? Right? Remember at the very beginning we talked about the assumptions coming from the statistics of frequentist people and the statistics from the basing people. um those come with different assumptions. Are those assumptions holding true for your machine learning case? And that would really be the best answer I can give to that question. But it should work. The workflow is generic. It works with any model. Okay. Next question. When we omit one element out, does the total amount data points available with that element make a difference? For example, if there were very few F data points to begin with, would that skew the results? So, it could, but not necessarily. The this particular data set is very very large. So, within the training data set, um, no matter what element you leave out, there should be around 70,000 samples. The testing data set usually has around 5,000 samples. in general for this particular problem that number of samples is sufficient to generalize to the task right and you can see based off of this uh figure from my friend Kongming's paper that if you leave out any of the other elements there is enough information here now if you are talking about fewer and fewer samples then basically what you need to do is introduce better uncertainty estimates for what you expect to get right so if you have say only three florine samples left out of your training data, right? Because there's only three florines in the case for this problem, it's not going to matter much because even if you have 5,000 florines, it's still bad. Right? If you choose a different element like venadium here with an R square of.979, that venadium was left out of the training data. So even if you only have three venadian samples, you can still expect your model to generalize. One more question. Is there a way to include the uncertainty of the samples themselves, say measurement error into this workflow? Oh boy. So this is actually a paper I was writing or not writing but reading last night and the answer is sometimes for basian modeling for basian neural networks. There was an recent archive paper which discussed how to incorporate uncertainty estimates as an input to the model. And in that case they found that they could include uncertainty as an input to the model in only a few cases of basian neural networks. So remember basian neural networks are kind of the gold standard here. Um they said it worked for ensembles and they said it worked for one other uncertainty method which I am forgetting but let me send that paper along to you really quickly. Okay, I'm looking and so far we don't have any other questions. A lot of thank you for a great webinar. Well, I don't see any more questions. So again, Ashley, thank you so much for a wonderful webinar. We really appreciated it. It was my pleasure. Thank you so much for having me.