Lecture 5: Decision Trees and Tree Ensembles | 2025 EdEC Machine Learning Bootcamp

Thank you. Uh appreciate everyone attending. Uh so so I'm if you don't know me, I'm David John or DJ Gier. Uh go by both, but either one's cool. Uh I'm currently machine learning scientist 5 in Sizzle, also jointly appointed in Ralph. Um a little bit about myself. I started out uh my machine learning journey began back in 2007 when I was I got accepted into a a research experience for undergraduates at Oklahoma uh where I was also going to to undergrad at the time and got to work with uh then a a um relatively new assistant professor named Dr. Amy McGovern uh who you might have heard of at this point. Uh but back then AI was a very niche sort of thing. Uh, and there weren't a ton of people doing it in the field. It was were kind of coming out of out of the AI winter of the late 90s, early 2000s. Um, but I got my start with storm classification with C meanings clustering and decision trees and then got got to was interested enough to keep doing machine learning, did another internship with with Amy over the following year and summer and picked up a a programming language that was also kind of new to the atmosphere sciences called Python. Yes. >> And and turns out Oklahoma happened to be a a very good place to be if you're interested in machine learning and Python at in the late 2000s early 2010s because we had a lot of uh I would say pioneering people uh trying stuff in in on on both sides both in the machine learning realm. kind of a a core group of people like Amy um Bappa Lakman or Lack if you if you happen to know him, Mike Richmond, uh Kim Elmore, uh and and other people are encouraging us like Kelvin Drogo Meyer who was kind of a big proponent even back back at that point. Um and then uh I also got introduced to a broader AI community in the in and the AMSAI conference and the AMSAI committee which included people like uh John Williams who was a project scientist at Encar at the time now was a weather company. uh Sue Halped who was was at Penn State at the time then came to ENCAR and then I I continued to do machine learning and weather related stuff into grad school and worked on extreme rainfall and then renewable energy thanks to Enkar putting on machine learning contests related to wind and energy uh and then I I was able to come out to visit John and then Sue uh as a graduate visitor became an ASP posttock uh in 2016 used that opportunity to get into deep learning when when deep learning started really being a big thing uh and and then stuck on and built our what's now the miles group uh after being hired as one of the first machine learning scientists here. Now the car wasn't doing machine learning they've been doing it since like the late 80s uh the first machine learning conference in atmospheric science was held in Boulder in 1988. So uh you can actually look there's a paper on it and if you we uh Sue led a history of AI and atmospheric science paper as part of the AMS centennial. So if you want to learn more about all those all those details including that that first conference and the first way some of the first waves of AI uh go check that out. Uh what I want to cover today more broadly is uh brief histories of decision trees and decision tree ensembles. uh decision trees, how how they work in their sensitivities. Uh we're talk about random forests, uh how they work in important hyperparameters, gradient boosting, uh another ensemble technique which make which is super powerful uh but but does require a bit more tuning than random forests uh the benefits and drawbacks of resistry ensembles, some applications of these in the atmosphere sciences, and then a brief introduction to explainable AI for tree methods. So, so now I'm going to take us back on a on a on a journey back far into the past. Uh, a lot of people think AI begins in like 2012 or 2022 even like when when oh chatt came out now that's AI. No, we we are going much much further and we're going to follow the journey of not the uh of in particular uh this one guy named Leo Briman uh who who if you don't know who he is you your life has probably been affected by him in one way or another you don't realize it uh because he's had some involvement in in in quite a few areas of statistics and machine learning uh he started out as a I believe in philosophy and then while he was at Colombia for his masters basically got convinced that it would be better to do math instead because it was hard to find a job in philosophy. uh and then got he went to C uh Cal Berkeley for his PhD uh in in math went to UCLA as a professor did some more fundamental theoretical math work including uh he has a theorem where he's co-enamed with like Claude Shannon of information theory Shannon entropy fame uh and then left his tenure at UCLA professor job to become a statistical consultant because he thought well apply statistics seems like an interesting area to get into and so he consulted with with people including you like different cities in the state of California. He he he did some like uh statistical modeling for ozone prediction uh like next day ozone so like pollution ozone not stratosphere ozone u because you know LA 1960s and 70s ozone was kind of an issue at that at that point um and he also worked some classified research in the military and in that work he found that like regular statistical models uh didn't seem to work all that well for actually predicting stuff and and and he found that actually optimizing on on on on pred predictions turned out to be really like to get much better results for actually predicting the future. Uh and and he found that decision tree models are a particularly good way to do this. They're fairly robust to noise. Uh and he he and uh some other colleagues uh including Jerome Freriedman who's also on in this book wrote a book called Classification Aggression Trees that came out 1984 but this kind of summarizes their work from the 70s. They wrote a book instead of writing a paper because they didn't think they could get it published in a statistical journal at the time. Um and and the cart method uh the cart version so there's other decision tree methods but the cart method it it it if you if you use scikitlearn which I'll talk about a bit more cart is the kind of underlying model for for for all the for the trees in in in scikitlearn and in a bunch of other applications. um he can uh in 1980 Leo uh basically went back to academia got a visiting professor position back at Cal Berkeley where he's working with the other statisticians there. So so people like Gar Friedman, Rob Tipani um so some of those also um uh uh people also invented the bootstrap. So if you're familiar with the the nonparametric statistical technique the bootstrap that kind of that's one of the things that kind of came out of this group and Leo one of the things he did with this was he uh like oh this could be an interesting way to to uh train ensembles of machine learning models in particular decision trees and so he admit bootstrap aggregation or bagging and and that paper came out 1996. uh there are also a bunch of other people doing decision tree ensemble work uh in 96 there's a paper about another algorithm called boosting and this particular is called ada boosting so Leo did work in ada boosting he did some work in neural networks in the 90s he actually had retired in 1990 but he was still like as a merit still like a lot of his like work he's not remembered for today was happened after he retired it's kind of uh uh so so even for for the more senior scientists on in the room and on the call there your your most impactful paper may may still be in the future. Uh thank [laughter and clears throat] >> uh and this work kind of culminated uh in 2001 where three really key papers came out uh all like kind of in a short span of each other and they're all kind of related. Uh we have so the original random forest paper came out 2001. uh the gradient boosting paper. So so boost there's a generic boosting method on on trees and then uh Freriedman and I think this help from Brian and other folks wrote basically figure out how to generalize this to arbitrary models and how to uh but trees seem to still work the best at this and and how how to like regularize them and get get them to perform really well. So that led to the gradient boosting paper and then uh they also wrote this very provocative paper that I highly recommend everyone read called statistical modeling the two cultures. Uh but basically it's sort of the clearing call to to like the machine the philosophical basis of of like a lot of modern machine learning development and basically saying that we should focus on prediction because predictive models tend to perform better in in a lot of empirical real life situations. Um and and it inspired a a back and forth within the p like response papers to the original paper that are all published together as a as one as one volume now uh from other famous statisticians. uh uh and and his response to them. So uh someone wants to get into the meeting. They they heard me talking. They're super excited. So >> but highly recommend you read this paper. Some of the things like some of the things it also talks about like like the uh trade-off between interpretability, accuracy come out of this paper. Uh that sometimes you hear about oh if we want more interpretable means we may need to sacrifice accuracy. There's been research since then that says that mean that's not necessarily the case. You can find accurate but still more interpretable models because of something called the rashimulum set which I think is also discussed in this paper. Uh so read the paper and then read other follow on to it. Uh Cynthia Rouen who's done a lot of work with interpretable machine learning especially in the past uh 10 years or so uh she she has a paper that just came out on archive this past year that that kind of follows up on this and I can share the link to that later. Um so we have random force we have gradient boosting uh but we're missing one key piece we need uh like so the random force algorithm was originally written for and then u Adele cutler worked with Brian Men also wrote a basically a package in R to supports random force so you could call the the forran code in a nice way uh and there are some Python machine learning packages by in in the late 2000s but they weren't super performant or they linked to some other compiled library or some other language So they weren't they were hard to install. They were hard to read the code. So a Google summer code project that started in 2007 but then was taken up by uh some French machine learning experts in uh in RIA uh after that uh turned into what's called was now scikitlearn which is pro like although I I think some of the like like tensorflow has eclipsed it in terms of number of github stars it's still I was looking at the pi stats is still I believe the most downloaded machine learning package uh even today uh and it has basically The philosophy behind it is is it's really easy to install. Doesn't have a lot of dependencies. The interface is very simple to use. Uh things like caris borrowed from scikitlearn. The psych learn borrowed from R. So it's not didn't invent some of the stuff but it made a really good user experience, really good docs. Um we're going to learn how to use that today uh and go into a bit with that. Uh for for the examples for this for the talk, we're going to use uh some data from from my days at Oklahoma. Uh so so when you're predicting severe weather uh we often look at soundings. So vertical vertical profiles of the atmosphere are usually taken from a weather balloon. You can also kind of get get forecast soundings by taking a column out of your uh atmospheric model. Um and and the severe weather forecasters like to calculate all kinds of statistics. Uh so things like cape and shear and uh all these other acronyms down below I won't bother to explain. Uh but but sometimes it takes a lot of interpretation to figure out what's going on with those. So so sometimes what another thing the forecasters have done is they created a database of of events where they knew there were significant supercells and were that produced like strong tornadoes uh as well as uh large hail. So like produced like 2 in or greater hail. So uh so this is kind of a conditional algorithm called the SARS or the sounding analog uh system. Um and you can actually grab all the data out of what's called sharp pie. So, so it's built off of a another package developed at SVC called Nar uh by uh a number of my friends who led the effort on this. Uh so, so grab the data out of sharp pie uh and we're going to actually do some machine learning on because it's it's got some interesting features, got some interesting linearities uh and but it's a relatively still a small data set so so easy to good for tutorials. Uh so we we get most of the soundings are coming from the Great Plains area. um especially sort sort of the high plains. So sort of looking at uh uh kind kind of western uh Nebraska, western Kansas, Amarillo, Oklahoma, Topeka and sort sort of peak there. Denver is actually surprisingly under represented in this data set. Uh but then we have we have some points from all over the country including even a couple from like eastern Washington and California. Uh the the distribution uh is a little wonky. So, so one would expect hail to hail size to have a uh like kind of kind of a gamma distribution exponential, but but it's if you look at it, it's really kind of peak in weird areas. And this is because when people report hail, they tend to do so historically at least not by actually measuring the hail size, but but comparing it with some other reference object. And some of those objects are things like mothballs, pennies, nickels, quarters, uh tennis balls, baseballs, apple/ teacup, uh softball, and grapefruit. Um some of these are a little bit more variable in size than than but but a value has been assigned to it. Uh so so if you so there's some inherent uncertainty in this data set and also some gaps because like like there isn't a good analog for two and a half inch hail uh uh in the data set. So we actually don't have any two and a half inch hail reports in this in this particular data set. Um so so knowing your data and plotting your data is a very important part of machine learning is the first thing you should do and and look at we look for weird things like this and try to understand why why is it weird is it noise or is is there some other system imposed that is seeing your your sizes. Uh but what we're going to do is 2 in is kind of a good splitting point on this data set. Uh and it's usually the point where where hail is going to wreck your day and your car and your windows uh if you get 2 in or greater hail. So we're going we're going to focus on that threshold with this data set and we're going to do apply some different machine learning algorithms. We're going to compare in this case a global machine learning algorithm versus local machine learning models. So a global model in this case linear regression or logistic regression which is linear but but you apply a a logic or logistic transform to make it range from 0 to one. U so you can use it for classification. uh basically you you're trying to max you're trying to fit equally well everywhere in the distribution for all your all your data points so you get a straight line uh in kind of this uh two in these in this 2D cape and shear space and and we're using cape and shear because this is a popular thing for severe storms stuff a lot of things are done in the space because that a lot of storms you kind of are trading off like low low high shear low cape and high cape low shear and occasionally you get both um But we and and it has like kind of a linear fit. But but is it but there's a question like is this act is linear the act right best model. And if you want to do things that are nonlinear you may like you could fit a like higher order linear regression like a like a a polomial type regression or you can use other models. And so some of the models we're going to talk about today are in the decision tree family which can localize fits uh over sub areas uh and and does so in a really fast and efficient way. Uh another local model you may have heard of is uh pay nearest neighbors. Uh this is also sometimes referred to in meteorology like analoges. Um there's there's other variants of with uncertainty like like Gaussian mix like Gaussian process models are are are kind of key nearest neighbors except you put a Gaussian around every single uh neighbor you're trying to merge together. Um and and Gaussian and K nearest neighbors can be very powerful but it has a couple major drawbacks. It's extremely local. you can set how much locality by how many by K. Um but but it also is not really a model per se. It's you're you're basically taking what like what like whatever points are in the vicinity of the point you care about and and seeing what they're doing. Uh and as a result you have to carry around all your neighbors uh anytime you want to do an evaluation. So it's not really summarizing anything. It doesn't scale very well. Uh if you've run any analog type models, you will notice they they tend to start slowing down a lot after maybe 100 or a thousand examples, there are some ways to optimize it. You actually using tree based neighbor lookup methods um that can that can allow them to scale a bit better, but you still run into issues. And this is where so this is where decision trees and their ensembles kind of act as a really good trade-off by kind of treating different parts of the space as different regimes, but doing so in a in a very efficient way. So what what is a decision tree? Um the one there's a lot of different variants of decision trees for we're going to focus on part trees which which follow a few different rules. First is they have decision nodes uh which basically in this case it's a very simplified one. You ask a yes or no question. Usually you have a variable and a threshold. you're wanting to know is is the variable greater than um like in this case is the wind difference height wind shear greater than 15, no or yes. If it's yes, you go down the yes branch. If it's no, you go down the no branch. Um and then you um you may have other decisions you're trying to make. Uh and eventually you get to what's called a leaf node. So so trees have branches and then they have leaves at the at the end of the branches. Um and and at the at the leaf for a classification problem, what you can do is is basically take every all the data that ends up at that particular leaf, you you you count is so it's large hail versus small hail and 76% of of the samples are large hail, then your probability there is 76%. Um uh so so you can you can derive out whatever output quantity you want that way. Uh so how do we actually grow these decision trees? Well, we we start with all the data and we we currently don't have any divisions of it. Uh if you were to treat this as a leaf node, this would basically be your climatological probability. So, however many examples in here are are large hail, I think it's about 50/50 in this case, uh versus small hail. Um so, so let's say 50% chance of hail, just that's what the model is going to predict. Um but if you want to refine it, what we do is we we we step through every possible question you can ask. And that could seem infinite. So we limit it to yes or no questions in that format of variable greater than threshold. Uh we pick the variables we just iterate through every single variable thresholds. We uh uh basically pick unique thresholds between different points. There's other there's also other ways to do this. You can sample you can uh make a histogram uh of the thresholds and pick different histogram bends. Um but but the point is we just want to find reasonable candidates and then we want to exhaustively evaluate them on our training data and see which one actually splits the the data into two subsets that improves your your accuracy or your whatever your your scoring function is the most. And so in this case we we find the difference of height or 15 as I showed before. Uh and and basically we draw what it looks like in the space we draw a line and on one side of the line we have a higher chance of hail. So instead of 50% everywhere, we have one side that's 68%, one side that's 19%. Um, and this and this gives us our our biggest loss. And then we repeat the process. We but we do so recursively, which means we we we go down like the large hail branch first or we go down the small hail branch first and then we split that up uh and we do that down each branch. And so we get to the next level and we and we now split the data even further. We found u in the space that that while most of this upper area tends to have large hail, there is a part that had that tends to have a lot of small hail. So so our our large hail probability went even higher up uh over here and much smaller over here. So that's an improvement in in the splitting and we see the same over here where where there's actually another regime that seems to pop out that has uh in in kind of the storm potential energy or the cape space. uh whereas you have high potential energy, you're more likely to have large hail even if the windshare isn't that high. Um so then when you get to the leak notes uh so I talked about how we do kind of classification uh you can make it also the deterministic prediction. You just look for the mode or the most common label based off of all the examples that reach that point. I'm going to close the door. Um but you can also use these for regression. So so if you have a continuous value problem, you want to make predictions on that. Uh if you want to make a deterministic prediction, you can you take the the values at at the at the leaf and you just take the average of them and and that is your uh uh that is your answer for that for that particular tree. You can also do probabilistic versions of this. So so you could take the mean and the standard deviation of that of the values in that leaf node. Uh you can also calculate quantiles if you use a quantile loss. Uh so there are quantile regression trees and quantile regression forests that that that that are a thing. Uh if you're if you're interested in other nonparametric things, you can also do multiple outputs. So, so if you have like two like like say two related classes that that you're trying to predict, you can actually have the tree basically take the mean on each of those um uh and handle multi output losses and it actually preserves the coariant structure conveniently because because you're basically all this is derived from the values of your training data at at that leaf node. It doesn't act none none of the actual input values directly are responsible for your calculation. So that that makes one of the benefits of that is if you have some noisy data that means your tree is can be somewhat uh robust to that noise and and and this is a and this turns into a very nice property of trees for a lot of applications especially real world ones I should say. Um so how do we actually calculate the losses at at um at each uh we generally use this formula called information gain is sort of the underlying idea behind it. So, so it looks kind of complicated, but it's basically saying what's the the score at at your current uh decision node? Um, how much that does splitting the data into two subsets and then calculating the scores on those subsets, how how much does that improve your basically increase your information? Uh, so so this all comes out of Shannon entropy theory. uh which is why entropy the not not the physical entropy but the information entropy uh is one of the ways to calculate this. Uh basically what you're trying to do is if you're so if you have a multi this is for classification. So if you have a a problem where you have like two or more classes that you're trying to predict like say winter precipitation type you have rain snow sleep freezing rain um there's four classes but the same the calculation applies you iterate through the classes and you look at the probabilities of each class and you and for what's called the Jenny index which uh it's generalized inequality index it's used for measuring how wealthy or how unequal society is but you can also use it for decision trees or entropy um is p * log base 2 of p because you range from zero to one. So, so log base 2 works works for this. Um, basically we we uh sorry, let me let me get back on track. Um, so so when P is closer to 0.5, you you your your impurity tends to go to 0.5. This is true for both of these. And then as you get further from 0.5, as things get sharper, you get you have a lower impurity. So so your class is purer. uh and thus and and that's why you have these negatives in there because you want to we we want to uh we want to we want to minimize the impurity uh basically as as we're growing the tree. Note that this is a a local optimization. This is only done on the examples that are in a particular node. This is not done across the entire data. So a tree is actually suboptimal in terms of a global minimization uh compared with like a neural network or a decision or or linear regression which does a global optimization. But this can be really handy for for like data that has lots of regimes and lots of lots of different subsets that are very different from each other. Um so so then what what kind of things can you can you vary with your decision tree? So one of those is is the depth of the tree. You can keep like like you can keep growing out the tree till until you have like one example in each node. If you're or basically your node is basically has all examples of the same class then you can't really split any further. Um and what happens is basically we we we are able to get sharper predictions. So we so so there is still room to be gained on this particular set of training data. Uh but but as you start drawing smaller and smaller regions the representativeness of the examples in that region becomes less and just pure to the problem of just a few samples and so so the the likely result is noisy. So usually deep trees tend to overfitit to data. Usually um there's a couple ways to account for this. One is stop growth early on. The other way is to um do what's called pruning. So you take a validate an independent validation set and you use that to say like does the validation error actually increase when I do the split if so prune it uh and make a shorter tree. Uh cart generally it tries to avoid the pruning part because it because it is slower. This was being done back in the days of of of of when compute was a lot more expensive and data sets are a lot smaller. Uh but there are methods that do pruning. You can you can do pruning. Usually in practice these days we don't work with individual decision trees for a lot of problems. Um and and this is for a few reasons. One decision trees are very brittle. So they're very sensitive to but splitting but you pick a splitting threshold and if you change your data even slightly a different splitting threshold will probably occur and your tree will be very different. Um so so so they're interpretable. You can you can read out the rules that going down each branch but those rules themselves are not may not be very robust. Um that said, you can take that fact that they are kind of brittle and highly variable and if you grow them out the like one of the benefits you get is basically if you grow it out all the way a tree becomes essentially a nearest neighbor's lookup table but but really fast because it's you're doing it in tree space. So, so if you know anything about data structures, um I basically any kind of tree data structure had has a complexity of O of login. Um in login if you have to look up like in examples but or yeah whatever. Um so so that means that that that as trees scale really nicely as a result. They're not as nice as like maps where or like hashmaps where you can go O of one, but they're better than than a list which is like O of N where you have to actually go like look through the entire list. And so so and and also the the other thing decision trees do that's really cool is is you can is automated feature selection because they exhaustively go through a bunch of features. They will only pick the best ones and they will ignore features that are useless. So if you have a lot of noisy features, trees are a great way to to kind of filter that out. But it's really Yes. Oh, [laughter] just just in ter in terms of like the tree depth and kind of the robustness of the rules. So those things are like the I don't know if it's the top of the tree, but like the first rule set is that usually going to be more robust than like the bottom levels of the tree. >> That's generally the case. Yes. >> Okay, cool. Thank you. >> Yeah. Yeah. So yeah. Yeah. Things at the top are going to apply to the whole most [clears throat] of the data set things as you get deeper in the tree not so much. And there's ways to actually take that and turn to a calculation that I'll discuss uh in later slides. Thanks. >> Uh I think this is a good stopping point before we go into ensembles. So happy to take some questions. >> Yeah, I just had a question about the thresholds. Are those all predefined or are they updated iteratively uh to match the training data for instance? >> So so the the procedure is is is designed to be nonparametric. It basically just looks looks at the distribution of of values and and and just finds split points. Basically if there's like I thought I had an example somewhere in here. Uh yeah. Okay. Yeah. So I I kind of skipped over this part. If the example So the inputs are are one, three, and four, you take the average between two points and and so the split candidates for this set are two and 3.5. Okay. >> Uh so so basically and if there's like multiple ones that have the same value, then you just just kind of stack those and and then uh you can still average between the two values and then that makes for robust split points. If you there's other ways to do it. So one way is you can just randomly sample values in that across from that within that distribution like uniform random sample. Uh you can there's there's there's other there's a number of other rules but but this is the one that's generally used for cart. Uh there's also a histogram way to do this where you you put plug your data into like turn your data into a histogram. So it's still non-parametric uh and that and by just looping through splits in the bins uh you but it's a lot faster. Um, so, >> so determining those split points is what the machine learning is actually doing. >> That that's a lot. Yeah, that's a lot of the work is is deter split point and determining the variable. >> Okay. >> So, you might have kind of just answered this, but you said that the earlier decisions are tend to be more robust. So, how do what determines that first decision? >> Uh, you basically you apply the same procedure everywhere. So this looping through every variable and and all the split thresholds and then you pick the one that on your training data maximizes your information gain. So so it does the best of you try a bunch of candidate splits and you pick the one that does the best on your training data. So so it's a greedy algorithm. It we're not accounting for like like in practice we know this is probably not the like globally optimal way to do this but but it's it's fast. It gives you good it generally gives you good results. Um and and it's something you just then repeat recursively through through the tree. In fact, the like if if you ever use R, the decision tree package for R is called arc part because it's recursive partitioning. So you just keep you basically keep splitting your data up and into smaller and smaller boxes. Um other questions anyone online? Sorry, I guess I have one more question. should look at this example because uh I guess is it always a binary tree if you have multiple uh kind of branches as long as there's only one? Yes. Like in this example? >> Yeah. >> Yeah, that's a good question. Yeah. >> Yeah. I would say for cart it's a always a binary tree. There are other decision trees that handle categorical data where each category is a different uh u split basically takes you down a different branch. uh what Clark found was that you can represent a lot of nonlinear relationships by just have by by still using binary trees but doing multiple splits on on different variables or the same variable different thresholds. So so from a computational perspective binary trees are the most efficient way to do this. They do come with some drawbacks uh and there are so there are a lot of other ways to do trees. There are other algorithms out there um kind of if you look through the the the history of these things but but car is pretty robust for a lot of problems and so and it's fast and pretty straightforward to code up. So so that's been kind become and it's open source. Uh there's another one called um uh like uh originally ID3 and then C4.5 and then C 5.0 by Ross Quinnland. Uh if anyone in here has used the cubist algorithm like R of sometimes uses that for projects that's out of that basically it takes makes a tree and then builds rule sets out of the tree. Uh that's its own algorithm. Um and there's other other flavors of tree stuff out there. Cool. Thanks. >> All right. Uh I think I'll move on and we'll we'll stop for more questions after we get through some of the ensemble stuff. So now we'll start uh move on to random force. So so the idea of a random force is that it it's designed to to basically take advantage of some of the weaknesses of single decision trees. So this is an ensemble method. Uh we have two forms of randomness. The first we take original training data and we resample the training data. We do so with replacement. Uh so this is like bootstrapping. Uh so this and or bootstrap aggregation or bagging. This is this is kind of the bagging piece of it. Uh the idea behind this is when you resample with replacement to the same size as your original training data. Some of your examples get duplicated and some get left out. So this kind of implicitly re-emphasizes different parts of the space uh and and make and makes each tree a bit weaker in terms of their like overall predictions but more but when combined together more robust. The second part of this is we um instead of so so originally we were going to like try exhaustive test every single variable and every single splitting threshold when growing our trees. We have a lot of variables that and a lot of thresholds that gets really expensive really fast. So why not just randomly pick a subset of the variables still exhaustively step over every possible threshold and that's what random forest does. Uh so we we u and we and we do the random subsampling of the variables. uh we repeat that process at every single decision node as we grow the as we grow each tree. So this means growing each tree is really fast. Each tree is independent. So you grow them in parallel if you have a parallel computer. You can you you can uh you can do that. They're all independent. And then you take your predictions for each tree and you just average them or you or if you or you can have them vote um for for regression and probabilities. It's basically an average of the of the of whatever leaf node probabilities you're getting from each tree that gets you a prediction. Uh, also one of the things that Brimman also did was he grew the the whole tree out to the end. So, it's essentially a nearest neighbor sort of thing and didn't do any pruning. Uh, pruning he thought was too expensive. Um, just keep the whole tree around. It also has the benefit of of of the error should be if you have like one unique answer or all unique answers at each node, uh, your each individual tree should be like wildly overfit to that data, but that means it has no bias. Uh, and then averaging reduces variance. And so we solve the bias variance trade-off with with with this ensemble. Uh and without having to do any tune any like very little tuning like all this is just standard decision tree very simple rules uh some stoasticity but it makes for it takes the weaknesses of the trees and makes them strengths and makes for a very robust model that doesn't need to be like you can you can take 100 100 nodes square root of the number of features that works as a good first start on a lot of problems. So, so I I consider a random forest to be like the second best model for for just about any tabular machine learning problem. Uh so, so what does this look like when you go from a decision tree to a random forest? Um this is not growing. This is actually a short forest, so not growing the full tree out. Um but but you you kind of get some some more gradients in your in between different samples. You also notice it's not as sharp particularly in the uh if you if you look at like like this this corner uh the probabilities are higher for hail less than 2 in. Uh they're about the same or a little bit lower for for this this quadrant. Um but just picking up on other regimes like this kind of this corner one kind of shows up uh because of the randomness you're able to to pick more samples in this region for some of the trees and thus um you'll get you you're guaranteed to get an improvement over a single decision tree. That's like a theoretical guarantee. um and is fast and um and and performant. Uh some hydroparameters. So so if you're using scikitlearn, what do you what do you actually care about? If you if you do want to tune, what should you tune? Um in estimators is just how many trees are in your forest. Um basically what happen it like you'll if you keep adding trees, you will get you'll you'll continue to get further improvements but asmtotes because of the central limit theorem. uh you also will sometimes get basically you have to do more computation the more trees you add both during training and inference and then the makes the each tree can be kind of large stored me in memory and in disk if it's a really large data set so that can that can turn into a problem so you want to find a balance uh but 100 is usually a good starting point uh max features so how many features do you sample uh this can range from like square root is of the number of features is generally the default option and that works pretty well it doesn't work as well if you have a lot of noisy features. So, so you may want to bump that up to to a to a different value or use some kind of tuning algorithm to pick a reasonable value. Um, and then controlling your tree dep um most of like the the original approach is to just let it grow out. Uh, but if you're using really really large data sets, the trees can sometimes become like unmanageable. Like I've accidentally saved out like multi- gigabyte size decision trees doing this. So, uh, the parameter I would recommend most to try is what's called max leaf nodes. So this will will preferentially grow out branches where basically you you have a lot of examples and your performance is really going look like really improving. So grow those out first uh and then then find the next best to grow that out until you hit that max leaf node number. If you pick two that will then correspond to how deep the tree gets. You can also just cut the tree off at a certain depth or you can say uh grow until you hit a minimum number of samples uh which which allows you to basically grow branches where more of the examples go down deeper and the ones that that don't get as many examples uh shorter. So that's random forest. Any any questions about about random forest? In what case would you prefer um a random tree as opposed to a random forest? Looks like a random tree is a subset of a random forest, right? >> Yeah. Well, so a random tree is going to be even less accurate than a regular decision tree because it's uh you're not exhaustively searching over all your data. Uh there there is a so so the en the randomization and the ensembling are combined together what make it really performant. But there are other variants. So there's another variant called extremely randomized trees where in addition to picking a random subset of your variables, you also pick a random subset of your splits to evaluate. So it's even faster. you get even it it gives you a [clears throat] bit more variance but you get the trees are even weaker at that at that point. Um it's inside learn as well. You can try it out uh see see if it see if it does anything for you. They're going to be a like it's not too much of a gain in performance as to why most people just use random forests for for a lot of stuff. >> I shouldn't come in all these diagrams the trees grow downward instead of upward. >> Um [laughter] I think it's a it's an artifact of the computer science and like tree trees in like you tend to grow down or most people read from like you start you usually read from top to bottom in the in the in the western world. Um >> these are roots not branches. >> Yeah. >> Yeah. You can think of it like that uh there's like that as seen on TV uh tomato vine thing where you like you hang it on your deck and the the tomatoes grow out of the top and down. >> There you go. >> So So you can think of it like that. that uh but yeah it the the mathematically the trees are upside down compared to real life. Uh but yeah, you can flip this all up the other way and it it still works too, but convention is usually top to bottom. All right, on to gradient boosting. So, so random forest h has a lot of things going for it, but it kind of hits a hard ceiling in terms of how much you can optimize performance. uh the trees get really really big. And so so there's other ways to do ensembling with decision trees. And one of those is is in the family are called additive models. And the idea behind an additive model is you have some kind of simple model that makes an initial prediction. And that prediction is usually pretty bad. But then you can if you take the errors on that prediction and make fit another model that predicts the errors of of the first model. You can add those the residuals, you add those those predictions onto the first one, you get a more accurate model. And then you keep doing this. So you make another model that adds on to the previous predictions and so on and you and you can do this endlessly and you can do this with like splines with uh polomials with you pick a whatever basis model you want. Trees happen to be a really nice basis model. Uh so gradient boosting while the method is generic uh trees are are generally the preferred method because you can you basically the idea is you grow small trees um and you take the original tree grow small tree get some prediction just like you would before um and then you predict on the uh basically the residuals of the pre tree prediction sum those up to get the next the total prediction and you keep repeating that process and so on. Um and and this is called this algorithm is called boosting. Uh basically adding another ensemble that predicts the errors. Uh you may wonder like can you just add infinite items sort of eventually your errors get small enough that you're uh that you're overfitting to the noise. Uh and so so gradient boosting one of the things it adds is is an idea of a u what's called the learning rate. Uh so this is this is different from the machine learning learning rate related. It's a constant you use to to shrink the the contribution of each tree. You think of each tree as a step. Um similar to like gradient descent. Each time you're you're optimizing your weights, you're taking a step toward down the gradient toward toward an optimal result. Um in this case, you pick a constant value that that that just scales the the prediction from each tree downward. Um and that that helps the model keeps the model from overfitting or helps. Uh another thing you can do is add stoasticity. So, so you can there's a subsample parameter where you instead of doing like the resampling with replacement, you just subsamp randomly subsample your data and then you do a different random subsample for each tree and as a result you're fitting to a slightly different data set each time and that while makes the model stoastic, it's not you can't it's hard to reproduce unless you have the random seed. um you you get the benefit of of making a model more robust to to to noise in the data and then in estimators just like rainforest it is uh basically how many trees do you fit uh and this you have to play with the learning rate and in estimators are kind they kind of work in tandem um so if you have a higher learning rate you can get use fewer estimators to get to a certain point but you may miss some of those global minima so so it can be beneficial to use a smaller learning rate and more trees or to find the balance between those do um and and here's an example from the scikitlearn documentation that kind of shows some examples of this. So so if you don't do any any regularization um basically no learning rate no subsampling after you get good performance up to about 150 trees but then your performance your test set performance starts getting worse again. Um if you just subsample it kind of doesn't really buy much. It doesn't get worse but doesn't really get better. But if you combine the two, learning radian subsampling and limiting your like and even feature subsampling, you can get much more robust test set performance going out even to like 400 or more trees. Uh the caveat with with these is because it's an additive model and you need to train the first model before you train the next model. Uh so so it's not par it doesn't parallelize in the in the respect of of the of the ensembling. you can parallelize the the uh variable and and threshold evaluation and a lot of the packages that the more advanced packages that do gradient boosting that's where a lot of the parallelization comes in. Uh and also usually gradient boosting you tend to do better shorter trees um and then you're just adding bunch of small trees together and that also helps keep it robust. um how um well again it's another tunable thing uh you can play around with um and because gradient boosting basically ends up being of of all the like tabular machine learning methods gradient boosting across many different data sets is generally the best one uh if you tune it right you you will get uh the best perform better than like neural networks better than random forest better than SVMs uh there's there was actually a study done in like 2007 where they had a bunch of like benchmark data sets and found that gradient boosting and it uh generally did the best. Kaggle also proved this out uh at least in the space of tabular problems and time series type and sometime series type problems. gradient boosting uh is generally the best because it's very it has the three properties of being very robust to noise and regularization and how you grow the ensemble to you optimize but then you're also still robust to to noisy or or high dimension really high dimensional large data sets and also because the trees are small you're not getting into the the the overfitting like noisy branches and it also makes it faster to evaluate if you have like a thousand trees you don't want to go down like 10 levels every Uh so so you can still use scikitlearn. It has um a great a very robust gradient boosting implementation. It's not super fast but it but it works well. A good place to get started. There's also supports something called histogram gradient boosting which which simplifies the split thresholding sort search quite a bit. Uh and is borrowed from another package that we'll talk about in a second. Um but if you're a power user, you really you got some large data set you want to try this on, I would recommend going to either XG boost or light GBM. These are the two big gradient boosting packages. XG boost basically the winner of many competitions does has some parallel optimizations and uh like C++ optimization and other stuff. It also supports random forest various losses. Um has other optimizations for training and inference speed. It it can run distributed training. Uh you can also do GP run on GPU if you want to. So so it's it's very powerful. Um but but I wouldn't start there if you're just playing around with it. Starts scikitlearn like GBM is another similar one. They use the histogram rating boosting. It also has a bunch of scale optimizations. So this is stuff used by like people on Wall Street or you know hedge funds and banking like like those are power users of these kinds of models because stock market data is extremely noisy. Um uh so so that's um uh gradient boosting. Um it it's still it's still heavily used. Uh doesn't get as much attention, but it it's it's still like like if it's not if it's not something where you want to throw a CNN or a transformer or something like that at it, gradient boosting is a good place to start. uh questions. What are people I show hands I guess who has used any of these decision tree or random forest or gradient boosting methods? Has anyone used any of them in their work before uh in the room? >> Good. So you got a lot of people try I heard a a buzz online. Is there Well, >> someone said that. I did. >> Okay, cool. Totally >> highly recommend trying them out. They are really like like the advantage of scikitlearn is also really easy to install into like if you deal with xg like gpm because of the GPU parts of them they're they can be a little bit more painful to install but scikitlearn is meant to be really really easy to install uh and plays nicely with numpy map lib and and and it's optimized with sython it basically is python that compiles c uh so you just have to install syon it's like the main dependency you may not have otherwise it it's it's really robust so some applications of of uh decision tree AI products. So we have random forest we so so one project we've done in row is a uh fuel moisture content estimation. So we take satellite data so there's actually a grided high-res grided problem which you think oh this would be a perfect problem for for a um like like a CNN or a unit. No actually it works better with a random forest or a gradient. I think we end up using we've done both random forest and gradient boosting on this. Um but the idea is we have sparse data of fuel moisture content. Basically you take some leaves or some sticks and you and you cook them in an oven and you measure your mass before and after and basically cook all the water out. And that tell um and and and then like the thing the less fuel moisture content the more the faster something will burn is is kind of the the underlying equation of this. And it varies quite a bit spatially, but our observations are extremely sparse and and uh if you're trying to predict fire spread, this is a very critical thing to getting good fire spread predictions. So, we taken like veers and goes and some other channels which you can kind of get sense of how green your your the land is uh out of those different resolutions and time time ranges and combine them all together. So a lot of noise variables um and and then fed that into the random forest and we get these really pretty maps and we actually see like di cycle variations especially in like the live fuel moisture uh like you see rain coming through and suddenly things green up it it can pick up on that and adjust. We've even plugged it into a fire model and found that the fire spread is much more accurate when you use u the heterogeneous fuel moisture content from from this kind of algorithm instead of a homogeneous uh assuming the same fuel moisture content everywhere. Uh another approach that was also sort of a kitchen sink model is called NATO. Uh this this was actually believe Brian Himple who's like I think he's tried as kind of a computer scientist who's really Mel didn't really know anything about meteorology but he found that let's just throw all the ensemble information into into a big gradient boosting model and see what it picks and it turns out actually made makes pretty high skill predictions of like this is hail but also it originated tornadoes since NATO cast uh but but he's expanded to wind and even tried doing tornado deaths which is a little Mac but u the data does exist for that if if you want to fit a model um but yeah you can basically get it gives good predictions it's all online if you want to take a look at it um but again it's one of those we we can take a bunch of variables and filter them out we can run spatially run over lots of data and it's it's a pretty robust algorithm uh there's also random force based decision tree or random force based uh guidance based off of like global on forecast system output developed at Colorado State uh that that the SBC uses operationally now for for issuing their their like day four to eight outlooks uh and and and basically even before it was officially operational some of the forecasters are using that to decide if they're going to draw like a 15% contour or not and and it's able to get a couple days lead time. So it's a huge value ad in in the space where where the predictions themselves are kind of noisy but but there is still some scale to be extracted if you have if you have the right like guidance that can combine that can look at different regimes and combine all this different information. So so then I really talked about decision trees I think summarize some of their benefits generally they're very accurate for tabular data problems with fairly minimal tuning uh gradient boosting being the slight exception to that. uh the robust to noisy and corrupted data. You can some of them can explicitly handle missing values. You don't have to like imputee something, you can just like say like it's a nan and then instead of breaking the whole model, it'll be like oh it's the nan branch. We'll just go down the nan branch or or we'll have some way to deal with that. Um it does feature selection as part of the algorithm. Uh so if you don't know what's important in your data, you don't have to like guess. You can you can just let it run through the tree and the tree will figure it out. uh the output calculation is only a function of the training data which uh is good in terms of it won't give you really weird results if you it's le like the garbage is less smelly compared with a neural network or or or SVM uh garbage in garbage out kind of thing uh you can do the training in parallel and there's different ways to parallelize it that have been exploited now and people scale these to massive data sets that are used by lots and lots of users and downloaded like millions of times a day kind like like like this is these are these are real world models that are used on a lot of real applications they don't get as much process neural networks but but they are kind of the like like most people that say they're doing MLE linear regression or or or gradient boosting uh you can also get some uncertainty quantification out I didn't talk about this a lot but you can not only you can get probabilities there's ways to actually estimate alitor and epistemic uncertainty out of the models by looking at the variances between the variance between individual trees um and there's I will talk about a little bit more. There's some explained AI methods that are both cater directly to decision trees or can be applied quickly on decision trees that can get give you some information about the problem. The drawbacks um I say high floor relatively fixed ceiling unlike neural networks where you can get you can do a really bad job but you can also keep growing them and scaling them out to the point where where they're like chat GPT um for better for worse. U tuning only gets you so far. uh the predictions also they're not decision trees are not differentiable they're not smooth the you get jaggedy bumps because you because each like subset is very independent so if you need something that's that approves of smooth predictions uh decision trees aren't it um this can be something you have to know about your problem uh but but this is a trade-off uh if you want to also you can't do back propagation through they're not differentiable although I think there are some soft decision trees that you can back prop through so so there are variants that that that do some things to do that every time. If you have new data, you have to retrain the whole thing from scratch pretty fast, but you still have to you can't do transfer learning or online learning with the decision tree. It's it's you have to rebuild the whole thing every time. Um, you do need to do additional pre-processing like a lot of traditional ML methods to account for spatial temporal features. you can plug in individual grid cells and I'll show the example of that in the exercise but the meaningfulness of it uh and if there are spatial correlations you have to like add extra features that handle that whereas neural networks you can impose that in the architecture uh and it can handle those sort of things m much much more much much more smoothly. Um they freance models can also because they essentially become a a nearest neighbor lookup database. They can have large memory storage footprint compared with neural nets which are essentially regression models. So you can make giant neural nets with billions of parameters that are also hard to store. Um and you like in terms of their extrapolation behavior they don't extrapolate outside of their training data values because it's just the nearest neighbor model and anything anytime you go outside your nearest neighbor will basically just extrapolate the same value from the nearest neighbor. Random forest al will will then further compress things because you're averaging examples. So, so random forest t if you are looking to actually have a model that predicts extremes, brain and forest are not the one you want to use because they they they tend to dip dip those out. So, as part of the trade-off being robust to noise is they have low variance. Uh so, um with that do another round of questions. everyone feel like they understand trees. Um, does anyone have cons like like it's from a sense of uh excitement about trees or are we appear trees or we kind of like ah this kind seems kind of interesting. I can I can vibe with this to like no neural networks or physical models or uh uh I'm done with this and I want [clears throat] to go get some coffee. >> [laughter] >> I think it's one of those things you have to learn by doing, right? So yeah, you can get a high level conceptual idea, but until you apply it, you don't really know how it will work in your own research like workflow and analysis. So it's like, yeah, I can see how this works, but I'm not sure exactly how translates to what I'm doing right now. >> That's right. >> Can I ask? Yeah, >> I was just curious if people are kind of surprised about the performance of decision trees given how simple they are compared to neural networks or I feel like when people talk about machine learning they're like neural networks are like the best anyone surprised I guess I personally I was when I decided trees I guess I feel like I've kind of c like caught on to that in the discussions I've been in randomly throughout the last couple years is that the random forest has been really useful for one thing that comes to mind for from a like observation data point of view is we often have to deal with things like detection limits or like maybe data that doesn't fully capture the system. Do you have any thoughts on how you might go into these decision- making like decision trees or random forests and like find a way to address that, you know, like or just like how how do I even think about how to do that? >> Yeah. So, that'll be the next section. We'll start getting into a little bit of that. Uh some ways to to to start to interpret these a little. You're trying to see like how well is it learning? I mean I mean some of this is like like the big insight from a lot of machine learning stuff is having a training set and an independent test set that you that you haven't trained on or tuned on and if your model still performs well on that independent test set that's maybe a sign that you're doing well or in cases where you're not um you like having some of the uncertainty predictions can also give you a sense of like in this example we're kind of near like the model is struggling to get to converge or struggling to make a coherent prediction or it's predicting climatology ology or or the uncertainty bands are really high. That's probably a sign that you're near your signal your noise for um uh so I don't know that's answering your question. >> I don't honestly know either but I just think I'm just trying to think about it from like again the types of data to feed into these >> yeah the I say another benefit is you can you don't have to like normalize or make any assumptions about your data. fairly as long like like the if it's continuously valid thing uh even if it's weirdly distributed because you're just looking for split thresholds uh the model like the random forests are pretty like trees are are are robust to a lot of weird distributions and uh noisy questionable things uh and some of the tools to regularize them also help help filter for some of the the noisy kind of stuff um or or other maybe more questionable patterns in the data. So, so there's um so so that that can be a benefit. One of the one of the weaknesses of neural networks is that they can like their failure modes can be it they can surprise you like there's some things you can do to them that they won't that are fairly robust changes but sometimes slight shifts in distributions of values will will throw them off into some weird land that's that's very different. And that's where you see like hallucinations and uh especially if you're doing anything auto reggressive that's that's where things get weird. >> Yes. If you go back to slide 14. >> Yeah. Maybe that's exactly what you're talking about here is like in the right hand side there there's a lot of outlines. Right. >> Sorry on the right side. >> Oh yeah. >> You can see there's a lot of outlines there. And maybe that means the model's not working well for those particular predictions or how would you try to address that I guess with incision trade or random variable? >> I mean some cases growing the model more helps. I mean some cases you just have to get more data to fill in like if you're near the edge of your distribution you you would need to sample more in the space right arguably [clears throat] >> is basically how you would deal with it. If you if you have like a regression neural network problem then how you how you what what basis you're using what what kind of like assumptions you're making like like is it a linear or quadratic or a cubic or like like what order polomial what kind if you're are using yeah sort sort of gaussian what kind if you're doing like a gaussian process what kind of distribution you're doing >> all those can well in kind of how it extrapolates around sparse data that's where you have to bring your knowledge of the problem into play is what's appropriate way or what how do I want it to do [clears throat] is a constant extrapolation the right thing or do I need to do or do I do I want to linearly go out do I do I want do I want to ramp down to zero or to the mean >> um like >> I guess in this case like you're talking about environmental conditions but it might be that the history of the storm is really important like MCS like >> actually got spawned somewhere else and it transported into these environmental conditions >> and this might be a good way of like identifying that I guess that maybe the environmental conditions aren't the sole thing you want to be predicting thoroughly. >> Yeah. >> So speaking of decisions, I think you a little bit especially like the first diagram. So this is now building you know the body of different machine learnings. Um and so maybe this is a better discussion for next week. But >> how do you approach the sort of qualitative versus quantitative process of deciding which type of model works better? So yeah, this photo tree this beyond just this one looks better. How do you sort of decide between which is appropriate for your problem? >> Yeah, that's um some of this is is um I would say knowing the kind of data you're trying to bring into your problem. So, so if it's tabular data that that uh like if it's not tabular, if you're trying to do stuff on images or or or or time series or text or audio like anything that's kind of signaly and and things that you expect to to have some continuous value electuous values and have some variances and have like multiple kinds of pieces you want to plug together, that's where neural networks of various sorts are are the way to go these days. I wouldn't probably bother with decision trees on those. But you have like time data or time series something that you know has a like you may have few examples but a lot of features. Um then then uh basically where you need to do some reduction of features that's where decision tree ensembles really shine. Um if it's something that that you you don't you don't have a lot of compute budget to train on and you just want you want something that or you don't want to have to spend a lot of time tuning. And that's also where decision tree ensembles kind of are are a good fit. But like like [clears throat] if it's a continuous value problem with relatively few dimensions, sometimes you like a linear regression or logistic regression is also a good thing to try too. Uh usually like kind of like linear logistic regression or other generalized linear models. Uh great for I mean some work for large data problems but uh but small data relatively straightforward like continuous um monotonic relationships, regression, linear regression is great. If you have like kind of less monotonic relationships, lots of noise, tabular data, decision tree ensembles, that space is good. You have structured data where you where it's grided and you care about things like there's connections between the grid cells are not independent. Um and and you you you really you want to preserve properties or embed assumptions about that structure. That's where like deep learning type models are I think are the are the way to go and then tune within that space of models. Uh there's also some there's a test [clears throat] that you can also do that uh Cynthia Ruden has a paper on it but basically was calls finding the rashimon elbow. So you start with a really simple model like a really short tree or or or like a polomial with only like you know like a like a linear model and then you add like a square or whatever and you grow your complexity. you track like basically make a plot of complexity um uh like train basically the y- axis is uh uh like your test set or validation error and your x-axis is complexity and basically if you um see that you're as you increase complexity if your error keeps going down then you should keep keep making it more complex but then as soon as your your error stops going down uh and and you have flat air but increasing complexity that's you should go back to wherever that where the elbow is the um and and that's and use the simplest model at kind of in the elbow space. So that's called the the rashimon set is the idea that that there are multiple machine learning models or statistical models that all have have like have different weights but give you the same error. Um and and so of that set there should be more complex models and simpler models. >> Thank you. >> All right, let's let's move on. Yeah, >> we want to have time for the exercise. >> Good good discussion questions. Um, so last part of this I want to talk a little bit about interpretation. So we already talked a bit about interpreting trees, but how do you actually interpret a tree? Uh, basically you can read rules often by going down particular branches and seeing what what the result is and you can see like what is the branch splitting on and like by shear if it's a lot of shear versus a lot of shear. You have different regimes. And then then we have a then basically the size cape is the next thing we want to split on both. And we have the low shear high cape uh get get you hail but also the high cape um this is like high cape medium high shear and then you have higher higher cape and lower cape and then that gives you different probabilities at the end. Um so fairly interpretable you can read it off. You can actually turn these into rules, the tagular form, but trees can get really really deep and really really big. And if you're doing ensembles of trees, it gets even nastier. Uh so there's a couple other methods you can use to to get some approximations of what what there are the most important variables in your data set. The first is called impurity importance. Uh if you if you play with scikitlearn and there's a if you look at your do dur on your model, you get model.feature importances underscore. This is impurity importance. So this is calculated directly from the tree. basically iterate through every node in your tree ensemble and you see okay this variable appears and this it does this well and this many variables go down it you you add that all up into its performance score then you repeat that process you step through the whole tree and do that so it's just a function of the ensemble um so it it's really fast calculate uh the problem is it's it's representation of the overall performance but not of the different regimes it's also sensitive to uh like correlating variables, it can give a it can sometimes give a like if the underlying feature is split across multiple variables in your data set, uh impurity importance will will also split the importance across those variables. And so so you if you're just trying to look at the ranking that can throw it off. Um permutation importance is another method. Uh this is model agnostic. The idea behind it is you shuffle your your data uh across the the example or sample dimension uh and then run it back through the model. really see how much your error changes on your predictions. Uh you can do this with any method. Uh it's model agnostic which is really cool. It run it runs well on decision trees and and ensembles. Um you can even run on physics based models if you really want to. Um it it can also account for correlated if you have correlated features. You can you can shuffle two features that are correlated at the same time or you can uh like uh just take one out if you know they're highly correlated. Um in terms of scoring you generally get fairly similar results but you get correlated field but some some of these are sometimes it will it will shift the order around. Um uh otherwise it's more about relative ranking that say the exact don't read too much into the exact values. Um another cool thing you can do with feature importance is because it's apply it's a it's a function of the data it's also stocastic so you can repeat it multiple times to get a more robust result. Uh but you can also apply it on different subsets of your data. So in this case I conditioned on elevation and we actually get a different ranking for the low elevation uh cases versus the high elevation cases. So this motion stable mixing ratio which tends to be lower at higher elevation um rises up from like near the middle to to near the top. Um otherwise the rankings are pretty similar. So so it's a good robustness test that can be a way to dig into your data compare with you can run the same the permutation forms on multiple models and see if they get similar results. You can run this on spatial fields. You can shuffle out a like like if you have a a temperature field in your AI emulator, you can take that you can shuffle it from a different example, run it through the model and see how the the prediction changes. Um other discussions, so model analysis methods are great because they they are an actual causal pertabbation on the model because you're you are changing the the data that it's getting. Um you can also they it's also very flexible, easy to code up. It's gonna be a little slow because you do have to do the shuffling piece. Um but but it also accounts for nonlinear relationships. Um and uh you can run non ML modeling functions. Drawback is is slow. It's stochastic. Um low mean importance doesn't always mean the variable isn't important. U so so so sometimes that can be a little it means it's less important, but it doesn't mean that there aren't cases where where it isn't useful. Um and and the problem with just while I read all the explanable AI for the most part is that it's an it's a summary it's an it's an attempted model of your model. So so it's you lose information. It's also hard to validate what is a true attribution and and sometimes you can also confound things by like oh I already selected important variables. So the fact that they're all important probably means that you because you applied your knowledge to it. Uh but you can throw like in a noise variable if if you uh if you're concerned about that. Uh some final takeaways I say again the random force are the second best model for most tabular problems. Uh but if you uh if you want to like really get some performance improvements and gradient boosting is is generally the way to go for for a lot of things um and unless you want to do deep learning and that's we already had lots of talks on that. So that that's the end of the lecture portion. Um happy to take a few more questions and then we'll jump into the exercise. Um thanks. So well well there's there's one question online from um was can you say more about defining the features of the data yourself for random forest versus the algorithm figuring out what features are important? >> Um so so the what the variable importance is doing is trying to it's it's trying like it still requires the random force whatever ML model you're using to figure out what variables are important uh or to to like actually find the variables. There's other methods of feature selection that you can do a priority based like based off of one your knowledge of the problem. Also you can look at correlations between your inputs and the output. Um you can look at correlations among individual variables. They all they all are imperfect di analyses of the information but you can combine them together to get I think a more if if things line up across multiple like scoring methods or correlations or in your physical knowledge then that probably means that variable is actually important. if if you're getting a value that like like uh like shark attacks and ice cream ch sales, things that are correlated but not um just because but but there's another confounding thing that it's summer and people are at the beach. Uh like like you may need to add another think about the causal structure of your data. There's a whole suite of causal methods that are out there if you not beyond the scope of this course, but there are ways to to to start getting into that if if that's something you're interested in. Right. Thanks. Other any other questions? >> Let's move to the size. >> All right. Yeah, I I I will um I think did Daniel have a he said he had a slide of how to >> Yes, it's in our chat if you wanted to pull it up otherwise. >> Okay. Yeah, let me get in there. there will be a little bit of sped up time on this because of getting this done late last night and uh being a problem right now, but I do have it locally so I can I want to walk through at least uh like the over the broad overview of the notebooks and that and that will help you all debug uh like getting everything running um just for the sake of time. Um let me switch to to this tab. So if you want to play with I say if you want to play with the hail data uh that I have a I have a separate notebook that's hail decision trees um both in terms of decision trees on hail and that they are truly the king of machine learning methods u the slides to your slide devotes Okay. Yeah. So, so um accessing boot camp preservation um today it's R430808 uh which which um uh should show up on your u Jupyter hub when you log in probably only from 10 to 12 10 p.m. Okay. But all this code is all CPU based. You don't you couldn't you can run on Casper, but you also like you can run it locally. Um, and then you can make an interactive job and do all do all the things here. You want to do a Jupyter hub? Here's the instructions. Those this shouldn't have changed any from from the last lectures. Uh so so we I added the new material uh last night or just got merged this morning. Um so so if you have um saved work that you want uh if you've been messing around with the extras don't want to lose your work um you can basically use this command called get stash um get stash push uh message saving for prior prior work get pull origin main that will pull the latest code from the repo and then get stash pop um that will that will then repopulate um your your your changes to the previous lectures. as well allowing you to add the other the new stuff and without having to commit it to the to the actual repo. You also commit it and then just do a get pool, but but you may not want to commit. So, and if if you don't care about your prior work, uh you can get fetch origin get get reset- hard origin main and that will reset it to to the main branch. Um so either way, um I would probably do the second one. So um anyway jumping into the notebook. So so the first one this is just optional hail decision trees. Um kind of you can remake the plots I have in the presentation and some other ones and you can play with play variable importance and some of these and look at some of the other data and features. I also have some evaluation stuff I didn't really go into in the broader talk but but slide >> oh sorry if you're into an evaluation there's also some some information on rock curves and reliability diagrams in here uh that you're welcome to play around with um and and um you can see all the depths of of of training the model and whatnot. So um but for our we want to go to uh Kristen's inso prediction data set. Uh the there is a slight update to the environment going on. Um I still need to actually populate this into my cond environment. Um so I I will do that after I go over the the the notebooks. But if you want to install from scratch, um once you have updated your your repo and you're in and you're in the lecture 5 directory, you can actually just you theoretically should be able to run this from your notebook. Um if it I think it will work uh condensate-fironment.l. So if you're not familiar with environment files, basically it's a YAML file that has a list of all your dependencies. So you can use this to like make new environments or preserve existing environments. Um so so that so that um because there are some newer versions of that change some of the things that are having issues with with with Kirstson's original environment. So so uh we have some we have an updated one. It should be it should be installed pretty fast. Uh and if it doesn't I'm I'm going to make it make my own and then you can link there's a way to link to that. Um so the data we're using the ERST data from lecture one but with a bit of a twist. So instead of doing a classification problem, we're going to do this as a regression problem. So you saw lots of classification lecture regression. We want to uh in this case we're going to take the ST field and then use that to predict the the and so the the NO 3.4 index u at t plus l where l is the lag in months. Um so we we we load in the data. Um I'm assuming it's in the utils directory. So if it's not there, uh Kirst may need to um give you the full path, but you can modify the path to wherever it actually is uh on on on Casper if you need to. Um and then everything loads in. If it loads correctly, you should get this plot showing showing the new 3.4 index over time. Uh we then extract. We we can set our lag right here. Uh so this is the thing to to to mess with. We're not filtering out. We're we're going to do in neutral event since this is a regression problem. Um not just n elino leninas. So leave in magnitude at zero. Uh and then I have a wet mask. So so this basically filters out all the land grid cells so that we don't have any nanss that we're going to pass through the model. Makes the data a bit smaller. And then we split our data into training validation test sets. And you can run this cell. You'll see this is an example where we have far fewer examples than we have features. So this is a good decision tree problem. Um model training. So we have a a box here where we will actually train all of our initial set of models. I also have links to if you want to dig in spend some time digging into the user guide and API for for for for different models and like for gradient boosting centries and random forest and psychit learn this good time to do that. Um and then here's a plot of the model predictions. They actually all do pretty well. Uh and you'll notice the the random force and gradient boosting are are are much smoother. The decision trees are very janky. Uh but also they they're under they're underdoing this particular Nino uh the 20 the 2017 El Nino event I think because it's not like bigger than anything in the training data. So that's I think that's what's going on there. Um but but but helpful to know for compilation purposes and then um so so we have some exercises. I have a little like running person because we're in Boulder uh for each for each exercise. Uh the first one we're going to plot uh so once you train your trees we're going to do some feature important scoring. This is the impur port. We calculate that for each of the models and we make a map. You'll notice that the like like the hot spots are all in the the El Nino region. So that's good. But then also where there's not really any structure seems kind of random. Uh and as a result, uh you can play around with the settings to see how much you can change that with your different models. Um and then another exercise we have is alternate data pre-processing. So one way to deal with spatial data traditionally is with is do you fit some principal components or eofs to it and and so first principal component is looks looks a lot like your INSO field or INSO variations um and you can plot the other ones if you want as part of the exercise and then then we'll train model we'll actually run PCA on on the data we'll train models on on on principal components um we actually get some slightly different results um all leave it up to you to to actually evaluate which one is better. And then you can also look at which princ like in terms of the feature importance, which principal components are the most important for each model. The first one is the most important for everyone, which is somewhat reassuring to see, but but there's some surprises further up. Uh and then as a bonus exercise, another [clears throat] XA method called partial dependence plots that you can that that you can also look at. And basically what this allows you to do is for an important variable, you can see its sensitivity. And part of the goal of this is to look like see just um how like if I plot different PC like like these different partial dependence plots how how much does uh like what is different between the more important variables and the less important variables and kind of think about that you you'll see some differences. Um you can also apply partial dependence in 2D which is kind of cool. Uh so so here's an example of that at the bottom and feel free to plug in different different numbers and uh basically change the number that gives you a different PC um so so that uh that is uh like so so at this point it's like kind of have fun. I'll get the con environment uh raised and if you run into issues uh raise your hand and we'll we'll come see what we can do to address them. uh code should all work, but if it doesn't, let me know. The the one issue is you may like data path uh Kirstson has helpfully put put it up on here if if you uh because it's not in the GitHub. So, you need to you can copy it into your copy of the utils directory if you haven't done it already where you have to pull from hers and just update the path part to that. >> Um the meantime, I'm going to get my environment built and see and see if I can get to >> Yeah. Yeah, I'll put I'll put those up for Sorry. >> They're also in his slide deck if you go to the lane page. >> Okay. >> I'll I'll leave that for here. >> Yeah, that works. It's right there. actually technically