Lecture 4: Pre-Trained Machine Learning Models | 2025 EdEC Machine Learning Bootcamp

All right, we're going to go get started. Good morning everybody. Welcome to our fourth lecture for our machine learning boot camp. And this morning uh we have Katie Dean who is a scientist in CGD who is going to uh walk us through uh her experience uh on pre-trained model. So um thank you Katie for being here and yours. >> Okay, great. Thanks Gunter. Yeah, thanks everyone for being here. This is going to be a little different than the format of the previous lectures. Uh there's not a notebook for you guys to go through, so it's going to be more of a showand tell. I do have some code sprinkled in, but it's going to be more about kind of my experience working with pre-trained models and this concept of using computer vision or image detection and applying that to earth science data. And I'm going to lecture for the first hour and then Daniel Howard's going to lecture for the second hour um on a different topic on AI and WP models. Um so, you know, feel free to ask questions throughout. I think I have plenty of material. So, we'll just, you know, see how far we get. Um, and the slides are in the landing page if you have the link to that. Um, and I guess for folks who can't see the screen super well, you can join the Google Meet and see the slides that way if that's helpful. All right, so let's get started. So, what how did I get here? Why am I here in front of you talking about machine learning pre-trained models? Um I thought I would say a little bit about my journey to working on machine learning research which I kind of you know fell into it during my um PhD posttock transition. So I was thinking about this problem of parametric uncertainty in a land model, but really sort of parametric uncertainty in in numerical models in general is very complicated. And um we were thinking about how we could explore the parameter space without having to run thousands of numerical model simulations. And this is a diagram that I tried to make as a grad student connecting the different parameters in the community land model to uh how they influence carbon and water cycling. And so it's very complicated and I couldn't you know figure it out manually. So when I came to ENCAR as a posttock we were thinking about how we could use machine learning in this space and there had been some work emerging on using machine learning to train emulators to uh map the responses of different parameter inputs to outputs and numerical models. But that meant that I had to learn some sort of neural network tool which also meant I had to learn Python. So I didn't use Python or any machine learning packages until I came to ENCAR as a posttock. Um but we settled on caris and tensorflow which is what you guys have been using in this tutorial. So I started you know writing some code just very simple neural network uh code kind of similar to what we've been doing in this tutorial to try and tackle this problem. So just to say a little bit more about how we set up this problem. We were trying to train uh a neural network emulator that was pretty uh simplified. It had two hidden layers. It had between, you know, five and 10 nodes in each layer to give the neural network land model parameter values as input and train it on a set of model simulations that we ran with the numerical model. So in this sort of supervised learning setup, we had input parameter values and model simulations that we had run and then we wanted to train the neural network to learn the mapping between parameters and the output. And the broader goal here was so that again we could with a trained [clears throat] machine learning model be able to input new parameter values that the numerical model hadn't seen and be able to predict the output uh in a matter of seconds versus you know running a model simulation which can take hours to days depending on the complexity and the length of your simulation and then we could optimize those predictions to observations. Um, so that was kind of the setup for this initial machine learning project and that's how I kind of came into this work. Um, but I also wanted to mention one exercise that we did on interpretability. Uh, I wouldn't quite call this XAI in the way that Kirsten was talking about last week, but um, we did some playing around with this um, technique called variable importance or feature importance. Um and so what we wanted to do was kind of figure out were there certain inputs to the neural network that were more important to the predictions or the predictive skill. Um so what this is showing is when you take your saved model and you sort of freeze that model and you say okay one by one I'm going to shuffle the information content in each of the parameters uh and then test the resulting predictions. So the x-axis here is the error, the mean squared error. And what it means is like for each bar um the sort of size of the bar, the larger the bar means that when you remove the information content from that parameter, your error goes up. So that tells us that those parameters are more important to being able to get good predictions from your model. Uh that's kind of the ethos behind variable or feature importance. Uh and you can do this for different outputs, right? So, I have two examples here of carbon and water variables. And they're different, right? It's the same parameters on the y- axis, but you see that different ones are important for different metrics. And then we can sort of connect that back to what we know about the physics in the model and start to understand why these parameters are more important for the prediction skill and what that tells us about the numerical model and maybe the assumptions going into the model, the structure and also um the values that we're choosing for these parameters. Okay, so that's just a little bit of an aside. Oh, before I move on, I wanted to mention that >> was the result uh surprising or did you was >> Yeah. Um I think it was it was surprising that we saw the dominance of the one parameter for carbon flux, but we didn't see that for water flux. Um that was surprising to me, but we kind of had some intuition for the specific parameters that were going to pop up as important. Yeah. So I wanted to mention here that getting started with this code this caris this Python code um for both building the neural network and also doing these interpretability tests um I didn't have to reinvent the wheel so I adapted some code from DJ Gier who was also postto at the time um that he had written for a tutorial for some other scientific application but I was able to take that code and adapt it pretty easily to the variables and the parameters that I wanted to look at. So I guess you know making the point and I'll make this again throughout the presentation that you don't have to start from zero. There's a lot of resources out there. Um and hopefully these tutorial materials are also a good place to start. Okay. But I was asked to talk about image detection. So let me just talk about how we got started on this project. Um we were interested in how we could use machine learning to automatically detect weather features in climate data. Um and the scientific motivation was we wanted to understand the different sources of extreme precipitation events and um you know their impacts and how they come from different sources. But at the same time we were seeing this emerging application of machine learning image detection um applied to earth science data or climate data. Um, and so we thought, okay, how can we leverage this work on feature detection and apply it to things like fronts and atmospheric rivers and other things that we'd like to understand and then think about how those features um are connected to extreme precipitation. So that was sort of the broader motivation for this project. Um, I guess I'll give a little bit of an aside that this concept isn't new. This concept of like image detection, computer vision, it has deep roots in machine learning and deep learning communities, but it hadn't necessarily been applied to climate or earth science data. Um, and there's kind of a a spectrum of different ways you can think about applying computer vision to climate science. So, I like this slide from Caric where he's kind of outlining the different ways of that you can classify objects um you know just identifying their type or detecting them or actually what we're doing which is called segmentation where you have the machine learning identify the outline around the image and instead of just saying it's a cat or a dog like tell me exactly where the cat and the dog are in the image. Um, so there's, you know, a subtlety there. And if you see the phrase semantic segmentation in these slides or, you know, in reading about this, that's that's what that means. Um, and so we wanted to be able to apply that technique to detect things like fronts and atmospheric rivers. Um, I'm not going to talk too much about the fronts work. I just included one slide if people are interested. But um I want to note that we were adapting a pre-trained model here, right? So we didn't do the training, the original training ourselves. Um that was done by Jim Barrett and Ken Kungl um who were both at NC State. Jim now works for climate AI which is like a machine learning climate startup tech startup. Um, so we could really leverage the work that they had done to train a machine learning model, specifically a CNN to detect, automatically detect weather fronts of different types. Um, and the paper here, their 2019 paper has all the details on how the CNN was trained. Uh, they used kind of a unique data set from the National Weather Service that had handlabeled fronts and they used MERA reanalysis data to set up the supervised learning problem. So they had some labels of the fronts and they were able to train the CNN to automatically detect those. Um yeah, what else do I want to say about that? So we wanted to take that pre-trained model and apply it to a data set that it hadn't seen before. So some simulations with CSM. Um and the temporal resolution of the data really mattered here. So, we wanted at a minimum to have three-hourly frequency of input because these features are so dynamic. Um, we wanted to be able to capture them um really dynamically. So, we ran this test and we were able to um detect fronts of different types in CSM simulations applying the pre-trained model to the new data. Um, and there's some more details here about the specific fields. the spatial domain was only North America because that's where the training data came from. The National Weather Service data set only had handlabeled fronts over North America. But that doesn't mean we couldn't apply it elsewhere. We just happened to, you know, we stayed with the same spatial domain that they used. And this is a example showing about a month of CESM output um as applied to the front detector. That's all I'm going to say about the the France project because I'm going to spend most time talking about our current work which is atmospheric rivers. Um but more details in the papers here if you want to read more. >> Yeah, >> I was just wondering Katie. Great. Um when you said you got the model, how big was the model? Do you know like this huge amount of data or >> like is it kind of compact? >> It's not very big. Yeah, interesting question. I would have to go back and look at it's you know they fit fits very nicely in my workspace you know I don't have to like >> worry about the space of the it's basically what we did in the previous lectures where you were saving the model out >> um and it's a CNN it's I think it's a little more complicated than the CNN's we were working with um last two lectures but um nothing crazy yeah >> just want to mention that when we save a model, all we're saving are those weights and biases. So, it's a really small amount that you'd have to ultimately save to make these predictions. >> Yeah. The the more data space intensive component is the input to the machine learning model itself because like I said, we were using like three-hourly time frequency. So, to do several years of that, you have this like massive time series of data and this is just North America. So we didn't have but for the ARs we are doing it globally. So I'll talk about that like the >> input data and the outputs to a certain extent because you have again three-hourly time shot time snapshots of the features that that's more space intensive than the model itself. Okay. But before I get into that, I just want to say a quick note about like why we would use machine learning instead of let's call it traditional detection. And here I'm going to use specifically the example of atmospheric rivers. Um so we could say okay why what you know what is machine learning giving us beyond traditional detection. Traditional detection being you know we're going to identify an atmospheric river based on the fact that the water vapor is over this amount or the wind is you know greater than this amount. Um for atmospheric rivers specifically there's a lot of uncertainty across different detection methods. The uh art project atmospheric river tracking method intercomparison project has this very well documented. So this is a figure um from a paper by Christine Shields showing each line as a different method of detecting atmospheric rivers. The metric here is AR events hitting the west coast of the US. So latitude is on the x- axis. Um, and you can see the huge range in the number of events that each detection algorithm is accounting. So, we know we're kind of entering this space where there's huge uncertainty and we have to think carefully about the pros and cons of different detection methods. Um but using machine learning allows us to remove those thresholds and not say okay we're defining an atmospheric river purely based on some quantity has to be above or below a certain value or it has to be this you know the geometry has to be this this size or shape or whatever you want to say. It kind of removes um some of the subjectivity in that. Um and it can also help with the computational expense. Once you have something trained, it could be faster to run a detection algorithm. Also, we were interested in uh not just applying this to present day data, but applying it to simulations with future climate change. Um, and so I'll show a little bit of that at the end. But if you have a machine learning model, you could say we don't have to worry about the the base climate state changing with those thresholds uh that you're imposing. So, [clears throat] but it's kind of an interesting question about how we can apply these methods that are trained on present day climate to future climate. So, we're still thinking about that as part of this work. Okay. But how do we get the training data for these kinds of applications? So I mentioned with the friends project there was a very specific set of labels that we could use or that Jim and Ken could use to train the CNN. Um how are we going to do that for atmospheric rivers? When you're supervised learning, you need some input data and some output data. And our output data ideally would be those segmentations of atmospheric rivers in climate data. So that's where this collaboration with Lawrence Berkeley National Lab came in. Um they were starting this project that they called Climate Net where they wanted to build a database of handlabeled features. Uh and they were specifically interested in atmospheric rivers and tropical cyclones. And the snapshot here is showing how these features can be traced um by humans, by experts, scientists going into different data sets, looking at different fields and tracing an outline around the feature where they think it is. Um and that's how this training data set was built up. It was by asking lots of different people to use a tool to label features um based on their perspective, their intuition on um you know what they think an atmospheric river is um which is also kind of subjective but it's kind of interesting that you get different you know humans perceptions of these features built into this training data set. Um so and then I'll just mention the the paper on the bottom left here is where the training data set and the training of the initial machine learning model is documented. So there's a lot of rich detail there but I want to um show this animation of how this actually works. So in this online tool, it was just a website. Um it samples a random time point from some climate model data. It's actually CAM 5 simulations that they used for labeling the training data. Users can toggle between different output fields, for example, water vapor, winds, pressure, whatever they think is most helpful to them to identify a feature and then quite literally tracing out a boundary around the feature. um labeling it as either an atmosphere group or a tropical cyclone. There was also an option to uh for users to submit a confidence level for each feature which is kind of interesting like uh I'm not quite sure this is an atmospheric river or I'm not you know totally sure on the boundaries here. Um we haven't integrated that data into this project. I'm actually not sure where it is, but it would be great to um think about that in terms of some sort of uncertainty quantification for the futures. Um and so, okay, so formally, how did they do this? They held tag what they called tagging or labeling campaigns at different places um to collect, you know, hundreds of labels. We held one at Encar um with Caric Cashath's help. Um I think this was in 2019. Maybe some of you were um involved with this. Um we just all met in the Damon room and Caric went through kind of um some general guides like dos and don'ts for um how to think about how to label these features um because you know I certainly wasn't an AR expert coming into this. I knew a little bit about it, but um you know I think it was helpful for people to have some rough guidelines about what to look for, how to think about you know identifying these features. And so after they held multiple labeling campaigns in different places, he estimated about 80 people participated in this. There was an extensive quality control process to make sure there were no blatant errors and what people had labeled and how you know they had classified things. 459 images were finalized for training. Now, it doesn't sound like a lot, and I mean it it isn't a lot, and that sort of points to the difficulty in crowdsourcing um these labels, and I'll talk about some different ways of doing that at the end of this lecture. Um but also, I guess the point is in a single image there's multiple events. So, you're not just working with like one event per image. It was quite often as you can kind of see in this snapshot here multiple features maybe five or six total um were identified per image. So okay so once the training data was collected um we could start actually training machine learning models. Um, again, we weren't really part of this process, but um, if you're interested in the the details, there's some papers you can look at here for the initial architecture that they used for climate net. Um, which I think was like a series of CNN's. It was pretty complicated. Um, but you might recognize some terms in this diagram, the convolutions and, um, you know, concatenating and pooling and things that we've talked about. But I'm not really going to talk about this architecture because we ended up actually using um a different sort of simpler lightweight architecture that was developed by a subset of the team. Uh and they called this CGET. Um and the papers here document this really well. And what we liked about using this model, it was faster, it was more scalable, it had fewer train fewer trainable parameters. They also introduced this idea of spatiotemporal tracking. So what I mean by that is in the current state of things the detection model would just take a single time snapshot of input and give you a single you know map of where the atmosphere and tropical cyclones were. Uh but it didn't say anything about how those features were connected over time. So if we want to understand the features themselves and how they're evolving over time, we need to do some like analysis after the um after applying the machine learning model. And I'll talk about that later. But uh we liked that they had that feature possible in this architecture. So we wanted to go with this model and all the code was on GitHub and freely available. That's really important for this project. Okay. So when I when I say when I talk about training these models, let me be very specific about what we're doing here. The input are maps of different climate variables. So I mentioned that the tool had the labeling tool had different climate variables that people could look at. So specifically the ones that they chose for training were total column water vapor, uh 850 hectascal winds, and sea level pressure. And so what you're feeding into the CNN is four maps. Um, and you're learning, you're trying to learn the connection between those four input maps to the human labels of ARS and TC's. So this is just an example of how those features were labeled either, yeah, I don't remember where I pulled it from, either by the humans or the machine learning. So the ARs are in orange and the tropical cyclones are in blue. And yeah, I want to acknowledge the team here, the sort of subgroup of climate who trained this model. Um, the it's a CNN architecture. Um, the CG and CGNET stands for context guided. So just just a little bit about what they're doing here. They're using a CNN in the way that Kristen demonstrated applying CNN's where you're using kernels over filters. But they also have, let's see if I can get my cursor do that. They also have this what they're calling a CG block. um which is when they say contextg guided they mean that they're using more information from the surrounding area to understand the feature itself. So it's kind of like saying I can get more context on what an atmospheric river or tropical cyclone is by looking around the area where it's labeled and not just the specific area itself. That's my understanding of it. And again we're you know doing a lot of applying these pre-trained models. So um not not the expert on that but um you know they did a lot of testing in terms of the architecture and getting something that performs well. Okay. Right. So not reinventing the wheel looking at the code in the GitHub repository. They also provided a lot of nice utility functions similar to the way that Kirsten set up her notebooks where she had, you know, utilities that you were reading in to calculate the Nino 3.4 index or do the standardization. We had the same sort of um resources using the code here. So, it's um using a different machine learning library called PyTorch, but it's very similar to what you're doing with Caris and TensorFlow in this tutorial. So we could take that um code and apply it to um our you know systems and use it for applying this pre-trained model. I guess I want to say one thing which is we we were working with a pre-trained model but we did retrain it ourselves and the reason that we did that was because we didn't have the saved model um that Peter was asking about. we we just had the the training data that they used um and the code. So we basically did this, we repeated the training process just so we could get that saved model ourselves. Um, and so we could kind of better understand the process of training, but we didn't have to like mess with the architecture of the CNN or splitting the training data or things like that because that had already like all those decisions had already been like thought through for this process. Hey, I'm just wondering like from your personal experience, how much do you get into the utilities and try to understand how the whole thing works? How much do you use as kind of like a a tool that you apply? uh initially a tool but then as we especially as we tried to tack on things like the tracking um I had to really dig into what they were doing in the code because we wanted to apply it regionally instead of they were doing it globally which was very computationally uh expensive. So yeah that's one example where like oh I've been relying on all these like functions and now I need to >> really like dig into them a bit more. >> Yeah. Yeah, I'd say it's a mix. Yeah, another example of that is the setting up the environment. So, ideally, we would just use the same cond environment that they provided in the GitHub repository, but we definitely had to do some customization to get it to work for our purposes. Um, I don't remember exactly what the issues were. John Trudale did a lot of this work, but um, you know, it didn't work perfectly out of the box, which is to be expected, I guess. Um, and then another funny thing is like we we found that we could only use a specific type of GPUs on Casper to train and apply this model. Um, my guess is it's probably because the the environment is quite outdated now. We were pulling something that they created in 2020ish. Um, and so, you know, you run into these issues when you kind of like don't update your packages, but at the same time it works. And so I'm not going to change anything. >> [clears throat] >> which is I think very relatable. Um so yeah, so we you know did everything in a Jupyter hub framework and used a similar kind of request to what we've been doing in this tutorial except we were um requesting GPUs on Casper um and didn't have to you know two GPUs um only for this for this application. So it wasn't a huge wasn't a huge hardware requirement. Okay, let me talk about the loss function as part of the training process and the metrics that they used to assess how skillful the model was. Um, and when you have segmentation problems like these, you often have an issue called class imbalance where one of the categories dominates the input data. In this case, the category that dominates is actually the background, none, right? because you see most of your pixels in the image here are not a feature. Um, and so 94% of the pixels in the climate net data set were none. So you don't want the machine learning model just to be trained to know what the background is. You want it to be trained to know what the features are. So you can use different loss functions. Um, for example, this weighted cross entropy or jakard loss. Basically what the weighted one is doing is waiting by the inverse number of pixels I think. So it's placing less weight on the background and more weight on the things that are less frequent. That's my basic understanding of it. But you want to like think about these issues when you're training a model like this. But how do you formally quantify the fit between what the machine learning model is predicting and the human labels? Qualitatively, you can look at these images and say, "Okay, well, that's where the human labeled a feature and that's where the machine learning predicted it." Oh, maybe it's in the right space, maybe not. Um, there's a metric called intersection over union, which uses the overlapping area of the features to mathematically quantify how well the human labels line up with the machine learning predictions. So in image detection problems like this, you can use this metric to say okay, I want to, you know, maximize the IOU like one would be a perfect match or I want to minimize this loss function. Um, and I'll show the the training plots in a few slides where we looked at those values during the training process. Okay, normalizing the input. So, I think we talked about this last week or two weeks ago. Um, when you're inputting fields into some sort of neural network or machine learning model, if they're coming from very different um ranges or units, you want to normalize them so they're not like one is not weighted more than the other. And so again, the inputs here were total column water vapor, 850 hector pascal winds, and sea level pressure. their mean values are very different. They have different units. So when we say normalizing the inputs, we want to do something like zcore scaling where you're subtracting the mean and dividing by the standard deviation. So you can put some sort of distribution centered over zero. There's different ways of doing this, but um this is the method that they use with cgnet. Later in the presentation, I'll talk about the question of what value you use for the mean here. Um because we found that the model is very sensitive to if you're using the mean value from the training data or from some other data set that you're applying to this model. So it's a subtle point, but I'll get into that more um later on. Okay, so here's the training of CGET when we replicated this on our systems. Um, y-axis is the loss function and the x-axis is the number of epochs. So, we've seen plots like this in this tutorial. Um, I mentioned that there were 459 samples from the training data set. Uh, they split that up into 398 per training and 61 for testing. So they did not have three sets. They just had two. And I think that's because they were a bit data limited and didn't want to and wanted to give as much data possible to the training. So there's only one line on this plot which is the training line and the testing data was reserved. So the machine learning model didn't see it at all as part of training. Um and they did that split um based on years of the labeled features. Okay. Okay. And then I I had mentioned this metric of intersection over union. A perfect IOU is one. Um we can look at the performance of that metric also over the process of training. Uh the top plot has the specific classes that are coming out of the machine learning model with the mean and black and the bottom is just the mean repeated. Um so a couple of things here, right? the the model trains pretty quickly to its like sort of base IOU value. The blue is the background. So it's like learning the background really well. But again, you know, there is a class imbalance problem. One is at the top of this plot. Um and then you can see the performance for atmospheric rivers is like hovering between 04 and 0.5 by the end of the training process and tropical cyclones a little less. Um, and this was kind of as as good as they got for the IOU. They weren't really able to improve the performance much beyond 15 or 20 epox. So, we didn't we also didn't extend the training further. We could test that, but I don't think you would get much improvement in the skill. Um, so, you know, it's definitely it's not perfect, but I think it's comparable to other, you know, image detection applications. And then the stars in the top plot are how the IOU performed on the testing data. Um so just sort of like taking that testing data and looking I think across all the samples at how well um the machine learning model generalized to the out of sample data. So there's a slight drop you know in the star in where the star is relative to the lines which is expected um especially for ARs but it's still you know in the sense of the mean it's still performing fairly well. So these are the kinds of things that we look at again this had already been done but we wanted to kind of understand like let's walk walk through the training process and let's understand how they did that. And then at the end you do the same thing you guys have been doing in your notebooks. Save the model so that you can then take that saved model and apply it to new data. >> Okay. Yeah. >> Um if you when you when you look at this loss function like are you happy with it? >> Yeah. Um I think that based on what they saw they didn't get much decrease beyond this number of epox. So yeah, >> because in one of the example that cuz you made us go through like you had to wait for like 200 epox to actually see the final >> like decrease. >> Yeah, it's a good point. I mean we could we could test that. Yeah, I think in terms of like the the matching the IOU matching it's it's not it's kind of plateauing. So um yeah, it's a fair point. Yeah. So it so it mean that if you were if you if you move forward with all of this, it means like you you thought you would your model will give you something that you can work with, >> right? >> Yeah, it's but it's always going to be contingent on how well it trained and the training data that was used. Yeah, absolutely. >> And you say that the loss value itself is not that important. I mean like percentage wise how much you decreased because wasn't that >> so you would always go like a according to IOU or >> yeah I think they were placing more weight on the IOU in reading the documentation. Yeah. >> Yeah. >> I was just going to ask how do you know an IOU that's a good value that would be useful like for an an application? >> Yeah. Yeah. I was thinking about this as I was talking about it. I I think they were basing that on other uses of IOU in the literature, not necessarily even in the climate data context, but other computer vision applications, especially ones that have this large class imbalance where you see that it's getting the background really well, like it's almost perfect, like because a one is perfect. >> Um, and maybe using as a metric like, okay, between 04 and 6 seems like as good as we can get, especially from this limited number of samples. But um yeah, >> I guess as you do it, you learn it, right? >> Yeah. >> Okay. Another thing we can do is look more qualitatively. I was kind of curious again about how the human labels compared to the predictions in the test set. So I'm just showing five examples here of human labels and the associated predictions from the machine learning model. And remember these are like different expert labelled maps um and different time points in the model that was or the data that was used for testing data. Um but you can also look at go back and look at the underlying climate data and again just as a qualitative like un I wanted to understand this problem better. can look at the water vapor fields for what people labeled and qualitatively assess like does it make sense that there's an atmospheric river there. Um the other interesting thing here is that sometimes two experts labeled the same time point. Um so those two that I bracketed here are actually the same point in time of data. Uh so they have the same underlying water vapor. But two different experts labeled the features differently. So if you look at the left column, it's kind of interesting. Like one person did more of like a swirly around some of the >> So the predictions are actually the same because it's the same um sort of underlying data that's being fed into the machine learning model, but there were two different human labels for that time point. So I just wanted to point out that's kind of an interesting thing that came up in this data set. Yeah, >> you instead of using human levels, if you use some quantitative threshold to determine uh your training data and then use that um training. >> Yeah. And I see the code just before. >> Yeah, that's that's been done um using something like the Tempest extremes package which goes through and uses traditional huristic methods to label different features. I think for atmospheric rivers specifically like I showed there's such a huge spread in the detection algorithms that one of the motivations for this is like let's do something that's threshold free that's purely data driven. But what if if you use all those method like and generate sample based on those method and see how the model >> you could absolutely do that. Yeah, great point. Yeah. >> Yeah. Jesse Katie, I mean, how much of this I'm just thinking of the atmosphere river space because Charles site is different, but how much of this is reflective of the fact that maybe we don't have an excellent definition of like what an atmospheric river actually is? Like it's kind like sometimes it's kind of like well, you know, when you see it and then but then it seems like maybe when experts that's not true. So I mean is there kind of like is there kind of like a minimum threshold like we'll never get better just because we don't there is no official definition in a quantitative sense of what that certain pro is. So I don't want to be all pessimistic. I'm just like yeah >> yeah I think that's a fair point. Yeah. Yeah. And this is yeah only going to be as performative as the set of humans that we got to label the features um and how it might compare to other methods. we can later I can show you how this method fits in that spread of AR detection methods. Um I think for our motivation we were interested in eventually being able to combine different features like combine fronts and atmospheric rivers into a single detection method. But how you do that um detection of ARS is hugely contingent on the the assumptions that you're making. Yeah. So I I don't have a really good answer, but I think it's an interesting question about how machine learning fits into this space of like climate features. And maybe it's just more of an exercise of like, okay, we can take the tools that we learned from computer vision and apply them to climate data. Okay. And then just to to finish this like formally, we're doing this by the IOU metric, but this is a way of sort of qualitatively looking at the at the labels. Okay. So now let's take our train model and apply it to some data that it hasn't seen. So some simulations from CSM. So then what we do here is we take you know output from CSM simulation. We take those same input fields, normalize the input, we run it through the saved model, uh, and then we get labels of the features in CSM instead of the training data. Okay, but I just went through this slide in like 30 seconds and this process actually took us, you know, months to do this and so I'm going to walk through that now. Um and I want to give a acknowledgement to Tegan King and John Trudale for all their work with this data processing workflow. Uh so again we start from our labeled features that we got from this climate net data set. Uh there was a quality control process that we weren't part of but I know that that was very you know time inensive going through the labels checking for errors. We ended up with this process data from the labeling campaign. So we had both the the underlying climate fields like the winds and the water vapor and the masks. We have it separated into training and test data. We have to calculate the means and the standard deviations for normalizing the data. We trained a couple different versions of CGET because I was interested in the sensitivity to the input fields. Kind of like what I was talking about with the parameters. How much does it matter if you're using all four inputs versus only two or only one? So I'll talk about that a little later. And then we have this step of processing the CSM output for input into the machine learning model. And that was where the bulk of the work went in. [clears throat] Um you know if we had a direct variable output from CSM at three-hourly frequency that was great. A lot of times we didn't. We had to calculate the water vapor from humidity and pressure. Um we had to interpolate the winds to the specific pressure level. We had to extrapolate high elevation regions because the machine learning model couldn't have any missing data in the field. So the winds had to be a continuous field. Um we had to remap the unstructured mesh from the CSM high-res simulations to a regular latl long grid because we couldn't feed in the unstructured data to the machine learning algorithm. And it expected each file as a individual time point. So we had to separate it into three-hourly files which is where a lot of the space constraints come up. But once you have those two pieces, you can run the process data through CGNet, generate the masks that I was showing previously, and then finally maybe do some science. Um, and then I'll just mention that this this idea of remapping the unstructured grid from CSM took me down this rabbit hole of how to do this in Python. And now we have tools like UXray. This was kind of as UX-Ray was emerging but um I got really interested in different ways of analyzing and visualizing the the atmospheric unstructured output in Python. Um and so I wrote a blog post for the Earth system data science initiative if you're interested in that. Um okay so the reality of data processing right it was a lot of staring at screens like this. So, this is the the Dask dashboard. If you're familiar with using Dask and trying to understand what's going on under the hood, um I think it can be really helpful, but I've found that it has quite a learning curve to interpret what's going on. So, I can look at it, but you know, actually understanding where the bottlenecks are has been challenging. Um so, what I would say to that is just definitely ask for help. um HPC help from the consultants or um the Earth System Data Science Initiative holds these collab events. There's one in a couple of weeks. I went to the last one and sat with Drew Cameron and just went through the Dask dashboard at every step of this regridding and you know managing these high frequency high resolution data and after a couple hours we figured out what the bottleneck was because he was helping me like understand what's going on in each of the bars. So yeah, I just wanted to mention that um as something that is challenging, but there is help available if you need it. Okay, so back to this picture of applying the trained model to CSM. We have the CSM max, we have our saved uh CNN. Uh because now we have like more data than we need, right? We have 15 years of three-hourly output. That's, you know, over 43,000 samples. Um, we can feed that all into the CNN and generate for each sample a map of the labeled features. Um, and it's not a super computationally intensive process. It took about five minutes per year on two Casper V100 GPUs to run the CSM uh, input through the saved model. Um, I found myself doing this a lot. So, one thing that was helpful was to make a function where I generate masks. what I'm calling the you know orange and blue plot as masks um using some pre-trained model because I was doing it for different sets of inputs I was doing it for different data sets so it was helpful to kind of formalize that and then you could make you know an image by stitching together or visualization by stitching together the features uh the feature maps one by one um but again there's no information from the output of the machine learning model initially that connects the events or connects the time snapshots together as a single event. So visually you can tell like there's an atmosphere where birds moving through multiple time steps but actually quant you know doing that quantitatively and like making that into like a labeled event is something that we had to do post hawk. So we ran some other scripts to do that. There's kind of a cool um feature that happens with this tropical cyclone off the coast of Japan or not quite Japan yet, but as it moves up towards the coast of Japan and then it morphs into an atmospheric river which I think is cool. We also didn't um we haven't looked at the connection between the two events because we've really just focused on ARS. But yeah, that's another place we could look. Okay. So, for the rest of my time, I guess I'm going to show a lot of maps like this. And this is how we're quantitatively assessing the overall frequency of ARS in the output of the machine learning model. And this is again borrowed from projects like ARTMP where they make these global heat maps to show how often an AR is detected at that particular grid point. So you can look at these maps and say okay are we getting the ARs in the right in generally the places we expect them and we are. Um okay so what about the sensitivity to the different input fields? I mentioned that we trained different sets of models with different number variables. Um so this is the same plot that I showed on the previous slide in the top and then the the corresponding plot for tropical cyclones. When we trained a model just on water vapor and meridional winds we got pretty similar results though surprisingly the detection of ARS went up in some places which was kind of surprising. When we just trained on water vapor, you could see that most of the information is is still there for the ARs, but tropical cyclones weren't detected at all. So, that's telling us a couple different things. It's that the water vapor is probably the most important field for the AR detection. Um, and tropical cyclones really need winds to be detected. So, we could do this more formally with some of the XAI stuff that Kirsten talked about last time, but we just haven't done that yet. Okay. Sensitivity to the normalization. I found this kind of interesting. I initially wasn't thinking too hard about the values of the means that we were using to standardize the data to put into the machine learning model. Um, but it turns out it does make a difference. So the top plot is when we calculate the means and standard deviations from the training data, which is kind of your standard approach. But what if we use the means and standard deviations from the what we call the inference data, the CSM simulations? Those are slightly different. Um, and I guess here we're looking at a difference between an AIP simulation on the top and a fully coupled simulation in the middle. And the difference plot is on the bottom. The differences for this comparison are are only on the order of 2 to 3% in certain places, but there is a difference, right? And maybe it's saying something about atmosphere only versus fully coupled climatology and how that maps onto the standardization of the input fields. Um, but what really made a difference was whether we weighted the mean in the calculation. So normally when I take a global mean of a atmospheric field I'm going to weight by the cosine of the latitude because of the variation of the grid box size. um we weren't doing any waiting or they weren't doing any waiting in the original CNET uh CGNET code which initially surprised us but now I guess after talking to them about it because the CNN is not aware of the geometry it's not topology aware uh they would argue that you should use the unweighted mean in the standardization but I think it's still I don't know it still kind of confuses me a bit because it makes such a big difference like these percentages are on the order of 20% pretty much all larger. And this is the difference between when you use a weighted mean and the standardization and an unweighted mean. But the same input data, same everything else, but it changes the detection on that order of magnitude. So it really yeah it makes a difference. We can compare our results from climateet from cgnet to a reanalysis product. Um, so this is where we compared the CSM atmospheric rivers to MEA atmospheric rivers. Um, and right now we're trying to dig into why we're seeing these also pretty large differences between the reanalysis and the climate model. Um, I'm not going to have time to go through all the slides because I know we're almost at the top of the hour, but one thing we're digging into is the differences in resolution. So I noted here that the the CSM input is at the resolution that CGET was trained on which is quarter degree. Um but the MERA input was on its native grid of a half degree. So we ran some tests. I'm just going to skip through these about showing the difference of the resolution. Yeah. Um and then we also compared the historical CSM with the future climate. Right. And so here's where we want to think about how ARS might be changing in the future and under the assumption that we can use this machine learning model that was trained on present day climate and apply it to a future climate state. So there's definitely an assumption there. Um and we have to be careful about how we justify that. Part of the reason or part of the way of justifying that is looking at other data sets like reanalysis but also thinking about the normalization. Are we using the values from the training data or are we using the values from the future climate because those are going to be different. Um, and there's a difference here when you change the values based on PAM data or RCP 8.5 future climate. Okay. So, I talked about a little bit about event tracking. Um, we were able to apply code that they had written with part of the CGET project to connect these features over time. Um, and that's basically a post detection script that goes through checks for how these features are connected, assigns IDs to them, and there's some assumptions that go into this. And it's it's pretty computationally intensive the way that it's written. So, we just did it for a specific region. And again, I don't have time to talk about this, but If you want to see examples of how we did this and the differences that we found with climate change, they're all in here. Okay. Future work. I want to mention a few things going on. Um Christine Shields and Tegan are both working on generating labels for polar specific atmospheric river detection. So this is kind of going back to the initial part of this workflow and generating a new set of labels um for Arctic and Antarctic atmospheric rivers. So we can train machine learning algorithms that are tailored for that specific problem. So that's going on. Um some of the folks that worked on the original climate net and cget project have expanded their scope what they're calling climate net large. This is interesting. and they're now uh using professional crowd labelers to generate you know thousands of labels instead of the you know very sort of uh grassroots labeling campaigns that we held. This is quite expensive so I don't think it's feasible for us to do for our project but they have this setup where they're uh having expert labelers generate labels of ARS TC's and they've added blocking events as a feature. Um and so through this process they generated an order of magnitude two orders of magnitude more labels than uh they had for the original data set. So this is just out and no machine learning models have been trained on it yet but it does provide a huge um potential for thinking about this in the future. So, some notes there. Some also some tools I wanted to include in case folks are interested in um using open source tools to generate labels for images. These aren't specific applications for earth science, but I think that they could work in this space. And then I'll just end with um you know a few a few of my thoughts on applying these methods to our our science here. Even with a pre-trained model, you have to do a lot of data processing. Um, but we were able to leverage a lot of existing open source code and packages. Um, I personally found that doing these sensitivity tests across different inputs and resolutions, thinking about this normalization impact were useful for me to understand what was going on in the machine learning model. Um, and we couldn't have done this without these strong collaborations with machine learning scientists from other places who actually did a lot of the leg work of training the models. So, thanks. Before we take a five minute break, do you have do any more questions for Katie? Just wondering like you know so a year and a half to do a lot of the work. I'm wondering how much of it is like you think out the workflow beforehand and then you implement it or how much is exploratory and you sort of have a general idea of where you're going and you find things you test them. Just saying if we're trying to incorporate these sorts of methods into our workflows do we have to be really thinking that there's a lot of unknowns that we have to explore? >> Yeah, it was very exploratory. I knew I wanted to get to the point where I had those maps comparing CEOs ARS, MEA ARS, and the RCP 8.5 future ARs, right? I wanted to get to those three maps, right? But the decisions that you make, like I didn't even realize the MEA one was generated based on a different resolution of inputs. So then I felt like I had to, you know, go down and explore the resolution impact. and then it seems straightforward to apply it to the future climate simulation, but the assumptions on the inputs and how to normalize the data, you know, hadn't thought about ahead of time. Um, and then adding on the the tracking, like I said, I had to go through that super fast, but applying that to a different like not doing it globally because it was so computationally intensive, figuring out how to adapt that to a region um was also not something that we anticipated. We thought, you know, we could just apply the tool and Yeah. So, >> so you had to have so some faith that you'd find the end point. >> Yeah. >> And I didn't even talk about the underlying precipitation data. That's actually most of what we're working on now is like, okay, we know where these features are. Uh what is the precipitation underneath them in the model and how is that changing with climate change? So, scientifically, that's what we're interested in, but it is like a lot of work to get to that. they're building the infrastructure to be able to science question right >> Daniel C is going to talk more about pre-trained model and uh yeah >> yeah so the um second half of presentation will focus more on like actually uh running some things uh ideally you'll be able to get a GPU do that unfortunately as I said earlier we don't have enough for everyone but um maybe if you ask between your neighbor and yourself like you can try and uh uh see how it works out um I recom uh to as again to just go over briefly um if you you get on the Casper system the Jupyter system and you go to Casper previous batch you could select your project account for this workshop um I just sug just just setting eight CPUs because that's a quarter of the four GPU nodes um 90 is also a quarter of the memory available on each node um but uh selecting the compute pick capability 8.0 one model is the A uh A100's or you could select down here, but this is just more generic and better to select the comput capability there. You can also do other ones you want to try for H100 if they're available or V100. If you do the V100, that's a 7.0. Um that one will work with the first example, but not the second one. Yes, >> a little bit bigger. >> Oh. Uh yeah. So, the main thing to selection is this one. So, the 7.0 7 V100s, 8.0 noise 800s and there's a couple H100s but that all depends on the availability on the system currently as they're kind of um hard to come by. Um so nonetheless uh back to just the main presentation I'll just go briefly over kind of like some history of you know numerical weather prediction as maybe most of you kind of think about you know uh so you know we had initially like back in the early 1900s even proposed the idea of numerical numerical weather prediction. It was mainly all kind of physics based processing that's informing the those uh uh you know theoretical kind of approaches. It wasn't until the eniac computer that was created in the 1950s a 500 plot machine you know uh we we now have pedaflops we manage here at encar um was that was one of the first uh actual numerical weather prediction uh process was then done as proposed initially from uh uh Lewis Richardson. So um you know since then since the 1950s you know the amount of compute needed for these physics based model uh models have grown substantially you know from the ENIX starting at 500 flops the Cray one here at MCAR was 160 megaflops that's a 200 km resolution model from you know jumping ahead to the 2000 2002 earth simulator uh that's 35 teraflops at 50 km resolution um the model was a pedop 250 pedlops and more recently in frontier and the theory system on E3SM doesn't a single exoflot for a 3 km resolution uh model. However, for achieving any of these sort of advances in greater resolution, fidelity, accuracy, um that's like you know desired out of like you know more and more compute um you know we're expending more resources you know requiring larger larger computing systems. They're increasingly power hungry. If you ever think about the power usage of kilowatt hours that go into megawatt hours nowadays that go into running of your models that you run on the HPC and all this is very carbon intensive thinking about the uh amount of you know energy being put into to run these systems. So uh more recently in the sense of NDP forecast run via advanced deep learning methods it that presents a potential opportunity to do a lot of this work more efficiently as GPU hardware tends to uh be a lot more efficient in how it processes uh the the matrices and algorithms. if you learned about these past couple weeks and doing these forecasts. Um so uh the main thing that what uh uh the AI component is replacing in these uh uh NDP forecasts is the actual forecast step. So you know the time step from like one time step to another t plus delta t um is what's changing. It's not necessarily yet addressing the uh post-processing or a simulation kind of data simulation kind of steps. Um but there are research work at least to applying like GPU algorithms in those spaces as well. um as uh and plenty of other opportunities to consider. But uh you're more than welcome to refer to some of the resources uh and other talks on these topics if you want to like to learn more specifically on some of these kind of broader uh how the systems are kind of built. Um so uh you know beyond the pure business based forecast that uh Richardson kind of proposed you know you know machine learning kind of in uh weather prediction actually kind of started in uh the 1980 late 1980s 90s and so it's been going on since but it wasn't until like this past decade or two where we actually had access to like GPU type uh hardware that was able to achieve like global scale or or significant tackle significant problems that would be of interest. So if you like to learn more of like kind of the history of practice of AI environmental sciences uh there is a paper a review paper in BAMS by uh Sue helped and DJ um and as well as others uh you're welcome to review that if you are inclined um as for future visions of AI for NWP uh you know current roadmaps like move towards AI empowered for next generation multiscale climate modeling for and you know this particular paper you know focusing on mitigation adapt adaptation but a key point here is a lot of the current models that I'll be discussing are really trained on physicsbased data output from uh you know like ERA5 is the most was one of the most popular data sets used to be training all these models. Um and there's yet to be a cohesive pipeline direct from earth observation into training these models into outputs. And so um there's an envisioning up towards like how do we you know also navigate that data ingestion from the observation step through the output step and having that also maybe feed into its own feedback loop. Um so uh that's what some of the calls to action of some of the prominent researchers in this area are making. Um so in terms of uh deep learning neural networks in numerical weather prediction uh you know GPUs are what has enabled this predominantly through the specific classes but algorithms they were able to accommodate and you know perform at uh extreme scales to to work to function for global uh you know forecasting models and so um you know some examples and there's many others uh you know forecast net has adapted for network oper operator transformers as a transformer-based model um there's graph nal network based model such as Google's graph FCcast and also Gencast alongside it. Um and so all of these are are highlighting the emerging capability of AI for numeric weather prediction. Um you know as a you know this gets cut off in this little slide kind of uh showing here but you know this kind of just highlights the way in which the uh uh the forecast net operator is is running you know it it translates its variables to a 4year space and and segments it into uh if it didn't cut it off you can see it more easily which I'll switch in a little bit to the actual notebook. Um but the it splits up the the uh uh for your space into into 20 distinct kind of like uh uh tokens that will be inputed into the training training model. And so uh there's methodology to that process and you're more than welcome to like read the paper that they put out to describe that as well as any of these you know many other models in a sense competing and comparing and contrasting with each other in terms of different methodologies of how to do this most effectively. um as I referred to earlier unlike energy use um you know the the AI model tends to be about like in terms of if we measure a a uh uh an cost unit to each of these types of uh functions you know ECMWF and how they talk about their AIFS model you know a physics based forecast that they relate is would be equivalent to about 180,000 like cost units but equivalently doing the same in an AI model inferencebased statistics based forecasts ends only being.3 cost units. Um that's only the inference step however and so the elephant in the room remains of the actual generating like the A5 data as well as actually training that model. But once you actually train the model, the forecast itself is much much more cheaper. And so you can get, you know, a thousand times more forecasts out of an AI model than you can out of a single physics based model in terms of uh, you know, that that could be up for consideration in terms of the different types of science might unlock in terms of, you know, studying the ways in which an AI model might produce a forecast and degrees of accuracy or not and all the sort of space and opportunity. You know, the fact that the the energy use, the resources required are much lower, it democratizes access to these uh models significantly. Um and they're not just you know you know uh uh novelties in the sense there are examples of AI for NDP being pretty uh accurate and having skillfulness. So um in this uh uh graphcast paper you know they uh compared it to uh HR um and per their comparisons on like uh on 1380 targets there uh improved accuracy about 90% of the time. So uh seeing these sort of like comparisons and where the AI model actually performs better than other traditional models is actually uh pretty indicative of their uh you know uh there's the reliability in the future moving forward in terms of uh uh valid valid work to be done and understand them better and perhaps even being used in more operational context moving forward. Um but I'll say at least as a before moving on to final exercises uh what created AI for what made it successful is really the the availability of data and how it's practiced in an open science way. So um RA5 again you know it's it spans 1979 to uh to basically today on a global 31 km grid um about a p pabyte and a half of data um but that is a freely open source and available model that you know anyone can download um you know we have it we host it here as well on encourage dex system um many other HPC sites have it on their uh systems as well um but uh you know the the intention of that model and other ways in which uh AI has been practiced nowadays is under the ethos of open science um you know the way in which of sharing not only the data but also the software so um and emphasizing that you know it's important thinking about you know when we are working in the space to realize ways in which we are making the work available in a fair and care way in the sense following the set principles not only in the data sense but also for the software you know how are you making your software easily usable extendable you know can it be easily installed and put on another system which I'll show in how like through studio has been done um you the these sort of like methodologies of how we disseminate our work can more easily democratize the work to other people especially in the context of these uh uh AI models where a lot of them can actually run on local PC they don't necessarily require large scale data center pl GPUs for every single type of model and so if you want say a local person that maybe does not have access to uh an HPC center to run um you know maybe their their risk model they want to run you know there's plenty of opportunity for them to be able to run these things on a smaller like much more affordable system. Um, but they can only do that if the software and ecosystem enables that where it's accessible and usable and under this fair and care kind of practice so that others can actually uh get on board uh uh relatively easily. So, um there's additional resources if you want to go through and like see some uh related talks in this space. Um and then of course I'll go through uh the Earth2 Studio kind of uh GitHub and it's and how that's implemented. Um but any questions from that kind of mini history lesson in a sense? >> You know you you showed your your two numbers between running the physical model as opposed to the you know um machine learning version >> and then you said of course the elephant in the room is the pre-training and processing and everything. >> How much does this elephant cost? >> Uh well this one says 15 billion cost units. So that >> that's pretty substantial. >> So you could run let's see math here. >> Yeah 15 billion divide by 180,000 for the physics space forecast in terms of >> it's 10,000 and I think this is actually and I need to double check how this was like uh shown but I'm they're referring to the creation of 5 itself. I'm not sure here they actually measured the like once area 5 was created like say even training the model which I imagine is also greater than the physics space forecast. So that's the probably the mini elephant also in this picture that's not shown. >> Well things considered it's not it's cheap to run the model. >> Yeah. Once you get there >> I mean even our physics based forecast. Well, well, I think I think still each physics I I need to look the the the costumes for each of these, but each physics model is still like, you know, on the orders of like tens or hundreds like tons of carbon, for example. So, it's it's still it's not insignificant that we shouldn't be like, you know, pushing aside even for the physics space model. But the AI model is just much less, you know, could feasibly be like, you know, order of a number of light bulbs worth like one for an hour, whereas like the other things is much more. the physics based model I guess but like maybe 20 days maximum in terms of >> I think these are like 14 day forecasted yeah I'm going to switch over to the actual notebook itself um because it shows better there um let me scroll up here uh so u all those kind of just converted from this notebook there's a but nonetheless and I'll try to zoom in better at least like see if you can do that easily I do this and then go to this side of the screen. I think it should look better. So, okay. Get to the right spot. Thank you. So, um again, this notebook is intending to use the Earth Studio package. Um it's built by primarily Nvidia, but with support with the uh weather and climate community in terms of empowering researchers and scientists in that space. Um this version of the Earth Studio is the 0.10 version of it. Um it is in a sense you know in active development but it you know is robust enough that you know I felt confident using it here today. So um the main requirements are actually as I said it can run on a on a local PC or laptop and depending on the model you could you don't necessarily need like a 40 gigabyte GPU you could you know manage with a you know a consumer grade class like video game GPU with uh like you know RTX uh the 3000 series for example. Um but uh uh nonetheless you know the basic requirements kind of there in terms of what's needed. Um but if uh each submodel does have a different kind of requirement that you can look through the documentation to see what's needed um for installing it through studio it's actually relatively straightforward. Um I've actually already provided provided it for you on the uh uh Casper system. Um so uh when you are in Jupyter Hub there's now an Earth2 Studio kind of icon you can click and you can actually just import Studio from there. Um but so uh when you load this notebook you can just point it to run that kernel as well. Um but once you have that kernel running uh which is I'll zoom out to show you like that's this this kernel here should be in this dropown box here. Um uh once you have that running, you can then actually run, you know, import it to show studios right there. Um one of the main thing at least I did address just here is to make sure that it's cached and where it saves files goes into your scratch. Otherwise, it might quickly eat up your home space. Um so, um and I think there might be still some bugs in that space. So, please let me know in terms of some models saving to work places I don't want it to save. Um, nonetheless, uh, once it's installed, uh, they they actually, uh, uh, recommend, um, using, so we already have it installed a password is what I meant to say. And then, um, to actually then further install it if you need to. Um, they actually recommend using this uh, UV uh, system. Um, it's a slightly faster even than conda that you might be familiar with in managing cond uh, Python environments. Um, you can there's a page here linked if you have not used UV before to to get it installed. Otherwise, it's as simple as like UV add and you point at the GitHub repo. And if you want to add a specific model such as forecast net or C bottle or maybe um some some data tools within Earth Studio, um there's just you know additional kind of like uh uh calls to to that kind of command to make that installed and make sure the dependencies are there. Um once once that's all done, uh you know, so for example, I've kind of installed the majority models and the one that's on on Casper currently. uh yeah see what all the the dependency uh dependency packages are actually installed there and if you want you can look through that um but otherwise it's uh you know works relatively straightforward from there um so uh about earth studio uh it uh it does have these uh multiple interfaces and interoperability between various components uh the main ones are the prognostic models uh that's for the actual forecasting steps uh and the diagnostic models where you might say uh transform form the uh outputs from a from a forecast to say pre precipitation um or other variables of interest and then of course like components within through studio that manage like IO or if you want to like uh impart perturbation to a a forecast in order to produce an ensemble forecast and also managing the the data sources and data ingestion um there's other utilities as well but this basically just gives an overview of like the various uh components in Earth2 studio um but uh of the prognostic models available so these These are available across the community in terms of what has been um you know developed already and then you know committed to the repository and actually added as a compatible model. Um you know if one model doesn't work you probably would want to reach out to that team. But I believe most of these have have been tested relatively well. Uh so if you wanted to run gaffcast there's you know the small version you can run or the operational version there's versions of kangu you can run you can run microsoft aurora or the fushi or afs or stormcast or the uh uh you know fc forecast networks uh uh forecast nets svn svno model or uh I believe uh d ym is from dkz if I'm not mistaken um but anyways there's plenty of models there maybe in the future be some encry models as well but there also within the credit framework work similar kind of like you know cool chain to to work with uh various kind of models as well. Um you know as for diagnostic models there's a precipitation model based off the uh uh FCN and for network operators um as well soil radiation wind gusts uh tropical cyclone trackers uh sea bottle that's for downscaling and infill um and cordiff um so uh all these are kind of available to play with and that's why I'm kind of bringing this to you. I want you to play with this model. This part of it is to like you know explore and have fun and see what see what you can do and compare and you know look at the outputs if it doesn't make like you're the better expert than I am in terms of the uh uh you know the the the technical theory around um you know all the the climate and weather variables of importance to you um I would just want in terms of providing this particular framework through earth studio this is primarily providing an on-ramp to see what some of the other models are currently doing and allow you to explore them in a more cohesive easy to access uh manner through this uh suffer ecosystem. Um so uh as for data sources um as I before I got ahead of myself just a little bit ago you know you can pull in various other uh um data to to instantiate any of the models you're you're working on. Um so I think for for most of this workbook I'm actually pulling in from G GFS which you can pull from you know erra 5 directly um which actually sources from encar um or uh many other kind of data sets that are made available through various repositories that have um that have it open source and available that are linked into it uh through the uh classes available in earth studio. So again for all of these kind of methods on each of these pages when you open the notebook you can click this uh API link here you can actually look see more information about the actual uh how to how to like interact with these kind of modules and and and and uh components of each of them uh of the earth studio. So for this first example I'm going to go through um basically just going to be running a diagnostic inference with a forecast net and uh precipitation u uh uh diagnostic uh through AFN. Um so the basically it will predict a new atmospheric quality quantity from the predicted fields of the prognostic that being FCN in this case. Um so the important classes from E2S that we'll be using are the prognostic model that is the models.px FCN also use the uh diagnostic model DX.precipitation.fno or there's no dot there actually but it's correct there. Um then we'll also clarify the data source we're pulling from. So be from the uh the data.gfs. GFS and then we're going to use a ZAR back end for managing the uh the output into a ZAR data store. Um and so uh you know as an example like you know here's like you know what forecast net as a prognostic looks like when you say just run it in the forecast mode and you can see you know it's it's for this particular example um uh you know they they showcase uh uh you know uh wind intensity directions uh in terms of various uh locations and they're comparing it from the uh real to uh truth. Um so I think forecast net was one of the first uh uh prognostics developed by Nvidia. Um and so there's actually forecast net 3 which is also in this model but it does require a uh stronger uh GPU um that they didn't want to like risk not having access to during this session or for some folks here. But this this particular first example as I said uh should really only require the V100s. Um it doesn't require that much memory on the GPU to make them function. Um so uh as as for the precipitation uh uh diagnostic component um it's basically taking those uh uh from 20 atmospheric variables produced from FCN and then uh you know sourcing uh using that source 5V analysis data set to produce that that uh uh prognostic it'll then uh you know uh put that through its uh uh uh cycle represent to represent the accumulated liquid and frozen water that falls to earth uh through rainfall and No. Uh and so uh this you know this little uh simple diagram just kind of represents you know how it's it's running this in terms of the uh you know if if no backbone um you know having these batch cycles to run through and then actually you know finally produce precipitation. Um and we kind of see a little bit of that as you uh look as we see the outputs from actually like running some of these cells. So, um, that, uh, I think I might need to actually also, uh, I'm going to I always get nervous running things live, but I'm going to scroll down really pretty quickly, um, because I want to make sure I reset, uh, my, uh, kernel, um, and kind of go back here. Um, but you can reasonably, you know, one run one of these models within this, you know, last 25 minutes of this, uh, workshop. Um so what this will do at least is it will uh it shows some of these like warnings and things just because of the dependencies and not liking some of the uh uh or warning of future deprecations and stuff. You can usually safely ignore those. Um but nonetheless uh it's uh instantiating you know basically the uh uh forecast net model making sure it has access to the pre pre-trained weights. If it doesn't it'll redownload them. Um if uh and then same thing for the diagnostic model it'll uh you know instantiate that that uh sorry P uh prognostic model sorry diagnos model correct yes same thing for that. So basically if it'll instantiate these like kind of models and classes within uh the Python uh environment and then if it doesn't have the pre-trained weights it'll like it'll download them ahead of time. Uh uh so uh when we set up the same other components of like the the data source, you basically just say data equals GFS for the IO component you you say the uh you can specify it being the ZAR back end or as you saw earlier there's many other options if you so chose to how you maybe want to write out the data. So each of these are relatively like you know oneliner kind of setups for each of these models. And there can be different um ways in which you can uh uh you know adjust or or pass different arguments to these to to to uh cater these to to whatever specific situation you might need in terms of the workflow you're working on. Um once you have that set up uh we're basically going to be defining a diagnostic in this step here. So we're going to you know uh specify the time that we want to start our model run from the number of time steps we have. Uh I believe this will run at like six hour time steps. Um the prognostic model itself. So we just pass in that uh uh the prognostic model here. We'll also pass in the diagnostic model. We'll pass in the data source being GFS and we'll also pass in the IO uh ZAR back end. Um we also want to make sure we put in the output coordinate system you want to be in. Um as well as uh the device you want to run on. typically want to run on the the the torch device. This is actually defaults to being on the on the u GPU. So torch device is GPU. If you wanted to run it on the CPU, you can, but you'd have to be a lot more patient and make sure you have enough memory in your RAM. So uh from there, basically, you know, when we have that, you know, we have our run diagnostic, you know, we have the time start there. Uh you have the number of time steps, the prognostic model, diagnostic model, data, io. Um and this basically just you know shows that it fetched the uh uh GFS data that's going to initialize the model based off the uh the time step uh you know tag we we we input and it will uh make sure we have all the data. So it finish fetch fetching it right there. Um and then actually run the inference. So it basically just does you know uh uh nine uh uh you know cycles there to get that inference completed. Um and we'll we finish with the actual like output object here being the uh you know coordinate system of that latitudes the the time steps we're working on and then the actual uh uh uh total precipitation variable across the whole uh uh data frame. So uh once that's all output this actually takes a little bit to process because it's a large file to create the image and sometimes I have to run this twice so I get nervous sometimes running this cell. Um, nonetheless, uh, you can then get an actual like output in terms of the precipitation it predicted for, uh, this, uh, June of 2021. Um, so, uh, you know, that's a that's a simple kind of run through of that process. There's lots of knobs in there to try to play around and like, you know, dig through the API and see what other kind of uh uh methods you might want to change or different sort of uh uh uh inputs or or you can even customize the way in which you're you're uh processing certain variables. Um but nonetheless, it's meant to be an easy to process an already existing kind of workflow that you're more than welcome to to play play around with and uh try out different methods. And so um before I like uh you know move on like the the the to the next example you know all of this is in a well doumented kind of like uh user guide um that they do provide for uh you know earthview studio. So if you want to like say learn more about like agnostic interfaces and how to like uh interact with these models you can see how um each of these are are defined and perhaps adjust them as needed. um and uh say in the future if you ever wanted to like add your own pre-trained model information there's you know uh these this automodel system um that's I think preferred within the earth studio kind of framework in terms of how do you then upload say a uh set of weights that will be stored in like say the NGC model registry which is like you know pre-trained uh uh AI models or the hugging base registry or if you want on like Amazon S3 storage um this one though often cost money you to manage the process of of uh you know how much you want to uh how much Amazon is paying for say the egress cost for those downloads. Uh but nonetheless um there's plenty of like kind of pages here to look through and you know you can go like directly to the API itself for each of these models and see the um you know is there a model anyone in the audience wants to look at quickly. I can like click on one of these and we can just see what they say. Anyone have a favorite that they've heard about recently? They want to chcast. Thank you. So, um yeah. So this just gives a description of that model and it talks about the you know uh different parameters you can pass into that and uh the the data formats of each. Um and so you know it's very detailed and like honestly like yes this is much more like you know comfortable for a lot of how software engineers interact with other kind of packages and things. Um, but you know, this at least uh I I don't want you to be afraid of it. And if it does seem intimidating, you know, there's plenty of colleagues here at Encar that I think be happy to help you navigate like learning how to use these sort of systems. But it's uh very well documented from from how I see it compared to many other projects I interact with sometimes. And thus then it's uh you know uh I I'm much feel much more confident like using a model like this having you know clearly defined kind of variables and at the end of each of these page pages there's typically um also provided examples like mini notebooks you can uh look through in terms of like how they how someone else already set up this model. So if you wanted to say click on um you know an example notebook you can uh uh click on that kind of page and it'll like show a um kind of setup and same process that I just did a little bit earlier and um you know uh give you that same output and so like you can see like how easy and how like at least the the number of lines needed or amount of like effort it needed to do certain types of processes. If you want, you can then download the source code or full zip code of like each of these uh scripts. Um any question given that first example? So I think I'll move on to the second example. Oh, go ahead. I guess I have one like so you're running these models on on on data that already there but what what about running this model on your own data on the own forcing that you'd like to try? uh that would be part of like the you know specifying either like you know a different like public data vo if it is like already existing like in a initial uh uh um data set. If you want to say like you know running your own kind of like perturbation or like forcing function as you said um that that would be dependent on navigating um examples such as uh uh like the a perturbation kind of process or there's other kind of interfaces to manage that. Um, I'm not exactly an expert in terms of fig out what maybe is the best recommended process. Um, but uh uh you know I'm I'm confident you could also even ping the community to like issues in the GitHub repo in terms of like you know asking uh that question but I uh I think through some like recipe guys for example I remember seeing one about like you know defining your own forcings. Um I'm not sure I can quickly find it or not, but like maybe like I can show this. Maybe one of these will like um be indicative of like what you maybe are wanting to do. Um I guess what I'm getting at here is like if if you keep running the script in your notebook, you're running like one model over the same data. So like you know over and over and it's cool but like if you want to use it for you to do something else to see something else >> like what what would you like what would you do which like >> does that make sense? >> Uh I think I'm like because like yeah the majority of like the way which workflow is currently set up for most of the examples is on like you know ancast kind of metric you know and like you know you have some existing data set you already have a train model and can see how it performs on that. Um but uh the the uh there is effort and work to like you know uh uh generate different types of of outputs and you know areas of interest in the research space um uh for specific say functionality. I think that'd be maybe discussion to have with like the Earth two studio developers and or uh maybe coordinating with the DJ and the credit team in terms of like maybe some of the ways they're thinking about implementing their model that is uh uh and how that research community is for forming. Um yeah, >> would it be fair to say that Earth Studio is more for if you wanted to evaluate how good these AI models are on your specific application rather than trying to give it your own data like throwing CESM data rather than ARPI. I think a fair statement. >> I would I think that's pretty fair. I think there there are opportunities to like get out of that boundary, but there'll probably be a big uplift I think in terms of just like you navigating the specific ways in which it ingests like a new forcing or some other you know other data set you want to feed into it because a big key component of it is it's trained on you know certain types of data sets that have be for formulated in like say the GFS format or has to be formulated in like RA 5. Um, and if you try to input model that's not of that same type, the AI model won't really know what to do with it because it's it's not you you have to massage it into the way in which it was trained in a sense. So, um, if if that's a limiting factor, then you probably want to find some different approach. >> Yeah. >> Could you also expand a little bit on how Earth 2 Studio relates to credit or and how they're the same or different? I know a lot of people have heard about credit, but maybe it'd be a nice way to connect. Um yeah, in that case I would wish like DJ was in the room >> with someone. Um so uh I believe so studio as you said is like mostly inference based and maybe can speak more uh after I finish um whereas like credit is uh focusing both on training and inference um and uh is at least developed here at Encard and I think it's more has that uh focus on like the research community. Um I don't know if you have any other thoughts Negan in terms of what you think of priorities. >> Studio is mostly just an inference workflow for um like big trained modings from the industry uh some of the more famous ones and it can support credit trained models as well like for example it can support and other models if it it's added to the model registry. credited on the other side as more of a platform for I I see it as very uh nice platform for distributed training creating these train models on for CDSM like with CSM data or like some with cand outputs different types of outputs. So I think their use cases are different with credit. You can do inference workflows as well but some of the graph cast type models are also credit. support which is one framework for supporting many different streets I think is great for distributed training and for >> right and I think to reiterate Negan's point I think there's a uh active issue currently where you know DJ uh was you know intending to work with some of the studio developers to integrate some of like uh you know credit ecosystems of uh pre-trained models into into the through studio. Um, so it's an opportunity for collaboration and kind of work and kind of just uh uh I think as I was mentioning earlier, it's it's the the democratization of access because not everyone is going to be needing to, you know, train AI model that don't even have the resources to do that. That's going to be the most intensive operation of any like AI workflow is actually training any of these like large scale like weather forecast systems. However, the inference component is readily and should be easily accessible. And I think making that easily accessible is a uh important goal for for any of our work in terms of having a broader impact of our science. Um so yeah I think I'll move on at least to the second example which is going to be using uh sea bottle. Um so uh so c bottle or climate in a bottle is a generative foundation model for kilometer scale global atmosphere. Um you're welcome to uh click on these links in the uh in this uh materials if you look at it now or later to read some of the related papers. Um this one however does require the at least the 40 g gigabyte GPU I believe. Um actually I'm not sure if it I think it's just the 40 GB. It might need a 81 but nonetheless it requires a lot more memory. So you can't run on the V100s nor a typically a uh workstation PC um unless you happen to have you know one of those GPUs on a workstation. Um, nonetheless, uh, this one has a different set of, uh, uh, you know, components you're going to select to to pass into the model. So, uh, we'll be using the C bottle 3D, uh, uh, uh, data source as well as then the, um, weatherbench 2 data source. The diagn diagnostic model is the actual infill model. So, this one is the, you know, filling in the gaps within and going downscaling into a higher resolution kind of uh, model output. um to cottle infill and the super resolution model is then the Cabel SR. Um so uh running this uh again you can uh you know we have the you know just basically u you know import the data studio kind of like objects components you want to work on uh you want to make sure you're running on a CUDA device um uh you set the the package you're in and the data sources you're going to be using. So, we have the Cuttle data source. Um, uh, and then making sure we pass that data to the GPU device here. Um, we're going to load the the the window size of the super resolution we're going to be working on. Um, so like if you wanted to look at different areas of or or domains of the region you're working on, you can adjust these sort of windows for example. Um, you have the super resolution kind of uh uh prognostic component. Um, so you're going to load that model as well. Set the output resolution you're desiring. Um, and then the window defined previously. Um, and then you have the infill model uh we set up with the uh cello infill. Um, and the uh input variables we're we're interested in and number of steps we're we're uh uh telling it to do. Um, so then uh once it's all done and I believe that one finished. Okay. Sure. So then we uh can go to the um generate generating this synthetic climate data for C battle sea bottle 3D. Um so this might be a step you could like bring your own data in this case and then C bottle can then downscale it to uh what uh how it needs to. You just have to I just recommend reading through the documentation to see what the requirements are of that model in terms of the types of input it would expect and can actually reasonably work on. Um so uh I'm going to let this run. Uh I'm not sure we'll report the exact amount of time it takes but nonetheless it's a you know relatively uh quick generation to you know should be able to finish within the last last 10 minutes here. Um but uh once that completes uh we'll then run the uh you know super resolution step on that ER5 data that uh that uh was gener synthetic data we generated um or on the real data here sorry um so then uh but nonetheless we do use the C ball infill model to predict the missing variables uh then perform the super resolution. So I think it'll just run in sequence to that. So that one just finished and then this one will start. So it did uh download some of the um or double check it had the variables from uh the ER5 data set and then it will uh run that uh infill step as we uh needed to produce the uh uh input for the next step. Um so then uh once we actually get that um you know that super resolution infill that we wanted we would then actually uh postprocess that um output. Um so this just you know gives various different uh uh you know plots to to to uh show what we just created. Um instead of having it just run again I could have it run again just for you know showing that it works nonetheless. Um now the top left here is the synthetic data that C model 3D produce at low resolution. Um and then we have the super resolution of the same at the top right. Um and then we have the ERA5 data low resolution at the bottom left and the infill of that of that um uh data on the bottom right. And so um uh whether or not any of these are accurate that's for a researcher to look at and do their own analyses and statistical assessments on. Um but uh it it goes to prove a point that there's an you know interesting work happening in this space in terms of some of these models and um you know whether or not you read a paper and then like oh that looks interesting. I wonder if I could try it out myself with model with you know workflows like this you you can um so you can uh uh so long as like some of those developers are encouraged and uh to actually add into um you know some of these frameworks like Studo you can actually download these models and with a few lines of code um you know compare contrast and double check what exactly the the paper author uh and team are are claiming in their in their paper and maybe do your own kind of like assessment analysis on some of the models that they're developing. and see if how well they work, how well they don't work, what maybe needs to be improved. Um, you know, I'm hoping maybe through many many of the other kind of like ideas or or topics you maybe learned over the past few weeks, you might say come to one of these kind of like models and be like, "Oh, wait. It's I I'd like to maybe try to implement this kind of step and I'm kind of like aware of where where now where I need to But nonetheless like you know this sort of like framework at least gives you uh gives you the first step towards understanding maybe where you want uh want to go and what uh would be interesting to explore. So um that's about all I have in terms of like these um examples we're running through. Um but uh there's plenty of other ones uh we could try out. Um I'm not sure we want to do that live at the moment but nonetheless [laughter] um you know the the if there's specific needs in terms of say if you start playing around with Studio because it is now available um here let me show it really quick uh zoom back out here um so if I open up uh you know a launcher here um this environment studio and I might actually I'm going to edit this later probably put beta in a sense of just because like it it is like a work in progress it is 0.1 1 um of like their release in a sense. Um I think there's still a lot of active work in that space and you know things may not be perfect but um nonetheless you know it is there and it's uh you you can start run uh pick up your own notebook and create your own kind of workflow and play around with it and um if you have any issues you know feel free to reach out to to me or others in the uh CSG can always submit a ticket to consulting as um was just recommended earlier. Um that's the rchel.youukar.edu to you. Um but yeah, um appreciate your time and hope you have fun exploring um infant space weather forecasting.