Lecture 6: CAMulator & CREDIT | 2025 EdEC Machine Learning Bootcamp

Hi everyone, it's me again. Um, [laughter] um, after Daniel's talk, I thought it might be kind of cool to talk about an emulator that we have in house. He talked to Earth, he talked about Earth2 Studio, which is a platform for us to actually use emulators um, easily. But I wanted to talk about that we actually have emulators inhouse and we actually have a platform for you to generate your own emulators um through the miles group which is led by DJ. So I'm going to briefly talk about what an emulator is, talk through emulator um which is one of the emulators we have and then also talk about the um credit platform which is how we can actually generate these at Encar. Um and then I just want to reiterate I'm the messenger here. The main lead of CMulator was Will Chapman who's now a professor at CU Boulder. All right. So, just to define emulators. Emulation. Oh, you're just making up. Okay. Emulation refers to the ability, this is from Wikipedia, so trust that as you want. Emulation refers to the ability of a computer program and an electronic device to emulate or imitate another program or device. So when we talk about machine learning emulators in earth system science, what we're referring to is the ability of a machine learning model to emulate or imitate one of our traditional models. It's as simple as that. So there's two main types of machine learning emulators. There's the auto reggressive um model which is a lot more similar to like our climate or our weather models and then there's the generative model which is kind of a mind game um if you're not as familiar with this type of model. Um but camulator is an example of an auto reggressive model. So basically all that means is that you have input that you put into your model. It outputs something. That output is then passed in as the input to make another output which is passed in as the input to make another output. And that's how we roll out our forecasts. On the other hand, there's a generative model which actually learns the distribution from your training data or generates states from a learned compressed representation of climate data sets. And I think the best way to understand this is through an example. So climate in a bottle is an example of this. It was generated by Nvidia. Basically all you give this generative model is these conditions that it must follow. And one of them is the day of year, then there's time of year, and then there's monthly average sea surface temperatures. And from these three variables, it'll generate a climate state for you. Um, it outputs 11 surface uh variables and four pressure level variables. Um, and I linked both these papers at the bottom here for climate model and came later if you're interested in learning more. Um, do we have any questions about >> Yeah, >> I guess to kind of the more in context are the ideas that like autoressive is more prediction focused and generative is more like analysis kind of. >> So generative you can actually in a predictive sense. um where you can't really necessarily like follow the trace of a specific like weather feature, but if you put in like your monthly SSTs in sequential order while the atmospheric state isn't going to be like identical, you could in in that regard and I think they show an example of that in the app. Yeah, there there also some ways it so if if you actually uh the way like in a bottle some of the models are set up they just like kind of sample from a completely random distribution but if you kind of like the other stochastic methods if you have like structured random noise that has is correlated in time and space then and you know the random seed for that noise generating process you could force a trajectory that has a stoastic forcing to it that's so like each trajectory is deterministic but but with a different stochastic route. So, so there are ways to mix the two. You have to you have to be more you have to be more intentional about it. Okay. So, just some general background history random things that popped into my mind when I think about machine learning emulators in the earth system sciences. We've mainly been focused on weather time scales. So far, most emulators are trained on Aero 5, whether that's good or bad. Um, we've been dominated by atmospheric emulators, but ocean emulators have started to appear and have started to be coupled to atmospheric emulators. Um, typically when we hear of climate emulators, they're referring to an AIP style simulation. Um, though this is quickly changing as I just mentioned with the coupled emulators. Um, and then there's this new focus on substance to seasonal uh time scales that people are interested in. Okay, here's a quick promotional video of the AI model that we're going to talk about. This is Camulator. Um, this is specific humidity at the lowest model level. And as you can see, it looks quite physically realistic. And I'll I'll show that it actually is. And I'm going to step through how this was actually trained to get what we're seeing in this great video. I think what's kind of cool is you can see the pulsing of the specific humidity. I'm assuming is associated with the dal cycle. Actually don't know the the temporal frequency of this video. Do you? >> I think it's six hours. It's >> six hours. Yeah. Just let it play for a little bit longer. Okay, so let's talk about camulator's training data. So when I say camulator, [clears throat] this is an emulator of CAM. CAM is our community atmosphere model here at ENCAR. And so what Will has done is he generated a 35-year CAM simulation and saved out six-hourly time steps for from this model. This is using um an AIP style run using SSTs from 1979 to 2014. So we basically have this long giant time series of six-hourly time steps for a variety of variables. And what CAM is tked to do is take an input at time t and predict 6 hours later. And that is all it's trained to do is take in t predict six hours later. There's no auto reggressiveness in this training process. So in terms of the training data, I want to talk about some variable um definitions. So we have prognostic variables. These are input and output by the model. We have diagnostic variables. These are only output by the model. We have um and then we have uh forcing variables and this is only input into the model. So I took the liberty to mess with this table and put it into an input table and an output table. So in the green highlight we have our input features and in our out table we have our output. So on the left hand side we have our prognostic variables and our forcing variables. These are our inputs. Prognostic has UVT and Q. These are all on 32 um hybrid sigmo pressure levels. Um and these are all 2D plus vertical. Um then we have two surface variables. So we have surface pressure and near surface air temperature. And then we have our dynamic forcing variables. So we have incoming solar radiation and sea surface temperature. And these vary dynamically with the simulation. So we have an update every six hours to force the model. And then there's the static forcing variables which is um surface uh geopotential which is to tell you about the top topography um of the earth and then the land sea mass to let you the model know whether it's an ocean or a band. >> So what's the resolution of the angle of the output for that? >> Uh one degree >> yeah one degree >> one degree >> 192 by 288. So it's still somewhat degree technically but >> all right. So then um do we have another question there? Okay. So then um we output UVT and Q at all 32 pressure levels. Then we have our surface variables that we also input. Those are output six hours later. Then we have all of 15 diagnostic variables that are also output. So this includes precipitation, clouds, wind stress, uh 10 meter wind speed, and a bunch of fluxes that are all output by the model. So all of these variables are included in our loss function when we actually update the machine learning models weights. So we're actually minimizing all of these these variables, the loss of all these variables. >> Is that cam 6? >> Yes, I think I have. Yeah, cam 6 and CSM v2.15. Okay. So, I made a simplified diagram of this um architecture of this model. Um, and I have the the real one that's in the paper. So, I'm just going to step through this one and then we'll jump to the a little more daunting one from the paper. So, basically all we have is some atmospheric variables that we put into our machine learning model and these four scenes. This is passed through um a bunch of layers that's in this uni unit struct like structure. Then um they've also included these um conservation. So they have a non- negative correction, dry mass correction, moisture budget correction and a total atmospheric energy correction. And then we have our outputs. Some of these outputs are then when we actually roll out the model are then put back as inputs into this model. And then the the coloring just represents the types of layers. But this is not necessary to understand for this talk. So um just to mention some of the terms that we've talked about here. Um the data is normalized using a zcore normalization. So it's subtracting the mean dividing by the standard deviation. All of the variables are standardized this way. The loss function is a latitude weighted mean squared error. It's trained for 190 epochs. The optimizer is atom W. The learning rate is 1 e to the minus5. It also has a scheduleuler that um updates the the learning rate as you train and the batch size is 32. So these should all be slightly familiar names to you guys in terms of actually training a machine learning model. So this is the schematic from the actual paper. So the the top half here is um similar to what I already showed. So we have our inputs. It's passed through the model. We have our physics um conservation. And then we have our output. And then the uh B, C, and D of this panel describe the actual architecture um of these individual layers. What's interesting about and really cool about this model is it's on the order of a thousand lines of code. Um it has 750 millia parameters. It's about 0.5% of GPT chat GPT and it's only 2.8 8 GB in size. Okay, so now that we've trained this machine learning model, let's talk about how we actually roll it out into the future. So typical to our general general circulation models, we have our input in in these GCMS. We then have our grid scale dynamics. This is just for sub um grid physics. And then we have our conservation corrections and we get an output. That output is passed in as input and then we roll it forward again. Exactly the same process here. Instead of grid scale dynamics and subgrid physics, we have an encoder and decoder. We do those conservation corrections and then we take some of those outputs and use it as input into our model. So basically what if going back to these tables, we have our input variables and we have our output variables. A subset of these output variables are then passed as input into the model and we can roll this out in six hourly time steps. So let's say we wanted to roll out camulator for 200 years. I mean this is part of our climate model. So we want to make sure that this is actually stable. So we are rolling it out in terms of six hourly time steps to get to 200 years. And this is showing the column integrated heat content of cumulator. Um the actual camulator run is in all those squiggles. This is forced by the 2000 climatology um sea surface temperature. And what we see is if we calculate the long-term trend, which is in this dash black line here, it's relatively flat, which is what we would expect um given given this situation. Okay. What if we want it to look like observed the the observed past? This is um CAM and CAM from 1979 to 2013 SST. So CAM's in purple now. Camulator is in teal. Um and this is a 12 member ensemble of camulator compared to CAM 6. And what we can see is when we force it with observed SSTs, it generally follows um CAM which is what we would hope. Uh interestingly as well, we also get this slight um increase in the column average um key content which is suggesting that it's actually able to get this warming trend as well as internal variability. What I find really interesting about camulator is even though we're also taking these six-hourly time steps, it can get these large long-term modes of variability. So, here's an example of the precipitation response to Enso in CAM in DJF and in CAML um in DJF. And the pattern correlation here is 0.9, which is really good. So, I find that pretty neat that we're able to even though we're taking those six-hourly time steps, we're actually able to get this this signal. Um, I also just wanted to include some climatological mean biases. Um, he's also included here um some observed or reanalysis type features. So, we have GPCP and AR 5 for precipitation and temperature respectively. Um, but it's important to note that we're emulating CAM and not the real world. So we're trying to match CAM and not the gray dashed lines here. Um and this is just um yeah those biases. >> I have a quick question. >> Yeah. >> What provides the randomness of the 12 member ensemble? >> Uh >> is it just perturbed initial conditions? >> I think it's perturbed initial conditions for for >> I think it's Yeah. >> Yeah. Just purve initial conditions. We're not doing the other stoastic stuff >> for ST. >> Yeah. >> Yeah. SST it's >> after about 14 days we it basically gets into that like but when we see the the error growth and then by 14 days you're you're essentially in different trajectories. >> Gotcha. [clears throat] >> Okay. Another thing um that I think is interesting is what is what happens when we have out of sample distributions which is typically what we use climate models for. So um what Will's done is he's taken the 2000 SST climatology and he's added in 2 Kelvin uniformly and 4K or 4 Kelvin um to camulator uniformly. So the uh the dash lines are for cam and the non-dash lines are camulator. The blue is an example of the 2000 climatology. The orang-ish brown is for when we add the two um Kelvin and the green is when we have um the 4 Kelvin. And what you can see is from the the top we have our lowest temperature model level all the way to a a 200 to 850 x pascal integrated heat column heat content. And what we see is that as we move away from the surface since we're not we're forcing this basically forcing this surface through the SSTs. As we move away from the surface you can see that the green really diverges. um between cam and camulator. All right, that's camulator. Do we have any more questions? >> Yeah. Yeah, Kirsten, thanks. Maybe you or DJ could talk a little bit about like the motivation behind why Camulator was developed and some of the use some of the use cases that you see going forward. >> Yeah, I can I can talk a little bit. I think part of the motivation of it was we wanted started with want to train a model on something other than error five. So so pretty everything was train has been trained on error 5. Uh the problem with error 5 is that it has well is a good a good representation of the atmosphere like the atmosphere as it is based off of the IFS and the data simulation practices at ECWF. It does have limitations. There's artifacts from like bringing in satellite observations that show up in the fields. There's uh unrealistic jumps and uh when whenever a new uh data simulation update update stuff happens and sometimes the models kind of struggle with with with dealing with those. Um some of those we wanted to see be able to yeah extend this to cl kind of climate time scales and see how can we roll out stably for a really long period of time. I also have more control like when you train off of a a climate model run everything should be self-consistent so you don't have some of the weird jumps that you get with with a weather fields. We also will decide to use average fields which um in some way will like damp out some like finer scale weather like signals but but for client purposes if you're wanting to do longer term averages it makes sense to do that. Uh so so there there are some performance benefits there. Uh we want >> means using six hourly average bills instead of instantaneous. >> Yeah. Yeah. Um some of those Yeah. We we had developed physics constraints set the weather scales. They want to see if those would help us at the climate scales as well and just have another application of credit but that using an incar based data set. So we would have kind of our entirely our own product from end to end and and not be relying on another center for for data. uh in terms in terms of what we can use this for. I think that the there's a lot of different applications. I mean in the longer run you want to couple a camulator with a mulator and a climulator and a CI later and so on and have have a full CS simulator. Um uh and and we can also use this as a so so one application if you're running like uh you only care about the upper atmosphere or you only care about the ocean or you only care about the land and you but you want it like a a coupled atmosphere that runs really fast and isn't your bottleneck on getting the results you want. the emulator has very attractive compared with prescribed forcing which isn't interacting with your your system because you there are a lot of studies they're showing if you have a coupled atmosphere versus a prescribed atmosphere you get very different results. So so there's so being able to like one goal we want to have it is is actually coupling this with not just other AI models but with physics based models and and uh seeing how how how effective that can be. Uh also ocean spin up I think is another use case that's come up. Uh if you can spin up an ocean with like a mulator uh like that currently I think takes like two months of compute time uh for the part of the run you basically throw away. Uh so if you can do that in two hours uh or or a day or whatever then that's that kind of changes a lot of what you could what kind of science you could do. Um I think there's other applications we can talk about more in the Q&A. So I I'll stop there. >> Okay. >> There's a question online. Um how long compute time did it take to train? >> 3 days on 32 GPUs. So uh and we could probably speed it even more. >> Is this enough? Uh I I mean compared with so like like a lot of if you're at I feel like quarter degree it usually takes a couple two to three weeks to to train. Uh there's probably we think there's some ways to speed it up with how you do the data pipelining. Um we could probably get even get this down to one day if we if we were really ambitious. So some of this is like how much like multi-step training do you want to do? How like how well is it converging? Um how how many variables are you predicting? So, so there's there's some light math you to to to play around, but but order days I would say for for training a camulator uh at at one degree for me clip will probably take weeks. Uh if we want to do a mea clip later um which we we want to do um >> because of the higher resolution spatial resolution. >> Are we sticking with the you later naming architecture or are we taking suggestions? I think that would be a problem for discussion but CGD and emulator team that's thematic thing you want to keep doing for now always rebrand at some point [clears throat] it does seem to be popular >> sorry go ahead you have a question yeah >> oh wait for me sorry yeah no well maybe to come answer I'll just you know you trained on cam six but cam 7 is definitely going to be here any day now and So, uh, like was there plans of like, oh, just retraining it again on Cam 7? Yeah, I guess like going to the future plans like anytime a new version of these models comes out, do we want a new emulator for that or is it kind of only when there seems to be a desire for it? Yeah. Yeah. >> I mean, that's something I've discussed with Dave Lawrence is have you I don't say if you have a firm commitment, but certainly you you once you have a new version of CAM Yeah. you do you do some test you do you do some long runs and train and then train a train a new off that it's it's it's quite feasible to do to have and then then do some scientific validation on it and and release that or some mix of them with with w with a new CSM release I mean >> yeah cool thanks >> yeah Judith >> can you talk a little bit about um that problem of um AR models or auto reggressive models with the smoothing and how that was dealt with there. >> Um I don't think the smoothing was necessarily dealt with. It does smooth um but I I think whale has a new upsampling or um decoder part of the model that um makes the smoothing a little bit less. But yeah, typically with auto reggressive models, you ultimately end up smoothing the output of your field as you um roll it forward. >> So Delan has this this thing that if you don't train one time step, but two time steps, it's one way to avoid the smoothing. Say that >> one still it seems to help a little bit. It doesn't get completely get rid of the problem though. And a lot of Dale stuff is also at much coarser resolution. And so it's not clear that it's helping at the final resolution. Also seeing his like from the act people running his stuff for predictive purposes seem like the model actually seems to struggle quite a bit. So I I think it's like two time steps isn't like there can be some advantages but it also comes with the computational overhead of having to predict a whole extra state. Uh some of the stoastic so like doing cops training seems to also help with smoothness. uh having more explicit vertical like um interactions also I think are would would would help some of the smoothing issues because I think under the underlying issue with the smoothness is we're not capturing all the processes we need to capture there's basically the neural network is trying to is just I can't I can't resolve that so I'm not going to I'm not going to predict it so unless you do it if you do it stoastically then then that issue kind of goes away but because you're sampling from a distribution rather than predicting the mean but if you have more information to condition your mean so that it's got a smaller space to smooth over. It should get less smooth. >> Thanks. >> Yeah. >> Kind of related to that, um, do one of you guys want to sh talk about a little bit like the differences between single step training and multi-step training and what that applies? >> Yeah. Um, so can I didn't mention this here just to keep things simple, but camulator actually has two steps of training. The first is it takes a six-h hourly time step and then the second version of training that it does is it does take um two time steps. So it does 12-hourly and then calculates the loss. I think it's the average loss based on those two time steps. Um and also includes the conservation then they found when they included the conservation um in the earlier training that the model didn't do a good job at training. So that's why it's it's later on. >> There's another question from Alison in the chat for for the cany. Can you say more about how many times did you need to retrain to figure out batch size number of box hyperparameters? >> I do not know that answer. Do you know that answer? >> Uh more than we care to admit. [laughter] We there certainly a lot of especially during the initial um like credit development process we we uh there were a lot of iterations on architect on figuring out a lot of the details a lot of like oh this didn't work right let's try something else and uh thankfully many lap like in addition to our sizzle allocation CGD and then cute are also happy to donate some of the GPU allocation under ratio for to to to this effort so um [clears throat] >> but now that we have it I've actually used this architecture to train something else which I'll briefly talk about later. Um, and I didn't have to adjust anything. So and that we have >> say I think that brings up an interesting point that like I think hyperparameter optimization for like pretty big models like these is kind of like you know a challenge right now in the field and it doesn't necessarily follow like the same paradigm of this kind of traditional hyperparameter screening in other models where you might be able to do brute force or even kind of have a you know smart kind of sampling procedure that the resources are just so much that there's a lot of unexplored space. Yeah, these bigger models. >> Yeah. Yeah. We're going to have to rely more on either proxies from like training a smaller version of the model and seeing if that those those parameters generalize to a bigger version. Um kind of relying on how fast is the loss going down. So only train it for like an epic or two and like then then just being really aggressive about kicking out candidates that don't seem to be doing well. Uh being more responsive to you know comp computational balance. Um, we've done this kind of more manually and you kind of gotten a feel for some of this, but shown there ways to automate it more. It's just a it's a it's an intensive process to like there's comp doing this is a lot more infrastructure than doing like a random forest for instance. >> Yeah. >> Yeah. Two questions. Um, in terms of actual numbers, like what how long does it take for us to do a six-hour forecast to compare to CAM? Like what's the savings there? I think it well said it's 350 times >> 350 times >> faster. >> Yeah, we can do 480 simulated years per day with the emulator. uh at least the version in the paper compared with I think 14 simulated years per day for for CAM and and that that's also 480 simulator years per day on one 8100 GPU uh versus 10 14 simulator years per day on 10 DO nodes maybe 10 Shyen nodes but still uh like like it's a m a lot like a lot faster for a lot less compute so if you want to do large ensembles came later you can easily scale that And the second part is um so obviously we've seen six hour outputs and perhaps it was trained to do this on 6 hours and 12 hours. Um now you know if I want to do let's say long-term prediction with with cam I want to see what happens in 10 days. I'm guessing you can do that with cam data but can you trust what you get what what it gives you? Yeah, [clears throat] well that's what some of our slides are showing is like yeah we we only train out to 12 hours but it but it does seem to be picking up on teleconnections and other like structures in the data that have longer term uh that basically are slower signals. Uh and I think part of it is we have all we have we have a full set of vertical levels compared to where we only were sampling a subset of of the state of the vertical state. So, so we capture a lot more features and those seem to propagate through time really well. Um, the one the one thing where where there still seems to be some open question like like we added CO2 in a in a newer version of Camulator as a field. Uh, but the model does and it seems to pick up if you if your SSD and CO2 are consistent with each other. The it seems to pick up on the right global warning system, but they're they're inconsistent. If you hold one constant, let the other one go. It you you get some non-physical stuff going on. So, so there are there are some processes that that I wouldn't entirely trust just yet. Uh, especially if you're like [clears throat] interested in the radiative forcing effects and which was kind of an important important climate process. >> But there has been like green function work that's been done showing that it's actually kind of okay. >> I think maybe the last question is a different one. I think we don't know if it captures divergent trajectories in a chaotic vision. So it might not get an error growth right unless we're saying yeah it's not not not I think it's doable we know that from other models but that was not what intent of this particular >> yeah yeah this is more intent like more direct to climate time scales and and [clears throat] we are we are seeing issues where it it varies by the model some of them seem are are like hard to knock and like don't seem to have as much of the chaotic error growth and some have a lot but too much it really and they tend to blow up. Um and with with stocastic forcing we can we can get I think good ensemble diversity uh and with consistent like like being able to do it consistently through time you can kind of calibrate it out but but the like deterministic time steps definitely are underdoing chaotic the chaotic error growth of course there's also the counter argument around this is is chaotic error growth is a property of of a model rather than a like but how much is it a property of the underlying system. Um, so so that so that there there's some maybe not the best place to get into some of the debates on that, but but there are but that that like the the fact that the ML models are are kind of like have this sort of implicit conditioning of larger scale, medium scale, and smaller scale onto the system. Whereas like numerical models tend to only be local and their in their updates in terms of like spatial updates. um it kind of leads to to different sorts of evolution structures um for better or for worse. I'll just also mention um credit is exploring a variety of ensembling techniques for these emulators and I have some listed here on the slide including perturbing the latent space um using a variety of initial conditions scaps and diffusion methods >> steps within the AI >> yes >> is Donald contact with that >> yeah Donna Kar has been working on that he he does like a sort of Scabs, but not exactly scabs. That that seems to work late in space, right? Or where does it where is it implemented? Like if it's not in the physical model just trying to >> we have both. So we have scaps as scaps um and that's not fully that work is ongoing not fully tested yet and it has to do with the question if you can in the chaotic regime how we needed if we need to treat initial condition vertebrations initial condition perturbations or if we can use scaps to represent model error and initial condition uncertainty then this is some this is something we're working on actively and then the diffusion is is a different that I think John might might be the lead that's completely different. So I would think of it as a different >> stochastic scheme or different uncertainty scheme entirely. >> Yeah. Yeah. Yeah. The one John's working on is a is injecting [clears throat] noise into into the latent space of the [clears throat] of the model at different scales. >> Part of fenn know it's a different model. Right now you're doing it in WX form, but you can do it accumulate, but they both have the same underlying architecture. It's not there's nothing special. It's it's fairly model agnostic the way it's set up. You just have to find once you train a model, you just fine-tune it with a stochastic component. Uh and you can get and you can calibrate your model that way. And and you can also get a latent space where you can do interpolations between say ensemble members and your lat space and generate additional ensemble members. So So John's writing a paper up on this now. uh it there's some overlap with some of these techniques that like like uh so like ECW had the AIFS CRPS paper coming out late last year and then where where they basically do optimization on CRPS but then do they have lat space probabations to help generate the ensemble admittedly they also train this from scratch. Uh Google recently did did a version of this as well as Nvidia. Um and the big advantage of it is it's a lot faster to train than diffusion and and a lot faster to inference and it's easier to get a calibrated ensemble even with a small number of training ensemble members. So so and there and it's showing performance improvements and multiple groups have essentially done very similar things and they're all getting good result like significant improvements their forecast skill. So like like Google did ran their model in real time during hurricane season. And for a number of hurricanes, they were getting both improvements in track and intensity uh and were competitive with like the the consensus of all the other hurricane models. So, so there's there seems to be some it's doing some of the right things here basically. >> So, I'm going to take us back to the slides now. >> Um great discussion. I definitely think we should continue this um during the the question and answer, but I just want to finish up these slides here so we can we can get to that. Okay. So, everything that DJ's been talking about is the options within the credit platform. So, credit has integrated pre-processing for reanalysis, reforcast, and model data. It has a library of existing and new PyTorch neural network prediction architectures, scalable training and inference on ENAR HPC which is the biggest thing for me when I ran an emulator um analysis tools and plotting and then we've um recently deployed one of these models for forecasting in a weather um in a subseasonal forecasting challenge but also for handcasting and projections. So in terms of data sets they have data sets from A5, CAM, GES, KONUS 404 and CESM is also now there. Um some of the models we have Wexformer and Camulator are very similar architectures. Um Fushi um and Simudra are also included. Sudra is an ocean emulator. Um then there's the physics constraints and then these processing options so that ensemble generation is is there um vertical interpolation, regretting um diagnostics, all that type of stuff. Um this is an awesome platform that you can use. Um DJ is the lead of the miles group which is creating this credit platform. Um and if you have specific questions about that, I've used it. Can't say I'm an expert in it. Um, but I can help as along with DJ and, um, Charlie and, um, the the Miles team. So, um, in terms of how I used it, as someone who's never really worked in the emulator space, um, I was interested in subseasonal time scales. So, I wanted to create an emulator that incorporated, um, the ocean as a prognostic variable. So, it's input and output by the model and also land variables. So basically I took camulator but I trained it with SST soil moisture and surface temperature as input and output into the model. And this was a relatively simple um thing to do because the whole architecture was set up. There's a config file that you edit what your input and output var and your forcing variables are. So you basically just edit that, give it the associated files, and you're able to do all of this process with very little hands-on on the actual emulator portion. And then there's also um this whole section on grabbing GPUs and how much memory you need and all these things that can be that's all incorporated within the credit framework. So I was able to run sub CES emulator, which is what we're calling it. Um, but I'm willing to take names um for the subseasonal version of CES emulator. Um, and I am actually working with John on um ensemble generation with subc emulator. So, what we we're actually submitting this to the AI weather quest weekly. Um, well, I should say Charlie is really submitting this to the AI weather quest weekly. Thanks, Charlie. um and we have 341 members and we're getting this from the gas initial conditions and then we're also perturbing that 11 times each of those 11 times to get the 341 members using an the the typical S2S um perturbation method. Um but like we've already talked about there are other ways and we're going to explore um this latent space perturbation option um and maybe some others in terms of how how well this does at quantifying uncertainty in the forecast. Um, I don't know anything about this, but there's also a regional AI weather model that's been developed um to support fast dynamic downscaling and uh near real-time high resolution regional forecast. I'll direct you to Kyle Shaw and DJ for questions on this, but I also just wanted to let you know that this is also happening in the credit platform. Um, and then lastly, some ongoing challenges in the emulator space, very broad. How do we actually evaluate these emulators in a systematic way? Um to evaluate if they're even representing physical processes well um uh and doing this in a systematic approach. There's the fast versus slow time scale processes. Um the climate time on climate scales we have the ocean, we have the atmosphere, we have the stratosphere. These all happen at different time scales. and how to incorporate each of these time scales within your training um is an open question. I believe DJ is working on this in terms of um incorporating different um portions of the loss function for fast versus slow um time scales. And then there's this big problem of out of sample fidelity. If you don't train on a future warmer climate, can it actually represent a future warmer climate? And that's all I have. So