OpenStack Ops Radio Hour - August 28 2026

let us know when it starts and I'll recap and then we can just chat. >> It just started. >> Okay. Uh welcome everyone who just joined. Um we were just chatting before the recording started. Um this might be a brief one because uh a lot of the community is uh still on vacation, coming back from vacation or going through their inbox after vacation. Um but we're joined by Rickard. Ricard who was mentioning that uh last time uh he had issues running Windows on OpenStack and it turns out to be AMD CPU bugs and you said that you had to have uh the latest microode as well, right? >> Very latest microode and you have to turn off the HV TLB flush uh enlightenment for now. >> That's good information for my firm. We are testing um Zen 5, but um we've already worked out that if you if you if you tune your CPU flags to be optimal for performance on one of the vendors, you then can't live migrate to the other vendor. Do you have mixed CPU in the in the hypervisor fleet in your deployment card? >> Not yet. Uh so we are running an entirely new cluster for this and then we're going to migrate in a whole a whole bunch of um slightly older hypervisors and at that point we'll keep them in a separate aggregate. I think >> that's very smart. Yeah. Um so we are thinking separate aggregates or even possibly separate uh like regions >> so that uh there's really no question about them live migrating. Um so we use nested virtualization flags and we use hardware accelerated htable flags. >> Uh and both of those are not the same flag between the vendors unfortunately. So they just there's no chance of them live migrating. And if you disabled those you could have a lowest common denominator virtual machine CPU flag set that would live migrate but would have worse performance. out. They've managed to fracture the binary compatible x86 >> world. >> Congratulations, guys. Um, >> very slow clap. >> I'm just trying to get this down because maybe this is useful information. And then um yeah, so the other thing you said was uh server cost went from I think you said 60 to 140 or something. >> 41 to 170. >> Yeah. Um >> yeah, >> we were having a similar experience. Um uh a dense storage server uh not a hypervisor was quoted at a 10x price of what it was a year ago for us. >> Oh, those have um 22 NVMEs. Um and so it's not just CPU. Well, not just RAM, but also flash memory. Um, it's just um totally insane. Um, so we're not decomming anything anymore. We used to have like a four-year hardware life cycle, and now it's um at least 88 months is the rule. So, >> that that makes perfect sense. I suspect we'll be in a very similar mood. Uh so we had a couple of uh people join. Welcome Allison. Welcome Jeremy known as Fungy. Um we weren't quite sure uh what the attendance would be like. Um so um we kind of just dove in because it might have just been a couple of us, but uh you know the agenda is pretty open. We just covered um the live migration stuff because Rickard had a follow-up. he found the issues that were stopping Windows live um Windows working well on OpenStack last time. So that's dutifully recorded here and then we were talking about supply chain u but we didn't actually start at the top which is the contributor's guide and then the open in for live um build did you want to comment on those? >> Yes. Um so for um our current audience and also those who are watching it offline uh or after the uh after the meeting the um operators contributor guides the link is in our um group etherpad it got a little bit of update. So now it has um up-to-date information about the matrix channel where everyone can join to have like a semi-synchronous chat communication with other operators who are in the channel and it also now mentions the ops radio hour. So um there are still more things that can be updated in the guide. So if anyone is interested either um test things out and you can propose a patch yourself or you can also hop in any of these communication channels or the OpenStack discuss mailing list to let us know and then um eventually someone will get there to make those updates as well. So I'm hoping that this will help with the visibility of the ops radio hour and we also have a little bit of work. There is um a meetings page that OpenStack has the community maintains. So we are trying to see how we can add the ops radio hour to to that page as well to keep that one up to date. Also, we ran into a little bit of challenges with the schema definition for the YL files that that particular website web page is using. So, there's a little bit of delay on fixing that, but we are working on that one also. And in terms of visibility, I also put opening for live um on the agenda for today. And the opening for live um is a uh as the name suggests a live show. Um we've been running it mainly on Thursdays and we are chatting about any topics from the uh the open infra ecosystem um and any news items and case studies and discussions that are relevant for um opening for projects and the ecosystem around them. And um open it for life recently turned into a podcast and it could also feature some conversations that are ops related. So maybe there could be some opening for life episodes that are a fusion between ops radio hour and opening for life. Um, but I would also like to give the the mic to Allison because she is um organizing these shows so she can describe it much better than what job I just did. No, >> I think that was really good. I think the thing that I wanted to bring to this group is that you know I know y'all have like a lot of like informal discussions around specific topics and I think that for ones that are particularly um high priority for different operators I know like in the past it's been upgrades um I think live migration is another one that's been really popular but as if there's folks um who have something that they want to talk about we're trying to make it as discussionoriented as possible versus sometimes it became more presentationorient oriented which um was not necessarily as popular with the audiences that we were attracting but also it was just not as engaging with the live audiences um so I definitely want that to be seen as a channel for operators um and so that they can kind of come in and it can be one person we have two hosts that will um be the recurring hosts of the show that are members of the foundation staff Kendall Nelson and Wes Wilson and um so it can be one person. It can be a couple people, but really if there's a topic that deserves some discussion, I think that that would be a little bit more of a formalized setting, then um that can be an extension of this where it's a little bit more uh it's a little less casual, but it it can still get conversations in front of other OpenStack operators as well as other Open Infra operators that might so it might be relevant for other technologies. Um, but I think it's a really great opportunity and operator deep dives in terms of like having a user on that talks about a little bit more about their environment has always been very popular. So, um, we also welcome that where it's a bit more of an in-depth case study style um, discussion but still still not a presentation but something where folks can answer questions or go a little bit more in depth in what they're doing with a particular technology um, is something we're definitely interested in as well. Um, so I can drop the um form where we're collecting topics into the ether pad, but I definitely want um this group to feel like this is a a channel for them or channel for y'all to kind of come talk, discuss different topics um and kind of be part of that um that program. So really a channel to increase the group's visibility further rather than something to with what we are trying to also achieve with the offs radio hour as we talked about bringing in some guests and have it like a half radio show and and half conversational about any ongoing challenges that that folks have. So it's it's rather an extension to all those plans and ideas. >> That sounds good. I'd give it a go. um especially if it's a discussion format, you know, sort of drive debate about the things facing us. >> Um so I put a couple of things on here. Um I've been discussing with Ilico um we have transformed the performance of live migration on our cloud and um I did talk to our subject matter experts on this and invited them to join ops radio. Fortunately, today didn't work. Otherwise, not only would they be here, but I would have I would have boosted this, you know, session in terms of getting people to attend who are interested, but they will do it next month, all being well. Um, and I had to sort of, you know, um, help them get get over themselves because they're like, "Oh, it's very modest. Oh, it's very it's very specific to our environment." I said, "That's not really true, actually. It's it's a it's a huge change." and twothirds of the uh areas are I think universal. So a big one has been just tuning your your hypervisor operating system. So typically Linux that's um that makes huge changes and we've had to update the kernel and to change some settings those can be shared obviously. Second thing is some patches have been submitted upstream to OpenStack and more in flight. Um it was um rather conservative about some things. Um you know to to when the VM goes live in its new location um we have that down to a median of about uh two 2 point something seconds of um network blip with no throttling in most cases. So um almost all the workloads that are on the cloud are you know happy to be live migrated without interruption. They don't disconnect. They don't break their own SLOs's you know their service level. Um it wasn't that way when we started using it and we're doing thousands of live migrations per day now. So that's um I assume that is of interest. I think other other ops radios people said yeah that would be good to actually hear like in detail. >> Super interesting. Absolutely. Um >> those are all numbered. >> I would love to >> feel free to plus one there. Um it just helps uh to have some evidence rather than just me saying, "Yeah, sure. There's people are interested." Second thing, um I also just off the top of my head mentioned coke. I've been busy this week. We're getting our cloud product certified for Swift, which is a financial network that has strong requirements for um how you manage anything that impacts that. Um so if people are interested I'm actually more I'm not certain about this but if people are interested I can talk about it and the thing is that um swift is asking for many things that uh you may need to do for Dora as well. Um in our case we are deemed a critical infrastructure provider. Um maybe lots of open stacks aren't in that boat but um you know it's almost like if you succeed you may get there. Um and the stuff isn't um mysterious or magic. It's things like um patching your machines, having support from hardware vendors, having um a paper trail for how people and well having lockdown access to the machines that host swift related things and having a paper trail like who approved it. Is your access logged? Is it is it time limited? Um so um I I'll put that there and you know when we start discussing the next ops radio hour which will be back in prime time you know um maybe we can have a brief talk about that. I see some uh yeah I see some capture of this at the bottom. Thank you. Um that would be very interesting for us. We we also run uh swift and I could ask someone from our our side who uh who deals with that to join as well. Yeah, it would be great to make it a discussion. These are always discussions. The purpose is for it to be a chat, not a presentation. That's why it's not slideware in front of you. It's a it's a doc that you can just type into. Um, so are you actually going to be um audited or is it just something that might happen? >> We we are yeah every year. So it's an arduous process. >> Yeah. So we had um some things were non-standard inside Bloomberg. the way we accessed our machines for instance um and we've um largely just consolidated with um other thing other infrastructure inside Bloomberg that exists and already passes such audits. So for instance um access control, password storage, secret storage um logging um you know there is actually um a cyber security operation inside Bloomberg that looks at you know anomalous patterns and things and we're shipping all our logs to them. Um so we've kind of become very um very normal. It's just that um our product is the the biggest um set of machines at Bloomberg. So you know um it's a lot of load. So you have to be careful when you're on on boarding to the the metrics portal or the you know our syslog endpoint that you don't just crush them >> you know because typically there's some team like yeah we have we have eight servers you know we have 10 servers then we're like we have uh you know 8,000 but it's good to hear that other people are succeeding in getting uh Swift certified. Last year it was um it's kind of a difficult one. We had to do some nifty footwork. This year we're much more just, you know, documenting what we do. You know, here's a screenshot of our release process. You know, it goes from the uh the contents of a release to the individual chain sets to the actual code diffs and that's tracked in version releases and you can see which one which deployments they've gone to. Um we were already doing that but um we got the developers to be a bit more orderly. So you can always go from you know the the like business requirement to the code changes and then to which machines did it get to without fail so that there's no mystery. Um that was important and um interestingly um there's some things that are um there's some tension there because they have for instance requirements about um support. Well, we're going to have to keep some machines longer than the hardware vendors will support it because we can't afford to refresh the fleet that fast. So I don't know how that's going to end up if the auditors say, "Oh, this machine's too old." Or like, "What do you want us to do?" >> Yeah. Um so I think we have um I think old machines go from vendor support to some kind of third party thing but it's not the same. So >> good just found >> to be quite reason sorry I think we found it quite reasonable once you engage in a discussion about why some things are are slightly different from the spec dogs they will write it down they will consider it come back to you so but it the process took quite a long time. Yeah, just on a on a different topic entirely. Um the Epic CPUs you're using 9575 FS other than the the Windows issue and CPU bugs. Um have they have they been good for you? We haven't actually used them in production. >> Yes, they've been amazing. Um I we couldn't believe the performance numbers we were getting when we first benchmarked them. They are outstanding. Uh we did um benchmark I don't know if they're quite I don't I can't I can't remember how to decode the epic part numbers but we've got some Zen fives for test and uh we found the performance sky high but the memory latency not as good as the uh the Intel Xeons. Do you >> can you share anything like that? >> I don't actually know. So we we benchmark by running a lot of our internal like batch workloads on them. Uh and I they're mostly streaming. So I think maybe the we wouldn't have had the memory latency u issues. Okay, that's interesting. I may go back to look at that. >> So um Bloomberg is in the business of uh being you know uh distributing market data among other things. So um actual uh market price updates, ticks >> flow through um so there's a lot of latency sensitive workloads but also news. Bloomberg delivers news to the Bloomberg terminal um you know with um as much speed as it can. So if they can beat Reuters then they win. Um so we have some teams who measure literally like how long um you know an individual request takes to be encued for a thread to process it. Um so um slight variances in latency across CPUs is very painful for them and that's going to be a huge challenge because when we keep machines for more than four years you know the variance across our fleet is going to be huge and um the app team say to us oh you know I have the all these VMs but this one runs slow you need to move that to a fast CPU and and our answer is you don't get to choose actually to fix this we're going to make them all equally slow which they won't like. >> Yeah, we we had exactly the same. Our equities team when they got the new AMD CPUs refused to run anything else. They were very demanding. Um we uh we fortunately most of our fleet is uh single socket. So we don't have a a really painful NUMA issue but the older machines are dual socket. So occasionally a VM can have tra memory traffic across the motherboard basically and that's exceedingly slow. So there are edge cases where memory latency is horrible and we were thinking those machines were on their way out and now because of supply chain they're not. So we may have to do swaps like moving old machines from the prod network to the dev prod network. Logically speaking, they're actually they're actually all in the pro prod network, but the ones serving the prod workloads are different from the ones serving the dev workloads. So, um, dev may actually be the the place that gets squeezed first from this supply chain catastrophe. >> Yeah. >> Which they also won't like, but um, you know, um, >> yeah, >> we're in the area of hard choices right now for this. >> Absolutely. I think a lot of our hardware that we would have decommed is going to go into our our cluster compute uh and that will yeah increase the variance there make people unhappy but I think that's what we have to do. >> What other um customer needs are you sort of uh fielding right now? >> It's going to be a lot wider. So we we tended to have mostly dealt with our um uh systematic uh hedge funds uh but uh we will deal with all the discretionary uh teams as well now uh and I think they will have quite different uh different patterns of use. So I think we're about to discover what's going to be different. we um we traditionally serve you know the sort of uh I I guess the 80% easier workloads you know and the 20% most challenging workloads by bare metal hardware >> so for instance that database team or there's multiple database teams but those kinds of things but with supply chain um the business is looking us to solve very very um highcale IO and all of our storage is currently on seph Great. You know, we have huge SE clusters and the VMs all have, you know, durable storage where they don't they don't lose data and it's always shared. So when we move VMs around, they're just always pointing to the SEC cluster. But we can't get the IOPS for something like an Oracle SQL server. >> So we're looking into, you know, locally attached storage for VMs, which we actually had many years ago and eliminated, but has coming back. Um, that's a challenge. Is that something you deal with? >> That's very interesting. No, we we run a lot of flash uh pure flash arrays for for things like Oracle. Um and I hadn't actually considered local attached storage. Um but that's very intriguing. >> So, um this is just something that I'm supposed to be, you know, making recommendations and doing some research in. But, uh I did find a paper. I'm not going to try and dig it up right now, but IBM has a paper about um locally attached storage for VMs in OpenStack uh exposed via you know QM U KVM probably maybe the stack you're using and um you know there's there's lots of different options you can have Nova ephemeral storage which just uses the hypervisor file system um but there's lots of layers it's kind of slow and the other end you can have PCI pass through where the VM just sees the actual hardware and does it itself. >> Oh yeah. >> But that's very um that's very inflexible because the VM actually sees the PCI bus and it owns the entire device. So if you have an 8 terabyte NVME, the VM gets 8 terabytes. You can't have four VMs with two terabytes >> of course. Yeah. Um but the interesting thing I wanted to share um since you're potentially interested in this is um the IBM paper reports um pass through of NVME namespaces. Uh >> actually know what NVME name spaces are. >> So um an NVME device can have logical partitions called namespaces and they can be exposed as different um virtually virtual you know block devices by vert.io. So you you can split an NVME device into multiple slices. And uh it even says that um they can each have their own encryption keys. And so um when you delete a VM, instead of zeroing out the storage to make sure that there's no data retention, you can actually just have the device drop the encryption keys and then it's um it's gone forever. So um I will uh I will share it on this ether pad after this meeting. That's that's >> that's something that seems to be somewhere in between you know it's like the Goldilocks you know uh porridge u baby bear you know mama bear's porridge was just right or whatever um because um you know just no ephemeral storage in the hypervisor's own file system. We used to do that but it's not particularly fast. What people want is most of bare metal performance but they want VMs. So if we could pass through NVME name spaces with I think IBM reports 80% of native performance but we can still take a big machine with a lot of storage and slice it into thin slices for lots of VMs then um that looks like the winner but um not easy to get from you know white paper to in the product. So that's something we're looking at. uh we don't have any um storage appliances in the in the cloud. The legacy uh non-cloudy infrastructure uses um sand and there's all kinds of vendor stuff in there but it's so expensive because we just we just use so much storage that uh they're trying to get away from that which is why we like seph and blue is actually um >> on the board now and actually really sort of a a major part of the se community at the high level. But um you know even if even if you tune SE to the max the truth is that your traffic goes over the network and then it goes again over the network to the number of times that you do replication. >> So two times well the minimum of 1.5 if it's um eraser coding but we don't do that. We do replicas. So we do four replicas. So you've got basically at least four times the number of writes over your precious data center network. um that just doesn't scale. Our our our network is the most overs subscribed. So even if we were the most leading geniuses at SE in the world, you just it can't do all your IOPS for all of you know because we have punishing workloads in some of the applications that previously just got to buy all hardware. Uh and now uh that is very challenging. So >> I totally understand that our compute crushing our storage is a frequent thing causing unhappiness. Yeah. We we re I I'll actually share that we recently um had a failed engagement. So we were onboarding the Spark um people you know sort of Hadoop replacement thing and um you know they were they were on track to just use up all of the bandwidth of a of a large SE cluster. So we we pushed them back and said um we'll we'll come back to you on that because the business wants everyone on cloud because get better utilization and uh better um oper operationalization you know because the machines don't don't go down because of a dim failure. They just get moved around and it's all hunky dory. But physics means that if they're if their compute is enormous, their RAM is enormous or their IO is enormous. It's harder and harder to just just throw in the cloud. You actually have to do special things. Absolutely. >> You've been happy with um you said pure uh you said pure array. >> Yeah. Yeah. Pure flash arrays for block storage and uh uh vast for object and file storage. Um it yeah it's been good. Uh the vast especially I quite like um it it's it can scale its kind of compute front end independently of its uh storage. So you can kind of like optimize IOPS you can you can uh you can serve independently uh which is very nice. Um and we have run some um I I don't want to call it a hack because it's working really well but serving um kind of block devices as NFS files under loop back we found this actually worked really well for some use cases. >> Interesting. So we um there is kind of a a pollution of the new stuff. The uh the new cloud-based uh infrastructure does allow the uh the VMs on there to go talk to the old NFS servers like NFS appliances. >> And I think that's just that they they treated that as like a a wrinkle like an old thing that has to go away but it just isn't going away. So, um, yeah, >> native support inside your cloud might be that might be a good trick for us to, uh, help retire those, you know, um, I'm not going to I'm not supposed to name vendor names too much, but um, yeah. Um, having this whole modern cloud, but it still depends on the old overworked NFS servers is kind of unfortunate. So we are on the small side attendance wise and I'm always conscious that you know meetings shouldn't run on when people come when we come to the end. So are there more topics we can talk about usefully today? Ricard, did you have other things you wanted to to share? >> No, I I've just been enthused by the discussion. So I'm good. We also have loose joined. Um I don't know if loose you have I'm probably pronouncing the name wrong but if you have any topics for today that you wanted to bring up. >> I don't know if he's talking. He says oh all good thanks. Just enjoying the wisdom. He's in the chat window. >> Oh yeah I see it now. Yes. Thank you. >> Yeah. >> Um yeah I didn't really have any more topics. Um just trying to keep it organic as always. Um those are the best discussions. I don't know if you all have any um like topic ideas for upcoming calls that could help um reach out to folks who could potentially talk um to those. I I also caught I think Chris you mentioned that you needed to do a little bit of um I don't know um boosting people's egos in terms of this topic is important enough to talk about it. So um I don't know but what um what we could do to um like relay that to our wider not just audience but but the the folks who are joining these calls that that the the things that they do are important and can be informative to to others so that they shouldn't treat them as oh it was just a little thing that took us three weeks to fix but it's not important. Um, so I I don't know if you anyone has any good tips for for that, how how we could um encourage people a little more to uh to talk about what they do if they can. >> I mean uh I do try and do that. I have brought along previously another Bloomberg colleague of mine who talked about you know high availability OpenStack architectures. That was about three sessions ago, but it will have been recorded and I think next month I'm very very optimistic that I can bring in live migration um experts. But uh I'm thinking that the the topic of um fast storage for VMs might itself be juicy for the community. If we said like you know hey um do you have a cloud that now needs to host more challenging workloads? Are you looking at you know ephemeral storage PCI pass through uh or maybe fancy uh you know commercial block storage devices um you know let's talk about it if you have experience share it if you if you're new to this challenge you know there's others of us um getting into it should we say so um that's just an idea we could try for the you know two meetings hence do you think I I love that I just noted it down. >> Okay, so next week, next week, next month, I I will try and bring our people, it's not all they do, of course, but our basically our cloud optimization experts to talk about live migration and I, as you say, I had to tell them, no, I think this is dynamite. We should share. And then the one after that I I think we can propose the topic of how do you folks do high scale you know many you know high IOPS um high bandwidth uh storage to VMs but I don't have someone to bring to that because we don't do that we want to do it right so I'm I just want to be very clear that sometimes I can I can bring the expertise and sometimes I'm looking for it so we should probably drive that discussion hard on open stack discuss matrix and the other is Elico because >> you know maybe we can get someone who knows I mean I have talked very briefly with Kevin Carter I think from Rackspace >> they do they do stuff like that you know they're they're enormous and they're recommitting back to not only OpenStack but um you know upstreaming their changes and you know being more um unified with the rest of OpenStack. So um maybe we can attract that attention. I don't know. I I have an email chain with me where he was going to get back to me about what they're doing and then it just I'm sure it uh he got overlooked by other things, but um maybe I can pick that up. >> Oh, you you already asked him, then I'll just I'll give the action item to you. I I thought to reach out, but if you are already in a mail thread, then I'll I'll let you that up. If you don't succeed, then I can still help. >> I was discussing it just in the sense of swapping information privately, you know, because he's he's ahead of us. But if if if he thinks it'd be more worth his time to spend an hour talking to the community about how they do, you know, top-end storage stuff for VMs in OpenStack, that might be great. So I can definitely try and re like reawaken that through. He's a pretty senior guy. I think he's a CT CTO or something now. So I can't promise anything obviously. >> No, he's also great. I think I think he's usually open to these kind of things. So I'm I'm hoping. Um, and yeah, if you if you don't have the time to reach back out, just let me know and I can I can also um ask him. >> Yeah, it's not so much my time in this case. It's it's just um I don't know presuming his time is not something I'm going to do, but we can try. Um the other the only last thing I think for this meeting is um we don't have uh we haven't picked a date for September. Uh >> so the last is the 25th. 25th. Uh, I'll just uh check my calendar. I'm sure that's okay. Should be fine. >> Okay, then I'll lock it down. >> Oh, I see Ricard saying um So I I think we're aiming for uh end of October for the storage talk. So some time to get people to agree to just chat in general sense about what techniques work. That would be fantastic. >> Yeah. >> Um you know buy something might be the answer. We're not against that at Bloom. Um we've gone a long way with pure open source you know software defined storage software defined networking. Um but if we can put remaining workloads in our cloud by hooking up pure flash array or whatever it is um that might be the answer. So hearing that might be good. My boss has previously been um against that. But um the constraints on us now are um outlandish. >> It's sort of like you know here's a here's a five pound bag. Here's not just 10 pounds of stuff but you know there's another 20 coming. So >> yeah, >> simultane storage we we might be looking at SE. So there's some, you know, cross, you know, >> seating there. >> Oh, we do a ton of SE. Seph is the block storage for um more than 100,000 VMs at Bloomberg on 8,000 hypervisors and 15 deployments. But it's also our object store. Um to be perfectly frank and forthcoming, the object store role for Seph is a bit less successful because the web tier that converts block storage into you know HTTPS uh verbs has needs a lot more investment to be as good as the rest of Seph. Um but that is still what we're running and in fact we've assimilated competing object stores. There was um something started by an app team based on um Mo which was sort of a open- source but also commercial thing where they kind of just decided that the open source bit wasn't for real. >> Um all those customers are now on our sethbased S3 compliant object store. So we we have so much SE it's crazy. Um so I'd be happy happy to talk about that. uh we could even get SE experts although I can't promise because I haven't even mentioned this ops radio to them but we have some people we have some upstream contributions and not just like minor things like they actually contributed a fix to um multi-sight replication which was not reliable kind of important that you know if you're relying on data replication across site that it actually keeps your data right so um you know I think we're finding some topics that might get people, you know, uh excited to join in. Right. So, >> I love it. >> Yeah. Okay. Um so, we have next next uh ops radio date picked out. I'll put it on my calendar. I will Rickard, I will share the the research paper that I found. It's just a white paper PDF you can download. >> Um I'll put it in this thing. Um if you or if you want, I can send it directly to you. Just email me. You don't have to share it on this recorded public thing, but my work email is corgan2 bloomberg.net. You're welcome to connect with me that way. Um, but I'll put it here for everyone to, you know, and that's that's an area of research for us. But unless there are any new topics, um, or any final questions, I think we should, you know, do the good, uh, meeting organizer thing and, uh, close for now and, uh, see you next month. Any final thoughts? Okay, Eldico, you want to close out the recording and we'll say goodbye for now. >> Yes. >> See you in the autumn. >> I I will do that. It was a great call. Thank you all. And I'll send out the recap and we'll try to encourage folks to uh sign up for these topics to to talk about them and add some more. >> Perfect. Thank you so much. Thanks for always keeping things going. >> Happy to. Thank you. Bye folks. >> Say out