OpenStack Ops Radio Hour - June 26 2026

Now we are >> okay. Thank you. Welcome everyone. It's a small gathering. This is the OpenStack operators ops radio. If you haven't joined before um think of it as a zoom meeting once a month where we just get together and chat about um OpenStack and being operators of OpenStack. Um the people who are joining today seem to have terrible connection problems. My um and I know I'm repeating myself, but for the recording, my experience is I cannot be on my my company's corporate VPN and stay reliably connected to Meepad. It connects and disconnects. It connects disconnects. It's awful. So, I have to drop that and I tell my colleagues, hey, I'm not connected for the following hour. Um so, that may be a a tip. Um, the way this meeting works is we have a ether pad and if you haven't found it, if you click on the dot dot dot menu at the bottom of the screen, it says more actions. Uh, you can do open shared document. So, if you aren't seeing an ether pad in the in in the in your view, that's the agenda. And it also has um you know um the uh the host, which is me, the attendees and the topics. and then you can just type into it. So what we do is we work on this doc document u to try and capture some material, some questions, some comments, some some shared info, hints and tips. Uh and you can probably see I thinko is actually writing that the tip that corporate networks don't work with this very well. Um I like the fact that we tried to use Meatpad because it's infrastructure hosted by the open infra foundation but um yeah it's not Zoom. Um you know big companies have special arrangements with Zoom to make it work well and scale and stay reliable and Meepad is just over the over the internet. So it's prone prone to these things. Anyway, the agenda today um pretty light. Um it's summer and uh people are probably busy. Um here in New York it's the last day of school so people probably traveling that kind of thing. U but the topic that did come up is the operator's contributor guide. Um what does still need updating and what other action items does the group want to take? But then there's um uh anything else that people want to discuss. And then um the question of the next meeting. Um I haven't looked at July 31st. Um I do think that we could try on you know in a month's time. Um so the context for the operator's guide which is the first agenda item is it's out of date and um people have looked at it and said oh it seems to be you know needing an update. So I foolishly volunteered to update it but we need some sort of requests some content some I guess bug reports. One thing is it doesn't mention ops radio I believe and it emphasizes um mailing this to IRC. IRC is um difficult for many um people in the community these days. For example my employer it's um real IRC protocol is banned entirely. We're not allowed. And even a website that wraps IRC often triggers the content protection agent. So it's really hostile and it's just kind of annoying. Um the ones that are lively and work the mailing list I think has become preeminent again and uh matrix um there's a special channel for operators there. uh but an eternal topic is that uh people who are doing OpenStack hang out in half a dozen other places former Twitter, Reddit, LinkedIn, Slack. Um there's other suggestions for other chat technologies. There was one last month, I forget the name. Um but I I think probably the the simplest thing these days is probably the mailing list. OpenStack discuss. Hey Mark. Um I think you're one of the people who was dropping in and sorry joining and then disconnecting. So it's hard for us to know whether you've got anything out of this meeting yet. Um >> yeah, I'm on my personal laptop. It's working now. >> Okay. Um so it's not clear whether we have quorum for this meeting. So I'm just I'm just kind of uh trying to see if we if we can get a discussion going. If if we don't have enough today, we understand that it's summer. It's kind of common that these things kind of peter out. Um, we don't have to take the whole hour. I'm very conscious that you should always end meetings when when you can. Um, but uh the the suggested topic this week was going over this doc. I think the link is actually uh somewhere deeper in the ether pad, but I'm not going to spelunk and find it. Does anyone have comments on the doc? Like would it be valuable if it was up to date? Um, one of the things that we're hoping to do is suggest that people who are new to OpenStack do get on the mail list and do get on matrix and we actually kind of coales somewhat because if there's 20 tiny communities, it's not as effective as a couple larger communities, I would say. Um, I have to kind of say this repeatedly. This isn't a this isn't a lecture or a presentation. And I'm not presenting slides. It's meant to be a discussion. I know some of you have have audio problems or connection problems, but like uh if it this isn't going to work if it's just me speaking for forever. >> Let me ask a question from the from the group. Um is this the first ops radio hour to anyone? You can Okay, Ricard. Yes. Um I assume maybe Mark 2 then if if you're with the same same company. Um, >> yeah, first for me. >> Awesome. Well, welcome. Um, I'm curious like what stage are you in with with OpenStack? Have you already deployed and running it or or um are you looking into deploying OpenStack in smaller or larger scale? So be we've been running it for years but um we're massively expanding the scale of our open stack platform as we're VMware. >> How is your how is your OpenStack experience so far? Like is it running smoothly? like um you found this meeting and you came to ask questions about something that doesn't work well or want to connect with others who are also running OpenStack but you are currently not experiencing challenges. >> So our [snorts] biggest challenge right now it's running smoothly with Linux VMs. Um this our first time trying to run Windows VMs on OpenStack and we're not having fairy That's interesting. We actually had some learning with uh Windows on um OpenStack ourselves. I could actually talk a couple of things about that if you if you like. I'm just trying to get a note down about what we're actually discussing because it seems like the contributer guide doesn't have much meat on the bone for this discussion, but just generally like um getting into OpenStack and Windows is a great topic. Um so it's Richard and Mark Uh you can you can correct my language if that's not a summary you you agree with. So uh our hypervisor is KVM and um you know you you deploy OpenStack and you run Linux on KVM great right everything works it's just fantastic then you put windows on it sort of works but it's kind of sucky and it's weird right >> so what we found is [snorts] uh the Windows team had uploaded images to OpenStack and they worked but they weren't tagged as being of with metadata windows I couldn't tell you the exact thing for I can get it if someone wants to email me. I I'm happy to share. So, KVM has a different set of defaults that are good for Windows, but it has to be informed that this VM is Windows. >> Um, so that was a big step forward for us. Um, the second thing was um the Windows admins like to keep their Windows machines in uh set with hardware clock to local time which caused problems with things like logging and other things. Um and we were able to verify with them uh a registry change that told Windows hardware time is UTC which is our corporate standard and that worked well after some fiddling around. Um those are the two biggest breakthroughs we had with Windows and then um we did have a problem with um Go ahead. I think someone was asking a question or a comment. Go ahead. It's open. It's not a presentation. Speak up. Okay. Maybe that maybe not then. Um anyway, so we run um a fair amount of Windows on OpenStack successfully now on KVM. Um and um we've successfully got past a few other issues that were more minor. One of them was that um we moved to jumbo frames and for whatever reason Windows didn't notice that or didn't negotiate that. So it wasn't correctly setting the right MTU which is very important if you're going to have good performance on modern networks. You want a 9K MTU. Um so those are some of the things that we um stumbled across and fixed uh and now it runs it runs very well. In fact, we have um sort of voluntary physical to virtual migrations going on because we are in more locations than the Windows team is able to buy physicals. So, we actually became preferred after. But one of the things we had to do is um and this is a I guess human factors thing. We had to we had to actually reach out to the Windows team and partner with them. What we had before and this could be corporate pathology unique to my employer but I kind of doubt it is we had the cloud and we had the Windows people any problem the business people would be territorial so there's a problem with this server a VM running Windows on OpenStack there's a problem with the server so the Windows team is like oh it's probably some cloud stuff ask the cloud team to fix it the cloud business owner is like well has the Windows people investigated this first they're the first line uh escalation, send it back to them and tickets would ping pong and nothing was done. And so what we did was we set up a a working group. They said we have this issue and I arranged for them to have direct access to our cloud engineers and they just worked through um you know this this this ops radio hour is not really about how to manage teams and conflict but that was a bigger [clears throat] thing that was actually a bigger problem than any of the technical problems that I mentioned with the hypervisor and tagging and MTU and uh you know UTC setting. Um is any of that helpful? I'd love to hear, you know. Oh, two thumbs up. Wow. Okay. I'm glad we had super helpful. I'm just trying to not have a microphone. >> I really just can't make that out. I'm so sorry. It's like you're in a It's like you're in a force 10 gale. Like you've got a microphone standing on a yard in a storm. Like so. Um >> yeah, that's helpful. Definitely the um UTC something UT UTC thing is something we should check. Um, yeah, our current issue is we we migrated a bunch of Windows test machines, everything seemed fine, but then when we went to migrate rail workloads, we're getting a bunch of random BSODs. Um, and we haven't fully pinpointed the cause. Uh, part of it is we run Crowd Strike on the Windows VMs, which is not our choice, but we do and we're forced to. Um and in a lot of the crash dumps we've seen that crowd strike seems to be involved. Um but that's that's an ongoing issue we're trying to diagnose. I will say that um we used to be very um dated on our versions of you know our hypervisor platform which is Abuntu 22 our KVM QMU versions um and things things are better for every all guests uh now that we're on I think kernel 6.18 or something like that [snorts] um no idea what how you're running your open stack or whether that's an issue or a possibility to go up to date um do you know whether you are tagging the images so that KVM is aware and some I know that's thing with the mind. >> Yeah, we we do tag the images uh with with with with the OS type or whatever the the property is. And um >> yeah, we're running um OpenStack 2025.1 on Ubuntu 24. I'm not sure the kernel version. Um and we use cola anible to deploy >> and it is KVM presumably. >> Yep. >> Yeah. Okay. Um well that's actually newer than us. So you might have the opposite problem. So um yeah that's a that's a that was a key priority for our product the private cloud inside Bloomberg because um previously you know we we were known that Linux worked well in it but Windows had been bad and the Windows people were like oh it's just not really for us we need you know something else and we solved that. Um, so we partner with the uh very experienced Windows admins and it's very productive, but we haven't had the specific things. I don't think we've had a BB SOD for a while. One of the things I was trying to recall was we did have a problem with um I think SSD trim and Windows handling of block devices. Um, Windows and Linux have different views of how to drive block devices presented by, you know, KVM basically. Um, but that was several years ago now, so I can't share great info about it, but I can dig something up if if so what we saw was file system corruption and not a kernel crash. So, um, probably a different issue. >> Um, we do have crowd. >> Very interesting to know that as well. say again that >> that's very interesting to know that you also have crowd strike um and and we did we did find that our our machines were not very happy when we switched from vio store to uh vio uh but I don't think that makes a huge difference for our block storage on windows in particular >> um yeah we're all our storage well 95% of our storage is seth so Um we don't I don't think we match the storage configuration you have. We have a little bit of um light bits NVME over fabric. Um but I don't think that's going to really grow anymore. Um anyway, if um if any of the things I mentioned that we work through seem of interest, I can dig out more information. probably all on our Slack channel, something like that, or you know, PRs and >> yeah, I'd love to know what what metadata flags you you applied and then we can we can try those as well cuz yeah, we've been stuck for a little while. >> Okay. Um, cool. Well, I'm glad that we were able to uh, you know, crack open a topic and get something down. Um, so my corporate email is corgan2 bloomberg.net. I'm not going to put it in the ether pad, but you're welcome to email me. We can connect that way. Um, and then I will endeavor to dig out some some fixes that we did for Windows. Um, but uh yeah, we uh we are now u like the Borg assimilating all of the Windows infestations around the company. uh because we we made the effort to make our product good. But then of course uh a big vendor who I shall not name raised the prices for proprietary virtualization and now uh there's lots of people who wanted to carry on with um Windows on VMs the other way who now come to me. So um yeah, it's been a good experience. >> Probably the the very same reason is why we're expanding our OpenStack footprint quite significantly. Yeah. Yeah. Um, do you actually use any of the um, conversion tools from unnamed vendor to OpenStack? >> Um, I saw some presentations about that and they're actually quite good. >> Go ahead. >> And we're using a V jail Jailbreak. Um, >> yeah, I'm quite liking. We have a couple patches I'm hoping to get upstreamed, but uh, yeah, quite liking it. Yeah. And that if you're not familiar with it, that uses vert v2v in the background which is a more common tool. I'm just trying to get something down. I haven't heard of these things. Is B2B the the open- source conversions or toolkit? >> Yeah, it's a very like command line basic thing. And VGLER is a wrapper with like a UI UI and API around it that runs on Kubernetes. actually runs on K3s like that. It self deploys in OpenStack >> and >> very nice. Yeah. >> Uh the the company that makes it platform 9 I think they have their own OpenStack distribution that's you know like a paid thing but we do not use that. Um this is actually useful information for me. So I'm actually this is great. Um I still have um potential customers who come to me and um they're they're there they're there what they come to me is they say we'd like to use OpenStack but we don't want to become your customer. Do you have OpenStack kind of in a box that we can install and we're the admins um but we no longer have to pay that vendor. And I'm like, running OpenSack is is, you know, kind of complicated. No, we can't just give you our product dehydrated and you take it away and then you run it without our automation around it. They're very disappointed. So, you know, um maybe I can just say, you know, if you really don't want to come and use our actual cloud, maybe you can look into VJL break, you know, because I would like to help them um in whichever way possible to get off um the platform where the price rises are like, you know, in some cases 10 times, right? Um so I will look into V jailbreak. That's great. Thank you. Yeah, that's open source and free and yeah, we've had good luck with it for the most part >> and it's it's very user friendly. It's quite a big selling point for it. >> Um, okay. So, you're running um Windows and Linux or just Windows? >> Oh, both. Roughly roughly equal amounts. >> Yeah. Our old our old open stack which is is running walabe is 100% Linux. Um our new open stack yeah it's like a 50/50 mix or will be >> cool cool. It's very similar to what we're doing. We um we have um re andu guests and windows and uh the latest challenge is um a vendor image from Cisco. Um I'm naming them because I'm not badmouthing them. Um for um SMTP mail scanning um you know our mail scanning people say we want to use your cloud but we have to use this this image. We've decided this is the only way >> and uh it's net BSD >> and um we're actually um >> oh >> once bitt and twice shy now because their attitude is we loaded it in dev it runs so you're fine. You don't need to worry about it works on your cloud and but we know that if if this appliance is doing the mail scanning for the the entire Bloomberg, you know, enterprise and that breaks our incident support people will be like, well, how do we fix it? What's wrong with it? Is it the cloud? Get the cloud engineers on the line. Even if it's, you know, the middle of the night on a Saturday night, right? So, um, we're reluctant to pick up more mouths to feed in terms of different flavors or different OS families. So for us it's real, abuntu and windows and that's that's it. >> Um it's a shame because >> interesting for us because >> yeah there we have a whole bunch of things that are deployed as um I can't remember the name of it but it's like a like a package format for for VMware um >> OVA or something >> and and and yeah those are the ones. Yeah. [snorts] So we'll see how those go. We haven't we haven't dried those yet. Well, my my view is it's um like as as uh Riota said, do or do not. So, NetBSD, we support it or we don't. There's no such thing as, you know, allowing it on at their own risk because an app team, they're not actually taking the risk, right? If it's if it's running on our platform and then it breaks because we upgraded OpenStack and got a new >> KVM and there's some misunderstanding of how it views the network or storage. We broke it. They didn't break it. Right. So, um it's like gonna get the call. >> Yes. So, um we're having that discussion right now. Um you know, and people basically turn up with I've made all the decisions and the answer is your product has to be our platform with the following demands. And we were like, nope, sorry. Um not just like that, right? Like if the business decides um we do want to have free BSD support, we'll first of all support it non sealed vendor image right we'll just upload an image and have some tests does it boot does it consume storage does it do networking and then when we upgrade OpenStack we'll do those tests right um otherwise it's a it's a liability you know we don't have like open season upload your operating system on this cloud because when it when any operating system issue turns up, we are at least 50% on the hook. So, >> I'm not sure anyone asked the question that led to this speech, but I I'm interested in what other people are doing with OpenStack. Are they allowing any OS? You know, is it truly, you know, at your own risk, just buyer beware? Because that's not our experience, you know. No, I I think we would very much prefer to stick to a an unknown set of base images. We we'll see if we can uh we can hold that line. >> Yeah. One of our problems with Windows is our Windows team are well, it's not combined Windows Linux team, but most of the images are 2019. Um, and I don't know if the crashes are related to it being Windows Server 2019 or it's just, you know, a statistical thing [snorts] that most of the images we run are 20 Server 2019. Um, we would we we would like it if they upgraded to a newer version. So, one of the things we we did uh and um I'm not lecturing it just it used to be that people thought that oh we do we do all the cloud stuff but for Windows we just said look we really don't know right we don't know Windows we don't want to know Windows but what we did was we partnered with the Windows teams we said you are key partners of ours you will have image upload rights you will upload images you will test and then if you have any issues you have like open access to the engineers And you can have access to hardware test clusters that we have that are identical to the production spec hardware and they run in the data center. The only difference is they're not allowed to have production workloads. So you can upload windows there and you can do things even up to internal beta with everything representative right the the network firewall policy the um the uh even the the ACL's on the routers everything is legit right. So if it works here, we we've done a great job of d-risking for full production open set clusters and that's been extremely productive because they they have the confidence to kind of explore cloud and understand that uh it's it's really it's just like a different form of hardware, right? Like you you load them windows on it probably works. If it doesn't work, there's a bug. we might talk to the vendor or in our case, you know, they might ask us about storage performance or network issues, but [snorts] it's not so mysterious once we partner with them. Um, so now they're kind of um helping our product do a great job for the Windows users at Bloomberg, which is still, you know, huge. It's critical. So, um, that's how we kind of solved that. Um but we had to kind of take like a left turn because the the old way was butting heads you know cloud business sponsor Windows owner just we're like this you know that doesn't that doesn't help. uh any any more topics about any more to discuss about you know initial experience with with open sec group um I am conscious we have other people there's a question here Lucash about a specific bug and I think you've you've handled that in the uh in the chat somewhat is that yeah that's that's way outside and we don't even use OVN or call anible so it's lots I can comment I don't know if anyone on the call is working with cola anible or um is aware um like have looked into that um that bug. Yeah, I we grabbed the the patch link but I personally don't have information whether or not um like what the plans are in cola anvil. Does anyone else know? We do use cola anible and OBVN. So we would be interested in this. >> Okay. I mean what I usually do um two things. I write up a a quick summary of of these meetings and I send it out to the OpenStack discuss mailing list and I also post it on the um OpenStack Ops matrix channel. So those like the mailing list has a wider reach [snorts] so people can follow up there. If we don't have anyone here on the call specifically who's knowledgeable what the plans are, then the mailing list can be a good um followup um to see if if someone someone else knows or if multiple of you raise interest that yeah this would be great to have then figure out how to get to the implementation part and figuring out the um resolving the bug and and all those kind of things >> and the um the matrix channel could be um the matrix channel could be a good place to also discuss even the metadata and things that that you were talking about for the for the Windows WMS. Um, if there's anything that that's not too specific and you don't mind sharing on sharing it on the matrix channel or on the email uh mailing list, then others can benefit from the from the guidance and experience that you all have as well. >> I don't know, Lucas, I saw that you um that you just joined back. Um, I don't know if you have any other um things to say about the the BOG and the OVN BGP um agent support topic. I was just describing that um I'll send notifications and um recap to the mailing list and hoping that someone might jumps on the topic. Yeah, I would echo Elico. Um I agree because the purpose of this meeting is to try and get operators together to sort of elicit um you know collaboration between operators about their experience about running it. But when it comes to a specific request for this but this feature being added to this you know um OpenStack project um we don't necessarily have those people on this call. So uh I do think as Eldico said the the mailing list has the best reach these days. We we determined that really all of the other sort of splinter groups and all the other social media platforms are very small, very scattered. Um but there is also some attempt to say but you know we do like chat programs. Matrix is the one in that space. So I would take that question to one of those two places. Um but I will say that people people do review these uh recordings and then uh some conversations happen based on just issues raised in ops radio happen just spontaneously someone like I saw this question >> um raised in the ops radio and here's the here's the you know the the fix or here's the the release that that fix that feature will go out and that kind of thing. So >> I'm not [snorts] saying don't mention your your questions your your needs here. It's just that you know we don't actually have a caller Ansible representative here or you know a Nova compute expert you know that we don't have the developers here. Um I'm trying to sort of facilitate the operators community kind of reforming into an actual community that can actually communicate with the developers. Um, but this would not be the actual forum for deep questions on subsystems of OpenStack. Um, and I hope that made sense. I feel like I'm rabbiting on here. Um, >> can you hear me now? >> Yes. >> Oh, great. [snorts] Uh, >> loud and clear. >> Yeah, I'm quite new uh in OpenStack world. So BGP agent and and cola. Uh >> I'm afraid you dropped out again, Lucash there. Um we know we we heard half of that. Um you're saying you're quite new and then there was a a gap. Um, I don't know if you heard what I was saying, but um, Ilico had said that for very specific questions like this, you should probably go to the OpenStack discuss mailing list because that does get >> um, good interaction. There are lots lots of places where people discuss OpenStack. There's this, but there's also Slack, Reddit, Twitter, um, other chat platforms. But I think I think we're trying to emphasize matrix and the mailing list. Um I would take that there. Um although there is a chance that someone will see this and respond later. It has happened because people check these recordings and then you know there are people who said I can't make the meeting but I'll look at the recording after. But yeah if you need to get an answer for specifics like that mail list is best I would say. >> Okay. No thanks. Uh there's some chat going on here. I'm just reading it. Okay. Um, I'm not sure who Moose is, but uh um is this um I'm sorry, I'm trying to understand the chat retrospectively. >> No, it's just >> Oh, it's about the wiki being broken. >> Threads. Yes. uh if if anyone runs into the OpenStack wiki not loading or the the mailing list page is not loading momentarily. Um so the the background issue is that there are lots and lots and lots of AI crawlers on the web these days and they are bringing down all kinds of resources including open dev resources as well. the open death team is who's running the the wiki and some some other um resources that the communities um supported by open infra are using and so the team has implemented some guard rails but every now and then it's um the wiki and and other pages might still be slow or you run into some checks that think that you are also an AI crawler sometimes like my web browser definitely case every now and then seems to act like that. Um, so don't refresh pages too often or or too fast. Um, so don't be surprised. Um, just be be patient and try to load it again a little bit later if you if you run into a challenge. the team is aware but but AI crawlers really show up as um DDoS attack like we are talking about tens of thousands or hundreds of thousands um requests pouring in by the second kind of anyway some some kind of a crazy scale I don't know the exact numbers just trying to throw out something really big um but that's what's going on in the in the background it is unfortunate and um sadly it's not an easy challenge to solve. So that's um yeah. I also dug up the um mailing list um info looker. Uh drop this into the ether pad. So, I just dropped in the a link uh for the OpenStack discuss mailing list. Um if you haven't had that one yet, if you're not subscribed, um it is in my understanding the the mailing list is now running on Mailman 3, which is kind of a little bit of a fusion between mailing list and the forum. Um I haven't tried it. I haven't tried to use it in the forum way, but if like if you don't want to want to necessarily subscribe, there might still be a a way around to to use the mailing list, then you can always access the archives. They are public and visible to to everyone. Um, and I'll also dig out the matrix link from somewhere. Thanks, Silico. That's a that's a great reminder that the mail list is is is a lot better than it used to be. Another thing that um helps it um have good signal to noise ratio for your topics is to use the tags in square brackets. So, OpenStack operators issues the suggestion is to always put ops. Um, and other people will do things like Nova or OVN or Cinder or those kinds of things. Um, and then you can just filter for those. So, even though it is a bit of a fire hose, um, if you wish to just see certain topics, you can do that. Um and I think um Fungy Jeremy Stanley was one of the infrastructure supremos um recommended the web interface um as another way if you don't want to subscribe to so many emails into your actual mailbox. Um so while you were speaking Ilicico I did a sneaky look at my calendar. July 31st will work for me. Um I'm game for doing it if you're game for turning up and doing a recording and we can try. I will be around. Um, will people be available? Like it's middle of summer, so there's a likelihood of not having that many folks around, but even today like we got up to 9 and 10 9 10 that was great. So I'm I'm a >> I I agree with that. The end of July is is not great, but I think August 28th is also not great, right? And then I wouldn't want to say like, oh well, we're closing until end of September. That just seems, you know, too long, right? >> Yeah, >> we might have a couple of thin meetings, but I mean, I think we exchanged great information about uh Windows, you know, in this small group, right? So, um I'm game, so I'll put it on my calendar. Um if unless there's an objection of anyone present, the next meeting would be Friday, July 31st. Same ether pad, same meat pad. um we don't change that which is something the older coast suggested which is brilliant. So you just always keep this one link and you can find the agenda the zoom meeting link I mean pad. Um so that that works. Um, >> also, >> so I need go ahead. >> Um, just a just a quick quick thought like if if you all have any topics that we um that we didn't touch on today or didn't find the answer to. um like either drop it in the ether pad or or what would be even better just speak out loud because I always send out reminders to the upcoming calls. So if there's a particular topic that we could dive into um then people will be aware sooner like before the call and even if they cannot attend maybe they can provide some information. So it's it's easier to encourage some more folks who might have answers to questions if we know some of them in advance because it's a it's a bit random. OpenStack is a is a big community with a lot of project teams and we don't we usually don't have a representative from every single project team which means if you have a very specific question then we unless someone has experience with it on the call otherwise we it's less likely to be able to dive very deep into it. Um, so is is there is there any topic or or any other um bugs or challenges that that people are running into that you would want to try to have a an answer to on the call on July 31st or by then? >> That's a great question. While people are thinking um we did have a request I think three meetings ago for someone to talk about high availability and resilient architectures for OpenStack and I actually hold in one of my colleagues who's phenomenal at this and he gave a talk on it. it will be in the recording archive. Um because we run H8 control plane, we're we're 247 even during upgrades. We don't go down at all. Um you know, so um if that's something that you're interested in, you can put in Slack or not Slack, I'm sorry, you can put in Matrix or on the mail list, you know, a request and we I can try and get someone to talk about a specific topic. Um there I think we had universal information to share. Um our networking is kind of very niche. We use Calico for our neutron implementation which uh is not very common. So when people ask about um open v switch or um OVN issues, I'm sorry. I'm just like I have no idea. Sorry. Um but it's it's completely valid to to bring those topics. Maybe I can we can find someone who can talk about that. you know, if we said there's um I mean, hypothetically, we said there's new OpenStack installations going on. The operators would really like to hear from, you know, experienced practitioners of X technology, whatever it is. Maybe we could get a guest, right? Um my vision for Ops Radio was that it would be like one of those talk shows where there's like a host and then there's a guest, right? So, it can be something you can just tune in and sit back and listen to discussion. um we've managed to do that a couple of times but um I think we that would make it um really valuable to to sort of get the thoughts of a of a subject matter expert related to OpenStack and get it recorded and you know bring that to the community. So are there such topics that anyone can think of? Um, Eldica, I think, uh, you do a great job always closing the loop and bringing it back to the mailing list. Um, maybe we should, you know, try and drum up topics more actively. I I think I've been remiss in that. Uh, for next time, um, >> yeah, >> sometimes have a have a chat in the matrix room as well. Um, I don't know, Chris, if you have you started to update the operators contributor guide already. >> Um, >> I haven't started yet, but uh, I did get myself an action. I'll I'll honor that. So um I think we we have some material further down this agenda dock about some of the things that are obvious like um you know um making sure that um uh the matrix channel for the operators um this event for the operators the operators wiki uh you know is all highlighted and anything that's that's completely obsolete and wrong gets deleted. Um but no I haven't started yet. I haven't actually done a docs change for a while, so I'd have to kind of refamiliarize myself with how you do that. I have done it in the past. >> Okay. Um, let me know if you need help or if you want I can I can put in like the the matrix and and wiki pointers and you can do the cleanup. Um, I'm just thinking of Lucash and and others who might be newer. um so that we could start pointing more actively to to that document as well to help folks to figure out where to reach out and how to get um answers to some questions or guidance on on how to take next steps on things. Um >> yeah, that's that sounds like a good plan. Um we heard last time and this time that it is useful from people who are new to OpenStack. So I I definitely feel like we should I should make the effort. So um we'll clean it up, get rid of any outof-date stuff. Um there are other documents that we discussed previously where it's a bit of a bigger problem. For instance, the um the uh operator's guide, I think it's called, tells you how to run OpenStack and that is like 10 years out of date, 15 years out of date. Like it's so the question is is it even salvageable? I think maybe not. But the contributor's guide looks I you know I had a quick look at it and uh it does need an update but it is worth saving I think. So um >> that's where we should start I think. >> I think community processes evolve a little bit slower than the ways how you can and should set up OpenStack. So it's uh it's definitely simpler to keep the contributor guide up to date compared to the the operator guide which is also goes into I assume some of the details where it's a little bit of a preference on yeah I want to deploy my environment that way but what if I don't what if I have another setup in mind and then the operator guide operator's guide doesn't cover it. So yeah that that one is a bit more challenging. I don't know if there's anything that could be added there about the Windows VMs and and those kind of things or if that information is available someplace else. Um I myself have not um dug into those bits of the documentation um as I'm not running an OpenStack cloud myself so I I don't necessarily know others might do. >> Yeah. So um it seemed it seemed that in a previous meeting on the specific thing of an administrator's guide, it seemed like the the very idea of one guide being a single document that told you how to do OpenStack was kind of a a lost cause now because you know it mentions a range of tools most of which are no longer used. For instance I think it's Ansible Chef and Puppet and I haven't heard about Chef and Puppet being used by operators for years. So, Ansible of course is still huge. Um, but then there's all you know there's um a lot of people are running OpenStack is just an application on top of their Kubernetes cluster or vice versa and that it doesn't cover that at all. So, it's it's extremely dated and to really cover everything would be extremely difficult now. It would be a much bigger document. It would be continuously out of date. So, I think that is unlikely to be updated and be useful. But I think the contributor's guide it, you know, is just needs a a quick retouch. So, um, I've given myself that action and I'll I'll have a go and, you know, um, I think Elder, if you and I can contribute, collaborate on that, that would be great. I am conscious that we're used up our time slot. I'm sure that people have other things they need to go do. So, um, any other business, any other questions? Um, put in the ether pad, put it in the chat. Some of you have microphone problems. Um, last time you ran slightly over and I hate that, you know, because then maybe someone was late for another meeting. Always looks bad. So, I'd like to end on time this time. It seems like we're done. Um, so thanks for joining. Especially thanks if you tried connect many times and got dropped off. That's um very painful. Oh, Constantin says something would be interesting to hear is the experience with ML2 OBS. Okay. Okay, I don't have time to sort of comprehend that and respond before we run out of time. I've just dropped it straight into the ether pad. And if you don't see the ether pad on the screen, you maybe don't know what I'm talking about. Um the dot dot dot menu. If you mouse over it says more actions, open share document and you'll see it. I put it in for topics for next time. Something to do with ML2 ovs uh which I do not use so I can't comment on that. Uh, anything else people want to drop in or I think we we'll we're out of time. So, um, thanks everyone. Um, if you can if you want to join this again, I recommend arranging to be able to do it not on a corporate network because it doesn't work for me and it didn't work for I think uh, Rickard. Um, it just not a good experience. Um but uh we'll try and do this again on July 31st and put topics in the mailing list or matrix for next time and we will go over them and if there's a if a if a theme emerges we'll even try and get a subject matter expert for that topic. Can't say better than that right but for now I think we are done. >> Thank you so much for organizing this. Uh this this has been Brian. Thank you. >> Oh oh oh well thanks for joining and I think we we're still growing this. I think it's going to go from strength to strength. Thanks to Ildico for always closing the loop and keeping this thing going and uh we will see you next time I hope. >> Thank you. >> Thanks [snorts] a lot. Thank you. See you then.