OpenStack Ops Radio Hour - May 29 2026

Recording is running now. >> Okay. Thank you, Aldiko. Welcome everyone. This is the OpenStack Ops Radio Hour. Um, I know that um, we often get um, people who are new to this technology. So, I just want to do some admin notes to make sure you're getting the full experience. So, there's a toolbar on your screen probably at the bottom. Um, the document that we are collaborating on is accessible in the dot dot dot menu. Um, and it's called called called the shared document. So, if you're not seeing the etherpad on screen, uh, you should open that and then you can see all of us typing in there. Um, that's really the heart of the meeting. It's meant to be a collaborative uh, working session. So, we all collaborate on an etherpad. It's not a presentation. I don't have a slide deck. The second thing is there is a chat window that is native to Jitsi. Um, and there you can see a speech bubble um, icon on the toolbar and there you can do uh, open chat and there uh, me that opens on the left. And um, I'm just going to type in it now. And if you didn't see that chat, then you should open that. The confusing thing is that etherpad itself has a chat as well. We've occasionally had people trying to uh, join in the chat in the wrong place. Um, I think that Jitsi chat is better, honestly. Um, anyway, so that's that's some admin stuff. Does anyone have any problems or any comments? If if you want to just speak up, go ahead. >> I I wanted to add that it's nice if if if everybody could add them to the attendees list. Who's here? >> Right. So, um, if you have managed to open the etherpad, um, I'm I'm uh, on the document now and um, you'll see uh, if you scroll down a little bit, you'll see a section for today, 29th of May. Um, it says uh, recording, hosts, attendees. I've added my name, Mike, Francesco, and Thierry there. Uh, we also have, um, Fungi, which is Jeremy Stanley and Milan Kaiser. So, if you'd like to share your names on the attendee list, it's just useful because then the color coding of your, um, name there follows into your updates in the in the etherpad. So, um, it just makes it a little easier to follow. Okay. So, um, there wasn't a agenda, so I'll go put something on and I also put something on that was from Matrix. So, if you don't know what Matrix is, um, we are trying to get the community to agree to communicate in a limited number of places because the community has been fractured over across tremendous number of different outlets. Um, you know, uh, LinkedIn, um, Twitter Twitter or X as it's called now, um, Matrix, Mastodon, um, Reddit, uh, you name it, right? Uh, and that just means that all the conversations are, are kind of, um, partial. Um, Matrix is, um, the suggested modern place to go. Um, and I think we even have, um, links for the OpenStack Ops channel on Matrix somewhere deep down in this document. Um, so, anyway, so that's where the agenda came from, but, um, if there are other items that you wish to discuss, whether it's what we want to ask questions or whether you want to share experience, um, just go ahead and write under topics here. So, um, you'll see it says topics and then, uh, I'll go put, um, a suggestion of operators contributor guide, which we'll talk about, and then I put something about, uh, the question of data center moves, which is from Karli Hapenan who can't make it today, and I put some some notes about, uh, something that we have So, we've never actually moved clouds between data centers, but we have decommissioned clouds. Anyway, so operators contributor guide is the first topic today. By the way, anyone have any comments or problems or anyone struggling with the interface? Um I think when this first started, nobody knew how to drive Jitsi, and maybe people didn't get much out of it. Um I think everyone seems to be doing well today. Um so, operators distributors guide. So, um the questions that Ildiko posed posed are are people aware of this? Is it used? Is it up to date? Um And then someone responded, "Areas to improve. We should probably emphasize Matrix over IRC." So, I took another look at the doc today. I was vaguely aware of it, honestly, but hadn't looked at it in a while. Um and it doesn't mention Matrix, so um I'm quite happy to to have a go at adding that particular thing. So, um I don't know if we actually have an action um Uh my color seems to have changed. That's interesting. Um I'm now just putting Um So, I can get the information for the Matrix channel for OpenStack operators into this guide. >> Are others on the call aware that there's a there's a contributor guide for for operators? Like how to get in touch with the community, what the community is, and all the things? Francesco, I think your hand is up. >> Yeah, I started from there, but it was already outdated when I read it because it uses an out-of-date IRC client, doesn't have a full list of the available channels, it is outdated on the release model because it doesn't have yet the SLR SLURP model. The Gerrit tutorial was out of out of date, so I had to follow another tutorial. So, yeah, it's outdated. Sorry. >> Oh, don't apologize. I I put the topic in here just to kind of figure out what to do with it. Is it also something that people would want to use? I mean, not to throw things out, just have the information. >> Probably yes. Honestly, to me, it was helpful, even if out of date, because I had all the information to start with the community in one place. Okay, it was out of date, but at least I had an idea, more or less. Then later on, you know, I changed the IRC client, put out a bouncer, etc. But, meanwhile, it was better than nothing than going around with different docs, etc. >> Awesome. I mean, not all of it is out of date, but I'm happy that that you still found good use out of the document. >> Okay, so um I've already volunteered to have a go at putting in the Matrix info. So, if I could ask for people to actually just list here in the Etherpad the things that are out of date, I can try and crowdsource corrections to all that. I don't necessarily know all the facts. So, um you know, I will rely on people to point them things out, but I will do the hands-on. I think I think the thing about Slurp, yes, that's a very important thing um for operators, but also I think um operators have some credit for consistently raising the topic of upgrade difficulties, which I think led to that. Um so, in the depths of time we had we would have meetups, and people would say, "I find it very difficult to keep up to date because each update is difficult and you guys update twice a year and we can't we don't have the manpower to get there. Slurp cuts the overhead in half because you're guaranteed to be able to go from one version to a version two versions ahead quickly. Um it's called skip level update. So that's a very important thing that um should be in this contributor's guide, I guess. If it does talk about the release process, at least. Uh I see lots of people um typing away. This is great. So again, I have to emphasize this every time. Um it's not a presentation. I don't have a slide deck. We're here to share information and kind of get it down on the etherpad and I have heard people say that um they couldn't make this meeting, but they caught up on YouTube, which is why it's recorded. Uh and often they also say that there's some good stuff in the etherpad. Um so this is kind of um our job today is to not only just have a discussion, but to try and get some key points down here and um the next time we meet, you know, it's a wonderful to go back and scroll down and see all the stuff, remind ourselves about what we were discussing, what tasks we set ourselves. Um I'm seeing more and more detail. Yeah, summits um that's a hard one because um the old cadence of two summits a year is gone. I think it's at most one now. Um and you [snorts] can't necessarily predict when they're going to be. Um but um once we do know when the next one is, which I think is Shanghai in the autumn. Someone correct me if I'm wrong, then um we can at least get that in there. >> [snorts] >> Yes, there were there were some reorganizing of content between um some of the OpenInfra website. So there There a new landing page for for events, which I think will be better to link to from the uh the contributor guide. And the the other note that I also wanted to make is um like updating the contributor guide goes through Garrett reviews. So, it is an easy thing to collaboratively update if there are more people who would be interested in um like getting your feet wet and uh and contribute some fixes to the documentation. It is possible. Um but by all means all the all the notes about what is outdated is already really really helpful. So, thank you. >> Okay. Um so, keep on adding things that you'd like to see in that doc and I'll um I'm going to I don't know about commit, but I'm going to try hard to have it uh update or at least the changes proposed by the next time we meet, which is um a nifty way of um rem- reminding myself to mention that the next meeting suggestion is June 26th. Um thank you uh Ildiko for putting that in. I'm sure you had to go and look at the calendar and kind of balance some things. It works for me, so I plus one it. Um [snorts] and I do find that um you know, if we have a date and then we might notify people, we get attendance. But if we just, you know, 5 minutes before the meeting say, "Hey, let's do this." It's that doesn't work. So, we're all busy, right? So, um please consider June 26th and, you know, uh plus one or minus one it. For reasons for minus one in a proposed date might be that it clashes with an open source event or that um you know, it's a public holiday in many of the countries that that we are participating from. So, give that some thought. Okay. Um Any more on the contributors or in fact operators related docs? Um I did scroll back and look at the context from previous meetings. There are other documents that are aimed towards operators that are probably even more out of date. Um For instance, there's a an operators guide and there's an architecture guide. Um They seem to be kind of a complete loss at this point. They're so out of date. Um My feeling about that was maybe we should agree to replace them with links to whatever documentation in that space exists that is kept up to date. Um I'm just reading some of the comments that uh are going in here. Um So someone's putting a question whether the operating facing documentation should be >> [snorts] >> Yeah, so um I'm not going to try and drag all the links into today's agenda. But um they they are mentioned further in this further down this very etherpad. Um and I'd love to get feedback. Um If some people, you know, use those docs and just wish that they are more up to date and that they're like essential and wonderful, that would be nice to hear because I haven't heard that um and we did get volunteers to rewrite those who then just happened to move on. So you know, that was a false start. Um so if you've seen those docs and you just wish that they were updated, maybe we can work on that. If however they are as um kind of hopelessly out of date or maybe inappropriate at this stage, I'd love to hear that too. Um When I say uh inappropriate, um when they were written, it was a smaller project and there was less diversity and um I think there was a feeling that um, you know, that there might be consensus on the architecture and the tools, but the truth is that OpenStack architectures are wildly diverse, the tools are wildly diverse, there's you know, official ones from major Linux vendors and from independent vendors. Um, you know, and there's containerized, not containerized, there's fully Ansible, there's OpenStack on OpenStack, there's OpenShift, which I think is OpenStack on Kubernetes. Other people run it Um, in some cases Kubernetes on OpenStack on Kubernetes, like you can't capture all that in a document. Um, some of the biggest OpenStack architectures are actually not public. Um, I think um, Huawei has something like call like a cascading OpenStack architecture. But um, how that works in production has never been disclosed. In even the uh, people who worked on Huawei open source didn't know. Uh, go ahead and go. >> Um, just a quick brainstorming idea um, to see if we can, I don't know, meet in the middle. Since these calls are recorded, um, I was wondering if people have interest in talking about how their clouds are set up, kind of sharing case studies. Maybe because it's again recorded content, that could that could be still good information for people who are trying to set up theirs and try to operate theirs. So, so trying to share some stories and then in parallel, we also look into the documentation and see what the things are that are easier to maintain that we could still keep up to date enough on the side. So, kind of have a little bit of documentation for people on how to do things and also the information if people are open to share what they are doing as concrete examples could be like just showing both sides of how this is supposed to be working and this is what we do in real life. Just a just an idea. Um >> I think that's a great idea. >> Sure, show and tells at the at the summits periodically as part of the ops forum. We'd have like you know, kind of people would just sign up and go up and give a 5-minute, you know, spiel on you know, how they set up and how they did things and kind of made some interesting records. You You go back and find some of the some of the ones I guess they were forum sessions mostly so they wouldn't have been recorded, but there's probably ether pads somewhere and there might be links to presentations and things we can collect and then carry forward from that. Maybe get a get a volunteer or two for each one of these things to >> I can put out a um like a reminder when I have the the reminder emails about the upcoming calls to ask who has their operator story to share or some kind of a show and tell. And >> Mike has his hand up. Um go ahead, Mike. I think we just kind of it's free form, but go I since you're using it, go ahead. >> Yeah, it's fine. >> [clears throat] >> Speaking from a company that's uh deep diving the OpenStack ecosystem and trying to find the current direction of the product and what the main consensus is of how this should be deployed going forward. Uh like they all say, there seems to be quite a push for containerized uh and at the same time I hear in this this session just now that there are a lot of uh not open source deployment strategies. Uh I don't particularly would want to say that they need to open source them, but having more this is what we did without giving the details of how they actually did it, but a generic view a view of how they deployed their system would be really helpful for new for new joiners. I also added some links and uh some suggestions. We recently came across uh somebody from Rackspace that started doing something with Kubernetes and OpenStack called GenieStack or GenieStack, not really sure what the correct state naming for that is, but more of these examples would be nice. >> Okay, that's great to hear, Mike. Um so I can respond to that from the Bloomberg perspective. So we used to run our OpenStack distribution, if you will, on our project open source on github.com, but um it had um some terribly dated parts. So we stopped putting it on on GitHub because um it's not a good look anymore. Um it's not a mystery we were using Chef, which um was popular about 10 years ago and then became unpopular and now has a different vendor who've stopped it being legitimate open source because you have to register with them to download it. Um and it's it's kind of dead for this community and we haven't finished getting our stuff out of that. Um however, more positively, we do want to share how we do OpenStack and you know, results we have at at our scale. I can give general numbers about scale. Um whether that would be a white paper or whether we reopen it as open source once we're done, um is TBD, but I'm I'm glad to hear that you'd like to hear that kind of thing. There's not just how we do it, but there's other people on the call who have been running up OpenStack or maybe you're familiar with multiple deployments. Um the one thing that's um niche for us is that we we are pretty unusual in not using um OVS or OVN. Um Our stack is based on a pure layer three network. So, it's kind of internet architecture. So, BGP based routing right down to the hypervisors and the control plane nodes using Calico, which is a different Neutron implementation. Um but otherwise, our stuff is So, it's kind of interesting. Ours is very niche in that one bit in that one way, but in other ways, it's extremely old school. We don't do any containers for our OpenStack. We just we use the uh Ubuntu distribution of OpenStack. And um we are ripping Chef out and going to just plain old Ansible for deployment. So, um Well, I think we have some stuff we could share at some point. I have to get it reviewed by Cobra Communications if it's written or if it disclose anything more than outline, but that's something that I want to do and I'm it I'm hearing that you'd like to hear other people saying well, how they do OpenStack. Maybe we can persuade people to come and give a little show and tell on a Friday morning once a month. Hey, this is how company so-and-so uh does their OpenStack just in general terms like like like what I just talked about like for instance, we don't do any layer two networking whatsoever um in our OpenStack, which is fantastic for scalability. Um so, this is something we should work on. Um it's a business goal of for me, for my boss to get back to being able to disclose more of our stuff, for example. I don't have anything I can link to today, though. Um Uh okay, so there's some great comments here. Go ahead. >> I didn't want to put anyone on the spot, but I'm just giving that uh perspective that if you want to start with OpenStack today, there's so much to be found and so it's it's a lot a lot is scattered. Like we recently rediscovered the Geniustack stuff. There's just so much out there. And and all also very opinionated. So that's also the other part. But yeah, I I'm I'm quite interested in learning how other people solved their dilemmas and actually which dilemmas they found to learn from of course. >> I would um that that's in the category of war stories. Um so when we had uh two-day operator meetups events, we always had war stories at the end of the day cuz it's the in some ways the most fun and it's sort of like a an icebreaker before we would then maybe go for dinner or beers or something. Um I can tell you what not to do with older versions of OpenStack. Uh Nova network was a bad architectural choice because we got on it and started scaling just as OpenStack um you know, uh obsoleted it. Um and it assumed layer two and so we grew enough to run out of ports on our core routers and uh we just couldn't do any more IP space and we couldn't grow and um extreme pain. Um that those clusters were switched off only this year. So I'm not I'm not really dredging the historical archives too much. Um but yeah, so um I think to get people to come and talk about how they do OpenStack, I'd have to sort of beat the bushes a bit and get people um interested in sharing who maybe don't come to these things because they're you know, they've successfully made their architectural choices and they're now scaling their product but um there are people who you know, uh would love to share I'm sure. So I'll I'll take that as an action to see if we can persuade some people come talk. Um one person for instance Um so there was a a former um Open Iraq space luminary. Um that Matt Van Winkle who left OpenStack community for a while but came back and is I think running it at Geico now. Um Maybe I can persuade him to come and just talk in general terms about you know, things that they can disclose. I mean no one's going to say, "Hey Geico, I need your I need your code. Can you please upload it again?" No, but they could say like >> [snorts] >> "What's your network architecture? Do you do high availability? What's your How many clusters? Like how many Are you using regions? Are you using cells?" Right. So um I I talked about since his uh return to OpenStack at Geico and tried to persuade him to send people to one of the events. I don't think that happened, but I I can try that. Uh if other people have um you know, contacts who are involved not only in the OpenStack development community, but actually the running OpenStack, particularly at scale and production. Um feel free to try and invite them to um this meeting. Um and you know, one of the things we have done in the past is kind of had a guest speaker. So I think two or three sessions ago I had a the colleague who's really the tech lead at Bloomberg for OpenStack. Uh name is Tyler Schecky. Uh talk about um our HA architecture for instance. So um you know, a guest speaker would really I think uh enhance this. Um Yeah. So Um I should put another action. I don't know if Etherpad has a specific syntax for for actions to send out, but uh I'm just going to type something in here. >> There's also um like there's a comment um from Milan um on the chat um agreeing with Mike that people sometimes need to be experts before being able to deploy and run OpenStack in production. And there's also a question like what is the term for things like Cola Ansible, the Genie Stack, and the Rackspace Atmosphere, whether these are distributions or life cycle management tools or ecosystems. I think they are deployment and life cycle management tools, but if anyone has a better category to put these into, um please share. >> Yeah, I mean Cola Ansible would would be, you know, deployment deployment tool. Um but there's also Cola itself, which is, you know, it's it's technically a separate project, I guess, so it wouldn't get included. But since they build containers, um I suppose you could consider it to be a distribution. Other things use the Cola containers as well. Um Or you can use them without Cola Ansible, so it's a little bit iffy on that one. But definitely would look at it as more more of just a deployment uh life cycle management >> the Some of the other names there are definitely vendors. So, Rackspace recently recommitted to OpenStack and said that they are reconverging with um mainstream in terms of the uh you know, the software version cuz they they forked it many years ago and had been living off in their own fork, but they want to reconverge on on um main branch, if we can call it that. And they do private and public. Uh and possibly turnkey as well. I don't I don't remember all the things, but um they've they've come back, you know, full >> [snorts] >> full-fledged. Um and then Vexxhost, um also a public cloud hosting company, right? That's Mohammed, I believe. >> Yeah. Atmosphere is is technically speaking open source. I don't know how many people outside of Vexxhost are maintaining it, but Atmosphere is a very opinionated deployment of OpenStack into Kubernetes. >> So, I guess the point I'm making is sometimes you just have sort of community projects. I think Ansible doesn't have a an obvious single vendor. But, Rackspace tools, Vexxhost tools, Red Hat tools, Ubuntu tools are obviously from vendors who would love to actually sell you an OpenStack. And I think that's fine, by the way. We We do use Canonical, for instance. Other people use Red Hat. I think Red Hat's the biggest. Um And you know, the vendors do a lot of the heavy lifting. They They actually employ most of the engineers who are still writing code. But, so to answer the question of what So, what's the terminology? There's some that are just open source projects, community, and then there's some that that are open source that are vendor sponsored, should we say? And I think they're all fine, as long as they're really open source. Um contrasting with Chef, which was only ever um you know, part of OpenStack community amongst many communities, and has become not not valid open source. Um so, some great some great content here. I've given myself some actions to try and get some speakers on the topic. Um but, I'd like to move on just So, the a question maybe for next time that was that was placed in Matrix was um practical experience of data center moves. So, I'll be up front. We've never moved a cloud. Um Does anyone want to talk about that? Or at least Does anyone want to say, "I would love to talk about that next time because the person who posted the question was up front that they couldn't make it today. Karli Hapanan. Um I think maybe Eric, you've uh you've recently moved equipment at least. >> Well. >> [laughter] >> Did that count as a data center move? >> I just counted it as a full decommissioning. Uh we can certainly talk about that. Um I mean I've done I haven't done it technically speaking moving in moving it from one Uh I mean I've done a lift and shift of equipment from one part of a data center to another. So fully bringing down the cloud and bringing it back up again. Same cloud. Uh and I've had to do multiple forklifts out of you know an old version of OpenStack that was completely unupgradable. So I've got that side of it, too. Um So I'm I'm happy [snorts] to discuss it. >> Um so by the way, thanks to thanks to Eric for joining. Um not to overshare, but uh it's not so easy for you to join these things anymore, right Eric? So I do appreciate it. Um >> I'm actually >> So for us >> a desk farm, so I have to keep it muted and talk in quietly. So I'm sorry if I'm too quiet. >> No, you're fine. Um so for us um our OpenStack runs in private data centers. Um and there's no such thing as moving um one of our implementations between data center because the IP space belongs to the facility. Um if we were to move a cloud um to a different IP space uh it would break all the clients. And so it's just not um not something we've ever done. Um I think if you're in you know, um maybe a colocation space and the vendor needs you to move to a different one because you've got too big. Um and they can keep you can keep your IP IP space that that would be a a fascinating challenge that I've never gone through. One thing that um we have done, as I mentioned before, is decommissioned clouds in place. Um the old OpenStack clusters that we started with uh well, several iterations, but we're all Nova network based, layer two networking. Um we turned them off this year, and I put some notes here because that itself uh is hard enough. Um And it's you might not know this because if you're starting you probably haven't thought about finishing a cloud, but um it's harder it's harder to finish it to to close it down than it is to bring it up. Um for instance, ownership um if you have VMs running on your cloud and um you send out emails to the people that you think own them, and they don't get back to you, what do you do? Can you just destroy them? Um I would advise against that. But you also have the problem that what do you what are you going to do? No one's No one's getting back back to you. Um so I put some notes there. I don't know if the people assembled here wish to uh discuss that, but um basically um you have to have a systematic owner ownership system somehow. Uh in our case you know, uh anyone who has an asset on our cloud has to have an entry in an ownership database. If they leave or if they change teams, it has to be transferred to somebody else, so there's always someone, and they can't say like, "I don't know about it, so I don't own it." They do own it, and they have to learn about it. Um if if someone leaves and it the ticket isn't transferred to their replacement, it goes up to the manager and so on and so forth. Eventually, there's somebody who has to deal with it. Um What did we do when we couldn't find owners, and no one knew anything about VMs? We just shut them off. That's a really great way to get people's attention. It's like they don't know about your cloud, they don't know they're not on the mailing list for updates about the cloud, they don't respond to things. But you turn a the off, and they come and say, "Oh, our CI/CD stopped working. What's going on?" Or things like that. That's a real example for me and that was for um a critical mission-critical app within our company. Um So, you know, you can turn VMs off, you can set quota to zero for a tenancy. Um and that's a great way to um you know, determine in a non-destructive way whether anyone cares when you're turning things off. And um I wanted to mention um there's been um many times when the the boss says, "Oh, just delete it. No one cares." Turns out that people really do care. And it's not that they're negligent. It's just they have no idea what's going on. So, if you in a large company with a lot of turnover, I mean, we don't have a little high turnover rate, all right, but we have thousands of engineers. They move projects. They come and go. Um industry a low rate relative to industry, but still it does happen, right? They don't know what's going on until something goes away, right? So, that's the comments there. I see I see some comments on the data center move thing. Um >> Yeah, that's me. >> Uh so, fascinating. It's not something I've had to do. So, I'm actually just reading it so eagerly like if we had to do this. >> [clears throat] >> The other thing that um I think helps data center moves and decommissioning of old architecture is uh we've really tried to get our clients to be um nimble. I was going to say agile, but that's kind of overloaded. So, nimble to me means infrastructure as code, VMs or containers um as replaceable items that they can build, you know, from some kind of pipeline um because then you know, you have a bunch of VMs on a cloud that maybe has to have downtime, maybe it has to physically move, or maybe it's been replaced by a different cloud implementation that uh is so architecturally different that, you know, there's no kind of um bridge between. Um those nimble teams can just you can give them loaner quota and say you know, go build on the new one. And when their tests pass, they can just decommission the old one. Um you know, being cloud, it's easy to do a uh temporary quota. Um so, people can kind of build all new while still having all the old. And then only switch off the old stuff when they're actually happy, right? It's not like in the case of physical where it's very difficult to buy all new and run in parallel with, you know, existing physical machines. That's quite a expensive uh proposition. So, um we've done that through limiting certain ways of working with OpenStack. For instance, we don't allow people to upload images by default. We don't allow people to do snapshots by default. Um we provide the images and generally we want people to just, you know, use cloud-init to customize their their instance and then finish off building their stack on it um through automation, not through hand making a golden master image and then making VMs from that and then, you know, next quarter hand updating it. Like it should should be infrastructure as code. Um I love the note about find a company that has experience moving racks. Um >> Yeah. >> We've been running data centers for >> That's a huge one. Uh Chris, that's a huge one. Find so find a company that has experience with it. >> So, we've been running data centers for decades. Um but things that have happened in my time on cloud, um racks have been assembled in one facility and then by the time they're installed at different facility um there's a whole bunch of bad connections including DIMMs because New York City roads are terrible and uh the whole region has you know it's a snow belt area of the country. So a fully burned in rack in one data center maybe a systems integrator who assembles these things for you doesn't mean it's good when it actually is in your data center. That was a horrible experience. You know you think the capacity is about a day away from going live and then it turns out it's it's it it's a mess. And [snorts] uh we've we've also had um a rack got knocked over whilst it was powered up and in service. You'd think it'd be impossible but um apparently it happened. So um I would take these things extremely extremely seriously. Uh if you're moving computers I would assume you might lose them. >> The fact is a thing CPU servers are very >> destroyed. Um Sorry Eric, I didn't mean to speak over you. >> No no I said don't put the big CPU servers in the top of the rack. >> Yeah. Um You They might get stolen. They might get destroyed. Um a power power cycle and then a physical move might just be the last straw for a hardware that was perfectly fine when you last used it. And then if all goes well you still probably should have a validation at the new location with the new cabling that you know for example that both power cables are fully seated into the power distribution of your rack. Go ahead. >> So Chris uh the reason why I mentioned the company that helps you move the stuff the one we found actually wraps the racks uh completely airtight. So it doesn't dissipate the heat that's in the systems. So when they arrive depending on the distance that you need to drive, for us it was maybe two two kilometers or three, they arrive still being warm, so they actually don't cool down completely. >> That's fascinating. That's great info, Mike. Thank you. Um yeah, I I'm not uh I'm not involved in logistics so much. Um I just hear about these things. Um for us um racks are in fact assembled offsite. They do ship in trucks and um that's hard on them. So, um you know, um if someone's asking you to do data center move in a hurry because oh, it's just you know, just power it down and then power it on, it's fine, right? A fully assembled rack of hypervisors is a huge investment and it's very delicate because, you know, connectors are meant to just stay connected in a data center, it's stable temperature, not when bouncing up and down in the back of a truck. Um and certainly not if you knock the rack over while powering it up. Um so, I've done a lot of talking here. I always emphasize this is meant to be a working session. Are there other topics that people want to add to today's discussion or do anyone want to dive back into something we've contributed uh already or discussed already? I know that uh Jeremy, you said you were interested in the discussion about the operators contributor guide, but I haven't actually heard you uh speak up. Maybe you've been typing away and thank you, but um did you have any comments on you know, what we've discussed regarding that doc and the other docs? >> Not beyond what I put in the etherpad, just trying to make sure that everyone knows where the source code for those pages in that chapter uh live and um the the the history there. So. >> Okay, so you're probably the one who did the uh updates in black. >> Yeah. Yeah, that that was that was my contribution. >> So, for those who don't know, um Jeremy, whose sweet name is Fungi, is effectively infrastructure benevolent dictator of infrastructure for OpenStack for Open Infra, not for all of OpenStack out there in the world, but like the Open Infra website. If Jeremy doesn't know it, probably nobody knows it, so thank you, Jeremy, for the the links there. I'm going to obviously be the first person to use that because I've boldly said I'm going to try and propose changes to the contributor's guide. Um and I'm still I am kind of looking for consensus on the other documents here, the operator's guide and the architecture guide. Just to repeat, I'm you know, I've lived through We had a whole meeting at one of the summits where people said, "Oh, yeah, these are important documents." And I have the experience I I know I know HA OpenStack like the back of my hand. Um that was Adam Spears, I think. Great guy, right? But then he no longer works on OpenStack, so that didn't happen, right? So, um before effort is made on those docs, I'd like to hear that people really would value them. And if the question is I'm sorry, if the situation is people haven't even read them, um maybe we can ask people to take a look and get back to it next time because um it's a non-trivial job because they are so thoroughly out of date. It's like writing them again from scratch. Um yeah, we we did add um some disclaimer on the security guide, I think, just because the security SIG had the ability to to review changes for the for the man for the security guide manual, and um uh just to basically note that it was so severely out of date that we were worried people might actually make things worse if they were followed recommendations in it, so. >> Yeah. So, that um So, Jeremy's very extremely diplomatic. But, um you one might say that the architecture guide and the operator's guide are actively harmful at this point. So, it's sort of like they should be fixed or or um retired, I think at this point. Um anyway, so I I no one seems to have comments on that. I do see some things that I haven't pulled out from the etherpad. Um so, one thing I just noticed a small thing mentioned of these meetings would be nice. That's a very good point. So, if I make the contributor's guide reference Ops Radio hour, maybe we'll have people funnel into this meeting that way because it's still a challenge to get the word out that this these things happen. I'll do that cuz I've already said I'll update the doc. And then, um right down at the bottom, a new question, would the foundation consider a Matrix alternative or are we too deep into that now? Um So, I guess this is a a >> That was me [clears throat] that >> suggestion to use Zulip. >> Yes. >> That's you, right? >> Yeah, that was me. Yeah. Um something we've been using at new company extensively, heavily. And I kind of like the way that it it the whole topic driven method works as opposed to having to make a new channel for everything or having everything lumped into just one long amorphous blob. Um you know, ala IRC Matrix kind of too. Um It's kind of nice to be able to it kind of nicely fleshes out the topic. So, we could have a a discussion in the Ops channel and then you basically start a topic and talk about the operator's guide and it would all be in there and you could collapse that topic if you weren't interested in it or you could, you know, focus in on it and then it'd list it all down, you know, the topics down the hand side underneath the channel. Uh pops ones with activity up to the top so you can spot them and and run down them and ignore the ones you want and mark all the messages read that you know, if you don't have any interest in that sort of thing. Um and pretty much anyone can create create a topic um depending on the permission settings. It's you can anyone could create a channel. Um just a kind of as a you know, higher level thing. Um I don't know. It's got lots of external integrations like Git for example. Uh just kind of nice for you know, if any So developers wanted to use it and have like Git notifications posted into a Zulip channel. Um so it kind of does some of the Slacky things that people like about Slack but it's a lot cleaner. It's open source. You can run your own. >> So yeah, so I didn't mention Slack too much earlier. Um but Slack is what we use on my day job um but it costs money. >> Right. >> Um I did I did hear that um projects that are part of the Linux Foundation are claimed to now be able to have um a better level of Slack. So to be specific, a level that doesn't delete all the posts after 3 months. And I actually posted that on the foundation board mailing list. But um there didn't seem to be much interest quite frankly. I think Julia said that she thought it was a specific thing for the specific team that had arranged that which was the SA Foundation which is now under the Linux Foundation just like the Open Infra Foundation is. Uh I don't actually know, right? So that was um an attempt to see whether we could do Slack. Am I trying to promote Slack? No. They're a commercial company and um my experience of using Slack for open source projects is miserable because the content doesn't last, so every time you get on, there's like posts from the past few weeks, and then everything else is lost entirely. So, Zulip sounds great, but honestly, there's just so many suggestions now, right? We've got >> Yeah. >> I didn't even go through all of them, right? Officially, the foundation says mailing list, IRC, and now semi-officially Matrix. And then, but people are actually out there using And the foundation is also on Mastodon. People are out there using Reddit, LinkedIn, um There are teams that are still on Slack. Um Eric, you probably remember the local organizers for the Mexico >> Latam, yeah. The Latam one is still on there. >> Latam Slack channel. Um >> Hey, I can I can I can chime in on this a little bit. Um so, in terms of Slack, um the news have been true in terms of uh Linux Foundation projects currently having access to that level of uh Slack's Slack, which does not delete the messages after 90 days. That is for 2026. So, I expect that to be renegotiated. I don't know, maybe on a yearly basis. I don't have that level of information, but the the point is that it is accessible now, but I don't see I haven't heard that there would be guarantees that it's going to stay like that forever. And since Slack is a commercial company, they can change their minds on this at any point in time and just get out of this um this agreement that they currently have with the Linux Foundation. So, that is not something that we can build on in a stable way for the future. In terms of other platforms, the foundation actually does not mandate that projects have to use IRC, projects have to use Matrix. IRC has been an OpenStack project community preference from the beginning-ish. In terms of like Matrix versus Zulip versus other platforms, one thing to consider is you always lose people when you switch to a different platform. So, unless there is a like a critical mass in terms of yes, we all want to move to Zulip and we truly want to have offline or about chat communication like semi-synchronous or async communication on a chat platform, then then you can consider moving, but it's it's not really a foundation mandate that the ops group has to be on Matrix or that you have to be on IRC. It is more around the consideration of if you would decide to move to Zulip, for example, would there be enough people who have the buy-in to follow or would you just further fragment the community? Because fragmentation is the the biggest risk at uh whenever you move platforms. Um I personally, I like Zulip, too. I tried it once in an experiment when it was evaluated for the team um that I was working with whether or not to move to. I personally liked it. Um I I think it's a it's a different concept than many of the chat platforms. So, I haven't really heard of mass adoption just yet. Although, again, it's a it's a nice tool um for chat. It's um more unique concept than most of the other ones follow. So, if there are enough people here who are interested and maybe also volunteering to maintain it. Uh the setup that um that the community prefers then there's there's nothing from the foundation side that would prevent you from doing it. Um but it is also something to think about that has some maintenance impact as well, which would be great to have people to volunteer for. >> Thanks Zille. So I'm going to say some potentially unpopular things now, but I just this is something I feel I've been fighting for a couple years now. So the foundation actually has no ability to control what OpenStack operators do whatsoever. Um we don't work for the foundation. We don't have a um you know, a contract that says that you know, we communicate in certain ways. Some things are mandated for instance, if I want to join the board meeting, I do have to join the Zoom meeting. But um just from first principles, there are other OpenStack communities out there using OpenStack who just choose how they can um collaborate themselves, right? For example, a lot of them are not on the mailing list whatsoever. They're just not on it, right? Um the foundation says the mailing list is the primary thing or has said such things in the past. But um you know, people may join and then find that it's too much, right? Um and then there are very valid comments that say that every effort's been made to make the mailing list work better. There's um a rich system of tags and then there's a web interface that makes it like a chat platform. But I don't know whether we can demonstrate that um the operators of OpenStack have really um sort of come back to the mailing list if you will. Um the dedicated operators mailing list was retired 8 years ago. Um so um I for instance have made an effort with Matrix. Um there's a dedicated operators channel on Matrix. Um but I think that it's about three messages a month or something or or a week, sorry, at most. So, I mean, okay, we we couldn't describe that as lively, right? >> Um I mean, depends on how you define lively. [laughter] >> If we take your messages and my messages out, um >> I mean, there there is there there is some conversation conversation there sometimes. So, um but I don't know >> that bad. >> I don't know how successful it's it's being, like >> So, >> the scientific SIG is on Slack and every now and then they have a conversation and then it's a a long period of quietness. So, just because they're on the Slack, they don't have more communication. But Jeremy also has his hand up, so I want to hear that speak before we wrap up. >> Yeah, I I was just going to say from the perspective of the the mailing list, I guess it depends on what you consider the operators being engaged on the mailing list. There's uh basically ever since we combined the mailing lists, um I I as someone who does follow the OpenStack Discuss mailing list see multiple questions from operators every day being posted and being answered by maintainers of the related subsystems in OpenStack. So, I mean, from from that perspective, it it seems to be working really well. Um we we were convincing not just the operators the OpenStack Ops mailing list into OpenStack Discuss along with the old OpenStack Dev mailing list, but also at the same time basically we were using that as a target for all of the discussions that were previously happening on the OpenStack forum uh and then it's successor the OpenStack Ask service um which was a a Q&A platform we tried for a while that wasn't really uh working very well either. And and the thing that none of them had that the mailing list has now is that it's putting the people who have questions and the people who have answers to those questions in one place together. But to your point, yes, there are also definitely very vibrant operator communities for OpenStack on Reddit who are only going to discuss things on Reddit with people who are in the OpenStack Reddit and the same goes for people who are doing it on HNN or Stack Overflow or all the various other forums. Some people just prefer a community within a particular forum to a community that is blessed by a particular organization. >> Yeah, it's very fair to say the mail list does have more activity and if I may have over spoken, what I was saying is I've repeatedly interacted with OpenStack operators or potential OpenStack operators who just not on the mail list. Even people who wanted to host one of the meetups that actually didn't happen were just not on the mail list. Um So, it's not universal but it I think it is more likely than the Matrix channel for instance. I would agree with that. I'm still going to look at Zulip. It sounds cool and I still run an OpenStack Ops meetups thing on Fosterdon on Mastodon. Anyway, we're out of time. So, thanks everyone. Please do come back on June 26th or give us feedback if that doesn't work. But I think we should wrap up because probably people have hard stopping have to go to other meetings. So, thanks very much and Ildiko, if you would close the recording and we'll see you folks next time. >> [snorts] >> We'll do. Thank you all.