OpenStack Ops Radio Hour - February 27, 2026

So for the uh February 27th, 2026 OpenStack Ops radio hour meeting, the recording is now running. So uh Chris, the floor is yours. take it away. >> Thank you, Elderico. Okay, welcome everyone. Welcome back. I guess we've done quite a few of these now. Um so I hope you can all see the ether pad. That's the um the rough agenda. Um we don't we don't have like a rigid structure. Um so if people want to call out and you know add topics, they can write them there. Um but we do have kind of a special guest. So last month there were questions to me about um how do you run OpenStack in a high availability mode and um I started to answer the questions thinking I really wish I could just ask Tyler who's really our technical lead on our private cloud product which is based on OpenStack and I said could you just join the next meeting and he said sure. So he's actually joined. So Tyler has been uh instrumental in building this to a key platform for Bloomberg uh and has presented at OpenStack uh or open infra conferences in Berlin and uh has upstream contributions. So um he knows his stuff uh better than I do these days. So um he's here if uh if any of the people who had questions last time are here they can ask or we can just you know um uh you know discuss HA architectures in general um we run uh a mix of architecture so we don't have one giant cloud that must be up all the time we have uh between 10 and 20 instances but each of those is also getting large these days >> and has enough scale fail to be um a scaling problem and also enough business logic dependent on it that if it's down it's you know it's a major outage. So um you know the HA aspect of it the control plane is a very important thing that we uh you know put a lot of effort in. we have uh most of those key services clustered or in HA mode um you know uh database load balancer uh message bus uh storage control plane etc. So are there any any specific questions on on on how we do all that that Tyler might be able to uh you know comment on. So uh in case anyone is um new to this um this is actually supposed to be a working session. This is not a presentation. So if no one speaks it's going to be very short. So um please do actually uh you know participate. Um it's not a slideshow, right? You're tuning in to actually talk about the work we do. So um really need people to actually talk rather than just sit there. Right now I've got only one person on video, no one's speaking. So it makes it impossible to to get anything out of this. So >> don't be shy. >> There are still people joining it seems um at least in the past minute or so. And um based on the survey, HA and HA configurations is one of the topic areas that people are curious about. I'm not sure if everyone is ready to ask questions. Maybe some folks are new and trying to figure out how to do HA in the first place. I don't know if you all could maybe share some good practices that you had good experience with or some things that you ran into and you had to resolve or any challenges in terms of how to operate OpenStack in high availability configurations that you think would be great for people out there to see because um there will there will also be people who will be watching the meeting recording on YouTube and um so if if there's any lessons that you learned good or bad um and you don't mind sharing that would be I think very useful for folks on the >> Sure we can we can talk about some lessons learned I think we are deep enough in that um we've made some mistakes and then recovered from them one of the things we had when we first started our current product was we had skew between the different deployments so we had you know two or three clusters and they they were initiated at a slightly different time. So they ended up with a different version of the DRO, different version of the Linux kernel and slightly different versions of OpenStack. Um, and it was kind of a pain, but we could deal with it. But as you get to, you know, um, approaching a different order of magnitude, it's unsustainable. So what we did was we actually um got to a point where we converged uh the exact software version of all of the deployments to one release as we you know we cool we call it and um we also stopped making local changes to machines in the vast majority of cases. So there's in the production fleet there's no machines where the Nova configuration is slightly different because we had a different workload and we really wanted to just do a special favor locally. Um we um we treated as infrastructure as code. So if we need to do a feature it has to either be universal or we have to uh have you know toggles in the metadata for a particular installation. Um but the code is universal. So when we do testing um it's actually representative because we have hardware test clusters. They're small but um the key point is that actually puts some mileage on the configuration that we use in production. It's not one of many or close. It's actually the same. It's exactly the same. Um the main differences we see are test clusters. You can't afford to have them at the same scale as your production installation. So problems of scaling are harder to spot. But of course we can do things like impose a punishing load and push things to their limit like maybe make cues grow, make memory under pressure, make storage slower, those kinds of things. And um I think to a large extent that's allowed us to scale to as I said between 10 and 20 production installations um without a growing amount of um you know uh local changes and local pain and local divergence and inscrutable bugs because this cluster is different from that cluster. We can often debug our problems on the test cluster and then deploy the fix to the production cluster and it's it's effective. Um, does anyone want to speak about how the problem of skew that they've had? Is anyone actually uh running more than one installation and they're different? Uh, I see Fungi would like to participate. Does that mean you want to speak? I I don't use this other than monthly. Go right ahead. >> Yeah. Um, I I don't have a ton of experience operating OpenStack clouds surprisingly. Um but uh as a as a user we see a lot of challenges um related to version skew between different versions of OpenStack. um just because uh we run uh a lot of crosscloud workloads and being able to to have one single client that works with you know a dozen different versions and vintages of of OpenStack APIs out in the wild in public clouds becomes challenging really fast. Um, so I I can I can kind of understand even just from a from a user pain perspective that version skew becomes problematic. Um I I did have one uh related question um which was were there specific services within OpenStack that that you found more challenging to make highly available and and for that matter even I a lot of people have different definitions of of what highly available means for for their deployment and you know where do you draw the line between like is it making sure that the the APIs are are available continuously or do you accept interruptions in in API access and is it just making sure that the workloads that they're managing stay running all the time? What what is what what is your definition of highly available there? >> Ter you want to just take that? >> Yeah, it's actually a really really really good question. Um there are many different ways to look at highly available uh and there are different different designs that you can take with OpenStack that your business may be willing to accept risk of some form where another business may not be willing to accept that risk. I think you know when you look at uh OpenStack the regions feature for example is really interesting in some cases because it's so well integrated with uh a UI like Horizon. So somebody might want to, you know, one business may find it, you know, valuable to go to Horizon and have a single pane of glass or even have a single endpoint where, you know, their entire cloud is a federated uh asset. However, that it kind of means that you are solely relying on one global keystone instance. And there's been a lot of work in the space where if you go look you know a lot of people have had um a lot of thoughts gone into how you operate my SQL alongside keystone in a way that you know provides that uh you know the level of high availability or stability. Um but at the end of the day it really if you do that kind of design you do have a single point of failure in sense where like you are solely relying on keystone and it's supporting services to to drive your cloud whereas other businesses say we just we can't accept that risk at any point and they have totally separate endpoints totally separate clouds. Uh high availability to one person may not mean high availability to uh another. Um beyond that is is your next question. Your next part of the question I think was about like certain parts of the control plane. Um we've also we've gone through you know Chris it's been seven or eight years since we started like the last major iteration of our our design. We've gone through a lot of different parts of the the growing pains experience. Um early on surprisingly one of the things that caught us offguard early was actually uh the Nova metadata service. So when we did our last um major kind of overhaul of the infrastructure and cloud became and OpenStack really became a you know an important you know essential out critical part of the business. um the you know number of VMs grows and as VMs are you know being used occasionally people are quering metadata what's its host name what availability zone is it in all those kinds of questions but that means though that essentially you know as your number of VMs grow so does your control plane load and for us it eventually got to the point where we're trying to figure out like where is all this database load coming from and we eventually kind of narrowed it down to uh the metadata service was the predominant you know source of load uh across the cloud just with the number of VMs increasing on it and simply doubling the TTL of that cache >> gave us so much breathing room to the point we ended up um doing some our own kind of internal changes to the metadata service to to reduce the load uh on on our infrastructure in a way that is not general purpose but it you know worked for us and it allowed us to to scale a little more. Um and beyond a certain point you can use there are existing um things in upstream OpenStack that that can afford you ways to shard out load for things like the metadata service. you can use cells within a a particular region uh to grow you know nova Nova's control plane alongside the number of VMs and instances and things like that I think there are um CERN and other I think believe CERN uses cells I think they they've published some >> yeah we we use >> yeah so there there's definitely use cases out there where where people are are leveraging that kind of functionality uh one of the other challenges that we've seen as you get really really really big is neutron because neutron does not have that equivalent functionality for you know there are no cells for neutron. Neutron is really one bigger instance that has to begin coping with the larger number of uh instances across your um you know alongside nova. And so now we're getting to the point where we're trying to figure out like how do we scale neutron? So high availability and scale it's very it's a very fluid discussion and it's a very hard to to quantify. Uh it depends on how you end up architecting uh OpenStack and what choices you end up making and what you're what risk you're willing to accept. Another more recent um uh example of talent of solving HA Tyler. Um so for unnamed vendor of NVMe over fabric um they delivered uh an overdriver but it was not it did not support HA mode. So they're like you know here we go we we've we've you know we do OpenStack use this. And then we talked to them and said well we're not using a nonHA storage driver. Come on. Like we've been HA with Seth for uh a bazillion years. I mean it's it's like 12 years now or something. Um so they thought about it and they said well actually we can we can support it. It just might not be perfect or you know there's there's this said well we're only evaluating your storage technology now anyway. Give us HA and then you know if we have a bug then we will work with it. You're our partner and we want to get a different storage back end. um into our product and you know not necessarily unnamed vendor but that technology looks very promising. Um so you know um that was a little challenging for them because they're not just able to say here's some software download it install it they have to upstream it. So they actually have to do OpenStack the the proper way which is they need to uh you know meet with the community get their stuff reviewed get it merged and only then were we willing to take it. So um you know it was a learning process for them that um although we wanted to do a a relatively small engagement with their technology they're still expected to both do open source properly and to support HA architecture because a non-HA storage backend is an absolute non-starter for us now like it's just not like what do you mean storage is down like >> they to them ha meant something different I think when we first you approached the vendor. They were new to OpenStack and uh when we asked if if they were HA they were thinking like you know are you providing HA enterprise storage and they were like yes yes yes you know of course we're providing HA storage they just didn't associate the fact that the the Cinder driver has to explicitly declare itself as highly available with regards to allocation and that kind of stuff and it it was a lost in translation exercise that as they became more familiar with OpenStack they were um you know they're able to support it. >> I I correct me if I'm wrong, S, but it's fair to say that their their stuff one of the weaknesses is when they when some of the associated actual storage servers are having a problem, the the storage service also has a problem, right? In terms of maturity, it's a little less seamless than Seph where when a Seth machine is having a problem, Seph just kind of evicts it and uh yes to a degree. I mean this is one of the I would say maybe one of the ch more you know challenging things as you're you know as we try to kind of use OpenStack as an enterprise product and similar things with other aspects of um other things in the Linux Foundation like Seth OpenStack is one of its um utilities and powers is that it has so many features It's so interoperable. Um, there's so many things you can do with it, so many different ways you can bend it and shape it and use it in your infrastructure. But that inherently means that, you know, with all of that flexibility, with all those choices, with all those options, it's hard to get total coverage on, you know, and test confidence across so many different options. Um, so there's always going to be bugs and issues that you have to, you know, prepare for and help contribute back and fix uh as part of, you know, you know, what works for you? Bunky, did that answer your question? Anyone want to sort of ask qu related questions to the topics that we brought up? There are other things on the agenda. Go right ahead. >> That that was great. That that uh that was quite thorough. Thank you. >> Uh another thing I wanted to mention, we talked about skew. Um I always want to be honest about, you know, like the struggles we've had. So we only just retired from production, but we haven't switched off yet our Nova network-based legacy OpenStack clusters that started like more than 10 years ago. And so of course um we did have skew in that sense because they they were they were a evolutionary dead end because um you know Nova network wasn't going anywhere became deprecated and we couldn't drag those clusters over to a neutron based architecture in any way. So and they still had workloads I mean diminishing number but they I think the last workload only got off like within the past month. So um you know it is a problem for anybody. Um you know you do some open stack you get some workloads on it that's it. You kind of took a vow right you're supporting that until that's off. Um we're very happy to say that we're all um you know um caracle set caracle I think and uh neutron with with calico. So um we finally uh we finally got that albatross off our neck. Um we've got other albatrosses but that was that that that infrastructure predated almost every member of the team being a member of the team. So it was increasingly difficult support burden. Um >> okay so >> um >> go ahead if I may ask because in terms of and high level of the service absent we have been running the cloud for many years now. So we have plenty of recipes um yeah that is working for us like lot of things rely on it proxies etc etc but just as a question for the people in in in the call in case they are facing this and we are curious about it now we are migrating to OBVN and this is something that worries us because we are moving from minus to OBN and one of the areas we're focusing on now is how to provide a proper available OBN setup um for Nitron so I wonder if you're already using OBN or in which networking stack are you? >> So that has to be a question for the the non- Bloombergers because we haven't touched OVN ever. I can tell you what it stands for and that's about the limit of my knowledge. I want to be very forthcoming. Um but but please this is again meant to be a working group. If someone even wants to say yeah we're new to OVN and we're looking at it that would be like you know welcome >> because from our side we try to follow as close as possible upstream recommendations like you know you run a cluster uh OBN database you have some layers of u relays that interface between the DBs and the hypervisors but yeah but I'm curious if someone out there with more experience have faced some problems with the setups and they have tuned or refine their their setup in a way that could be useful for others because I guess many people now with the application within splits they will be moving to open vets OBN that is the recommended default upstream I uh I don't know if anyone's willing to share that they even run OBN other than yourself uh someone's saying would like to speak just go right ahead >> yeah we're we're running OVM in our uh stack setup. Uh and that is just because that it was the recommended uh setup when we when we started out a couple of years ago. So far it's been running good. Um I would say it's a little bit complicated to to um troubleshoot. Uh uh but but it's been running stable. We have a fairly small small setup. So, so I couldn't speak for for you know uh any HA setup with OVN but but for for our single uh single cluster with uh not too many machines it's it's running it's stable. Um the main issue is uh when you're trying to debug why does the traffic go the way you expect? uh it's >> very complicated to to look in in all the uh all the tables and trace all the traffic um but but uh yeah when it works it works and it usually do so >> great thanks for the contribution Andreas um we had special constraints uh from our security people that actually wanted to stop our product doing full software defined networking and keep it as some sense as simple as possible in sense that no no overlay network, no NCAAP, Dcap um you know so uh we we use Calico for um like um a layer 3 neutron implementation without most of the SDN the fancy SDN features. So from our point of view OVN looks uh fascinating and full of toys but I never love to play with it. Yes, the we are starting to to embrace and adopt all these fancy features and also discussing as well with our security team what the implications from the point of view will be for for many of these things because yeah things are more complex but many things are encapsulated that so everyone should have a good visibility of what's going on >> so one thing um I would love to do is um get more guest speakers if you will or guest participants um but I don't even know an OVN like advanced practitioner in the OpenStack space. So if anybody does know somebody um and you know they could reach out to them and see if they would join for a different session where that could be a focus because um yeah it's it's a unknown territory for me. Um >> I think from from the from the experience I had in the summit in Paris I think the people from stack it they have quite some expert in the area. So I think maybe that would be a a good community to to maybe one one month come here and and discuss a little bit these topics. >> Okay. I mean I' I'd love to coordinate with them, invite them try, you know, try and uh demystify this and um you know uh so if someone could put either could reach out to them or give me contact info, um I'll try and make that happen. Um so uh that's quite a lot on the first item on the agenda. Uh so I guess I'll ask for any more questions, comments, observations, experience with uh OpenStack HA in particular. Um >> the only thing I'll add to the OVN discussion, I'm with Chris. I don't really have experience in in OVN. uh but more to say like when you're do when you're talking HA opent stack uh moving from one technology to another or drastically changing how you deploy parts of your stack it's a it's a big lift and shift so when you're do talking about HA deployment when you're looking at OpenStack for an enterprise use case be be very intentional uh do scale testing think about how you're going to use it see where the the limits are and and size things appropriate you know design your architecture to size things appropriately. Uh because the last thing you want to the last place you want to find yourself in is starting to grow into OpenStack or starting to grow into an OpenStack deployment and finding out that you know you're out of you're not able to scale uh some part of the control plane or some aspect of OpenStack alongside the needs of your business. Uh I think one of the big my understanding of OBVN is one of the big reasons it came too was because that there were uh scaling issues with OBS as it grew into a very large uh control plane which you know going back to the earlier point about when Neutron gets very big it can get it can get hard to scale and so you need to intentionally see where those breaking points are and and you know for your use case and be prepared to deal with them ahead of time because retroactively shifting things that are in the air can be hard. >> I agree with that and that actually brings up all of the pain of the older cloud which I won't bore people with too much but for example it used tagged VLANs which are finite. It used the layer 2 span which means that you have to have large switches and you eventually run out of ports and um the way that Nova network worked is each tendency had an IP subnet. So you had to decide how many subnets do I have which is how many tendencies you have and how big are they. So we had to have some small, some big. Lots of people got big tenencies and put two VMs in and other people got small tendencies and filled them to the max. And we abolished supporting any of that I think literally two weeks ago. So phew um okay uh great great discussion about HA and as Tyler alluded to we could never get to the bottom of all of it but um you know as as previously if people have specific questions after this uh you can reach out to me on um on element well on matrix I guess element is the client uh we're trying to get that chat room off the ground um that seems to be something that the community thinks is okay other than I mean much better than for instance, the you know the old sort of Twitter community which is dead and gone. Um we are again just some people heard this but our our project is not currently open source but we do want to re-release it in a in a a form we're proud of in the future including a buildable HA version. We did have that for a while but some bits of our stack became non-free due to vendor changes. So uh but um as of right now you know we are open to talking about how we do things if they're interesting. Uh but I do want to move on through the whole agenda. Um so OpenStack backups um I can take this question for us. We do the most obvious and stupid annoying thing. Uh apologies to any anyone who thinks this is great, but basically we don't provide it in the cloud, but uh VM owners can talk to the actual backups team and they install a net backup client to install it and then send their data out to the existing corporate data protection complex and ultimately that has I think a staging area on disk and then giant tape robot libraries which I think is super cool, right? But what's what am I contrasting that with? Well, there are continuous data protection things that plug into OpenStack and become self-service. You're like, "Yes, protect my data." Uh, I won't name vendors. I'm not supposed to favor vendors, but um I think some of that stuff is really cool. I wanted to go there, but um you know um we already have a backups and data protection team and uh we also can't afford to back up the entire cloud. Um I think that they wanted it to just be teams who go and talk to the data protection people and then do the normal thing. Um, so if people would like to volunteer something maybe more interesting than that, please speak up. Um, you know, I'd be interested to hear um other other things or you know, you can probably talk about vendors more than than we are permitted to. I don't know that could be true. Um, if if the person who asked the question is here, I could ask them. Did that answer the question? Maybe they're not here. Sometimes you have a problem that people put a question on and then they don't they can't make the meeting. So we can't actually close the loop and make sure that they got the answer. From an infrastructure engineering perspective, I would advise people it's probably a better option uh to leverage the capabilities of OpenStack and and design, you know, going back to designing infrastructure intentionally. Uh you can avoid full backups by you know intentionally making sender volumes and putting your data on those sender volumes and using cinder volumes to carry your state as you need if you you know use the instance in a um declarative fashion. So if you correctly model your infrastructure so that you're you know able to just rebuild a VM separate and have the VM's configuration the software and everything separate from the data you can then your backup story becomes easier because your data is separate from your operating system from your operating system configuration. You can redeploy uh your infrastructure, patch it, secure it, test it, uh stage it, all those kinds of things separate from your data. You can back up your data uh and manage it just through uh Cinder at either the block level at either the the you know database or file or object or whatever you want to do. So, you know, full system backup, it it may, you know, I I would probably steer people away from it unless you really really have a need for it or, you know, a legacy use case. I will just argue with Tyler, which is weird, but um uh keeping your data safe and your application working on the cloud the way he said is a great technique, but it's not technically backups in the sense that it's not an offline disconnected uh thing that would would be compliant with um regulators who need to hear about tape and storage and and and those things. Um we do have some of that. Mostly what people want is I want my data to be safe and if I have a hardware problem, I want it to be rescued and I want my application to keep running. actual backups. You know, the regulators need to sometimes see that you actually have tapes on a shelf or dis replicas that are disconnected so that you know, for instance, if you had ransomware loose in your network and encrypted all your apps that you could actually recover from an earlier snapshot, that kind of thing. Um so we don't have a lot of um interaction with that because basically the app owners if they need real backups it not necessarily tape but you know tape like disconnected immutable um they just they install at the application level uh and the other thing about Tyler's answer of course is having a cinder volume where your precious data is kept is is of course u very resilient if your storage is very resilient. We run Seph with four replicas which is kind of expensive on on NVMe so that uh we just don't we really don't lose data if it's in a cinder volume. Um we did in the past have ephemeral storage which could go away if there was a hardware failure but exactly the wrong set of clients used it. So we withdrew that because they actually lost data. And one of my favorite um you know stories over a cup of coffee is uh the conversation where someone said Chris we're friends right? Surely you have my data in one of your secret backups. I like we are friends but I don't have secret backups. I'm very sorry. Um you know when we said your VM is going away it went away. So um okay so trying to move along if that topic is is done. Any any further comments? Uh, how do people automate their clouds? Um, I think our answer is all Ansible all the time at core, right, Tyler? Um, there's many other things to be said, but um, we used to do a mix of Chef and Anible. Um, Chef is going by the wayside. Uh, but we have our own um, take on it. Um, not the OpenStack Anible playbooks. they kind of were opinionated in ways that clash with other architectural decisions that we made or had imposed on us. So we have our own um and we're not containerized. So it's kind of traditional in that sense. Um anyone else want to talk about how they automate their clouds particularly if you're running a large one. The other thing that I'm sorry >> the other thing that came up with uh Anible I think when we started out using OpenStack it you know Anible was a great first step tool in terms of you know provisioning infrastructure as code and treating infrastructure as code. I think one of the challenges we had with Anible as we you know the size of our cloud grew is eventually your ability to maintain your cloud you know you you can't you stop being able to rely on workflows that are manually orchestrated. So even you know as you have lots and lots of clouds running a deployment across lots of them with anible and having complex deployment and release processes it becomes unwieldy. So for really really large organizations or if you have a lot of clouds, you'll probably find eventually that it's necessary to invest in um you know how you actually use Ansible, how you deploy Anible and putting an API in front of Anible so that you can begin kind of gathering metrics on your deployments and and things like that and treating the actual Anible infrastructure itself as code. I I believe uh is it Luis was was trying to get a word in? Um just go ahead. >> Yeah. Um because from my side I'm curious at the level of managing OpenStack not only the infrastructure but how people automate the I don't know how to call it like operations business processes things like how do you handle new projects? How do you handle when things go wrong and you have to evacuate machines? What sort of uh workflow engines do you have there? In our case, when it comes to infrastructure, we continue to use the same tools we use at the beginning. So, a lot of puppet involved. Uh but then on top of that, when it comes to interface with our repair team in the data centers, we have I don't know things like run jobs where we are able to delegate task to them. So, they're able to train our hypervisors, put things out of production, bring things back to production, these sort of things. And then we could be using as well things like I don't know mistra workflows to also automate some other processes to make our lives as operators a bit easier. So I'm curious what people use for these sort of things beyond an or maybe anible as well for some things but apart from the the deployment of open stack itself more like the operations on top. >> It's a great question. I see um Franchesco has a hand up. Go ahead. Just my two sh two two cents on automation. Um we are using forman for the full life cycle management. So instead of anible we're using for and puppet. >> Great. Thanks for the thanks for participating. Um yeah we actually do use sworn. Um for us uh there are layers of um automation that are not critical to the core operation of the cloud but there are things I think mentioned like evacuating hypervisors. Um and we do have um in the cloud reliability team is called uh a lot of airflow. Um the nice thing about that is uh it does integrate with um our anible but it also integrates with nonopensac systems within Bloomberg. So for instance, it can create tickets for a team who are the owners of a tenency. Um and um I think I also heard a question about how do we handle tenencies? Um that again is um basically an integration between our OpenStack and other Bloomberg systems. So it's kind of further away from my area of expertise, but we did have to make that basically self-service. Um ultimately we none of us have done you know like openstack project create for an end user in 10 years. >> So so I guess it's similar to our case because in our case for instance that workers will be some many in-house uh pling between uh in our case open stack and service now that for instance is the platform that people would use in in our uh in our organization to request new projects in open stack and these sort of things. I think I heard service now. Yeah, I haven't uh I haven't used that. Um again, when you when you share a unique perspective on these things, if you if you'd like, you can put drop it on the ether pad here because there are people who come and read it after and get answers or at least think, you know, find out what we talked about and maybe it spikes some conversations. Um so basically we've done we've got Anible, Airflow, Puppet, uh Foreman, the usual things. Um so any any more on the on the automation thing? I mean obviously it's another deep topic. Um and how are we doing for time? We still got time. Uh so the last item um or last substantial item update on ops radio survey results. Um I wonder Elica would you like to take that maybe? >> Yep. Yes. Uh so there's a link on the ether. Um I don't know if you all want me to try to share my screen and show what's in there. It is the email that I sent to OpenStack Discuss. So if anyone is subscribed to OpenStack Discuss, uh you probably received it already. If not, then the link itself is pointing to the archive with my summary. So, we closed the ops radio hour survey this Wednesday, two days ago. And um if anyone hasn't seen it, then it was targeting to get information on where people are finding the meeting series and what time would work for folks. um what cadence and um also what topics people would like to discuss. So um are you all looking at the uh the email summary uh through the link or do you all want me to try to share my screen? I don't know how that works with the recording though. Have I'm not sure if I tried that before. Um I mean if you talk through it I'm sure people can go and get the details. I mean do you have a a take on the sort of like key findings? >> That sounds good. So in terms of where people are finding the meeting series um OpenStack discuss mailing list was mentioned most frequently um almost 50% of people and then the rest is uh the scientific sig slack workspace was mentioned we also did the opening for foundation did a direct email outreach to send out the survey links to operators So that was mentioned. The matrix channel was also mentioned which I was very happy to see. Um and we had uh Reddit in the list and then OpenStacks Summit and the the YouTube channel. And um so we have a little mix of um the the main communication channels for for the community, the OpenStack community, and then some other um ways how people find the meeting series. We did ask whether or not the current time slot works for people. um 80 almost 87% of the responses said yes. Um with only 13% of responses saying no. Um in terms of preferred cadence, bi-weekly and monthly are in a tie with um over um like what 43.5ish%. And then weekly came up and every two month came up as other answers for preferred time slots. The current time slot is not the absolute winner. However, it is in the top three. Um 13,4 and500 UTC got the the highest votes. So my personal assumption is that people are respondents are likely in a bigger percentage um maybe in Europe um or east coast of um North America or something along these lines. But um I think every single UTC slot got some votes. Um and if you look at the um the email uh summary then you will see that there is a little bump in the 01 and 2 UTC and then it goes down and it starts to come up at 800 UTC and then it grows um it goes up uh and the highest is400 UTC that got the most votes and then the the curb curb curve starts to go down. Um and um the the absolute low is yeah towards the uh the 22 2300 and then the 3 4 500 um UTC. So we have folks all around the globe who are interested um in participating. um 87% of people would support to run these gatherings in um in an alternating time slot. So this is something that we could try and in terms of topics that people would like to cover that was a very wide range of everything. Uh many people would love to hear real life experiences. Uh what are the pain points? what are the bugs and issues and how people got around them and got them fixed. OP oops uh war stories um came up because people like not to feel alone with any challenges that they are facing. Um automation that we just talked about today, backups and disaster recovery and what do we do now that freezer is dead. Um HA also came up. monitoring was mentioned multiple times. Um documentation is also something that that came up. Um one thing that I personally found interesting that things like uh training and skills development for operators came up as a topic idea which I don't think we explored too heavily in the past. So that is something that I found very interesting and we could consider the group could consider to put it on the agenda and also um getting involved and contributing to OpenStack as an operator came up which we touched on a little bit but I don't think we um we dug into it very very deep and then things like deployment methodologies, configurations, upgrades and patching and how to automate that and how to do that um those are the uh to me uh less surprising topic mentions because these things come up um relatively often. So there are a lot of things that people would like to talk about and um this time slot is in general not a bad one to stay with but we do have options to explore in terms of hopping around uh time slots a little bit and whether we um do bi-weekly or monthly um that is uh a question to the group to decide but in terms of survey respondents they would welcome welcome either of those options. So, this is roughly um how the the survey um responses turned out and all of this um is in my summary email on OpenStack Discuss and I will share the the >> I wish I could but I can't help with emails here. I'm sorry, my my home assistant just talked, so I don't know why my pod. Uh, that's great, Elico. Um, so I think a key uh limitation of doing these bi-weekly would be um hosts. Um, this might seem really self- serving, but I've done the last few. I'm wanting to try bi-weekly, but um I don't think I can get to weekly. Um, >> oh, people don't want that either. Um it was like maybe it was a very low uh percentage. Let me see. Uh yeah like eight eight n% which was um technically two um two people saying that weekly could be a good idea and every two months was one person. Um which was representing 4% of the responses. >> Um it sounds like um a rough consensus that we're doing okay for now. Um, I don't know how to see whether we should go bi-weekly experimentally. I guess we could set the next date to be roughly two weeks from now and see if we have uh involvement. Um, I haven't actually looked at the calendar. So, um, did you have a suggestion for the next uh, meetup? So, um I mean do you want like two weeks from now is uh let me go back to my calendar. There we go. Um is March 13th and then if we keep the monthly cadence that would be March 27th. I don't know if folks from this group are going to CubeCon EU because um that is that week. So if that is a concern then we could um do March 20 which is between the the two and four weeks um from now. So those those are the options. Uh, >> so do you do you have a meeting on Friday the 13th is I guess the question for the bi-weekly cadence first. >> I mean u the people who who um joined today should probably speak up. Um I think the the options that Eldico put forward are get I mean run this in two weeks time in four weeks time or split the difference. Um I can try and run it. The other person who has run some of these is I think busy today. Um but uh it's really kind of a question for the community like how often should we try and do this? >> So people on the call um are you interested and available and have topics to come and get together in two weeks on March 13th? raise your hand or say yes or >> um I if I could speak that I would say that there isn't doesn't seem to be a strong mandate from the people here to change anything. Would that be fair? Does anyone want to say that they disagree? Um so maybe we should just go monthly for now and then I think if we if we started to have too much um content then we could say let's just do more meetings. Um, so I'm seeing some thumbs up now. So maybe I'll take that as people agree with what I said. So, uh, is the time in in four weeks clashing with Cucon EU? Is that what you said? >> So it's the it's the same week. I think the conference ends on the 26th. So if people are going to EU, then they will likely be traveling home on Friday. But if this group is usually not attending that conference, then it doesn't really matter. But it's a bigger conference um and it's in Europe where at least some of the folks on this call are so it it it might be something that people attend. So that's why I wanted to bring bring it up to make sure that we're not colliding with it. Um if it's a if it's an issue for this group. Is anyone going to CubeCon Eve from from the folks around today? Seems like cricket so might not be an issue. Um, >> so Friday the 27th seems to be unopposed, should we say? Is that is that accurate? Um, okay. So, not much response from the room, which doesn't actually prove anything, but I saw one thumbs up. So, uh, March 27th, is that what what you said as >> I mean that's technically the the last Friday of of March. So, that would be the four weeks monthly cadence. Yes. >> Okay. So, um, we're running out of time or let's let's say we've we've filled an hour, which is great. Um, so let's let's let's put an asterisk on that and put it in the matrix chat. Uh, we're thinking this date. Um, if people want to make Ops Radio and that's a bad choice. We're open to moving it. Let's just see if that sparks conversation. If not, I'll put it up at the top and put it on the calendar and uh and plan to to run another one of these in a month's time. if someone has contact info um or wants to talk to an OVN practitioner about you know joining this as a guest speaker the way Tyler did today um so that they can get more into you know how their experience is how you scale it um that would be great but let's see if we can actually get the guest speaker I think this is the first time we've had an official guest speaker so I want to say thank you to Tyler um Tyler and I have worked together for years but you know I asked him if he would do this um and he said yes so He's a volunteer. Um, so please please thank him. >> Sorry. >> I always forget about the noise. >> Um, no, I actually I I'm glad to hear the room's kind of coming alive. I do emphasize this is this is meant to be a working session, not a presentation. If you gave me a score for as a presentation, I know it would be very low. It's meant to be we get together, we talk. So, um, thank you Tyler for joining and sharing your perspective and, uh, I'll say any other business because we've got about 2 3 minutes left. Any feedback on on this session? Would someone like to say that what they what they learned or was it interesting? You know, speak up or press the emoji button if that's all you're comfortable with. I actually enjoyed that. very interesting to slowly see the slowly be more uh engaged with community and better understand the perspective the community is moving with because I am quite new to participation in the community direct directly. >> Yeah, I would say we're new to our OpenStack journey. We're still just starting out, but I definitely find these sessions useful and very informative today around the the HA stuff and and all of those. So yeah, any more deep dives on that is is greatly appreciated. >> Okay, I love the feedback. Um, you know, it makes it all worthwhile. So, uh, I'll do my best to get an OVN practitioner. Um, but I don't know any. So, you're going to have to help me a little bit. Um, I will ask other people, you know, we maybe we can drum someone up. Um, I think the guest speaker makes this uh, you know, uh, really great. Um, I'm thrilled with it. So, we'll try and do more of that. This was actually the sort of vision I had for it. When we set Ops Radio, it was meant to be like, you know, a host and a speaker and then other people can kind of phone in. Even though we're all actually on Jity, um, this is actually what I had in mind. Um, so let's keep let's keep going with that. Seems to be something that people enjoy. Uh, we are we're an hour in, but not quite at the top of the hour, but of course people need to grab a coffee and things. So, I think maybe we should call it. Ildico, do you do you agree? Should we stop the recording and say? Yes, I think we are ready to do that. Yes. >> Okay. Thanks very much folks and see you next time. >> Thanks. Thank you. Have a nice meeting. Bye. Bye.