Foundational Infrastructure Working Group, August 6 2026

Okay. Then uh hello uh welcome to the dash infrastructure working group meeting 6th of August. So let's check the agenda. Um so we have one topic paid runners for PC run open STCI outside. Uh-huh. CC environment. Who brought this topic up? >> Hi, it's for me Sasha. Hey. >> Hi. Should we start with that or >> usually we start with the topics on the agenda and then go over the dashboard to some uh don't uh u push people wait until we go over the items. >> Okay. I I try to uh make it short. Um so Reuben um mentioned that um we can check microstack to see if we can use microstack to uh run the um OpenStack CI in GitHub actions. Okay. And um yeah um I started this P for us. Uh the first finding was that a micro ST is not maintained anymore. Instead we uh then continued with devstack. But apart from that we um we did workflows on uh GitHub actions to see if we can uh do stuff like um uh deploy bosch um create um download stem cell, upload stem cell, create a VM out of a stem cell etc. Okay. And we were running in uh some problems. So um uh my latest runs for instance the deploy bosch wasn't working. Another problem was that um create an VM out of in stems is not working. A lot of nested virtualization problematics. Um I also switched from KVM an open stack onet virtualization to Kimu which is an emulator. I also ran on problems when I switched. However, I have now two questions. Um, Net supported also here. Um, regarding the two questions, um, Ned and I, we uh, tested and used, um, the free runners of on GitHub action, we wanted to ask if, uh, the if paid runners um for GitHub actions are available on the Cloud Foundry repo or Ned, maybe I've um, I miscommunicated a question. maybe a few community. >> You're correct. Yeah. Um the issue with the free ones is um they're using chemo which is very slow. So we weren't able to get a director running. It's always timing out. Um question is maybe we can increase the timeouts with more time for it to provision. Um >> well this is it my end >> sorry >> uh the the timeout I played around with with so >> um maybe it's not running at all um yeah I haven't tried increasing yet >> um so an alternative >> I I would have a question have you tried locally the setup and does it work on um um virtual machine or on a laptop. >> No. Um I only did it as a workflow on GitHub actions. >> Maybe there are other issues which are uh somehow hidden and we think that it's the uh uh free runner. I would suggest that this you get a well feeling how it works the setup locally on a machine which will uh um and what the requirements for that machines are uh regarding CPU run and then we can have concrete discussions whether this can be run as a GitHub action. >> Yeah. It's a point I take with me. Um, however, the the other point the paid runners. Um, is there a possibility for us to use paid runners on Cloud Foundry because right now we use the free version and doesn't even make sense because the whole do it in GitHub action um approach um is u because and I don't want to uh miscomment now Reuben. So apologize if I might miscomment him. Uh the whole uh backlog item was because GitHub actions are supposed to be cheaper than Conquest and I wonder and Ned as well wonder if it's still the case if we switch from free runners to paid runners. >> Uh we have to uh address this question with Chris Clark the program manager. I he has more contractual details about our enterp GitHub enterprise license. I can not answer the question >> but uh I I mean uh we can do to um I don't know how uh whether we can in parallel ask the question to Chris or we need first a good understanding what we need here and then ask the question >> can you forward the contact to me and N or should They do it over you. >> I'm going on vacation uh uh beginning of Monday. That's why most probably I can get a bottleneck here if we do over me. >> Mhm. >> Um yeah, just to clarify, um I think we can run most of the pipelines in GitHub actions. um like bumping the dependencies, unit tests. >> Um the issue we faced was particular with PSTs because it requires a director to be provisioned and VMs from the director. >> Uh Reuben highlighted though that especially the bets are in focus here. Um, and what Sasha mentioned is if um we're unsure if the if we end up having to use the um paid GitHub runners, is that going to be preferable over concourse? >> We can answer this question when we have uh understanding what is required here. maybe one uh virtual machine which started for one hour to validate and then shut down again is not so much of a problem. Um basically the bets need to run through. >> Yeah, >> this is required. Yeah, I think as far as coordinating between concourse and OpenStack that's or and uh GitHub runners that's it's not straightforward but it is fairly it's done a few times. There's different ways to have hooks or um either GitHub actions triggering by concourse or vice versa. So I think for the things that don't require IAS access um those could those could live in in whichever environment suits the team the best. Um we the the the root issue it seems like is we need an IAS we need an OpenStack booted up. Um, and that could be something that we trigger um, in Concourse if it's possible to boot like a single VM microstack. Um, and that could be targeted either from Concourse or from GitHub actions, I think, depending on how we set up the networking connectivity. Um, so yeah, I don't I don't know. So I don't know whether using a paid worker or not through GitHub would would help us necessarily but I think I think the the the root problem to solve is how do we go from having no OpenStack IAS running to having an OpenStack IAS running and um in the case of other IAS's we those are just public cloud and we can we can ask for instances to be booted up and use bubble or other mechanisms but because we don't have an IAS is booted. We need to to figure out that it's like a pre-bubble up sort of step that will give us a working open stack that we can target with for example bubble. >> Does that make sense as a framing >> um the open stack environment I [clears throat] think was resolved by Sasha. I don't remember the name of it. >> It's dev. Yeah. And I think that's able to provision fine. The we run into the issue when trying to get um Bosch running actually. >> Is that because Devstack is not performant enough on the hardware that it's running on or is that some other issue? >> I think it's the virtualization. um the free runners use um Kimu compared to >> um the paid ones. >> Um so I was running into the timeout issue. Um Sasha mentioned he tried playing around with it but still no >> is is so that devstack is available from from a provider. >> Yeah. as a GitHub action >> as a GitHub action. Okay, good. Oh, and then that's why the page runner Yeah, would make sense. Yeah. Um I don't know. I think it's probably a a Chris question in terms of costing. I think the the the other possible way of doing this would be to boot some version of devstack ourselves with scripts in GCP and then have concourse workers target that for the bats. I don't I think that's probably the place we want to check costwise whether it's cheaper to pay for a GitHub paid runner with the faster OpenStack or And there's hidden cost in here of actually doing the work like write a write a a concourse some concourse tooling that would boot a VM and installed install devstack on that something like a devstack on that and then allow that to be targeted by the either the GitHub actions or the the concourse jobs. Mhm. >> Um I I don't have a good sense of cost either way, but that will probably if we're if we're trying to look at this through the lens of what's the most costefficient way to do it, I think the comparison is going to be between the time it would take to to boot and run a a sufficiently large machine in GCP with a devst with something like a devstack orchestrated by us installed on it versus the cost of running that worker. um in GitHub actions. >> Surprisingly, defec was, if I'm not mistaken, using around 500 megabytes of RAM, >> so it's not really um that much consuming. >> Yeah, I don't have concrete information. >> Yeah, I don't think we need really precise numbers, but if we have ballpark like if it's If it's 2x cheaper to run it, to like write a script to run to to boot a VM that's big enough in GCP and install Devstack on it and run that, maybe it's worth the effort. But if it's if it's like 10% more, it's probably not worth the time and we we would just want to pay for the the GitHub action worker. Um, what do you think about that, Ben? >> Um, I don't um I don't think that we have until now paid workers. We have to explore what does it mean for our contract? >> Yeah. >> Um, >> is that something you could ask Chris about? Um, just to find out if it's possible. >> Yes. Uh what kind of work paid work do you have any configuration or >> I think it's custom sizes there. Um what we're mainly aiming for is the virtualization KVM support >> but you say 1 GB and two CPU. So what are the configuration rough configurations? >> I I it looks like devstack is not the main consumer. So I'm I'm unsure about Borch. If you run the B test, we'll need a directory setup and I guess one or two VMs at the same time. Yeah, >> I would guess we want at least half as many CPUs as we want to run VMs. So probably I would say probably four or six CPU cuz we want one CPU to be able to manage the IAS itself and then at least to have a CPU for the Bosch director um and all the worker processes on that so we don't have contention um because we have to remember that that that one virtual machine is going to be running all of the OpenStack orchestration layer plus all of the Bosch orchestration layer plus the I guess the test code won't be running there but then any VMs that get spun up or torn down as part of bats um will need to be fast enough that they don't stall out um or or cause the the open the underlying OpenStack VM creation time to to spike. So my guess is probably four minimum Maybe we could try with two and see how that works. >> And around 68. >> Uh, it sounds like dev stack was 500 megabytes. Is that right? >> So maybe >> Yeah, don't quote me on it, but um yeah, during testing that's how it looked like. Yeah, it might be interesting to to like like Bon was saying like maybe we can get this working locally or if not locally then than then as an experiment in in GCP or something um till we get to a point where we can run bats against a devstack instance and then understand what the performance requirements are there and that would and let us know either what what it's going to cost to run that in GCP roughly because of CPU size and memory or what kind of a worker we'd have to request from GitHub. >> Yeah. Yeah. My point was to be able to iterate fast and get understanding of the setup it is is it working reliably or not? How what are the requirements regarding CPU and memory? and then we can go from there because triggering a process to buy workers and so until before we know whether it's a reliable setup and what are the requirements. Um >> yeah um unfortunately we can't test it on a paid worker to confirm that it's going to work. Um, Aram, do you think it's worth the effort to set it up in concourse in GCP? >> I don't know necessarily. I don't think I think before setting it up in concourse would be fine, but I think my thought was can you can you manually boot a devstack virtual machine in in GCP and then and then run run the bats from your from your local desktop or from a from another machine in in GCP just simply to validate that that that that some to figure out what configuration of devstack will allow um the bats to run successfully. >> Okay, >> does that make sense? Rather rather than trying to set up the whole pipeline, just just see if you can like >> manually get a devstack instance running in GCP and then kind of manually or or have a script that that runs bats until we get that to pass successfully. And that will hopefully give us a sense of what what size devstack instance we need and also maybe give us a sense of the complexity of having to do that in concourse if if the paid workers for GitHub uh don't work out somehow. >> Okay, >> with that with that we can calculate also the cost. If they are I don't know below let's say one two 300 two 200 per month uh then I think we can just go because it needs to run only when we release or uh do new commits on open stack >> not so often the VM you can roughly calculate the costs. Yeah, I mean we could the pipeline could be set up so that bats aren't automatically triggered on every commit and we could just we could that could be a manual trigger which would then um either run the run the paid worker or you know boot boot the VM in GCP. Um [snorts] I'm thinking mostly in Concourse cuz I think mostly in Concourse but don't I'm not this isn't me trying to say we should absolutely use Concourse necessarily. It's just the the language I'm using because it's faster than saying both. Mhm. [clears throat] Okay. So, we need to um deploy OpenStack and the B director in GCP. See um if it's working. Yeah, I think so. I think once you get OpenStack running, I hope I don't know if Bubble works for OpenStack, but maybe you can just use BBL to you can point that at the OpenStack instance and have it bring up a director for you. >> Mhm. Um, >> okay. >> That that might be the way to go. >> Um, >> also like uh download upload stem cell create VM out of the stem cell or is it really just Okay. Not just it's open stack and bosch. >> Yeah. I mean I think Yeah. So I think for the bats you you need a director present and the way a lot of times we get that is to to bubble up and then once you have the director then whatever the bats do which I think are you know most of the Bosch operations um there's probably good good examples in AWS or GCP about how to how to set up an environment um once you have the IAS credentials to to to run to run the tests. Okay, so sounds good to me. >> Thank you all for tackling this and good luck. I hope it goes well. >> Thanks. >> Okay. Uh Aram, you mentioned you are going to add other agenda points. Do you have something? >> Oh, no. I was just updating the agenda and making sure this this thing from uh Slack chat was or from Zoom chat was in there. >> Uhhuh. >> I don't I don't have anything. >> Anyone else having something else? >> Yeah, very quick announcement. Um we spoke about it before. Um, we have issues with the GCP CPI pipeline where we get an error permission denied. Um, I was finally able to isolate it. Uh, it's a concourse edge case I guess. Um, I found out we have um met GCP metadata users that are created on the VMs on the workers and they are not created in the same order. So when Concourse is sharing I mean they're not you have they don't have the same UIDs on each worker and when um Concourse is sharing images between workers um we got some of the home directories with the incorrect UID um owner. So this was merged in concourse the fix and I'm assuming when we they push out the version haven't unchecked and we update this should finally address the issue. >> Cool. Um I I ran an upgrade yesterday and we got uh 8.25 I think or 725. Anyway, the we got the latest version of Concourse as of yesterday. So I I think maybe we pulled in that fix unless it's more recent. >> Yeah, I'll take a look and check. Thanks. >> Sure. >> Okay, then let's go to the dashboard. I'm opening first all the pending to be pending for merch. Okay, this is the implement snapshot based disk type conversion for the disk uh CPM. >> This one um needs a review. Um I have asked Sebastian who's working on Azure from our side to take a look as well. >> Should I assign Sebastian? [clears throat] >> Uh yeah. Thanks. >> Okay, I updated the state. Speed up integration tests. Uh so C rabbit is fine with it. Chris uh works by Chris. All comments from Chris has been addressed but there is no approvement. >> I'm happy to approve this. I had a walk through yesterday. Um >> okay >> the the gist is that there were um so basically two two things changed. some in some places we didn't need to do as much tear down because uh the tests weren't mutating state necessarily of the um of the integration environment. Um and in other cases there was um uh there was some oh this and also we we're we're saving the runtime information from RSpec about um when uh which tests are slower which leads to better splitting across the the end workers so that we end up with a a closer a shorter like a a tighter time bound. where sometimes we would have like one work one one set of processes would finish in 5 minutes, the other one would finish in 20. Um, so it should even out the work between the different threads. >> Mhm. Okay, [clears throat] I'm going to merge right. >> Does it have an impact on the cleanup? I mentioned some of the >> um the the cleanup is only these are integration tests that run on a single VM. So, it's just uh directory temp directory cleanup stuff that happens. Oh, >> okay. Let me see. There it is. Oh, we have a lot of approvals. Yeah. [laughter] [gasps] Or approvals. Okay. And uh I don't have any other option than merging this Asia trusted launch security profile. This needs um review Bholder left over some more automation. Okay, you walked into this um so boot still uh leaks resources. I notice again last week when a new boot release has been done, there were again orphan bubble environments. But somehow I didn't have time. Is this improving the somehow the cleanup left over? >> I think this adds some cleanup to >> Yeah, this this adds an onfailure hook. I think that >> Okay. and then and then pulls pulls the behavior out to a function. >> Yes, this is more synchronized or uh centralized. Uh >> yeah. >> Okay, this is good. Maybe helps more. Yeah, I'm thinkful changes. Um, this one we can skip for now. I might make this I might make this um I'm working with Colin on this. This is a um some tooling we put in place for being able to test uh the Golang replacements for NAT sync and um in the health monitor. Mhm. What is the motivation uh behind reimplementing in go? >> Uh both for both NATS and these are there are open uh draft PRs for replacing a rewrite of NAT syncher. >> I saw those >> in the uh motivation is NATS tends to we we've been using the Docker CPI and we see the NAT syncer show up as as a really high CPU consumption. Um, and also just the these the two processes don't interact with the database. So we don't have to worry about any of the director SQL OM stuff in Go. We can just run the processes and um hopefully reduce some memory and CPU footprint on the VM. >> Okay. Yeah. >> Well, it is in lower. >> Okay. This is about the trigger back port. I also worked into this comment last week. Um, >> yeah. >> So, >> I think yeah, let's see if anything comes up from there. >> Yeah, still waiting for changes. Ubuntu noble. Um so this is the fips contribution for noble fips support for >> I think there is a there's an open comment from code rabbit and somebody asked the person to address um that so I think we're just waiting for contribution on this >> yeah [clears throat] we'll wait just Okay, Chris already walked into this and has comments waiting for changes. I was surprised um the person is um I don't know from where Yeah, I don't it was nice to see somebody contributing but I haven't I didn't recognize them. >> Yeah, >> but yeah, the state is we have assigned Chris is assigned and waiting for changes. Okay. Pending preview. [clears throat] We talked about this one. So, we discussed this last time. Um >> yeah I I think it needs at least one more review. >> Yes, this was the conclusion that we want to have another review on this. So it is uh pending the state is correct. Um, love to see some reviews on this. Um, >> Mhm. >> Yeah, this should >> Yeah. Okay. and uh also in correct state. This is also waiting for review. Once I get a review or two on this, I'll I'll um uh there may be some some challenge to merging this because vendoring a package requires blob store access. Um but I think if I just run it through CI once and the bumper will then automatically synchronize the blobs um into the blob store and it should be fine. >> Mhm. Um, these were some co-pilot recommended changes for for fixing workflow security warnings which seem to not impact the workflow running. So, seems fine. There's there's like three or four of these. >> Should we just match those? So, >> I think it's fine to merge the workflows have run successfully. So, >> okay. Be right back. Just a second. And this one we also set now we want to have another review on it. >> Um >> okay >> it's a smaller change but someone else has to say Google. Yes, in a very special place. [laughter] Um, >> yeah, this is also waiting for review. >> Well, we'll try to take a look at that. Um, >> this one. Uh yeah, the one with the one in um in config server. Yeah. >> Okay. Yeah. Server. This is also waiting for radio still. We talked about this trust touch and put in review. Okay. Then um we have couple of bumps here. Um >> um yeah, some of these are a little I'm a little confused about them. I'm the the Bosch bootloadader and the Bosch disaster recovery test should be autobumping. So I'm going to look into that those pipelines. Um, the Bosch io worker code I don't believe has autobump on it, but I I don't know that we have good tests in GitHub actions for these. So, I'm I've been a little reluctant to bump those. They seem fine, but I don't have as much context on the Bosch IO codebase as other things. Um, we'll look at the disaster recovery test and see if I can um figure out why those aren't autobumping to a newer version because the pipelines should be doing that. >> Mhm. You mentioned porch put folder and disaster recovery. Yeah. >> Yeah. And there are a couple of open stack related maybe someone from SAP can look into those. >> Yeah, I think the OpenStack one is failing I think because the the GitHub actions are just failing full stop. Um, oh, and then the JSON bump there, I think that's just a I think that's a bump to the JSON gem. So maybe that doesn't need to be once once the Ruby OpenStack is deprecated, it's not important. >> Yeah. Mhm. I will leave them in the inbox then. >> Cool. Thank you. That seems good. >> This one is the only new. Yeah, this one I'm working on reproducing it. Um because it it was supposed to be fixed. Um so Aram, we discussed this last time. Um for some reason, we're not testing the health monitor in the CI and we ended up with a release which had a broken Ruby gam. >> Interesting. um >> there. So is uh did you add a test for the Ruby for the Ruby Health Monitor now? I thought those specs were running. >> No, haven't looked into detail yet. Um was just trying to reproduce it today because it's supposed to be fixed now with the latest director. [clears throat] But um yeah um I'll check the CI as well after that how we we missed it after I managed to reproduce it. >> Okay. Yeah. Cuz I'm I'm surprised that we missed it. I wonder Yeah. I wonder if it possibly was caused by something that happened when when Ruby got updated from 32 to 33 or or some some other stranges. But yeah, let me know if you figure out how it how it slipped through. Um, >> yeah, of course. >> Okay. And this is the only issue. The other ones are older. Okay. Uh, promotions, we don't have any. Then we are through. >> Nicely done. >> So then uh do you have anything else? >> Not for me other than have a good vacation. >> Thanks. Thanks. >> Are you gone for 3 weeks or? >> For 3 weeks? Yes. >> Nice. Well, enjoy. I hope hope you have warm weather and no forest fires. >> Yeah, it's [clears throat] Yeah, hopefully. Thank you then. Uh have a nice day, evening, morning and take care. See you. Take care. Afternoon. Take care, folks. Bye.