Foundational Infrastructure Working Group, September 3 2026

Hey, Hello. >> Hello. Hi. I had about the email quickly and I were discussing and we had no idea either. >> That's fine. Meeting is recorded though to keep in mind >> already. >> Yeah. >> Oh, I know. Is there everyone? >> Morning Carol. Morning. >> Morning. >> Good afternoon. Hope you all are having a good Thursday. Hello. >> Hello. >> Welcome back, man. Thanks. You recovered enough from vacation that you'd like to run this or shall I? I need to prepare. Okay, let me try. Happy to run if you'd rather catch up a little more. We see the meeting notes I shared. Yes. Yeah. Yeah. >> Yeah. Looks good. Oh, I see that one topic moved somehow below. Hello to the foundation infrastructure working group third of September. So we have some topics on the agenda. I will suggest to start with those. The first one is discussion move validation to GitHub actions. >> Yeah. Hi. Um so I've got currently the idea to um replace the corresponding concourse pipeline uh for the validation of the stem cells um by Gab actions and I just want to discuss it here in this round. What do you think about that idea? Currently it's on a concourse based pipeline and the idea is to just brutalize that conourse based pipeline by get a vector corresponding workflow to validate it. >> I think if we can we can satisfy all the complexity that seems fine. Uh historically, well, historically we use concourse cuz concourse was invented by people working on glide foundry. Um and one of the things that GitHub actions hasn't been good at is is anything involving manipulation like running like managing directories and such. But I think if I don't think that we do any of actual deploying of stem cells necessarily in the stem cell pipeline. I can't remember. Um that that part might be harder to transfer, but it seems like a good idea. It makes it more visible for folks. Yeah, it's also I think it's a good uh point to also further reduce the costs because uh and we need yeah we can cut the the responding resource for for the conourse. Yeah about the G actions because only as far GitHub actions are not not counted >> right. Yes, in general we have a guidance to prefer GitHub actions if something can be implemented with GitHub action because at the moment they do not generate costs uh yeah on the and uh yeah S mentioned yeah we need to check about uh security whether the access to yes resources is needed or something like that which uh needs to be taken can with more care when it's running in GitHub action. Another complexity will be the FIPS work which has kind of particular access requirements. Um we we publish a record of the FIPS artifacts having been generated but the FIPS artifacts can't be distributed or made public by us. Um I I don't know that that will necessarily be be harder to do in um in GitHub actions, but it's worth noting that the security model might be be different in terms of what's exposed and how how those GitHub actions are accessible or not. Um happy to discuss more. I think probably the the right move is to start a a discussion in a in an issue either in the community repo or in um in the posh linux stem cell builder. Um I don't know sphan if we want to have a if the TOC would like to have like a a sort of umbrella issue for migrating to GitHub actions. Not that we're necessarily choosing to do that, but as a way to track sort of what's moving or what's what what sort of consistent problems we're able to or having to think about solving. >> Um maybe if we manage to move and we have certain uh warnings from that we can share those. But I don't see that we need an umbrella at the moment. umbrella item uh umbrella issue for that. Let's see how this evolves and I will uh if we have some interesting learnings I can try to shadow something from TC perspective in a way which makes sense. Okay then uh we can continue with the next one. It's on uh RFC which has been opened this week. Um yeah, I encourage you to look into this. Uh it's about enabling uh Bosch to deploy to a new infrastructure. So they the proposal is to have uh a new CPI and new stem cell. Um so basically yeah yeah you can check um so also here statements about how this will be maintained and validated. Um and my proposal feedback was here that this can be done in one area uh in our working group like Ali cloud and open stack. So people can have ownership of this seems reasonable. Um yeah congratulations um Pascal. It just occurs to me, I don't know if you have been following recent meetings, but I know that Reuben is working on a a Nyxbased um stem cell um builder sort of replacement that would allow much more reproducible and and and uh much easier building across potentially different architectures. I think he was experimenting with that for ARM 64. Um, but as a result, it it made the builds more reproducible and that that might be something which is easier for us to uh roll into a um a GitHub action um potentially. So, he definitely would be somebody to I'll tag him on the document, but he's definitely somebody who may have ideas as well. I'll check that. Thanks for for the information. >> Yeah. >> Do we have any other topics for the agenda? So, okay. If so, then I will jump to the dashboard. I started opening the pending merch prioritized uh items. So the first one regenerate fakes using counter fight to defaults days one appro from Reuben. This looks to me uh good to be merged. Yeah, I think for anybody mostly this will be something that I will work on cleaning up, but the this does change the package name of one of the fakes and I believe that that's the blob store fakes and those are consumed by the Bosch CLI and possibly the Bosch agent. So there will be a little bit of manual work when this next um uh this next uh update this update lands for those packages. But it it should be it will be trivial but it should be um it will require manual intervention. >> Okay. also has been approved. So, okay. Uh oh, it's documentation AM you already reviewed and I will just merge. Thanks. This is making the gob based uh CPI default in B deployment. Uh yes. So um I think at some point we wanted to uh replace the Ruby version completely with the Golang version and this would then be the first step uh to replace it in post deployments and then move forward from there. This I think we discussed in the last meeting. >> Yeah. So we can match this right. Um so um I don't know whether we we don't have any releases from B deployment. Should we put somewhere something noticing that uh this is changing or um yeah I don't Yeah, we don't do release notes for post deployment. Yeah. Yeah. That's maybe a whole other complexity to consider is whether we want to should be tagging this at all. >> Um I don't think we need until no one requested until now and uh >> don't push for complexity before. Yeah, maybe we can put an uh message in the B stack channel. I don't know. Uh or just I don't have any good idea yet. Okay, this is also has a couple of approvals. So ready for merch. But okay, this is also changing the default size to BS5. you will have changed. Um I mean uh like changing the default CPI. Uh okay. Uh we need reviews here. Pending review. I assign reviews. It looks like uh ready for March for me. We have one review. It's not Well, here we still need uh to iterate on it, right? Net uh or is already code rabbit uh happy? >> So, [clears throat] I had a review from an approval from code rabbit. Unfortunately, like a couple of minutes ago, one of the pipelines ran and bumped the the bump push packages run um created a commit. So, I had to remove it and that's why the approval is not here, but it's done. It's tested. Um I actually included uh tests for both Ruby and Golang. [clears throat] Um I I think we should release one last version of Ruby CPI because it hasn't been updated in the northern year and um we had discussion with some more someone that um found a security concern that was addressed in separate PR that was merged. Um yeah so um this PR um runs unit tests integration life cycle and bats for both Ruby and Golag separately. >> Mhm. >> And the only also the boom packages and bump dependencies are working. I already run those and they [clears throat] updated master. Um, so after this we just need to try to u create a new release pretty much. >> Mhm. So uh it's ready for merch from your p from your point. >> Mhm. Yes. >> We got also an approval from uh Aram. So then I'm going to march and I aram I believe um after creating a new release we can deprecate Ruby. >> That seems great. >> Yeah, sounds sounds really great. Um I thought something interesting um the uh bats and life cycle which includes um creating a director um with Ruby it takes almost an hour more compared to Golang. So Golang compiles a lot quicker. >> Wow. That's that's great. Okay. Uh this one has also one make inside selection texture aware. Is this changing something in CPI behavior? how instances are selected which um resolves uh most probably this needs to be mentioned in the CPI release or so most probably will be mentioned automatically by the so it's only one I select between ARM and Intel basic. So won't change anything else. Okay. We have another two. I missed. So this is adding posgress Q8 support to back and resto. We have an approval. Do we need another to wait for another prom or >> I don't I think so. I don't know how much you all use, but I think adding it doesn't seem like it will be a problem. [clears throat] >> We don't use this one. We don't know. We don't have expertise in backup and restore SDK. Cool. Okay, then I will go and march. >> Are you been able to uh review this arrow in the meantime? >> Yeah. Uh okay then. Uh mostly comments and uh just bumping the API version which goes from 217 to to 2026. >> Mhm. I see. >> So TLS 1.2 seems Fine. That's basically okay then. Uh, let's find go with one and go for waiting for changes. merge forward workflow. So we've been having discussion on this forward backward. Uh so there's an update from you Pascal. Yes. Uh I've posted updates. >> So is it now ready for review? >> Yeah. Yeah. Um, I think I've also placed one commit. Um, I'm not sure if if that is pushed. If it's not pushed, then I'll I'll do it in a second. >> But in general, it's ready for review. >> Okay. I will move then uh request review here and move it to pending review discussion. >> Okay. Buntobo Fips. >> Thanks for working on that, Pascal. I'll take a look today. >> No problem. Is this now again ready for review? Think. Okay, looks like call L has been resolved. So I will then move this also move I don't see any open comments then I already request reviews and move to code review changes. So, code rabbit is still not happy. I just ask then uh needs to be retriggered. I think >> man uh the I think if I trigger a review or dismiss code rabbit's comments, it will rerun. But I think there's that approval pending. If you scroll down a little bit, there's a checkbox just right at the very top in the first Code Rabbit block. a little higher there. There's a trigger review right below approval pending. I think if we do that, it will rerun. It might come up with something else, but I can I can babysit and make sure this gets addressed or resolved. I >> No, something happened. Okay. >> Yeah. Okay, then so use code name and test comment. This uh we saw this came up a week or two ago and they haven't uh it's a very small change but they have not um signed the CLA. Uh there's one other change similar to this that seems a little strange. Uh seems like a very a very strange drive by PR. >> Mhm. I think I think if we don't if they don't >> bother to sign the CLA that's I don't know I'm not sure what this what they're interested in but >> seems a little a little odd to create a PR for for a character change. >> Yep. >> Let's see. This is a similar one. Different person. Uhhuh. Okay. Okay. Let's uh wait a little bit on those. So, this is still waiting. Uh Rebby waiting for Kot Rabbit. Yeah, we're we're uh excited about this. This is uh something that Matthew Coker had noticed when he was as a comment here when he'd been working on this. Um the AWS CPI does not play well with BPM because it uses uh slashdev to do um manipulations of the heavy stem cell in order to prepare for uh using that. So if we could if we could move to an API based approach for heavy stem cells that would I think improve a little bit the security posture in the system and make the CPI able to be run under BPM. Okay. So Yuri we are try to address the feedback. Okay. This is waiting for changes. Okay. in the correct state. Um, so and go with the uh the bumps. I will ignore the bumps. Oh, do we do we need to look into the bumps? I don't think we do. I think the Bosch boot loader. Um, so Ned, I believe, wanted to wait for the OpenStack ones to see if the CI changes resolve them cuz they they should be pulling >> Yeah. >> those in. And then Bosch bootloader I don't have as much context on, but it it seems Wayne, I don't know. I'm only picking on you cuz I know you've worked on it in the past. I don't know who is taking ownership these days of BBL but um these uh dependabot uh bumps might be an indication that the that the dependency bumping in the the BBL pipeline is not working as expected. So if you're not a responsible party then feel free to ignore me. >> Yeah I I'm not involved in BBL. Okay, there >> that might be Reuben maybe. >> Reuben a little bit. And then Ramon is an occasional visitor, but I think he's he's no longer working with a company that's doing much with Cloud Foundry. So is uh he's only an occasional visitor. >> The ones for OPT, I think they should be outdated now. I'll take a look and close them. >> I will leave now uh the B portfolio ones but if uh yeah most probably we have to discuss the ownership or who is interested to look into the BBO. >> I I do believe there was an issue opened um to move those bumps to the concourse CI. Mhm. Yeah. But someone needs to Oh, the conqueror CI should actually resolve those. Yeah. Someone needs to Why the bumps are happening when we have the concourse bumping. Yeah, that was the issue. Okay, this is the waiting for nothing pending. We have already assigned. Okay, then issues. There's a new one, August 26. Open this one also. Okay. Uh oh, this is from unit. Uh >> yeah. Um so we hit an issue with the best CLI internally. Um so I had to go and check what was changed in the latest B release and I noticed that um the version in the Bops when comparing the latest and the previous release shows one version um for S3CI in this case but the actual package inside the push release um was different Um, so I'm assuming this is happening in the concourse pipelines. Um, like the order they run or something. I haven't really had the time to look into detail. Um, just opened it so we can uh track it. That's interesting. Thanks for for raising that. and know we've had some S3 blob store not concerns but just we've had some issues with with customers running into different versions. So I think I'll I'll pass this on maybe somebody can investigate on our end and see um or at least they'll be interested to hear what the outcome of this is. >> Yeah, it's not specific to S3 CLI. Um we just noted noticed it with that package. So, >> Mhm. >> Okay. I will leave this in the inbox for now to still continue looking into it again. One net. >> Yeah. Um this was a followup to a comment that Aram left on one of the PRs um for the OpenStack uh concourse CI. [clears throat] Um yeah what he suggested is a better approach for the oh the docker images for the different uh CPIs uh consume BH packages that are required. Um so this is already implemented for Borch but uh the Google CPI and now the OpenStack CPI um use this um in quotes worse approach um where we hardcode the porch CLI version for example in the actual Docker file instead of dynamically um wrapping Yes. >> So in short um this issue is so we can copy what BSH is already doing and apply it to the Google CPI and the OpenStack CPI uh Docker images or any other that are using the same approach. Okay. So this is not resolved with the PR we merged. This needs to be done on top. >> Okay. >> Okay. Then I will also leave in the inbox. So pending or waiting for or let's put waiting for changes. Yeah, this is uh system test. >> This is this has been here for a while. It's um I just noticed when I was cleaning up some PPL environments. Um the the BBR CLI pipeline leaked has leaked one environment. There's still some pipeline that's leaking um BBL environments that are using the default BBL- env um prefix and I haven't tracked that down yet, but this is just a note >> for or to to reexamine this pipeline. There's a there's a complex level of insurers that I think need to be maybe reordered. Yeah, would catch you. I also noticed that this PBL uh Yeah, pipelines are leaking a lot of environments. M [clears throat] >> yeah it may be worth looking through our pipelines and seeing I I don't I think the the the right move is maybe to look through and find things that are using un using the default prefix and make sure that every every bubble end uses a pipeline specific prefix so that we can track >> Mhm. >> where that's coming from. >> Yes. Uh this will be nice. Uh so some of the working groups did uh something like that already. Uh yeah uh CF deployment and AR >> I can provide here I will provide here uh PR which already did something like that for reference. Cool. Thanks. Um, adding to that, I don't know if it was already discussed, um, but one of the jobs in the AWS CPI pipeline is constantly failing with PPC limit reached. I'm not sure if it's um, leaking resources or we're just hitting a limit. Um my my guess would be that it's the we don't have a very high limit of VPCs that we can have and my guess is that we are uh we're leaking some environments somewhere and then are not able to um to create new environments. Um I've been manually using the script that I just dropped into the Zoom channel. Um if you are authenticated to the uh CFF GitHub [snorts] project that will interactively search for anything uh older than a particular date with a prefix and then give you a list um and then if you choose it will destroy those um all the resources within the VPC and then destroy the VPC itself. [snorts] I'm not sure. >> Could you put that one here in this uh place? I am keeping those scripts in this uh >> uh yeah, happy to. Um can you slap me that or something? Send it to me. >> I put into some link. So I have some utilities to clean up uh um stuff. >> Yeah. Yeah, that'll give me uh that's great motivation so that I don't just have it sitting on my desktop. Okay. Yeah, you can try that one net and see whether it resolves the issue. >> I'm unsure if I have access to AWS. Um but I'll take >> Okay. >> Oh, it's AWS. Um yeah, this This one is for GCP, but maybe there's something in that script that will >> Yeah, I'll take a look if I can. >> Um, >> and direct workers on the BPM cannot read CPI bing configuring job directories. >> Yeah, I think we've talked about this while you were away. This is a a bug we noticed when we added BPM jobs to put CPI jobs into BPM. There were a few CPIs, I think maybe the plug-in capacity in um vSphere that it expected to be able to look at sibling jobs for configuration um and that was broken by BPM. this has been resolved, but this is just a community awareness um issue that uh was posted. We could resolve it cuz it's been shipped and just leave it here for history sake. >> Mhm. Yeah. Okay. Then uh we want went through all the issues and PR we have promotions. We have one. Excellent. So this is an area which is not so well uh uh filled. I think it's fine to have your resource appro here also already working in that since couple of half a year or something like that seems great so you are fine with maring yeah I can't prove right now just that sounds Congratulations. >> Thank you. And okay then uh we have everything discussed. Is any new topic arrived into the agenda? No. Any one something? >> I don't think there's anything else. And we have uh saved 14 minutes for uh yeah then enjoy the day, the morning or the evening and take care. See you next time. Yeah. Bye-bye. Rest of >> the week, folks. Bye-bye. You too. Bye. Bye. Bye. Bye. >> Bye. See you. Bye-bye.