Foundational Infrastructure Working Group, September 10 2026

Hello. >> Hello. >> Wait, there are two updates from Oh, no, this is from today. Date was just not changed. >> Morning, Hey, you're on mute. >> Yes, good evening. >> Hello. Welcome, everyone. >> No full webcam. >> Yeah, first time I'm not I'm not at home. Usually I'm not dressed as for business when I'm at home, so it's not a good time for a cam. >> You put on a Cloud Foundry shirt, I mean >> Oh, yeah. Specifically for >> That's nice. >> That's commitment, man. Wow. >> I think I have one as well, but it's under here. >> Oh, wow. Okay. Did you guys wear it specifically because we had this meeting today or just >> It's the only thing I I wear. >> [laughter] >> During business days, right? It's important, right? To have that code. I thought it was part of the CLA, right? You you cannot write commits without having the shirt. It's not like a thing. >> Thanks. >> Shall we go ahead and get started? Um Reuben, would you like to drive? Or would you like me to drive? >> Yeah, I can do it. I got lost meeting. I just in the zone. And I was like my wife said, uh "Weren't you supposed to have a meeting?" I was like, "What? What time is it?" And I was like, already meeting was already done. Um yes, tough. Welcome to today's Foundation Infrastructure Working Group uh meeting. We apparently have an agenda item. Azure CPI release plan. Who wants to know about that? >> Uh I saw that Reuben put that on the agenda. That's right, Reuben. Behan. >> Yeah. Because I was looking at the doc earlier, but I don't know any more context other than I I know he was the one who wrote it down. >> I believe Sebastian asked about it. He's like uh he recently did a lot of changes, so I'm assuming he wants to uh get a release with them. >> But he can click the button, right? >> It's not in the CFF. >> Oh, it's not? >> It's never been moved off of um Pivotal then VMware now Broadcom uh account. Um I looked at moving before I got access to an Azure uh IAS account. Um and now that I have one, I haven't I haven't been able to create I last I looked, I wasn't able to create the resources I needed. But I haven't looked at it in 9 months because of other stuff. Um >> Okay. >> I will take a look at the pipeline right now and see if I can um see if there's any issues with releasing or not. Okay. Thanks. Um yeah. And let's go over to board. This is the strip or the uh let's do the this one is first, right? The other one is a consequence of this one. But I see we have two approvals. >> Yeah. >> Go ahead. >> Uh yeah, this is about removing um the Ruby source code. So, I I have um kind of divided things into four PRs now, and let's see if there's more, but yeah, this is the second one. So, everything in the source folder is gone. >> Looks good to me. Okay. And then the other one was this one, right? This is the CI related one. >> Yeah. Yeah, this is um also almost finished, but this will make the bats test not pass because I've already removed Ruby from the CI. Here from the Docker image, I mean, and uh the bats test will fail. So, there's a a subsequent PR that I'm working on right now, and that is going to move the bats test to go. >> Okay. Um so, we will schedule this today, and then next week there will be the other PR, right? So >> Yeah. Yeah. >> This is also 3 hours goes. >> Yeah. >> [laughter] >> Uh let's leave that one. >> Um This makes sense. Was that um Rosetta variant already merged in the stem cell builder? Last week? >> Um that was released. I think there's one comment on here. We were going to wait for Matthew to feed in. Um so there is um this publishes the variant, but um would have published a FIPS variant um which we can't legally um distribute because of agreements with Canonical. >> Yeah, they they won't have that. >> There was a guard There's a guard in the code to do this, but I think I think it probably makes sense for us to remove FIPS from that. >> Reg X. >> Um Reg X, yeah. >> Okay. >> I can work I can work on getting this merged, but yeah, once this is done it we should see those variants on the at least for uh Resolute shop on Bosch IO. >> Nice. Will there be docs on Bosch IO how to use >> I don't know. Uh I think it's just you can run it on You can run that stem cell on Docker when the host is arm. Um Most of what happens is there's a certain number of binaries that are x86_64. So that Go. Anyway, you uh >> Okay. Uh then we'll wait. Feel free to merge this. This is seems non-controversial. Uh Hi Bay Hahm. Uh I heard >> I am >> You are interested this? The updates >> There was an ask in the channel so yeah. >> Yeah, we just discussed it. Aram is going to look into it or he's going to look at the state of the pipeline, right? That was >> Yeah, I just I just checked. Everything is green. It hasn't auto released because there were no detected um vulnerabilities, which is the I don't know so I can go and release a patch now um unless there was a request to release a minor. Do we >> We'll take it right, yeah. Yeah, thanks. >> Not sure if a patch is a correct one. Um he did add a lot of functionality, I think. >> Okay. I I can go ahead and click them. Release minor button unless we think it should be a major. I haven't been tracking. >> I think minor sounds about right. >> Okay. That's easy. It's just one button. So. >> Wait, there's already a final release? 1 minute ago? >> AI angels >> Oh, no. I must have I must have accidentally clicked it. Oh well, we have a new minor. I can click I can make a new with a new patch I can make a new minor if we want. >> Yeah, almost probably is. >> [laughter] >> It's probably fine. Uh here, I did a thing. Um >> Yeah, it looks okay for minor or for patch, yeah. Yeah. >> I mean, there's this which is a bit breaking maybe, I don't know. But storage accounts are Mhm. 2 2 TLS 1.1 has been out of support for a long time. >> [snorts] >> Yeah, right. And this should be fine. >> Yeah. Yeah. >> Um, yeah, that was that. Um This one has at least one approval from that. And from Code Rabbit. >> It had one from Ahmed as well. Um just Code Rabbit came up with the comment. Uh but from what I saw, I believe this one should be done now. Uh Aram, you also joined in the discussion. >> I just um made a I changed the title of this cuz it just used to say Ubuntu Jammy Noble, which was confusing me a lot. Um this seems great if we're ready to >> Okay. Broken links, docs. >> Yeah, this one's pretty impressive actually. Um I just found the This was from before. Um instead of relative links, uh we had like links to push IO. >> Mhm. >> I see. So, should we wait for this, or do you want to just commit this and go? >> Um uh the rest is fine. I checked it, so >> Okay. Should we just commit this and go? >> Yeah, yeah, that good. Sebastian already addressed a lot of good rabbit comments as well. There were two comments. >> Oh, another one. Sorry. Now you need to approve, I think. >> Yes. >> [snorts] >> Yep, done. >> Nice. Heavy stem cells. >> This one is from me. I mean, just for context, SPR should address the issue that the post create the end cannot work with heavy stem cell outside of EC2. So yeah, now it's fixed. >> Is that like inside it would attach it, right? And how does it work outside? >> No, it should use these EBS direct APIs. So it works without the any AC2. When I address that >> Yes, there's image APIs now. >> Yeah, like that. I assume. >> Because that was a a long-standing limitation in Azure, right? That we had to attach them and then copy things over. But there's no APIs for that. That's what you're saying. >> This is not Azure, this is AWS. >> Sorry, AWS. I meant AWS, yeah. >> Yeah, so it doesn't need attaching the volume directly to the EC2. It might need more more human review because I address all the what reviews come in, but yeah. Maybe need more human reviews on this. >> Yeah. I can I can ask Matthew to take another look. I know he's he was playing with this because of the limitations of BPM and mounting a dev device or the heavy stem cell. >> I see. I requested a code pilot review 30 minutes ago, so. >> Okay. >> I see that things No, it was like only just before the meeting. But yeah, uh is Matthew already assigned? Oh, he is. Yeah, okay. Yeah, if you could ping him, that would be great. Um thanks. Thanks for looking into this. Thanks. Okay. We have a clean board. Um What's this one? Oh, yeah, that's the CLA thing. I will convert this to draft. Another one. I'm also going to convert this to draft. >> Do you have any uh context? >> Um this was something that I've been running locally to clean up uh BBL ends. Um they had wanted me to just drop this in the >> Okay. >> Um in the Google repo as a utility. >> Beyond, do you want to take a look, or should we just merge it if it's uh something you have already been using? >> Might be good for somebody one other person to run it. Um basically it takes out >> I can check. I can run and then I'll approve and then Yeah. >> I wanted to mention is this related to the spot instances that are orphaned from the bubble pipeline? >> Um I don't know that they're spot >> spot instance, it's orphans from the from the from some BBL resource. Um I was seeing orphans from the BBR re- uh pipeline, and I changed the prefix that bubble uses so that it matches the pipeline name, and I haven't seen any more from that pipeline, but there is there is some pipeline that that tends to leak maybe one or two environments a month. Um but it's using the generic BBL env-pipeline, uh BBL-env- prefix, which means it's not clear where it's coming from in in which pipeline. >> But I noticed today um one of the tests that uh the bubble pipeline does, it um deploys CF, which means it's a lot of VMs, and it uses spot instances for it, which makes the test a bit flaky. Um which I guess that might be fine, but on failure, it doesn't really clean up every time. Um so, currently there are a couple of orphaned spot instances, which I believe are from uh the public pipeline. Um >> This This looks at resources starting with VPCs and then deletes everything on down from that. So, this it might be worth trying to run this if if there is a VPC stuck around as well. If there isn't, this might need a different the script might need some tweaking. >> Okay. >> What was this one? >> If you you can find something which can be fixed, that would be great. Yeah, the pipeline >> Yeah, I mean the yeah, the root the root issue is addressing um is addressing where those pipeline where those environments are leaking, but I haven't been able to find them yet. I think because they are generated as part of a put step, the um the fly search doesn't hold those logs, and it makes it very hard to track them down. >> Any on this one? >> Uh this should be ready to go. I've redeployed the go fetcher several times. >> Okay. So, we just uh go? >> Yeah. Worst case scenario, where everything. >> I'm okay with that. It's almost end of business day, right? So, it will be uh the night shift people who fix it. >> Yeah, it's just Friday tomorrow, so. >> I'm just referring to Aaron who works European nights. >> [snorts] >> Okay, that one we had and these are something. These are automated. >> Those are automated. Um So, this is uh this is a new PR that Clay worked on yesterday. We noticed that uh Golang came out with a new miner and we don't automatically pick this up. Um there's a script in the repo to run which will dump the last the out-of-date miner and add the new miner. Um think uh just just move this into a CI task so that we can trigger this CI task to create a PR that will uh >> Yeah, so still manual, but then from CI. Or is it >> going to also trigger automatically? >> may actually trigger on new Golangs. I didn't I didn't look, but um this was a kind of end of day um I think it might trigger a I think it does check Golang and might trigger a PR creation. So, at least we'll see a PR. >> hour we hours we do it. >> Yeah. >> Okay. >> Um so, I think there may be there's some discussion in there about maybe needing to modify some creds or something, but but effectively that just just automates a process that used to be automated by a Joseph Palermo. >> Okay. >> This is not I'm to require changes to the other pipelines uh for the push >> this is this this is only for bumping the available package in the Bosch Go lang package release. >> Okay. >> And the others would still need to do work to consume it, right? Unless they are on the the one package, right? So, if they are on >> Yeah, if you Yeah, if you if you pull the minor or the major package, then you'll always get the latest minor. Yeah. If you pull the specific minor, then you're then you're stuck. Um there's a commit on this repo recently that shows like what happens when you run that. Um when you run this when you used to run the script manually. >> This one? >> Uh yeah, no, it's add 127, remove 125. It's the one just below that. The >> Oh, yes, one. Oh, so it needs to re-pipe as well. >> Needs to Yeah, needs to rebuild the pipeline. Um once the PR gets merged, it changes the known versions. It rips out the deprecated version, and adds the new version. Um and then the pipeline has to run once to pick up the blobs for the new version before before everything is green because this this change only updates the data and doesn't actually fetch the resources from upstream. >> I think this is a good idea. >> I agree. >> Um but you're you're saying it's not ready, right? >> I think I think we should have folks look at it more than just me at the end of the day and Clay as he was walking out the door. >> Look at some digital folks. And uh I will assign the peoples. Okay. And this one is done in waiting for pending review. Okay. But Um I have a PR people need to look at, but it's a it's also in our working group, but it's UAA related. >> Are you ready with that? >> Yeah. We uh did a bunch of testing already with it. So >> Mhm. >> It works. I don't think it's going to be relevant to Bosch maybe. I mean it is It is a generic feature, so you can use a certificate. Yeah, I mean we we did discuss for the um What was it? >> But the NATS client possibly. >> Yeah, the NATS client stuff, right? So if we want to do >> Or were you talking about the you talked about this for the dynamic disk. >> dynamic disks, yeah. So basically with this you can exchange do an OAuth call. Um there's a spec. for this this one RFC. Uh, mutual TLS client. So, you you basically do an MTLS call [snorts] to the UAA and get a JWT token back. And uh, the client you can configure to to map certain things from the certificate to certain fields in the the token, right? So, we could actually use this um, for that potentially. So, we basically can have a client UAA client uh, that is configured to work with the the NATS um, CA cert. Or yeah, NATS What's it? No, it's the Yeah, it's the NATS. >> I think the NATS we did approach with NATS we uh, but we wanted to have a broker who can somehow with the Bosch APIs. >> Yeah, so you you want to be able to from a from a VM you want to be able to talk to the UAA to get a token >> Yeah. >> and then use that token with the with the director, right? So, this is the >> That was the concept. >> Yeah. So, this is the UAA part of that, right? Uh, and then you can just go to the So, we don't have to do MTLS on the Bosch director at this point, right? We could just do a normal call. We do a token exchange with the UAA with MTLS and then do a normal call with a JWT token which contains the the agent ID, right? Uh, in a field. Um, and go from there. >> Yeah. Yeah. And great. Hope fingers crossed that this this get in. >> Yeah. And this is the uh, for for Cloud Foundry it's quite useful, right? So, it's for workload federation. So, if you want to authenticate with your app against maybe your GCP or something or your uh, AWS account. You can set up AWS to trust your UAA, right? And then you can exchange your instance identity search with the UAA, get a token, then exchange that token with GCP and authenticate against your database. That's what this allows. So, for all the people with UAA experience, go look at it. It's only 132 commits. >> [snorts] >> So, >> I don't know the number of the tokens. >> [clears throat] >> Yeah, a few tokens. >> [laughter] >> A few tokens were spent in this process. Um but yeah, so I did these balance reviews and I got like I I went through 15 iterations or something with it finding stuff and then eventually it started to quiet down and be be okay with it. Also, I'm not a Java person, so that didn't help. Uh but I have people that do I mean, one nice thing about working in a big enterprise is that there's a lot of people that know Java, so they looked at that at that part of it. >> [snorts] >> Um so, it was fun. >> Yeah. >> Uh but I saw Marcus is on vacation still, right? So, >> Yeah, still some people on vacation here in Germany. But next week should >> Yeah. >> Next week people will be home just in time for the holidays to start. With more vacations. >> Yeah, we have summits. >> [clears throat] >> Yeah. >> [sighs and gasps] >> Promotions. >> Yeah, just created this one based on the latest inactive user report. We have to wait here. >> What's this RE thing here? >> Oh. >> Seems like someone is copy-pasting or having agents that are not well in >> Good catch. >> It is in in the commit messages, it's fine. But uh >> Yeah. >> Um >> I know extreme nothing. I think he was someone from Broadcom site working in security stuff or do I need >> Yeah, at security and out of the Toronto office, but he left. Um He left the company. >> Okay. >> Over a year ago, so. >> Because we've been updating uh with with we have been ping-pong with we have ping-pong with him to keep him active, but yeah, okay, then. >> Yeah. Okay. Then we will wait. Any other business? Oh, issues. Wait, we should go over issues. >> Um Um >> Nat, I saw you replied on that post-gress issue. I will go ahead and try to see about removing the blob from there and hopefully the bumper will work. I think it was just too much um too much committed that it's always a bit difficult when this when CI kind of owns the private YAML or had access to the blob store. >> Do you think it's just you or what? >> Mhm? >> What is this one? The bump? >> That's this one, yeah. So, uh Postgres was 18 was added, but the PR included a commit that put the blob reference into the blobs.yaml, but because the person doesn't have access to private.yaml, it's not in the blob store. So, >> This was not merged. >> Oh, sorry. That was dire. I saw an and just um extrapolated. >> Um but it sounds something Sounds very similar to the other issue I opened above the history's UI uh blob. >> [clears throat] >> There was a difference in the version in the blob file and in the actual package. Um >> Yeah, I think >> This one >> a We don't really have a great pattern for um for PRing in blob material unless the pipeline is >> Mhm? >> Get out of our first. It's great. But then we are running into usage limits on GitHub. >> Yeah. >> Exactly. >> it's not cheap. Uh it has other issues. >> I haven't looked into this, so I have no idea where it's coming from, the difference. I was just assuming it's the the CI um >> Don't I think it could be that the CI is there's something confused in the pipeline in terms of the wrong version getting pulled or something that seems a little spooky, but that's possible. Has this Is this resolved now or has it that has been bumped or is this still in a strange state? >> With the latest release the blob and the actual package or at least for S3 CLI they match. >> Okay. >> So it's like an intermittent thing. Um So the the package contained the older um S3 CLI but the blob was saying it had the newer one. >> Mhm. >> I mean it's not really an issue. Um the thing is we had a um run into a problem with the S3 CLI. There was a bug in um one of the versions that the BOSH director came up uh came out with. >> Mhm. >> So I was uh trying to confirm by looking at the blob and it ended up uh the information there ended up not being correct. Um so yeah. >> Yeah. Is there any reason why we not compile these things from source? >> I don't think we're compiling them from I mean >> In the release, right? >> I don't know. I mean speed is one of them. I don't know that we necessarily want >> compiled releases, right? So that's not a problem. >> Yeah, I don't know. Um I mean hopefully this will change into a different problem once we have the storage CLI um but I haven't seen any movement on that in a year. So. >> We have someone internally pushing for it. So it might happen sooner than expected. >> That would be great. I mean I would I would love for us to get off of the I would love for us to use the same ones that Cappy is maintaining. So. >> Yeah. And it shouldn't be any kind of operator visible change. It should just work. In theory. >> True. In theory, I like that part. >> So, this is just I think maybe this was the one that you posted as well, Ned. >> I don't know. So, the one I saw is on the push bootloader pipeline. The actual people bubble release. >> Okay. >> This one can be closed on, right? >> Looks like it. Yeah. >> Uh this was open just to track it. Yeah, so I don't forget. Um So, for GCP CPI and OpenStack CPI, we hard code the Bosch CLI version in the Dockerfile. And Bosch is actually doing it towards what cleaner. And so, this is to track that. So, this is something I brought up brought up in one of the PRs. >> Okay. This is a note to self, basically. >> Yeah. >> Wait is definitely open contribution. Yes. >> Yes. >> Okay. Um Uh how do we want to mark these? >> Um open for contribution. >> Okay. >> Yeah. That's it then, I think. We did it, people. Thanks. Anything else you want to discuss? >> Um I did have a little bit of news from Matthew who's been working on Resolute. Um I think he he mentioned in chat the other day that he has the stem cell down to close to 700 megs. Um and there are a few other things that he's looking at possibly tuning. Um one of them is to strip out the uh GDB debug symbols from all the Go lang binaries, which should save close to like 20 or 30 megabytes. Um and then I think there were a few other um similar kind of changes we might we might be able to make. >> So. >> Is that done for the agent? Does the global stuff >> all the all the blob stores, all the things. But I think this should all happen probably at the at the originating >> Yeah, it's not it's not Resolute specific then, right? >> No. No. Um but there was some he figured out some some more robust ways of clearing out the app cache which brought the brought the stem cell under from like I think it originally started out at like 1.1 or 1.3 gigs. So, almost almost half. >> Are there parts of that that we want to backport to other lines? >> Possibly. Um I would take a look um once the changes land. I I don't know if he's planning on making them that far back or not. Maybe. >> Okay. Nice. Yeah. >> Now, just real quick, um I tried to look into the VPC limit that we're hitting for the AWS pipeline. >> Mhm. >> I couldn't find my password for the community AWS accounts, and I think I had access um when I was working on the bots for AWS. If someone can check and reset that. >> I can log in and I can reset and and I'll slack you a password. I'll do that right now. >> Okay. Thanks. >> Yeah. Thanks. >> About the stem cells, I was somehow had to restart. I was notified around the application runtime deployment working group so that they can start validate. So, this is CF deployment. >> I I think the the alpha stem cells are available, so that should be >> Yes, they are available on Bosch VM, yeah. >> Yeah. >> Nice. >> And then once CF deployment is Oh, wait. Are we also already validating with the Bosch director? Shouldn't we do that? >> Um I believe that we should be good on Bosch with that the I'd double-check the pipeline, but I know Matthew did some work to the sort of >> I asked him >> parameterize the stem cell into the pipeline. >> Yes, he mean the our BOSH private channel, who is called BOSH, and he confirmed that on BOSH side, yes, it's fine. >> Yeah, the BOSH director has all the basically one things for every stem cell line. Yeah. >> And there were no major things? Do we need a migration guide? >> I don't think so. >> This just works with Has he tested it with CF deployment already or not? Matthew? >> Uh I think that he has to deploy with CF deployment and flag those errors for the for the folks there. Um there are I know the CHPST, which is a like a user it's like it's like down down like switch user and change my permission set that came with run it. Um is is something that Cappy needed to work on, and there's some some PPM stuff. I don't know whether those are in issues or >> We have a tracking issue, right, for uh RFC status. >> Which one, sorry? >> We had one for Noble. I don't know that we have one for Cappy. >> Actually. >> Oh, yeah. >> Yeah, I'm I'm keeping track here on everything. >> Okay. >> But there's no open issues, no open um >> This is the state I found a couple of weeks back. I just search for resolute stuff in our org, cloud foundry org. >> Mhm. Okay. Let's subscribe. But um good to hear that it's uh going or on its way. >> Yep. >> Thank you all for attending. >> Yeah. Thank you. >> Take care. >> Good evening. >> I think next week Ben, you and I are out, right? >> Isn't there the conference? >> Yes. >> If there's the conference, should we skip next week? >> No, it's not the conference. There's another >> Oh, it's something else. Okay. >> Oh, it's not the con- it's the conference. >> It's the conference. It's like uh There's an agents AI agents con in Amsterdam that we're both going to. >> I don't understand why you would be going. Why aren't you sending your agent? >> They don't support agent-to-agent protocol yet. But otherwise, I would have done that. I That was my first intuition as well, but sorry. >> No identity. >> [laughter] >> Identity was not supported. We still have to figure out how we're going to solve the identity thing. That's why we need to go to the conference. >> [laughter] >> So >> All right. >> Yeah. >> Okay. Well, I I'll see you all next week. >> Yeah, maybe you can run the meeting. >> I should be able to. Um and then I'm out for the last 2 weeks of September. But otherwise I'll see you all. >> The week after will be CF Summit bro. No, is that No, that's actually beginning of the week, right? So we >> Monday, Tuesday, yeah. >> Yeah, okay. Okay, then we will run that meeting. >> Yeah. >> Excellent. Take care. >> See you later. >> See you later. Bye. >> Bye-bye. >> Thank you. Bye. >> Bye.