Foundational Infrastructure Working Group, August 13 2026

Morning, Adam. >> Good morning, Ned. How are you doing? >> Yeah, doing doing good. Doing good. Can't complain. How are you? >> I'm I'm all right. >> Well, happy to hear. >> Welcome back, Ruben. >> Thank you. My internet might be horrible. Can you hear me? >> Yep, no problem. >> Okay. >> All right. I'm feeling this maybe everyone. Um Anyhow, welcome to the weekly foundational infrastructure working group meeting. Um I think a lot of people are on vacation, so it might be a short one. Uh there's one agenda item that I put on um from Pascal Zimmerman, which is about the PR to remove Postgres 13 from the Bosch um director, which I think is fine. I've asked um some folks from Broadcom to take a look. Uh And if anybody has any input from SAP, especially if there's any concerns about that, please give it a quick glance. Um Uh we'll open up the project thing. Reuben, do you have anything now that you're back from vacation? Any large-scale changes you'd like to make? >> Not to Bosch. Not right now. >> Okay. >> I would like to add something to the agenda before we move to the dashboard. >> Sure. >> Um I've sent it as a file in the chat. It's regarding the PM the OpenStack CPI pipeline, uh which we discussed last time. >> Actually, I I do remember something. I uh this was just before my vacation or towards the end of my vacation or beginning of my vacation. First week I still worked a bit. Uh and I built a Bosch NYX Linux stem cell builder, um which is an interesting idea, I guess. It uh cleans it up quite nicely and it supports building uh noble as well as I think the next one resolute from the same uh branch, right? So, it can all be shared beyond the same branch and there's no more um separate um pointers. So, there's no pointer to a a compiled agent or compiled or pre-built OS image, right? So, it can all be built from source, the agents as well as the uh OS um and it uses NYX caching and uh I believe it's fully deterministic, so you can like binary reproduce um a stemcell from a source commit. Which I think is quite an interesting property to have. Um so, I appreciate people taking a look at it. Um seeing if we want to take the or yeah, evaluating if we want to take this further. Uh I think especially with um us soon having three supported stemcell lines at the same time, it might be worth looking into um simplifying this a bit. So. >> Yeah, I think it's interesting. Um Ned, I got the file. I was going to start a thread in Bosch core dev if that's okay. I'll tag you and put the file in there. Yeah, I'm just cuz it's kind of hard to to paste the the markdown directly into the infrastructure working group. >> Of course, yeah. >> Do you think we'll have the time to quickly discuss it? And what you think about it? >> Sure. Let me add that to the Slack thread and then uh let me see if I have something I can read this in. Just a moment. What am I going to do with this? Try to figure out how to share this without sharing the disaster that is my desktop. Okay. We can do this. No, that's There we go. That's what I wanted. All right. Can folks see my desktop? Ned, I think I've managed to >> Mhm. Yep. >> share that appropriately. >> Um, I'll just try to summarize it quickly. Um, I used the GCP VM. >> [clears throat] >> Installed um, DevStack on it, which is your best stack infrastructure. From there, um, the life cycle tests running uh, fine. The integration ones. For the bats, um, a director was able to be provisioned. Um, we didn't use bats because the the CI was not using bats before. Um, so just directly provisioning a director. And all bats tests also, um, run fine on it. Um, cost-wise, so what I was thinking we make a um, snapshot with a DevStack installed. And when we want to run the integration and batch tests, we just spin up a VM from the snapshots, run the um deploy the director, run the tests, and destroy the VM. >> How long How much does that save the snapshot? Like, how long does it take to install devstack? >> Um I think that time If you scroll a bit down, it's pretty quick. Um it takes more time for the director to spin up. Uh well, maybe I didn't include it. It was something like uh 10-15 minutes, I think, for the devstack to to get installed. >> Then I might I wouldn't bother because there's also a like snapshots probably cost money, right? And then you have a cleanup issue, and then it might get stale or stuff. >> Mhm. >> But that's uh that seems really reasonable. I mean, if it would be more than $30, I would be making a problem, but this is perfect. >> Yeah, so the those are rough estimates. >> Yeah, I understand. >> But the snapshot with just devstack, it's around 50 gigs. >> Mhm. >> Um disk that we'll need. Um yeah, so it's around about 2-3 dollars a month, I guess. >> Yeah, I know about that. Mhm. But still I wouldn't bother with the snapshot because of the uh complexity of like keeping snapshots up to date by cuz then you need a pipeline to also go snapshot it if you need a newer devstack. Right? >> Yeah, yeah. >> Uh it's it's it's premature optimization in my mind. Right? Like I I would just go with installing DevStack and then we can always if we find it really annoying or be a big issue then we can optimize that. Not start with the snapshot. >> Yeah, yeah. That makes sense. It does require a couple of tweaks to to get the DevStack prepared for for bets, but that wouldn't be really a problem. It can be just one script. >> Yeah. I prefer that, right? Because if you get like a manually crafted >> [snorts] >> snapshots like we have some of these artifacts that were created by someone at some point with context and that we never can like can reproduce and don't know how they work. That's what I'm worried about with these snapshots. >> Yeah, so it's let's say it's about around 4 hours for the whole thing including the live cycle integration tests and bots. >> Mhm. >> Um which if we create the VM only when we run those tests and we were discussing the idea of having it triggered manually only. So when you want to release a new version then you trigger it not on every commit. >> That makes sense. >> Yeah. >> Um yeah, or we could have one of those labels. I think that they have that in other working groups like ready for testing or something. Um but that's on a PR and right? And we're not even talking about PR validation, right? We're talking about doing it after merge. Or do we want to do it on PRs as well? I mean it would be useful on PRs, right? To be able to validate before merging the PR that it still works. But not on every commit. Ideally, we mark it but maybe a label saying go run the acceptance tests. >> I think if this is going to live in Concourse, I think the first the first best step is to put it in the pipeline, but make it be manually triggered and then we can figure out how to how to optimize that for for PR flow and such. >> I mean, that's definitely the first step. With the integration tests, it's it's pretty quick. For for the bats, it it does need some preparation. Deploying the director took a lot of time, like something around 40 minutes. I'm not sure if we can use a compiled release somehow. I didn't look into it. I'm just wondering how convenient would it be to run bats in the actual PRs as that's going to take a couple of hours. >> Um each >> It should be possible to use a compiled release for the director, which would ideally speed things up. I think those are available on bosh-io. Um So And there's a pattern in bosh-io of using the um the agent compile step to to compile the CPI. So, you could do that prior to spinning up the the OpenStack the dev stack VM to to make sure that the the running time doesn't include any of that compilation. >> Um as for the alternative of GitHub Actions, Sasha did the testing there. Um I think he can give more details. We weren't able to get it um, running the director. >> I mean, this seems from my perspective this seems good to go. I think we can start it, try it, make it manual, see if the cost see how the cost looks and but at least get get openstack unblocked from from having official releases with with bots. >> Yeah. >> Yeah, if we have the green light from you guys, um Yeah, I think the costs are acceptable. I mean >> Yeah. Compared to the rest of our CI budget, monthly CI budget, this is not much. Yeah. Or CI spent, I should say. We don't have a budget. Yeah. >> Um, I'm seeing the thing about using a Bosch really it mean to use a source release. Um if you're testing on jammy, I guess you could you could use the pattern in Bosch itself where they they recompile using the agent. Um, I think it's the job is still called export release in like the it's a there's a there's a fan in step called like build candidate release or something. Um but you could also switch the testing to noble potentially since that seems to mostly the default. Kind of up to you or you could do both if if that seems like a good plan. >> Yeah, haven't looked much into the compile uh, release so I I'm not sure how it works so yeah, if we decide to go this route I I can I'll look into it and that's kind of speed things up a lot as well. >> Yeah. Uh, I'll put the job in the agenda somewhere once I figure out what to do with everything. I'll I'll put it in the dev I'll put it actually in the channel. >> Thanks. Thanks. >> Sure. >> Would you like to keep it for some time? See if everyone anyone has any comments about it. >> I would just go ahead. >> I would go ahead. >> Unless you feel like Bay on has strong opinions about this. But he's on vacation and I think he's fine. >> Yeah, that seems good. >> Expecting a very a lot bigger number for the the cost. So >> Yeah. Now I now that I see the cost it's like okay, just go ahead. This seems like a no-brainer. We should definitely have this. Yeah. Great work. >> Thanks. I'll forward it internally and you know, we'll we'll be start working on it. >> Cool. I added one last thing to the agenda just as a kind of heads up. I've been seeing a lot of bats failures across all three stem cell lines noble jammy and resolute. I haven't looked into it but somebody Matthew Coker took a quick look yesterday and said it seemed to be mostly due to capacity issues in the regions where targeting. And I I don't know whether this is uh because we're using an outdated small instance type or because there actually is capacity in Europe West 4 or something. So, we we may we may run into an issue where we have to switch regions because of capacity. Um >> I mean, we're we're using RAM, right? So, and AI is eating all the RAM, so >> Yeah. Um anyhow, that's just a heads-up that we may need to switch to a slightly more expensive region if things continue. >> Thanks. >> Um >> Cool. >> Um I think that is it for the agenda. Let me just double-check. Um All right. Um All right. Um Okay. I think we're done here. Uh I managed to lose my shared window. Can you all see the shared window? >> Yeah. Yeah, we can see the dashboard. >> Awesome. Uh all right. This I can see where we're going. All right. Um This we opened there's a number of Bosch releases that aren't syncing to Bosch IO. Um it was we were going to give folks that are responsible for the releases a couple of um maybe a week or so to make some decisions, but it looks like uh these releases changed their blob store to not be public accessible, so they can't be synced or updated to Bosch IO. So, the plan is to remove them. Um by like next week sometime. So, just to kind of clear up the craft and CI. >> Yeah, and then we'll actually not remove them from Bosch IO, right? The ones that have been synced already will stay. >> Um they may, yeah. I don't remember. Um and I think those blobs may stay available. I don't know. Um I'm inclined to take them completely away because I don't know that the blobs will stay available indefinitely, but we don't have anything like that. >> Bosch IO we have create we have we have our own S3 bucket, right? So, we ingest these things in our pipeline and then do a Bosch create release, which downloads the blobs, and then they go into the S3 bucket. So, the stuff on Bosch IO should still work. It's just the ingest that's broken. >> Yeah, that's fair. Um you know, feel free to add an note here. I think it's we should at least flag them somehow as no longer being updated because even though the repos are are open source, the even though the repos are accessible to people, the blobs are not. Um let's see here. Do we have context on this? There was some conversation. Code review was approved, but we don't have other approval and then I'm >> This is some flaky behavior we um fixed internally. But I don't have the the details. >> Okay. But this seems like a just uh create or gets, right? Like >> Mhm. >> Do we need a full review on this? Like seems so like it seems so small. >> All right. I'm happy to approve it. >> I believe you read most likely tested the fix. >> Okay. >> Yeah, I have had not had my head in networking land, so I'm a reluctant to just drop stuff in, but if you all feel unconfident, I am feeling good about this. All right. >> It looks like there are changes required. Requested. >> Yep. >> Okay. I think it proceeds as >> Um This I mentioned. >> Yeah, there were 8 hours ago. Or comment from that And that's That person should uh Look at it. >> should probably look at it. Um just double check. Um >> There's just one disagreement with uh code rabbit. I think that's So, it seems like everything is addressed. Maybe just someone needs to uh So, agree or disagree with the disagreement with code rabbit. >> Yeah. I'm going to dismiss this. I'll look at this. Um I'll look at this myself. This is I'm I think I'm going to suggest that we do this merge forward, not automatically, but we instead um do it manually for a while and then and then we can turn on the animation cuz I don't necessarily think we want to um run it full speed. >> We first want a human to hand hold it, get some Yep. Makes sense. >> I'm about to sneeze. Mhm. Sorry. I'm allergic to mornings. Um Stash that for myself. Um That's going to move to draft in a minute. Um Ned, you want to talk about snapshot based disks? >> Sebastian did a pretty good review on this one, and I'm still going over his comments. >> Okay. >> This is still work in progress. >> Okay. Cool. Uh I'm going to move on. Feel free to flag me with this if it's uh if you're ready. And also if you have approval before um before next month next week, feel free to merge. Uh let's see here. Um anybody have any context on the trusted Azure launch profiles? >> No, I have not looked at it. >> I'm generally inclined to just merge this I think um this person is works with Azure and obviously has a ton of context. Um We don't have a lot of people with deep Azure experience. I can tag um Brian Upton and see if he Yeah. is able to notice anything. Um Any other >> But not I agree with like his PRs I tend to just I don't know give it a a of time but then end up merging it because they tend to sit for a while now. We want to review them and he's the expert on Azure anyway, so >> I'm going to mark my approval and then >> Mhm. >> I think we just rebate. I think we merge this has been here for 2 weeks and nobody seems to be >> Mhm. >> picking it up. All right. I think I think similar with this one. >> But don't we need something for in the stem cell for this this as well? >> I think this is the maybe the change for the stem cell. >> Sorry, I was looking at the I see. Yeah. I was looking at on my other screen at what this feature does. >> Yeah. >> But is that like that's affecting the stem cell, right? >> Yes. It's adding some uh some packages for EFI. >> But has does this have like impact for for other >> need to run to sign utility. It does add them across the board. >> Yes. Like this affects all stem cells, right? >> Yeah. >> This should probably uh This could do with some testing from somebody else. Yeah. >> I'm going to tag you on this one, Ruben. Maybe you can use Sorry. Maybe you can use next to make it amazing. Uh >> Cuz I think it's a feature that probably we want to look into. Sounds like a useful thing for a bank. >> Mhm. All right. >> More security. People get really happy from that. >> Everybody wants more security. >> Yes. More. More for free as well. So, yes. >> Okay. I think this just adds an extra check for DHCP being able to be forced by the network, which seems seems fine. This is in the agent. Once added Once added this kind of extra boolean This seems seems fine to me. Does anybody have any concerns about that? >> Seems fine. >> Right. This is Um Just I think this just removes the deprecated post crash 13 job. Or post crash 13 is no longer supported. This removes the job from This removes the package from Bosch. And add some instructions for people who might still be using post crash 13. Seems fine to me. Any concerns about merging? >> No. >> And on those two. Okay. Well, luckily we've gotten those two done. Um Yeah, I saw you approved this. So, I'm I'm even more excited to not dig too deeply into it. Um >> This one is a little bit It's a bigger change. Um I had one concern. We discussed it with Sebastian in the comments. >> Mhm. >> Since we um had an issue with when we implemented dual stack for AWS. >> Mhm. >> And we had um an instance where uh the mapping was incorrect. >> Mhm. >> This was the only concern, honestly. Um I think it would be nice for someone else to also take a look. Well, but it looks fine to me. >> Yeah. Anybody else that is like likely to be able to Yeah, I think I'm I think I'm good to merge this. We're not changing interfaces or anything, so. Um this is I'm hoping to land soon. I think there are um Colin Shield has been doing some testing to try to reproduce the SQL bug uh in Ruby 4 and Ruby 4 with or without YJIT um where we were seeing some memory corruption in high concurrency uh SQL queries in the BOSH director. Um So, once we uh figure out uh a little more about whether YJIT is implicated there, we'll we'll go ahead and merge this. Should hopefully see some improvements, maybe even some speed up in test time. Okay. >> Would you maybe add a comment with that context? Because this has been open for a while, so otherwise people might forget. That we're waiting on that. >> And just for context, this I think this change will require a little bit of CI dance because a PR shouldn't really have the ability to upload a blob to the blob store. I mean, I could probably go find it in CI, but the the rust package will need to be sort of vendored by CI so that the blob ends up in the blob store before everything will kind of work correctly. >> All right. And then you're going to take a look at the EFI. And let's see. We have a bunch of go lang bumps. >> I'm going to ignore them. I'm going to ignore the debt bumps for right now. Um Okay. Um let's see. This is Um Right now, this should resolve some of the dependency bumps from uh Dependabot. Turns out we didn't have uh any CI job for the disaster recovery acceptance test. Um this should hopefully um at least do regular updates of the the Go line code. Um and then we'll run a dry run lint, not actually run all of the tests because they get run elsewhere. Um >> And is this on GitHub actions or is this uh Concourse? >> This would be in Concourse. >> All right, I see. Never mind. >> There There was a There was a pipeline um uh There So, they're run in the Bosch director, but then there used to be a BOSH Lite pipeline that was independent, I think, of the director. And then I think it got folded into the director pipeline, and then this pipeline got removed from Concourse. So, um this would just add back a sort of basic test coverage, linter, and such for the for the pipeline. Um And then this uh this change is um a switch to using code.cloudfoundry for config server. Um this can't happen until we get the um code.cloudfoundry redirector uh updated. Uh I have access to that um and tried to deploy it with another um CFF dot com person, I mean. Um but the didn't seem like the CF application hosted by anynines had been deployed in a really long time and some of the uh credentials were missing. So, we we had access, but we didn't feel like we're actually pushing the change. So, at some point um that will um that app will be updated, which will allow us to redirect to uh subdirectories of repositories, which should allow us to make this change to the import path. All very boring, but useful. Yes. Um I think that might be >> With regards to the uh slimmed down pipeline one, I see a lot of commented out code in the pipelines. Is that intentional? >> Um that could probably be deleted. I I just wasn't I didn't I didn't want to lose the git history, right? I didn't want to lose the yeah, the git history. I can I can rip that out. Um >> Otherwise, it never will be. >> Yeah. Good call. Yeah. Um yeah, just tag me on. And uh add a note to do that. Um Cool. I think that is everything that's not a Dependabot um thing. And then we have Maybe we have some promotions. No promotions. Very exciting. Any new issues? Um Nat, I think you had one update on this. >> Yeah. Since the issue was resolved, um I will eventually get get to this one to confirm if we're missing something in the pipeline or not. >> Okay. >> Okay. And that one rewritten in uh Go line to help monitor. >> Uh I have a PR open for that. Um I think once we I would would like to hold those in reserve until we get the Ruby updated and YJIT stuff landed, and then >> Mhm. >> Um and then as far as sequencing, I'd like to um land the NATs sinker first because that's a little less complicated than health monitor. >> Well, health monitor, isn't that like part of the plugin interface? So >> Health monitor has a plugin interface, so we would need to uh it's it's part of HM, which is like an event responder. Um the Go lang rewrite uses external like external binary execution to to do the plugin model. Um but there there's a lot more moving parts in that one, and I would >> Yeah, makes sense. >> I would like to land everything all the other sort of pending stuff first. >> And then one remaining part, right? Rewrite the director. >> Mhm. >> Go ahead. >> Like 40% of it's already in the Bosch create and CLI being go like so probably fine. >> Yeah, hard going to be. >> Got to be easy. Um all right. I think we have done it, folks. You have 16 minutes of our life back. >> Thanks. >> unless there's anything else or the kind of the orders, they say. >> Would appreciate some little feedback on something. >> Yeah. >> Like So we um >> [clears throat] >> it's I guess an edge case, but um internally we had someone um resizing the persistent disks on one of the deployments. >> Mhm. >> So the Bosch agent um expands the partition and then resizes the file system. So in our case, it managed to expand the partition, but um some of the disks were corrupted. So when the agent tries to resize the file system, um the kernel is saying no. >> Mhm. >> The the thing is the tool that we're using for the resizing uh just returns a um what was it? Not permitted the error. Doesn't say the reason for it, so you >> Yeah. >> have permission to know it. So someone needs to check the kernel talks. And that's fine. Um my concern is that we are we were running this in CI so in Concourse. >> Mhm. >> And in return one of the pipeline runs failed. But the next one was successful. >> Mhm. >> So the the issue is that the agent once it expands the partition from there it doesn't check if the file system was actually resized. >> Mhm. >> So I played around with the on this one and um one of the solutions was to try to resize every time. So if no resize is required it just returns zero and continues. Um but I didn't feel like this approach. The other one was this one when I when I tried to add some logic that checks if the partition and persistent disk have a size difference. But it got It's I guess we got too complex and I thought that this approach as well. Um I ended up closing them but Ramon noticed the PRs and was asking if we should please document this somewhere. But I'm really not sure if it's worth investing uh time to fix it or I'm not even sure how to document it. It's It's a very particular case where this happens. >> We have that like documentation page for agent failures. All right? For failing VMs and stuff. >> Mhm. >> And then like if you have a particular error string and you have something like you already found like you had a way to you found something in the kernel logs and you had an agent string, right? [snorts] So it's just the next person Googling ends up in on a useful page. I think that's >> [snorts] >> that's the aim. But you could also just document it here. It might already be, but like the particular failure I I I don't know if Bush I always better index than get up. >> Mhm. Yeah. Okay. I'll check. Yeah. >> That could be. >> The the the main concern is that um after the next pipeline run was green Yeah, that's annoying. This was something intermittent. It worked, but actually the disks remained uh with the smaller size of system. >> Yeah. I wonder if there's a way to like if there's a if there's an agent sanity check to see if the expected partition sizes match the actual partition sizes. >> This is what I tried to implement, but uh I I think it's too complex. Um >> Mhm. >> And I'm I'm not very confident in it. Um because the sizes are not one-to-one. There might be a bit of a difference, which is uh a bit of um 100 megabytes of delta, which we already have somewhere in the code, but Yeah, I really didn't like the this approach, so I ended up proposing it. I'll see if there's a place for it in in Bush I/O to >> Yeah, just as a heads-up. Yeah, that seems like a good I got option. Yeah, I don't know. Like maybe or maybe there's a way to write to write something to the disk on failure um when the resize fails, which is like a signal to check that the resize should be re-attempted or something. >> Yeah, that might be a simpler approach actually. >> Yeah. >> It's not perfect and you'd have to figure out a way you'd have to you need some kind of break glass tool to like get rid of that maybe get rid of that sentinel file or something, but that might be a way out of this. >> I couldn't think of a graceful way to to handle it honestly. Yeah. >> Yeah. Nice job finding a complex problem. >> [laughter] >> Now you got the the prize. Prize is also having to deal with it. >> Yeah. No, I was good after this the second one was green. I was like yeah, that's So everything's fine. But someone was actually checking the disk disk sizes, so unfortunately. >> Yeah. Maybe next time just don't look so closely I guess. Um Cool. Well, thanks thanks for posting and probably worth Yeah, if if there's a good place to stick it on Bash IO that seems seems valuable. >> Yeah, I'll take a look. Thanks. >> Thanks. Cool. Thanks y'all. Have a good Thursday. >> Thank you. Thank you. >> joining. >> Yeah, you're welcome. Bye now. >> Thank you. Bye-bye.