Foundational Infrastructure Working Group, August 20 2026

Hello. >> Hello. Good morning folks. Hello It's probably my turn today, right, to run the meeting. >> If you're up for it, I'm a little spaced out. >> I have no guarantees about not being spaced out, but I will try. Okay, let's go. Thank you all for attending the foundational infrastructure working group meeting. Um, let's start. Um, agenda I think in concourse Did you put this here? >> Um, yeah, I did. Uh, I thought that the PR that's linked in the sub moment there should should have fixed this by adding the deploy key to the list of accepted deploy keys, but it doesn't appear to have made a difference. So, was curious what the correct change is there. And this is for the >> Bats. Uh, >> yeah. Did you verify that the changes were propagated? >> Mhm. >> Did you experiment with changing settings at random until it worked? >> I haven't I haven't gotten around to that yet. That's slightly undercaffeinated. >> I see. Yeah, first things first. >> Okay. Yeah, that's what I would do next. >> Okay. >> You should have the the permissions to do that in that repo. >> Figure out what's going on. Maybe the because the deploy key was created before the automation ran. >> Is that conceivably an issue? >> No, but I thought it just changed the setting. >> Yeah, agreed. I think something like allow admins or something to to override. I think something like that is what deploy keys need but I don't remember exactly. >> Okay. >> Um but it takes a while like but this was merged last week. No, when was this merged? >> Yeah, it was merged last last week I believe. Could also be that the actions are not syncing. >> Sync pending sync or settings. It had that orcs one of the the remove individual access to repos has been failing pretty consistently and I assumed that that was just a standard. >> Don't know if it's an Uh, this commit had a a failure of the remove individual access, but it had >> Okay. But this one has been running for 2 hours. Sync orc settings. It has been green. >> Yeah. >> Is is this the orc settings though? I don't know if it's in there. >> Uh, it's not the remove individ individuals. >> Yeah. >> Oh, think. No, that that's also another one. Not sure. But you said the setting was there, right? Uh the setting is in the community repo. >> Yeah, sure. But was it synced to the uh target repo? >> I'm not sure what setting is supposed to be flipped. So, I wasn't able to verify that. >> It's great. Um, >> let's not rabbit hole in this. I can Yeah, if there if the only if if I Yeah, I I'll get some coffee in me and then I'll do this clicking around and being confused. I don't mean I need to drag everybody else through the um >> sharing is caring. Yes. Okay. Um, >> you too could be a member of the foundational infrastructure working group. >> Endure the pain. Uh, document file system smaller. That's that one. Yes. Clean up one. That one that one. Okay, let's go. convert it to draft until that and that was merged then I'm assuming yes so we got an approval from rabbit the thing is the prerequisite this merge. Now we just need to have someone look at this. Is it big? That's a lot of documentation. That's an environment variable. That's just some template changes. More guidance. And here's the actual spec. Okay, that's relatively small. Just one test being added. This is for the Azure CPI. Um, given the that this is [snorts] just adding tests and it's for the Azure CPI and it's by the one Azure CPI maintainer, should we just go ahead and merge this? >> I think we already merged the functionality to PR. >> Yes, that was a prerequisite. So I I feel like we should just merge this. Any objections. >> Oh, does this address the failing pipeline? Aram, do you know? >> There was a failing pip. There was a there was a PR that I merged earlier that was supposed to address the failing pipeline. So I let me check and see if that's green or not. Believe I said I was going to test this. I didn't. So because that's basically this, right? Like uh would be good if someone could test this does not break anything. And I agree. U the Azure pipeline is currently green so we should be good to go was was already marched. Yeah. If someone else has time to look at this part uh that would be greatly appreciated. I'm uh currently deep in UAA stuff. Um but otherwise I might look at it next week. Um >> this seems interesting. >> Yeah. So we've been getting a lot of tickets now. Uh and we also verified this on our landscapes that um all DNS requests are being sent to system D resolve and BDNS and on investigation we found that u push DNS becomes default route all the time our code is not setting it anywhere but uh it still becomes the default route. So in the PR uh we actually just u set it to default route no wash DNS to default route no and um yeah that prevents the duplication of requests but we have another um concern here and we want your opinion on this. Um there was a colleague who was saying that uh uh okay you set this pos DNS to default route no but then everything falls back to um systemd resolve and systemd resolve cannot handle a very high load environment. we could not test it and uh I think this colleague of ours tested it and he um had 50 concurrent requests but it didn't fail under those conditions. So we are now um yeah we want to know if we should test it and if yes um how should we test it or if you think this is not even a limitation. Just to add a bit more context, um this is a live environment. Um it's a VM with what was it? 200 >> 34 containers. Yeah. >> Yeah. Containers inside and they were seeing around 15% failure [clears throat] with system CTO I think was the command. >> Uh system D result. >> Yeah. So they're seeing a lot of failures there. >> Um they were speculating that um system d resolve just can't handle that amount of traffic but we haven't really have not had the time to try and reproduce it. >> Mhm. Yeah that that part we couldn't reproduce it. I mean we got we haven't gotten around to but this we verified and this is the case each request is being each request is being duplicated and >> but it's being sent by by whom to to are clients sending two requests or is it >> no one request one unique request is sent but it is forwarded to both posh DNS and to system D resolve >> forwarded by the kernel or by Who is doing the forwarding? >> Not sure about that. I mean, but I just did the dig command digg google.com and I could see um one request going to um bosch DNS IP and the other one going to s um system d resolve IP and the bosch DNS one uh responds with refused. But is it going at the same time? I I thought that since like um >> yeah, at the same time >> would like receive it and then it would go to uh be sent forwarded to Bush DNS, but you're saying it's being sent to both at the same time. >> Yes. Yes. And um we also thought that Bosch DNS would only be getting the internal requests and not everything. >> So we were not sure why this google.com request goes to Bosch DNS. >> I see it seems odd. I wonder if it's I wonder if the the the splitting is happening with within dig based on something in resolve confetting an Etsy uh a systemd resolver that's then forwarding it. It seems like if the if the systemd config change fixes it, that probably means it's systemd that's doing the forwarding. It would be interesting to see if I mean yeah interesting to see if >> I find it weird that the client is doing requests in parallel to multiple things right >> yeah it could be >> it it could be based on the the contents of resolve.com I think I don't know what the expected behavior is there it seems like it's probably something internal to systemd because otherwise this >> um the the PR to prevent this which changes in systemd deconfig wouldn't wouldn't impact this. >> Yeah. Um so currently what we saw as the um uh ETH0 the physical interface and the the the dummy push DNS both have default route set. >> Yeah. Um during testing when we disabled the default route for push DNS um we stop seeing the the packets and in return we don't see the refused anymore otherwise we see them duplicated so systemd resolved is still answering the same the same requests. >> Interesting. So this is actually like the the configuration for the network interface not not >> like not actually related to DNS at all. This is a this is a change to the this the virtual the virtual network endpoint. Interesting. Uh yeah, this may be some >> I I believe no I was going to say something left from Jeremy but I think Jeremy takes a different path looking into that. >> Yeah, Jeremy Jeremy is um putting Bosch DNS in result conf right and uh taking over result. >> Yeah. Well, the good news is the person who knows the most about this was Joseph Polmo, so I think we can safely blame him >> since he's left. [laughter] >> Yeah, I think it's his fault. Definitely. >> Yeah. Um I I don't I can't imagine that there is a problem with removing a default route from the Bosch DNS nick virtual virtual network that we're creating. But I I'm I am definitely not certain. Um, do is this something we that you folks would have the possibility of testing at at e either on like an isolation segment or a subsection of like a test environment where it's it's actually gets set across a good portion of the VMs or possibly all the VMs in a deployment. Um, I'm just I'm I'm I'm wondering if Yeah, I'm just I'm a little worried about a change at this lower level impacting having broader impact, but generally this seems reasonable. >> We didn't see [clears throat] we didn't see any issues with the internal resolution. Yeah, >> if that's [clears throat] what you were concerned about. >> Yeah. Um, seems good. So, what's the conclusion? We're going to wait for reviews or we're going to >> My thought is that it seems safe. I might be worth if anybody has a giant hoorde of tokens, they might want to >> Mhm. >> see if um they or if you have a quota left on Copilot. >> Oh, yeah. I can do this one. if you're concerned for a particular um like edge case something we can test. >> Yeah. >> Yeah. I don't I don't have anything concrete in my head. I'm just sort of most mostly unfamiliar with the DNS setup and how that interacts with network creation. So I'm I'm I'm cautious but I don't actually >> I will leave this Don't mistake the c Yeah. Don't mistake the caution for knowledge. >> Yeah. No, let's leave this open. Wait for co-pilot to come back. Then we can come back to this one. >> Mhm. >> Um, >> thank you. >> Yeah, >> before we continue regarding systemd resolved, so from what we were reading, um, it's designed to handle small amount of traffic. >> [clears throat] >> Mhm. >> And in our case, we have a lot of containers um with a lot of traffic. Um do you think it makes sense that we're hitting some limitation? I think it's possible, but I also would be surprised since system dresd has become the default for iuntu if there were like catastrophic We're not we're not the only ones running big containerized workloads, right? So >> I would suspect that uh Kubernetes would be running into this as well, right? >> Yeah. Or um >> what's uh like containerized workload system open shift. >> That's Red Hat. Um >> that's Red Hat. Um anyway, Abuntu has something that I'm forgetting about, but yeah, it seems it seems unlikely that systemd would be would have been selected by anonicle to be the core DNS to be the DNS the primary DNS resolver if it wasn't sufficiently performant. It could very well be that we have done something really or like done some very poor configuration that might be impacting it. >> The main thing is it's u using one thread compared to other DNS tools. So it's not multi- threaded. >> Um >> we'll try to reproduce it and like try to isolate the issue. Um yeah, we'll yeah see. >> But it's separate from this one, right? Like I think this one like it currently is already in the the critical part anyway. >> So >> yeah. Yeah, it's separate. So this one is just setting it as not default route and the other one we were just concerned that when this is out of the picture then is systemd resolved the uh like good enough to u resolve all the requests because this was a concern raised by a colleague. Yeah, I don't it seems it seems possible that we've configured something poorly, but it seems unlikely that systemd would be not able to handle >> Mhm. a lot of requests, but many companies do strange things. So, it's also there's some chance there's just one finding which is medium. >> I can take a look at it. Okay. >> Thank you. Thanks a lot. Thanks for digging into this. >> This one we said we're going to wait until something happens, which was that that was yesterday. Good to merge, right? Seemed like it. Maybe click through to see if any of those have made any if there's been any. We do have to do the slack dance. Allow the thing. Yes. Allow the thing. Cloud foundry. Slack. Yes. Go do the thing. Yes, I know. It's great. Do the thing. Security is great. Um, wait. Where was it? No, that was not. Did I close it? Um, can I now close it again? I'm I'm looking at the various issues that were open just to make sure there's no comments on those. Okay, I wait for your go ahead. >> Um, no comments. Two out of the four have a thumbs up, so I think it's fine. >> Okay. I'm can merge this one. Oh, it already has two approvals. Go. Oh, I'm VMware again. Great. Um, that one. No, this one. Um, I looked at this one. You added a thing. I forgot to approve, but um I looked at it. [snorts] Thanks. Thanks for taking a look. This is why are the pipeline changes? Oh, because you now need to set up. Oh, yeah. The pipeline changes are to add um the defs bumping u or um test dependencies and to add the lint test which wasn't there previously. There was there was a llinter running in GitHub actions but >> what's this? Oh, this is a fixture. >> Um no, so I broke apart the uh so there's that's a fixture. There's a there's a priority binary which runs on successfully on Windows and on on um Mac OS and Unix. Um and that changes from using nice on that's a change from using a nice script on Linux which didn't work on Mac OS making local testing impossible. Um, and on Windows we were using an inline PowerShell script. Um, which if you go down to the OS file system Windows test one more down you will one two more down from there I think. Oh maybe I Anyway, I also made the tests be using using the ghost uh suffix. so that they get compiled specifically for a particular >> Mhm. >> um operating system. Uh it may have originally been in the file system test code. Um but there was an inline pcript PowerShell script that was being used to test the niceness behavior. >> Uh that's this one then. Oh, >> I think that the one below that. Oh, no. You're right. OS Exec uh command runner. Yeah, this shows up as split because I made two different uh versions. Want to play Linux and one for Mac? Yeah, I think this one uh could take a review. I mean, the impact is actually not that big. How do we feel? >> Feel relatively good. I've seen it run. >> Yeah, it's it's a test refactor basically, right? Yeah, I'm fine with it. Wait, there's no approval on it actually. Only from a rabbit. I had an approval. This is the documentation for the thing that was just merged, right? We merge the tests. This is the docs. >> You already looked at it. I'm fine. >> Yeah. Mhm. >> Wayne making a PR something Nick groups related net your feature is being used. >> Oh no no that was implemented by Felix. >> Oh was it Felix? Sorry [snorts] but did you do the test then? Um, I worked on some issues that came up afterwards. Um, I can take a look. I haven't seen this one. I mean, this seems quite impactful. It's an agent change um obscure feature. So yeah, would love uh your you taking a look at it now. Thanks. Okay. Okay. So, you looked at it arm this. Yeah, I looked lightly at this, but generally I've been following the work that Matthew's been doing to convert all the uses of the Warden CPI in tests to use the Docker CPI instead. >> Yeah. Yeah. So the aim is to get rid of the warden CPI then I'm assuming >> get rid of all her all rci dependencies and then announce the deprecation of the warden CPI. >> Nice. >> Let's go with this one. Okay. >> I pushed the commit to address the Okay, cool. Okay. AI didn't find anything um or just something test related but uh nothing fundamental with this approach. Um are we good to go? Yes. >> Thank you so much for approving it. Yep. Boom. Um, yeah, this one needs testing. Luc is there. I said something needs testing and then Ramon appears. >> I I had this inkling that someone was talking about me that I need to do something. >> Just say his name to or say something needs to be tested three times and he arrives. Uh yeah. No, I don't know if you have uh time for this. Probably not. Uh it's uh secure boot boot loader changes that we would like to see tested in some other environment because we feel like there might be a fallout >> in some >> secure boot sucks so much >> but it's secure. >> Yeah. >> Enterprises love it. >> Yes, of course. Um. Oh, Proximo box. I don't know if you've all seen, but there was a discussion about Proxmox CPIs. There's two now. They are comparing them. Yeah. Um, do we want to go with PVE or Proxmox? I don't know what PE stands for. >> I have no idea. It would [clears throat] makes more sense for Proxmox as that was everyone is knowing, but Oh, so it's Proxmox V. >> V. Yeah. What? What does that stand then? What's that? >> I have no idea. >> Virtual environment. I just call it Proxmox because Jesus. Do they have something else? >> Product overview. >> Do they have more products? Oh, backup server virtual environment. I see. Okay. The proto is okay. So it's probably more correct. >> Yeah, >> they even have a mail gateway. Nice. Although in the context of CPIs and cloud configs, there's nothing else proxmox, there's not going to be any confusion, right? You're not going to target any of the other products when talking about product. Yes. But this is the read me. And then we have a Okay. Although I feel like we probably first want to figure out which CPI is going to win because this is there's going to be cloud properties in here that probably not work for both CPIs. We could ask them to harmonize that. Um, There's changes requested. that this discussion is still ongoing. Nice. Who requested this? Oh, your your quota has been reached. Serum, we do a balanced or a light one. Okay, the set is here. Okay, so we're waiting for a response here. This is in draft. pending review. Probably still pending review. Um not trusted. CLA has been signed offer. Sure. Um, but there's reviewers assigned. Why is there no rabbit here? Don't we have rabbit enabled for the push acceptance test? review co-pilots or request a co-pilots and then we have this one finally in a state that it needs to uh okay >> okay okay this one is this one is ready to go there is I put a note up at the top of this. There's some amount of either kind of handholding with CI or um I or somebody will have to grab the private YAML and do a blob upload. I think that the CI will will handle this once it's merged because it will it will notice that there's a there's a CI job to bump the Rust package and then that that job will run, but the release will be in a strange state briefly. So, um, once I if we're ready to do this and if if you want to approve it, I can, um, I can go and babysit getting this through CI today. >> Okay. And then all the rubies will be so much faster. >> I will hold off on merging. You can go ahead and merge when you're ready, >> please. Yeah, I'll try to take care of that today or tomorrow. >> Okay. Thanks. This was another one. >> Um, I have modified go fetcher. I haven't had a chance to redeploy it. Okay. Still blocked. Um, a bunch of the autobumps should go away once a bunch of those are some of them um, Ramon are for bubble which I think means bubble doesn't have a golang dependency autobumper um, and then a bunch of those are for the disaster recovery tests which should start which should go away once we have that pipeline working. >> Right. Which one did I click? This one. >> And the last one >> run test job. Oh yeah, that one. Yeah. Um, that one is in waiting for pending review then, right? This is for >> Okay. >> This is for the Golang package. So I think almost all of the Bosch package releases use Warden. Um, and this So there's going to be a flurry of changes to >> Yep. >> up update these. >> Makes sense. But there's still comments here. >> Yeah. >> And a lot of documentation changes probably. >> Not so much. The pipelines are really it's really just in the pipelines where the warden CPI was being used to validate that the um that the package will deploy and run the tests. But it >> I don't you're not really going to deprecate warden. This is the prep for deprecation. I think there's a there's a full RFC in the wings waiting to uh talk about deprecating warden. >> Okay. >> Yeah. This is just making so that it's uh not as impactful as in all the work has already been done. >> Yeah. >> Check. Okay. Wait. This no there issues. No, nothing new. Nothing in the inbox. Um, we have 13. It's the last. >> Yeah. Is there any um are there any promotions we need to >> Yeah, that's the next one. Promotions. Uh there it is. Nothing. We finished it. We did it. >> That was fast. Thanks. >> Nicely done. Thank you, Reuben. >> Thank you all. Take care. Or any other business, right? That's always what you need to end with. Any other business? >> No. Okay. Thank you. Oh, bye. Take care. >> Thank you. Bye. Bye. >> Bye.