Foundational Infrastructure Working Group, September 17 2026

Hello. Hello. Happy Thursday, folks. Give it another two minutes and then we'll start the meeting. Seems like we might have a very light crew today. Yeah. All righty. I guess we'll start. Welcome to the Foundation Infrastructure working group. Um, looks like kind of a skeleton crew here today, but we'll try to get through what we've got. Um, can folks see my screen? Okay. Can you >> see it? >> Yes, we can see it. >> Okay. Right. Um, let's see here. Start with this one. We don't have any reviewers. Um, let me know if either of you have context on any of these. Welcome, Amed. >> Yeah. So on this one, Ned uh is working on cleaning up some logging uh stuff on the agent side. So this is just a prerequisite, I guess, that he'll use the features here in a different PR. Not sure if they're up or out of draft yet, but uh they should be coming soon, I guess. >> That seems good. Um, this was me. Um, there were some changes. I think the root of this is that there are uh there's some changes in the way that the JSON parsing error format uh sorry the JSON parsing error error message string comes back and we were testing very specifically um for that. So, and even though uh the go c the bos cpi go it has a go mod file that points to golang 126 in CI it tests against 127 now that's been released and the new version of go has some slightly different error messages and um I believe also now fails compile if there are any govet errors Um, so this uh I don't know if there were vet errors here, but this this basically fixes the ways that the error messages uh were set and matches so that it works in either um in either 126 or 127 or in in this case uses a slightly smaller um substring of the error rather than the full the full text. And the rest of this is all uh vendor updates. I think I may have one review on this. There's one approval. If anybody has any thoughts, let me know. Give approval as well. Looks looks good to me. Cool. Thank you. Welcome everyone. Hm. >> All right. Uh, dynamic IPs. I'm guessing maybe this has to is related to IPv6 or no. Does anybody have any context on this Azure? Oh, sorry. This is Bosch directly. Never mind. I don't know nothing about this. Okay. >> The only one >> there are two people who played with IPv6 I think and those are not in the call. >> Okay. Right. We have a code rabbit approval but nobody else. Um so I'm going to leave this as is. pending review. Oh, this is uh this is a response to uh a comment on the issue that I had opened previously. So a commenter found that it the fix pre that I previously pushed uh doesn't fix the exact error. So this is a response to uh change it up a bit. So it's a bit complicated. So that would be best if someone can take a look. Okay. What is he trying to do? >> Um, do you have you tagged the original commenter? >> Uh, yeah. On the issue, I tagged him. Okay. >> Not on the PR. >> Might be worth tagging them on the PR. Um, >> yeah, that makes sense. Um, and then I'll Yeah, we can pick this up. I'll I don't know that I'll have time this week and I'm out on vacation, but if if Yeah, please just keep keep pinging folks and make sure they're both tagged. >> All right, thank you. The fix it just just mono the fix itself is not that complex but the train of thought leading to the fix is a bit of a >> Mhm. >> thing that needs some taken care of. >> So yeah. Yeah, that makes sense. Over there. Just looked at it. It seems okay. Mhm. documentation even. That's amazing. Yeah, this seems to be highly Azure specific >> for their new API endpoint. >> Yeah. Inclined to approve. Does anybody have any other anybody anybody else have contact on Azure that >> Yeah. >> reason? >> Uh the other guy who norm signed for height or something. >> Mhm. Good. He's in the reviewers list, I think. So, >> yeah, >> this one. >> Exactly. >> I mean, yeah, >> they are on the same team, so maybe he already took a look. >> Yeah, it would be kind of nice to get another >> Azure expert on there. This is all just documentation. It looks like that's fine. Let's see. Um, this I think we uh I looked a little bit at this but don't have haven't had a chance to give it a deep dive. Um, >> wasn't Ruben creating something for this? >> Uh, Reuben >> was all in one stem cell stuff. >> Yeah, I think it was more of an all-in-one thing. I'm going to ask Reuben to take a look. This is um I think specifically using the new um Rosetta compatible stem cell for Mac OS and being able to run run a Bosch director in in a in a Rosetta comp in a Rosetta compatible container. >> Oh, okay. >> For local development. Um yeah, I'm going to leave put Reuben on there and look at that pending discussion. Yeah, this one is pending review as well. It has to do with a create end of deploy on the director and apparently a colleague found that uh in some cases stem cells get deleted and then you can't redeploy because the stem cell has gone. you have to manually upload it. >> And this happens in specific cases when uh on a second run when the stems is already present. uh when on second run the create M >> uh the VM deployment create M fails >> it goes partially through uh the run and deletes the or rather it doesn't delete but it sets the current stem cell as an empty string >> and on the next run >> there's this cleanup that goes through and deletes all the unused stem cells. >> Mhm. So what the fix does is it does not set the current stem cell to the false to the empty string. It's a bit of a behavior change because uh this way we won't uh when create runs it tears down the VM it won't uh remove the current stem cell but it shouldn't be shouldn't change the outcome because in the happy path uh the when the VM comes up again it will set it M >> but it's a bit of a complex thing and I' be happy to have a someone take a look give a review. >> Yeah, >> that makes sense. I think I guess I might be worried about use cases in CI where where we end up tearing down the direct we end up tearing down a create VM but then the stem cell is left over in the IAS. It sounds like that might be a consequence here. Yeah, that can be a thing. And I I also think was why are we if you're so afraid of cleaning up when in the next run, shouldn't there be like like a maybe a flag or something? Uh I'm not sure because uh what this change does is doesn't clean up but rather uh just uh so the it just removes the flag the thing that actually sets the current stem cell that's being used to nil which will trigger the cleanup to delete everything. >> Okay. Yeah, maybe I'm understanding the issue. >> Yeah, I'm not explaining it very well. So, >> it it sounds it sounds complicated and and interesting. So, I put a note for our team to take a look and Reuben maybe or Ramon be be good. >> Yeah, I need to deep dive more into it before it's needing to >> Yeah, this seems seems cool. I mean, I'm it seems like a thing we should be addressing. Um. >> Mhm. >> I'm not sure. >> I joined a bit late. I'm sorry. Um, there should be an issue linked to it. Uh, it was opened a while ago, [clears throat] but it might have >> might not have linked it to the PR. >> No, no, it's there in the description. Uh, but yeah, it might have more context >> in the beginning. Yeah. >> Uh, here. Okay. >> A while ago. So I don't really remember the details. >> Okay, cool. Yeah, it be it would be good to get some eyes on it. I think a lot of the stuff has moved into like ready to merge because code rabbit has approved, but >> yeah, so have to >> code rabbit should stop approving. He is not an approver. >> I agree. Um, okay. Yeah, I flagged this for folks to take a look at there. Thank you. >> Thank you. This is cool. Um, it's interesting to see the dusty corners of things. Um, is this ready? Let's see. I want to that. Yeah, heavy stem cells in AWS is uh is a long longstanding issue. >> Yeah, I'm excited to see this change happen because it will um it will help with being able to put the Bosch CPI or the AWS CPI into BPM. Um because right now the heavy stem cells have to use device drivers and um >> and such >> maybe we can also if heavy stem cells is a thing we can also maybe like the light stem cells uh turn that down a little bit maybe. >> Yeah. >> Yeah. Exactly. Um, and I will I guess re-trigger trigger coil review. >> Okay. >> Joseph, that was a tried it a long long time ago. heavy stem cell stuff. Um, I don't know. >> I don't remember. Let's see here. Being a not Azure expert, I'm kind of inclined to trust Toby and Shade or Sebastian on this one. Um, >> yeah, I I think so, too. We'll do that and then maybe we'll come back in a bit. Um >> yeah, co-pilot cannot do a lot, but this is way too much for him too. >> Yeah. Let's see. I think this one is in the wrong column. This is Yeah, this should be ending. Oh, no wait. That's is waiting for changes from code rabbit. Zoom that and request that. All right. Head secret acting. Yeah. I think there was one comment on co that was addressed but yeah the the backstory here um we noticed that after migrating from jammy to noble the bush agent logs jumped six times um so after some >> um yeah this one is actually not about it the previous one was um so it seems we started forward ing all the debug bush agent logs >> which is not happening on um I will open separate PRs for it but need to add some stuff to the bush utils. Um this one in particular I accidentally noticed that we are forwarding in plain text uh the signed URL which contains tokens and other information that maybe should not be Yeah. Visible. >> Yeah, those are the the the blob store ones, >> I think. So, yes. Yeah. Okay. >> Um, yeah. So, I'll be using this one later in the I think the push agent. This is >> Didn't we already have one where we can just put like the redact flag on it like or are you now just redacting only the tokens from that URL? I think the idea is to reduct the entire thing. Um, and there there's also a I think review uh method that's going to be used in the code. >> Well, I thought the Bush director already did already had this. You just need to say which uh you just need to say just make this uh make this a secret deck. is in >> I don't know where it was anymore. It was a long time ago when I >> when I did this. >> I think this is for the agent though. The dire this is the director's all Ruby code. >> Oh yeah. Okay. So the agent does not Yeah. But you mean in the agent logs? Is that >> Yeah. >> Yeah. So yeah. So but then you always need to go the the agent logs can only be viewed when you have root access. You cannot do like B logs >> those access right >> I think you were working on something related to Ramon. Um >> it was a long time ago. I I for sure. >> Yeah. Um so the way Jami was forwarding logs I guess it's different in Noble and once we migrated um like the the amount of logs um increased six times and I noticed we're now forwarding all the the actual Bosch agent debug logs uh to our logging system. Um, okay. Um, but this thing with the signed URL I think has been there for like for a while. Um, >> yeah. So if the if there is too much logging now going on which is not wanted then we we just need to reduce this a little because I think it's it's from journal C uh we are including the we are just forwarding the journal uh CTL from the journal from watch director to the logs now I know there's some PRs like in the past months when someone tried to change several things But >> yeah, if if this was if this is solving uh something from the agent. Yeah, fine. That's good. >> Yeah, the was both I think >> I'll review the comments. Um >> okay, >> for the push agent, uh someone did make a change and forward the the logs to a file. And the idea I think is to have them locally and not forward them to external uh systems >> which was okay with Jamie but yeah noble works a bit differently but that's going to be a separate PR in the agent. >> Yeah I mean it can be worth having those or anyway yeah we should talk about that when that that comes up. >> Um thank you for digging in. Um, I need to put some people in here. >> Yeah, the the funny thing is the Bosch DNS stopped sending logs, but the B agent >> Yeah. increased the amount with no >> That's strange. Well, possibly DNS is less involved like it it is doing less because it's not the primary resolver anymore. >> Yeah. Yeah. It's a a lot less loss now. Yeah. Um, I think I'm just going to go ahead and scan and see if I will leave these for them to take a look at. Um, all right. That one we talked about. That one we talked about waiting for things. That one we're talking about. Um, >> another cleanup script. Yeah. >> Yeah. It's all we need. Uh, okay. I am to leave that there. And then then a few Bosch boot loader. Um actually it's good that you're here, Ramon. Um Bosch bootloader has a couple of um security bumps from dependabot and I've been leaving those cuz it seems like it should be something that the pipeline takes care of. But I also didn't want to commit without somebody with more context. >> Not those kind of things. It does almost everything except for uh for these ones. Um so yeah, just merge them. >> Okay. >> Merge them and if it fails, it fails. We will see it in the pipeline. That's >> cool. >> Are these um is it because they're not in they're in test packages? Well, that one's not. Neither of these look like they're safe to >> Oh, yeah. I'll tag you and feel free to do what whatever makes sense. Um, >> yes. Uh, as as a as a general comment, I I added a flag to the Golang bump script in the Go release that will allow it to um bump test dependencies as well as non-est dependencies. So that maybe that flag will be useful. I don't know. >> Okay. Well, if I need some more context, I'll Yeah, I'll text you. Um, okay. Um, this I think uh so this blob was added as part of a PR, but it didn't it because the person who PR the changes didn't have uh the private YAML. It wasn't synchronized to the um to the blob store. So we're seeing failures in the pipeline trying to generate the release. So I I think that the bump jobs will pick up excuse me will pick up the newest version once this has been deleted but right now it's skipping. >> Yeah, I would I [clears throat] think so too. Uh normally would override it but because it's not even able to access it, it will fail. >> Yeah. So yeah um I will quickly add my approval then we can um so this is something that the what's called iuntu advantage or abuntu pro um that uh will wake up and try to see if canonical is present in the uh I think this is maybe the I guess it's in jammy anyway I guess it's a new new change for this >> okay I approved it and merged it >> oh thank you appreciate Great. Um, I'm inclined to merge this unless folks have concerns. >> No, I have no concerns. Okay. >> Buntu, can Ono really like to push stuff he can't remotely? We need to we need to rethink if we want to stay with Canonicle at this point because they are making really bad choices lately. >> Oh, I'm not excited about the idea of switching base OSS. >> Yeah, but now they're putting Rust everywhere and like they they changing >> everything is going to be in Rust. We're rewriting the the Bos directory in Rust. >> Oh, nice. Yeah, it's going to be great. >> It's memory safe. >> Exactly. Um, all right. This was the thing you mentioned, Ned. It sounds like >> Yeah. Yeah. about the lo forwarding. >> Yeah. H I wonder if there might be value in breaking up the Bosch agent into like a a standard log and a and a a debug log potentially. >> Yeah, we Yes. So, we can do that. Um because I if yeah I haven't looked at this. I'm just thinking out loud but sometimes the debug level stuff is very valuable on disk but um probably not valuable to be sending all the time. Although it might be very nice for folks who are debugging agent problems to have the disc to have the the logs get off the VM. But um >> yeah, but it should be a flag then turning it on and off. >> Yeah, >> we we weren't really getting anything with jummy. >> Mhm. >> Because we we need to look I I think it's systemd uh because we that's the whole R unit, right, of RS log that we use. >> Mhm. Yeah. RS lo and journal are both doing things together and then we are we are sending it to the log directory and then and then create a sim link to the B old B wash log directory. >> Yeah, >> I think that might have already been done but I'm I'm not sure. >> Okay. Anyway, this seems Yeah, seems valuable and seems cool. I'm not sure what the I don't have a thought yet about what the right move is, but um maybe on Resolute it it makes sense for us to move the Bosch law the agent logs into the standard RVCAP logs RV gaps logs location. Um cool. Thank you for tracking this down and figuring out a way forward. Oh, people made me check it. Yeah, I didn't want to, but they sell six times more logs and there were questions about >> Yeah, tell them to upgrade their log server, maybe. >> Data's cheap. >> Yeah, exactly. All right. Okay. And I think we're in a good spot with all this stuff. I'm going to move over to issues. GitHub is a little slow. Um, is this a Bosch thing or a UAA thing? Looks like a Um, this might be worth tagging UAA folks. um because they're probably better able to tell us if our the configs were sending are sufficient or not. And I I think they may actually be the the UAA release itself may be in control of these values or at least have control the defaults. Yeah, I'm assuming we're not printing error any error because we're not getting any. I'm sure we have some placeholder for the returned error. It seems to be empty, I guess. Let's see. Marcus, I think, is My network is very slow today. Anyway, while that gets looking looking up, I was going to tag Marcus, who's one of the UAA contributors. Um, seems like this is already in light. There's a PR open. seems like this maybe interesting. Okay, Sebastian is looking into it. Leave that open. Okay. It's good. Okay. seeing anybody from UA. I'm I'll tag one of the UAA maintainers in here. Um, I think that might be the end of current things. Um, I don't know that we have any promotions. This is new. This was opened last week, so I think we usually give two weeks for those. Um, I don't know if I can approve this or if it will matter. It looks like we're good base and merge. Welcome. This seems good. And it looks like we have this was the person I was looking for before merch. All right. And merges there. Excellent. I think that might be the end. I think that's all folks. Unless there's anything last minute agenda items. >> No, I've been out of the game too long. So, >> well, as always, you're always welcome back, Ramon. >> Yeah. >> Well, next week is the EU summit, the summit again, so we'll be there. >> Awesome. I wish I was going to make it, but I will not. Um, all right, folks. Thank you for coming. Uh that was the foundational infrastructure working group meeting and I hope you have a lovely Thursday. Bye all. >> See you. See you next week. See you next week. See you in two weeks. Bye.