App Runtime Deployments Working Group, August 13 2026

Hi Johan. Hi Oliver. >> Hello everyone. Yep. >> Hi. Hi [clears throat] Oliver. Hi Milena. >> Hello. >> Hi. >> Hi. Hi. >> Hi. >> So, Dave appears to be offline. Let's get slowly started. >> [snorts] >> This is test one. So. Then welcome to the working group meeting. I checked the items from last time. There is nothing more to do for today. So, um the Windows validation is still failing because this small agent fix has not yet been released. So, yeah, took a while to find it, but eventually if you get give Claude the right hint, it immediately finds the problem. Yeah, the uh Windows money script was forgotten when this new parameter has been had been introduced. And uh yeah, now we just have to wait for a release. Um >> Um you just pinged Yowan for that and he >> Yes, can Yowan do releases? >> No, no, that's the problem. He has not uh uh access. Uh the release pipelines are still on Broadcom side. Uh but he pinged Broadcom already one or two days ago. And uh uh there's just a side discussion on SAP side. Uh maybe if he can speed up here a little bit to get the release release pipelines in the community as a uh >> They're still running on a on Broadcom IT? >> Uh exactly. Also, the this is what uh uh Yowan yesterday wrote. Also, so he's just Also, he's aware and he pinged also Broadcom Broadcom colleagues uh we requested already yesterday uh so that we get a release and catch and bring it into a new UCF release, but it depends on Broadcom. That's that's the problem here. Yeah. What What other question? It's It's good that you fixed it. Thank you for that. >> Yes, so >> also uh the task of uh Yowan or or the community responsible for the Log Regatta. >> Um yeah, Yeah, well, it it wasn't at first really clear where the problem is. So, I mean one cell was not one that the window cell was not up and running and >> [clears throat] >> you first need to find out which job fails and why. So, using Windows shell with type and so on and eventually we found it and then it was trivial to fix. Yes, okay. Okay. Yeah, yeah. Small fix. >> [clears throat] >> Good. Okay, good. Yeah, so yeah. But, once this is this is released, it should automatically turn green again. Okay, RFC 55 identity aware routing. Yeah, same problem. I am still waiting for a new CFC live release because that one will contain the new commands at policy the the policy management commands which are required. Otherwise, you can't use this feature. So, when this CFC live is released, then we can merge the two PRs. And then, Oliver, this would be a nice task for you is to activate the new ident the the new cats test provided by Reuben and here we need these small changes to make the test work. So, create an additional domain with this this is also a new flag or that one is new either of them. And a new security group rule and then the test should pass and we could activate it on the experiments experimental environment. Yeah. But, only when CFC live is ready. >> Mhm. Mhm. Yeah. Yeah. Can I I create here shot small backlog item. >> Okay. >> Oops. >> Okay. CF smoke test pipeline is quite unstable. Yes, we noticed. Um Need to add retry attempts, clean up before deploying CF again. Um yes. So, uh yes, it is unstable. So, one problem is that we see a also in other pipelines a lot is uh where was it? The UAA authorization error. So, we see this a lot. >> Mhm. >> Best guess so far is that uh Ardesh is >> Also >> joining. That uh the Bosch director is temporarily overloaded and UAA is unresponsive. I think I analyzed it once. But didn't find very much in the UAA logs. >> [snorts] >> Uh so, if anyone um has some enthusiasm left, he or she can try to analyze this in more detail. Uh what would definitely help is also better error output. What was the UAA response? And I'm not sure if a retry is implemented. Yeah. So, yeah. >> Uh retry from what to what? Which call? Um >> Yeah, this is the director >> Mhm. >> retrieving an access token from its own UAA. And I have no idea if there is a retry implemented or not. Uh Yeah, I mean you can this can be easily done with with cloud. >> Mhm. >> I would say. >> [clears throat] >> Yeah, but I also can't tell why it this this happens now so so frequently. It didn't I've never seen this in the past and then since some or street deployment version it appeared quite appears quite often. >> Mhm. >> [snorts] >> I can you see if it's stable a little bit back when this flipping comes more often or so was it in the past more permanently green or so in other words is there maybe a starting point with a code change? I here we see it now there's consistently green with a few exceptions. But still >> Well, it's related >> Okay. >> to I've had the first boot loader version or the first deployment version contain within June 18th. Yeah, it's it's hard to nail down but or perhaps UAA test takes a bit longer to start. I don't know. So yeah, would be nice if someone could investigate a little bit deeper here. Very first step would be better logging. To understand what's happening here. >> I mean what we can do is we can add also backlog item for this not with the highest priority yeah, but if you have time. You can look into this. >> [clears throat] >> Okay. >> So, hi Dave. Um Uh Dave, quick recap. Uh Windows validation is failing because we are waiting for a log aggregator agent release release which can only be cut from Broadcom uh infrastructure. >> Okay, I'll reach out to the team ask them about that. >> Sorry, you have >> I I said I'll reach out to the team and ask them about that. >> Okay, that would be great. I think Yovan already contacted them, so I would expect the release soon. Yeah, yeah. And uh RFC 55, the identity where routing, here we are still waiting for a CF CLI release with the new commands and then um Oliver will create a story to integrate everything and activate the new tests that Ruben provided. >> Mhm. >> Okay, uh good. Then [snorts] new cats release. Who added this? Is there anything? >> I'm not sure who added it, but I created the the release a few days ago. We had a a need for a change. Um we're doing some internal testing. Yeah, we're doing some internal testing and uh so Rob added a way to skip uh basically it makes the um the get build pack into the dependent test actually check for a compatible stack. Um and if it can't find one, then it then it skips it. >> Okay. Um I think I saw >> one that you then made a a change to and removed Cif Linux FS3 afterwards. >> Yeah, yeah. But there wasn't another PR with a similar pattern, right? >> Um there is a new one, yes, that I think I merged yesterday. >> Ah, here the .NET Core as a path. Yes. Ah, yeah, mhm. Um >> Because this has assets that uh pre-compiled per stack. So, this just gave operators or your your people the running it the way a way to um provide their own assets if they have a custom stack. >> Okay. Mhm. Okay, so if it's going in the direction of custom stacks, it's not exactly wrong because we at SAP will also uh have to deal with custom stacks at the end of the year. Just wondering if it's okay if we spread now this this stack list uh on on in so many places. Mhm. Yeah, just >> there's definitely the the possibility of some refactoring there to consolidate that. >> Yeah, yeah, we we should not sure if maintain one list supported stacks for cats and then deprecated stacks in in cats and then have uh and and uh and no experimental stacks maybe >> I think the only challenge there is things like like the .NET one, I think it associates the stack with the corresponding asset. >> Yes. >> So, it's a question of being able to maintain all of the information that's needed in all the different places in in one go. If it's just associated by the directory, then maybe that's not too bad in this case. I'm curious why there's a safe Linux FS 4 directory inside >> Yes, that's what I mean. >> And then there's a safe Linux FS 3 inside that. >> Yeah, maybe Milena, can you recheck this? This still looks like a copy and paste problem. Um, I think what was FS 4? Yeah, Matroska. This one also has [laughter] an FS 3. Okay, this got a little bit messed up, so maybe we can strip down what's not needed and um >> This This is the nested sticks feature. >> This is the [laughter] super nested windows. >> looks like it's some of those were 4 years old. I don't know how that Yeah. >> I need to check this one, sorry. >> That's okay. >> Yeah, no problem. Yeah, feel. Okay. Good, but Okay, we somewhere somehow we were able to support this. This is .NET 8 and FS 4 was Yeah, okay, something else. Okay, good. >> But yeah, well definitely once we've got everything Well, our testing is finished, I'll ask Rob to take a look and see if there's a refactoring possible there to just streamline that a little bit. >> Okay, good. >> Um, there was another PR opened last night. We'd seen some issues with uh Windows tests where the um the app that we were using was running out of memory. And so I opened a PR just to increase the um the memory requirements for that. Um, so it just goes from 256 to uh to 512, I believe. >> Okay, so this is just >> And then it just And then it also doubles the um the amount of memory used when it's testing that the that value is exceeded. >> Okay. Yep. Okay, looks looks innocent enough. >> Yeah. >> So, we'll approve it and Off top also approved. Okay. Yeah, good. Then let's merge it. >> Cool. Thank you. >> Okay. Good. Anything more on cats? >> No, I'll probably cut a a new release today just to have those changes available, but neither of them are breaking, so it'll be another minor. >> Okay. Good. Okay. Then the last point uh on Monday, the CredHub root CA certificates expired. It does so after a year, and then you get lots of orange jobs. Um And I first thought it was the just the CredHub server certificate, but it's No, it's the bigger uh it's the CA edition, so you have to delete a little bit more and restart a few jobs, and then okay, it recovers. Um So, yeah, I have a reminder for the ARI Concourse, but I didn't I haven't set one for the ARD Concourse. Um yeah. Maybe >> a reason we don't have a a longer-lived CA? >> Uh I guess it's all um uh uh set by the Caravel secret secret general controller. This is managing a lot of stuff and possibly there is an option, but it's not so well documented. And uh yeah, and we're also hoping that the Concourse working group provides a central Concourse we can just use. >> Mhm. >> Um yeah. So yeah, I don't know. Oliver, can can you just take this as a story or I mean, it's basically just setting a reminder and doing what's what's written in the wiki. >> Um, you showed here the description in the wiki, no? And >> It's in the chat in the in the channel. Mhm. >> I think uh I can create a story. I said Is it now we [clears throat] have to replace it, right? Because it's expired or soon expired. >> No, it will expire in 1 year from now. >> I said you okay okay okay. It's just then that we uh have it on the radar, so to say. >> On the radar or we'll >> Mhm. Yeah. Yeah. >> Hope is that the Concourse team >> Mhm. >> takes over >> Mhm. >> managing. >> Um yeah, I can create a story maybe anyhow also for our internal Red Hat instances uh currently documenting uh which ones we have and uh when they expire and have to be renewed. So, this would be then one one more. >> Yeah. Okay, cool. Good. Okay. Any other business? >> I was wondering something which I really decided to ask directly here. For Linux FS uh 5 when it should be is there a timeline where it should be become default? >> What is timeline basically comes hopefully from us, Elena. As it means I want to have it delayed until we are prepared on SAP side. Uh that if we switch to the default on the community release, yeah, that we prepared on SAP side that the switch will not directly come to SAP and if we are then prepared then I think we should start a discussion here in the community. I know from Dave you already said that on Broadcom side you are prepared, no? So it is >> Yeah, we we are fine, no? >> So so you are okay, so it's just some homework on our side. And if we are prepared on SAP side then from my perspective we can uh flip or make the default then FS 5. So Are there any other Yeah. >> Yeah, yeah. >> Yeah. Or you can Dave, do you see other >> I mean >> aspects that we have to consider or >> This basically means we make it the default and provide an ops file for opting out. So for the community this is likely already now perfectly fine. And >> I guess the only question would be when is the date that we that say if Linux FS4 is no longer supported that we have to do this by. >> Uh it's next [clears throat] year in April. >> Okay. >> Uh so we have still >> We have some time left. >> time left. Yeah, yeah. As honestly we we we we we wanted to do this somewhere in the next couple of weeks, yeah. So we already working on it. >> Okay. >> Somewhere in in September, October time frame, I would say. >> Mhm. >> Yeah, that that should be fine, yes. >> Yeah. >> I mean, this this is not critical. That's just uh one moving around of files. More questions. If we flip the default, uh does this also have a relation to the buildpack detection order? So, that we have to flip here also the order somewhere. From FS5 to >> I would >> Say no. >> I mean, the community, not no. >> It >> It's just I think in on SAP side, you know, because we have this thing. >> Buildpacks are ordered by by stack >> the default I mean Default is FS5, it will prioritize FS5. >> Okay. So, if FS5 is the default stack, then it should also take the FS5 buildpack, right? If you push something without any further specification. >> Mhm. >> And does this mean then if you just flip the default, which is one just one config flag, to FS5, will then static buildpack will be uh set on >> Yes, if you don't specify a buildpack when you push or in your manifest. >> Mhm. >> I I would say this this is all fine for the community, yeah. That would actually Yeah. Would not not much that would happen. >> Mhm. >> Yeah, good. >> Okay, then I will uh commit No, not commit, but create the PRs which are for making it for preparing for I mean, internally. And then provide a PR for making it the default here. >> Mhm. >> Yes, yes. >> Yeah, I I guess the only thing would be the sooner we can switch the default, the longer we give folks to figure out if the if it causes them any problems >> before it goes live. >> Yes. I mean, yeah. Yeah, we need We need to find out if there are any leftover issues with FS5. I haven't tried anything. It's doesn't seem to be so critical, but yeah. Okay. >> Yeah. Agreed. >> Okay, good. Um >> Um uh Stefan, there's this Slack thread with the runtime team currently ongoing. I don't remember the details anymore, but I know you are involved also. They detected or currently figuring out something uh with the switch of the nobles themselves. Do you remember this this discussion uh >> What I found out in one of my apps is that it's a Java app and VM metrics memory metrics suddenly uh report host metrics instead of container metrics. And that coincided with a switch um with a redeployment of the cells and then um Vladimir found out that this is actually related to um the switch from Jeremy to noble and that again involves C groups V1 to V C groups V2. The real root cause is still unclear. Is there is I don't know. Is a certain flag not properly configured in either the stack or the system cell or in the build pack or in the JVM. That's still unclear because JVMs should work with C V2 at least uh something that And I'm not aware that I see the same behavior on Kubernetes where we should have I think Cgroups V2 and but I didn't had the time to look deeper inside. I just wanted to share [laughter] my observation and then see what Yeah, maybe the runtime people know that already, but yeah. Seems like that's the case. So, that's something I guess we will get more reports of that uh everybody who reads metrics from the VM. The container itself is properly configured, so you have your container metrics and uh the container metrics on process level are correct, so it's just the JVM metrics, so yeah, it's difficult. >> [clears throat] >> Okay. Who who produces the JVM metrics? The Yeah, it's I I I I I saw it by accident because I um currently in one of my applications I had an uh out of memory, a very strange one, which I tried to find down, that's why I looked at these metrics. And then I just searched for the overall available OS memory. If it goes permanently down, then yeah, OM is still in. And suddenly I see, "Oh, I have now tons of gigabyte of memory." That looks surprising, much more than my container has, and yeah. >> Okay. >> Oh, can I ask? So, I found it by chance, but um there's something behind it. But I I'm I'm not >> in the Slack channel? that's no bug yet and there's no cuz I also have no idea whether where this can also be a JVM. issue >> Also possible, yes. Do you know which one you're using? >> Yes, a machine 25 it's open [clears throat] JDK 25. So yeah, if I had time I would create a hello world and then try different VMs and see I can derive something. >> Yeah. But yeah, the problem won't go away. So maybe after my vacation. >> Okay. >> I can look for it in detail. >> Okay. >> [snorts] >> Good. Yeah, vacation I will also be on vacation for 3 weeks. Aftab will take over the next meeting. Thank you, Aftab. >> Yeah, I know. >> Good. Anything else for today? >> Uh not from my side. Thanks. >> [clears throat] >> Okay, then we are done. Thanks for attending. Enjoy the last hot summer days here in Germany. >> [laughter] >> And see you again in 4 weeks. >> [laughter] >> Enjoy your time. >> Thank you. Bye-bye. >> Have a nice break. >> Thanks, everyone. Bye-bye. >> Bye.