Submind YouTube summaries
Thumbnail for App Runtime Deployments Working Group, September 10 2026

App Runtime Deployments Working Group, September 10 2026

Watch on YouTube

Video summary

The September 10, 2026 meeting of the App Runtime Deployments Working Group focused on reviewing recent pull requests and addressing operational issues within the deployment pipelines. The session began with housekeeping matters, including a substitution for Johan who was on vacation, and a discussion regarding technical difficulties with accessing meeting links due to keyboard layout differences. The primary agenda involved clearing backlogs from previous meetings, where most items had been resolved or merged. The team confirmed that several pending reviews were either completed or required additional time for validation, particularly concerning a syslog filter change that needed further testing before final approval. A significant portion of the discussion centered on breaking changes and specific technical hurdles encountered in Google Cloud Platform (GCP) environments. The group addressed a major update to the GCP Terraform provider version 8.4.0, which shifted application load balancers from external to managed types, causing downtime for bubble environments until proactive fixes were applied. Additionally, the team tackled issues related to MariaDB connectors and self-signed certificates, specifically within the Trilium environment where VPC peering configurations caused validation errors. Solutions involved enabling specific flags to skip certificate validation for certain services while maintaining security protocols for others, such as disabling hostname verification for private IP connections in Credit Hub. The meeting also covered new test suites and infrastructure stability issues, including a failing job related to fresh cuts that crashed due to volume mounting errors on the Diego release. This issue was flagged as a potential backward compatibility problem that required investigation from the SAP Diego team, with a promise of a fix by the following day. Furthermore, a critical update regarding the CredHub test app was resolved when a colleague bumped the Go version to 1.26 to avoid support expiration, only to find the pipeline failing due to missing dependency updates. Running `go mod tidy` successfully corrected this dependency issue, allowing the pull request to pass and unblocking the deployment process. In conclusion, the working group successfully identified and resolved most outstanding items from their backlog, ensuring that pipelines remained stable despite external provider changes. The team agreed to merge the remaining approved pull requests once final reviews were complete, with documentation provided for rare but necessary manual interventions in GCP configurations. With all immediate technical blockers addressed and a clear path forward for ongoing maintenance, the meeting concluded on a positive note. The participants scheduled their next gathering for the 24th, leaving the specific agenda for that session open for future discussion while wishing each other a pleasant evening.
Read the full video transcript
It's fun. Nervous huh? It's exactly 5:30, so maybe we'll wait a little bit. Um >> Hi guys. >> Hello. >> I am. >> Take it deal. Yes, so my So, you love and show and you you do like to him. >> Hi. >> Hello. >> Hi Jan. Sorry, I was going to send a meeting link, but that glad it worked for you. >> No, it's yeah, in the end I just like I followed the link in the Google Doc. That seems to be okay. The one in the community repo doesn't seem to be the right one. >> Yeah, okay. Maybe you look at it and fix it. In the meanwhile, I will do some housekeeping stuff. Yeah, and I cannot pronounce I don't have this German keyboard thingy, so >> No problem at all. >> [laughter] >> I would I would then use an OE as a combination, which is >> I think so. I could find a way. >> Like my mail is written and stuff like that, so it's OE. >> At least yes, it should work now. >> Oh, all that. >> Yeah, yeah, yeah. Cool. Um, all right. I think so we could start. Um, I just checked Dave is still He he's online, so maybe he'll join. Um, we could start. Um, Um, maybe it will be just look at the the quick ones. Ah, here's Dave. Um, Hi Dave. >> Hi Dave. >> Good evening, everyone. >> Hello. Good evening. Good morning for you. >> Yeah. >> So yeah, we were just almost um, starting. Um, just to let you know your uh Johan is on vacation, so I will just substitute for uh substitute him for now for this one. So I think so we have clear agenda for the last I would say the last two weeks the last time we met and uh I don't think so we have any leftovers from there except maybe some PRs. Maybe let's go to there. Just checked it. I think so this was merged. The only one is missing and um yeah. Um this one syslog filter. I think so you can already provided it a very in-depth um review and a feedback and yeah, so we are just waiting. I think so it was all um yeah, adjusted. So, maybe we just give him uh some time more. I also need to go through it so just deploy it and see how it works. Last time before this one I checked it almost takes like yeah, 8 minutes, but I mean this also includes the one um because in total um originally without this change there are two tests. So, his only test uh Yoris test are five. So, I think so that doesn't bring any major difference. So, yeah, you look at it. So, 8 minutes is quite I would say a normal one, but I think so they are running it against locally some kind of uh this CPhone kinds not on a yeah. So, let's see they're a little bit on that one. But they're still open. And the other one was this I think so was also merged. There was some prerequisites like the CLI and the copy everything. I think so it's already merged and I think so we have a backlog item on our side to enable them on her money so it can be then tested and validated. Um what else we have from the previous one? I don't think so we have anything. Yeah, this was a small add-on. Uh that was nothing special. Uh this was also solved thanks to the copy team Philip and yeah, Stefan uh Johannes also. Um this was >> Mhm. and the issue with this uh J connector or the the MariaDB connector. So, it was affecting that one. And uh if you have a self-signed certificate that uh you know you're missing, that caused a problem. And it happened on one of our Trillium environment because we are using that one um or for for for this with external database with a self-signed certificate. Um yeah. And the last one, I think so this is also clear the issue with the UAA. So, I think so all are well uh taken care of except the one that little bit needs more review. And maybe then we start with the current one. So, the quick wins, I think so they are small just PRs that are opened and need review. And in a sense, I already applied it to the pipeline. So, this is related to opening a connection. I mean, accessing the CredHub. So, we need to make it in um in three attempts. So, retry. So, at least it does not fail because the whole job is well deployed. Fine. Everything uh is fine except when it's trying to access CredHub, it fails. So, that's why we try at this one. So, maybe a small um as a quick wins if uh get approved, we can merge this one. The similar is also for the cards one. Um so, if you look at for example the ARD pipeline, at least the cards I think so the plus here. So, you can see it exactly here. So, the whole job is fine. And then we have a problem with accessing it. So, this make sure that now retries is um is is is is making it successful. Mhm. That was the quick wins. Um the other one maybe I should have put it in other way. Maybe I talk about this one. So, I bring it up. Um So, there was a breaking change uh in the GCP Terraform provider and I mean the first and the most important part was the ALB, so the classical one. So with 8.4.0 Terraform provider, they are I must find from type external to external managed. So I mean new application load balancer. So the first time was observed with the with Stefan ARI working group. We are all the bubble environment were down. So you honest just contacted us that this happened and we just paused the pipelines for all the bubble environments or the jobs for all the bubble environments and yeah, make sure that we got a new release and this not happen because surely it will happen. So yeah, taking some proactive action against that and then taking action against one by one with each GCP based uh yeah, environment. The only one which caused a little bit headache was the Trilony. Why? Because uh we enabled this passwords with this direct VPA VPC peering password. So Terraform does not aware of that one. So maybe I should also bring that one. It's important. Um Yes, there is a PR already open. Yeah. So um to to to uh to when when there is a bubble up, there is our issues or when you bubble down the private service access subnet, this has a dependencies and this has to be done manually. And this is well documented now. So everything information is there. You can give it a try. Uh what has to be done on the GCP side? I mean this is very rare, but still it will happen if some time we want to bubble down and bubble up again, but this is well documented the whole process. So this was the one hiccup that we have at Trilony and the second one was that the MySQL 4.8.4 we are using it. The caching share to password is in used. So, before we don't have this issue, why? Because the default one is I think so this is well documented. Um So, if you are already using it, you don't you can't you don't have a problem. You can use the MySQL native password. But, if you are deploying a new one, you can't use this one with the flag. So, it's no more available. What this mean was that um the password exchange between our encrypted channel it should be done now because they don't support now. Or, the other option was to manually log in into the database and change that type for the user which is not suitable for long term. But, to make it short, uh we um come up with a solution and that was to enable this one also yeah um expose them in a way that we can deploy it. I mean, the best solution would be is to also um had a collaboration with the UE colleagues how it can be done in a in a better way, but this is the one that we did we did. Maybe let me I should open the PRs. At least that will give a picture clear. Uh so, you can see here. So, an ops file that does this and and I mean these are the only one are affected who are they using the MariaDB connector. So, UE is also the one and the other one was the um the the credit hub. So, we have enabled these flags. So, you can see enabled skip all validation for the UE and for the credit hub we introduce or got that certificate for the database injected here and then also disabled the host name verification because this was also a little bit problematic. The UE worked with this enable skilled uh skip all validation uh is because the TLS is on, the certificates are ignored means the connection should be established no matter what is happening with the cert because if it's trusted or not. But, the credit hub was a little bit problematic. Why? Because we need to validate the CSR for the database that the server identity is verified, but we had to disable the host name verification because it uses the private IP. So that's why the certificate has a SAN the SAN field only for the deal host name of the instance. So that was causing a problem, but nevertheless we implemented validated locally and then after the fix was done, we also enable it on the pipeline. So these are the PRs which are left to enable them. So you can see that. And if I check the Triloni pipeline, so you can see here. Uh this one was this before and you can see here. Yeah, there was no job and if you look at here, we see here that we are grabbing the SQL start injecting it and let them use because the credit needs this. And pipelines are green. Um all of them are merged. Johan did a review yesterday before he was going on vacation. The only one which is left is I think so this one. The documentation had the one, but I will keep as for anybody if interested, they can just review it and then we can merge it. Um any question for the this this this this breaking changes from GCP side? If not, then I will move to the next one, the brand new items. I mean, it's almost why I say new is this two days ago there was a new test uh suite that for the contest for the async receive recursive delete behavior is implemented and this also need a kind of a review. I think so. We implemented, yes. So yeah, this is also something we will have to check and I already requested from the approval reviewers. So, yeah, you're welcome. It will be good. Um The cuts uh again um So, I created ident this issue 2 hours ago. Why I created it is because of this failing job. The fresh cuts are failing. Um they are failing for some other specific reason also, but if you scroll down the app is crashing because there is a failed to mount volume errors. So, there is option of force like the work grids, the work type, but in the end of that is the app cannot start. So, I open an issue on on the Diego release because it's it's it's related to Diego. I provided the logs and the symptoms and what has to be done. And I also pinged the the the the colleagues from the Diego at SAP side to look into it and they promised that by tomorrow maybe they in a daily they will just have a look and uh yeah, let's see. And we'll propose a fix if it should be done from the NFS volumes release side or it should be done from there because they promised that it is a backward compatible. If you look at the PRs, I think so they mentioned somewhere. Uh backward compatibility is not a not breaking change, but it happened that it's a breaking one. So, yeah. Uh that's for the cuts and the last one is the same one. I think so it was just a part two missing and it the truant is um green, looking fine. There are no issues with that. I think so. I hope I covered any everything. Is there anything from your side, Jan, Dave, Stefan, anything, Milena? >> I have a small PR to address that I could use a review on. Um a colleague bumped the CredHub test app um Go version because 1.25 is going out of support. So, he bumped it to 1.26, but then that is failing in the drafts pipeline. >> Yeah. >> Because it needs Go mod tidy running. So, I just ran that and opened a PR. >> Yes, this one is failing. Okay, then. This one, you mean? >> So, that This is the PR to fix I hope that will fix it. I just ran Go mod tidy and it did did this. I'm like, "Okay, if that's what you want it to do, I'll I'll check that in." >> Okay, yeah, okay. >> [clears throat] >> So, the PR that broke it I merged and then it went through the drafts pipeline and uh yeah, caused a problem. >> Okay, okay. I definitely think >> So, this is the drafts in >> This is the CF deployment and and this one is the drafts. >> This is the one I was looking at. Yeah. >> Okay, okay. Um, so it's basically yeah, right >> I was looking at. Uh, if you scroll up a little bit, keep going up. Uh, yeah, I think you keep going up. Right, that's the top of the screen. Right here. It just basically says, "Okay, Go mod." Yeah, and it needs Go mod tidy running. Now, I hope that the PR I I opened will address that and it should go green at that point. Um >> Yeah, I just want to run the test. I hope I have a permission to do it. >> Let's see what happens. >> Yeah. >> [clears throat] >> I'll give it a shot, then. >> Yep, there we go. It's happy. >> Yeah, definitely we don't have that branch yet. It's the exact one. I mean, we already have jobs, but they're no. No, no, no. Do we have No, okay. Cool. Uh Then I will just merge this one and unblock this. Oh, yeah. >> I'll keep an eye on that. Thank you. >> Yes. Yeah. I will release. Yeah, okay. I need to release the pool. >> Yes. >> Otherwise it will complain. All right. Oops. >> Perfect. Thank you. >> You're welcome. All right. Then I think so. We are done for today, I would say. If there's nothing and then I wish you a very, very beautiful, nice rest of the evening and also morning day for for your side, a nice day. Then we see each other again on 24th. Something is there already on agenda, but I will leave it as as a suspense for next time on 24th >> [laughter] >> That's good. >> All right. See you then. Have a nice evening. >> Have a good evening. See you then. Bye-bye. >> Bye. >> Bye-bye. >> Bye.