Video summary
The September 10, 2026 meeting of the App Runtime Deployments Working Group focused on reviewing recent pull requests and addressing operational issues within the deployment pipelines. The session began with housekeeping matters, including a substitution for Johan who was on vacation, and a discussion regarding technical difficulties with accessing meeting links due to keyboard layout differences. The primary agenda involved clearing backlogs from previous meetings, where most items had been resolved or merged. The team confirmed that several pending reviews were either completed or required additional time for validation, particularly concerning a syslog filter change that needed further testing before final approval.
A significant portion of the discussion centered on breaking changes and specific technical hurdles encountered in Google Cloud Platform (GCP) environments. The group addressed a major update to the GCP Terraform provider version 8.4.0, which shifted application load balancers from external to managed types, causing downtime for bubble environments until proactive fixes were applied. Additionally, the team tackled issues related to MariaDB connectors and self-signed certificates, specifically within the Trilium environment where VPC peering configurations caused validation errors. Solutions involved enabling specific flags to skip certificate validation for certain services while maintaining security protocols for others, such as disabling hostname verification for private IP connections in Credit Hub.
The meeting also covered new test suites and infrastructure stability issues, including a failing job related to fresh cuts that crashed due to volume mounting errors on the Diego release. This issue was flagged as a potential backward compatibility problem that required investigation from the SAP Diego team, with a promise of a fix by the following day. Furthermore, a critical update regarding the CredHub test app was resolved when a colleague bumped the Go version to 1.26 to avoid support expiration, only to find the pipeline failing due to missing dependency updates. Running `go mod tidy` successfully corrected this dependency issue, allowing the pull request to pass and unblocking the deployment process.
In conclusion, the working group successfully identified and resolved most outstanding items from their backlog, ensuring that pipelines remained stable despite external provider changes. The team agreed to merge the remaining approved pull requests once final reviews were complete, with documentation provided for rare but necessary manual interventions in GCP configurations. With all immediate technical blockers addressed and a clear path forward for ongoing maintenance, the meeting concluded on a positive note. The participants scheduled their next gathering for the 24th, leaving the specific agenda for that session open for future discussion while wishing each other a pleasant evening.
Read the full video transcript
It's fun.
Nervous huh?
It's exactly 5:30, so maybe we'll wait a
little bit. Um
>> Hi guys.
>> Hello.
>> I am.
>> Take it deal. Yes, so my
So,
you love and show and you you do like to
him.
>> Hi.
>> Hello.
>> Hi Jan. Sorry, I was going to send a
meeting link, but that glad it worked
for you.
>> No, it's yeah, in the end I just like I
followed the link in the Google Doc.
That seems to be okay. The one in the
community repo doesn't seem to be the
right one.
>> Yeah, okay. Maybe you look at it and fix
it.
In the meanwhile, I will do some
housekeeping stuff.
Yeah, and I cannot pronounce I don't
have this German keyboard thingy, so
>> No problem at all.
>> [laughter]
>> I would I would then use an OE as a
combination, which is
>> I think so. I could find a way.
>> Like my mail is written and stuff like
that, so it's OE.
>> At least yes, it should work now.
>> Oh, all that.
>> Yeah, yeah, yeah.
Cool. Um, all right. I think so we could
start. Um, I just checked Dave is still
He he's online, so maybe he'll join. Um,
we could start. Um,
Um, maybe it will be just look at the
the quick ones. Ah, here's Dave. Um,
Hi Dave.
>> Hi Dave.
>> Good evening, everyone.
>> Hello.
Good evening.
Good morning for you.
>> Yeah.
>> So yeah, we were just almost um,
starting. Um, just to let you know your
uh Johan is on vacation, so I will just
substitute for uh substitute him for now
for this one.
So I think so we have clear agenda for
the last I would say the last two weeks
the last time we met and uh
I don't think so we have any leftovers
from there except maybe some PRs. Maybe
let's go to there.
Just checked it. I think so this was
merged.
The only one is missing and um yeah.
Um this one syslog filter. I think so
you can already provided it a very
in-depth um review and a feedback and
yeah, so we are just waiting. I think so
it was all
um yeah, adjusted. So, maybe we just
give him uh some time more. I also need
to go through it so just deploy it and
see how it works.
Last time before this one I checked it
almost takes like yeah, 8 minutes, but I
mean this also includes the one um
because in total um originally without
this change there are two tests. So, his
only test uh Yoris test are five. So,
I think so that doesn't bring any major
difference. So, yeah, you look at it.
So, 8 minutes is quite I would say
a normal one, but I think so they are
running it against locally some kind of
uh this CPhone kinds not on a yeah.
So,
let's see they're a little bit on that
one.
But they're still open.
And the other one was this I think so
was also merged. There was some
prerequisites like the CLI and the copy
everything. I think so it's already
merged and I think so we have a backlog
item on our side
to enable them on her money so it can be
then tested and validated.
Um what else we have from the previous
one? I don't think so we have anything.
Yeah, this was a small add-on. Uh
that was nothing special.
Uh this was also solved thanks to the
copy team Philip and yeah, Stefan uh
Johannes also.
Um this was
>> Mhm. and the issue with this uh
J connector or the the MariaDB
connector. So, it was affecting that
one. And uh if you have a self-signed
certificate that uh you know you're
missing, that caused a problem.
And it happened on one of our Trillium
environment because we are using that
one um or for for
for this with external database with a
self-signed certificate. Um yeah.
And the last one, I think so this is
also clear the issue with the UAA.
So, I think so all are
well uh taken care of except the one
that little bit needs more review. And
maybe then we start with the current
one. So, the quick wins, I think so they
are small just PRs
that are opened and need review. And in
a sense, I already applied it to the
pipeline. So, this is related to opening
a connection. I mean, accessing the
CredHub. So, we need to make it in um in
three attempts. So, retry. So, at least
it does not fail because the whole job
is well deployed. Fine.
Everything uh is fine except when it's
trying to access CredHub, it fails. So,
that's why we try at this one. So, maybe
a small um as a quick wins if uh
get approved, we can merge this one. The
similar is also for the cards one.
Um so, if you look at for example
the ARD pipeline, at least
the cards I think so
the plus here. So, you can see it
exactly here. So, the whole job is fine.
And then we have a problem
with accessing it. So, this make sure
that now retries is um is is is is
making it successful.
Mhm.
That was the quick wins. Um
the other one maybe I should have put it
in other way. Maybe I talk about this
one. So, I bring it up. Um
So, there was a breaking change uh in
the GCP Terraform provider and I mean
the first and the most important part
was the ALB, so the classical one. So
with 8.4.0
Terraform provider, they are
I must find
from type external to external managed.
So I mean new application load balancer.
So the first time was observed with the
with Stefan
ARI working group. We are all the bubble
environment were down. So you honest
just
contacted us that this happened and we
just paused the pipelines for all the
bubble environments or the jobs
for all the bubble environments and
yeah, make sure that we got a new
release and this not happen because
surely it will happen. So yeah, taking
some proactive action against that
and then taking action against one by
one with each GCP based
uh yeah, environment. The only one which
caused a little bit headache was the
Trilony. Why? Because
uh we enabled this passwords with this
direct VPA VPC peering password. So
Terraform does not aware of that one. So
maybe I should also bring that one. It's
important.
Um
Yes, there is a PR already open.
Yeah.
So um
to to to uh to when when there is a
bubble up, there is our issues or when
you bubble down the private service
access subnet, this has a dependencies
and this has to be done manually.
And this is well documented now. So
everything information is there. You can
give it a try.
Uh what has to be done on the GCP side?
I mean this is very rare, but still it
will happen if some time we want to
bubble down and bubble up again, but
this is well documented the whole
process.
So this was the one hiccup that we have
at Trilony and the second one was that
the MySQL 4.8.4 we are using it. The
caching share to password is in used.
So, before we don't have this issue,
why? Because the default one is I think
so this is well documented.
Um
So, if you are already using it, you
don't you can't you don't have a
problem. You can use the MySQL native
password. But, if you are deploying a
new one, you can't use this one with the
flag. So, it's no more available.
What this mean was that um
the password exchange between our
encrypted channel it should be done now
because they don't support now. Or, the
other option was to manually log in into
the database and change that type for
the user which is not suitable for long
term. But, to make it short,
uh we
um
come up with a solution and that was to
enable this one also yeah
um expose them in a way that we can
deploy it. I mean, the best solution
would be is to also
um had a collaboration with the UE
colleagues how it can be done in a in a
better way, but this is the one that we
did we did.
Maybe let me I should open the PRs.
At least that will give a picture clear.
Uh so, you can see here. So, an ops file
that does this and and I mean these are
the only one are affected who are they
using the MariaDB connector. So, UE is
also the one and the other one was
the um the the credit hub.
So, we have enabled these flags. So, you
can see enabled skip all validation for
the UE and for the credit hub we
introduce or got that certificate for
the database injected here and then also
disabled the host name verification
because this was also a little bit
problematic. The UE worked with this
enable skilled uh skip all validation uh
is because the TLS is on, the
certificates are ignored means the
connection should be established no
matter what is happening with the cert
because if it's trusted or not. But, the
credit hub was a little bit problematic.
Why? Because we need to validate the CSR
for the database that the server
identity is verified, but we had to
disable the host name verification
because
it uses the private IP. So that's why
the certificate has a SAN the SAN
field only for the deal host name of the
instance. So that was causing a problem,
but nevertheless
we implemented validated locally and
then after the fix was done, we also
enable it on the pipeline. So these are
the PRs which are left to enable them.
So you can see that.
And if I check the Triloni
pipeline, so you can see here.
Uh this one was this before and you can
see here. Yeah, there was no job and if
you look at here, we see here that we
are grabbing the SQL start injecting it
and let them use because the credit
needs this.
And pipelines are green.
Um all of them are merged. Johan did a
review yesterday before he was going on
vacation. The only one which is left is
I think so this one.
The documentation had the one, but I
will keep as for anybody if interested,
they can just review it and then we can
merge it.
Um any question for the this this this
this breaking changes from GCP side?
If not, then I will move to the next
one, the brand new items. I mean, it's
almost why I say new is this two days
ago there was a new test
uh suite that for the contest for the
async receive
recursive delete behavior is implemented
and this also need a kind of a review.
I think so.
We implemented, yes.
So yeah, this is also something we will
have to check and I already requested
from the approval reviewers. So, yeah,
you're welcome. It will be good.
Um
The cuts uh again um
So, I created ident this issue 2 hours
ago. Why I created it is because of this
failing job. The fresh cuts are failing.
Um they are failing for some other
specific reason also, but if you scroll
down the app is crashing because there
is a
failed to mount volume errors.
So, there is option of force like the
work grids, the work type, but in the
end of that is the app cannot start. So,
I open an issue on
on the Diego release because it's it's
it's related to Diego. I provided the
logs and the symptoms and what has to be
done.
And I also pinged the the the the
colleagues from the Diego at SAP side to
look into it and they promised that by
tomorrow maybe they in a daily they will
just have a look and uh yeah, let's see.
And
we'll propose a fix if it should be done
from the NFS volumes release side or it
should be done from there because they
promised that it is a backward
compatible. If you look at the PRs, I
think so they mentioned somewhere.
Uh backward compatibility is not a not
breaking change, but it happened that
it's a breaking one. So, yeah.
Uh
that's for the cuts and the last one is
the same one. I think so it was just a
part two missing and it the truant is um
green, looking fine. There are no issues
with that.
I think so. I hope I covered any
everything. Is there anything from your
side, Jan, Dave, Stefan, anything,
Milena?
>> I have a small PR to address that I
could use a review on. Um a colleague
bumped the CredHub test app
um
Go version because 1.25 is going out of
support. So, he bumped it to 1.26, but
then that is failing in the drafts
pipeline.
>> Yeah.
>> Because it needs Go mod tidy running.
So, I just ran that and opened a PR.
>> Yes, this one is failing. Okay, then.
This one, you mean?
>> So, that This is the PR to fix I hope
that will fix it. I just ran Go mod tidy
and it did did this. I'm like, "Okay, if
that's what you want it to do, I'll I'll
check that in."
>> Okay, yeah, okay.
>> [clears throat]
>> So, the PR that broke it I merged and
then it went through the drafts pipeline
and
uh
yeah, caused a problem.
>> Okay, okay. I definitely think
>> So, this is the drafts in
>> This is the CF deployment and and this
one is the drafts.
>> This is the one I was looking at. Yeah.
>> Okay, okay.
Um,
so it's basically yeah, right
>> I was looking at.
Uh, if you scroll up a little bit, keep
going up. Uh, yeah, I think you keep
going up.
Right, that's the top of the screen.
Right here. It just basically says,
"Okay,
Go mod." Yeah, and it needs Go mod tidy
running. Now,
I hope that the PR I I opened will
address that and it should go green at
that point.
Um
>> Yeah, I just want to
run the test.
I hope I have a permission to do it.
>> Let's see what happens.
>> Yeah.
>> [clears throat]
>> I'll give it a shot, then.
>> Yep, there we go. It's happy.
>> Yeah,
definitely we don't have that branch
yet. It's the exact one. I mean, we
already have jobs, but they're no.
No, no, no.
Do we have No, okay. Cool. Uh Then I
will just merge this one and unblock
this.
Oh, yeah.
>> I'll keep an eye on that. Thank you.
>> Yes. Yeah.
I will release.
Yeah, okay. I need to release the pool.
>> Yes.
>> Otherwise it will complain. All right.
Oops.
>> Perfect. Thank you.
>> You're welcome.
All right. Then I think so.
We are done for today, I would say. If
there's nothing and then
I wish you a very, very beautiful, nice
rest of the evening and also morning day
for for your side, a nice day. Then we
see each other again on 24th. Something
is there already on agenda, but I will
leave it
as as a suspense for next time on 24th
>> [laughter]
>> That's good.
>> All right. See you then. Have a nice
evening.
>> Have a good evening. See you then.
Bye-bye.
>> Bye.
>> Bye-bye.
>> Bye.