Video summary
The Jenkins Infra team meeting held on August 4th, 2026, began with announcements regarding the upcoming release schedule and team availability. The weekly release version 2.576 was scheduled to proceed after a security advisory concerning Jenkins Core is issued later that day, while an LTS backport containing critical fixes would also be released simultaneously. Team member Damian noted his absence from August 10th to 13rd due to vacation, and the group discussed meeting logistics for the rest of the month, ensuring at least two members are present for each session. Additionally, the team addressed expiring credentials, with plans to renew tokens for Artifactory, NPM, and other services by late August to maintain secure access without interruption.
A significant portion of the discussion focused on cloud budget management across Azure, DigitalOcean, and AWS accounts. The team reported positive financial performance in July and early August, noting that spending remained well below forecasts for most platforms; however, they identified an unexpected increase in consumption within their sponsored subscription that required investigation to locate specific resource groups or DNS requests driving the costs. Conversely, the AWS account showed a successful reduction in spend compared to previous months. The meeting also highlighted a critical deadline: without securing new credits from Amazon by December 31st, the team would be forced to migrate their Continuous Integration (CI) pipelines for Jenkins Core from AWS to Azure, necessitating an immediate restart of the credit request process which Mark had handled successfully in prior years.
The agenda then shifted to technical milestones and work-in-progress items, covering infrastructure maintenance and security improvements. The team completed several tasks including standardizing Docker update pipelines on single property calls, renewing various API tokens for Algolia and plugins.jenkins.io, and resolving permission issues that previously blocked the deployment of testing bot plugins. A notable discussion arose regarding a potential security vulnerability where removing an old API token from certain cloud providers like DigitalOcean did not immediately revoke it, prompting plans to test this behavior across different platforms before renewing credentials again in three months. Furthermore, progress was made on migrating public AKS clusters to the Azure 2026 subscription by establishing cross-region service endpoints for storage and Docker registries between Sweden and US East regions, though challenges remain regarding private image caching which will be addressed once a solution is found.
Finally, the meeting concluded with updates on software lifecycle management and upcoming deprecation efforts. The team decided against forcing immediate JDK patch upgrades based on Oracle's critical security advisories after reviewing that no catastrophic issues were present in the current versions. They also discussed dropping support for Windows Server 2019 agents to align with newer virtual machine standards, as well as planning for the end-of-life transition from Debian Bookworm (Debian 11) to Trixie later in the year. With most immediate tasks assigned and minor issues moved back into the backlog or removed from active milestones due to their lower priority relative to upcoming security advisories, the team adjourned with plans to reconvene next week after the release of version 2.576.
Read the full video transcript
We are recording.
>> Hello guys. So uh it's the 4th of August
2026. Uh in the infra team meeting we
have myself Jay Franco. We have Damian
and we have Mark Wait.
So moving on to the announcements. So
last week's uh weekly release was 2.575.
Uh there was a minor issue on the docker
image publication but it was fixed soon
after
and this week uh it's 2.576
which will proceed tomorrow after the
security advisory that goes on.
So what are the what other announcements
do we have? So team capacity Damian
would be off uh from
from August 10 to 13th. So he would be
back on uh Friday,
the next Friday.
And for our priorities, so we still have
uh migration to puppet, I mean migration
from puppet to anible on the infra road
map. And um in the current epic, we
currently are trying to bring down our
AWS course so we can um extend the
credits at least until Jan
>> if I just may just a quick check since
we are in the announcement and tip cap
capacity. Y
>> regarding the upcoming team meetings uh
we are the fourth next week will be 11.
I won't be there. V is okay to run it.
But Jay, you will be his backup if he
has any issue. Is that okay for
>> Yeah,
>> the next one 18th of August, both and I
will be there.
>> Uh it's not 18, it's 19. Sorry. So I
will most probably run the n the 18 and
then I will go in holidays. But we
should have the two upcoming ones and
the next one 25 both you and I will be
there J. So we always have at least two
person for running the the meetings in
August.
>> Okay.
>> Yeah, sounds good.
>> Okay. So,
uh moving on to the upcoming calendar.
The next infra meeting would be next
Tuesday, which is the 11th of August. Uh
so it would most probably just be me and
Airway.
uh the next weekly would be next week
which is 2.57
again uh next Tuesday
and the next it would be released
tomorrow after the security advisory
so uh I'll go ahead and
>> just just one point uh tomorrow it's not
after it's part of the advisory the
advisory is coped to genkins score which
mean we will have both a weekly release
2.5 uh 57 76
>> and LTS with the same fix. It's back
port to the LTS line. That's really
important because that mean we will have
many things uh to run and we will have
to update all of our controllers.
>> Okay.
>> Okay, Jay. Um since it will most
probably happen later time, I most
probably will take care of this with
eventually the help of if it's
available.
>> Cool.
>> Okay. Yeah.
Uh so you'll go ahead and send the um
email on the Google discussion
on the Google group.
Now
I I did receive a question yesterday
from from the release lead from Shalini
Sudaran and her question was hey is this
the right message to post to the infra
team about the packaging and release
repositories and their updates. So, I
believe she posted that and I assume
that we're okay with it that we we're
settled on how much or how little we're
going to do to those those dependent
repositories
>> regarding the the back ports that Yeah,
exactly.
>> Yeah, her message was the right one.
It's me who missed it during the early
days and they already fixed his comment
because he was asking to have it
earlier. But uh in fact that was the
second time that Shellini posted it.
>> Okay, perfect.
>> So just just to confirm she did it on
the right time with the right content in
the right location and she did very well
to post it again a second time because
no answer on the first time. That was
really the best thing and she did the
due diligence.
>> Great. Thanks. Okay. And and Daniel and
Kevin, whoever's working the security
advisory, they're okay with with what
we're doing in our backports. Okay.
>> Yep. Um, that would have been better if
I would have done the backboard thing
last week because uh that that that made
Danielle a bit late today, [snorts] but
that's that's on me.
>> Okay. So, moving on. So, we have some
credentials that are expiring in the
next three weeks. Um, so the most recent
one is the contributor stats and plugins
as your fileshare service principle
credentials. So I'm going to be taking
care of that. It should be done by
tomorrow. Uh, I was held up with
something.
Then the NTM token has been renewed.
Oh no, sorry that was the net 5. My bad.
So I'll be taking care of creating the
issue for the npm token and also taking
care of uh the token renewal
and on 22nd August uh the artifactory
admin token for rpu expires. So
I uh I think I think Damian's going to
take care of creating the issue as well
as total renewal
and we don't have any major event coming
up in the next 3 weeks.
So we can move on to the cloud budgets.
So for Azure our uh August spend up
until now has just been $213.
So July uh good news. So we did what you
underspend and stay within our threshold
even lower than the forecasted at 2.2.
So it was uh 2182.
So we'll meet our threshold. It looks
like we'll meet our threshold today as
uh this month as well.
And uh something about this note Damian
do you want to elaborate?
Hello.
>> Sorry. Yep. Um, no.
>> No, no, it's it's it's still on hold
because uh um it's a heavy one and I
need a to be around for doing this. Uh
either you and or me and you or but yeah
was missing time. So we delayed no
emergency but the goal is to decrease
even the consumption.
>> All right.
uh for the Azure sponsored subscription.
So we have around 80 credits 80k credits
left uh till
28th of October 2027.
So uh July's spend was 5.5K a little
higher than the forecasted 5.1K.
So uh and August looks like it's going
to be even higher than what it was for
the month of July.
Uh
yeah, so putting it at $793
as of now from uh the 1st of August.
>> So for the digital ocean credits,
>> if I may just before going back to as
your sponsor, were you able to look
where the increase in consumption
happened or didn't didn't you have?
>> No, I was able to uh look where the
consumption happened.
>> Okay. So that will be a good exercise
for you. Um part of the reporting for
the issue is also if possible identify
quickly where the money went. That
doesn't mean it's a call for action on
on resour cloud resources but at least
we understand that can be DNS request
that can be a specific resource group
that consume a bit more but we we need
to locate otherwise it doesn't mean
anything for us to track costs. We need
to act. So in order to act first let's
see what the culprit is that's a quick
one and then we can discuss during the
team meeting okay so we won't do it now
and I let you continue but when we see
an increase that is not expected from
last week that's better to check okay
>> room for improvement for the world team
>> got it uh for the digital ocean credits
so we have 6.4k 4K until the 2nd of Jan
2027.
So July spend was uh around $500. So it
was even lesser than the forecasted
forecasted amount which is a good sign.
And for August, it looks like we might
uh be even we might reduce the spend
even more per capit at 460, but we'll
know by the end of the month.
For the AWS sponsored account, we have
around 30k credits until uh the 31st of
Jan 2028.
31st of Jan 2028 as of 1st of July. So
the July spend was lesser than u the
June spend which was 5.8K.
So it looks like we are heading in the
right direction in terms of our epic. We
are bringing cost down for the AWS
sponsor account. And uh for August the
spend up until now was around $487
forecasted at pi.4k
which is a really good sign. So kudos to
the team
and for the JROG artifact usage. Uh I'm
still not uh I'm still not too well
versed on how to look at where the
consumption is more on the dashboard,
but it looks like our for it looks like
our August consumption has been pretty
high at least for the last 3 days. So
it's been at 6.77 dB and the forecasted
rate is really high. It's forecasted at
50.7
if we keep going at the current rate. Is
there anything
Damian? Are you aware of anything that's
bringing the consumption up? No, I need
to I will fill check the July cons
consumption because that wasn't on the
dashboard and I forgot to show you where
it was. I will take the opportunity to
see what increased exactly on at least
what we can I'm not sure there is
anything we can do if someone starting
again to download thousands of data
since Grog doesn't want to allow us to
use API rate limits or anything else we
cannot block IPs there is no thing that
we can do except reporting to them that
maybe we will consume more
>> and then that's their problem but I have
to say I'm not sure they will do
anything because uh uh they told me in
the past months 14 time that we are not
entitled to support and we should
contact the license manager to get a pro
subscription
so I guess yeah automate not enough
people at their support that's already a
first barrier so yep looks like they're
in bad shape so let's not put too much
pressure on them just letting them know
no action for us though
>> okay so uh the allegoria API usage for
July it's been around 86.4K
which is like lesser than our uh max
rate limit 400k. So it looks pretty
good.
Okay. So if uh nothing to add there I'll
move on to the issues that we've uh
completed in the current milestone.
So
sorry just one point uh AWS so we are
out of the I emergency
>> but we will only be able to run CI
genkin on on Amazon until December
which mean if we want to be effective we
will need to move CI on 1 December in
Azure unless we get credits from Amazon.
>> Okay. So that's really important that we
start again the process to request
credits. I believe Mark you did it last
year, the previous years, right?
>> I think so, but I haven't seen Have you
seen an announcement from Amazon that
they're again they're again open for
proposals?
>> No, but I um since you took care of it
in the previous year, I wasn't really
paying attention except waiting for you,
waiting for feedbacks or validation. So
maybe I should start this so you are not
the bus factor and we don't burden you
with that part. But I might have
question on where where were you looking
at or what were you watching to get the
news because maybe it's an email that
you are receiving or a page you were
checking and I'm not sure.
>> I and I'll I'll keep looking. We could
also AWS certainly has their open-source
team and we could if we need to we can
ask them the question hey is there is is
there an opportunity
or yeah will they will they consider
funding an open source project for
another year.
>> Cool. But yeah that's really important
call to action because otherwise we will
have to work on moving back seno and as
your sponsored subscription and we will
need to fix problems uh billing problems
here. Then
yes, we can move on. Thanks.
>> Move on to uh yeah the issues that were
completed in the current milestone. So
under keep infra and maintainable.
Thanks Mark for the uh Linux for the
Ubuntu upgrades on the S390X machine. So
uh I saw that it was completed. Unless
you have anything to add on over there,
I move on.
Okay. Uh so standardize docker updatly
pipelines on single property call. So
this was an issue that was completed by
me. So I noticed that the internal uh
properties block of the update pipeline
library was overriding any of the pi any
of the pipeline steps the properties
block that was in the individual job in
our repositories. So I went ahead and
fixed that. So it uses the internal
updatly uh properties block and it works
very well now. It fires on a daily basis
and yeah it retains the properties. So
this was a follow up on from the next
issue which is the daily chron wasn't
firing on a bunch of repos. So again
because of the internal properties block
which was fixed and the JROG artifactory
u so I can confirm now that I and the
entire infra team now has access to
JROG [clears throat]
and can view the dashboard for the cost
spending and analysis.
So thanks Damian for uh giving me for
making sure that I had my access.
Okay, moving on to keep infradate. So
the netifi token was renewed. Um that I
went ahead and took care of it. The
Azure credential by packer images build
expires on um 26th July. So that was
taken care of by Damian. So thanks for
that. And the Algoria right API token
for plugins.jenkins.io
IO uh that was going to expire on the
29th of July has also been renewed. So
thanks Damian again for that.
Um
>> just just a point on this um we noticed
a word behavior
where the old token at least on Algolia
uh because on Azure I think I did a
mistake and I didn't took care of
testing carefully but on Alolia it looks
like that removing an API token from the
UI doesn't revoke the token
>> and so then I did another mistake that
is quick to to verify. I didn't push on
the right uh branch on our secret
system. So the new secret was never used
by Infrasci.
In the case of Algolia, I removed the
token and launched the build. So using
the old token should have failed
immediately,
but the token was still valid, which I
don't understand. So I need to to retry
this at a moment in time before next
renewal because that mean if someone get
their hands on the token, we cannot
revoke it immediately. And second,
what's the point of renewing token every
3 months if a given token is never
deleted until it's it's removed. So we
have to be careful. Jay, looks like you
had the same issue on digital.
>> Yeah, I had the same issue while
renewing the digital ocean PAT.
>> Yep.
>> So yeah, it does like what do you say?
It does come up as kind of a security
risk. So we'll look into that. uh I kind
I'll test it out before I'll test it out
while renewing our uh
>> Azure service principle credentials so
to see if the I'm able to replicate the
behavior
>> you you won't be an Azure I'm sure not
for Azure tokens the problem is on other
cloud systems
>> oh I see
>> in any case it's worth doing the
exercise also for you for your for your
own process for your own internal mental
process. So in any case do it like this.
>> But yeah uh for the others uh there are
weird things and we need to test cloud
provider by cloud provider.
>> Mhm.
Okay. Uh moving on to some support tasks
that were completed in the current
milestone. So CD release fails with 403
forbidden deploying testing bot plugin
to report.jenkins.ci.org
or releases.
Uh it looks like maybe you've closed the
issue.
>> Yes. So um I don't really know how the
fix went, the problem was finally
solved. It looks like that between the
first uh forbidden errors that show that
there was a problem in repository
permission updater that wasn't updated
properly permission. It was creating the
users, the groups and the permission
schemes but it wasn't uh associating
them properly. So no permission were
granted to no one. However, that a few
days later the problem was fixed and
they were able to release their plug-in.
That's why I closed the issue. Looking
back at the changes that happened
between both in in that interval, there
were a few code changes in RPU that
looks like could have had the impact of
fixing the problem. But I can't be sure
because the code that has been changed
wasn't directly related to permission
management.
So I'm not really sure oh it went back
to normal use. But what also happened at
the same time were a bunch of Grog
artifactory issues on their platform. So
maybe and only maybe we we had the API
call that say set up these permissions
as expected but the result from the API
since it does a get to get a status then
a post to change and then the get to
verify the new status if the initial get
things was already wrong maybe the there
are hidden bugs in RPU for word cases it
works no need to spend so much time I've
let Marcus Winter and Tim about this
>> thanks thanks for briefing on the issue.
Uh and the next support issue that was
closed was review need to have the tool
and plugin in installed everywhere. So I
saw that it was open by Daniel and also
uh I looked at the discussion and we
agreed not to have it on have it removed
on the infraite. Damian care to
elaborate on this as well. So with
that's an old issue that had been
reopened because the tool env plug-in
was back installed on some of our
controllers. the thing happened because
we have um something in PUPET that
detect the G genkins configuration as
code and based of the in the content it
automatically detect that oh I need this
plug-in otherwise starting the
controller will fail because that gask
directive won't be known by Jenkins
and in fact when detecting some of the
tool installation we use on search CI
and trusted CI was mistakenly setting up
tool env instead of another plug-in.
Fixing that mistake and again removing
again to envure that that problem is
gone. We still have a few issues. Daniel
mentioned that we installed the world
pipeline uh meta plug-in that install a
bunch of plug-in when we only need a few
ones. So we still have cleanup to do on
that area. But tov was really
problematic one.
>> Okay.
Uh so I can move on to the work in
progress. Mhm.
>> Okay. So for the work in progress uh
under support we have enabled previews
for the plug-in modernizer stats repo
PRs.
So um yeah I'm not sure what work
>> I can I can take it. Uh for this one uh
it's because uh in a few of these
website projects they need uh pull
request preview on a specific branches
that is not the main branch because they
will have the current version of the
website that need to be kept maintained
and they have a new branch that also
need its own specific uh review
deployment inifi not in production until
they reach the right amount of quality
and then they deploy it to production.
finally but it's a kind of develop
branch uh a develop pattern uh on git
when you have a develop branch that is
not a production branch we can do that
on netlifi and in infrasci but we need
to change slightly the configuration to
allow this
uh
I might be wrong I think for this one is
they just need to enable netify pull
request so that will be even easier
uh so these are things I'm going to take
care of. It wasn't urgent compared to
other things.
>> Uh but yeah. Uh so right now that's one
of my next upcoming test today or
tomorrow to unblock them.
>> Okay, cool.
>> Let me
>> uh yeah. So the next task um build
pipeline plug-in tests now fail with
cannot find Chrome binary. So Mark, this
was opened by you.
>> Yes. and and I've made no progress on
it. It is still correctly assigned to
me. I won't make progress on it because
I'm not panicked about it right now. I'm
still Damian's done all all the things
that that really we could do from the
the infra side. It's now a question for
the plug-in source code to decide should
it adapt itself to be more like all the
other places where we successfully run
selenium tests. And and that's that's
just a code change. We could I' I'd
propose we leave it open. I'd rather
leave it there just so that we're all
aware that it's still there. I'll get to
it when I get to it.
>> All right.
>> Agreed. Um I I propose we remove it from
the milestone so it's not an active
infra item, but the issue still exists
for auditing and for tracking. Is that
okay for you, Mark?
>> I like that. Let's remove it from the
milestone because things in the
milestone, we like to finish in that
milestone when we can. And and this one
I I it is it is not high on my list, but
I also don't want to just close it
because it's still a real issue. It it's
the plug-in needs to be adapted to do
the do its selenium test better.
>> Yep. You can move on. Uh
>> okay. So under keep infra up to date. So
uh prepare release packaging and docker
back ports for uh the LTS release
tomorrow. So uh I think Damian and Air
are going to be taking care of this.
>> Uh so yeah won't have time for taking
care of it. I did packaging and release
in emergency earlier today because we
had to unblock Danielle for the security
staging. I'm working on a Docker one
that must be done today in order to have
a proper Docker build tomorrow.
>> Um
I I was late that this should have been
done last week. I missed it. So, yep.
Room for improvement here for me.
>> Yeah, I mean everyone
everyone has it sometimes. Okay. Uh so,
thanks for taking care of this. Uh I
know it's going to be late in my day, so
I really appreciate it. Uh moving on to
the next we have a few credentials that
need B2B uh renewed for stats plugins
and contributors.jenkins.io.
So I'll be taking care of it probably
today or before uh early in the day
tomorrow.
So that should be done by me.
Do you think it will have an impact on
the advisory that start tomorrow? Uh I
think later after your lunch time.
>> No, I think I should be done with
everything related to this.
>> Okay. So So Daniel is not likely to
publish the advisory during the early
part of the European day. It'll happen
most likely towards the afternoon.
>> Agreed. The goal was just to it's also
um let's say a little game just so that
Jay uh is aware
>> uh stats and contributors have
absolutely no relationship at all with
theory. So you can go whenever you want
>> should be safe
>> but plugins genkins say will need to be
rebuilt if there are plugins that
require something related to advisory if
there are plugin with a change log. Once
the advisory is published under code,
it's only the core. But if the plugin
[clears throat]
advisory is only core, no plugins
>> in theory, but if there is a plug-in in
last minute or if a plug-in need to be
updated to say, hey, you need that new
core version in order to work as
expected. Usually the plug-in is
released but not published on plugins
genkins and that will happen after the
core release when the update center is
run again and then plugin site build the
front end. So plugins genkins io must be
working uh when you go on lunchtime
tomorrow J.
>> Yeah.
>> I I think that will be perfect but most
probably start with this one.
>> All right.
>> Good for you.
>> I'll take care of that. Yeah, perfectly.
Uh so the next one is prepare update
center root CA rotation. So Damian seems
to be the only person with the access to
do that, the power to do that. So uh
>> yep uh I've released the new
certificates uh same status as last
week. I'm waiting for a co-contributor
to do it. We have an advisory and the
co-contri contributor here is Danielle.
So most probably let's wait for the
advisory
on old until after the advisory.
Uh, and I'm waiting from uh, VDC
security officer to be back from
holidays
because I need to find a backup and we
don't need more people to have access
uh, to the the CA key. By default, I
will ask Vadic, but that most probably
means does he want and does he think
it's a good idea? depends on because we
can renew the security officer once a
year while usually the genkins board has
a two years mandate if I'm correct. So
depending on if Vadc feels like uh if he
want to go back or is okay to have
access if he if he's not because Daniel
is not Daniel declined which mean I will
have to find someone on the genkins
board that just started their mandate so
we have people that will last as much as
long as possible also VDC had ideas on
how to use um Shamir algorithm uh maybe
has find anything Shamir algorithm is
something that say if you need to
access. You need two people to validate
with the GPG keys when you need it.
I I think that could be interesting, but
I guess it means we will need to
generate update center certificate on
the go and change it once a year. And
yeah, not sure if it works or not. So
right now I'm assuming as less people as
possible but waiting for the advisory
and vic.
>> Yep.
So uh for the next issue it's been
reopened a couple of times. So
[laughter]
>> yep it was still open
>> from 22 to 20. It was still open. My
bad.
>> Yep. So you missed the in incremental
publisher the docker file.
Um, okay.
>> The update to Node GS24 was not missed.
You did it, but you did it with the
minor version that was published when
you did it, but it wasn't tracked by
update CLA.
>> Yeah.
>> So, I've um incremental publisher docker
file wasn't tracked
already on 24x.
So, you did the almost all the work. It
was just the recent changes.
>> Got it.
>> Just just a note, the issue still open
because we fail to bump a plink. GS
version. It's still failing and we
didn't have time to work on it. I guess
it's not GS. So I can ask cloud code to
fix the problem for me.
That that was a subtle joke.
>> Only the one who knows will understand.
the the the negative things hidden here.
So since we need to spend whether or not
we use an LLM, we need to spend a spend
a bit of time on this one. So that's why
it's back in the work in progress.
>> Okay.
And drop Windows 2019 support.
So, SI platform was informed
about the upcoming
infra deprication of deprecation or
depreciations. I'm always missing mixing
both words.
>> You You got it right the first time.
Deprecation. The C is hard. Depreciation
is what happens when your money
decreases.
>> Oh, okay.
>> Depreciation is a money thing.
deprecation is. And shame on the English
language for having two words that are
so close together. Sorry about that. I
apologize for my entire culture's
problem with that.
>> At least you have two words for two
concepts. In French, both concept are
using the same word. So it only depends
on the usage context. That's why I keep
making
>> Okay. And that Yes. Good. Yeah. So yeah,
it's
welcome. Yeah. Italian has several
several words for love, right? And the
English one. Thank you very much. Make
my life complicated. [laughter]
>> So controller container image already
duplicated in June if I remember
correctly.
Um issues open for agent container
images.
Container images. Um Pierre opened and
ready to merge or SSH agent
whip on the inbound agent pair PR.
Uh, next step
winpy
on CI genkin sou because right now the
CI of the WP component
uh is currently requiring to run on
Windows 2019 virtual machine agent
because the VS code uh setup here.
>> So we need to fix the VS code part on
packer image and then we can move it to
the latest agent we have available.
I think that should be easier than in
the past because uh the problem was on
Windows 2022 but now we are running on
2025 which work way better
before removing all infra 2019 stuff.
Any question on this one?
>> Nope.
>> Can move on. Okay. So I'll move on.
Yeah, by the way, I can't see uh you
guys if you just nod because I'm using a
screen,
>> right? I forget you. You don't have you
don't have two screens, right? So,
you're dealing with sharing one screen.
>> I have to rely on verbal.
>> Okay. I will be more careful at
verbalizing.
[laughter]
>> All right. Uh so, keep infra sane and
maintainable. Uh there's a
so there's an issue open for
what do you say defining a common node
with timeout and retry function for uh
website builds on the pipeline
algorithm.
>> Okay. Nice to see that you're using
claude [laughter] to experiment with
stuff. I'm wasting my time with code to
make people who like this content and
learning things on the code I have to
fix.
Uh the real value of this issue is to be
sure that the website builds especially
on CI genkins IO when their spot
instance agent is reclaimed then it's
automatically retried like we do on most
of the other builds. It's just
transplanting that retry things. Um the
new thing is the timeouts. I'm trying to
see if I can have a timeouts block in
the pipeline outside the node directive
which mean if we reach the time out
before an agent is allocated then we
retry the build because that allocation
could come from many word things in the
past. Uh I'm currently failing to have
the expected behavior. So I think I will
go with the timeout on the build things.
So if the build is stuck in the agent
that will retry with a new agent but
that won't prevent us retrying to
allocate the agent if something is doing
word in C and you can say anyway we need
the retry and the timeout in any case.
So I was just trying to polish the
thing. Uh I don't get the best behavior
I expect but I will have the
minimalistic required behavior.
>> Okay.
Uh for the next work in progress we have
migrate public AKS cluster to the Azure
2026 subscription.
>> Yeah I did see that there was a blocker
but uh I think you'll need to elaborate.
>> Yes. Um
with all the issues we had with machines
allocation in US assist especially on
publicates uh cluster uh the idea and
the recommendation from Microsoft is to
run it somewhere where we have uh enough
resources in Sweden. Most probably we
already have the ephemeral agent of CCI
running in Sweden data center. However,
we still have a few elements, especially
what we call the endpoints. Public gates
will need to be able to read from the
storage that is still on us. We cannot
move everything on one shot. We will
need to move by ports. As such uh we
need to set up what is what are called
service endpoints which are internal
manage services such as Microsoft
storage, Microsoft uh Docker container
things registry for caching uh or
Kubernetes services that we expose
internally. We need these endpoints to
be cross data centers. It we accept the
latency and the temporary cost due to
cross region transfer. However, we need
to first ensure that it work before
working on the rest.
I fixed one of the issues that was the
build reports uh from search CI uh for
the private monitoring you built last
week. We fixed that. So now we see that
at least for storage we can read write
from Sweden directly to us. That mean we
have unblocked one of the big
requirement for public gates. However,
the second one that I haven't solved is
the Docker registry because we have a
lot of private images that only lives
inside that registry. If I'm not able to
find a way to use the same technique
from Sweden to usist
insert CI because yeah, insert CI if we
fail it just automatically and
transparently falls back to docker up.
So we don't really care in the case of
publicates if it fails it fails and it's
a hard failure. We don't want the data
center to go down because we failed the
migration. As such, I need to finish the
transition of search CI controller which
still lives in usist inside directly the
new thing. It will use that endpoint or
not. But at least we will have uh we
will have solved the second requirement.
As such, I'm going to write this
requirement in the publicates issue and
move it back to backlog because it's
currently superseded by the search CI
issue.
Is it clear or or do I need to elaborate
or write on? Okay,
>> pretty clear.
So we can move on to the
AWS
issue which is activate user define or
do you have something else to add
uh for the issue below?
Uh okay AWS uh I think that one will be
closable. I need to check with survey
when it will be back. We added a lot of
tags.
>> Uh it doesn't help at all. We see
things. It's doing what we expect in the
issue scope. It doesn't help us to draw
conclusion on the center of coast. Uh so
maybe will have ideas and that's why I
want him to to be sure. But that one
stay here will take it over and most
case we close it before in this
milestone.
>> Okay.
Uh did you have anything to say about
the search CI migrate the controller VM
from uh CDF to Azure sponsor?
>> No, I'm reserving work on this.
>> Cool.
>> Need to command and set up task list.
>> Yeah. And the last issue which
you can um
>> um so we received messages from a user
about MySQL 8.0 O uh support
uh pay
is paying or is costing us additional
dollars in CDF subscription
on 31 July because
or upgrade to 8.4.
[sighs]
So, uh,
perform the A.4 upgrade
for the fun. I just wanted to see, uh,
was it really that good? That happened
really smoothly. So, that's at least
that's the good thing compared to the,
uh, to their old services, but that work
very smoothly.
But since Mato
never was put in production,
TB was empty
was not used. It wasn't empty. It was
not used. Removed these unused resources
CDF because these these things were
costing money not a lot but still
costing money on CDF. If we want to
resume mbo and half time we will
recreate the same database with the same
terapform code from scratch and set
things up in the other subscription
where we have credits. That's why I
haven't removed the resources in mata
mode.
>> Okay. So this one back to backlog now.
>> Good for you.
>> Yep.
>> Anything else on this on the work in
progress?
>> Uh no, I think we're good for the issues
in dry edge. Uh, I think the one that's
worth discussing the most is we'll have
to start the JDK patch upgrade campaign
for the new JDK versions that's being
released by Adoptim.
So on the JDK patch upgrade campaign, I
reviewed at least did some review of the
of the issues reported by Oracle in the
in their critical patch update. didn't
see anything that ju would justify us
forcing those upgrades onto the security
fix that's coming out this week. So, so
for me that was the choice to not put
those onto our container images is a
good choice. Let's stay with the
existing ones that we installed 3 months
ago for now. Yeah.
>> Yep. That's the exchange we had with
Steam, but I think it was private.
Sorry, Mark. We
>> No problem. That's that's those kind of
security things should be done in
private. So that that makes sense. It
just having done the review myself, I
thought, you know what, it I was I'm
worried about those because if there's
something really serious in the JDK, we
probably want to fix we want to update
to it, but I didn't see anything
particularly disastrous
fixes there.
Um Jay, are you okay to take this issue
once you are done with the other tasks?
>> Yeah.
>> Uh at least starting uh because the
packer images changes for instance can
be done.
>> Uh we need one last release of packer
image with NodeJS update and many
updates but then that could be the next
packer image update.
>> Cool.
>> Uh everything should start after the
security advisory. Good for you.
>> Got it. Yep.
>> J takes it can start after the at the
Zish at least for the infra items for
the non-infra items Mark Tim high who
are maintainers of these items will be
able to follow up and you will only have
to track things in that issue.
>> Got it.
>> Okay. Don't merge things too quickly on
this one.
>> That's an exercise. Uh, just one l I
tried to ask Clo and Jiminy to help me
write this issue and I'm really bad at
prompting because I've wasted two hours
of my time. I keep telling them, please
use the same formatting as the previous
issues. Here is the link to issue. Can
you get the row code, copy and paste and
update the links inside
it kept trying to do fancy things like,
oh no, here is a title in Mar. No, keep
the same formatting. Yes, I'm keeping
the same formatting. Here's the result.
But you kept the titles. No, I didn't. I
used the same formatting and it was
arguing with me and I was like, okay.
So, I ended [laughter] up copy and
pasting and doing the things myself
faster. Five minutes.
>> Yeah. Takes it takes it takes a little
trial and error, but uh
>> yeah, I'm not willing to spend hours on
changing text.
>> Yeah. Taking a little trial and error is
a polite way to say it wastes time.
[laughter] But yes, that's great. We
should we should we keep we keep
learning. That's good.
>> Certain things are not terribly
efficient.
>> Yeah, I'm not a die and retry game fan.
[laughter]
>> Okay,
>> thanks Jay.
>> So I don't think uh I don't know not
sure if there's any other issue we're
discussing discussing from the issues in
Rage. Uh we have two other to had in the
milestone the two um website request for
GSO.
>> Okay, cool.
>> Needed for GO need letly
infraci pipeline
minor changes
to be set up
same as above. So that's why we add
them. I don't have other issues right
now.
Um, Kubernetes 135, we can discuss that
in September.
Bookworm campaign is not emergency. Mark
just need to check when is a bookworm
end of life.
>> Bookworm. So bookworm is Debian 11. Uh
12
12 we got plenty of time.
>> Okay.
>> 11 11 reaches end of life uh in October
and I've already dropped it from from
for instance the git plugin and others.
So 11 is off my list. Its official end
is October of 2026.
>> Okay. The the reason is because uh um
bumping from bookworm to Trixie in the
current state of some of our images will
bump the version of LDAP, OpenSSL and
VPN
and Yep.
>> Yeah. So end of life date site says we
have almost two years before before book
room is end of life.
>> Good. Um because yep uh we need just to
clean up the docker compost stack that
allows local testing of both VPN and
LDAP altogether before being able to
handle this this update.
>> Okay. And I I guess yeah. So so it's in
LTS support right now. So the the the
nonLTS support ended in July of 2026,
but we've we've confidently relied on
their LTS support for a very long time.
>> Bullseye Bullseye end of life is 3 weeks
from now.
>> So 11 is gone.
>> Okay, cool. So that means maybe we could
have um October fest issue in this part
where we would want someone to resume
the build stack uh for these two ones.
So we can build and run them locally
with the right Elm file docker compose
whatever. So we can quickly iterate on
this in the future. We have some pieces
but they are not just not all tested all
together.
I don't have other issues that looks
worth mentioning or putting on the
milestone. Do you folks?
>> No.
>> Okay.
So, we'll see each other next week. I'm
going to stop recording.
Um,
and Jay, let me take care of publishing
the notes so I can clean it up with the
all the minor things I need to update
and put on my uh list that will help me.
That's why I'm asking for this.
>> Okay, cool.
>> So, your work is done. See you next week
for people watching us.