Video summary
The meeting began with announcements regarding the recent release of Jenkins 2.580, which was part of a security advisory and successfully deployed despite a minor issue where the Docker container image tag did not reflect the latest code commit immediately; this has been resolved in the subsequent 2.581 release. The team also discussed their capacity planning for the upcoming week, noting that Jay is on sick leave while Tamalmer may take time off due to kidney issues, potentially affecting the schedule for the next major release and LTS updates. Additionally, the roadmap was reviewed with a focus on moving configuration management away from Puppet and resuming automated Jenkins statistic collection, alongside new initiatives related to account management and future infrastructure tasks.
A significant portion of the discussion centered on cloud budget optimization and cost management across Azure and AWS environments. The team highlighted successful efforts in consuming GitHub-provided Azure credits until October 2027 and optimizing CI Genio costs on AWS, which are expected to last until January 2027. However, a critical decision point was identified regarding the future of their AWS usage; if Amazon does not renew their generous credit allocation, the team plans to migrate operations to Azure or other providers. The meeting also addressed challenges with external services like Groq and Docker Hub, which are facing rate limits and LLM-related disruptions, prompting strategies such as implementing rate limiting, using custom mirrors, and potentially building their own infrastructure within China to bypass network restrictions.
The final segment of the meeting covered ongoing maintenance tasks, including the retirement of Windows Server 2019 templates in favor of newer versions, updates to the Update Center root certificate rotation, and the cleanup of stale issues. The team also tackled a specific problem regarding Artifactory login failures caused by changing outbound IPs that were not yet reflected in their LDAP configurations. Furthermore, plans were laid out for migrating public clusters from US data centers to Sweden to improve capacity and reduce costs, along with testing strategies to manage IP changes without causing widespread disruption to users accessing the infrastructure. The meeting concluded with a triage of open issues, assigning responsibilities for upcoming milestones, and confirming that no further questions remained before ending the session.
Read the full video transcript
Hello everyone, welcome to the Genkins
infrastructure weekly team meeting.
Today we are the 8th September 2026 and
around the virtual table we have myself
tamalmer
and Robson. Welcome Robson.
Um Jay won't be there is in sick day and
Mark depends because it's quite early
for him. Uh so maybe we'll join later.
Let's get started with the
announcements.
So last week we were able to release
Jenkins 2.580.
Uh it went well for the release process
itself. It was part of the security
advisory. So it was on Wednesday and not
on Tuesday. Uh just the nitpicking
information the github the g tag for the
genkins cage container image and only
the container image uh was on the wrong
commit doesn't change that the docker
image is okay. It has the security fix.
It has genkins 250 as as expected. It's
just that the base image has uh a bit
less changes than it would be expected.
No worries. We recommend to use 2.58.1
that has been released earlier today. Uh
and we have uh located the the problem
that won't happen next time, but
absolutely no issue and it's only the
Docker image. It's only a tag on the
code. So no worries. That's the binary
itself.
This week we were able to release weekly
2.581.
Our release went well. No issue. Can you
confirm?
>> No. No. No fastly 503 annoying errors or
Okay, cool.
So, nothing else on weekly release. Do
you have something else on these
releases?
>> No.
>> Okay. Uh continuing team capacity. So,
Jay is off.
Uh I'm wondering I might take uh an
expectedly tomorrow's I'm not sure alpha
day or the full day. I don't see
anything but I might need some sleep due
to the kido. I will let you know in
detail survey but so yeah maybe I will
um I will be hope if that's the case I
will update the notes here uh if that's
okay with you on our bus
the road map deserve an update I will
send it a bit later uh mainly because we
started resuming the work on stats let
me open it on the screen just to be sure
we are all on same side I've opened the
link I'm clicking on the filter
infrastructure and services so the
current task is Move configuration
management out of pupets. That's work in
progress. So that one is correct. Uh
resume and automate Jenkins statistic
collection that should be in current.
And for the future related to accounts
that's still future, but we have two new
items that need to be added in the near
term or current. So that's just a slight
update. Our priority remain unchanged.
By the way, the epics. So that's the
same thing as what we have on the road
map but one one layer lower. So we have
pupet the top priority the campaign to
update Ubuntu 604 uh can be started at
least on the agent side and some sub sub
elements.
Uh we also have the autumn
2026 work for CI geno that's related to
CI genu running on AWS. We have finished
the billing and cost optimization
efforts. We have credits up until
January 2027.
So the upcoming four months will be
about finishing consumption of these
credits for CI genio and determining the
future. Can we have more credits from
AWS or do we need to move somewhere
else?
Any question on the road map and the
epics with these priorities?
>> No, I'm good.
uh topics worth mentioning not top level
priority but that might bite us in the
upcoming months mirror bits uh that will
be an issue in the upcoming milestone so
that one will be tackled very quickly uh
enginex ingress retirement and
kubernetes 1.35 upgrade uh I've put them
in this order because I believe the
engineix ingress retirement it should be
safer to fix it before bumping
Sorry,
>> sorry Robs. Oh, no problem. I thought
you were asking something. Um, so yes,
engineix ingress retirement should be
something we should start looking at. Uh
so as reminder do we need another
ingress controller or do we try to move
our charts to the new gateway API that
involve installing gateway controllers
and creating many objects instead of
sing simple ingresses.
In theory that should work however we
have risks around the way we use ingress
for updates genkins IO and get genkins
IO. So the gateway might not be the
right solution for us.
And a word on sponsoring. Uh I'm still
working with NUPS. Their plug-in start
to have a good shape. It's still private
source. Uh but I'm able to create
ephemeral agent on the cloud. Uh my next
step will be to report to them and try
to run the bomb builds on their
infrastructure.
Uh I will take I will let you know where
are they because as soon as the bomb
builds has walked
as large walked uh I will in that case I
will share the my token with you so you
will be able to test on your own and
then I will meet again with them to see
what will be the next step mainly open
sourcing their plug-in so it can be
hosted on plugins Jin say
hello back Robson
Okay. So, any question about
announcements, priorities, weekly
releases, team team epics and top level
tasks?
>> Not for me.
>> Okay, I'm continuing with the upcoming
calendar. Next infra meeting will happen
next Tuesday as usual on 15th September.
Same day we will have the weekly release
2.582 as usual.
Uh thanks survey for completing the
date. So the next LTS release will
happen on the last day of September.
That will be 2.58.1
that has been selected as the next line.
Mark started the work to create the
branches everywhere. I don't know when
the release candidate is planned for.
We'll see next week.
>> Do we have to care about the release
candidate? It's only a pull request in
Jenkins call.
>> Yes, it's just a fire. It's just for
information.
>> Okay.
>> No, no actionable for infrastructure
because it's not an automated process
that we have to watch. Uh that we used
to have it because before your work uh
that was generating a bunch of bomb
builds and costing a lot on CI genite
but now we we don't care about this.
>> Uh yeah, good point. We need the
backport preparation at least for docker
image.
We don't have any publicly announced
security advisory. Uh and we don't have
any credential expiration in the
upcoming three weeks.
No next major event either.
Question on the calendar.
Okay Robson, do not hesitate. If you
have a question, you can interrupt us at
any moment. Of course.
>> Thanks.
>> Okay. Cloud budget uh on the Azure CDF's
paid subscription
uh in August we were below 2.2. So, yep,
our the expected amount of money we can
spend is 2.5 monthly.
Um which remind me, let me write down I
need to do a summary for CDF billing
person. So Mark wait
uh summary before the end of year
because they need to know how to manage
the money on their bank account. So they
need to know when we will increase the
billing on the credit card on that
subscription.
We still have public cluster to move at
least from USist to Sweden data center
but ideally out of CDF subscription into
the sponsored subscription. So that's
less money that the CDF has to pay at
least for the upcoming months.
Sponsor subscription. We have increased
our consumption. That's good. We have
credits. These credits expire on
September uh 1, 207.
Oh no, sorry. No, they expire next year.
Expire on October 2077.
So we have to consume these credits
which have been given by GitHub in the
form of Azure credits. We are consuming
them. We should deplete on August next
year unless we had more more things that
includes public cluster but that doesn't
include CI genkins out of Amazon.
The increase is mainly due to search.ci
genkins. That's the controller used by
the genkins security team to verify the
private fixes and run them against
genkins score and other plugins. Um,
since we have more and more advisories,
maybe we will start tracking cost with
them because that's one care monthly and
that start to be uh not something
punctual. So maybe some optimization
could be be done on this one.
Any
questions so far on Azure credits?
>> No problem.
>> Okay. Digital, we have increased the the
consumption as expected because we are
now resuming the
uh the war consensus and that that
implies we increase the disk in order to
store the statistics. So we see a slight
increase. We still have 12 months.
That's far away and it's even after
credit expiration.
Depending on what the next steps, we
might need to add a few virtual machines
in digital. So we most probably will
need to contact digital to ask unlike
the two past years not to keep the same
amount but increase the amount of
credits.
Uh in the past year we weren't consuming
too much. So we just ask them to extend
the expiration date of the credits. I
believe we should need a bit more
especially if we plan to move out of
cloudflare to custom VPS machines on
that provider because bandwidth is
really cheap at digital
and on Amazon uh great job with the bomb
optimization. So we see a visible
decrease that's is not only because it
was August
So that means we will have CI genio
running on Amazon until January 2027. We
now have four months to decide uh if we
can get more credit from Amazon. If they
give us 60K like they used to to do at
least once per year we can cover all
2027
assuming a few optimization but more or
less we are we are covered. If they
don't then we need to move to Azure or
somewhere else find solution and
implement it. I've opened a new epic to
keep track of this. That means we stop
the efforts around cost reduction on
Amazon.
Any questions on this one?
>> Cool.
On the sponsoring area, Grog uh is
hosting repo genkins.org or where we
host everything every Maven dependencies
and binaries uh in complement to our
distribution system. Uh like everyone
else nowadays they are being hit by this
crappy LLM. So that mean a bunch of uh
outbound bandwidth thing is they don't
answer and they don't provide solution
to block ips put rate limit or other
solutions. So we know that we will
explode the the download bandwidth but
there is nothing we can do about it.
It's a public service.
So, yep, we'll see if we have news from
them since they should be back from
holidays as well in September.
Alolia as proposed by I believe we can
stop tracking this weekly. Maybe a rate
of tracking it monthly should be better.
Uh since they increased our threshold,
we don't get hit by LM anymore. um we
implemented rate limits and and blockers
on some specific IPs. They also added
dynamic blockers on their own
infrastructure with the help of
Cloudflare. So that also improved the
situation for them because same they are
being hit really hard by the LLM and
that makes no sense.
So that's all. So no major problem on
cloud budgets. We need to work on
renewal of AWS or moving somewhere else.
That's the main topic. Any questions on
the cloud budget?
Cool. Let's move to the task list.
During the past milestone, we were able
to finish the following tasks in the
category keeping the infrastructure up
to date. We got
an LTS new release. So, we had to deploy
to our own infrastructure the same day.
Nothing specific about it. Do you have
questions about that issue? That's a
regular one.
Um, so we had terraform state
credential. So the credential used by
our terapform project to access the
shared states. We have one pair for each
project. So each project uh cannot write
down on the others. It was expiring
because we cannot use credentialless
workloads.
So it had to be renewed. Nothing
specific here. Uh I handled them last
week.
Any question on this one?
>> No,
>> because initially I wanted you to do it
but uh with the tasks I assigned to you
that was too much. So that's why I took
it over as we agreed but just wanted to
be sure it was acknowledged by you and
nothing else.
Uh we closed the GDK critical patch
upgrade campaigns.
So we are still trying to discover the
upcoming rate of updates since both
Oracle and adoptium during the summer
agreed upon the monthly release cadons
for GDKs. So now we will have to update
GDKs uh patches by patches monthly and
not quarterly. That will be a huge
change in the ecosystem.
Um
I believe we might want to adapt our
process
just the reporting part because yes now
with a monthly update uh the cadons on
the docker images might be a bit
different especially for LTS or weekly
we don't care we deploy it once a week
but for our LTS controller maybe we will
have to wait uh regularly
the main impact will be on the SIG
platform not in infrastructure. platform
for genkins project because that mean
maybe we will need to backport way more
often the GDKs
uh for infrastructure that will be a bit
less work for us
on the support category
there was a new plug-in uh hosted on
plug-in genkins IO not not discovered
anymore but the person wanted something
fully automatic geninio needs up to 24
hours before automatic discovery.
Everything is fine. Thanks Mark for
taking care of this one.
No question so far.
Okay, continuing then. Uh on the
category keeping the infrastructure sane
and maintainable, we have a new mirror
in Romania. Uh thanks for the people at
Zerolag for that and thanks Har for
setting it up. Oh, which might be
important.
We have to update our goip database
because right now our mirror system
locate that mirror not in Romania but in
China.
Can you remind me? Do you remember if it
was set up as country only?
>> No.
>> Okay. Because it's located it's seen as
located in China.
Um that will be a way to bypass the JIP
locator. We could set that mirror in
Romania.
Shortterm set it to country only
to bypass this because that mean it will
only be used by requests coming from
Romania. I don't remember if we have a
second one in Romania country as well.
worst case they will compete but yeah in
any case that we need to update so we
will need to add in the new milestone
the resuming the chrome joatabase update
uh has been migrated to the new
subscription and require a few changes
that uh that we deployed to other
controllers uh that has been done
nothing specific here everything went
pretty Mos,
do you have a word about bomb costs
mainly reporting what you saw in costs?
>> Uh yeah, so we don't have uh we can't
really uh it's difficult to measure the
specific cost of the bomb build. But
looking at the total cost of uh in AWS
we can see that from June to July we
decrease by 25.3%
from July to August from another 8.5%
or a total decrease of uh 31.6%
since selection of split test.
I can let me pass uh stat resume.
>> Thanks.
The interesting part is um compute cost
seems to be really low compared to all
the other costs mainly bandwidth
especially in AWS that make you pay for
that when you download dependencies
docker images layers or that you
transfer files between agents or between
availability zone.
That would be one of the cost
optimization if they gave us more
more uh more and more of things. either
we add cash per availability zone or we
stop doing multi availability zone again
because yep we saw that cross
availability zone doesn't work because
when they decide to be out of compute
capacity in the in in Amazon they are in
the war region and not as so multiple
availabilities only protect you from
fall tolerance which we don't really
care if the zone is down the zone is
down
thanks for the reporting survey
and next issue. We were able to finish
migration of search CI genkinio
controller out of CDF to sponsored
subscription. So we don't pay for that
one. That's not a lot of money. It's
like 20 bucks monthly. However,
um not a big gain or billing,
but we verified that we can
have things running in Sweden
because search CI controller and agent
are now running in the Sweden Azure data
center which has way more capacity than
the USist one that we use. So we can
start moving things around step
component by components. We can have
cross region the added uh costs and uh
latency are absolutely okay as per what
we saw in CCI. So that's good news. Uh
we may have more flexibilities
to have less suffering from Azure not
being able to scale their data centers.
Any question on the done tasks?
No.
>> Cool. On the closed as not plan issue,
we
uh so we decided to embed the not with
time retry pipeline library for only our
website to a more generic build website
uh function. Uh let's let's put the
effort on the right area. You started to
prototype something. reminder is that we
have seven websites and we want to build
them all with always the same way to
decrease the attack surface on our
infrastructure
uh to provide a better user experience
and to be sure that we don't change one
website and forgot about the other
breaking the pull request when we do an
infrastructure change. That's the goal.
So we close that issue in favor of a new
one that is inside the tri edge list and
as such it has been closed as not
planned.
And then the support part, thank survey
for cleaning up. That's a cleanup for
something that has been fixed month if
not years ago.
Now work in progress. So all the issues
that we started to work or worked on but
are not finished as part of the
milestone because that's that scope way
on a way larger time window.
We have a new support issue. Someone is
not able to log into Artifactory. uh
looks like they don't have an account in
Artifactory yet, but when they try to
login with their genkins
password, that's give them an error. The
problem could be we expect user to be
part of a group or existing plug-in
permission scheme. So, we need for them
to finish their plug-in hosting request.
I was thinking about an egg and chicken
problem, but we would have seen that
earlier. There is something else could
be artifactory which changed their
outbound IP and it's not detected by our
LDAP yet. I thought it was automatically
updated once a day but maybe maybe not
worth checking.
Could it be new outborn IP not allowed
in our LDAP?
check our automation
uh because Artifactory like GitHub
publish JSON API with their outbound IPs
that change sometimes when they change
their network infrastructure and our
LDAP is closed. So it only allow a few
external IP including Artifactory. If
the user logged in and they use the
newborn IP, LDAP refused the connection
and that could be a word or message on
the site.
Any question on this one?
No,
>> of course I also recommended to the user
to not go the way of manually [snorts]
raising plugins.
The the way to go is CD with exclusive
mode which mean no human should be able
to release a genkins plug-in. That's
safer for everyone and easier for
everyone. That's just a few option to
change in a YL file in repository
permission updater in the hosting
request. Uh and there are many
techniques to trigger the release only
when you need it. So it's only playing
with the labels of your pull request.
That should be quite easy.
Next topic uh open but stale that will
stay with us. Um so the rate limit from
docker
it's not authentication. It's only CI
geno. We haven't seen it since a few
days, but it still happens from time to
time, especially when we have a bunch of
pull requests.
Um,
we evicted the problems related to our
transparent proxy. It was working as
expected, but sometimes Docker C uh was
trying to deal with TLS while we don't.
Not sure what the problem is. Seems like
a word infrastructure or network error
or a bug in Docker client.
uh for now we keep it open to continue
and documents. One suggestion uh we have
an idea that Seran and I had will be to
try to implement
uh custom registry on all images on all
the from directive of docker files. So
we could optionally specify the ais
docker mirror. That's an explicit mirror
that you need to add a prefix in the
name of your docker images on both tests
bake files docker files.
But that will mean instead of having to
fall back to docker hub the docker
engine will directly get the internal
Amazon copy of the docker mirror which
is not rate limited internally. That
will be a solution.
that require a bit of tooling. It's not
transparent though. It's not like in
Azure.
Let's try to support
custom registry to allow usage of AWS
internal is here.
Docker
mirror. Any questions so far?
>> Okay, next one. create sonar cloud
token. We have that contributor. We
started to build a pipeline library to
allow CI geno to scan things from sonar
cloud. CI genio is public. So we have a
problem here of any token here should be
should have should be really scoped
because it's most probably already being
bound. It's a public genkins instance.
Um problem is the organization we have
at center cloud an oss plan does not
allow to have organization scoped token
which mean we need a p if in order to do
this so we don't have a solution yet the
user asked uh sonar cloud if they can
allow that feature for us on oss no
answer so far the alternative we
proposed that need to be review by the
gensec team and the administrator of the
genkins CI organization will be to
create a doomi bot user with shared MFA
owned by the genkins infra team. It
should only have read only on the
repository in GitHub and we will use
that user to create a personal access
token. So if that token is spawned the
actions that the the attack the
attackers would have will only be read
only that will limit the coverage and
would still allow the sonar cloud
scanning to run for our plug-in and
contributors. still a benefit.
>> Are there any question on this one?
>> And Jenkins CI admin to validate the
technical
user proposal using a P8
Ilcore. We've pinged the Adreion.
He's still analyzing. I guess he has
other priorities. So the ill score is
still is still showing bad results. We
might need to sync with Adrian because
it's not only the ill score on plugins
genkins IO but that's also on any
genkins instance in the update plug-in
list. So there might be improvement or
things to disable in the upcoming
releases.
Waiting for Adriina for a full state
analysis before we we check what we can
improve in monitoring.
Finally, support mirror selection in
update center. So we have that chi China
based user who complain about the only
mirror for downloads genkins the tuna
university sometimes blocks user outside
the university network inside China
leaving uh them to receive four or three
errors when they updates their plugins
which is quite the problem. problem we
have is that we used to have mirrors in
China but none of the mirror
administrator answers and none of them
provide us a way to scan the mirror so
we cannot include them inside our
infrastructure.
We have disabled the tuna mirror and
right now the user said it's okay for
them because they are redirected to I
think it's Taiwan based mirror which
mean their ISP allows connection outside
the great firewall of China. We haven't
heard from any other users in China. So
maybe we broke other user. We don't
know. That's the only mirror we have
there.
We gave the user solution and we pushed
forward for them. So they contact
Alibaba or any other organization so
they can get us a mirror at least the
mirror that we can have for HTTPS and
air sync for scans.
So right now waiting for the user
or the user to get us a sponsor or
contact.
We'll add back tuna. Uh so no okay
question is should we add back tuna
mirror? How much time do we wait?
Because right now we are totally blind.
We don't have a way to measure things
from within China's network.
My proposal is that we keep it disabled
for one month and we see if people uh
complains if no one complains we remove
it because tuna administrator do not
answer at all when we contact them. So
either we have the bad contract contact
address or they just don't care don't
monitor or whatever. Is it worth running
adding a mirror that is not able to act
if something goes wrong?
I believe without anyone complaining in
China, I would prefer having less
mirrors but having people happy when
they complain.
However, that's still worth finding a
mirror. So, Tuna is disabled.
Let's
wait until I propose end of September.
Yes, I'm going to
>> before making decision
as they don't answer to our emails.
We should work on building a mirror in
China ourselves. So the goal will be to
find the right
uh the right level of uh sponsoring.
There are many VPS VPs inside China that
we could use with a two CPU 2 GB machine
on the right hard drive that will also
allow us to use it as a mirror for
update center providing the metadatas
updates every 5 minutes inside China
instead of uh Europe or US.
So that's the current status.
Are there any question on this one?
>> No.
>> Cool. Thanks. On the keep infrastructure
up to date, the update center root CR
rotation um waiting for the new CA to be
uh released in genkins call. Once it be
it will be released at least on the
weekly line we'll have to work uh with
the gen key security team on how to up
what's will be the timeline for updating
it inside the update center.
So right now stale
>> Windows 2019 what's the status survey?
So we don't have any more um we the only
remaining consumer of Windows 2019 image
agent
is windp
uh that will take care of migrating to
2022 or 2025 vidally
and uh I've removed the packer image
[snorts] uh the next step will be to
remove the Azure image. It's in Azure
Gallery as they're not used anymore.
It's uh done via Terraform
or AWS MEI. We have to wait until WinP
is migrated. Then we'll be able to
delete them manually.
Um another step will be to delete uh
um Windows 2019 template in your
controllers.
Uh
still waiting for WP. Uh to be sure that
not deleting
WP
uh template the template used by WP for
now.
Uh my brain just reminded me uh that the
other gallery can't be removed
immediately. We still have 2019 agent
labels in search CI. We need to clean up
the agent template first.
They are not I don't think they should
be used or if if it is it's not because
it's 2020 nine. Uh but we need to
update.
>> Yeah, we'll ask um be asked to think
like
I'm I'm not sure but I think the last
time we mentioned those Windows
templates they were not really using
them. So to be yeah
>> they they used Windows agent on search
CI that's a fact. But do they care if
it's 2019 or 25? I don't think so. I
think they even already started to use
2025 without even knowing. That's I
think was the conclusion in the add
windows 25 15
of the 20 five uh template by default
using them by default
>> but we still need to work on the cleanup
first before removing the other
galleries because if we remove the
gallery and we restart the certi
controller it will make it will throw a
bunch of errors on the other VM plug-in
and that that might have impact I forgot
about this one.
>> No problem. I'll update my message in so
>> cool. Thanks.
Any question?
>> No.
>> Okay.
Um, infrasane and maintainable category.
So, you added this new one. We did not
have time, but that will be the next
step. We will try to merge the dock
builds
that require Windows 2022 base image and
host into 2025 because it's possible in
theory.
>> Oh, we have already docker and docker
agent using a windows 25 agent. Uh the
only remaining is the docker is docker
ssh agent.
>> I have docker and docker agents project.
>> Sorry the project opens. uh and really
solution.
>> Can can you confirm that I'm not
mistaken that it works on both AWS EC2
and Azure virtual machines because
that's not the same.
>> Yeah. Yeah. Cool. And
>> next we is docker SSH agent and if I
remember correctly you were blocked on
SSH agent. No,
>> no, it's is a pull request is ready for
resume since
Yeah, it's ready.
Okay,
next steps will be done. Find if any
other
consumers
either
>> project or uh controller agent templates
>> just to be sure what need to be cleaned
up on infraside. We are also consumer
somehow. We have the release using 2022
agent on AKS and the blocker was
previously that 2025 wasn't available as
AKS node. It's not the case since June.
>> So we should be able to update the node
pool and move to the 2025, right?
>> Yes.
>> Cool.
Or we could get rid of the Kubernetes
Windows nutpool and use as VMs.
>> Yeah, but that's
Yeah. Uh that's another help desk.
That's not really Yeah.
>> Yep. Depends on the timeline, right? We
have to choose one or the other.
Uh then we can get rid of Windows 2022
just in time for Windows 20 27 or 28. I
don't know which one will be the next.
Thanks. So we keep that issue in the
current milestone
stat generation. Uh
J uh so you may you added May and June
2025
visible on stats genino.
Thanks. Uh July in progress,
August by J.
Then next step automation
up to March 2026
and
we have to contact KK
to resume
March
up to today from his basement machine.
There is a whole issue about removing KK
as a bus factor.
I'll see uh I'll see with him if he's
okay to check with the board if we can
delegate that to someone else or if he
wants to continue.
But I haven't heard from Kiki since he
left cloud B again. So, yep. I hope he's
doing fine.
Okay. Any questions so far on the work
in progress? No issue to remove. We
continue working on this.
We have a few issues to tri edge. Thanks
survey for keeping care. So the
following issue I propose we add them to
the new milestone that's part of the
triage. I will update the labels after
the meeting.
So the CI genio persisting a new SM
issue while cleaning up few word
elements. We had we have the explanation
from Mark that mean we can at least
persist the change to avoid bite
surprise because it was only done
manually on the controller and now we
can persist it as gaskas uh code that
also mean we should do an audit on what
g tool we are using across the
infrastructure at least on infraci and
eventually release we should get rid of
git tool instead of the command line
tool
build website we mentioned it earlier so
you started a prototype. Uh we will let
you comment.
We have mirror bit bump. We are sharing
the work on this one. That's no major
change, no bad surprise. A bit of Elm
chart to add the new option uh that will
help us to investigate when a mirror is
disabled
>> and allow also the view sort to be uh
yeah that's what the new new option not
sure if you will use it or not. That
will be interesting to test. But yeah,
uh clean up auto link references. I
guess team is no
>> team is okay to run the script. I have
to prepare that.
>> Okay. To run the cleanup script as admin
of genkins ci
um the new publicates cluster. Uh so
migration to Sweden
plus uh sponsored subscription.
So the one last thing for the plan I
need to do on this one for the public
cluster I want to test if I can manually
move a public IP object in Azure from
one subscription to another.
The reason if possible I will want to
avoid changing our inbound IPs because
many users that are trying to reach
genkins infrastructure without even
knowing they reach that cluster [gasps]
they usually contact their network team
and ask them to open the IPs on their
firewall. If we change these public IPs
that are the entry point I don't
remember if it's one or two I think it's
only one in Azure. I don't do dual IP
stack load balancers. So that IP if if
it changes
uh then I will prefer keeping the same
even if it means stopping the all the
services for one hour.
Uh I would prefer an outage but not
having bunch of user having to again do
the on the same dance. Hey hello can you
open this new IP blah blah blah because
that mean we will need to communicate on
many lay many things just for changing
an IP.
So I need to run a test. I will create a
duplicate uh set of IPs the same as we
have for the public V6 and V4 but not
allocated or allocated to a virtual
machine. We don't really care. I will
try to move them to the new subscription
in the new location to see what happens.
Depending on that we will have two
slightly different plans. If it fails as
usual we create the new IPs the new
resource the new cluster the new pods
and then we move we use a DNS move from
one to the other. Each service is a
CNAME to the cluster uh public DNS. So
we can move service by service like we
did in the past. It's easily to it's
easy to test. We rewrite our EDC host
locally on our machines and we can do a
full testing.
But that means changing IP. If my
[snorts] technique works, I will still
create public new public IPs that will
allow independent testing and eventually
doing blue green uh migrations.
But when we will move everything, we
will move the IPs manually and recreate
the load balancer. So full outage but we
will have a different IP. These are the
two plans we have. I need to be sure the
second plan is possible before asking
people to choose otherwise I will just
waste people time.
Clear for everyone?
Yes.
And finally a bit of Terapform code.
That one should be quick. That will
unlock uh our ability to use recent
Kubernetes providers on our Terraform
project.
>> Do we take uh resume show IP data which
>> yes I think I think we should.
>> Yeah,
>> thanks for the question and I propose
that we keep the NodeJS 2224 for the
last service uplink uh for someone in
the future. Yeah, we are almost in
October. So, we will have either October
fest or or any uh really highly
motivated student with a good LLM that's
dependency update in NodeJS. They will
be able to do a bunch of test and
updates, I'm sure.
Okay, that's all for the the issue
triage. I will take care of updating the
labels, milestone stuff after the
meeting and after my lunch.
Now, are there questions and things you
want to discuss
before we close that meeting? Things to
clarify, question, whatever.
>> Good for me.
>> Okay. So, Robson, did that meeting was
useful for you? Did you was there
specific part you were interested into?
>> Very interesting.
I learn
uh for for you and Harin
my first time and I prefer to listen.
[laughter]
>> Okay, no problem. Do not hesitate to
join our elements matrix channel if you
have question that is here to discuss
asynchronously
but you are welcome to join that's
public meeting for public
infrastructure.
>> Thanks so much.
>> Okay so if no other topics and no
question I'm going to stop screen share.
So one last chance for asking question
otherwise I will stop recording.
5 second timeout. No more questions.
Then
I'm stopping reporting for people
watching us. See you next week.