Submind YouTube summaries
Thumbnail for 2026 09 08 Jenkins Infra Meeting

2026 09 08 Jenkins Infra Meeting

Watch on YouTube

Video summary

The meeting began with announcements regarding the recent release of Jenkins 2.580, which was part of a security advisory and successfully deployed despite a minor issue where the Docker container image tag did not reflect the latest code commit immediately; this has been resolved in the subsequent 2.581 release. The team also discussed their capacity planning for the upcoming week, noting that Jay is on sick leave while Tamalmer may take time off due to kidney issues, potentially affecting the schedule for the next major release and LTS updates. Additionally, the roadmap was reviewed with a focus on moving configuration management away from Puppet and resuming automated Jenkins statistic collection, alongside new initiatives related to account management and future infrastructure tasks. A significant portion of the discussion centered on cloud budget optimization and cost management across Azure and AWS environments. The team highlighted successful efforts in consuming GitHub-provided Azure credits until October 2027 and optimizing CI Genio costs on AWS, which are expected to last until January 2027. However, a critical decision point was identified regarding the future of their AWS usage; if Amazon does not renew their generous credit allocation, the team plans to migrate operations to Azure or other providers. The meeting also addressed challenges with external services like Groq and Docker Hub, which are facing rate limits and LLM-related disruptions, prompting strategies such as implementing rate limiting, using custom mirrors, and potentially building their own infrastructure within China to bypass network restrictions. The final segment of the meeting covered ongoing maintenance tasks, including the retirement of Windows Server 2019 templates in favor of newer versions, updates to the Update Center root certificate rotation, and the cleanup of stale issues. The team also tackled a specific problem regarding Artifactory login failures caused by changing outbound IPs that were not yet reflected in their LDAP configurations. Furthermore, plans were laid out for migrating public clusters from US data centers to Sweden to improve capacity and reduce costs, along with testing strategies to manage IP changes without causing widespread disruption to users accessing the infrastructure. The meeting concluded with a triage of open issues, assigning responsibilities for upcoming milestones, and confirming that no further questions remained before ending the session.
Read the full video transcript
Hello everyone, welcome to the Genkins infrastructure weekly team meeting. Today we are the 8th September 2026 and around the virtual table we have myself tamalmer and Robson. Welcome Robson. Um Jay won't be there is in sick day and Mark depends because it's quite early for him. Uh so maybe we'll join later. Let's get started with the announcements. So last week we were able to release Jenkins 2.580. Uh it went well for the release process itself. It was part of the security advisory. So it was on Wednesday and not on Tuesday. Uh just the nitpicking information the github the g tag for the genkins cage container image and only the container image uh was on the wrong commit doesn't change that the docker image is okay. It has the security fix. It has genkins 250 as as expected. It's just that the base image has uh a bit less changes than it would be expected. No worries. We recommend to use 2.58.1 that has been released earlier today. Uh and we have uh located the the problem that won't happen next time, but absolutely no issue and it's only the Docker image. It's only a tag on the code. So no worries. That's the binary itself. This week we were able to release weekly 2.581. Our release went well. No issue. Can you confirm? >> No. No. No fastly 503 annoying errors or Okay, cool. So, nothing else on weekly release. Do you have something else on these releases? >> No. >> Okay. Uh continuing team capacity. So, Jay is off. Uh I'm wondering I might take uh an expectedly tomorrow's I'm not sure alpha day or the full day. I don't see anything but I might need some sleep due to the kido. I will let you know in detail survey but so yeah maybe I will um I will be hope if that's the case I will update the notes here uh if that's okay with you on our bus the road map deserve an update I will send it a bit later uh mainly because we started resuming the work on stats let me open it on the screen just to be sure we are all on same side I've opened the link I'm clicking on the filter infrastructure and services so the current task is Move configuration management out of pupets. That's work in progress. So that one is correct. Uh resume and automate Jenkins statistic collection that should be in current. And for the future related to accounts that's still future, but we have two new items that need to be added in the near term or current. So that's just a slight update. Our priority remain unchanged. By the way, the epics. So that's the same thing as what we have on the road map but one one layer lower. So we have pupet the top priority the campaign to update Ubuntu 604 uh can be started at least on the agent side and some sub sub elements. Uh we also have the autumn 2026 work for CI geno that's related to CI genu running on AWS. We have finished the billing and cost optimization efforts. We have credits up until January 2027. So the upcoming four months will be about finishing consumption of these credits for CI genio and determining the future. Can we have more credits from AWS or do we need to move somewhere else? Any question on the road map and the epics with these priorities? >> No, I'm good. uh topics worth mentioning not top level priority but that might bite us in the upcoming months mirror bits uh that will be an issue in the upcoming milestone so that one will be tackled very quickly uh enginex ingress retirement and kubernetes 1.35 upgrade uh I've put them in this order because I believe the engineix ingress retirement it should be safer to fix it before bumping Sorry, >> sorry Robs. Oh, no problem. I thought you were asking something. Um, so yes, engineix ingress retirement should be something we should start looking at. Uh so as reminder do we need another ingress controller or do we try to move our charts to the new gateway API that involve installing gateway controllers and creating many objects instead of sing simple ingresses. In theory that should work however we have risks around the way we use ingress for updates genkins IO and get genkins IO. So the gateway might not be the right solution for us. And a word on sponsoring. Uh I'm still working with NUPS. Their plug-in start to have a good shape. It's still private source. Uh but I'm able to create ephemeral agent on the cloud. Uh my next step will be to report to them and try to run the bomb builds on their infrastructure. Uh I will take I will let you know where are they because as soon as the bomb builds has walked as large walked uh I will in that case I will share the my token with you so you will be able to test on your own and then I will meet again with them to see what will be the next step mainly open sourcing their plug-in so it can be hosted on plugins Jin say hello back Robson Okay. So, any question about announcements, priorities, weekly releases, team team epics and top level tasks? >> Not for me. >> Okay, I'm continuing with the upcoming calendar. Next infra meeting will happen next Tuesday as usual on 15th September. Same day we will have the weekly release 2.582 as usual. Uh thanks survey for completing the date. So the next LTS release will happen on the last day of September. That will be 2.58.1 that has been selected as the next line. Mark started the work to create the branches everywhere. I don't know when the release candidate is planned for. We'll see next week. >> Do we have to care about the release candidate? It's only a pull request in Jenkins call. >> Yes, it's just a fire. It's just for information. >> Okay. >> No, no actionable for infrastructure because it's not an automated process that we have to watch. Uh that we used to have it because before your work uh that was generating a bunch of bomb builds and costing a lot on CI genite but now we we don't care about this. >> Uh yeah, good point. We need the backport preparation at least for docker image. We don't have any publicly announced security advisory. Uh and we don't have any credential expiration in the upcoming three weeks. No next major event either. Question on the calendar. Okay Robson, do not hesitate. If you have a question, you can interrupt us at any moment. Of course. >> Thanks. >> Okay. Cloud budget uh on the Azure CDF's paid subscription uh in August we were below 2.2. So, yep, our the expected amount of money we can spend is 2.5 monthly. Um which remind me, let me write down I need to do a summary for CDF billing person. So Mark wait uh summary before the end of year because they need to know how to manage the money on their bank account. So they need to know when we will increase the billing on the credit card on that subscription. We still have public cluster to move at least from USist to Sweden data center but ideally out of CDF subscription into the sponsored subscription. So that's less money that the CDF has to pay at least for the upcoming months. Sponsor subscription. We have increased our consumption. That's good. We have credits. These credits expire on September uh 1, 207. Oh no, sorry. No, they expire next year. Expire on October 2077. So we have to consume these credits which have been given by GitHub in the form of Azure credits. We are consuming them. We should deplete on August next year unless we had more more things that includes public cluster but that doesn't include CI genkins out of Amazon. The increase is mainly due to search.ci genkins. That's the controller used by the genkins security team to verify the private fixes and run them against genkins score and other plugins. Um, since we have more and more advisories, maybe we will start tracking cost with them because that's one care monthly and that start to be uh not something punctual. So maybe some optimization could be be done on this one. Any questions so far on Azure credits? >> No problem. >> Okay. Digital, we have increased the the consumption as expected because we are now resuming the uh the war consensus and that that implies we increase the disk in order to store the statistics. So we see a slight increase. We still have 12 months. That's far away and it's even after credit expiration. Depending on what the next steps, we might need to add a few virtual machines in digital. So we most probably will need to contact digital to ask unlike the two past years not to keep the same amount but increase the amount of credits. Uh in the past year we weren't consuming too much. So we just ask them to extend the expiration date of the credits. I believe we should need a bit more especially if we plan to move out of cloudflare to custom VPS machines on that provider because bandwidth is really cheap at digital and on Amazon uh great job with the bomb optimization. So we see a visible decrease that's is not only because it was August So that means we will have CI genio running on Amazon until January 2027. We now have four months to decide uh if we can get more credit from Amazon. If they give us 60K like they used to to do at least once per year we can cover all 2027 assuming a few optimization but more or less we are we are covered. If they don't then we need to move to Azure or somewhere else find solution and implement it. I've opened a new epic to keep track of this. That means we stop the efforts around cost reduction on Amazon. Any questions on this one? >> Cool. On the sponsoring area, Grog uh is hosting repo genkins.org or where we host everything every Maven dependencies and binaries uh in complement to our distribution system. Uh like everyone else nowadays they are being hit by this crappy LLM. So that mean a bunch of uh outbound bandwidth thing is they don't answer and they don't provide solution to block ips put rate limit or other solutions. So we know that we will explode the the download bandwidth but there is nothing we can do about it. It's a public service. So, yep, we'll see if we have news from them since they should be back from holidays as well in September. Alolia as proposed by I believe we can stop tracking this weekly. Maybe a rate of tracking it monthly should be better. Uh since they increased our threshold, we don't get hit by LM anymore. um we implemented rate limits and and blockers on some specific IPs. They also added dynamic blockers on their own infrastructure with the help of Cloudflare. So that also improved the situation for them because same they are being hit really hard by the LLM and that makes no sense. So that's all. So no major problem on cloud budgets. We need to work on renewal of AWS or moving somewhere else. That's the main topic. Any questions on the cloud budget? Cool. Let's move to the task list. During the past milestone, we were able to finish the following tasks in the category keeping the infrastructure up to date. We got an LTS new release. So, we had to deploy to our own infrastructure the same day. Nothing specific about it. Do you have questions about that issue? That's a regular one. Um, so we had terraform state credential. So the credential used by our terapform project to access the shared states. We have one pair for each project. So each project uh cannot write down on the others. It was expiring because we cannot use credentialless workloads. So it had to be renewed. Nothing specific here. Uh I handled them last week. Any question on this one? >> No, >> because initially I wanted you to do it but uh with the tasks I assigned to you that was too much. So that's why I took it over as we agreed but just wanted to be sure it was acknowledged by you and nothing else. Uh we closed the GDK critical patch upgrade campaigns. So we are still trying to discover the upcoming rate of updates since both Oracle and adoptium during the summer agreed upon the monthly release cadons for GDKs. So now we will have to update GDKs uh patches by patches monthly and not quarterly. That will be a huge change in the ecosystem. Um I believe we might want to adapt our process just the reporting part because yes now with a monthly update uh the cadons on the docker images might be a bit different especially for LTS or weekly we don't care we deploy it once a week but for our LTS controller maybe we will have to wait uh regularly the main impact will be on the SIG platform not in infrastructure. platform for genkins project because that mean maybe we will need to backport way more often the GDKs uh for infrastructure that will be a bit less work for us on the support category there was a new plug-in uh hosted on plug-in genkins IO not not discovered anymore but the person wanted something fully automatic geninio needs up to 24 hours before automatic discovery. Everything is fine. Thanks Mark for taking care of this one. No question so far. Okay, continuing then. Uh on the category keeping the infrastructure sane and maintainable, we have a new mirror in Romania. Uh thanks for the people at Zerolag for that and thanks Har for setting it up. Oh, which might be important. We have to update our goip database because right now our mirror system locate that mirror not in Romania but in China. Can you remind me? Do you remember if it was set up as country only? >> No. >> Okay. Because it's located it's seen as located in China. Um that will be a way to bypass the JIP locator. We could set that mirror in Romania. Shortterm set it to country only to bypass this because that mean it will only be used by requests coming from Romania. I don't remember if we have a second one in Romania country as well. worst case they will compete but yeah in any case that we need to update so we will need to add in the new milestone the resuming the chrome joatabase update uh has been migrated to the new subscription and require a few changes that uh that we deployed to other controllers uh that has been done nothing specific here everything went pretty Mos, do you have a word about bomb costs mainly reporting what you saw in costs? >> Uh yeah, so we don't have uh we can't really uh it's difficult to measure the specific cost of the bomb build. But looking at the total cost of uh in AWS we can see that from June to July we decrease by 25.3% from July to August from another 8.5% or a total decrease of uh 31.6% since selection of split test. I can let me pass uh stat resume. >> Thanks. The interesting part is um compute cost seems to be really low compared to all the other costs mainly bandwidth especially in AWS that make you pay for that when you download dependencies docker images layers or that you transfer files between agents or between availability zone. That would be one of the cost optimization if they gave us more more uh more and more of things. either we add cash per availability zone or we stop doing multi availability zone again because yep we saw that cross availability zone doesn't work because when they decide to be out of compute capacity in the in in Amazon they are in the war region and not as so multiple availabilities only protect you from fall tolerance which we don't really care if the zone is down the zone is down thanks for the reporting survey and next issue. We were able to finish migration of search CI genkinio controller out of CDF to sponsored subscription. So we don't pay for that one. That's not a lot of money. It's like 20 bucks monthly. However, um not a big gain or billing, but we verified that we can have things running in Sweden because search CI controller and agent are now running in the Sweden Azure data center which has way more capacity than the USist one that we use. So we can start moving things around step component by components. We can have cross region the added uh costs and uh latency are absolutely okay as per what we saw in CCI. So that's good news. Uh we may have more flexibilities to have less suffering from Azure not being able to scale their data centers. Any question on the done tasks? No. >> Cool. On the closed as not plan issue, we uh so we decided to embed the not with time retry pipeline library for only our website to a more generic build website uh function. Uh let's let's put the effort on the right area. You started to prototype something. reminder is that we have seven websites and we want to build them all with always the same way to decrease the attack surface on our infrastructure uh to provide a better user experience and to be sure that we don't change one website and forgot about the other breaking the pull request when we do an infrastructure change. That's the goal. So we close that issue in favor of a new one that is inside the tri edge list and as such it has been closed as not planned. And then the support part, thank survey for cleaning up. That's a cleanup for something that has been fixed month if not years ago. Now work in progress. So all the issues that we started to work or worked on but are not finished as part of the milestone because that's that scope way on a way larger time window. We have a new support issue. Someone is not able to log into Artifactory. uh looks like they don't have an account in Artifactory yet, but when they try to login with their genkins password, that's give them an error. The problem could be we expect user to be part of a group or existing plug-in permission scheme. So, we need for them to finish their plug-in hosting request. I was thinking about an egg and chicken problem, but we would have seen that earlier. There is something else could be artifactory which changed their outbound IP and it's not detected by our LDAP yet. I thought it was automatically updated once a day but maybe maybe not worth checking. Could it be new outborn IP not allowed in our LDAP? check our automation uh because Artifactory like GitHub publish JSON API with their outbound IPs that change sometimes when they change their network infrastructure and our LDAP is closed. So it only allow a few external IP including Artifactory. If the user logged in and they use the newborn IP, LDAP refused the connection and that could be a word or message on the site. Any question on this one? No, >> of course I also recommended to the user to not go the way of manually [snorts] raising plugins. The the way to go is CD with exclusive mode which mean no human should be able to release a genkins plug-in. That's safer for everyone and easier for everyone. That's just a few option to change in a YL file in repository permission updater in the hosting request. Uh and there are many techniques to trigger the release only when you need it. So it's only playing with the labels of your pull request. That should be quite easy. Next topic uh open but stale that will stay with us. Um so the rate limit from docker it's not authentication. It's only CI geno. We haven't seen it since a few days, but it still happens from time to time, especially when we have a bunch of pull requests. Um, we evicted the problems related to our transparent proxy. It was working as expected, but sometimes Docker C uh was trying to deal with TLS while we don't. Not sure what the problem is. Seems like a word infrastructure or network error or a bug in Docker client. uh for now we keep it open to continue and documents. One suggestion uh we have an idea that Seran and I had will be to try to implement uh custom registry on all images on all the from directive of docker files. So we could optionally specify the ais docker mirror. That's an explicit mirror that you need to add a prefix in the name of your docker images on both tests bake files docker files. But that will mean instead of having to fall back to docker hub the docker engine will directly get the internal Amazon copy of the docker mirror which is not rate limited internally. That will be a solution. that require a bit of tooling. It's not transparent though. It's not like in Azure. Let's try to support custom registry to allow usage of AWS internal is here. Docker mirror. Any questions so far? >> Okay, next one. create sonar cloud token. We have that contributor. We started to build a pipeline library to allow CI geno to scan things from sonar cloud. CI genio is public. So we have a problem here of any token here should be should have should be really scoped because it's most probably already being bound. It's a public genkins instance. Um problem is the organization we have at center cloud an oss plan does not allow to have organization scoped token which mean we need a p if in order to do this so we don't have a solution yet the user asked uh sonar cloud if they can allow that feature for us on oss no answer so far the alternative we proposed that need to be review by the gensec team and the administrator of the genkins CI organization will be to create a doomi bot user with shared MFA owned by the genkins infra team. It should only have read only on the repository in GitHub and we will use that user to create a personal access token. So if that token is spawned the actions that the the attack the attackers would have will only be read only that will limit the coverage and would still allow the sonar cloud scanning to run for our plug-in and contributors. still a benefit. >> Are there any question on this one? >> And Jenkins CI admin to validate the technical user proposal using a P8 Ilcore. We've pinged the Adreion. He's still analyzing. I guess he has other priorities. So the ill score is still is still showing bad results. We might need to sync with Adrian because it's not only the ill score on plugins genkins IO but that's also on any genkins instance in the update plug-in list. So there might be improvement or things to disable in the upcoming releases. Waiting for Adriina for a full state analysis before we we check what we can improve in monitoring. Finally, support mirror selection in update center. So we have that chi China based user who complain about the only mirror for downloads genkins the tuna university sometimes blocks user outside the university network inside China leaving uh them to receive four or three errors when they updates their plugins which is quite the problem. problem we have is that we used to have mirrors in China but none of the mirror administrator answers and none of them provide us a way to scan the mirror so we cannot include them inside our infrastructure. We have disabled the tuna mirror and right now the user said it's okay for them because they are redirected to I think it's Taiwan based mirror which mean their ISP allows connection outside the great firewall of China. We haven't heard from any other users in China. So maybe we broke other user. We don't know. That's the only mirror we have there. We gave the user solution and we pushed forward for them. So they contact Alibaba or any other organization so they can get us a mirror at least the mirror that we can have for HTTPS and air sync for scans. So right now waiting for the user or the user to get us a sponsor or contact. We'll add back tuna. Uh so no okay question is should we add back tuna mirror? How much time do we wait? Because right now we are totally blind. We don't have a way to measure things from within China's network. My proposal is that we keep it disabled for one month and we see if people uh complains if no one complains we remove it because tuna administrator do not answer at all when we contact them. So either we have the bad contract contact address or they just don't care don't monitor or whatever. Is it worth running adding a mirror that is not able to act if something goes wrong? I believe without anyone complaining in China, I would prefer having less mirrors but having people happy when they complain. However, that's still worth finding a mirror. So, Tuna is disabled. Let's wait until I propose end of September. Yes, I'm going to >> before making decision as they don't answer to our emails. We should work on building a mirror in China ourselves. So the goal will be to find the right uh the right level of uh sponsoring. There are many VPS VPs inside China that we could use with a two CPU 2 GB machine on the right hard drive that will also allow us to use it as a mirror for update center providing the metadatas updates every 5 minutes inside China instead of uh Europe or US. So that's the current status. Are there any question on this one? >> No. >> Cool. Thanks. On the keep infrastructure up to date, the update center root CR rotation um waiting for the new CA to be uh released in genkins call. Once it be it will be released at least on the weekly line we'll have to work uh with the gen key security team on how to up what's will be the timeline for updating it inside the update center. So right now stale >> Windows 2019 what's the status survey? So we don't have any more um we the only remaining consumer of Windows 2019 image agent is windp uh that will take care of migrating to 2022 or 2025 vidally and uh I've removed the packer image [snorts] uh the next step will be to remove the Azure image. It's in Azure Gallery as they're not used anymore. It's uh done via Terraform or AWS MEI. We have to wait until WinP is migrated. Then we'll be able to delete them manually. Um another step will be to delete uh um Windows 2019 template in your controllers. Uh still waiting for WP. Uh to be sure that not deleting WP uh template the template used by WP for now. Uh my brain just reminded me uh that the other gallery can't be removed immediately. We still have 2019 agent labels in search CI. We need to clean up the agent template first. They are not I don't think they should be used or if if it is it's not because it's 2020 nine. Uh but we need to update. >> Yeah, we'll ask um be asked to think like I'm I'm not sure but I think the last time we mentioned those Windows templates they were not really using them. So to be yeah >> they they used Windows agent on search CI that's a fact. But do they care if it's 2019 or 25? I don't think so. I think they even already started to use 2025 without even knowing. That's I think was the conclusion in the add windows 25 15 of the 20 five uh template by default using them by default >> but we still need to work on the cleanup first before removing the other galleries because if we remove the gallery and we restart the certi controller it will make it will throw a bunch of errors on the other VM plug-in and that that might have impact I forgot about this one. >> No problem. I'll update my message in so >> cool. Thanks. Any question? >> No. >> Okay. Um, infrasane and maintainable category. So, you added this new one. We did not have time, but that will be the next step. We will try to merge the dock builds that require Windows 2022 base image and host into 2025 because it's possible in theory. >> Oh, we have already docker and docker agent using a windows 25 agent. Uh the only remaining is the docker is docker ssh agent. >> I have docker and docker agents project. >> Sorry the project opens. uh and really solution. >> Can can you confirm that I'm not mistaken that it works on both AWS EC2 and Azure virtual machines because that's not the same. >> Yeah. Yeah. Cool. And >> next we is docker SSH agent and if I remember correctly you were blocked on SSH agent. No, >> no, it's is a pull request is ready for resume since Yeah, it's ready. Okay, next steps will be done. Find if any other consumers either >> project or uh controller agent templates >> just to be sure what need to be cleaned up on infraside. We are also consumer somehow. We have the release using 2022 agent on AKS and the blocker was previously that 2025 wasn't available as AKS node. It's not the case since June. >> So we should be able to update the node pool and move to the 2025, right? >> Yes. >> Cool. Or we could get rid of the Kubernetes Windows nutpool and use as VMs. >> Yeah, but that's Yeah. Uh that's another help desk. That's not really Yeah. >> Yep. Depends on the timeline, right? We have to choose one or the other. Uh then we can get rid of Windows 2022 just in time for Windows 20 27 or 28. I don't know which one will be the next. Thanks. So we keep that issue in the current milestone stat generation. Uh J uh so you may you added May and June 2025 visible on stats genino. Thanks. Uh July in progress, August by J. Then next step automation up to March 2026 and we have to contact KK to resume March up to today from his basement machine. There is a whole issue about removing KK as a bus factor. I'll see uh I'll see with him if he's okay to check with the board if we can delegate that to someone else or if he wants to continue. But I haven't heard from Kiki since he left cloud B again. So, yep. I hope he's doing fine. Okay. Any questions so far on the work in progress? No issue to remove. We continue working on this. We have a few issues to tri edge. Thanks survey for keeping care. So the following issue I propose we add them to the new milestone that's part of the triage. I will update the labels after the meeting. So the CI genio persisting a new SM issue while cleaning up few word elements. We had we have the explanation from Mark that mean we can at least persist the change to avoid bite surprise because it was only done manually on the controller and now we can persist it as gaskas uh code that also mean we should do an audit on what g tool we are using across the infrastructure at least on infraci and eventually release we should get rid of git tool instead of the command line tool build website we mentioned it earlier so you started a prototype. Uh we will let you comment. We have mirror bit bump. We are sharing the work on this one. That's no major change, no bad surprise. A bit of Elm chart to add the new option uh that will help us to investigate when a mirror is disabled >> and allow also the view sort to be uh yeah that's what the new new option not sure if you will use it or not. That will be interesting to test. But yeah, uh clean up auto link references. I guess team is no >> team is okay to run the script. I have to prepare that. >> Okay. To run the cleanup script as admin of genkins ci um the new publicates cluster. Uh so migration to Sweden plus uh sponsored subscription. So the one last thing for the plan I need to do on this one for the public cluster I want to test if I can manually move a public IP object in Azure from one subscription to another. The reason if possible I will want to avoid changing our inbound IPs because many users that are trying to reach genkins infrastructure without even knowing they reach that cluster [gasps] they usually contact their network team and ask them to open the IPs on their firewall. If we change these public IPs that are the entry point I don't remember if it's one or two I think it's only one in Azure. I don't do dual IP stack load balancers. So that IP if if it changes uh then I will prefer keeping the same even if it means stopping the all the services for one hour. Uh I would prefer an outage but not having bunch of user having to again do the on the same dance. Hey hello can you open this new IP blah blah blah because that mean we will need to communicate on many lay many things just for changing an IP. So I need to run a test. I will create a duplicate uh set of IPs the same as we have for the public V6 and V4 but not allocated or allocated to a virtual machine. We don't really care. I will try to move them to the new subscription in the new location to see what happens. Depending on that we will have two slightly different plans. If it fails as usual we create the new IPs the new resource the new cluster the new pods and then we move we use a DNS move from one to the other. Each service is a CNAME to the cluster uh public DNS. So we can move service by service like we did in the past. It's easily to it's easy to test. We rewrite our EDC host locally on our machines and we can do a full testing. But that means changing IP. If my [snorts] technique works, I will still create public new public IPs that will allow independent testing and eventually doing blue green uh migrations. But when we will move everything, we will move the IPs manually and recreate the load balancer. So full outage but we will have a different IP. These are the two plans we have. I need to be sure the second plan is possible before asking people to choose otherwise I will just waste people time. Clear for everyone? Yes. And finally a bit of Terapform code. That one should be quick. That will unlock uh our ability to use recent Kubernetes providers on our Terraform project. >> Do we take uh resume show IP data which >> yes I think I think we should. >> Yeah, >> thanks for the question and I propose that we keep the NodeJS 2224 for the last service uplink uh for someone in the future. Yeah, we are almost in October. So, we will have either October fest or or any uh really highly motivated student with a good LLM that's dependency update in NodeJS. They will be able to do a bunch of test and updates, I'm sure. Okay, that's all for the the issue triage. I will take care of updating the labels, milestone stuff after the meeting and after my lunch. Now, are there questions and things you want to discuss before we close that meeting? Things to clarify, question, whatever. >> Good for me. >> Okay. So, Robson, did that meeting was useful for you? Did you was there specific part you were interested into? >> Very interesting. I learn uh for for you and Harin my first time and I prefer to listen. [laughter] >> Okay, no problem. Do not hesitate to join our elements matrix channel if you have question that is here to discuss asynchronously but you are welcome to join that's public meeting for public infrastructure. >> Thanks so much. >> Okay so if no other topics and no question I'm going to stop screen share. So one last chance for asking question otherwise I will stop recording. 5 second timeout. No more questions. Then I'm stopping reporting for people watching us. See you next week.