2026 07 28 Jenkins Infra Meeting

Hello everyone, welcome to the Jenkins weekly infrastructure team meeting. Today we are the 27th of July 2026. Around the virtual table, we have myself Damien Du Portal. Uh Jay, Rémi, and Mark Waite. Hervé is in holidays in Brittany enjoying the sun but not too hot. Lucky him. Uh okay. Uh announcements. So last week we were able to release weekly 2.574 without any issues, especially on the Docker image parts. So thanks Hervé for fixing the the pipeline. The rece- the magic recipe is uh parallelizing less thing and putting them back in sequence is faster and less prone to errors. Uh we could parallelize with the right amount of work but uh the patch that Hervé published is really cool. This week uh we released with almost no issues but someone here named Damien Du Portal, aka myself, uh forgot uh one one line of code somewhere. >> [snorts] >> And so that failed. So every detail is on the pull request. It's already fixed and merged so the problem won't happen next week. Thanks Mark because your nitpicking command about duplicating uh flags helped me to find the missing parts a few lines after this one. So when I looked uh at this and saw the error, I was like, "Oh, oh yeah, there is another occurrence." So yep, fixed. And already available. So Jay, whenever you will feel like you will have time, you can upgrade if not already done. I haven't checked yet. So thanks. Continuing the announcements. Sorry, last week I cancelled. I had network issue. Thanks folks. Thanks Mark for for being there. Uh I still published the note on milestone, so we had a snapshot of the cloud consumption at least. So, that gives us a trend over the weeks. A word on team capacity. He always still off. He will be back in the office in theory in the 3rd of August, but he will have limited availability on 345. The reason not announced yet, but publicly knowledge security advisory on the five for LTS and weekly release on the core. So, I will take care of it. Um Wednesday, so tomorrow, low availability for both chair and high. bility missing a t Are there other team capacity weekly release announcement related things? Uh priorities no need to sync on the priorities still the same. Um yep. So, no I don't think it's worth spending too much time here. We are in summer. And let's let's have a look at the calendar. Next infra team meeting happen next week on Tuesday, same time 4th of August. Next weekly release will happen on Wednesday, the day after on 5th August at the same time as the LTS 2.5 68.2. That will be a security advisory. Uh weekly as master branch has been locked, so it's public knowledge, but it's not announced yet on the security advisory mailing list. We have one to four credentials expiring in the upcoming three weeks. Uh we already treated a few last week. So, Jay, is that still okay for you to take care of these two upcoming credential because you are assigned to them. Okay, just wanted to challenge that. Cool. And don't forget to announce there is an NPM token expiring. There is an issue to create. So, it's it will be middle of August. Uh and there is one on Artifactory. Only me or an Artifactory administrator can renew it. So, I will take care of creating the issue. >> Uh I can take care of creating the issue for the NPM token renewal. >> You did the previous NPM token renewal, right? We gave you the access to NPMGS. Cool. I wasn't sure that that I wasn't sure so that's why that's why uh I didn't ask clearly. I was thinking, "Okay, if Jay doesn't have access, I will ask Hervé to take care of this uh to give you access and delegate it to you." So, if if we're cool, thanks. Okay, that's what I have on calendar announcement. Are there anything else you want to discuss on these topics? Everything good? I see you speaking Mark, but you're muted. >> Thank you. Next LTS is on the calendar. Good. All right. So, because we've got that next Okay, that's all what I was worried about. Thanks. >> Oh, yeah. Yeah, yeah, I got the the initial date from calendar. It was planned that day as as usual. It's the advisory that moved the weekly that the same day. Uh okay. So, of course that means advisory and LTS mean we will have a bunch of backports things to not forget about. And of course, I will be present in case something goes wrong. For instance, the the fix I pushed on the weekly release might need to be backported to the LTS line of the Docker container. I will take care of that later today, uh Mark. Thanks. Uh anything else? Okay, cloud budget. Unless you want to deep dive, we are in good shape everywhere. >> [laughter] >> That's the summary and that's really cool. Uh we can still move more workloads from Azure CDF to sponsored public gates, search CI controller. Everything is ready for that and we will work on it. Uh but taking it slowly, it's summer. We don't have RV, we are only two. Uh so losing a third person is always a pain. Digital Ocean has a slight increase, but nothing nothing insane. We are still good. AWS, we decreased a lot the consumption thanks to the move to the EC2 fit plugin. That works very well. Uh I think we can we can really think about uh moving that plugin as plugin of the month soon. Uh Mark, just for sharing, uh there's been, you know, something named stakeholder meeting where Jesse was absolutely shocked to hear that we were going to drop the EC2 plugin and stressed, like, "Oh, there might be paying users of some products that could be using EC2 and having the same problem." So I That's why I pointed him and he two or three weeks ago, I don't know if you saw, he he asked questions on the GitHub issue of the EC2 plugin. But the problem is way more complicated and now there's been a bunch of everything hidden somewhere. But yeah. It's not as if that's the kind of things we advertised one or two years ago, but we did. And Jesse said, "Oh, oh that's true. You did. I forgot about it and many person missed it." >> Yes. >> Right. Right. And and we got tired of telling people that this is a problem. So but I love that I love how EC2 the EC2 Fleet plugin is working and and I I appreciate that they had the ability to expand and contract the status page now. That was a big usability improvement for me when that arrived. So, thank you to that. >> Yes, absolutely. And I also And there is also one big reason because EC2 plugin has fixed things, so that improved the EC2 plugin for people still using it. However, the feature where we can mix heterogeneous instance families on a given fleet, that is really useful and delegating that to AWS cloud that knows way better than us orchestrating their own instances, clearly that has been uh really impressive in the cost management. So, we we are still really on a tight schedule for AWS. I don't remember when we I have to contact uh our friends. I say that I was going to contact them for asking for a renewal of credits. I just hope we will have uh yep, 6 months more like they did in the past 6 months. Uh but yeah, we will do. In any case, we have the Azure sponsored subscription where we can improve things. But uh uh Hervé started to work on the bomb part. I expect a bit of improvement here as well. If we could have 0.5 or even 1K less per month, that would be a great improvement. But in any case, that will be worth the effort. JFrog, JFrog as usual. Yes, we consume No, they never answer for solution. Yes, we don't have the right to claim support, but yes, they still sponsor us, so it's a kind of status quo. That's engineering at its finest. It's not the best, it's not the worst. We are really grateful for the sponsoring, uh but it also is limiting us on some areas. So, let's continue like this unless they disagree. And Algolia is performing and faring way better. So, thanks for them. Looks like we are not hit by LLM anymore. So, that's good. These are good news. That's all for me for the cloud part. Are there questions, thing you want to live? Cool. So, let's move on the milestone. Uh we were about to complete the following support tasks. Uh first, for one of the GSoC project, they requested for a GitHub page. So, no task done in the end except pointing them to the right documentation, but that was part of the GSoC process. The mentee, the person working on the project had to ask question on the right issue tracker, get the answer, and do something from that. Uh they have access and permission, and Chris was able to to enable and confirm it was working. So, good work. Let's continue. Uh thanks, uh Mr. Saylister, for helping the Jenkins project for 1 year. They gave us a Germany-located mirror for the past 13 months, but that mirror is being deprovisioned. So, we removed it from our system. We are really grateful. They still provide an India-located mirror, so thanks for that. Uh there were issues related to CD releases of many plugins. Uh so, one was an old one. Now, we are running RPU every 2 hours instead of every 3 hours, since the expiry time is 3 hours for a token. That let us 1 hour to have an agent virtual machine spawned in trusted CI, the build to run, the build to process all the API and update all tokens. So, we have 1 hour, which is way better than being just just in time. The rest are mostly CD related thing, no infrastructure issues. On the infrastructure issues though, the plugin site website was failing because spot instances on CI and Azure capacity issue for the production deployment. Uh so, commented and fixed. We still have improvement. Uh I believe having a pipeline library that takes care of spinning up node and retrying if the node fails to non-spot will help on the CI part. And also, we could have improvement on the Azure part. Uh retrying other instance families. One of the main thing is that our websites are scattered and have different ways. We have most of them using virtual machines for two reasons. Some are using Docker to build the websites, which looks good for contributors, but when you want to deploy to production, just a few node commands should should be sufficient. So, that's Jenkins I am thinking about. Uh we should be able to use container instead. The memory footprint and the time to get something will be better because Kubernetes agent spawning is better than in the past. On at least on Amazon. Uh also, Gatsby, which is a framework we use for the website, requires at least 6 GB of memory to build a less than 100 MB website. That looks insane for me, but maybe I'm too old now. So, we need to have virtual machine to be sure that it works. So, yes, spot instances. Any question on the support task we were able to fix? >> No, feels good. We made progress. We did. >> Thanks. On keeping the infrastructure up to date, we had two credentials. They have been renewed. Nothing else to say unless you have questions. Okay, on the keep infra sane and maintainable. So, Jay, I was able to fix the infrastructure part for search CI. And that was really easy to add the monitoring for search CI acceptance test based on the framework you built one or two months ago. So, congratulation. It took me less than 1 hour to do everything from end to end. That was really smooth. On the infrastructure part, the the I realized that we don't need to create one specific endpoint for the storage which lives in US East while search CI agents are in Sweden data center of Azure. Azure provides what we call service endpoints at the subnet level. On the subnet, you specify what kind of managed service by Microsoft such as storage or AKS or any others you can reach. And in fact, we were using microsoft.storage endpoint. And they also provide a global global.storage endpoint. You have to change. It's one or the other. They are mutually exclusive. The global storage allows you to reach storage endpoints which are on other data centers. Of course, that cost us a few bucks. But here we are publishing JSON files on each build. That's good enough. So, after fixing that storage endpoints, everything else was really smooth. That infrastructure part is important because we will need it for migrating the current public cluster. And eventually all of our infrastructure out of US East because US East for Azure and AWS is always out of capacity every week. >> [clears throat] >> So if we want our service to operate smoothly, the recommendation on both Azure and Amazon would be to move to other region including Sweden. Right, looks like they don't pay a lot for uh cooling the servers. Of course that's Sweden, right? And they have plenty of space available and plenty of machines available. So yep, since search CI works smoothly here, maybe it could be interesting for us to move the infrastructure. Knowing that we can reach cross location if we even if we pay a bit means we can have by part migration. >> Mhm. >> Um thanks Mark for taking care of that um uh close as not planned your up uh issue. >> So and I have had one on previously closed just to share so that the two of you are aware. Um we had long ago a case where ci.jenkins.io would sometimes lose track of its date stamp on some jobs and set them to zero. Well, it turns out and a Jenkins user found a case that causes exactly that repeatably. However, we weren't affected by it. So the flaky test handler plugin has a bug. The bug corrupts build.xml files in in certain cases and it's repeatable. I've I've submitted a duplication case, but we don't have flaky test handler installed anywhere on ci.jenkins.io. So that's not the problem we saw and there's another user, but at least one instance of that has been found. >> Cool. That which maps to the problem we had that was the same DataDog plugin when being upgraded to a new breaking version because they did not test it back in enough >> was corrupting the build.xml files. >> E- E- Exactly. So, the So, the the the bad behavior for flaky test handler is the same bad behavior the DataDog plugin had. Right? So, So, it's corrupted build.xml file is a real thing. Yeah. >> Which is interesting because that means it can happen on other plugins. So, that could be a feature, if possible, that will help taking care of this build.xml file such as I don't know, repopulate date time or something. There could be something in the core. >> Right. >> But, not sure about the complexity of such a feature, but user-facing at least a feature that say, "Hey, sweep everything, text time, and rebuild them." Okay. Cool. Thanks, Mark, for the sharing that. Yeah, I remember that comment. Okay. So, moving on work in progress unless you have questions. Cool. Um in the category keep infrastructure sane and maintainable. Uh Jay, you have two issues that you are working on. Can I let you the mic to explain the status and the the problem that you want we want to solve with this issue? >> the first issue is about a problem we identified while working on the issue below. So, we noticed that the update key pipelines for most of our Docker jobs, they have a properties called in the in the pipeline script. So, that pipeline that properties block is redundant as it's being overwritten by the internal properties block used by the update free pipeline library. So, right now we the team and I we identify the issue and we agreed that we would want to standardize it to only have one properties call and which is the internal update free pipeline library call. So, yeah, we'll we'll have to get rid of the redundant one in all our repos. And the issue below is we noticed that the daily cron wasn't actually firing in most of our repos. Again, from the same issue that it was being overridden by the property the internal properties call. So, the work has been done there. The pull requests are in review and it should be closed shortly. And the daily cron should be firing as intended. Yeah, that's it. >> No question from me. Mark, was it clear? >> Clear for me as well. Thank you. Strange strange strange and bizarre bug. Thanks very much for fixing. >> That's because we haven't listened to Jesse years ago when we built the pipeline libraries and he said one should not have properties in both pipelines and libraries. He's he prefer having it on pipelines, but for us it's too scattered. We have too much occurrences, but we should have step on only define them inside. We try to merge and it doesn't behave well. Thanks. On the pipeline library, that's what I mentioned earlier. I haven't started yet, but we should provide an infra.node with timeout and retry function at least for the websites. Uh that's scoped to all these machines. So, we would have no GS built properly based on are you running on CI Jenkins CI or infra CI and retry if it if you have spot instances and you detect that spot was reclaimed. That will help getting build up to completion no matter what. And it will also help to not fail the production deployment if something happen. Not started yet. But all prerequisites met. For next milestone. So, we keeping in milestone because that's not a lot of code. I also want to experiment a bit with cloud code to help me on that task, especially the pattern matching to detect what could be the common properties between the seven repositories. That's what I talked to you Jay earlier today. And what are differences? I want to see if it's as efficient as what I have in my head. Next issue. Uh we have two problem with JFrog. One that they're still avoiding. Uh Hervé asked if Hervé and Jay could open issues on their support thing. We are not entitled to automatic support. Yes, but they still we still have a contact email when when they break everything on the whole, we can contact them. They don't guarantee any time for the things to be back. They pay for it for us, so that's okay. However, that could be interesting if Hervé or Jay could open issue when we have a running outage. So, there's been long discussion for nothing, so I will ping them again. However, uh so, first support adding Hervé and Jay is still not answered. But uh being done again. There is still an action for us. We still need to add Jay in my JFrog portal. Because if I remember correctly, Jay, you still don't have access to my JFrog, so you cannot check the consumption at least on the weekly meetings. So, there is still an actionable for me here. Any question? Jay, can you send me privately the email you want to use for this one? That should be the usual, but I keep forgetting about it. So, having your written one asynchronously that will help me. Thanks. >> I'll do it right away. >> Uh next issue. Uh moving public gates cluster. Uh so, issue to fill. That's on me. Now that we have fixed the Sweden US East service endpoints challenge within Azure. So, I was waiting for fixing the cert-ci thing. And blocked by cert-ci controller migration to sponsored subscription. So, filling the content of the plan is still thing I can start working on. So, we have content that I can give up to the rest of the team. But the operations on this one are blocked by finishing the cert-ci controller migration out of the CDF subscription. Finally, we have the So, Airwave found a way to enable tags on AWS on the tracking cost by tags. He enabled a few ones. I've added others. We We start to see patterns. But it's still really inefficient. Everything has been done in these usual clouds for not helping you track what are the sources of your cloud billing, of course. So we can already compare the the costs per kind of agents. Uh the EC2 fleets, for instance, we can have cost per EC2 fleets. That include a bit of bandwidth, but not totally. So that could be interesting for CI agent in say you to track the costs of a given builds or a given pipeline at least. For instance, we could have an order of magnitude of the consumption of the bomb, at least on the compute part. Of course, we don't have the cost of the bandwidth. So we are back to to beginning like uh if we consume more or less machines or mean machines of minutes machines or hours of machines, that could look interesting, but if it triple the cost of the bandwidth of what we download at same time, that could absolutely defeat all of our efforts. So it's a first step, at least an indication on how can we make the bomb build a bit less constraining on the resources. But we don't have anything, and that's a cool community Hervé. We still still don't have anything for Jenkins to say, "Hey, I have that build, and that build will push tags to every underlying hidden resources behind. It's cloud costs, it's whatever." So then someone could use telemetry, tags, whatever to say, "Oh, that build took time and cost that much." So maybe telemetry could be the way to to get it, but it's cloud by cloud, so that's yeah, a tiny walk. Still something interesting to propose to the community. We have more tags tracked, but need more to really have a real real trend. Any question on this one? I will have a few more tags that I found and close it because we know it works. So yeah, we don't have any other action able here and we'll see if it helps us in the future. On the keep infrastructure up to date topic. So we have two credentials. Jay, you are going to take care of this too. Nothing to add here. No questions. Um, on the update center root CA rotation, meaning generating a new certificate authority that will have a lifetime of 10 years. I'm the only one with access to the key. So I'm the only one right now as infra officer. We need to find someone else. That discussion I believe should go to the board. I was proposing the security officer, but I don't know if Vadek plans to be there at least for the upcoming 5 years. The challenge here is once you have access to the key, we cannot go back. We haven't implemented a technique that will say how it's a one-time thing and at a given time in So there are solutions, but they are really complex to implement. So we still get the same trade-off of hey, let's have as less people as possible having access to the key of the CA. So the CA is public. I've generated it because that means contributors or people without access, that's specific access, could still use the certificate, put it inside Jenkins as a secondary certificate to validate data that come from update centers. Which mean we can start to track the period where both certificate will be accepted. And then we will work on the next steps. But as soon as we get this, as soon we will have more Jenkins version able to handle the changes. Because if they are not, like the LTS that will happen next week, when we will change the certificate authority on update center, that mean every version of Jenkins up to today won't be able to update or to get plugins. That should happen in 2 years, but we are still late. Last time there were 3 years period. So I've put the thing. It's uh I guess Daniel is the next step is the the the link with the core thing. I will ping him next week if I don't see activity from him, and I will try to either find get help or do the pull request myself. Maybe it's an easy one. Are there any questions? Important note, we will experiment what happen if we change the metadata of that cert CA. Because we did it. It's not Kohsuke anymore. It's the Jenkins infra team with our team email, so it's not Kohsuke personal email anymore since Kohsuke is not really present in the project now. Yes, we have to plan for 10 years. Finally, dropping Windows 2019. Trust me, if I could have done that one or two weeks ago, I would have and I would have prevented me wasting time. But yeah, uh we have a SIG meeting later. I will uh officially bring the topic to the SIG uh platform. The goal, I will want to drop Windows 2019 before the end of summer. As soon as we can, that's better. But, of course, we have to communicate to user at least under change looks. >> Yes, but I think it's the right choice. Absolutely, considering the the awful experiences we've had. I can't imagine causing that for our users. They they don't deserve it. Let's Let's push them off of it just for their own benefit. >> Abs- That's absolutely all I would have put. That's like, we are protecting We are protecting you. Trust us. We suffered the pain already for for the war back. So, yep. Uh, I say G later. to uh, bring the the big officially. Okay. Finally, two support tasks still open. Um, one is still CD release failing. I thought that one was fixed. I guess, uh, okay. I I won't bother you. I will check, uh, but I guess it's closable easily. The other one, uh, I will need help, Mark, because I did not have enough time. I spent my time on Windows 2019 instead. >> Mhm. >> The Selenium version on the build pipeline plugin does not supports, uh, recent version of Chrome of Chrome or Chromium. When I said recent, I mean it's since the past 18 months. So, I guess there is something to update. I checked the dependabot or renovabot, uh, Selenium bump, but that's just a patch and it still have, uh, the same error message. >> Yeah, I think Bill, you you have done the heroics there. Thank you. That one is my responsibility. You can You can send it to me if you'd like and because that plugin needs to somehow be changed to do what acceptance test harness does or what other tools that use Selenium do. It is alone in this failure in the top 250 plugins. And when we have something that's alone like that, we need to make it not alone by making it the same as the other people as the other as the other components. >> Okay. >> So, whether that's switch to another another test tool, whether it's kill the test, um last time I I I tried to kill the test, Basil correctly told me, "Mark, that's really bad. You shouldn't kill tests." >> [laughter] >> But but I'm I'm tempted, right? Because because I'm not I'm not actively interested in maintaining it. So, we'll we'll uh but that's clearly mine. You've done heroics on that. Thank you, and I should take it the rest of the way. Let's not bother you further with it. >> I I've played a bit with Playwright. With or without its MCP tool sets. Uh so, as a human and with the help of whatever automated system, um it it works quite well. Uh looks like Microsoft has put their effort somewhere, at least, because Playwright is backed by Microsoft. Um they update it quite often, and it works with a bunch of web browsers. And I I've been impressed compared to uh let's say my Stockholm syndrome with Selenium from years ago. >> Yeah, yeah, you and me both. The post-traumatic stress from Selenium experiences is still there, but >> Yep. >> the the the compromise is lots of other places use Selenium. There are some though that use Playwright now inside the Jenkins world. So, in Jenkins plugins, I think maybe even pipeline graph view. So, so there've been some there've been some nice moves towards Playwright. So, it's at least worth consi- >> And and uh thanks for opening that issue because no matter what our preferences are, it takes time to migrate to to change tool. And that cooked an issue that we we missed in from the infrastructure. We broke things for good for good reasons, but we broke them. So now at least the Chrome binary is used. So um the issue from infrastructure is closable. However, I let you drive the the way you want it, Mark. You can close it or keep it open for tracking and >> go ahead and let's go ahead and close it because the as you said, the infra tasks are done. The plugin task should be tracked in the plugins own issue could be tracked. That's not that's not something. >> on the other hand, since the change come from the infrastructure, that will be worth having closing the issue with a workable solution. Okay. If other plugins are have the same problem. That's why let's keep it open, assign it to you, and then you will report here from the plugin fixes if that's okay. Okay, cool. Thanks. We don't have other work in progress issue unless someone raise their hand. And on the triage, we only have two issues with triage that I'm proposing for the upcoming milestone. There's been a request for GSoC to enable preview of plugin modernizer stats. Um J, maybe I will ask you for help on this one, but if it happens, that won't be on uh starting Friday or eventually Monday next week. So no worries, no need to check. I will explicitly ping you in elements where if I need you on this one. The other one I'm going to uh we will have search CI now that the security team has finished their changes. I will should be able to move that controller. I wanted to do it last week, but yep, it was blocked. Uh now it shouldn't be. So I will move it the sponsored subscription, so every search CI things will be running on Sweden on sponsored credits. Uh the rest for me are not needed unless someone has uh something Oh, Mark, maybe I could get your help on this one. >> Oh, yeah, that one's a That one's complicated. That's been there for years. So. >> Yep. Um I believe that the solution that James proposed is not that bad because it's Now, it's only on the pom around used by only core ATH and the third one. It's not present on the plugin pom. So, the impact is uh less more uh Let's say it's straighter. Is that the right word? The the scope is is smaller than it used to be in the past. And looks like there is solution either drop it and use a new XML processing. I believe that one Uh the thing is the work to do is not that much. It's more about the way to communicate it to developers and reach a consensus. I think reaching the consensus will be the harder part here. >> Mm. Okay. >> And James told me about it privately saying it should be way easier than in the past and less scarier. Uh he said I could even propose the subject, but my my call for help is because I'm missing uh time and availability for doing that. >> Yeah, this this one Um I'm I'm scared of this particular area of things, but I'm happy to take it because I'm I'm I'm better suited in particular because I'd much rather have you and Hervé and Jay doing other things. I can I can squander time on this one. We've had so many good things from you on other areas. Let's not waste your time on this one. >> And it wouldn't be wasting. I I like I like this kind of task. However, I'm missing time and I think you will be way more efficient because there are some part of the XML processing that I I'm not really efficient to do. And that requires someone with the right instinct and the habits of working on these parts. Uh so, I'm not adding it in the milestone, but I'm pointing it to you for your own personal backlog just in case if you have some time. If you don't, that's not a problem. We're not blocked anymore by this. Okay, that's all for me. Uh the decrease bomb cost by splitting tasks more efficiently is on hold for until Herve is back. I don't have anything else. Do you folks? >> Nothing else for me. So So, the after we've stopped recording, I've got a topic. >> Okay. So, I'm going to stop recording. So, for people watching us, see you next week. Bye-bye.