2026 09 01 Jenkins Infra Meeting

Hello everyone, welcome to the Genkins infrastructure team meeting. We are the first day of September 2026 and around the virtual table we have myself portal and uh Jay is off. He will most probably not join. We might have Mark later eventually. Maybe if he's uh awaken and available. Uh but oh no, Mark. No, Mark is back from Alaska. So Mark won't be there. >> [gasps] >> Okay, so last week uh Genkins weekly release 2.579 was released with no issue, no problem. No fast error, no docub rate limits, no internal uh infrastructure issue. Uh this week we will have a weekly release but only tomorrow due to security with the at the same time as the LTS. So no weekly 2.580 today. uh team capacity. So and hi we are back uh to the office and Jay and Mark are back tomorrow. Uh we have the links on the current priorities they haven't changed. Free topic worth mentioning as a reminder the same as the past weeks. Uh we have an upcoming patch of mirror bits but mirror bits doesn't follow semantic versioning. So that patch could have breaking changes at least for us. Engineix ingress retirement it retired since March. We need to find solution that most probably will be a subject for us next trimester. And Kubernetes upcoming 1.35 uh upgrade 1.37 is available. Uh usually we have free updates a year. So that will be the second of this year. Finally, a word on sponsoring NUOPS. The status hasn't changed. I did not have time to check further, but they built a plug-in. Um uh I'm waiting for having something that works at least one time in their environment before sharing with uh air and other that could be interesting. Uh no no sense in uh making you waste your time until they have something that's that is a PC that works. Right now I've asked them a few elements including implementing SSH agent protocol because it's only inbound agent. They they use the Kubernetes plug-in as a base and it doesn't have SSH agent of course but that loop promising. Any question on announcements? >> No. Well, good. >> Cool. Uh let's have a quick look at the calendar. We have uh a meeting next week. Uh I will raise during that next meeting uh question about the timing because it will be harder for me to attend with this timeline since Mark uh it's um moving it earlier will be hard for Mark to attend moving it later will be hard for Jay to attend. So maybe we will need to have a um uh even even weeks uh a earlier and odd weeks later or that kind of solution. But let's discuss it when everyone will be there. Uh next week is tomorrow and on the 8th September next Tuesday we will have 2.581 as usual. Next LTS is tomorrow. The3 is part of the incoming security advisory. Uh thanks every for taking care of the back ports. So it's the second time I missed the back ports. So proposal is that we have starting from next LTS each time we discuss uh we need to plan the back port. >> Yeah. Having that line in meetings and I'll update the release checklist to >> make it more uh foolproof that uh we are we have our back port ready. >> Yep. Um the idea will be to have the backport one week before any LTS security advisory or not whether or not the the security release advisor or whatever leader sense the issue. Uh the reason is because if it's a security advisory they used uh specified planning. We know there will be a an LTS. We didn't know until this weekend that it will be a security advisory or at least not publicly. We cannot risk even if we know earlier in advance, we cannot risk saying oh it's a backport for security if it's not published yet. >> So what do you think about the 7day backport before any LTS? >> That's an abit that's always the same not change whether it's security or not and we create that a bit. >> Yeah. and you proposed a few areas where we could uh implement this to avoid. So thanks for this idea survey and let's try to do better last week. So again my fault so let's improve there. So tomorrow next security release and we have a credential expiration uh tomorrow no in two days sorry three uh the terraform one uh I'm not sure if I I will create the issue I'm not sure who will take care of that uh I might need help because it's it's one that can be annoying >> okay >> um let's discuss that uh privately but yes I will create the issue and add it to the milestone on cloud budgets. Just a reminder, it's only August because we are the first day of September. So checking the costs that are not aggregated on each cloud provider yet uh makes no sense. Azure CDF is good less than 2.2. We are below our threshold. We can do better, but we are good. A big increase in Azure sponsor subscription though. So the the the deadline for depleted credits move from September to August. Uh but search CI required to build many things. So we have a big security advisory and that will be the case more and more. We can think about optimization but for now it's absolutely fine. So let's move on. But that explain the pick that shouldn't happen uh next next week. If I remove search CI and replace by the July 3CI consumption, we are at 5.4 digital. Uh we see that the rate limit uh effort went well. Uh we might have a slight increase next month due to the statistic work on census because we've added um a disk uh we incre I've increased the disk from 100 to 500 gigabytes. But that should be fine. And that should be like 20 bucks monthly. So that's absolutely in the lines and June. No, that's Oh, yeah. Yeah. Uh AWS, so congratulation. Looks like your work on the bomb builds had a huge effect. Let's confirm next month or in the upcoming weeks. But yep, we have consume in August 20% less. So even with the August holidays effects, we still had a bunch of builds in CI geno and it looks like the the new bomb uh particularly the pull requests optimizations and test replies uh are incre are improving the situation. I haven't updated the current rate. So yeah, I think we should aim for January, eventually February, but we're fine. I'm resuming the work on the renewal. Uh I need to send email. I did not have time last week. Grog used a lot of consumption. Still no answer from them. So I will consider uh Yep. Let's try to report monthly and and we'll see. Alia, we are fine. Nothing else. Any question on the blood budget? >> No, I don't. >> Cool. Okay, let's have a look at the milestone on the task we were able to finish first on the category. So since last week in the keep infrastructure up to date the back ports array there were can you confirm there were only a few backports related to the GDK patches right? Yeah, on uh on release updating the docker packaging and agent for Windows uh set where including critical patch patch uh nothing on packaging as there was no new commit and so useful base image upgrade. >> Thanks. Um there were a few fastly token expiring last week. So trusted and release uh were done. Uh I messed up the Cloudflare Air R2. I thought it was on 28 and not 27. So I woke up on update center uh not being updated updated. So the behavior was interesting. The poor archives.jenkins.io you digital ascent machine was handling all traffic by default as fallback. Uh the impact for end user where uh I saw less than 1% of timeout request on the logs. Uh and the update center metadata wasn't updated for 6 hours. >> Could have been worse. >> Yep. Uh being alone working. Uh yeah, it's not not easy during August. Uh there were two fastly tokens also updated the week before. Just one note, we have one in infrasi for purging these free websites, but that credential has never been used and the pipelines are not it's not implemented. The CDN push is not implemented on the free website and never has been. Isn't it in trusted that this p is done? >> Yes. On trusted on release on trusted it's done on >> Okay. So yeah the token not used. Okay. >> Yeah. >> Uh also I saw that fastly is having word issues. So fastly had the 503 errors that we had or 504. they were uh transient and I saw people complaining that the dashboard for the API token that report if it was used and when was it used last time sometimes it take two days before being updated. >> Okay, >> so uh it took me some time because I updated some and I run the bill with the new token with the old token removed and everything went fine and the dashboard say token never used and I was like how is that magic can happen just so you know. thing >> on the support part plugin score had an issue uh and Adrian required log so thanks survey for taking care of this one >> yeah I've yeah >> yep go away >> uh so um it appears that uh it did not update since a long time really uh I've push Adrian made a fix for the error that was in this first log. Uh but after deployment uh I encountered 403 errors whenever I tried to search something on the front end. I've send him send him the new logs about this error. But uh yeah not so he got it uh his log and uh I told him that in the fixed pair he pushed so it will be go it will going in the other desk and about the outdated data. >> Cool thanks so we will focus. I thought that the runs were monitored. I I remember a pro. So maybe it's not. Maybe it's just the the front end page and >> could check. >> Not now. But yeah, uh that would be worth checking. >> Yeah. >> Do do you want to check? >> Yeah. >> Okay. I don't know what that mean. >> Don't say to open an LBS issue saying uh um audit phs monitoring and we can close it if it's already done but yeah thanks so close the j responding slowly issue. Um, we had the same problem. I think it was Saturday morning. We saw a bunch of pagod duty errors uh due to Jira being slow. Uh, I've received the logs, the access log from last time from the Sunday that create uh that uh pushed mark to create that issue. I will check this uh because yep with with the gen team eventually they might have tools to see if we see wet patterns. Um could be interesting to see if they can apply a rate limit uh on the front end uh Apache engine they have and that's something we should start to have as well. I I saw an interesting uh article about the kernel administrator speaking about coolers and how it's >> [snorts] >> uh not a loss battle but it's really difficult. They implemented an but they are now increasing the proof of work level and it's now uh penalizing users on phones that can can't access. So yeah, >> that's why the rate limit per IP is a good enough layer even in the age of FMS. Uh it will penalize the user using the same outbound IP. But that's another problem that means they are either behind a big corporate firewall and that's their organizational problem if they don't monitor their users uh or they are behind a country firewall. That's another kind of issue. uh but in any case uh applying request rate limits should be something we should study on many of our services. We already did in bond with level for archive genkins because it's a download service but uh um will be worth checking the the amount of requests per IP. Of course we should be able to open our own outbound IPs. Yeah, but as they said in that article, more and more those creator are using residential proxy and so you get a bunch of we have so many different IP. It's difficult. It's difficult but still it improved because uh last time the CDF checked and last time we checked our logs on CI genu we had less than um we had something like 70% of the request were like 10 IPS. >> Okay. Okay. >> So yes they will improve their ability but that will make more and more annoying for them. I know LLM will ease this because they can scan for residential uh security flows and get there but yeah I mean >> I Y go ahead. >> No, I've posted the article I was speaking about in Jenkins Safra. It's an interesting reading. >> Mhm. >> They went into all this level and yeah, but yeah, in the CA in our case that that will be a good enough solution. That's not perfect, but we don't have the ability to >> uh to push more and >> especially for J. >> Anyway, moving on. So, it was closed on CCI. I was able to set up CCI with the wall common pattern that we have. Um so, usually when we move across subscription or data centers, each of our controller has different components at least controller and a bunch of ephemeral agents at least. uh we used to have a non-clean pattern where some common resources were either on controller side or agent side. You move one of these components and you break everything. So instead let's create a generic components called commands where you have usually the private links the access to the the private endpoints to the registry and other resources like this one. So done for search CI following what we have built on infra and trusted. Uh now search CI is using uh our docker internal mirror. So if we pull images from search CI that wasn't a lot of use cases but now it has automatic proxy and caching that was a requirement for both uh moving the CCI controller on sponsored which is not the case yet and we will have to work on it and the future public cluster move to subscription and Sweden location. Now work in progress. We have had a we have a new mirror, right? What's you added it but it was >> as down. I don't know if m bits can give us more information about why it's it has down in the last scan I've made. Uh it they did not make uh an initial sync before. I've told them that they've made one. We now have the scan showing that it contains file but it's still shown as down. >> Okay. Uh >> and there is a date when you click on the second link uh in the desk this one. >> Uh it's shown as uh with a date of uh standard uh not here maybe somewhere else. Maybe in my um in my yeah in no in my comments of the mirror bit list output we can see that the date is >> yeah default one yeah that's a common bug usually I try to I delete it and add it back >> okay I try that there are changes that will explain um mirror bits with the new patch version 0.6.2 two, we should have a reason why the mirror is down now. >> Uh otherwise, you have to search for the logs in data dog. >> Yeah. Okay. >> Uh while I'm there, can you please update the runbook? >> I've done it. I've updated it. There is a link to that. Uh look in the link uh reference uh after uh just before my first comment where I said uh I I did it. Yeah, go a bit uh up you GitHub Notch is not showing you the link but uh there is a link to the runbook a reference a GitHub reference >> if I log in. [clears throat] Okay, cool. Uh it's because I wasn't logged in on >> uh so can you please uh follow the instruction to update the runbook? It's not >> it's not complete. Okay. I Yeah. Okay. I'll take >> um you you have um um backup of the configuration to a local ML file to make. >> Okay. >> Maybe the runbook is not complete and it's read me. Double check this. >> But we the backup is better because the backup can be reused to restore data later. >> Yeah, maybe we can keep only the backup and Okay, I'll take a look. >> Please do both. There are reason why we have these two informations because we don't have the same information in both. >> Okay. >> Uh thanks for taking care and thanks for them. Um don't hesitate to remove the mirror and add it back. >> Yeah, fine. Yeah. uh >> and then trigger a rollout restart of the mirror bit service just to be sure that the HS system doesn't fail to synchronize Let me know. >> Uh okay, you can put it here. >> Okay. Um so uh note with time out and retry function for websites. I had a P for implementing a common reusable function for all websites. Uh I played with LLM because that's an easy one. Uh low risk. Uh it's working locally with my local controller. I have local controller for running tests because unit test is not enough for a new thing especially when it's about agent allocation. Um the main challenge I'm still not sure is the timeout. Depending on whether you have node and then time out or time out and nodes it has different behaviors on pipeline. So I just need to to summarize everything on on an issue and see uh what choice we could have. In any case we need a timeout but the behavior is slightly different. >> Okay. >> Uh more on this later. That's a low priority issue. That's why uh I only played locally. CCI controller uh resuming work need a task list updated. So it's only one virtual machine with one disk. Uh no, but yeah, we still need a few few changes. I'm not sure if I should take this issue or you or J. Right now the task list update is for me. Then we'll discuss and decide who can based on their workload. Is that okay for you? >> Yes. >> Update then team discussion on who takes care of it. This who takes care of it. Uh statistics. So my parts until Friday I was able to set up posgrql database load the data from Andrew from the usage machine and set up the SSH access restricted access from census to retrieve data from usage. Uh the database was up and running within disk and then I handed over to you. Can you summarize? >> Yes. So yesterday I've um clone the Jenkins new go binary uh into senses in op folder. I've built it. I had to uh install make I will open a new nuet to add make in the requirement of census agent. >> Mhm. >> Uh sensus machine. Um I also create a pjass uh file in a root folder with search mode 600. It's for now it's just okay we will as you said my goal was to get a binary uh importing and reporting for now uh as we have and we will uh refine that. So I got the import of air sync usage stats uh files from usage to sensus with sync. Yes. And uh launch successfully uh cm port uh then the reports and I committed the results to statistics. So we now have the May 2025 stats [snorts] and I'm uh continuing. So uh I've started the import of I've made the import of June 2025 uh files. So the import of one full month take about 8 hours and uh the generation of the report for a month a complete month for now just May take about four and a half hour. So ju just um stopping just to be sure I have full context. One full month of logs imported in PG. These logs are locals or does it includes the sync copy from usage to census as well? >> Uh no the sync is not taking longer. The sync of one month is about three gigabyte. One month copy with air sync usage is >> 3. >> So that's less than one hour, right? >> Yeah. With a big peak. Yeah, it's it's it's short. It's a few minutes. I think it's 5 minutes. It's copy about between 40 and 60 megabytes per per second. So it's quick. The full months of log imported in with the jen the import uh command of the goinary takes about 8 hours and the generation as a report for one month takes about 4 and a half hour. [snorts] Uh I also noticed and I've I've made the sync up until March [snorts] 26 because there is no new usage stats. Uh afterward the last usage stats we have in usage are from uh the 10th uh 20th of March 26. So we will be able to made to push to create a new report and up into this state and yeah we [clears throat] need to contact Okay. Okay. >> Yeah. Okay. Cool. But uh yeah that is that will be a few work to have. Uh cool that led us enough to put the automation in place. Cool. Um have you checked the disk? >> Okay. Yeah. >> Disk usage to be checked. Uh I can tell you uh the reports generated how much I've I've uh I've got the public data on my >> Oh, don't bother. It's not it's not the CSV. It's more um the size of all logs and the pages. >> Yeah, I haven't looked at how much the database increased with those new >> is this usage to be checked. That's okay. Uh the the reason is to be sure. Do we need to put a garbage collect at the moment in time? >> Yeah. >> And because maybe that could be interesting to have the same logs on two machines on two discs. Yeah. That's the same cloud provider. So we can do better. But >> because the garbage collect will be sending the things in Azure. So we would have the data in two different cloud providers. At first I uh I've I think both May and June in the same folder >> with ash in census >> but uh my first run my first import run was not in the screen session and took a long time. I wasn't sure if I could about it. So I've let it I just deleted the June uh files and let it run on the May. And for the next one I thinking them in their dedicated folder. There is one folder per month. >> Okay. >> And same for the report. So we can work uh I think on that. >> Okay. Be careful that the the content of the logs the daytime of the logs content and the name of the file might be different months. You have an edge effect. You can file rotated on the 1 of September that will have August logs in them >> for the report. There are two parameters to [snorts] specify the last month and last year >> to generate. >> I'm I'm speaking about >> by one. >> Yeah. >> Yeah. you're speaking about the logs but the generation I I did not add any June u result and in my reports I don't have the the may result have been named June and I have to push a fix commit to just renames the last the last one >> if you can don't say to tag Andrew and check with him just to be sure there isn't >> there isn't a reason for this I remember weird things that made sense after his explanation and team uh because I remember Tim saying I need to fix this and then discussing with Andrew and then oh that's why >> so be care be careful on this one >> um okay have you reported the command and things somewhere >> book in the issue okay >> I I will in the issue I will report what I've done >> that's really important especially if you need to hand over the automation to Jay to me or someone else >> yeah yeah for now I'm I will report. >> Thanks. Need to report on issue runbook whatever. Okay. And remove the PG pass file. >> Okay. Uh useful discuss later in >> it's a it's a credential. It's store encrypted or not or it's not stored. >> Yeah. uh I was unsure how to pass the credential the password to >> interact >> to the binary uh and I did not wanted to put it in you pass the data link pos connection URL as parameter and I did not want it to have that in the restory that's why I went to set fact >> that that makes sense uh Andrew used um GPG encryption decryption of the PG pass >> okay >> play on site >> I doesn't I don't need >> I mean it's yep no problem but that's the thing if we purchase the password even for client side it's that's the reason I I I and I made that because it's this password is already in clear in the docker script running [snorts] the pass the postgrad skill >> mhm >> and yeah but yeah >> yeah um I agree but still uh >> yeah still I completely agree I agree Yeah, >> cool. Thanks for this. >> So, yeah, and you can see the statistics for main. >> Is that okay for you to document so I can take care of the next one? So, we are then two person able to run them so we can move to automation. Is that okay for you? I was thinking eventually ask Jay when he's back to perform July before that August that let us six eight months for testing automation >> my uh I've I've uh imported June and I'm uh the the June report is running and I'll stop there and I uh so July can be done by you or try to get your own on >> just a minute. Sorry, the words on the dows. Uh, okay. So, you're finishing June then and then I take July and August for uh cool. >> Okay. >> For thanks. Um, GDK critical patch upgrade campaign. So, that one was on old until tomorrow. Uh, we have the one that Jay took care for July and we'll add it back to the milestone and same will be closed tomorrow after the LTS and weekly releases. uh Mark alerted us that Oracle and Timarine and adoption will be monthly updated from now. So that's why we took the Docker packaging in a custom uh build because don't we try to remove dependent outside dependencies even if it's from Adopjam themselves. We might have some delays on the S390X. We'll see if such an issue is required in the future. It used to be required because it was a full trimester thing but yeah uh maybe we can automate creating one per month. There are many solution but just information before um on Daniel update center CR rotation on old waiting for Daniel's PR to be merged then we'll evaluate the update center integration integration and timings I don't remember if we can sign with two certificates We'll see. Uh but the new certificate and the new key are there. I'm the only one with access to the CA key. I might I need to sync with uh VDC. So we can discuss uh if he will own the if he will be the second person in case I'm dead uh or or if um uh if someone else need to be designed. That's a 10 year valid system. So, yep, we don't have many solutions here. Windows 2019, I think we have to wait for tomorrow >> to deliver new theent image. >> Um, I think that's already delivered. >> Yeah, we have the SSH agent. We don't >> Yeah, you're right. Uh, sorry. Yes. SSH agents release then winp unpacker images cleanup. >> Yeah. >> I don't think we have anything else on this one. >> No, I don't think so. >> Um on the support parts. So we have the sonar cloud token. Uh so I need to answer need to answer and the team will check the sonar cloud access to generate the token. So the the risk of such a token if it is exposed are the same whether we put it on GHA like today or inside CI genio assuming CI geno has a flow. The security posture is even improved because the time window when the token is available is shortened with the pipeline library that's the contributor provided. Uh so we'll have to move on. Uh I might ask you for help uh survey but if that's the case I will schedule a knowledge sharing session with you otherwise you can just ignore it by default. >> Okay. >> So you mentioned ill score. So Adrian has the loss. >> No he just told me he just responded to the he doesn't have access anymore to element. I will give him on another channel. Okay, follows up >> but I will work on it. >> Cool. Uh mirror selection in update center. We have not heard uh back from uh the tuna mirror. >> Yeah, >> but it is the only China based mirror. So we have restricted this one only to China. So the country around China don't have the same problem as what was reported. The contributor wasn't able to tell us about an organization that will be okay to host us. So um should we run a mirror in China by ourselves? That's the question. I think I will reach to the board for this one because if we exclude the tuna mirror, the China user won't have any mirror behind the great firewall of China. So right now we need to accept that some user will have the same problem as this one and they will need to have servers outside China if they want to have a reliable mirror. Um let's bring this to the board. I've already calculated uh with two different cloud provider that provide VPS um VPSs inside China uh and both of them are costing like 20 to 30 bucks monthly. So that's that could be affordable through CDF. Let's see. >> Okay. uh CDN such as Cloudflare and others. Uh eventually we can check on Cloudflare but with the update center Cloudflare is with air2 is really bad. It's not posix file system and mirror bits needs a POSIX file system to synchronize it needs sync or FTP and the trick we use with a data center is already problematic in that case that will be even worse. So that's why >> okay >> uh and the cost of fastly that could provide local CDN are out of question. Maybe we can have cloudflare as CDN in front of our server though to offload some of the cost but if it's 30 bucks monthly I don't think we should care. >> H um so sorry any question on the >> No no no good. Sorry, I was a bit too fast. There are three issues that should come back. So, GDK patch upgrade campaign. Uh, the one from July managed by J is back to the new milestone. Of course, um, decrease bomb cost by splitting test more efficiently. It's back to the milestone for you. So at least see if we can if we see something that uh in the cost we saw a global decrease. We cannot be 100% sure it's thanks to the bomb. However, that will be interesting to to see if the tags in AWS are just find and report that we saw a decrease. I don't remember if there were other tasks in that area that were directly related to infra or cost. Uh directly related to cost. I have something uh not publish that can uh to to skip uh combination that have that already passed in the previous build. That could be >> Yep. Yep. implement skip test already passing on previous build. That that's for pull request mostly, right? >> Yeah. Uh pull request but not eventually for the release to if they are on the same commits. Um but yeah, it's not no I don't think so. No, no, you're you're right for pull request. Yeah, open request. >> It's same commit but to be confirmed. Okay, I let you check with the usual bomb maintener. Not sure about this one but for PRs that will help. >> Yeah. And about cost but for later and that's absolutely not for now but there are some issue and discussion about testing more on bum for example they're ready for review. They're ready to merge request on course to test them automatically on bum to be sure that they pass something like that. I have something working but I'm not introducing it yet because it will increase cost and >> but >> it will increase cost cut but it will also makes the the new codes more viable. So in the end it could be yeah >> exactly that's a trade-off. Um let's finish the skip test part and see if we are uh because then uh we will be able to one thing could be Friday watch for the kind of instances we have on the cost. Um I'm interested into trying one or two weeks using virtual machines instead of pod kubernetes agents. >> Yeah. >> To see if we can move to deter first a deterministic cost because we know the cost of machines in advance. >> Yeah. Uh we that's easier than having Kubernetes abstracting away things [clears throat] on VM and even build on and just checking. >> You like doing mathematics, right? I [laughter] see. [gasps] >> Uh that will also open the ability to run more test using Docker. But that could be interesting. >> And see and yeah, absolutely. That's all I had in Oh, no. Um, yeah, I think that's all. I'm not sure if we should work on the NodeJS 22 to 24. That's uplink dependency updates mainly. That mean bumping the framework and NodeJS npm updates. See if it doesn't break anything and then move on on NodeJS 24 and then see if things are not broken again. >> Yeah, can be a task for LM to check the dependency update but >> Yep. But someone has to drive the LM. >> So do you think >> Yeah. No, I don't think Let's keep it for now >> in the backlog. Uh Jay is not there. We will see with him when he back. We will we will Yeah. Yeah. Let's see next week. >> Okay. Thanks. Okay, we're good. any other topics that we should discuss or >> No. Yeah, we'll see next week too. I'm thinking about Kubernetes upgrades. That would be good to put in our plan. >> Mhm. For next trimester uh ideally in September that would be good. Yep. >> But we are we might face the ingress replacement first. We have to be careful that cloud provider does did not try to be smart. >> That's not perfect for me. >> Cool. Thanks. So, I'm stopping screen share. No, already stopping recording. See you next week. >> Bye >> bye.