Flock 2026 Packit And Fedora: The CI Story Continues

Yes, so we can start. I I'm Christian I am from the Fedora CI in this presentation and I'm joined by not Nicola but Matthew from Packet team. And we're here to talk about Packet Fedora and Fedora and like what we've done all around. So we're just going to go straight to the what we're doing with what's Packet and like how it where how it's related to Fedora. And >> All right and I will take over from this point. So um we will start describing the what Packet does, what it already did and how we are basically improving on that and what other responsibilities we are taking over. So um we will start by the opt-in features that were the first one that were available. So what can we basically do is automating the releases to Fedora. How does it work? You basically opt in, therefore you need to configure it. So we set up a Packet config and what we can do is that when we get a notification from release monitoring, we create a pull request that you should check and merge. And uh apart from that there are also other features built on top. So after the merging we can also do the koji builds for you because you know that's the annoying part. It's basically a lot of manual steps that can be easily automated and we also know that a lot of maintainers do have some scripts that automate already. So that's one thing and also after the koji builds are done, we can also do the body updates. And from probably one of the most interesting features maintainers are interested in are the support for side tags, which is a relatively new. It's still a bit painful to deal with them. So, even with the automation, it's really hard to work around it. And I guess that's for the opt-in part. So, this is not uh what's automatic. This is uh your decision to do. And you should also be in control of what's going on. So, even if you create the pull request, you should check it. You should be aware of anything that you release into Fedora, because that's your responsibility, right? And uh there were there is also this initiative that we're mostly talking about today for this talk. And this is a package as a Fedora CI, which is the default. And what it does is that we basically react on the open pull requests in this Git. It was implemented as part of the Fedora change some time ago. And there has been a lot of steps going through. There have been some issues also implementing it. So, it's been a long process, but we are slowly there. And uh we are right now basically most of the functionality is implemented. So, when you open a pull request in this Git, you can see that uh you get the scratch builds done. And there's also a lot of uh TMT plans enabled by default, which is uh contribution mostly by Christian. So, you should check these They They should basically help you with uh with the mundane tasks that you are tasked as the maintainer. So, you should be checking those. And following that, how the pieces fall together will be described by Christian. >> Okay. Uh one thing that might be confusing is like putting out like the word Packit and the Fedora CI. Like how is it all related? Because Packit is not Fedora CI, Fedora CI is not Packit. But basically you can see like a graph. Do with what you want with it. Uh the lower parts is what the Packit as an opt-in does, like doing all the upstream koji builds and stuff. And then we upgraded so that it also does something by default and that is this Packit by Fedora CI. Does the koji build and then it runs all of the test. However, the tests are usually defined by Fedora CI. We have like some uh tests that are defined in TMT and then we run RPM inspect, RPM diff, check, and whatever other tools you would like. We also have some of the more legacy kind of workflows that go through a Jenkins and like that when you would do a body update, then it goes through Jenkins and then the tests and then back with all of the same workflows. That is what we have and that's like kind of the uh the line that we got between Packit what the Packit does and what Fedora CI does. So, Packit does all of the communications, is the tool behind everything. And Fedora CI is like more on the um the glue of like the tests and everything. Yeah. >> And what we have done more recently. So, we have done like a lot of migrations. So, we had the all of those Jenkins runs that now the everything is run on Packit, the scratch builds and TMT tests. We also we had like some Zuul, those are also gone. and everything is migrated. It was not very uh straightforward because we forgot the ELN scratch builds, but we got them in the end. Uh the more recent one is uh the changes from pagure.io. Those also came a bit quite fast, and we just like scrambled and like now all of the testing farm jobs are also given by Fedora CI as fast as we could. Then, there's also like some custom jobs that are done in Forge actions, and we don't plan to touch them. For now, or who knows. Yeah. And we've added more tests. >> [sighs] >> So, the most recent one and most reliable, I would say, is RM dep check. Thanks very much, Adam, for this. Where is Adam? >> Remember. >> Okay, thank you. And also license validate for SPDX validations because we have noticed sometimes the spec file like change, and the license is no longer in SPDX. Usually, that is available in RPM inspect, but it's better to like have it more visible. Something is wrong. And we're going to add like some more things like a Fedora review and stuff. If you have something interesting, uh let us know. >> Also, from the package side, I think that has been enabled maybe 3 weeks ago. We have a log detective integration, which is also kind of a collaboration with the log detective project, which basically, what it can do, it takes the logs from the failed builds, as you can see on the screenshot. So, we have a failed scratch build for Rawhide. So, it takes the logs and tries to summarize the issue, so you can uh instead of digging through loads of logs, you can just see uh summarized description of what the issue might be. Sometimes, of course, you will have to dig in, but uh well, you don't have much choice, right? So, you can also see the example from the dashboard on how the final result looks. So, it gives some analysis reference for the log detective, also the ID of our job, and then the explanation, which uh well, looking at it looks quite readable to me, instead of the long lines in the logs. And uh we intend to expand this support and integration further. And uh let's go into the future. So, what do are we planning? And uh this is one of the discussions that happened yesterday. So, there was also the proposal to discuss if we can do something like a PR-based gating for Fedora. Um I'm not sure what was the outcome of this discussion because I joined today, so maybe Christian can provide some context. >> I can like probably summarize. We had quite heated discussions, let's put it like that. But, we've come to an agreement that we'll try to like do it everything step by step. And the first step would be to bring more of like what we have with Bodhi and try to like make the PR jobs more reliable. We'll probably do uh first some change proposals to describe what we've heard like in that workshop, and then get some more feedbacks. So, keep your eyes out, and we'll see how things evolve. >> Uh yeah, and again, more tests. Please come to us. So, we're trying to like make everything as easy as possible. So, we have like one repository. You can describe like what test you have. Just to be a D&D test. And we will like try to like bring it in like step by step. And hopefully you won't have to like deal with anything that is related to Packet. Just like on the video side. And everything would be smooth. Let's see. And another idea that came when we were looking at Fedora review is like to make to have a way to like have less configuration files. Because we have a lot. Like RPM inspect, RPM lint, Packet, everything. I don't know if you like to have all of these in the package. Uh Personally, I like the how my project tunnel does it and set only one file. But this is opt-in. So, like I think would be nice, but like would also be interesting to for others to like pitch in and gather some feedback. So far, we have the feedback that they want to have the opposite. Yeah. And Jenkins. For those who know Jenkins, then we know why we don't like it. Anyway. For the user, at the very least, sometimes you see this kind of like page. But you don't know like, okay, how it failed? How do I navigate? Hopefully, we can get rid of it. So, that's on our agenda. And monitoring. Greg is not here, but maybe like through the camera. Hi, Greg. Thanks for everything. We're We hope to like get all of this kind of monitoring and like get notifications on uh the metrics and then if things are getting like too out of hand, then we can get know a bit sooner. Right now it's everything the monitoring side is like what is monitoring. So, it's just this Hopefully we can get it soon. Yeah. Um >> All right. So, if you want to get involved, uh the best uh thing to do is to open up a discussion. We have a public metrics channel. We also have a lot of issue trackers, especially for Packet. We have a lot of repos, so there are many places to open issues. Um right now we don't really have much bandwidth for feature requests and uh that many improvements, but we are open for community contributions. On the other hand, we would pretty much uh like to discuss those beforehand cuz there are also some PRs that sometimes get open and have question of quality, so yeah, it's better to discuss it before just jumping in and uh wasting your time in case it doesn't align with our vision of what we are trying to achieve or it's not really going in the right direction. I'm not sure if you have something to add to that. >> Well, no, just like we have the in the metrics channels, like contact us there. We have like uh a discourse. Wherever it is more comfortable for you, like just get to us and like let us know uh what to do better. And lots of questions, so. >> Okay, Carolina. Uh do we have mics or do I give a That's easy. >> Uh thank you. I'm a huge fan of CIs, uh as you may know because I open uh also issues to uh CI repos. Uh my question is the So, great work. Thank you. Improve on that. That's That's great. That's all of it is great. Do you have a single repository where we can uh send our issues to? Uh because there is a quite a lot of jobs. There are scratch builds, tests, uh license validate, and so on. I find it hard to find the proper place to uh open an issue to. So, if there was was one place to to like user to have user issues, that would be great or some other easily discoverable place. >> So, yes, there is, but I forgot to put it on slides. Sorry. So, hopefully everybody knows about the forge page. Then just go to CI and tickets. Just any questions you have, put it there, and then we'll find out where to put it afterwards. >> Also, we will try to update the slides and include the link there. >> Hey. Uh so, you had mentioned that there wasn't really room or resources to work on some features, but I'm curious, is there like a list of like, you know, if you guys did have the resources now, what were the things that you would immediately want to work on? And what would some of them be now to help peak interest in the room for people who would want to take a crack at them? >> Yeah, as for the package, right now we are kind of uh shifting our focus towards uh agentic handling of the CVEs. So, that's the primary focus right now, and therefore there's uh less people focused on package itself because uh um as you could probably see from the start, there are multiple moving parts, like the package is also the upstream automation, it's also the downstream, and then there's also the Fedora CI part, so it kind of splits the people in between those separate parts of the project. So, yeah, like the focus is right now definitely on maintaining the packet working and implementing what we promised for the Fedora CI. I hope that answered the question. >> And just a short comment, like in the same CI tickets or probably in the project, you would also find the project view and like what are our uh near-term goals and like long-term ones that we're trying to uh work on. So, you could uh probably comment on those if you want to get involved and then we kind of guide you if there's any need. >> Hey. Um let me start by saying that this is getting better every day and I thank you for that. Like I remember when the only CI we got what what like push it and see if it builds. That was the CI. And it's getting better. Like it's still sometimes frustrating when the jobs are failing and you don't know why, but thank you very much for improving this. >> Thanks a lot. >> My my two questions about the improvements. First, when when the job runs or fails, I want to be able to click it and go to the failure. I am overwhelmed by Ansible gibberish I don't understand and finding the the problem uh or seeing where the tests are now. Like oh my god, why the test is running for 2 hours? What's happening? I want to click the log and see that it's stuck somewhere. This is really hard and being able to see it like it happens, I don't know, in GitHub Actions, for example. You click it and you see where it is and you see what it is doing from your perspective, not from the infra perspective. That would be awesome. And another thing is is it possible, once we migrate to 4G, to generate feedback from the CI jobs for particular lines? So, for example, you would run RPM inspect or RPM land or Fedora review, and it would say, "On line 27 of the spec file, you have a mistake." And the UI would actually like display it there. Is that I think it should be possible. I don't know if it's important, but it would definitely be really cool. So, just I I see your face, so the first thing was more important than the second. >> I mean, how much do you like log detective? Maybe that would be the place. Let's see. >> Yeah, log detective like deciphering what it is telling me is slower for me than reading the failure in the log, so I don't much. It's a cool idea, anyway. >> We do have the custom views that we're trying to implement. >> So, there there is an attempt to sort of fix this. Um if you look at the RPM inspect is a good example, cuz when you look at an RPM inspect result, it gives you a kind of HTML view with the table at the top. And that's a generic feature in testing farm, like any test can potentially generate an HTML like that. And we've added them for some more, right? Like mini TPS has them. Right now, there seems to be a bug where it's not getting shown immediately when it should be, and I know Christian's looking at fixing that. So, yeah, we are trying to fix it. >> But like >> getting the exact line would be a bit difficult. >> Yeah, I would just get back to the question from you. Yeah, like if we had the bandwidth, definitely a a of like UX usability stuff. Like we are aware that things can be improved. Like functionally, I think we are on a good level. But yeah, there are some tickets also from Miro about like per architecture displaying of those or like uh uh displaying right away if one of the architecture fails and this kind of stuff. So like those are small issues. Uh like Christian is doing a lot of great work into like putting stuff into the shared repo like the plans. Uh but I'm sure he would enjoy if he had like someone to help him with that. >> Yeah, maybe just to reduce the worries that like there was like term called like maintenance. It feels like yeah, you are going to deprecate it. So no, it's basically like I would call it like active maintenance and slow moving forward because yeah, we need this to work. We would like to like push this forward. So uh we might have like a reduced capacity, but still we would we value VC value and that's so no no worries on that front. Yeah, maybe there was this mention about like the gating discussion from yesterday. So if anyone is interested, we would like to probably continue this discussion as part of Fedora CI SIG which is kind of being resurrected. Actually, the discussion between me, Christian, and Adam being published. So we will probably provide some info if anyone wants to join us and we would like to slowly move move things on. >> We would love to have more people. It's great. >> Yeah. >> conference to see a room full of people at this talk because it's me and Christian and Adam check on a video call every week and we're like does anyone care? Does anyone notice? Apparently, we do. So it's great. >> Yeah. And if if you can like we can definitely provide any support if anyone does not know how to maybe contribute, we can definitely guide guide you. >> Do we still have time? >> Okay, one more note. So I noticed when Packet creates an update, it's created under the Packit name. So, I assume it means that the package maintainer is not subscribed to that body update. I don't know if you can confirm or not. And so, it means that whenever a user submits a feedback to body, it's dev null. The maintainer doesn't know. And then that makes me sad because we QA try to convince people to actually, you know, put more feedback to body update. So, is there some plan to improve this somehow? >> Yes, we are aware of this problem. And if I recall correctly, we should have somewhere in documentation a page where you can configure the notifications from Fedora messaging bus, I think. So, there is definitely a way to get to those. >> Okay, if it could be done automatically by default somehow, maybe patching body, you know, adding some default CC or something, that would be awesome. >> Yeah, that's a very good point. And actually, we know about it. Sadly, not all the services along the line are able to let Packit act as on behalf of the user. So, so there is not a simple way how we can act as another user, maybe picking like a main maintainer or something like this if we would like to. But there are like these missing features on multiple levels. So, we need Yeah, currently, it would be also nice if we can pre-prepare the notification, maybe set up for people to realize or something like that, or at least make it like click and you have like this in URL all the filter or something like that. Yeah, but good point. >> I mean, definitely, like if you have ideas of how we can solve this. We know it's an issue, but we also don't really know how to do it right now. Like, give us some feedback or like ideas. That would help a lot. >> We are out of time. Thank you very much. >> Thank you so >> [applause]