Flock 2026 What's Cooking in COPR, Testing Farm, TMT, Packit And Log Detective

Uh, welcome, uh, to the PT presentation, descriptive name. And, uh, actually there there you might be confused that there was only my name in the in the invite, but, uh, actually there will be bunch of people who will show, and that's exactly for the reason that there are real humans under these projects and all the initiatives, and I would like to, uh, let you know these people that did the hard work. So, what's ahead of us, uh, what actually is the PT thing, and mainly what we are working on, and what you can expect for future. So, for the start, uh, PT stands for Packaging and Testing Experience. It sounds awesome, but it's basically a group of projects and teams, uh, in Red Hat, and, uh, with describing, uh, what we are, I'll start with how I see that the code flows from upstream through Fedora, CentOS Stream towards RHEL. There are various pieces together. And this is actually how our projects relate to this. Yes, it's a little bit of mess, but, uh, I would like to show it because, yeah, we are all over the place. That's that's the point. Uh, uh, and I am uh our aim is basically, uh, to unify stuff around all this space. So, provide same beha- behavior and same experience when you test, uh, when you build, ideally you reuse tests, etc. So, that's that's one part of it. And the second thing is we are trying to provide workflows. That's the experience part, so we are trying to glue things together, so it's some unified workflows, maybe integrating various services together. Uh, just a quick thing, uh, you might be won- wondering, uh, there will be also mentions about Log Detective, and, uh, I just want to mention that look detective didn't disappear. Actually, it was it just got closer to the Packit team. So, that that's just a what what to mention. So, there is like a Packit team covering multiple projects. There is Packit project itself as you know it, then there is also look detective project and email project by the same group of people working on all of this. And now finally to the important stuff and I'll pass my word to Pavel. >> Hello. Ahoy. I'm Pavel Raiskup from the Copper team. Anyone knows what the Copper is or should I give you some intro? Everyone knows. Okay, cool. So, uh >> [clears throat] >> What is what has been happening in Copper for past few months or maybe it's all already a year is that we were adding a support for hosting the stuff that we built in Pulp. Uh Pulp is a kind of generic great beefy machine that is able to host various stuff including RPMs and that's actually what we provide. So, um Right now, we have everything implemented or most of the stuff and we are moving and actually I'm not sure if you know, but you've been moved to Pulp in the meantime and we actually tried to hide this fact from you by by not interrupting you your work, but sometimes may happen something. So, sorry for that. It's almost done at this moment. And I mean, I wanted to thank to the Pulp team that is I mean, that there is a great synergy between the teams and they are always very helpful to help us and actually I mean, the preparations were very very long. We are like we were anticipating that these extremely large repositories like the RubyGems that have hundreds of thousands of RPMs could cause some troubles to Pulp and the team the Pulp team was great and they they prepared in advance and actually they scale very well very well now. Um Our team has the privilege to maintain Mock. That's that's the tool that's behind Copper and Koji and also Conflux uh uh these days uh and uh Mock has two new plugins these days. Uh one is for analyzing unused build requirements. It's called Unbreak. Please take a look if you care about minimizing stuff. The other one is for system monitoring stuff. I I don't recall the name of the plugin, but it actually allows us to track the amount of CPU and uh storage and stuff like that um while you are building and maybe uh mm analy- analyze the s- overall situation situation and give you more um more this or that. And yeah, what's uh and we actually I mean thanks to RPM or I mean the idea uh behind how how do we protect against attacks like the XZ one when the artifacts uh or the the provided artifacts in the build system are actually modified by the test suite uh scripts uh there was the idea like how do we how do we exclude uh the test suite from the ability to modify the build artifacts. And uh since RPM 6 .1, I'm not sure if it's out right now. It used to be better, like 2 weeks ago. Uh we have the ability to short-circuit check, uh meaning that you can run the build without running the check phase, and maybe preferably yes, run the checks check section later on after the after the artifacts are safely copied out copied out of the build system. So, this is the thing thing that we are working on right now. >> [clears throat] >> And yeah, let's hand it to Laura. >> Okay, so hello. My name is Laura, and I'm from the Packit team. I believe most of you probably do know Packit, seeing the familiar faces, but just very shortly, Packit is for a few years providing CI/CD automation for Fedora, so like upstream CI, and then I really sinking in Fedora, bringing the upstream releases, Koji builds, Bodhi updates. But the one big new thing in Packit for the past year was that Packit is currently also the default Fedora Dist-Git CI. So, we are very happy that we can cover this part of the Fedora as well. The switch happened in February, and actually it's probably good if you haven't seen too much of troubles with that. Previously, there was a Zulu based CI, and there was also the Jenkins based CI, but now everything on the PRs in Dist-Git is covered by Packit. And probably what you might have noticed is that the checks on the Dist-Git pull requests look a little bit different. So, as you can see in the screenshots, but we've also added some functionalities. Hopefully, the user experience is a bit better, and we've added some more tests, test plans. By default, we running the RPM length, RPM inspect, uh, and we have definitely, uh, more plans how to extend this. And, um, as you have, uh, probably listened to in the other talks, like there is happening the, uh, work on the 4G uh, and the migration of the disk is repositories, so, uh, obviously Packit needs to accommodate to that as well. So, uh, currently the work on our side is to accommodate our tooling for this switch as well. So, once the disk is repositories are moved to 4G, everything will work as it has been working, uh, in Pagure. Regarding the release, uh, syncing to Fedora, uh, we have a one new, uh, nice feature, and that's, uh, Packit, uh, supports multiple release streams. So, previously, uh, as we are reacting to upstream release monitoring, we've been always reacting only to the highest release stream, uh, but nowadays, you can configure and consume, uh, multiple release streams. So, for example, for Rawhide, uh, you want to get the updates for the major version three and higher, and for the stable Fedora, so you need to actually, uh, stay on the, on the two dot uh, major release. And, uh, in the space of the Fedora CI, uh, we are planning, or it's already in progress, uh, to provide one repository with shared tests. Uh, so it will be all transparent to you, uh, easy to contribute, uh, we can discuss on the pull requests the changes, um, and, um, the plan is also to make the test infra independent. And the last thing from us, uh, that, uh, was one of our heavy focuses, uh, from the Packit team, uh, was actually something that's currently not uh, covering the Fedora, but uh, real and sent to stream. Uh, so we are providing um, AI assisted tooling to automate the package maintenance. And here we are covering rebasing, rebuilding, and backporting. And as an external person, you can see this in the CentOS stream MRs. You can see some contribution made by EmirBot. We've been hearing a lot of talk also about the CVEs. So, with Emir, we are trying to help with the situation. And even though currently we are not covering Fedora in the URL you see, you can at least see the code. You can maybe get inspired. And for future, maybe we can figure out also something for Fedora. And now I will hand it over to Jirka. >> Hi everyone. I'm Jirka. I work on the Log Detective. Yes, Log Detective is now part of Packit team at least. And it's actually hooked up to Packit right now. What you see over there is the output of Log Detective on one of the failed builds we had like relatively recently. Previously, we have already been serving and analyzing failed builds in Copper for some time now. There was this little ask AI button now there next to each and every one. There was also the teach AI button, but that didn't get much use unfortunately. That being said, one of the reasons we could finally do this for Packit is that we moved from the original architecture which was almost 90% deterministic, well, more than 90% to a semi-agentic architecture where the agent basically decides which artifacts to look at, gets some additional information about what was actually happening, what was the expected result, which allows us to basically get much better responses. And amusingly enough, despite the fact that it's kind of an agent now, we're actually saving on tokens, like substantially. Um so, it's a win-win on all accounts. And the original interface on the www.logdetective.com is still on, so you can try it yourself. We have recently actually introduced an integration with OBS. Yes. So, there's also that you can analyze your OBS builds, not just your copper, koji, and such builds. And um now I will hand it off. >> Uh hello. Uh I'm Peter Splichal, and I'm coming to uh bring a couple of news from the TMT area. So, TMT, uh which stands for test management tool, uh we during the last year uh we are glad to announce a couple of tangible steps towards simplifying and unifying things. Uh so, one of them is TMT policies. They were introduced about a year ago. At that time, they were just available for for tests. Now, quite recently, there was added support for plans as well. On the right side, you can see an an example. It's uh it shows like um it's basically a way how to dynamically uh modify uh test metadata and plan metadata. And it's designed in the way to make it uh enough flexible, but that complex stuff will not be something that user will be face facing. Basically, the idea behind the policy is that uh for example, when testing things on rail, uh there is a defined way how the guest should be prepared, the defined way how the test checks should be enabled, and do this in a consistent way. And then like the policy would be just like the single thing which user, or for example, testing farm, or the CI would provide uh as an information, and the farm would apply these settings. So, for example, on the right side we can see like that the ABC check to be like a mandatory check for all tests to be enabled when RLCIS is testing. >> [snorts] >> So, uh this is what is what is being done on the right side. And for example, the new feature for the guest profiles or environment setup profiles would be another thing that like a specific prepare stuff would be inserted into into the plan, which would enable the guest profile, and the and the guest would be set up in that in that in that in that way. So, um these are the policies. Second thing, TMT recipes, uh quite fresh thing. Uh when the TMT run is completed, there is a recipe animal file which contains a reproducible information about about about the run. And it can be used to rerun the same thing once again without any changes or with some changes like the example below shows. >> [snorts] >> That it's possible to take the run, take the recipe, filter for example just the failed test, and rerun again only the failed test to like speed up speed up the execution if there was some infrastructure problem or something similar. So, uh those were the recipes. And TMT artifact install, another step towards unifying the way how things are done locally on your laptop and in the CI and testing farm. So, the new TMT artifact install plugin will allow will allow like provide in the same way the same way how uh for example, Koji a fresh Koji build would be installed on the guest. Uh so, uh two examples how to do this locally using TMT run or using testing farm request. Uh you provide a Koji build, and uh the very same way uh of installing the artifacts would be used uh on on the laptop and in the CI as well. The one one significant change which applies here for for the users is that we want to provide a simple way how to how to install packages which are conflicting when there is a single build and there are conflicting conflicting packages. So like now everything which is needed for the test should be like explicitly included in the requires. So that's about the artifact install. And a couple more things but I saw the sign that 5 minutes so maybe just a few seconds and the new cleanup step which is a dedicated step which allows the finishing tasks to run separately. Some improvements in the image mode area and also now it's possible to expose devices from the test runner to the guest. Improvements in the hardware specification and sole script which can be useful when running things uh when you need to do some common common setup before test. And finally per test adjustments in the discover step or adjusting plans when importing them from remote repositories can can also like give some more flexibility. So that would be news about TMT. Quite a few things from these are new and we would like like to invite you for for like experiment them try try them and provide us feedback and of course everything is open. So for contributions that would be nice. And that would be it from TMT. >> Hello. I'm Yevgeny. I brought some news from testing farm but I need to speed run it I guess. So first one AWS bare metals instances in testing farm we offer it now and user can request them by setting hardware requirements like in these examples and we increased instances variety in AWS so the provision should be more robust. Um testing farm includes air AI analysis for all failed and error test requests. Uh it is available within several minutes after uh test request finishes. Uh but note that it is in prototype stage and we are still actively fine-tuning outputs. Uh big change to token management. Um now all API tokens have an expiration date limiting their life to a maximum of of 1 year, uh but we provide a grace period option uh that allows user to choose how long the previous API uh token remain valid after a generation to allow a smooth transition without any interruptions. Uh there are several results viewer improvements. Now we show TMT warnings TMT warnings per plan and uh we improved error reporting per plans. Um yeah, testing farm in uh supports image mode aka boot C testing and um uh yeah, you can run tests there. Uh you can install any artifact there and uh we provide uh image mode composers for all Fedora sends at those stream we support. And uh yeah, architectures are arch or Intel platforms. Uh CLI improvements uh the skip guest setup option allows you to skip all default guest setup tasks. Uh most time-consuming one uh which can increase uh uh speedy speed up system provisioning. And uh we added a composers command uh which lists all available composers in testing farm and we added a list command which uh shows all your uh test requests and reservations. And uh the multi-host pipeline now supports artifact installation uh via TMT prepare how uh like Peter showed. And uh previously, it was available only for single host pipelines, but now this limitation has been lifted. And uh you can you can install uh copper builds, culture builds, brew builds, and other artifacts of your choice into multi-host pipelines. And uh the next steps uh in the next vlog, I I hope we'll bring single host and multi-host pipeline uh unification. And uh by default, there will be no TMT or TF profiles used, uh but uh we will provide specific policies for Fedora CI, CentOS Stream CI, uh Packet, etc. And of course, we will improve AI provided test results analysis. >> Back to Frank. >> Yeah. As you can see, there's a lot of going on. Uh Yeah. And just uh two things from DevConf, if you happen to be there. And that's basically it. Uh And we can move to the Q&A. I But just before that, I would like to thank all these roughly 20 people working on all of this. So, huge thanks for all the work. >> [applause] >> Questions? >> If there are no questions, I have a few things that I forgot about. Or or I mean, I I I wanted to save some time before, you know. Uh Uh of course, uh AI stuff in copper, uh we became addicted on Gemini reviews, because I mean, in our team, we very appreciate any reviewer. Reviewer is the topmost important person. And everyone should learn from us this, because I mean, producing the code is not that difficult today. It's really complicated to get it reviewed. And we don't know what to replace the Gemini reviews with right now. So, if you have suggestions, please let us know uh >> [clears throat] >> because Gemini is being stopped by Google, I think. AI again, Jirka Kyovsky is working on tool RPM meta, which is going to like uh estimate how long the RPM build is going to take and perhaps assign to the build a B fear machine to speed up things. Uh Uh architectures uh we are going to have a problem with Z architecture. If you can help us with that in the future, please let us know. As you may know, it is back again in copper natively, but within I think it's a year till IBM Cloud is going to shut the the S390 down. We need to find a replacement. Similarly, it would be nice if you knew about Power 10 architecture because Fedora ELN is uh is only I mean it requires Power 10 architecture. We currently have hypervisors that are Power 9. And a little advertisement, I am working on a uh Okay. on a project that is called No RPM. I care about security and the tool is able to parse or I mean expand values of RPM macros safely without let's say dangerous side effects like running bash scripts and stuff like that. I know I was integrating this into RPM build pipeline in Conflux and also in copper. >> So, as you can see, we can speak like all day. We should next time have a separate separate We should next time have a separate a separate room.