Flock 2025 Even More Release Automation

start. So, hello. My name is Matej. I'm from the Packit team and uh today we will be talking about uh new features that we have in uh our downstream release automation. And uh just out of interest, uh how many packagers do we have here? If you are a packager, please raise your hand. Okay, that will be most of us. Cool. So, and how many are using Packit on the downstream? Okay, that's a bit less. Okay, cool. So, I will start by summing up also what do we actually provide right now and uh what is basically what are the new features related to and uh how can we basically help you with uh doing the boring work that uh nobody likes to do, but well, somebody has to. So, I will start with uh the basic workflow that is. So, we have some source code somewhere on the upstream, GitHub, GitLab, Codeberg, whatever. And uh that's where the code lives. That's where people work on it. And uh then we basically need to get it to downstream to Fedora or Apple or whatever. So, what do we do right now is uh look aside cache and this git. So, we have uh one place uh where we have the spec files and uh sources that are used for the builds. And also, we have the look aside cache where we store the sources themselves, the archives. And uh yeah, so upstream releases a new software a new new version of the software. What we do, we bump the version, hopefully upload archive, and that's it. Most of the time. When no, then that's where the boring becomes interesting and you don't want that to happen. So, for that we basically have two jobs that can help you with that. That's proposed downstream where you where you basically control the upstream. So, it's the basically the push workflow. So, I maintain the sources myself, but at the same time I maintain the downstream. So, I can basically just push the new versions from the upstream to the downstream. And then we also have the other way which is pull from upstream. So, I'm just the maintainer. I do not maintain the upstream itself. And uh this one is basically a bit more helpful to people that do not maintain the upstream. So, I can do I can automate some stuff even though I do not have the right access to the upstream parts. Okay, so that's basically most of the work that we do. So, uploading the archives and some magic spec files to get the new version there. And yeah, that's the big part. And then well, after the release happens and after the maintainer checks everything and accepts the changes, the new version, then it gets merged and we go to the Koji builds and Bodhi updates. So, what does a Koji build do? We build it, we get uh as we get RPM package. And then we should test it and provide some feedback, Karma, also link the Bugzillas. That's what Bodhi is for. So, we can get some feedback on what are we releasing and how it looks like and whether it should be accepted into the Fedora itself. And after all after this whole cycle finishes, then we are at the user. So, that's when I can just run DNF and install Sifir, the new version. Okay. So, that's the summary of what's the whole workflow, and right now I will go into the new features that have been brewing since last year, and uh what we can do now more that we couldn't do before. So, one of the things that's uh it's not so recent. It's like autumn of last year, so right now it doesn't feel that new, but yeah, it's still new. Um we support non-git upstreams. So, if you have packages that do not have git upstreams, you can basically configure pull from upstream in such a way that you can uh just react to the release monitoring, and uh that's all you basically need. And maybe one question here, we will make a poll. So, there I've decided to put a snippet on this side, so you can also see the configs, and my question is uh who thinks that this is whole config, everything that you need, and this is all, nothing more? Who thinks that's whole thing? We have two people, three people. You'll get free stickers at the end. Yeah, that's all. Nothing else is needed. So, we just have the jobs, and of course, we can add the Koji build and Bodhi update, but uh we just have the pull from upstream, we trigger it on release, and we do it to Rawhide, and that's all. Nothing more. So, whenever we get an event from the release monitoring, um we will open the pull request to the Rawhide, and uh yeah. If you If you've seen some other configs, you probably notice that uh they can get a bit more complex. We also have a lot of customization available, so this can get out of hand pretty quickly. Um one thing here is that it's a simpler, it's more user-friendly in my opinion because you don't need to define a lot of things that you would have to. And also most of the time when you when you want to bump the release, you don't even need the upstream repository because you just need the archives. So, this one is uh simpler also in the sense that we don't even clone the upstream because you don't have to. So, we just take the archive, we upload it from the based on what we have the source in the spec file, and yeah, that's it. So, it's simpler, it's a bit easier, and uh yeah. You It's usually what you need. Okay, so that's the only Git upstream support. And one of the long-requested features was support for the side tags. That's uh relatively new, and yeah, you can see already the config is getting a bit more complex. So, uh yeah, that's not that really easy to read, but I will go through it. So, what what is happening here? So, we have a job for the Koji build. And we basically define that it's triggered by either commit or other Koji build, which means that when you want to release multiple packages, and they are built in the same side tags. So, in our case, um I've taken this uh from our own own packages, so you can You can that uh you cannot see from which package it's package itself, but you can also see at the bottom of the list of dependencies that is the old grand spec file that are needed for our package. So we can release all three of them at once in one update and they are built in one side deck. So the other parts are yeah, the package instances which basically used to test our own code. So we we we do the code you build only with the stage instance. And then there is also the requirement for the PR authors because there were some concerns about automatic code you build. So by default we just go with uh with our own pull requests. So if maintainer wishes to open pull request or push to branch he would need to specify themselves in the in the list. And yeah, and then the side deck group. So there needs to be a name of where it does where it needs to build. So in this case there are some nuances and well I would summarize this as side decks are very simple concept but once you start using it it's not simple at all. So what does happen here is that I will get back to the trigger. So one thing is that committing in this git triggers the code you build, but at the same time you might want to tag the build into You can also want to trigger it when something happens in the side deck. So that's why there is the code you build trigger, the other one. And yeah, I will walk you walk you through the release workflow and I will start by going through the instructions that we have for our own release card. So, this is part that happens on the downstream. So, once the release happens, you we need to check the pull requests that we propose. So, after the pull requests are merged, um you just wait for the koji build, right? Except not really because in case you don't release everything, you need to make sure that the respective builds are tagged into the side tag. So, there's also the other part that you need to do and that's manually tagging what you the packages that you are not releasing. So, yeah. Uh it looks simple, but uh it it gets more complicated with the side tags. So, what can you do with that? Uh in this case, I know that I haven't released this one package, so I will just post a comment. I want to tag it and I want to tag all Fedora releases. So, I post the comment, it gets tagged, and then the koji builds happen. So, yeah, that's the koji build part. And what else? Well, after the koji, what happens is the body update. So, exactly same thing. There are some differences. In this case, you can also see that there is allowed the builders. So, body update is triggered by finished koji build and uh who can build package and package stage, so we have the two instances. Uh If you want to build your yourself, you could probably just uh drop your name in there, your FAS user account. And, uh, yeah, there's side tag group. It's basically the same thing, uh, just the builders and authors or committers switches because this reacts to the Koji, not to the commit or merged pull request. And, uh, yeah. Koji builds, build, and then you just wait and see for the body update. So, you just get everything in once. You see all the change logs, and yeah, that's it. Um, uh, I've also linked, uh, some more complex, uh, configs. For example, I found the Podman, uh, which, uh, also uses the Koji side tags. So, if you are interested in that, you can also have a look. It's, uh, it's not as easy as our own configs. There is a bit more stuff there. Okay, I guess that's all for the side tags. And, uh, there is also one thing that has been asked for quite a long time, and, uh, that's the support for fast forwarding. So, just out of interest, who of our packages here enjoys fast forwarding from Rawhide? Okay, we have four, five people. Nice. So, this is, uh, gets a little bit more hairy here, um, with the syntax. So, what happens here is that, uh, you specify where do you want to propose the update. So, in this case, uh, we propose update to Rawhide. And then we also have the fast forward merge into. And we want to do that to branch Fedora releases. So, what happens here is that, uh, we basically just run the pull from upstream, uh, which means that we create the pull request with the update. And then we reuse the same commit and open pull requests into the other branches. So, you still can review and check what you are releasing. Also, it kind of relieves a bit of a pressure on us because we do not need push permissions on the this Git repos. So, yeah. As least permissions as possible, right? Security. And also, one of the nuances that I wanted to mention here is that you can notice that this the syntax is a bit different. If I go back, you could see maybe here. Yeah, you can see that the this Git branches at the bottom is a federal all with uh list. But, when I get back to the fast-forward, since you need to specify where do you want to fast-forward, it becomes a dictionary. So, yeah, it's not very nice, but uh there are no better ways how to handle this. So, the row height is basically a dictionary where you specify that one parameter for fast-forward, and then, well, the Apple needs to be empty dictionary. So, yeah, it opens the pull requests and uh with the same commit. Maybe one quirk of Pagure is that uh you get a lot of statuses on the commits because the statuses are shared for the same commit. So, when you open the pull request, you just see multiple Fedora CI builds or end tests. But, yeah, you you just need to go through it and check that everything's okay. Um yeah, and since I mentioned complex configs, uh I can get back to way how to handle duplication and that's actually uh contribution from community from Christians. So, thanks a lot. Um we haven't supported this and I recall some users uh using the YAML tags in the previous configs before we started supporting this. It was not very nice because uh you just randomly place a tag in the in the in one of the definitions and then you reuse it. So, this is uh something that is a bit more easy on the eyes. So, you just define the one top-level key and you can hide whatever you want there. And uh well, what would you like to hide there? Definitely targets and truths cuz uh I've noticed that some configs have a pretty long list. So, this is something that uh definitely helps cuz you can have them at one place and make sure that even when uh you have uh hardcoded values that when you change it, it propagates. So, I do not forget one job and then it fails. And uh one other benefit would be also the testing farm cuz uh there are some pretty wild configs with extra options and whatever else. So, yeah, there is a lot of duplication. So, this uh simplifies it a bit. And uh yeah, I would also like to shout out the distro aliases. We have converted to them recently. That's maintained by the COPR team. And uh yeah, there are also the openSUSE aliases that's maintained by Dan Čermák from openSUSE. And uh yeah, so this uh this is a pretty nice feature that uh a lot of projects would like to use and uh I also thought that it would be nice to mention it here because some people like scripting. So, if you need something like this, it's available. You don't have to make your own. And it also supports caching which we definitely use. So, yes. You can go through the multiple options. I've just taken this example from the readme itself. So, yeah. It's a very helpful thing. Okay, and I will slowly get into the more technical stuff. So, there were a few requests about releasing for Go. So, that's I've also linked the issue for the context. So, you can have a look and see what it's about. I'll just abridge it as they basically create vendor archives during the release automation and there's also some kind of license checks. And we have just added a hook that basically allows you to run something after we do our stuff with the bumping the spec file, change log, and creating the archives. So, you can also create more archives. And yeah, that's one of the most recent most recent being maybe 1 week 2 weeks old. And this comes from the Koji side that I support that I've mentioned at the beginning. So, someone ran into the issue where they do not wish to run the sync releases for pre-releases. So, yeah, that's not an easy way to automate. So, what can you do here is well, we've implemented an action which that you can use for uh basically indicating to us whether it should be run or not. So, as you can see there is the package project version and grabbing for some alpha, beta, or release candidate. And if it succeed and if this doesn't succeed, so it's exit code zero. Um it doesn't run. And yeah, docs are still work in progress. That's not merged, so you will have to wait a bit. >> Why the command is a list? >> Sorry? >> Why the command is a list? >> Uh all actions are lists. So, we specify all the commands in list. Okay, and one of the probably not that uh surprising things, we also built package as a container. And uh we've got some requests for making the arm builds. So, right now you can also run it on the arm machines. And uh it should help you with the reproducing of uh what we do locally, but uh yeah, all the tokens still not an easy thing to do. And yeah, there's also one more thing for reproducing the uh Apple 10 support is a work in progress. So, this will be coming, but uh there is a lot of dependencies missing, so that will probably take some time. And uh last but not least, we also have some dashboard improvements. So, one of the things that uh were problematic were the errors during the koji build submissions, which are now propagated nicely to the dashboards. So, when it fails, you can see it. And we also rebranded the views a bit, so it's uh it looks a bit nicer. We also provide the logs, so you can check what happened. And uh maybe one of the things I would like to mention is the ongoing user experience survey, which is a QR code. If you download the slides, it's also clickable, so you can open it. We would very appreciate uh any feedback. If I recall correctly, it's also related to the package uh maintenance workflow in itself, not just the dashboard, so it would uh it would probably steer the way we go with the future of the project. And uh yeah, that's all from me. So, these are the related talks from my colleagues from my team, other team, and uh you might be interested in those, so you can check the list, and if you see something that you like, please go watch it. And yeah, we also have a workshop on Saturday, so we would like to we would like you to join. Okay, that's all from me, so we can go with the into the Q&A. >> My question is, how many packages use sync to release automatic updates in a branch of or in Rawhide or branch of Fedora? And of course, up to making the pull request, you know, maintainer may want to click the button themselves. >> Um we have a statistics, but I don't remember the actual number, sorry. >> What's your guess? >> In hundreds. >> That's pretty awesome. >> Yeah, that is a good guess. >> this been uh have you pushing for this? >> Um we also had a push for the uh the adop- for the onboarding uh in like last year or so, and it was a pretty steep curve, so there is a lot of packages that onboarded in the recent years. 2 years, I think. >> So, they're progressively like Okay, cool. Any other questions? >> Uh one thing what is not clear to me with the side deck. So, if if I have a build which usually goes out as a single package, but then on the other release I need to do it as a side deck with another package. Can I do it without issues? You know, that you have a combination like you know not always release a side deck. Just maybe like every second release or whatever. >> Yeah, that's the thing I mentioned with the tagging. So, this basically always releases in a side deck, but for the packages that you are not releasing you just make them with the command manually you tag them into the side deck. But this you basically have to post the comment yourself, but the tagging happens on our side. >> Okay, so we have a metric for projects with at least one merged downstream PR with automated releasing or koji build or body update at that 720. >> Thanks a lot. Any other questions? Then if not, then thanks a lot. Also, thanks for the questions. And if you would like to reach out to us, we have this list of very communication channels where you can reach out. And also, you can bump at us outside after the talk. So, thanks a lot.