Flock 2026 Forgejo Runners on Fedora Forge

Hi, everyone. For your runners on Fedora Forge, what they are, what we offer, and how the automation behind them works. Hi, everyone. I'm Lenka Segura. I go by Lenka Sek on Matrix and us. And I would like to show you how far we are with Forge runners on the new Fedora Forge and what can we offer. As many of you know, Fedora has a new Forge hosted on forge.fedoraproject.org. It's a self-hosted virtual instance replacing Pagure for Fedora-related projects. The main The migrations from Pagure are underway. Um it's getting populated. There's a lot of projects already on board it. It can do everything you would expect uh a Git Forge to offer. You get Git hosting, issues, pull requests, wikis, and most importantly for this talk, uh workflow automation via Forge your actions. We offer runners. Uh the runners are available for your Forge your action workflows. They are scoped for uh per organization. So, uh on the Forge, you request a runner and they are scoped for your org. We offer no global runners. So, you know what you're running on your runner uh is just yours. Uh you don't need to share it with anyone. And uh you can request one via the issue template on Forge. I'll show you later how to how to onboard, how to get one. And important thing to say, they are designed for the lightweight tasks, linting notification, repo events, um issue stuff, pull requests. Testing in Forgejo dispatch, we offer that as well. Uh they are not uh designed for heavy CI workloads. So, as of now, on production environment, we have 20 runners serving 20 organization. One runner per organization. Uh what is a Forgejo runner? I guess uh who does not have any uh experience with Forgejo runners on GitHub or elsewhere? >> What is the question? Who has experience? >> Sorry? >> You asked who has experience. >> Who has what? Sorry. >> Experience. >> Yeah, who has experience? Who knows what the runner is, what the runner does? Yeah, but most of you, everyone. So, it's like a container I was warned not to use the name the the word agent or service that connects to Forgejo and is waiting for for a jobs to take. So, when it happens, when the workflow gets triggered by some event that you want uh Forgejo dispatches the job to the runner, runner picks it up, runs your workflow, and sends back the results, either fail or or or uh or pass. Um it's based on the same concept as GitHub Actions, so it is very similar. Uh this is how it works. So, you drop your uh action file into the dot for work dot Forgejo workflows. Uh the Forgejo dispatches the job, the runner spins up the container, um runs your workflow, reports back. And uh how it picks up the runner, well, it uh syncs on the run zone label. Yeah, uh There is a link for the Forgejo documentation. And we offer two runners, two types of runners. The regular runners, um they're for lightweight actions. Um they are, as I said, for lightweight stuff, rebel events. Or when you need something to happen when you open a pull request, an issue notification. Uh they're They can handle four concurrent um jobs. And they usually go under labels Podman or Docker, but that's negotiable. And then we offer Test and Farm runners. So, they are optimized for the CI, for the heavy CI workflows via Test and Farm. Um they differ from the regular runners that they have a lightweight image, they use Alpine, and high concurrency because uh that's basically only sending an API call and waiting for the result. So, we allow 20 uh concurrent jobs for that. And they go by label Test and Farm. Here is an overview of of how the regular runner looks like. We use the notebook worm, that's the Forgejo default, we didn't change that. All of them run as unprivileged, rootless, Podman containers, four jobs per runner, one runner per organization. Here is an overview of the Test and Farm runners. So, they use much smaller image, Node Alpine. Um it's like a four five times smaller than than the bookworm one. Uh 20 jobs per runner. And they just serve for dispatching jobs to the Test and Farm, which does the heavy lifting. So, no heavy CI workloads on our infrastructure. Um here is an excellent example that I took from uh the CI organization on Forge. They made uh this example of uh how to run a test in Farm Action. So, thanks to them. There is a source if you want to if you want to see how it looks like. So, they use runs on Test in Farm that finds the machine runner, uses Test in Farm Action. Those actions are mirrored on our Forge instance from the ForgEO. They supply the API key and then later create a Test in Farm. Or if you're not happy with any of these options, you can also onboard your own runner. ForgEO runners are open source. You can run one yourself. So, you can just register it against your organization. So, when you have a when you're a maintainer or an owner of an organization, you go to the settings tab, actions runners, and then you see there you have the runners listed available for your organization. And there's also a button to add a new runner. It will give you a token and you can register yours and run it on your own infrastructure. If you're coming from GitHub, that's going to be quite easy because ForgEO actions were designed to be very similar, not 100% compatible, but very similar to how GitHub actions are. So, this is a overview of what is the same. It has the same workflow YAML syntax, only you drop it in a different directory, not dot GitHub, but to dot ForgEO. The secrets bars are the same. GitHub is alias to ForgEO, so you can actually take your GitHub action with GitHub and vars, and they will work. They will translate to ForgEO. The same matrix strategies. The same triggers, yeah. What is different? Permissions. The keys and scoping during the action, that is different. That will get ignored for two doesn't support that. And caching artifacts have minor differences. They would not work as of now. And the runner runner image is much it's really minimal than what the GitHub offers. I think Ubuntu latest or something like that. Um if you're coming from Pagure, so the difference is very big. Pagure didn't have any built-in CI. Pagure was using Jenkins or Zuul. Um So that differs. The setup also. Here it's built-in in Forgejo actions. And for Pagure, you needed a an external CI service. The triggering was very different. Basically, the event left Pagure instance. There was a webhook to message pass and um the things were reacting on that. Well, here it's built-in. It's automatic. And the results were reported like as a flux or as a comments on the issue or pull request or whatever. And um Yeah. And well, here in Forgejo, it is the workflow is in your repo. And you can see it in the in the built-in tab of actions. Yeah, well. Technical pause. >> Sorry for the technical difficulties. >> I'm blindfolded without the phone. And I need you to see also. They are QR codes. Sure, go on. >> You mentioned that these runners are currently using Note 22. This is not a Fedora image. Are there any plans to replace the base images in the runners with a Fedora image? >> That Okay, that we're currently not using Fedora based images and if there is a plan to include them in the future. Well, it would be nice to have, definitely. Um we're providing default images and thank you Kristoff for providing Fedora based image that we could use. Fortunately, we cannot uh use those images but we have to maintain them, we have to store them. Uh well, anyway. And if you're not happy with the default images that the runners run on, you can always run your own image. You can include it in your action. Run uses and your image. So, that's just a convenience to have the default image. You can run whatever you want. >> I know that. I'm just asking about the default one. >> There was also an idea not to provide any default images and to force the users to always provide their own. >> The default still The default still The default still says Is it working now? It is working. Is it on? So, actually there is a default one which is network one. >> Yeah. That's the forge over default one. We're using the same for the moment. Exactly. That's the default one. We can continue. Thank you. How to request the runner? Okay, there is a QR code. It will take you to the forge tickets uh new issue and there is a template you can use. So, basically we want we need to know your organ- your organization name, your use case, how you want to use it, the runner type. Currently, as I said, we offer regular or testing farm runners. And if you have some special requirements, so mm if you need something that we don't provide currently, get in touch with them, write it in the template or contact us on our matrix channel. We'll see what we can do. We'll give you feedback. We can make it work. May- or maybe not. Okay. Okay, what's next? I don't know. Ah. Now, let's get to the technical part, my favorite part. So, this is, as you can see, a graph how it is implemented. Yeah, I have to stay close. So, you see, there is a Ansible container. Oh, thank you. So, there is a Does it work? Yeah. So, there is an Ansible container, Ansible playbook. That when we want to add or delete a runner, there is a runner definitions on the on the repository. It's just a plain YAML file. We store the secrets in Ansible private. Then, we run the playbook. What the playbook does it? Renders both things into one file and creates a secret in a in a OpenShift namespace and Kubernetes API. And also registers a runner on the four job pod. Then, what happens? There is a secret on the Kubernetes API and the rest of it happens on the VM. So, we have Kubert VMs on the same namespace and there is a watcher service that has privileges to watch that service watch watch that secret in the OpenShift namespace. And uh that service is watching like it's using uh Kubernetes watch stream, I think it's called. Yeah. It continuously watches the secrets when the secret changes, it fetches the runner definition base decodes it and runs Ansible pull that pulls the Ansible playbook, the inner virtual machine playbook that runs the that creates the Podman secrets and the volumes for the new runner and runs populates the service systemd templates that we have for runners and runs it as a system D service. And runner is online. If you want to delete it, then there's also a clean up uh clean up tasks that uh that the There are tasks in the playbook that match which runners are missing in case uh there is one runner less. So, then it mm deletes the Podman volumes, Podman secrets, uh stops the container, and the runner is gone. This is how uh is the overview what the infra Ansible playbook is doing. It loads the definitions from Ansible private and from the YAML files, renders the config, creates Kubernetes secrets, deploys the VM, and registers the runner in Forgejo. Uh the runner definitions look like this. It's just a simple YAML file. Uh We need to know the name, the scope. There will be always a scope because we don't offer global runners. Uh the label. The label also defines the uh container image that the runner spins and the capacity. Okay. Uh so, the runner VMs uh We have our main VMs on the OpenShift uh mm Kubevirt. And they are they have the specs. They have 30 GB of storage. Um they use Fedora base image and are provisioned by Cloud-Init that installs the packages and do the lingering and all the things. We have also some AWS VMs for more for like uh experimenting. Currently, the testing farm runners are running from the AWS uh uh, VMs. So, we can tell you that plug-in a AWS VMs works well. Yeah, and this is a the same overview from a little bit different point of view. Uh, there's a the Ansible mm, the in Ansible playbook what it does then the OpenShift layer uh, there is a service account with the role-based access that what your service uses and yeah, uh, the virtualization the Kubert VMs the AWS VMs uh, those are in dotted line because they are not automated yet they're manual thing. Uh, I have to add this to the automation when I can and here is what the Ansible role for Joe Runner playbook does the inner playbook that runs on the VM watches the Kubernetes secret. Um, starts the permanent secrets the the volumes and the system D services. Um, it's triggered in three ways by daily cron, by at booth it runs and also the watcher triggers it as well. Here are the important links to our Fedora forge. Again, how to request a new runner um, to the forge or documentation. Uh, where is our matrix channel? Maybe before you open a ticket try to get in the get in touch with us on the matrix channel first so we don't have duplications. And thanks for watching. That's all. >> Questions? >> For 5 minutes for questions. Go on. >> Uh oh jeez. Um I >> [laughter] >> I had two questions. Um first like what is the security model for all of this? Is it just the idea that each job is running a rootless container? Or is there anything other than that? >> They are well, the running them as a VM is one layer one one thing that we do also that they run uh unprivileged. We're experimenting with privileged ones, but we don't offer them. And the rule for that was that they have dedicated VM always. But if that answers your question. >> Right, but I guess you're saying for the potential future privileged ones, as of now it's a dedicated VM, but it's not like an ephemeral VM that's created just to do the one job and then turned down like it is in Copper. >> No, we don't offer ephemeral VMs as of yet. >> And then the second question was about the testing farm thing. For that, is it that each repository needs to have their own testing farm secret? Are there plans to make like a global testing farm secret that any repository could use? Is that even possible? >> Um make a global testing farm uh runner is possible, but we want to have them organization-scoped always. >> Okay. So, but each user needs to log into the testing farm thing and request their own API key and then put it into the repository secrets in Forge show. That's not set up by the forge team for you. >> Uh yeah, we only provide the runner like the the thing on the testing farm site. >> Right. >> The organization people have to do that, yeah. >> And are there plans to replace that in the future or for now it's just the way you trigger testing farm is through the forge show runner? >> Um replace the testing farm runners? >> Like I know there are various there are chatterings about replacing the testing farm forge show action with packet, but I don't Is that happening or for now it's just we're using the actions to trigger testing farm. >> [sighs] >> Um I don't think I can answer that. Um but basically we just offer runners. Um I don't know about anything plugin packets, but and like for testing farm you can use whichever runner. You can use the regular runner as well, only you will be a little limited. Like the testing farm runner is optimized for that the high concurrency of jobs, but >> all it does is make an API call that Yeah, okay, I get it. Thank you. >> Uh this is a very basic one. Uh can you provide an example of a organization because uh it is not clear for me uh what kind of organizations are you need to a runner I mean if if you need a runner you have to provide an >> Mhm. >> organization. What? >> Well, um the forge instance is uh like for hosting of Fedora-related projects. So, whatever was on Pagure previously and is related to um for Fedora somehow is an organization when they onboard the forge and can request a runner. Yeah, the forge instance is not meant for hosting personal projects. If if you mean that. Okay. Thank you very much.