Flock 2026 State of the Fedora Kernel

well. So, this is to the Fedora kernel talk. I haven't given this talk in a couple of years. Um and a lot's changed actually in a couple years. It's it's um for a long time the Fedora kernel was maintained in a particular way and that was fine and it was great. Um but we've we've moved to kernel org several years ago. Um it's probably been what, 6 years ago now? And so, as we moved to kernel org, a lot of things changed about how we maintain the kernel, how you might possibly test a patch, how you might maintain your own kernel. And that's why I'm here now. So, kernel org is the uh supposedly the always ready kernel. That's what org stands for. Um it is merged in with Rawhide on a daily base or merged with Linus's tree on a daily basis. Uh today is running a little bit late for my fault. I'll get to that later. Um but it it does try to actually merge in daily. We build daily um or at least 5 days a week. Sometimes I don't build on weekends. Um and then from that we have uh there was a repository quite a while ago called the uh uh no debug kernel because we built Rawhide kernels as debug kernels. We don't build them as debug kernels anymore for a lot of reasons dealing with performance that just made them unmanageable, right? We lost the benefit of doing so. But I do still keep the Rawhide debug kernel uh repository up-to-date with a once-a-week RC snapshot. Um it exists, but yeah, it it's not really no debug versus debug anymore. It's just for the people who only want to update once a week. Now, for stable Fedora releases, we follow Greg's tree. We follow Greg's tree religiously. I will build on weekends. I will build whenever it happens. Um So, that that is always managed upstream. The way that we actually do these things and so the the QR code here is actually linked to the kernel arc repository and so OS build is our master branch where we, you know, merge in latest latest branch and all those things. We also have a branch called Fedora-7.0 right now. Um there's, you know, what it whatever the current version is that we're building to stable. That's the the branch that we maintain for stable releases. So, like actually as of yesterday, uh 7.1 was released um last night, I guess. I time zones are weird. But I Sometime this week I will create a Fedora-7.1 release uh branch and that branch will have um not all of the patches that are in Rawhide. It will have some of them and that will be what we actually ship to Fedora stable releases. And so, the reason it only has some of them is because there are some some branches are some patches we carry in the Rawhide release that are um really rel specific and the policy on those is all of those patches have to to to actually do like uh if defined they have an ifdef on a config rel release. And if they have that um which RHEL defines and Fedora doesn't, then they'll follow that code path. We don't follow that code path, so technically those patches are no-ops for for Fedora anyway. But they're there for RHEL, and that's great that they're there for RHEL. My problem is I have a whole lot of bugs. I've got, you know, 500 to one bugs at any given Fedora release at any given time. And so part of the way of dealing with those bugs is saying, "Can you please deal with upstream directly?" Um It It works well. Um it a lot of times anyway. Um but when I ask you to deal with upstream directly, and if they ask you to apply to a patch, and that patch doesn't apply because of some patch that we have that's a, you know, diff, that's problematic. So, I try to get rid of all of those patches. Uh as many of those patches as I can. I Fedora is as close to upstream as we can possibly be. Now, realistically, we still carry 60-some-odd patches. Uh things like secure boot finally got upstream after 7 years, but uh the what what got upstream as kernel lockdown does not actually automatically turn itself on if you have secure boot enabled in your EFI. So, we still have to carry the patches that do that part of it. Uh you know, lockdown is enabled not not enabled if you're uh just just just due to EFI settings. And so, we carry those things to make those things happen correctly. Um strangely enough, I think every distribution is now carrying our patches to do the exact same thing. Uh for years, every distribution was carrying our patches to actually enable secure boot at all because uh we were the only ones doing it. But it's there. Um so there So these are the We do have deviations for a from upstream, but we try to keep them as minimal as possible. So one of the things that I will be turning on with 7.1, which I've not turned on with previous kernels, is there are some some deviances for FIPS mode. Uh some things that upstream will not take, but that Red Hat feels are important for FIPS mode. Um and and they're things like uh uh external hardware random generator, things like that. >> What's FIPS mode? >> [laughter] >> FIPS mode is a uh as far as I know, it's just a US um but it it is a a federal um Yeah, it's it's Well, so there's a certification process, and Fedora will never be FIPS certified because from my understanding, FIPS certification takes a couple of years. Uh well, yes, and a lot of money. And then there's um the the flip side of it is but there there are things that FIPS has said there are uh cryptographic things that FIPS has said are acceptable and are not. And so when you enable FIPS in the kernel, uh and it's a boot flag kernel boot flag FIPS equal one, um that will turn all of that on. Uh you you disable a bunch of cryptographic stuff that is not considered acceptable, and you only go into what is considered uh acceptable. Like I said, Fedora has a couple uh RHEL has a couple of things that we did not carry in Fedora because upstream wasn't carrying them, and I didn't find them critical. Until you find out that there are actually users. Uh even though we're not certified, we are still we can be FIPS compliant. And so, uh, we found out we had a couple of national labs users that were trying to boot Fedora in FIPS mode and it wasn't working correctly. So, we'll be adding those those patches into 7.1 and should be more FIPS compliant. Um, even if it's not certified. And it never will be because the life cycle of Fedora is shorter than the certification process. No, it really is. It's I think it's something like 2 years to get certified. >> That's like almost it's almost 30 months usually. >> Yeah, yeah. >> It is pretty long. >> Right. So, yeah. The life cycle of Fedora at all is never that. So, like I said, we do try to stay as close to upstream as possible, but we are different from upstream in a few ways. And so, um, you know, how are we different from upstream? Uh, when when we built things out of kernel arc, it is a source git branch. Um, you know, I if you're used to most Fedora processes, we have disk git. We have this, uh, you know, tarball plus patches and spec file or whatever. Uh, we do that for for a kernel, but we don't actually look at it. We don't manage the kernel in that way. We manage it in a source git branch. And that source git branch is, you know, the exploding kernel tree. It's great. You can look at it. You can do all of these things with it. Um, you can main- maintain your own branch if you'd like to. And you can do it in a way that's very easy to to merge with what we're doing, uh, from from a disk git standpoint, but we don't look at disk git. And when I say we don't look at disk git, if you are a, um, proven packager, technically you can contribute everything, right? If if you make a commit to to uh disk-it, it's actually automatically overwritten by anything we do. Um when I do make disk-it, it it I I'm I'm not even looking at it. Well, technically I do because I don't want to um I don't want to step on anyone's toes necessarily. But realistically, it just overwrites everything automatically. And so when I do the commit, I look at the diff and if somebody's done something that maybe I'll I'll, you know, reach out to them and try to try to make sure that we're in sync. And you know, the uh the other thing we've done is we've turned off pull request for disk-it at all. Do everything through the kernel arc branch. Um so if you want to see how we differ from upstream, there's there's actually uh two ways because the Fedora policy should say that, all right, well, we carry all of our patches and here's the upstream tarball. And when we build, we do actually still have the upstream tarball uh that is built and then we have one patch that is kind of everything. And that's not great at telling you exactly what we're carrying and why. So we do have a patchlist.changelog which has every patch we're carrying listed out. It it will have the the shasum you can you can actually, you know, check it out. But kind of give you an idea of what we're carrying, why we're carrying it, right? Ideally. Um but the the idea is Fedora is as close to possible as upstream. And part of why we are is because right now, Fedora is me. Um from my kernel standpoint and and so from a trying to get things fixed, you file a bug. I'd like to help you out a lot of times, but there's a whole lot of bugs. And really if you can go upstream and and deal with upstream directly, that's great. And if they can send you a patch, you can do all of that. Um So, what we have here, uh if you need to build your own kernel, and this is why that kind of all ties in. There's actually a file, uh there's a patch included in the spec file called the next kernel test patch. It's a zero There's nothing in it. Uh the whole purpose of that is of if upstream tells you, "Hey, try this patch and see if it fixes your problem." You can actually copy your patch to this and it will automatically be applied. You don't have to change the spec. You don't have to understand the spec at all. And suddenly your patch is applied. You can build it. You can test it and see what you think. Um That Yeah. For for single fixes, for Well, even if it's, you know, multiple patches, you can you can put as many patches in there as you want, but uh you know, it it's a single patch file that applies. Um If you're trying to test something upstream, that's a very quick way to do it without having to deal with Git, without having to deal with uh really anything. Now, if you want to maintain your own branch, someone like Asahi and what they're doing, you can actually just branch over what we're doing. And And there's a uh wonderful thing called makefile.variables. What makefile.variables does is it allows us to separate our branches and say, "All right, well, here's our upstream." So, when I do a like Fedora 7.1, which I'll do this week, um I say our our upstream is Linux 7.1.y, which is Greg's upstream tree that I'm pushing, right? But if you want to do your own branch that's based on Fedora, you can say, "All right, well, my upstream is Fedora 7.0 or 7.1." And you can then manage your branch all you want to. And when you when you do make this get or uh however you're going to you're going to do your stable releases, it will say, "All right, well, my tarball is now coming out of whatever tag is is specified in Fedora 7. x. And then I'll build my my patch list on everything that I'm carrying that's different. And so you can manage that forever and not have to worry about, you know, doing a lot of manual updates and changes there. >> [snorts] >> Uh last thing I I really want to talk about is is there's a uh I don't know if you know Thorsten, uh nerd in in Fedora. He has been maintaining the vanilla branches, been maintaining the uh Linux next vanilla trees and things like that. They're all in Koji. Uh but he's also come up with this great script called archify. And what this archify script is, which I'll link to here, um archify will take any any upstream kernel tree and basically throw the Red Hat directory archify stuff on top of it so you can just build your your your RPMs and manage it in the same way you would any other Fedora or Red Hat tree. Um it's useful if you want to if somebody tells you, "Test DRM next." You can use something like this and not have to worry about, "Oh, well, how do I deal with all of this?" >> [snorts] >> So. Uh at that point, that's kind of all the the basics that I have. So, if anyone has any questions, I know we've got like a couple minutes left. So, questions? There don't seem to have too many. Okay. >> So, um I'm sure as you're already aware, the kernel's been under a ton of CVE pressure as of late. So, I'm curious like both how is Fedora handling that problem currently? And I'm sure, you know, everyone could use more help in that space. How can, you know, the call to action for this room? How can people help >> So, >> this key part of the OS? >> That that's um the normal way that the kernel is handling CVEs upstream is they will not assign the CVE until something has been uh patched and then stable. Which really means the kernel the Fedora kernel never has any CVEs except for the last month when everything happened. Um technically those CVEs were not actually like there were no CVE numbers applied until, but there were times where uh with Fragnesia I think I built five kernels in a week. Um and that was it's the messiest patchlist.changelog I've had in a long time to be honest because I would apply a patch and then I'd revert it and apply the V2 and revert it and apply V3. I built five kernels. I mean, just just in one week trying to push that out. So, we're going to do everything we can to make sure you're covered. Um how you can help other than letting us know if you see something that we maybe didn't see, I'm not entirely sure. Um but you know, we we're we're going to try to make sure you're covered in every every possible way. And the nice thing is my turnaround time, other than when the initial when the S390X was down and we had the two two-day issue, uh realistically my turnaround time is about 8 hours for a kernel right now. That is the reason So, we do have um if you wanted to do say an RT build uh, if you wanted to you know, real-time build, if you wanted to do a All these things are in the spec file, they're fine, they're enabled. The reason we don't ship them in Fedora is because exactly that CVEs, I cannot start a build and then wait for six builds to happen because it's all serial. Um so, hopefully at some point that will change, too. We've asked for it, but um and you know, in the meantime, it's like these things are defined, they're enabled. You can build your own kernel if you want to do these things. Uh, in fact, even in the um in the kernel art tree I maintain a a long-term stable branch. When I say I maintain it, I'm not backporting the CVE fixes to it, but it's there. If you want to build it, um whatever is upstream is is guaranteed there and it's merged with the Fedora patches, so it works as a Fedora kernel. Uh, I just don't Like I said, we don't build it and ship it. So, uh, that and that's one of the reasons. Like I said, the CVE fixes I try to get them out as quick as I can. Um, and sometimes that means you know, it it's 2:00 in the morning when I'm pushing a fix or uh, I There's Justin who I test. He was He was He was talking to me about some of the AI stuff. It was like, you know, 2:00 or 3:00 a.m. and I'm waiting for builds to finish so I can push them out. So, we we do try to get them out, but it's it's um I'm not sure how the community can help in that method. >> And just a quick follow-up, how we can help is if anybody is hacking on Koji, we have a patch for a running these six builds in parallel in the work for a three or four years. >> Yes. >> We have no hands now, so if someone wants to take it and continue on that, we will be happy. >> Yeah, we Well, and we've asked for it for a long time. And actually, uh Terrell also has had these problems, and maybe that will force getting that patch pushed. Because the reason it was it was delayed for so long is is, you know, supposedly everything's going to complex. The problem is the kernel is still not in complex, and now we have these these SLAs, so we'll see. Right. >> Just saying that the complex will have the exactly same problem, so we will need the another patch there. >> Yes, absolutely. >> Um I assume you don't have anything surprising to say here, but it has come up. What is your recently What is your opinion on building alternative kernels within Fedora? >> So, the policy against alternative kernels um exists for a reason. And the big reason is, you know, from a support standpoint, um I you know, I don't just build kernels and push them out, right? I Every bug that comes in comes to me. I read every single bug. I promise, even if I don't comment on them, I read every single bug that comes in on on Fedora kernels. Uh the problem is when you have multiple kernels to deal with, and I have to process, right? Well, which kernel is this against, and which is this um That's why we don't do them. So, like I said, I I'm maintaining a long-term stable branch for the people who want to build it themselves, and that's fine. But as far as a built, signed, secure boot kernel for Fedora, I am pretty strongly against adding another branch, and I I don't see anything that's going to change that. Um even talking to the to the SUSE guys, I don't know if Shawn's in the room, but yeah, know, was saying, "Oh, yeah, long-term stable doesn't actually seem to fix anything for them." They added it. They They've got it, but it doesn't fix anything for them. >> Actually, it makes it worse most of the time. >> Right. So, I mean, unless like if I knew that it was going to solve a lot of problems, I might actually push for it. And frankly, since it's just me, I can kind of make it happen. But, I haven't seen it solve any problems. >> So, on the it's it's just you thing, you're awesome. Thank you. But, if someone would like to get involved and help, you know, so that maybe you don't have to read every single bug or or some you can you can you'd feel comfortable that somebody else is there. How would they get involved in the kernel in Fedora and helping you out? >> Right. So, there's there's actually two things there, and I'm actually very hopeful soon that it will not be just me. So, I don't know if you've like Augusto has been doing um builds. But, doing builds is not much of the job, right? Uh it doesn't doesn't take much off of me. I believe Augusto's going to be trying to do a little bit more there. And then, we do have someone else who's coming in to the rail kernel maintainer team from support. He's been in kernel support for for 10 years. And so, the biggest thing is like my Yeah, like I said, the builds are are not the job, really. It's the bugs and dealing with the bugs. Uh so, somebody wants to jump in, look at Bugzilla, look at open bugs against kernel, and see what you can do to help. You If you tell me you're working on a bug, please do. Uh if you want to submit merge request, so like all of these branches do exist on kernel arc. Um you can submit merge request, and I am more than happy to review them. I would love to have your help, please. Uh any anybody who's willing to help. >> Jason from the kernel >> Sorry. It's [clears throat] It's not really a question, but just following up that I get to work with a lot of different kernels and distros and you're doing an amazing job. And for getting fixes for new hardware in, thank you. Really appreciate it. And for anybody looking to help, I do the merge requests for fixes for our laptops and yeah, it's amazing. So, thank you. >> [laughter] >> No, thank you. Because cuz realistically, I mean, I want Fedora to work on everything as well as I can, right? And my my my goal is I I mean, you're a great vendor, but but the truth is like any vendor, I I just want all of my users to work and you make sure that they work, right? You are sending us patches, so I'm I'm grateful for it. I love it. >> [laughter] >> I try to be. We don't We don't have all of the necessarily overhead that that well does, so I can kind of do some of those things, but yeah. I I really do I want to make it as easy as possible. If there's something you want, you know, even with config changes, you know, like I've had people ping me on Matrix or IRC and say, "Well, can you change this config?" And I'm like, "Well, yeah, um you could file a bug, but I could just do it right now." And it doesn't take anything. >> Uh, we're out of time, but please catch up with him. >> Yeah, I'm available. I'm around. >> Fedora >> Thank you. >> [applause]