Flock 2025 When And How To Use Upstream Linux Kernels To Fix Or Improve Fedora

Yeah. Okay, hi everyone. After some uh changes here for the uh plenary session later, let's start with my talk. Uh welcome everybody. I'm Thorsten Leemhuis, and I want to talk to you about when and how to use upstream Linux kernels uh to fix or improve Fedora. Um uh important things first, uh Fedora's kernel maintainers are really doing a a a uh a great job. Uh the Fedora's default kernel is great. Uh it's basically the fun the right kernel for everybody to use. Uh yeah, but so often in life there are exceptions, uh special cases where you want to diverge from what a distribution does. And one of them, for example, if you want to efficiently prevent uh kernel regression from reaching Fedora. Uh another one is actually when you if you face a kernel bug and uh you want to report efficiently. Um and the third one is actually if you want to bisect the regression. So, to explain the difference between a bug and a regression, a regression is when something stopped working that used to work. And a bug here in the talk is includes a regression, but basically is often something that never worked in the first place. place. There are three other cases where you might want to use a different kernel, but we ignore those for now because those three are the important ones we care about. And if you are running in one of those situations, we have you have basically two options to run different kernels then. One is to install them via RPM and DNF. And there are actually two flavors to do that. Um one of them actually has uh offering various kernel series, so you have basically it's the most flexible. And the other one is basically installing manually. That might sound dangerous to some, but it's okay sometimes because basically bypassing DNF sometimes is a good idea. Yeah. Uh what's the best approach actually depends on what you want to do. So, let's take a closer look at when you want to use upstream Linux kernel to fix or improve Fedora. So, let's start with that. Um it was as well and start with something that sounds a little bit scary, but it isn't. Um the thing is Fedora leaves actually handling the vast majority of bugs to upstream. That sounds really like something bad, but it isn't. We get to that. Um um it's actually something good. I don't have numbers to back this up, but I'm used to work as a Linux kernel regression tracker. I didn't discover it stalled at the moment because I lack funding. Hopefully that will improve again or otherwise I need to to find another job. Um yeah, but that Fedora leaves things to upstream most of the time is as I said it doesn't mean at all that Fedora is doing something wrong. Fedora cares and you can see that by a change that was basically last week or maybe 10 days ago. There was a big bug in a in a new stable kernel that went out from kernel.org from the upstream developers that Fedora actually fixed or avoided by reverting the change and in in their kernel. If you want to see what kind of patches Fedora actually applies follow that you URL that that the patches are actually listed. Yeah. So, again that Fedora actually leaves things most of the time to upstream is actually a good thing because cooperating with others with others is in everybody's interest. Especially so for our friends at at Arch Linux and openSUSE Tumbleweed because they are using basically the same kernel as we do. Um it's just a different configuration, different packaging of course, but in the end it's basically the nearly the same kernel apart from a few patches and doesn't make a difference. So, any bug we have they face also and they also need to fix and instead of doing it three times in these three distributions or maybe even more, it makes sense to work on upstream and doing it in one place where everybody gets their kernels from anyway. Um like in every project uh upstream has its problems and things could work better there, uh but that's how it is and maybe cooperating uh with these better cooperation among these three distributions would be really really something that could help everyone here, but uh yeah, that's a different topic I I can't uh go down here. Another thing why it's good that Fedora leaves things up to most of the time is that fixing too many things on Fedora side can easily can lead to a hard to maintain mess. Those that look at the rail kernel actually likely know what I'm talking about because it's so so many patches, interdependencies, and it makes it hard if you want to rebase to a new kernel. So, it's also a reason why you want to leave to upstream. And uh yeah, every man in the middle obviously complicates things. So, if you have a bug and you report it to Fedora and expect the Fedora kernel maintainer to report it upstream, there's going to compression artifacts if if you want to say it. Things get lost and it's often better to directly talk to those people that um have to fix things anyway. Because Fedora can't often often can't do much in any case. That's because the kernel is mostly drivers, about 50 to 60% of code is I think it's it's drivers. And Fedora developers simply have no access to the hardware. So, they can't reproduce the problem or debug it and the thing is even upstream sometimes lacks lacks access to this hardware. Because yeah, hardware is tricky. I for example owned a Dell XPS 13 laptop. I think it was the model year 2017 and then I got a regression report from somebody else that basically had the same model and he had really great problems with the graphic drivers. The display stayed dark on boot up and for me everything worked. Yeah, it turned out it was nearly the same model, it's a different configuration. It had a higher resolution display, and that confused the Intel graphics driver, and then nothing worked. Yeah, yeah. That's just one of thousands or 10,000 of tricky problems that happen with hardware every day, and that's why testing on on real hardware is so important, and and why it's best to consider your system to be one of a kind. If you don't test it, there are likely there might be bugs that that nobody noticed, and yeah, because they can't simply simply be tested in VMs or CI's those drivers, or most of them can't. So, in the end, if if you don't test it, there's a pretty decent chance that some regressions actually might be not noticed at all before it's too late. Too late in this case means because they before they end up in the in the kernel.org release and new Linux version, and from there end up in in in Fedora in the in the kernel update. And if they are there, it's gets harder to remove them. Another reason why Fedora often can't do much is the kernel is a complex beast even when it comes to things like that are not drivers, because it's really the kernel is at the core of the system. It's doing a lot of things, and yeah, we only have a small team of Fedora kernel maintainers, and they um everybody can can only do so much. So, it's basically um often other upstream developers skills required to to debug and fix the bug. Fedora's kernel team um could also obviously could fix more bugs if if it was bigger, but it's actually quite small even with if it was be three or four times bigger. It likely couldn't take care of many bugs, because that's just an an enormous amount of work. Leaving things upstream is basically the best idea. There are also some of you might wonder, yeah, we have the kernel test weeks and we have the updates testing repository where our new kernels are tested before they're going out as upstream. But, the thing with those two is they can only do so much um because they're basically testing versions that are already out. That's great and they find bugs that are fixed. Um but, often it's um it takes two two to six weeks to debug some problems uh or most problems and probably fix them. Um so, at at some point um um there are simply no other choice to Fedora to to um to ship new versions that come out in between because they bring important security fixes. And yeah, so Fedora has to ship them out as despite knowing about regressions in there because the security fixes are important for some users or because old versions don't get security fixes anymore. That's a problem for Fedora. That's uh yeah. What does it all mean in the end? In the end, that means your best choice your best chance to prevent regressions from reaching Fedora is actually um is to test upstream kernels before while they are still in development. That's basically either the mainline kernel, that's the one that Linus Torvalds maintains like 6.15, the version that came out like 10 days ago or the next pre-release first pre-release of the 6.16 um kernel, that's um the the pre-release is coming out on Monday and the release is still weeks away. Because um yeah, and all all testing the the stable RC kernels that where new kernel versions from stable series are tested, those are those that end up later in a Fedora as kernel update. And there you have a little bit of time to to notice problem. And if you notice problems in these uh kernels, there's actually um um if you report them, the the upstream developers simply often kick out the the changes that cause regressions if they are known, that requires a bisection or uh somebody knowledge looking at it. Often sometimes uh that that uh the latter uh suffices. Yeah. And uh um they could simply kick this um uh um changes out there, but once the um a change that causes a a problem actually made it in into a release, it's not that easy anymore, then it often takes longer. That's why testing in the in in development versions of uh really is a good idea. Um ideally report uh those regressions encountered with uh in the in development kernels upstream because then yeah, no man in the middle and uh you immediately reach the people that can do something about it. Uh obviously with uh if the problem is somehow relevant for to Fedora in in a big way because it's uh breaking things left and right, it uh then you want uh to keep Fedora in the loop. For any regressions that made it into the kernel that uh Fedora actually uses and ships, um there you want to also uh in my opinion ship uh report it upstream, but obviously you want to let the Fedora kernel maintainers and the other um Fedora users know. So, it's best to keep them loop in the loop and basically file a a Bugzilla ticket and link to the upstream report so that everybody is on the same page. Yeah. Uh reporting other bugs, uh so things that never worked in the first place, uh reporting those to Fedora's in my opinion, if you ask me, is really a waste of time because Justin has way way way enough to do already with all all those stuff that's happening. He likely will never uh uh look into this. Maybe even upstream doesn't care uh about it. So, um you really should report those upstream and don't bother bother uh with it uh um with uh bothering uh Fedora with it. Yeah, it's just a waste of everybody's time. Obviously, um with all that I just said, um that just for the kernel and things that are happening in upstream code, everything related to packaging must obviously be reported to to Fedora, for example, patches caused problems caused by patches Fedora applied or the kernel configuration, placement of files, bootloader scripts, etc. and all this stuff. If in a doubt, simply ask for advice, for example, in the Matrix channel on on on on on on the Fedora channel server. Yeah, this concludes the when aspect and the the slide show even gets faster at this point, I hope, because there are still 30 minutes left and brings us to the point how to actually use upstream Linux kernel to fix or improve Fedora, which brings us back to the three points I mentioned earlier. The first one was how to prevent kernel regressions regressions from reaching Fedora. And one solution there is actually use a raw hide kernel either by running a raw hide all the time or installing it on on Fedora releases. The instructions how to install them on Fedora releases are at the end of the slide deck in case anybody is interested. I kicked them for for timing reasons. It works fine for many people. There are a few downsides. Um Yeah, the patches can can become a problem if you actually run into a problem want to report it. Sometimes it takes a while from upstream to get them downstream. Most of the time it's just 24 hours or a little bit more of a day delay, but sometimes it's occasionally it's it's a a week or something due to raw hide composers not working or something like that. Um there are sometimes additional dependencies that's not much of a problem. A came out and work in and technically but only often 10 out of 12 months because raw hide sometimes uses a different compiler. For example, like it was when Fedora 44 was prepared. And but the most important thing is actually that mainline for technical audience like the ones that you are and might not be the best fit because you want to avoid the most risky phase in the development cycle. That's actually the two weeks right after new release. It's a merge window it's called. So during that those that time not it's might be a good idea to not use the mainline. And that's actually possible with a more flexible alternative that's avoiding these down sides that's installing the kernels from the kernel kernel vanilla repositories. Um those are actually seven different coppers. These days the URL where the table is from is actually from here. Don't take a look too close a look. It's a lot of text and we get to a few important bits. The the important thing is it's those kernels and the RPMs shipped there are basically like Fedora kernels. And they're nearly built exactly from the same mechanism Fedora uses to build their Rawhide kernel and their their release kernel but it was without the Fedora specific patches. And yet for it's following upstream quite closely. So basically every morning your every European morning I I there's a new build and that's basically just a few hours behind Linus state of the art. Um there's one big downside here that lacks a trusted UE UEFI secure boot signature. So most modern PC won't consider it trusted and boot it. That's actually not easily possible to support this in copper. And that's how it is. There's nothing that can be done about it. And the other thing it's not maintained by the Fedora's kernel team. It's maintained by this guy that got a little bit more gray hair I think in the last 10 years. Yeah. So it's kind of so take everything with the gain gain of salt because those coppers are actually maintained by me. I I'm doing it for for 10 years now in the beginning without copper but these days I'm doing it with copper. For example, one of those coppers is actually the mainline copper is basically just like like raw hide kernel in the end was but without the patches. But for you people folks here actually the the more more interesting ones are these three. Actually the the one I want to talk first about is this mainline without merge window copper. And actually is a ships the kernel kernel version that within one to 13 weeks will become the base for kernel update for for the next Fedora for for for kernel update in for Fedora releases. So that really leaves a lot of time to test things and report problems and get the changes that are causing regressions out again before it enters the release and so to make sure the problems never reach Fedora either by by fixing them properly upstream or reverting the change. The copper looks like basically like every other copper. And enablement is also not different. Just enable it like any other copper DNF upgrade. And yeah, apart from the last step this is using mock mock you util to check if secure boot is on. And if that command tells you that secure boot secure boot is on you need to disable it either in your BIOS setup or through a process you can also initiate with mock you util. The the page I link to has details on that. Yeah. And if you say okay, yeah, I I'd like to help but mainline it sounds a little bit dangerous because there are lots of changes there. I I like to get something that's little bit closer to what Fedora uses then the Fedora Fedora RC branches actually the Fedora RC copper is actually the better one. It ships a kernel that basically uh round about 3 to 10 days later um, will become the base for our kernel update in Fedora. That depends on the timing where in the week we are. And um yeah, as a the the problem is the RC phase that's just 3 days long. So, if you really wanted to help test, you need to reboot and uh immediately report problems. Um, but maybe that's something for some of you that's still interesting because you can um uh um really avoid many problems there because yeah, the the kernel basically just 1 week later will end up in in in uh Fedora likely with small change like the the problem of the AMD graphics uh that multiple people hit that I mentioned earlier that was reverted that could have been avoided if somebody had at that stage um uh tested the kernel. There's also the stable RC uh um uh copper that's basically the same and uh most of the time ships the exactly same kernel version as Fedora RC copper uh except during the three or four weeks right after a new um mainline release that's uh like uh uh the state we are currently in like 6.40 15 was released recently. So, right now it ships that version or a stable version uh based on it like um 6.4 15.1 by now. And uh and uh so you can test that and that will um basically end up as uh the base for our kernel update that or later version will be the uh in in like uh two or three three weeks three weeks from now. Next week is a tech test week and then usually the week after or something like that it will go out as regular up to date update to Fedora uh 41 and 42. Yeah. If that all was a little bit fast, look at the wiki page. It explains uh the different coppers. It gives a few examples of the versions then then it's easy to understand. And uh if you wonder here, how dangerous is it for me to run uh um um uh these kernels from this copper. It's nothing much to fear. Obviously, it's a software, something can go wrong. But the thing is the installation is not a problem because if the kernel you install doesn't boot, the one you used beforehand always remains installed so you can simply choose it in your grub boot menu. And I'm sure people like us all have reliable backup and restore strategies. Do you? I mean, lately I had small problem and then it was oh, right now my backup is broken. Yeah. But it turned out to not be that bad. So it's a good reminder to to have your um backup in place. Yeah. There's also instructions for uninstalling the the the kernels from those coppers. Um It's basically just disabling the copper, syncing the kernel to the Fedora kernel, and rebooting, and um and removing the vanilla kernel packages. No no big deal. That works quite well. That concludes the first part, brings us to the second part, and time is running hot. I need to speed up even more. Um the thing is Fedora's kernels are often not yet The next thing is then you you is the thing I not mentioned earlier that you face a kernel bug you want to report. The thing is Fedora's often not ideal or even unsuitable base for reporting kernels upstream because some upstream kernel developers reject reports from cat patch kernels. It's a a big problem when it comes to Ubuntu or L. For Fedora it's not big problem, but the other problem is that Fedora's kernel often are considered a little bit too old because development sometimes is already a little bit ahead. And yeah, that's why you might actually use mainline repository to report bugs. And if you have a regression with a stable kernel, check if if if there's already a newer version that might fix this in the in the pipe, um which is also uh this process is also important to check mainline because it's determines who's obliged to look into and fix the issue. Yeah. So, the rule of clump is uh in the case of a regression in a stable kernel that Fedora ship, test the stable RC copper, test uh mainline, and then uh you can uh if both show those problem or one, you can report upstream and you know about uh whom to contact. Um the upstream documentation on reporting bugs um actually covers all this. Um I know reporting bugs to the kernel is a bit frustrating and hard. Um yeah, but if nobody do does it, uh we'll have uh we'll end up with lots of regressions in Fedora. So, really uh um uh it's good to get this uh out. And especially one thing I'd like to note, don't use blindly use bugzilla.kernel.org because some subsystems ignore it, and you might want uh uh to check where to actually report problem. Mo- most, yeah, yeah. Yeah. So, the third um issue where you want to use a upstream uh Linux kernel is actually if you want to bisect a regression. That's basically using Git to check um um which which which change broke things. Um so, you compile like uh 15 or 17 kernels for when it comes to a range from like if you have a regression from 6.14 to 6.15. That sounds like a lot, but it doesn't take that long. But, then you find the change that is um uh broke things. And there my my I'd say to you, just bypass RPM. That's not bad. Uh just compile the kernel as you would do as it's uh described in the upstream documentation. The thing is upstream documentation I actually wrote it. I tested it on Fedora. It works. It actually also has uninstall and install um um instructions. So, you need to remove one directory, and um then they are away. And um uh run the kernel install command um and that's the same kernel install command that's actually used by the spec file to remove the kernel, so it should be clean, yeah. That brings us to the space in the middle where you uh where I mentioned three other problems. One is where you might want to use upstream kernels. One is you just really need a uh kernel features not that hasn't reached Fedora yet. Just use the copper that ships it. And two other situations as you require a kernel patch and you're a kernel developer. That's basically use RPM or bypasses depending on your needs. It depends. Some of you might be interested in Archify. That's a little hack I wrote that fell off on the side from the stuff I'm doing to to maintain the the the coppers to the kind of vanilla coppers. It can actually add the infrastructure to build us source RPMs to any Git repository and um and then you can call build them with with mock or koji or whatever you want to do. Yeah, those are the six situations which actually brings me to nearly in time to the summary of the talk. Yeah, let me repeat again Fedora kernel Fedora's kernel is great. It's it's it's almost always the best choice to use. When it's not, it's not that different to to install or uninstall different kernels on Fedora. That works both with and without RPM. Uh when you need to bisect a regression, yeah, just bypass RPM. That's my advice here. If you want to report a kernel bug or regression, use the use the RPMs from the copper or maybe even if it's too hard for you, try a report bug with the with the kernel Fedora RPMs. Uh But what I really want you to do is as to help preventing regressions from reaching Fedora by constantly helping to test upstream. Um Uh that's why I'm standing here. That's why I gave this talk because uh Linux needs people like us to do this because hardware is so complex and uh uh um we need to the kernel needs to be tested in the field to to detect this regression. Uh you can't do that in VM. So, it's need people like us that are technically interested and to do this. So, it's your your chance to give actually back to the Linux community and it doesn't cost you much as long as everything works. It uh it uh it's just working like in every every other update. So, and yeah, Fedora can benefit a lot from it. And our friends at Arch Linux and openSUSE will also and upstream. So, just do it. And yeah, with that I'm at the end of the talk. I think there was about 150 slides. There's question? Maybe just ask an an I I can repeat it or >> One second. >> Yeah. >> So, currently when on the template in Mozilla when there is a kernel report, there is a section that says, "Have you tried the Rawhide kernel?" And I was thinking that maybe it's also a good idea to mention, "Hey, have you tried to reproduce this with one of the corp vanilla kernels?" Because also it's a not it's a non-tainted kernel. So, >> Where where is it on on the >> On the Mozilla. >> On the Mozilla >> On the kernel component. >> Ah, okay. Uh one of the Ah, okay. Yeah, the >> sections on the template is, "Have you [snorts] tried Rawhide?" >> think Justin is the better person to ask. I guess the text can be adjusted to do that. >> The text can be adjusted to do that, but from a bug reporting standpoint, um running the Fedora kernels isn't necessarily that bad, and I'll explain why. Rawhide has a lot of patches that stable Fedora does not have, but most of that is rail stuff, and it is actually if deft out to the point that when you are compiling a Fedora kernel, you will never hit those code paths. Those code paths are built for ELN repositories only. Uh now, when I do stable Fedora, I pull all those patches out, and the reason is um because when you get a patch from upstream, upstream says, "I'd like you to test this." If there's a lot of context diff, it might not apply, and people just don't know how to mess with that. Uh but from a reporting standpoint, we are much closer to upstream even with all of those patches cuz we're just not doing those code paths. So, you can report it, and if it works in Rawhide, it's easier than perhaps do another copr and all those things. Now, when you're actually trying to debug it and test patches, that's when this stuff becomes much more important because if, you know, people see a the patch doesn't apply, and they don't know what to do, that's a probably not going to happen here. >> [laughter] >> I think is it Yeah? Yeah. So, um I had this problem yesterday that my machine crashed uh when I plugged in the projector. >> Yeah. >> And um it's a tough thing because it hasn't crashed in 8 months, so um I wasn't accustomed to rebooting. Uh so, maybe I would like to figure out what went wrong and and report it, but uh I don't know. Like, do you have any advice how to how to try to debug it? >> um bugs like that are hard to debug. Um sometimes um uh what I would do in this situation is test the latest kernel if it's maybe working better there. Some Maybe it's a bug that was reported already, and somebody fixed it already. So, basically, mainline, test latest mainline, and if it's not, maybe install even older kernels again and check when it started, and then you can actually using uh um a Git bisect to find when it starts broken. At least if it's a reproducible mistake, yeah. Yeah. easier Yeah. Yeah, that's problems that I don't reproduce that those are problems. Yeah, I guess we're running out of time. I guess Yeah. Yeah. So, thanks for listening. Have a nice rest of the conference, and yeah.