Flock 2025 Finding And Fixing: The Fedora Kernel s Impact On Upstream Linux

Hello everyone. I'm Fernando. I'm here today to talk to you about uh my experience and my adventures in the kernel and more specifically in Flora kernel. So all right a bit of context. Last year after um at Rochester I was discussing with several uh community members about the kernel uh that I had interest in it and also how I could help. So as an aspiring internal developer I just uh I was already familiar with it because I contributed the past but I was not super super proficient person and especially not so okay. The easy path to me was to go to maxilla to insert my component component and start taking a look to the to the bugs. So yeah, I had a plan. The plan was look at the back, find the bug, fix it. Go back to look at Billa, find the bug, fix it. Sounds easy, right? Super straightforward. As you can imagine, right now that isn't the case. Um, when I started to look at the I found multiple kind of bugs up and I started to realize that I could not work on all of them. As I could not work on all of them, I say okay, so I need to organize them in a way that allow me to easy identify which ones are global by me and which one are global by me. So I created three different categories. In essence, we have fed specific bugs. So these are kernels sorry these are bugs are kernel that usually rely on federal configuration or feder RPM or fed back door or for example and you might uh heard about that GCC version updates in Fedora and therefore the kernel is not compiling anymore in Fedora that kind of bugs. Luckily in Fedora uh there is a lot of people working on these specific bugs and they usually fix super fast. They are very experienced. Also we are on them. They do a really really good job and they are doing very great. So I felt like I was not fast enough uh yet to deal with these kind of bugs because usually there were other people much more experienced than me in federal colonel that were able to tackle them tackle them down pretty fast right so then we have another category upstream bugs so the upstream bugs are usually bugs that break on a specific kind of version and they are not related to a very specific federal configuration just some configuration that's enabled the module and it's usually related to a core part of the kernel which is not a hardware driver and why I mention this specifically because that's another whole section drivers. So bugs are obviously kernel supported drivers also out of tree drivers and they usually are hard to debug without the hardware because you cannot reproduce them. You cannot make sure you're fixing them and it's super hard to work on them. So having these three categories where should I focus? So for the federal specific backs as I mentioned we haven't really handled very well. So let's Uh, move on. Then we have the driver bikes. Sure, I could work on driver bikes, but right, I I have a laptop and a workstation. I have one motherboard, one CPU, uh, one graphic card, one network card. I don't have hundreds of them because believe me, the amount of different drivers that I load into is super huge. And obviously it's possible for me to have all the drivers. So sure I go I try to filter uh by the specific drivers that I have access to and unfortunately there was not any bug or luckily because I was not experienced any bug and then I took a look to upstream bugs and I found some serious bugs that were upstream bugs. That means these bugs were not only affecting were affecting every Linux system that has this kernel version. So real examples uh I'm going to show now some real examples where I worked on uh and they were fixed in Fedora and they were handled uh debugged and everything with Fedora mine and the fix landed in Linux upstream which ended up helping distributions as well. So by the way I use federal row height. So I want to clarify that uh I found these bugs and they were in essence this is a here I have the four that are more important than some inner issues. The first one it was an UDP socket moveing listening bug. So in essence, if you had an application that was listening on a UDP socket uh with multiple threads, you were completely broken because in comet kernel uh broke it and suddenly all the all these applications were really only using one single um specific thread and the other were completely ignored. And if you were unlucky enough and were using some socket options, it will just not listen on it. So that's a problem. Then we have some IPv6 omitted information on IP root 2. And then we have one of the most important bugs we recently faced which is x86 architecture was only seeing a single CPU uh when IO APIC is disabled. And I don't going to get into specific, but believe me, a lot of laptops out out there are using uh are compiling the kernel with the this config uh disabled because otherwise the kernel won't boot on their um hardware. So these users were only having one CPU. And then uh I found uh x86 warning of a possible deadlock with federal row height. that this uh this possible deadlock was at the end a false positive. So it was not a real uh threat although uh it's important to also fix a false positive. So all right me results I got five patches merged into Linux kernel. Uh all this include a process of compiling the kernel, investigating, understand um what is happening, why it is happening, try to reproduce it in a virtual machine, try to reproduce it on bare metal and so on. And then these patches landed into multiple subsystems like netv x86 tracing etc. And uh these issues and especially the x861 was also reported in other distributions and they got the fix too. That's a very important thing because all of them referenced the Bakilla uh Fedora Baxilla bug where the work was done and say hey Fedora fix this and that's a very good thing that hey Fedora is fixing things for the others and we are collaborating and it's uh from a in my opinion from a distribution point of view it's important to be there and that people notice that you are contributing back and that you are not only taking care of the things that affect to you but that the things that affect to you and also to others and you are not the one waiting for others to fix your problems. So, yay. And also extra points when working on the uh CPU uh issue. I got the there was a whole discourse um thread about it because believe me, the problem was affecting a lot of people and someone took the effort to post that we don't care about bugs and that it's a waste of time to report them. One week later, the bug was fixed thanks to this this discourse and this report. So, I'm here to call out this comment and say no, it's not a weight of a waste of time to report bugs. They get fixed. Sometimes they don't they do not get fixed but because we don't have the resources of the or the people to work on them but they are valuable and and I I want to completely discourage this kind of uh comments or this kind of behavior reporting things it's very valable it's the only way we can know things are broken >> I'm not sure >> yeah the important is it on yeah the important thing is to report them at the right place because many kernel bugs are reported in the wrong place. That's what people should keep in mind for the uh AMD GPU that's for example a free desk free desktop issue tracker. If you report them to bzzilla.kl.org they will likely be not even seen by the developers. >> Yeah. So about that I'm going to go back to to answer that in a minute. >> So now your turn. All right, just to add one of the things that we need is that just to to get this understanding first, the kernel community upstream recommends that when you face a bug, you should report it to your distribution. Then someone from your distribution should take a look to it, analyze it, understand if this is a kernel bug or a distribution specific bug. If it is a kernel bug, report it upstream. That's ideally how it should work. So one of the things that we need is people triaging these Fedora kernel bugs because that is not happening and there are thousands of them and I'm not exaggerating it's literally thousands bugs and we need someone and ideally more than one person to take a look say oh right this is related to this config this is related to this driver and report it back and say hey uh Linux kernel community. Uh I noticed this specific bug about x86. I noticed this bug about this driver. I noticed this bug about tracing whatever. And then if they do not handle it, we cannot do much more. Of course, it will be very good if you are an aspiring kernel developer to work on it if you are able to. that from the point of view of a distribution what it is expected is just that they report back to upstream the bugs that are really an upstream bug and I didn't decide that that is how the NOS kernel community work and so yeah we need to someone to take a look to the bugs categorize them them and then if you have the hardware mentioned it's also very useful to volunteer so if you notice there is a bug reported to against any of your hardware you can say hey I don't know how to fix this because maybe I don't work in kernel I don't know C I don't I'm not even a software engineer it's fine as long as you I can give you an RPM and you can compile it and say hey this is happening hey this is the lock hey this is fixing the issue it is not fixing the issue whatever but having volunteers with hardware is very very useful and and it will be very nice to to to get more volunteers on this area. And then what if I'm not experienced? What if I am a software developer or someone that is willing to learn software development or or someone that wants to help but I don't know how. No worries. I'm happy to mentor. I'm I'm really are um reach out to me uh on matrix reach out to me on email I don't really care you can reach out on kernel uh room uh in fedora project.org work whatever reach out. Hey, I know Basc I have uh this experience and I want to continue uh contributing and I want to start contributing to Cano. Great. We will look together at the bugs. I will try to help you as much as I as I can. Of course, I I am not a super hacker that can fix everything, but probably can give you some guidance of how to debug most of the things. And with that guidance um even if I do not have the knowledge I will probably give you a contact uh contact point in kernel upstream of someone that can help you to debug it usually and this is probably a controversial opinion kernel community is welcoming and I know that that is not the general opinion but that was my experience so far and uh I try to be well for me. So if you want to help, please uh step up, reach out to me, reach out into kernel federpro.org and we will try to yeah get you started. If you don't know C, it's completely fine. We can give you some resources to learn C. You can practice with simple bugs. Uh I can just show you how to compile it. we can learn together and yeah next steps in federal so I want to work with feder community uh to launch an official internal mentoring program so in essence it's not like or richi or gock where we seek out uh interns from from the outside like students or people that just get out of college or whatever but we are more like um we have people in federal community that already want to do this work. Let's try to help them. And then I would like to coordinate this with the federal colonel QA. Uh there is some people at Riad that do a really good job and they they they really they really help a lot and also the kernel maintainer that we have that also works at at Ryad and he he does a really good job. So I would like to coordinate with them so we can easily get people on board. And the aspiring point or or the final goal that I visualize on my mind is that Fedora kernel to be an entry point for aspiring kernel developers and that when it's really hard to get into Linux kernel development and into Linux kernel community it really is but I think we could make federal kernel the place the the the start point for these people that wants to to start and that's all folks. Yeah. Any question? All right. Maybe it's a dumb question. I don't know. Um you had three categories, right? Upstream, Fedora, and drivers, right? How how is support done today for folks who are having issues with driver based issues? Is that things that we just kind of go out into like the ask fedora or just ask who has this or or how is those types of bugs usually dealt with or or are they deal with? >> Yes. So usually uh unless there is someone already in federal kernel that has that hardware, we actually um do not do much about them. We try to report them upstream. That's it. This is why we want more volunteers. If we get kind of uh a list of people with I need to think about it, but it would be nice to have a list of people with uh hardware like a map like this hardware. These people that they need to be obviously volunteers and people that we could that that we know that we can reach out to them. It would be very good because hey, we got this back about this uh AMD CPU and this person has this AMD CPU. Would you like to take a look to it? Something like that um could be great. Right. More of a comment than a question. Following up on that, um, because those bugs are hardware specific, >> is there a possible attachment point to the Fedora ready concept where we could sort of maybe organize people who have specific vendor hardware that has specific drivers, specific hardware into sort of sub communities? Um, is that something that that does that make sense around some of the driver bugs that help identify the potential pool of contributors who can triage things? >> So for for the partners that we have at Federal Ready, I think they already do some work like they take a look to these bugs, but usually they do not cover the wide variety that we have on our users, but I believe they they >> right the the vendors do. I'm saying, >> but is this an opportunity where >> we can help >> vendors move into that program too? >> I hope so. >> Uh if you finish uh it's related to that. Um the question is why are you doing this in Fedora at all? I mean of course it would be good to get Fedora involved in this but Arch Linux and open sour tumbleeed basically use the same kernel. So why not collaborate directly on upstream to find people that have the same hardware uh with support from Fedora and work together on a fix upstream because that's where the fix needs to go anyway, >> right? Good question. Um there is collaboration upstream uh a lot of times but this sometimes is is really hard like um people is people find a community like Fedora more welcoming than the kernel community and therefore they are whining to step up or to collaborate more in Fedora than in directly with kernel. Also with kernel we mix sometimes kernel developers and and users and there is some misunderstanding sometimes like people oh sorry people people that speak a really technical knowledge to people that do not so we the distributions we as the distributions are like the middle point there >> yeah as somebody working on upstream in this area it would be I did don't I think you're right there are some problems there but you don't have to do all everything upstream the collaboration with Arch Linux and open sus Tumbleweed that would be that would be important because they have basically the same problem yeah so now I'm now I'm stopping >> yes so so so yes to to give you that you're completely right this talk could be in essence in any kind of um conference distribution focused like arch conference or whatever uh open source conference they have exactly the same problem but uh it turns out I we are at Fedora so I contributed in Fedora but uh yeah you are completely right. So about finding volunteers to test uh the packages usually the reporter is the um hardware that reproduces the bug. So is there a problem to get the reporters of the bug involved in testing possible fixes? >> Yes. Uh there are two different kind of reporters. Uh yes, there are reporters that in essence collaborate a lot and they even sometimes they they provide reproducers. Yeah, that's uh really good. They provide a reproducer so you can play with it. But I must say that most of the reporters are people that have a crash or whatever they use the automatic tool for reporting. They send the back at the back and then they disappear. I tried to reach out to some people. Uh there were like two bucks in wireless card that they were I believe they were kind of simple but I could never test them because uh I reach out to them like hey would you like to test this batch and I never get a reply back. So at the end you need people that sign up as a volunteer so you know that you can rely on them and say hey can you test this? Of course, the first option is always a reporter, but my experience that they usually disappear. >> Another question. >> All right. Yeah. So, thank you very much uh for attending. Thank you.