LF Live Maintainer Session: My Life as a Linux Kernel Developer and Maintainer with Mark Brown

Like that. >> Thank you, Candace. Uh it's my pleasure to welcome you, Mark, to speak with me about your uh developer experience and maintainer experience and everything else you do. Uh would you like to introduce yourself before we get started? >> Um sure, I guess I can. Uh so um I'm Mark Brown. Uh I've uh been working on Linux for um a couple of decades now. Um So, which is uh terrifying thing, but uh in various capacities. Uh originally, I got involved in Linux uh as a uh Debian developer, um but then at work, I started you working on the upstream kernel um as part of uh developing some embedded products we were working on, um after which I moved to um full-time kernel development at um at Wilson Microelectronics, which was a chip manufacturer, before moving to uh Linaro and now ARM, uh always uh with an upstream focus. >> And embedded also, I think you uh logically, is that good for >> Yeah, yeah, like like uh yeah, so like like I said, the original thing was we were a product that used Linux, an embedded product that used Linux. Uh so, we we needed some changes to some of the upstream drivers. >> Right. And Mark, today's webinar, you can find it on the LF Live Mentorship Series. You'll find the debugging embedded and and also. So, yes, um uh So, what was what what was it like when you first uh started contributing to the kernel in terms of what was your first contribution? Do you remember? I mean, after 20 years, it's hard to remember after so many patches. >> Yeah, I want to say I think it's the only thing I've ever put into the networking stack, so it's um fairly remember, but yeah, it it was there was um uh an erratum for basically a hardware bug for the network controller that we are using on the um the board we had. Fortunately, somebody Harold Welte had already developed a fix for it but he had not up streamed it and it needed a bit of work to get up streamed. So I um grabbed the patches from him and up streamed them. Um yeah, that was and also introduced a work around for a um Actually no, sorry. Harold had developed something to improve performance and then I I I developed an erratum fix for the horrible erratum that the chip had and I up streamed them both. So yeah, actually I can't quite remember this the great length of time. >> Well, half the work is it seems like always getting the code first round of code getting it to work and refine it and then get it to in a place where we can up stream. That's that is the part, right? So up stream >> Yeah. Yeah, yeah and in this case the the other half of the battle was deciding that the hardware was in fact buggy and that it wasn't the driver doing something dumb that we needed to fix. >> Right. Right. We can yeah, sometimes we tend to blame blame something we don't Our first thing is to blame. Okay, maybe software isn't doing something right. >> Yeah. >> Go to hardware and then then firmware and then >> Yeah, it's a lot easier to blame the thing you can control. >> That's true. That's true, absolutely. So um So how do you you you do lots of things. I know you do you have gotten involved in the Linux next working working with the Steven Rothwell and also you have been active in embedded space it to more so than um I see you getting involved in large areas. Like you contribute to Alsa, you contribute to anything that you see something is wrong and you contribute to self-test, and you make sure self-test big thing is you make sure all of the self-tests are running in embedded environment um and all of those things. How do you do all of those things? And how do you manage your time or balance your time in various activities you are involved in? >> Yeah, so that that um that can be tricky. I mean I find for a lot of the things I do like the the self-test for for example, um there's a lot of small um bite-size tasks um that you that are um that don't take too long. And I find um so if I I find those are often good to slot in around bigger development. Uh cuz so sometimes when you're working on a a complicated thing, you need to take a break to um to refresh your refresh your mind, let let the problem percolate a bit. Uh and so picking up one of those um smaller things like looking to make sure that it see if the self-tests are working, trying to work out how to uh fixing them. Uh that that sort of thing is good to um fill in those gaps where you need to take a break. Um you don't necessarily want to uh completely step away. Um and laterally with Linux next, um it there's a very clear priority. With Linux next, it's um something that I have to or somebody has to get done every day. So um it's very it's very easy to um look at what's going with Linux next. Um like in fact it's running on a screen over to to my right there. Um it's very easy. You just look at the look see if it's doing anything. If it's doing anything interesting, then you have to stop and work on it. Um, otherwise get back to something else. Uh, and again that that's the thing that's really easy to slot um, other more smaller tasks around. Uh, cuz I I if I'm if I've got Linux next going, then I don't know when it's going to there's going to be some error that I need to step in and do something about. Um, so having a smaller thing that I don't need to concentrate on for as quite so long uh, and I can just drop easily. Uh, it is really great for getting that done. Um, in a sort of in a timely fashion. Um, the the the other thing I find with prioritization is I always try and think about um, when other people are waiting for me. Mhm. And I try um, so I try to um, um, I try I try to make sure that I always carve out some time in my day to be responsive to people as much as I can. I'm not always terribly successful at it. Uh, but that that's generally the goal is, you know, first thing I do look through my inbox see if there's anything that I really need to look at. Um, take a look at the ongoing reviews that I've got. Um, see if there's anything I need to move forward or can move forward. Uh, and things like that to so to try and make sure that um, um, that that I'm not the blocker for other people as much as I have I can manage to do that. >> Right. But yeah, that is the tricky part, right? With the With a lot of I struggle with that too. I try to um, but um, but yes, with the volume of emails we get, um, that is definitely the tricky part of trying to figure out, okay, somebody is waiting for us. >> Yeah, uh, yeah. Uh, and sometimes those things are you know, sometimes the emails are really hard to respond to and you you it does it's going it's going to take a while, but sometimes they're quick, you know, it's so some of it's about looking at them and dealing with the ones you can deal with quickly quickly. >> Right. So, it looks like you employ a similar strategy as I do. If the some emails takes like 30 min 30 seconds or 1 minute, I just respond. And others that take longer go on a on a different pile, but getting to that pile is something that you have to get to. So, that's uh yes. >> Right. Right. And uh you are right about that though that um um Linux-next it you never know if everything goes smoothly or you have to go uh do something about it. Similar to stable release testing or release testing. Some days stable release testing just goes smoothly and you don't even have to worry about it. On a on occasion it comes sense like, "Okay, something isn't working on my system or some patch is not working." Then you have to go figure that out. Right. >> Yes. Yeah. Yeah. Yes. With Linux-next I felt the first day I took over Linux-next after Stephen retired. Um I actually uh it it runs so smoothly that I actually stopped and spent um more time making sure that this um these scripts had actually worked properly um and they hadn't just uh abort you aborted and given up on me uh then I did then uh I did actually do anything with them. Uh but equally well, there's an interlock in that Stephen put in the Linux-next script scripts that tell that tells you to go to bed if it's after 4:00 a.m. >> Oh, I see. >> So, [laughter] um you know. >> The the natural sleep cycle for Linux-next. That's that's important. >> Yeah. Well, that's actually it worked for me as well, but yes, partly. That was that was Stephen's thing. >> So, for people in the audience that don't know Stephen, Stephen Rothwell, um back when I think 20 some years ago, started Linux-next as the integration tree, uh which is a lifesaver for a lot of us maintainers uh to go find integration merge problems, conflicts, and so on. And then when uh Stephen sends us something wrong with the Linux next, we usually jump on it right away and try to fix it as quickly as possible. So, anything to add about Stephen, uh Mark? Um you worked with him long probably closely. >> Uh yeah, there's Stephen like um doing Linux next uh like he did that he came up with the idea um and he did it for uh mostly single-handed for um I think uh 15 years, something like that. Uh and like do it do it uh especially doing it now every single day, I like I was filling in for him when he was on vacation, but um it's uh now it's other people filling in for me. It it really brings home um how much of an effort uh it is to to manage that. >> Right. And not only that, you also have to Mark, you also have to keep uh track of well, who's the new maintainer coming in and um hey, do you want to add your tree to Linux next? You know, even all of those kinds of things as well to take care >> Yeah, fortunately as you say because as you say it's quite important to a lot of people, uh fortunately that bit of things is usually um somebody el- the first I hear about is somebody else has already prodded somebody to do something. Uh so, you know, various core maintainers will say, "Oh, you need to get this into Linux next before I will start pulling it." for example. >> All right, that's good. Yes, that's good. One last thing for you today. >> Yeah, yeah, exactly. Delegation's always great. >> Right. So, um so, you know, on top of Linux next, um you're the maintainer for several subsystems and then you can you're an active contributor for various things. And um so, the other part of we talked about a lot about how you manage your upstream responsibilities, but how do you manage your responsibilities that come through work? You work for Linaro, Arm, and Wilson Electronics, I think. Yeah, you mentioned that. All of these things, how do you bring that? >> Yeah, so I I I've been very fortunate that for the most part my work has been upstream focused. So um that that makes life a lot easier. So for example, when I run Linux Next, that's something that Arm, my employer, I do as part of my job. It's not something I do on the side. Um so that makes the the balancing easier because it's already factored into the um the project management and so on. Um Uh but but it it it is it is tricky and I I think uh you know, it's been like previously I used to do a lot of onsite support for customers at previous jobs, uh which obviously interferes with the amount of time you've got to um to work on upstream. Um I think the only thing that's worked for me consistently is time boxing and saying, you know, I I've got at least 1 hour or whatever it it is at the time to that I'm going to spend focused on upstream activities. And then the rest of time um I work you know, I work on the the things more employer focused. Uh but you like like I say, I've been very fortunate in that my jobs have always allowed for um allowed for the upstream part. It's not been something I've had to do completely on the side without any support from my work environment. >> That is great. So you you found a way to um bring have a good balance of both that you whatever you are doing for upstream is also benefiting for your your employer and vice versa. So that always makes it easier to it's a great situation to be in Mark sometimes struggle with being able to do that effectively. >> Yep. No, I'm I'm super fortunate most maintainers aren't as lucky as me in that regard. >> So that's great. I mean what that means is that they align for you. Your interest and you're able to find a way to say, "Okay, this is what I'm doing, but this also benefits both your upstream and and employer." And in a lot of activities we do do benefit both. In some cases it's like being able to connect them and say, "Hey, this is how you are benefiting from my activity." >> Yes. >> to to tell your employer the same thing. >> Yes. Yeah, no, definitely. >> Sometimes it's hard to see especially with upstream. It's a very it is hard to say sometimes, you know, the how it's benefiting. >> Yes. Yeah, it can Yes, some areas it's easier like some for example some of the enterprise distros really do want to take in code that isn't upstream already. So that if you're working on their their things it's a bit easier than if you're working on an embedded thing where people just want to get their product out the door and they don't really care about where the code came from. >> Right. Absolutely. Yeah. That is that is that is the reason why I think it's it's great that you're able to do that. So now that is a good segue into managing stress and in terms of what is what is the thing that really stresses you out. We all have different stresses stresses when we talk think about our work day. So any stress factors that and how do you manage them? >> Um I mean I think the I I think the main thing is usually when it's a deadline and I've got limited amount of time to to do things. That that that I find is typically um that that that's typically what stresses me out. Um Or you know or or when I can't see how the the the available work is going to fit it into the available time. Um and um I I'm not sure anybody should be looking to me for advice on how to deal with that if uh in a sensible fashion. I mean I I think to a certain to a certain extent um it's as it's as straightforward as just not worrying about it or learning to just not worry about it or not worry about it excessively. Um but that's much much much easier said than done. Um >> Yes, especially when when you know that somebody could be waiting and you don't even know that they are waiting in because you you just forgot you didn't manage to see the email. Those are the things that but you're right that you just have to learn to manage that meaning we have to worry about it and at the same we don't want to excessively worry about like you said. >> Yeah. >> And >> Yes. Yeah and also from that point of view especially with the maintainership stuff some of it's um can be about um well maybe sometimes I'll do a merge in Linux next where I'll apply a patch and I'm not sure that I've done the right thing and that it worked. Um but I also I you know, I have to It's something you have to accept that maybe there'll be a problem, but as long as you're responsible in fixing it um then things should be fine. Um So it it So yeah, there there is there is an element of that as well, you know, trying to um make sure you've got a a realistic box on the um the severity of of the problems and make sure you um you know, if you're if you're letting things slip, know know that you have a recovery strategy and that um how to um you know, things are going to go wrong, how are you um um how are you going to deal with that or is there a good story on dealing with that? >> Right. Right. And I mean, um yes, that's strategy of how do you how do you get well? I mean, we're all human, we make mistakes and >> Yeah. >> um and and that accepting that we are not superhuman, we and saying that we should never make mistakes or set those bars high. Just set them Okay, I'm going to make mistakes sometimes and then you know, just say, "Hey, I made a mistake." and um and then I know how to fix it or I know how to go about fixing it. >> Yeah. >> That's kind of what you're saying, I think. Um and and that um don't come down on yourself if you made a mistake. >> Yes. >> That adds adds to more stress, right? You know, on top of the mistake, you you're now you're dealing with oh, how did I do that? How could I do that? Yeah. >> Yes. Yeah. Yeah, and and if you're overloaded, sometimes it it's it's okay to just low you know, lower the bar a little a bit if you're So long as you're doing it in a way that you understand and where you're um confident. And so like we we have a long process between applying a patch and it going out in a release. So, you know, how how confident are you that they if there is a problem somebody will tell you about it and it can get fixed before it get ends up in an end user's hands. >> Right. Yes. Yes. Especially for embedded and in some cases in this like some of the hardware you deal with there will be fewer people that might find the problem. So, you'll have to go about finding them yourself and >> Yes. >> [snorts] >> Yes, no, I I remember surprising one it was one colleague somebody was working on one of my subsystems but I explained to him I didn't really care if his driver actually worked or not because it was for hardware that I was never going to have access to. I cared about whether it was me like I was I was looking for problems when I was reviewing it but um you know, I I wasn't if it was a bug bug in the register right sequence or whatever I would not be able to tell and it wouldn't affect me anyway. >> Right. >> He so he it changed his perspective on what we were looking at. He was quite surprised. >> Right. And I think I think that a lot of people it's hard for a lot of people to understand the scale of the footprint of Linux and spans from really really small IoT devices to all the way to supercomputers and then multiple variations of the same chip or same driver that affects things. So So, we rely we rely heavily on sometimes we do our best and we do find problems and we reviews and multiple reviews that we go through and each patch could go through several people's reviews but there is always sometimes something slips and then we are responsive. So that's it. >> Yes. >> That's right. So that is the that is the hard part. Being able to stressing out about oh okay, what's getting into the users' hands and that is that is the >> the main line. Yeah. Absolutely. Um oh, yeah. So you do Linux next, of course, um is is larger than your work that you do for the community and I can I look at it as Linux next even though it's a part of the code and you're still handling all of those things. It's bigger than bigger scoped. And then what are the other things that you do um uh for the kernel besides code. Like for example, you're sitting with me talking to me. This is this is like mission, right? And then mentorship series and so on. >> Yeah. Um So there's not the I think the um the the biggest thing right now uh or I say the big is is certainly a big impact for the community. I don't know about my contribution, but it's something that you're involved in as well as the Linux Plumbers Conference. Um so um I'm the uh current chair for the organizing committee which mostly just means I have to run the meetings and remind people to do the things that they said they were going to do. Um So other people are are actually doing the you know, actually doing the heavy lifting there um including um people from the Linux Foundation who uh do an awful lot of the event organization. Um Yeah, I think I think that is at the minute that's the main um non-code thing that I'm doing directly with the the kernel is the uh is try trying to get Plumbers uh out this year. Make sure it's a good event that everybody enjoys. Hopefully it will be. Um it's in Prague. Um tickets are available now. You should all come. >> Yep. Yep. Yes, that is the first time chair. Um it is uh it it's always hard when I mean because you are getting other people and we are all volunteers. So, that's we are all carving out time from our everything else we do. So, yeah. So, but uh but hey, you're doing a great job. >> Yeah. Yeah, and and a lot of it really is just reminding people to do things. It's um um you know, making sure that making sure that we've uh the schedule. I see the the link to the event has been uh dropped in the the chat for people. Uh thanks, Candice. Um yeah, but a lot of it really is just reminding people of the you know, the conference is in uh quite a way out still. Uh but we still we we're at the point where we need to start getting we've got deadlines for getting things done. So, it's just um keeping it the top of people's minds when it's not the most immediate thing. >> Right. Absolutely. And then also asking them, "Hey, you you have to register or you have a speaker code sitting in your inbox. Come and get it." >> Yeah. >> stuff, too. Um but uh it's inside I don't What's your experience? I mean, I have attended before I got involved in the program committee. I attended lots of Plumbers Conferences. I really didn't think much of the how much work goes into it. What's your takeaway on that how the the work that goes behind the scenes happens? Did Did that Yeah. How did you >> Um so, it didn't surprise me so much because I'd been peripherally involved in running the um uh the Debian Conference when that was in Edinburgh, Scotland where I live. Um so, I I'd seen conference organization from the inside before and uh just to how much work it can be to put on a big event. Um but yeah, as an attendee to an event, you you never see um all the effort that goes into securing venues, making sure the venues are suitable, um working out the program for the event, um coordinating travel um for for people who need travel support, um sponsorships, uh this uh um and then just on the day just ma- making sure everything's going smoothly, um you know, having people in all the conference rooms, making sure making sure that the uh these these light projectors are working, that everyone's on schedule, that sort of thing. Um there's a huge amount that goes on behind the scenes, and it's not really it really shouldn't be obvious to anybody who's not actually taking part in the organization. Um the whole point is to put on a good event for people, not to uh worry them about what's going on. >> Absolutely. Absolutely. So, you have been in the community for 20-plus years, and >> Yeah. >> you are you're very active in the community in various aspects um of upstream development. And what keeps you here? Uh what keeps you uh get up every day and do what you do? >> Um so I mean, obviously some of that is that it is my job um and um that always that that always helps. I mean, that that that uh especially, you know, the it stops um it ensures I've always got some level of involvement, even if life's being very hectic at any given time. Uh it's a pull back, but um really a lot of it is the it's a combination of the technical challenges and um trying to make something that does the best possible job for people, uh and also the people we're working with is uh just a key part of it. Um I mean my my personal experience has been the community is for the most part really great. I mean it's a huge community, so there's always going to be um things that are difficult. Um and there's a lot of um difficult problems, which you again makes uh can make things difficult. Uh but yeah, it the it's um it's a people thing as much as it is a um uh as much as it is anything else. It That's um Yeah, that that I think that that that is probably at least as probably more many ways important than the uh technical challenges. Cuz I mean there are technical challenges in many fields. Um the kernel, you know, the kernel has some interesting things going, but so yeah, so do applications, so do web services. Um So do embedded devices. You know, there there um there's a huge number of other places you can put your effort. Um but the the community with around Linux is what makes it really special for me. >> Absolutely. Yeah. Cuz we um the community as a whole um yes, there are people that um um people are when whenever you people technology is easy, right? In some cases. And then when you people get people and people are all committed to the shared goal of making making Linux work for everybody. And they're also working to see, "Hey, does it run on my laptop properly, too? Does it help my uh users?" Uh kind of thing. And we're all coming with the same goal. Sometimes we don't agree on how to go about it, but we find a consensus. So that is the that is the thing I find um fascinating and great about our community. You know, I that's what keeps me here, too. So, because of the >> Yes. Yeah, yeah, definitely. >> Right. So, um so, what advice would you give to new developers when they are coming in um and to, you know, dipping their toes and trying to understand the community and stay here? >> Um So, um I think um it really helps to have um some goal you're trying to achieve. Uh and we which is usually it's not the sort of thing you would usually um turn up to just for the sake of it. So, I I think that's um that's usually the case for for most people. But, um you know, often you see people with things like, you know, um my laptop's audio uh audio doesn't work. Um I'd like to fix that. Or um or or, you know, I I've got this um uh project I want to do with an embedded device. Uh you know, I've got a Raspberry Pi and a hat for it and I want to uh make it into a display for my um my family calendar or something. I I don't know. Um But, yeah, it it it helps to have a um a thing you want to do. Uh like a a clear goal. Uh and then I think the the main things are probably um just talk to people. Um you know, if you've uh changed something, you send a patch. Maybe it's the wrong patch, but that that's that's fine. You can um it's it's a good way of triggering a discussion. Uh yeah, um talk to people and um don't get discouraged. You know, sometimes people are it's um people find it difficult because um in many areas of the kernel uh email response times are not good. Um uh and and and and sometimes that's uh simply a matter of just leaving time for other people to comment. Uh like if you if a discussion goes past uh by too fast, then uh it gets in the way of um other people getting involved. Um so if you're not seeing a response um uh then a bit of persistence and a bit uh maybe a bit of chasing um can help a lot. But, you know, try try not to get get discouraged by that. I think that's one of the biggest barriers people run into is um not is um is is seeing the uh or is getting a feel for the time scales uh on which you uh things will happen. Um and like and like with the like even if you you land a patch, um it it will generally be about 3 months uh to 4 months before um it turns up in a release. Uh which can be discouraging for people, but um if people are aware of what's going on, I um you uh you know, understand that there uh there is some latency um and have their expectations set well, I think that um that helps a lot. But, yeah, um talking to one one of the best ways to think about uh to find out about stuff like that is through talking to people. And I think that really is the key is um just um discussing things. Uh you know, put putting code out there um you know, replying to uh you know, if somebody replies to your email, uh listen to what they say and engage thoughtfully with it. And um Yeah. Uh kind of generic advice, but ultimately I don't think um Linux is that special in in this way. I think it's it's just it's a big project, but it's just software. Um it's easy to get um to or it it it's easy to get the idea that it everything about it is super deep and really some things are some things are really really really complicated and hard to understand. Um but a lot of things are very straightforward to understand. Um >> Right. It's a large project, too. So, there are a lot of things to do keep us engaged, and we don't get bored. So, that is the thing. >> Um yes. >> Boredom is not something that uh you would would associate with the Linux. >> Yeah. Yeah, so yeah, so if you really want to work on Linux, you will uh you will find something to do, absolutely. >> Okay. So, um I guess it's you know, this is like a we cannot um s- end the conversation without talking about AI. So, how it how did what how did it change did it touch your life and how did it touch your maintainer or developer life and uh what how what how it's been role being in your day-to-day life these days? >> So, I think there's there's two um big um impacts that I'm s- seeing from uh the LLM-based AI stuff uh with the kernel. Um as a uh both as a maintainer um one thing is that I'm seeing um a lot more patches coming in uh fixing um mostly fairly small issues. Uh and it is it's quite interesting the and many of these are fairly clearly generated with LLMs or some involvement from LLMs either um just from a sense I have that that's what's going on or because the submitter actually says this was you know I used an LLM while I was doing this. Um they're quite interesting like the have uh frequently got um new tools coming in um that make uh that result in large numbers of patches being generated. Um you know new linters that spot problems in code or um things like Coccinelle which make it really easy to make do widespread uh code transformations to uh clean up issues but those are all are all deterministic and regular and they they produce patches that all look um very similar to each other. So you might see a lot of patches from somebody but they'll tend to be roughly the same patch just done in a different place uh fixing a similar issue in a um in in a huge pile of drivers for example. Uh but with with LLMs um they're a lot less predictable. So they will um you'll see somebody who's clearly cleaning things up with LLMs but um but they you um you have to review each individual patch in a bit more uh detail because it's um either they just will be different fixes that the LLM happened to have helped them spot. Um or um the LLM is doing things in a way that's not quite so predictable and doesn't necessarily have everything joined up so you need to uh you can't have a um you can't have quite the same sense that yes, I trust this tool. This tool will probably do the right thing. This person said they they did X, they probably did X, and that's fine, which is what you can get to with a lot of tools. So, it makes the um review load um a lot higher. Uh there's just a huge numbers of patches coming in compared to what there would have been. And when the volume goes up, the um at least for me, maybe other people have a different experience, but at least for me, the um the things that would usually make um the the yeah, that would usually make it easier to deal with a big pile of patches from one source one underlying tool um don't really apply to LLMs in quite the same way. So, it's um it makes it uh tricky from that point of view. Um the other big thing um is um there's a tool uh called Sushiko, uh which is at uh sushiko.dev. Um I think hopefully Candace can put that in the uh the chat for people. Um so, that um grabs patches from the mailing list and does uh a code review using um LLM. I I think it's using Google's LLM cuz I think it's a Google project at the minute, but um And that's um that's helping in the other direction. So, that that that um that will um Thanks, Candace. Uh so, that that will um it's quite good at spotting that there is an issue in the code, uh which can save you time in analyzing it. And my experience has been that they tend to spot different issues to the ones um I might spot. It's going to get more into the weeds of details of drivers and things in a way that I wouldn't. Um although it can be a bit confusing at times because it will also highlight uh pre-existing issues in the code that aren't necessarily relevant to the immediate thing you're reviewing. Um but that does help quite a bit with the review uh workload. Um because it will um it will It's not going to spot everything. It's in no way a substitute for actually reviewing the code yourself. Um but it will um help point out areas of the code that need attention even if the um specific things it's saying are not great. It's usually um what the pattern matching is usually resulting in it um highlighting something that needs to be highlighted and does deserve some attention. Um so I I think I've I think There's a There's some balance between the two. I think the people sending patches are currently more than the um than the um help with the review. Um but it it it there um Yeah, I give if they are take it away take it away. >> Right. So, you are you are right. I think I I heard uh recently looking at the numbers uh 7.0 release had about 490 new developers or 400 and odd the new developers suggest from We usually see about 200. And then 7.1 I believe had like uh 390 developers. So, the new developers is clearly tracking the LLM use, I think, in this case. And you are right about Coccinelle that the kind of things we have done Automating stuff is not new to us. We have automated things in the past. Uh we had homegrown AI tools to find um CVEs and patches that go into fixes that go into mainline and they're going into stable. So, we're not new to it and we have kernel bots that go and pull the patches and compiled and found problems and tested them and all of that. Shishio is great cuz it also finds adjacent things and we had Smatch before that that did similar things. So, but you are right that we have new new people coming in. Shishio is good that it also finds adjacent problems cuz it gets us thinking. But, that also confuses new developers because so do I I keep getting questions like do I want do you want that fixed? Do you want it fixed in a different patch or this patch? So, it's like more traffic on the mailing list. >> Yeah yeah and sometimes the pre-existing issues it's pointing out are just um like they're they're they're often really like some sometimes they're false positives but often there there really is some issue there. Uh but sometimes it would be completely disproportionate to ask somebody to um you you somebody's fixed a type doing something level of fixed a typo and it notices that there's a structural problem with the whole driver or the whole subsystem. It would be completely disproportionate to expect the typo fixed to um wait for the um the bigger structural fixed. But, um yeah it's not very often not very clear to submitters um what the distinction is. I mean personally I tend to not push submitters to look at Shishio directly unless I I know who they are and I'm I know they're able to make that sort of judgment themselves and because I do find that it's it's often um it's a good source of things to work on but it can be confusing to newer developers. I just trying to understand what's a what's an urgent problem and what's a oh yes, we should really fix that problem. And what's a that is urgent, but it's it's a very big problem. >> Okay. And in general, we don't expect them to combine or even work on the adjacent problems and combine them with this patch. They are they're going to be totally separate anyway. Um they find an adjacent problem uh that they would send a separate patch depending on the situation. >> Yeah, yeah, usually uh usually it's some yeah, it should be something completely separate and it shouldn't um get in the way of doing whatever small uh improvement or fix. Um or even larger improvement or like it it's just off topic for whatever is being done. >> Right. >> Um So it it should be dealt dealt with separately. >> Yeah. Absolutely. So um I guess it's time to look at the questions um and see what what people are asking us. Let's see. We have about I think five open questions. Um would you like to just take them in the order you want to take them? >> Um sure. Um So I I think Wilbur um asked about how we start um building and developing. Um and I think we we sort of cover covered the getting involved um and advice for new developers uh a bit earlier. >> Mhm. >> Um but I I think my um my big um advice would be it would be find something you want to solve and uh then work from there. Um in in terms of like the mechanics of it, that's uh hugely dependent on what you're doing. So um I that most of my work has been either with um simulators uh for my current stuff on the arm architecture or with embedded hardware. So, the way you deploy software onto those is um especially embedded hardware is quite different to the way you deploy software onto say a virtual machine or your actual laptop. Um so you You know, the the very concrete first steps are going to be dependent on what you're doing. Um There are some good resources out there. I think kernelnewbies.org is still quite good. Um Be finding people interested in the sort of problem you're interested in and then finding resources from them is probably the good first step for how you start building things and working on them. >> And yeah, you could also look at the love live maintainer no mentorship series that offers a lot of good tutorials and 90-minute sessions with all of the developers. >> Okay. >> So, go and see if there is a match for that area. Mark has done one or two and there's a lot of a lot of maintainers have done that. So, take a look at that as well. >> Yeah. Yes, I should have remembered that having done some >> And we do have a lot of resources, I think. Yeah, like Matt Mark has mentioned. >> Um Yes. Um And um David Harris asked a similar question about um getting involved with security that I um don't know much concretely about the security um area, so I I'm not really qualified to provide uh specific advice on that at all, I'm afraid. Um other than the generic find the security people and look at what they're doing and um try to work with that. Um I think one thing that the I would say there is that the um testing side of things for security um is maybe interesting. Um the the making um it's all very well having features, but making sure that they they work properly. Um I don't know if there's gaps there, but um just thinking about what I see when I look at the testing stuff, security never pops up uh that much, which either means that the the testing is very good and the quality is very good, or it means that there's a lot of stuff that could be um be worked on. Uh and if there is stuff that can be worked on, then um that's often a good way into developing things. If you find a problem, then obviously you can uh you can try and fix them. Uh so yeah, hopefully that was useful. I would like to say I'm I'm afraid it's a bit generic because I'm it's it's kind of out of my area. Um And we have a question from uh Jeff and Philip um about um how I treat uh regression patches. Um So I think you have to give priority. Like if if something was working for people and then it stops working, then you have to give priority to that, obviously. Um So I will tend to f- um if it's something I can work on, I might look at it myself um directly if if um that's appropriate, but uh I'll tend to be prodding the developers who introduced the problem or who are responsible if it's not obvious who introduced the problem who are responsible for that area to respond quickly and ideally also working with these submitter if you know if they if you can gather any information about um about the problem that would help narrow it down while waiting for the the people to who knew about this specific area to get back to me. But yeah, I would always try and fast track anything that looks like a regression. And try you know try try and make sure that the appropriate people are paying the appropriate level of attention to to the issue. >> I just put in a link to Colonel Self-Protection Project that has a overlap with security and you will be connected with the security minded people in the maintainers and such in the community. >> Yes, and that's they're speaking of community earlier they're also a great group of people. So if you look Yeah, there there's They definitely have a big to-do list. So yes, that's that's a great idea. Thank you Shera. >> Of course. >> So Jerry Coolstra, sorry if I'm mangling everybody's names by the way. I'm I'm guessing a little here. Um He asks about finding entry-level jobs. So, um I'm again maybe not the most qualified person to ask answer that question. Um I mean I know the from the work recruitment I've been involved in what worked for me more years ago than I care to think about. Um the there are entry-level jobs out there. I think um um a lot of them are there's a lot of opportunity in the embedded space. There's a lot of smaller companies making small run hardware devices that need software engineers who are not necessarily just kernel engineers. Um but who will have to work on the kernel as part of putting the device together. Um so that's if you can find those sort of companies nearby. Um they're often a um a good place to look. Um and the other I think Yeah, otherwise it it's um support type junior developer type roles in uh the big companies. But um everybody knows who the big companies are. So, those can be quite competitive unfortunately. So, it's it's a bit tricky. Um The the other thing is obviously if you um become known for contributing to something uh it becomes a lot easier to get hired. Um but that that's you know there's unfortunate blockers for uh there you know you have to have the time to contribute. Um which is a big ask for some people uh for many people. Um but it it does I mean it it does work. Uh whether it's fair or not is a separate question. I should you I know you were involved with the um Linux Foundation mentorship programs as well. >> And yeah, you is one of my graduates that was submitted to us. So yes, this Mark you are right on. That's what I tell my mentees as well that um making contributions and staying with the company staying with upstream that shows speaks for itself in in terms of contributions and then people will find you. It's but it is a long road. Like we have we have done it too. Like we don't get our dream jobs. There is no concept of landing your dream job. You have to make a job your dream job or and you will have to work through the process of building a career, right? So yes, you have to start somewhere and even if it is like a support role that you're doing and part of it is kernel and part of it is user space you have to kind of work towards where you want to go, where you want to take your career to. >> Yeah. Frankly, some of those user space kernel space interface issues can be some of the more interesting ones. >> I find them very interesting. That's why that's why I kind of straddle between self tests and K units and then all of the other tools. But yes, absolutely. I think you do the same Mark. We >> Yeah, I'll I'll I'll >> Yeah, we are in some of the similar area. >> Yeah, I tend to stick to the top of the kernel or on the bottom of user space. But yeah, it's kind of similar. Um So David who says Perez asked about using AI without um compromising any of learning good quality or trust. Um I think I think the main thing I would say is there's two things I'd say. One is like for the the code um make sure you understand what's being sent and have double-checked it yourself. Um The other um If if things are wrong, it's not the end of the world. That that happen that happens all the time. Um but you know, try to have some make sure you have some understanding of what's being sent. Cuz I think the the biggest um the biggest issue I see with in terms of um the AI um or the the people using AI, it's it's not so much um it's it's not so much um to do with the code so much. Or at least the people using AI for smaller things. I've not so much experience of people using AI for large large features. I know there's been some big problems with that. So yeah, I would recommend sticking with smaller changes that are easy to comprehend in and of themselves. Um But yeah, the the biggest issue I've seen is when people are using AI to write emails. Uh because it it tends to be very very obvious that they've done that. And um a lot of people really really don't like seeing that. Um and get kind of frustrated talking to the LLM. So yeah, I I would recommend small bits of code or manageable bits of code that you can understand well yourself um before you send them out. And um don't use it for the emails. Or you know, if yeah, yeah, make sure you you actually write your own emails. Um also can can sometimes be I know some people report good experiences having it help them prompt find bugs. Uh like Coccinelle is kind of in that ballpark. So, that's yeah, that's another thing to look at. >> But this is actually a great question, if I may. If you are thinking along the lines of how can I learn from use AI as an accelerator and not compromising learning and code quality, you are way ahead. Because um that's really AI is a great accelerator. It's a huge It takes off a lot out of a huge learning curve. But you have to use it correctly because you want to use it to learn, not to just generate quick patches to that you want to get into the kernel. I think that's >> Yeah. Yeah, and and you you have to have a skepticism about what it's doing and a concern about the and yeah, you know, is you're right, it is a very great question. So, we have one more question from Jeff and Philip, which is a I I think I can answer quickly before we run out of time, which is just how long it took me to become a maintainer. Um and that was I I don't know several years, I think. Um then it it depends there there's different there's different stages to that as well. So, there's um I initially started working on drivers for for well, so now I was sort of was maintaining those. Um then I started sending changes for uh for for the subsystem and then I took over that. And that that took maybe another year or so. And then over time I also took over other subs existing subsystems for various reasons. Uh and I also wrote one new subsystem completely from scratch cuz it fit a niche for me and I just became the maintainer of that one by default. Um cuz I wrote the thing. Um but yeah I would say it was a period of years although there was the shortcut with the that one that I am I wrote myself. That's that's a bit of a a lift though to find an area that needs a whole subsystem. There are there are maintainers of drivers, there are maintainers of subsystems, there are maintainers of core subs. You know, there's maintainers of nearly every thing. >> Mhm. The road road is different for different people and also it does take time because you are building trust in Nobody gives you a maintainer ship maintainer role. You earn it. So. >> Yeah. And it's something that not you don't necessarily want to do either. Like a lot of people are highly respected developers who just want to develop and don't want to deal with all the review and patch management and process management stuff that comes along with being a maintainer and are very happy doing that. So it's not Yeah. It's not necessarily something that would be fun for everybody. >> It's a mixed bag. >> Yeah. >> Uh we have two more questions. Mark, we can just go ahead and take them. It's not a problem. I think we a little bit over not a problem. >> Um Yep. Sure. Yes, I've got time. So yes, so Paul Van Dam asks says Rust programming the kernel is getting more traction and there's increasing demand. Yes, I absolutely. Rust is like there there are several new drivers have been written um entirely in Rust and are just completely relying on it especially in the the graphics subsystem I think is the main one. I know other subsystems are experimenting with it as well. Um so yeah it's um it's really becoming a lot more important. Um they're going to be systems you buy in a few years where you're not going to have a display if you don't have Rust running in your kernel. So um um there um Yeah it's becoming more important and I I think there it's attracting some new developers as well who want to work on really low-level stuff in Rust. Um so the fact that the kernel has um is now somewhere where you can work on Rust is um is drawing them into working with us. >> Right. Tune in to some of the LPC Rust talks. We do we have virtual coming up in September in Prague. We have a few good like three or four good talks and Rust and C as well if you want to. >> And they're also like the Rust developers are also great friendly bunch of people. So if you're looking for introductory stuff to work on there's definitely a lot of open ground with Rust. Um so um Yeah it's if you're looking for ways to get involved and you're interested in Rust I highly recommend looking at that. Um and the the last question is um anonymous um so which is basically asking what the balance is between uh people using AI to patch or people using AI to review. Um I would say the I can see more people doing AI patching. I don't know how I it it's a lot harder to tell if somebody's using AI to review though. Um because they might be using AI to prompt them to look at something and then um understanding you know, understanding if it's actually if they all I'm actually told them uh something that makes sense or not. Uh so that their emails will come through and it'll just be something written by a person and you won't be able to tell that they used an AI uh an LLM for that. So I don't um I would expect that there is a lot more AI review going on than I am actually um directly aware of. Like I can see the patches. Um I won't necessarily see the review um and I I mean I know I'm seeing people uh respinning their patches um having fixed issues and they'll say I saw this from Sashiko. Um so I know people are looking at it. But I don't I don't know I I just don't have a sense of what the answer to that question is. Um I suspect more people are using it for review than patching. But I can't I I can't see that so I couldn't swear to it. >> So in Yeah, that's all we have in the questions wise. Any closing thoughts before we part? >> Not really for me. No, I I I think your your your questions covered everything and I would have brought up. Um you get you know, together with the questions from the attendees. So yeah, I No. >> Thank you so much, Mark. >> I I thank you for the opportunity. >> Back to you, Candace. >> Thank you, Mark and Shuah for your time today. And thank you, everyone for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today. We hope you join us for future maintainer sessions. Have a wonderful day.