Flock 2026 State of Fedora address

Hey everybody. It's still weird for me to be called the Fedora Project Leader and it's been a year. I don't know if it's weird for everybody else yet, but it's definitely still weird for me. Hey, I'm Jeff. I am the the not quite as new as I was yesterday Fedora Project Leader. I can't say new anymore. This is my second flock. So, let's get into it. Let's talk about the state of the Fedora Project. Um So, first, Fedora by the numbers. I'm going to show you some good things. I'm going to show you some bad things. I'm going to show you a weird thing. Um let's start with let's start with some good stuff. Um So, this is made possible by a really great tool that Matthew put together and it's actually super close to being uh something we can deploy so everyone can look at this data and examine it. I was able to run it locally with one small configuration change last week. Uh hopefully I I think I still owe Matthew a PR on this. Hopefully he's fixed it without me, but it's really great and I love playing with these dashboards. These are the Grafana dashboards that uh look into our count me data. For those who are not familiar with, when Fedora systems or systems that talk to our servers looking for updates uh communicate, there's a count me system we can keep track with of these systems in a way that lets us see um what they are. Um so, overall, uh this is what it looks like in the screenshots uh uh from the dashboards from the tool that show the overall state of Fedora, right? Everything that we count in the Fedora Project as an output that can that can talk and ask for updates um uh is represented here. We've I basically seen uh almost a million systems in the last week according to this morning um and a year-over-year change of over 9%. So, that's actually pretty good, but it actually gets better. This is for the everything. Um I love this graph. This shows you sort of the long tail of releases that are reporting in, right? All the way back to like Fedora 32. Pretty cool. On this graph on this graph. But there's data for forever, right? Uh it's really interesting. I should point out some of this some of this data This is actually just the count me data. We don't even This isn't the old data of when we were doing unique IPs only. This is the count me system, so it goes back to 2021. I deliberately did that because the unique IPs are have specific issues cuz it's hard to know if they're the same system or not, right? The count me should address some of that ambiguity. I love it. It's great. It's much better than the small. Um for those old in the room, I just identified all the old people. Um this is also one of the great things about count me because it has an age concept. That dark blue strip are all the things that are a week old. The light blue strip are all the things that are less than a week old. So those Those are sort of the things that pop. They're the popcorn, right? Those are not stable systems. Those are go on and off. They may be seen asked for an update once and then during their lifetime, right? Um the bottom part is the systems that have been updated for forever across multiple releases. They have what? I think it's like over 24 weeks of count me history. Um that's that's also super interesting. I love this graph. It sort of says that hey, updates work everybody. It's great. Um and and this is interesting. This is by variant like what what the counts look like. And and don't get hung up in the actual numbers. The trends really are what matter to me. Whether there will be issues with with the raw counts um because you can't ever know if you're undercounting or overcounting, but the trends are really important, I think. And I >> [snorts] >> I I to highlight a couple of things in here. The top-line is workstation. Um the dark blue line is cloud. Uh the yellow line is KDE Plasma and I'm going to point that out. Uh there's a dark green line that's hard to see that's close to the KDE line that's actually CoreOS. So, I'm going to I'm going to drill into this. Um and once you get the dashboards everybody you can drill into this too. There's there's so much data. Um so, let's talk about KDE Plasma. The the superstar on the year-to-year growth. 120% year-to-year growth for KDE Plasma. Um that's fantastic. Good job KDE developers. >> [applause] >> Um I mean I can speculate that this is because it was promoted to addition and it was able it because it was above the fold it's it's seeing more uh notoriety and then seeing more use. Um that's a speculation. I don't know have a way to ground truth that yet. But uh the but to talk about that, you know, what KDE Plasma is at 158K, right? And it's still growing. Uh workstation is at 297K. And it's still seeing 18% growth. So, they're both outpacing the overall growth of everything. Um so, that's also fantastic. In fact, you can I can make myself There's a knee in the last couple of releases. So, so it's picked up, too. So, they're not competing with each other. They're growing the pool of users. Um so, that's also great to see. Um Now we get into uh image mode. Like so, this graph is all the image mode varieties that that we currently track. But CoreOS outpaces all of it. Like it's huge. Um and it's still growing. Not not growing like KDE KDE's has a like a exponential curve right now. But that's a linear growth that that that's picking up. So, it's steady year-to-year, release-to-release, it's growing in use case. >> [snorts] >> And And is the by far the biggest part of that. The weird So, that's the good stuff. The weird graph is cloud. I don't understand this graph at all. Uh I I expect all the light blue I expect a lot of stuff to just pop on and off and not be a week old anymore. But what's happening in the last couple of releases I I don't understand that graph. Like, stuff has shown up like instantaneously as being 24 or more than 5 weeks old. It's like did someone dust off a gold image and then like turn it back on from a snapshot? I don't know. But but this sort of accounts for why like everything else I've showed you has been outpacing, you know, that green percentage number has been outpacing the 9% for everything. This is the one thing that's sort of a negative number that I've been looked at so far. I don't know. But but I want to go back now and take cloud out and run it the rest of project and see what the growth really looks like without this. I think I think this is just something because cloud is a user's pattern that's just different. It's harder to track. Um but like I said, I said good, bad, and ugly. This is the ugly thing. Um Um and then outside of Fedora proper, right? We have things that show up in the counting system because they are partially derived from Fedora and they're still pinging our server. And um things like um Aurora and and uh Blue Fin and Bazite, right? Uh these things exist. They're part of the larger ecosystem. And because we have data about that, I want to show you that because they are our friends. And when they do well, we do well. I would actually like to find a way to bring them closer to us, right? It's something we should work on. Um but but in this case, Bazite by far outweighs everybody else's use case and that is super interesting. Um and I don't know I don't know how we we work with them better, but it is great for the ecosystem cuz it's part of our larger ecosystem and we should celebrate that as well. Um so that are those are the numbers um for in terms of use usage. I want to thank everyone who contributed asterisk to that success. And I put an asterisk on that cuz we do a we have a hard time defining what contributed means. We spend a lot of time in the in the council face-to-face actually trying to discuss exactly that. What does contribution in this project mean? Um that being said I want to actually try to take a look at what contribution looks like right now. And this is where we get into the not so great part. Uh I want to thank Michael Winers of uh and his effort in the Fedora data working group um to provide us new set of tools. We have a whole bunch of data that flows through the Fedora message bus uh that gets captured that we've had a hard time making use of that that really speaks to uh project health and what's going on in terms of contribution. And his effort has really given us a tool to make it possible to delve on that. And the graph I'm going to show you is literally I asked him a question at the workshop yesterday and he was able to answer that question last night. Uh and I don't think we've been able to answer that question. Matthew hasn't even seen this yet. >> [applause] >> So So this is what it looks like uh over I think one 2020 2023 2026 a look at what active packagers are doing uh in our data collection what we are able to capture in terms of activity from and this is not all contribution but this is an important part what packagers are doing, right? And this isn't the number of commits. This is the number of FAS accounts associated with commits. And there's an obvious decline there in terms of the number of people who are participating in this project in that way. Right? Uh in 2020 you know you can squint and say that's pretty flat. In 2023, it's not flat anymore. Right, there's a decline over the year. And in 2026, while it's only it's only 4 months, um, that's pretty straight line. Um, so so that is, um, concerning. Um, I think I have a I think I have a, uh, curling analogy here. I Sorry, Eva. I said I wasn't going to do a curling analogy, but back when I was in Alaska and I was in Fairbanks and I was part of the curling club there, which is a volunteer community. It's all volunteer run. Uh, because of the way the town was, we'd have to make up 10 to 20% of our membership every year to to be a sustainable club. And we put in a lot of effort in turn doing learn-to-curls. We to get those What was this amounted to? 20 or 30 new members every year to make up for the loss of the 20 or 30 members who'd left town every year. And it took us, I mean, learn-to-curls, we would we would run 200 or 500 people through a no-commitment sort of learn-how-to-curl thing just to get them to show up again and play in a league. And then we'd make effort to get those people who showed up for the first year without a commitment mentored so that they would show up a second year and feel like they were stuck in and part of the community. And then once we got them the second year, they were in until for work took them out of town. Right? So, but that was a lot of effort. The new club I'm in on the East Coast now, they don't have to do any of that. They actually have a wait list for people. They make very little effort to do recruitment. Um, and so the story basically is like it can depend on the environment, not necessarily you're doing something different, but the environment's changed. Uh, the new club I'm in, this is their first year in like maybe a decade where they haven't filled up the ice and they're now wondering what they have to change so that they can actually do recruitment and they don't know how to do it Because it's been a decade since they've had to do any recruitment because they've just been um relying on the groundswell interest for the sport and for whatever reason things have changed. I have I could talk off camera about what has changed in the DC region in the last couple of years that might have done that, but not on camera. So, that's the analogy. So, we have So, what what is What is the sustainable path for this community now? I don't know. I mean it could be that we need to to really double down on outreach. Uh find a way to do some program programmatic work or the new normal for us may be that we have to do more with less. We may need to find a way to automate more work. Uh work that uh was interesting to people of my generation, right? In terms of skill development, but maybe that's not a skill development thing anymore that's attractive. Maybe it's the kind of thing now where uh learning how to automate work is the attractive part that would a that would attract the 22-year-old that I was when I joined this project. Uh it's something we have to think about. Uh but there's and I don't know what the way forward is, but I am concerned by that graph and I need to ask more questions of the data so we can get a better sense of what we need to do if we can if we can sort out more uh so I'm not just throwing darts at the problem. I do not want to throw darts at the problem. I want to solve the problem. So, what what I think what I think it is it's a generational switchover. What we're seeing right now. Uh I think I I I guess it I I want to dig into the data a little more, but I think we're just losing some people by attrition and we don't have uh the process in place to to speak to interest sort of the that next generation. Um so, we have to figure out what that looks like and change our processes. So, that's what I think I think the core problem is. Uh how do we address it? Um good question. I have thoughts. And the new curling club, for example, I don't have to solve this problem for my new curling club. I'm not on the board. I just can be a member. But unfortunately for all of you, I have to have thoughts on this. I have to have opinions. I'm going to share them with you. And and uh and see if we can figure out together how to solve the problem. Um it's the Unfortunately, I'm the FPL, so I can't not have an opinion on this. Um first thing, I think we need to modernize the contributor experience. I think we've already started to do that with some of the tools. Like the forge is a big leap forward. I think it actually brings us uh access to some workflows that are more standard in the ecosystem. We can have opinions about whether or not we are like everything else in open source because we are an operating system. Um but I think we have to try to do things the way uh the rest of the open source world has learned to do them while we have been maturing. Um And And this isn't a knock on us. We were way ahead of the curve. It's just that open source is one. And it won in a way that we didn't anticipate. And we need to see if we can get back in the line and standardize on some processes and workflows that people now expect um for open source development. Uh I also think we need to modernize how project governance works. Um I will I put my bright spot. I love the fact that Vesco has has chosen to do their skip um process for their term limits now. I think that's fantastic. It makes a hole in leadership so that new people can step up. Um and I think that's part of it. Again, curling analogy. My curling club did that for years, for decades. Uh it worked really well. Um and so I'm happy to see it Vesco do that without me actually having to nudge really hard to do that. That's fantastic. Um but we but we got to do that across all the government stuff. I think we need to examine all a lot of the council process and we can talk about that in the next panel. Um I think there's still room for improvement there for some Fesco stuff and probably some mindshare stuff. But but it is a conversation now that I have that graph, right? These are conversations we can talk about until we can make sure that uh we are doing governance in a way where we're encouraging new leadership, younger leadership, with new ideas and where the innovation is happening. Um I also think we need to modernize outreach. Uh like I said, if we're going to try to make up that gap, we have to do outreach in a more significant programmatic way. Uh and the problem is we don't necessarily have the resources to do that right now. So, we're going to have to figure out how to expand resources. Uh one of the things council has talked about um at it was in the face-to-face um in February and we we reported on it in the dailies. You know, and there's a community group that's already looking into this uh trying to find a way to have a way to take in community money for community interests that so they can we can support and broaden the support beyond what uh Red Hat can do with the budget. Like like Justin said, like Flock takes a lot of effort in itself. Sponsor money from other companies helps put this on. We need to go further and so that we can actually get Fedora people, community members, out to way more things with a mission to build up the contributor base again. And we can't do that without some resources. So, we got to figure out a way to modernize that and get access to more resources. Um and and we want to run an experiment. Uh the council actually wants to run an experiment with Open Collective. I don't know if anyone's familiar with Open Collective, but we we want to run an experiment and do a small money appeal so that we can learn how that works, uh have a framework where we can then do more and grow that in a way that I think we need to to rebuild an outreach program globally globally. Um and and the global situation's got more complicated in the last 2 years, 5 years, 10 years. So, um we have to do some work to modernize how we do that. So, uh the other thing I want to say, how much more time do I have, by the way? >> All the time you need, man. >> 13 minutes. >> 12 minutes? Oh, I can slow down. Uh so, I also think that we need to let ourselves experiment more. I think we we front-load a lot of discussion. I'm not going to say bike-shed, but it's bike-shedding. Um we front-load a lot of discussion and we we we try to get things very perfect before we have something to play with and build a community around a sustaining community around. I I want to I want to fix that. I want to I want to build a space where it's okay to do something experimental that we're not sure is going to work. Uh we're not making commitments to users. We're making commitments to just build community interested in in this innovating. And if it doesn't work out, we can celebrate the attempt and let it live on somewhere else if it can't be integrated into the project fully. Um and along those lines, um y'all may have seen that I've been shopping a proposal that I call Fedora Innovation Innovation Lifecycle, which is inspired by what CNCF does, but it's not what CNCF does. Um I have a I have a second draft right now in Fedora discussion. It's been there for like a month. The goal My goal is after Flock is over and uh that I'm going to start the formal process to go through Council's formal process. We'll do the formal discussion period, and we're going to work towards uh implementing it. Uh the key idea there is that we're going to add like a sandbox portion of things where where it where interesting innovative things can show up in the project. Uh, we can can be worked on and then we can figure out how they get integrated um, via technical implementation changes and things like that or so that so that we're not trying to front-load all of that for things that are too big for the change process. For those of you who are familiar, we have a change process that Fesco um, arbitrates. I guess arbitrates is the right word. Um, that is that is really good for small incremental changes, but it gets very complicated for large big ideas, right? Um, even even stuff like what Michael's doing with the data uh, working group is probably too big to fold in as a change proposal. It needs to live and breathe as a proof of concept somewhere before we figure out how to fully integrate it and that's what he's doing right now. I just don't have a I just don't have a good process to basically place that in while he's doing that work. So, he's sort of doing it on his own and that the intention is to integrate that, but it's doesn't fit until it needs to iterate and then we know what the right thing is and we can put it in. Um, and that's that's just uh, something I talked about this week. Um, so so I want to make sure that we're baking in experimentation into the culture in a way that we de-risk it. We're not over committing too early and we have a clear exit path and clear expectations. Like the part of this is like anything that runs through this process in the sandbox, half the things in the sandbox should fail. We should not put it in the sandbox with the anticipation that it's going to succeed. Half the things should not make it. That that's a signal that we're using it the right way, right? Um, but part of that is being very explicit about how getting out of the sandbox works. You need to have I I in my proposal I put like three pillars. Like tech you have to show technical maturity to get out of the sandbox or you have to show that it has um market fit or it gets solves a problem, right? Um that there really is a need for this thing. And then And then the third is like it needs to have um sustaining community like community engagement. It doesn't have to solve all three of those things, but it has to solve make significant progress on at least two of those. And then the next stage you work on the third thing and you get it integrated. Um I think it's a great way for um um some vendor initiatives. Um I can help I'll say vendor because we have multiple vendors who depend on this project now. Like I'm sure you all saw that Azure is now based on Fedora, right? Uh Amazon Linux is based on Fedora. Uh Uh Red Hat's products are based on Fedora. Uh and all three of those vendors could use this process to experiment and get innovate technology, make progress on it with the intent of getting it integrated, but it's nothing's guaranteed, right? The idea is to is to show that this is worthwhile inside the project and not have to do it outside the project. Um and that's And And And like And Michael's thing is not even a vendor thing, but it would fit. Michael Winder's thing is also would fit super well and it's not a vendor product. It's a community aspect or community idea. So, I think it works and I would like everyone to take a look at the discussion or the proposal and provide feedback. It's the second draft. We're going to I'm going to make it a formal proposal and we'll move forward. I'm anticipating that we'll get a chance to try it. >> [snorts] >> So, speaking of experiments, uh I want to also talk about Fedora Hummingbird. Um this is something that came up for council. Um Um and we we And because because of the nature of it, we unfortunately couldn't make the discussion public cuz we were in a infrastructure gap. We didn't we we couldn't we don't have private tickets right now that we could open up. Um but um this is a really interesting um, approach, a new idea, innovative idea on how to build an operating system from the sources that are Fedora, right? Um, and it it's relies much more on automation. Um, and it's trying to do what I think is one of the one of the hardest things and one of the more the most interesting things. And it's very much an experiment. And if we had had the sandbox or the lifecycle process already, it's an obvious fit. And so together uh, the I think the my my intention is once we get once we get the lifecycle process, my intention and my understanding is the intention of the of the technical leads for Hummingbird uh, they want to be in they want to be in the lifecycle process. They want to be in the sandbox. They want to prove that this is worthwhile. They want to show market fit. They want to show it's technically mature, right? Uh, and then work towards full integration with the project. So that's and it and we're sort of stuck in a chicken and egg problem, right? Where where we don't have a process for this big idea, right? But when I proposed the process, we didn't have a big idea to run through the process. Um, and I think we're going to we're going to sort that out now over the next 6 months. Um, and and as we run Hummingbird through this, we're going to iterate on the process itself and make the process better for the next thing. Uh, so that so that it's we feel more confident that it's the right process for the big ideas. Um, without risking the rest of the project with a crazy idea. I want crazy ideas to live and breathe, but de-risk them in a way that we can continue to produce the mature output that everyone's relying on because that's the success. So, uh, I just want to summary what am I got? 5 minutes? I burn 5? Um, yeah. So a summary, um, you know, the Fedora users metrics look great. Everyone who's been Everyone should feel proud about those metrics. Everything's everything looks great. Um those of us in leadership have some work to do around project health and sustainability. Um uh we have to we have to dig into the data that we have access to about project health and we need to start figuring out a way to make some changes that are day and with data driven and figure out which of those changes are are positive impacting um so that we can get to a better place in three or four years. Um I would I would like to be able to say in three or four years to run that graph that we're we're through a wall and we're back up on the on the contributors. Um there's a lot of things here um that are going on. I would like this to seed some questions for the leadership panels that are coming up. Uh council's up next then Fesco after that and I believe mind share is tomorrow. Um so um yeah, I think I think that's it. Um things are we've got some problems but we're we've got some good work that we're outputs and we just had need to um address the internal contributor stuff and I think we can get there. I'm not like I said, I think this is just a general I think this is a generational switchover stuff and we just have to figure out a way to encourage the next generation to participate. So, thanks for your attention everybody. >> [applause] [applause] >> I mean, I have time for questions but we can just save them for the council panel because it's not just my opinion that matters. It's everyone everyone in leadership's opinion matters on this. This is this is a leadership lift. So, uh all I can do is get out in front but I do want I do want council to answer the questions with me. So. Or you just want to let's just roll with the council panel. We can we can take questions on council. Unless you have a question for me. We'll We'll take questions. >> The decline in package contributions weekly or daily, how does it split between Red Hatters and non-Red Hatters? >> Good question. So, we There are a whole bunch of questions I I still need to ask. I said the the data stuff That was a graph that Michael run it ran up for me yesterday because I happened to go to his workshop on the data stuff. We We've had We haven't been able to answer these sort of questions until very recently, even though we've had the data, we haven't had the tools to look at it. So, he He's invited me to drop in all my questions that need to be answered in a repository, and I'm going to drop in like 80 questions this week and start answering that. Um so, that is absolutely one of the Like I want to piece that apart. I want to understand where that is. Um so, I know where the outreach efforts need to go. >> Hello. I would like to ask whether uh in the declining graph of the packaging contributors, whether Flatpak is also included. >> Flatpak? >> Yes. >> No, my understanding is the Flat I don't know. I But, they they should show up as commits because it's just overall commits into the system. And so, if it shows up as a commit into our source git uh or a disk git, it should show up there. But, I'll take that back to Michael and make sure. But, right now I don't know if it's granularized. It's just I think this is everything that >> Because maybe a a good thing to ask would be also whether there are packages that are no longer provided as RPMs, but they are provided as flatpaks now. >> Well, well, that that that definitely doesn't cover Flathub, but it it would cover our flat our Fedora flatpaks commits. >> Everything that's a Fedora flatpak is also a Fedora >> Right, exactly. Right, so cuz everything that's a Fedora flatpak is a Fedora RPM, so um I understand the sense of the question, but but Flathub itself I we can't see in our in our data. He's got He's had his hand up for a last question. Last question. >> Uh yeah, I mean, I wanted to say that the stability of the Fedora is getting better over the years, so maybe there's no no need to have that many people involved, but well, the generational switchover uh I don't see anybody here close to the retirement, so I I'm I'm in I I just want to protect the base of the great people that we have, and if you also count with the like uh let's let's keep them. How how do we Well, I mean, why are we losing the people actually? >> Uh why is hard from data, right? That's interpretation, right? Um we have to figure out we we have to go back and look at who was contributing, and then ask them questions. But, that gets into uh we're just asking the baseline questions first, and then we can delve into that. But, but why from data is very difficult. We'd have to reach out to people who have stopped and ask them, but it's um >> Yeah, yeah, I guess. >> Let's be friends, the old and new people. >> Oh, no, no, no. Yeah, yeah, obviously. >> I mean, the bet on the young people might be uh risky. >> Well, you know, when I started, I was a young person. So, let's talk about risk. Um we have to trust people. We have to trust that they want to participate, and we have to mentor them. Um we've gotten better over 25 years. The peop- the people who were at the start, like me, we were not good. But we built a thing. And the And then And I Part of it is from from my mind, part of it is like the goalpost have creeped, right? Like, what we consider high quality now is not what we consider high quality 20 years ago, right? But but we But because it's a volunteer organization, we have to have a space for people who are at the start, who are doing skill development, right? Um we had to figure out We had to figure that out. Um part of it could be that we can have people use the sandbox process and do crazy things that are not good. But But that gets them in the door. And And it gives us an opportunity for them to feel like they're part of the project, and then they skill up, and then they gain uh maintainership rights, because because we've seen them learn. We've seen them like fail and get better, and we know that they are trustworthy. Not that they're doing the right thing, but they're trustworthy. Adam, that was the last question. >> Oh, I'm sorry. I just had a I just wanted to It's more of a comment than a question. >> Go off. >> We can maybe go ahead and transition. We're actually on time for the council panel, so we can >> Save it. I want I want this to seed question for council cuz my opinion by itself is not it not as important as leadership's opinion generally. >> So we'll ask we'll go ahead and jump straight in. We'll do a quick transition. Coffee break will be after the council session. We have all of the folks, council members, if you can make