Flock 2025 The State Of EPEL

I'm Troy Dawson. >> How do y'all? My name is Carl George. >> I'm the Apple Steering Committee Chair. >> And I am the Apple Team Lead inside the community community Linux engineering team at Red Hat. >> And this is our annual state of Apple talk. What is the state of Apple this year, Carl? >> Lots of stats. >> Lots of stats. You know, Matt didn't give you any stats and I know a lot of you are stat deprived and so I've come to fulfill using his old scripts. Um the first one as usual is Velociraptor. This is doing the actual IP hits that uh are coming to Fedora's Fedora. Uh it's not as good as as the new thing, but for the old things Apple 5 through 9 uh it's good enough. They are no longer counting Apple 10 or any of the Fedora things after Fedora 48. So the slide's going to get outdated after a little bit. Uh as we can see go to the next one. I like the the others. As we can see Apple 7 is still our lead thing. It's only been uh expired for about a year. So it's it's starting to go down. It's not into the Is that say 35 million or 3 million 500? I think it says 3 million 500. I should put dots on those things. So it's starting to come down. Um and as as we see what side am I on? 9 is starting to go up. Uh 8's sort of leveled off. 8's the purple thing. To be honest, it's okay even though 8 was my first distribution that I did with rail, but um 9's better and 10 is even better. Okay, let's Let's to the next one. Now, you'll notice that we see eight and nine. 10 is actually out. Uh I was using um Matt's uh scripts to do things. And 10 is so low right now, it's in about 10,000 users in Apple, and it kept just dropping it off. It says, "That's too low. That's not even the a blip in the thing." So, you'll you won't see 10 on either of these. Um but this is using Bron Maybe the >> Go ahead, say the name. >> Brontosaurus Sapphire. It's it's a little easier than Velociraptor cider. Brontosaurus Sapphire. Okay. But looking at that, we actually do have about 3 million with with Apple 8. And nine has got this uptick. We'll see if it stays that way. Um I I know where that's coming from, and and I'm curious if it's going to stay, but I'm not going to tell you. So, next year, if nine continues to have this nice fast track, then I'll probably tell you where it's coming from. But if it just goes away, then I won't. >> Is it people going straight from seven to nine? Is that your theory? >> No. Uh I I I did see where where this actually is coming from. Um but uh I I'm not going to name names or anything. So, so those of you in this room that know who it is, sorry, I'm not going to tell you. Next one. Yep. Um and this is just an easier way to say to see that nine is I keep forgetting we're going the the other way. Nine is starting to eat eight's eight's numbers. And that's fine. Nine is a good distribution. I like nine. >> And this is still the IP based one? >> No, no. Uh Brontosaurus Sapphire is is Count Me. Yes. Sorry. Yeah. And again, 10 is here, but it's it's so small that uh it's it keeps getting just bumped off the thing. So, 10 again is 10,000 as opposed to 2 million. Now, this is fun. So, this is the number of packages, not how many downloads. This is the number packages. >> Source packages and coding. >> Source packages, I should Yes, thank you for clarifying that. Source packages. Now, the fun thing is is this is the first time you see the blue is Apple 9. Blue finally passed Apple 7. Apple 7 has been the standard. It was has been around the longest. 6 never got up to even close to as many packages >> of life now. >> That's true. >> For a year now. >> For a year, but I kept it on this chart just just to show that 9 has finally passed 7 as far as the source RPMs in Apple. Now, these little these little things are when 9 was released. This is This is when Rail 9 was released. We had a a whopping 2,617 packages when 9 was released, and we thought that was great. And we marked it through all the things because when 8 was released, well, this is when 8 was released, that was zero. We had zero packages for 6 months. It was It was bad. When 7 was released, uh we didn't quite have this amount of numbers. Anyway, so this is the last time you're going to see 7, but when 7 was released, it was about as 9. Now, let's look over at 10. We go over to 10. So, we hit 2,600 like What was that? 8 months before Rail 10 was released? When Rail 10 was released, we had 5,431 packages. Thank you. Not just me and Carl, but jeez, we >> You're in the top 10. >> Yes, still. >> I'm top 20. >> You're top 20. But, thank you all of you Apple contributors who've helped to make uh these numbers possible. It's helped us, it's helped a lot of uh end users. Thank you very much. These This number is for for when Rail 10 was released, 5,431. That was a lot. >> Mhm. >> And thank you. It was It was great. This is the last time you're going to see this chart like this because we're going to have to redo it just cuz 10 releases has really thrown me off. I don't know what we're going to do. >> Yeah, whose idea was it to add minor versions to it? >> It was >> We'll get to that. That's foreshadowing. Foresha- Yes, Tory. >> Filling technique. >> Yep. I think it was all of our idea. >> Sure. >> It was only instigated by one person. Okay, this is the other number that I really like is the number of maintainers because this is how we judge the health of of Apple uh is by the number of maintainers. Now, you'll see the the blue the little thin lines are the individual things and the blue, which is thick, shows the number of uh maintainers. And it it's it sort of drops each time. Wait a sec. How do I Total this per date and total overall. Okay, that's right. So, each each date time, we're do we do this every 6 months. We check this every 6 months. And we see how many is there at that date. And you'll notice that it drops whenever like um Apple 7 went away, there was a lot of maintainers dropped in Apple 6 and all that. But then we look at how many unique maintainers we've had overall. And that's where the orange one is and that's always been going up. So even though we lose some maintainers, we also get more that keep filling things in. And we expected this Apple 7 drop. But you notice if you can, it's a little hard to see, that we're on an upward trajectory still. So we're still able we're still gaining maintainers, even though some people built things for 7 and they've gone a few years ago, we're still getting newer and and newer people and that's one of the things I am very grateful for uh in Apple is that this is another one of our health maintain health metrics and I feel that we're still a very healthy thing. Despite that drop, we expected it, we're still going up. >> Ready? >> Yep. Okay. Oh, this is my the other the other health one, which I don't know if it's healthy or not. >> The trend is not in the positive direction. >> This is trend is not in the positive Rust has has sort of thrown all my my metrics out the door. Uh number of packages, you can see again Apple 9 has passed Apple 7. Source packages, number of maintainers um it's sort of interesting that we have 241 Apple 10 maintainers meaning maintaining more packages than Apple 8, which has has had 511. All of you packaging Rust, thank you very much because you guys wow, the number of us >> Thanks, Fabio. >> Yes. Oh, you guy. Well, actually there's a couple others, but Fabio, yeah, you are now our as far He's number one, right? Can I name names? Now that Now that it's not me, I can name names. >> Yes, it's not bragging. >> Yes, it's no longer bragging. Anyway, um Yeah, this has been a health metric we've had and uh Rust sort of threw this metric out the door. So, anyway. >> What we really need to do is keep getting the package number up, but get the maintainer number up also, so the packages per maintainer is lower. >> Yes. But uh Rust keeps adding more and more modules. Um Go did finally start bundling. I saw that as a change proposal. So, anyway, all of you maintainers, again, I've said it like four times, but I'm very grateful for you. Thank you very much. Let's go for the next one. EPEL 8. Has anything changed in EPEL 8, Carl? >> Not really. It's uh it's going strong and it will be end of life the same as RHEL 8, uh May 2029. >> May 2029. >> all we have to say about it. >> Yep. That's a long ways from now. EPEL 9. >> That's a little bit sooner date for something relevant. Um we have EPEL 9 next. If you're familiar with that, that's actually going to go end of life the same time same time as CentOS Stream 9, which is May 2027, but then regular EPEL 9 keeps going through the full RHEL 9 life cycle till May 2032. >> 2032. >> So, that's really the only relevant stuff we have for those. Interesting stuff what people are probably here to hear about is EPEL 10. >> Well, let's hit the button. EPEL 10. Carl, is there anything exciting about EPEL 10? >> Nope, it's boring. >> Oh, okay. >> So, in case you hadn't heard, EPEL 10 is a thing now. Um we have implemented minor versions in EPEL 10 now, uh which is something that's been tossed around as an idea in the EPEL community for a long time, uh and we finally made it happen. Took a lot of work. Took some external factors, too. But it's EPEL built for every minor version of RHEL, including CentOS Stream 10 as the leading minor version. We'll get into that in some charts in a minute, some diagrams. Um So, but to explain that better, I should do this visual representation of how EPEL used to work. >> Do you want to swap me places or do you not like >> closer to the notes. >> Oh, okay. Do you want me to ever point at things? >> Yes. Be my Vanna White. Now, so to tell how represent how EPEL 10 is different, I'm going to show these diagrams. Um Basically, EPEL 7 started shortly after RHEL 7 launched. And just the solid blue line here, single branch, was built against the solid red lines up there, just the current minor versions of RHEL. So, it was it was built alongside those minor versions, but because it only had the single branch and always bumped to the next minor version, it was effectively major version only. Um Those packages, those resulting packages, would be used on both RHEL and CentOS Linux and other derivatives like that. And that mostly worked. I say mostly because those rebuilds would follow behind RHEL by about a month. Um And the reason it mostly worked, while RHEL is very stable, it's not frozen completely. Sometimes there are library changes between those minor versions. Um And in this in this model, back in the 7 days, whenever one of those libraries a library would change in RHEL, some EPEL packages would become uninstallable or they'd even block upgrades of other packages, which wasn't great. Um The maintainer could either rebuild it immediately to fix the problem for RHEL users and then break it for CentOS users or they could leave it broken for RHEL users until CentOS caught up. Neither answer was good. Uh but because there was no good solution, we basically just kind of ignored it and said, "It'll be eventually consistent in about a month." EPEL 8 started out the same way. Um Started launched a little bit later after RHEL 8 just because of a few factors, mostly modularity, some other stuff. We won't get into that. >> Yeah. >> Don't need another therapy session. But, uh then a little bit after that, CentOS Stream launched. And that basically takes over as the major version branch of RHEL. And we had had this interesting problem where we still wanted to use EPEL packages on it. But, instead of CentOS being a month behind RHEL, now it was about 6 months ahead of RHEL. So, that was a bigger gap and harder to ignore when there was a library difference that came up. So, a little later into RHEL 8's life cycle, we launched EPEL 8 Next. And the idea there was, uh without disturbing existing EPEL workflows, we wanted to give maintainers a way to build against CentOS Stream when it was appropriate. Like for say a different a different QT version or lib LLVM or any of those things that sometimes change between minor versions. Uh the idea was that it wouldn't be just a completely separate EPEL, but rather a layer on top of EPEL for CentOS users. So, you'd have RHEL and EPEL and then CentOS Stream plus EPEL plus EPEL Next. That solved the basic problem. Uh it made it where maintainers could create a different build. Um and I mentioned we implemented that without disturbing existing EPEL workflows. It was optional and maintainers didn't have to know about it. They could just keep building in EPEL 8. And nine times out of 10 or even more, they wouldn't wouldn't even need to use it. Their packages would just work because of how overall stable RHEL itself is. For EPEL 9, we took a new approach. Um separate from that problem, there's another problem of just the package availability. Some of the charts uh Troy showed earlier about EPEL 8 lagging real far behind on packages being available. Uh we started EPEL 9 about 6 months, 5 months before RHEL 9 in attempt to solve that. Cuz really, the only way to solve that is just give the community maintainers more time to add packages. We could just, you know, add throw in more people at the problem would only get us so far and you know, hiring budgets aren't unlimited unfortunately, so the best answer is just, you know, make it available and then uh call to action to the community. So, that's what we did. >> I was going to say, it's also a community oriented thing. >> Yes. >> Um if if Red Hat threw people at it, then it wouldn't be as community I don't get paid to do KDE and you well, you don't get paid to do packages. >> I get paid to work on Apple overall. >> Yes. >> Specific packages, no, you're right. >> Yes. >> So, yeah, we want The idea was like, how do we enable the community to do more things and have packages ready? Uh from the Red Hat business perspective, paying people to work on Apple was more about we have constant complaints about say, I'm not upgrading from RHEL 7 to RHEL 8 because these packages in Apple that I know are unsupported aren't available yet. We wanted to you know, the business side, they wanted to remove that as an excuse not to upgrade RHEL versions because being on an older version makes the customer harder to support in general. Um So, that was kind of the justification to create the team that I'm on now. Um So, that was the first thing we did on that team was we started driving like, how do we start Apple 9 now? Let's not wait. Uh and the way to do that was we initially built Apple 9 against CentOS Stream 9 and then we switched it to RHEL 9 when that was available. Um We were in the community telling people that CentOS Stream is, you know, a preview of the upcoming minor version of RHEL and this was our chance to prove it. We're going to build against CentOS Stream and then it's going to work on RHEL 9.0 cuz it's, you know, a minor version ahead and to my knowledge, we didn't have a single bug report uh about incompatibilities because of that approach. So, it it basically worked. >> Yep. >> Um We still had the minor version problem after that. Uh so, we also made Apple 9 next same as in 8. Um it just kind of worked like this where we switched which one Apple 9 was building against. Uh but after that point, they're still separate branches. And that's when we started real really thinking about the problems that we had with Apple Next. It was often mistaken as a standalone repo. Some maintainers and users thought like, okay, well, this is the Apple I use for CentOS and this is the one I use for Rail and why is this one not installing right? They needed to be used together. There's a decision process process for maintainers when they needed to use it and we had no inheritance between if you built a package in Apple 9 next, you also had to rebuild it in Apple 9 6 months later, which for some things like choice KD stack is a very big task. >> Yep. I I also had the problem of >> [snorts] >> the delay. I had to decide when to release the KDE. I could build it fast and then all the Rail people were happy or >> The same condition problem. >> Yeah, then the Alma and Rocky people were were not happy or vice versa. >> So the So yeah, that was the observations we had about it and we started thinking about Apple 10 and we realized that Apple Next was really just a subpar bolt-on solution tacked on after the fact. And it was a a roundabout way to target two minor versions at the same time. So we decided the path forward is just to fully embrace minor versions, which is what we're doing in Apple 10 now. >> Yay. >> Apple maintainers are Fedora maintainers, so we looked at Fedora branching for inspiration. A new Fedora branch A new Fedora new Fedora major versions branch from Rawhide. So Rawhide is a preview of the next major version. Like right now Rawhide identifies as Fedora 43. Similarly, new Rail minor versions come off of the CentOS Stream major version. So CentOS Stream is a preview of the next Rail minor version. There's a lot of similarities there and so we just modeled Apple 10 after that. The Apple The main Apple 10 branch builds against CentOS Stream 10 you know, continuously not just at first and then we'll have separate Apple 10.0 10.1 branches that build against the corresponding versions of Rail. Builds that get done on that main branch, like if you built a package here, it just showed up in Apple 10.0, just getting tagged into it. Um, similar to Fedora. If you add a new package to Fedora today, it'll by default it's in Rawhide only. You have to go back and ask for the Fedora 42 branch if you want to put it in there also, or the Fedora 41 branch. >> So, as an example, uh, using my KDE example, I I built the KDE on this on the CentOS CentOS stream branch. When RHEL 10 came out, it was there. Um, I didn't have to do any last-minute rebuilds. It just came right out. And it was pretty slick. >> We have 5 minutes left. We should go quicker and get Q&A. >> Okay, let's go quick. >> All right. Um, so, that is how that's working. Uh, we launched it in in December, a little about 6 months ago. We also had a soft launch period before we announced it starting at Flock last year in August to give maintainers even more time to prepare their packages. So, instead of announcing it with zero packages, um, we were actually able to launch it with 10,000 packages, uh, source packages from about 3,600 source source packages. And then by the time of the RHEL 10 launch, which was last month, uh, we were up to 17,000 packages from about 5,400 source packages, like Troy mentioned earlier. So, that was really exciting. Lots of patting on the back all around. We were very happy with how that turned out. And like Troy said a couple times, we couldn't have done it without the Apple Apple maintainer community. Uh, I definitely did not build 17,000 packages myself, not even close. I was I'm responsible for a very small percentage of that. Troy a little bit more just cuz KDE has a a lot of sprawl with the dependencies. >> Yeah, but nowadays it's it's a small percentage. >> Uh, come and join us on Saturday. We are having an Apple Hack Fest here at Flock. Uh, some of the topics we're going to have, uh, we'll have an initial Q&A section. The idea there is Fedora contributors that are already working in Fedora and are new to Apple or want to get involved in Apple. That is your time. It's explicitly geared to how do I get started, you know, from Fedora to EPEL and get involved there. We'll also do some early EPEL 11 planning, talking about what we should do to keep improving things. What's our next step to make things better. Probably not as big changes as 10, but >> Hopefully not. >> Probably the theme will be starting even earlier somehow. And also policy revisions, documentation, all fun things like that. It'll be pretty free form. Those are just kind of rough topics to get us started. Maybe some package reviews. You want to talk about the desktops a little? >> Yes. I being the KD guy, I get a lot of questions. The biggest one is why are there no more no other desktops other than KD? Because real 10 has X has no X11. That was opposite of what I meant to say. There is no X11 in real 10. >> No Xorg server. The desktop also get after you for that. >> Oh, good. Cuz that's that that is a very distinct point. But Xorg server itself is not there. Um and that means a lot of the desktops that we've had have been X11 based. Uh KD is Wayland, Gnome is Wayland. Um I know some of the others XFCE is is working on it and LXQT is working on it, but they aren't there yet. Um I just wanted to let people know. That's answers to the question you keep asking me. >> And I'll elaborate on that a little bit more, too. Um Just like any other packet, other desktops can be added, but just like any other package, the dependencies have to be available as well. If your desktop depends on Xorg server, somebody could add Xorg server to EPEL 10. I'm not going to do it. >> to do it. >> I don't need it. Uh but there is a lot of demand and so far zero volunteers. So if that is something that is important to you, you can step up and maintain There's nothing stopping people from adding XOrg server to Apple 10. And I know that it does build. I actually tried to build it and then I quickly deleted the copper because I don't want anyone to come to me with issues for it. >> No. No. >> So, on the alternative is work with those desktops upstream to improve their Wayland support. I know one example, XFCE, they did a release with experimental support for Wayland. Um and talking to the XFC maintainer, Kevin, um he doesn't feel comfortable adding that to Apple 10 with just the experimental label. So, maybe when XFCE does a release where they drop that label, maybe we'll add that to X to Apple 10. That's probably the closest one to getting added. But, if you're involved in upstream development, start work hammering on their Wayland issues and help get that sorted out um so we can have more desktops available for people that like doing things different. >> Or, there's plenty of Wayland desktops in Fedora. Bring them over. Uh I would love Cosmic. >> Sway probably could get added. That's a Wayland desktop. >> Oh, uh actually I think it is. Oh, sorry. You've got your You've got the button. >> Oh, yeah. >> [laughter] >> Um real quick, we're out of time. Uh KDE 6.3 Uh we do have QT 5.15 in in Apple 10. It's going to stay there as long as Fedora supports uh QT 5. whatever. Um I'm not going to go to insane lengths, but uh a lot of people still want QT 15. And that's all I'll have to say. We're short on time, so We just had our Apple our second Apple Steering Committee >> Second ever. >> Uh voting. So, congratulations to Michael Lind. Wait. Wait. Why did I >> Michelle. >> Mish Mish Well, first off, Michelle Lind is taking a a break. So, congratulations to Robbie Cal Oh, my goodness. >> Calicote. >> Calicote. I just realized I've never pronounced his last name. Kalacote um was elected to replace Michelle. All the others, which would be Neil and Davida, are still there. So, congratulations to that. >> Real quick, if there's a package you want in Apple 10 that's not in Apple 9 or that's in Apple 8 but not Apple Apple 9, so so on, you can go to Apple apple.io/request. We have a guide about how to file a bug for that with templates and everything to communicate with the maintainer and get it added. >> And we're at questions and answers. >> A minute over, but I guess there's 5 minutes till the next session. >> Yeah. >> So, maybe one or two questions? >> We got one right in the front. >> With the new branching in Apple 10, do the package maintainers have to build for every single branch or is there automation for that? >> There's not automation, so just like Fedora, if you were it's up to you what which ones you want to build in. We do we're doing mass branching, so that way if you do it if you start it early enough, like if you requested an Apple 10 branch like in December, then you automatically got an Apple 10.0 branch and we also tagged the builds in the Apple 10.0 builds, we tagged them into 10.1. So, there's a little bit there to help maintainers along. We don't actually do a mass rebuild like Fedora does, but that's something we could look at in the future or some better way to tag things forward. >> Yeah, cuz it it's if you've got a CVE, you suddenly have twice as like within a year or so, you're going to have twice as many builds to do. Um the only Apple packages I maintain are the ones I've been bullied into maintaining and I don't need the extra work. Um so, it is within >> power to tell people, "No, I only want it in the latest branch." >> That was like I've had things where people have gone about my back and branched for me or said, "I will maintain the branch, so I branch it and assign them. >> Yes, to be fair. >> Maintain a ship to them. >> just block it, but you can say I don't have time for this. I can add co-maintainers. That's one of the two things you can >> like I've had situations where I've had someone said I'll co-maintain it, and literally 2 weeks after that I've branched it and assigned it to them, they often reassign it back to me, and I'm stuck with the branch. So like a lot of the Apple stuff is quite aggressive in the way people want to do it. Um >> That is something we should delve into further at the Hackfest, because that's like when we we have another process called the stall maintainer thing, where if you don't respond in time, I think we started that at 2 weeks, and then because people thought it was too fast, we bumped it out to 3 or 4 weeks. And I'm up to I'm open to like making that even longer or shorter, however people want to go with >> cuz there's like some packages that like because of the length of Apple, don't make sense to be in Apple, because they're already dead. >> 100%. >> Um but yeah, I've had situations where it's like this doesn't make sense to be in Apple, and there's no communication. It's just feels like a bully process. >> Yeah, we definitely don't want maintainers to feel like that. Um at the same time, the other other side of that coin is that uh we don't really own packages. It is, you know, packages that we are currently maintaining. So we want to strike a good balance between I don't have time for this, but enabling other people to do it. So we're still working on that balance. Come talk to us more about it at the Hackfest. >> That that does bring up a good point though, in that a lot of times there's no real need to rebuild the packages because RHEL is a stable ABI, [clears throat] right? So by the time you get to RHEL 10.10, and you've got all of these branches, um it would be it's horribly tedious. Why not have >> have all the branches for one. Yeah. >> Cuz you said something that I wanted to comment on. >> Yeah. >> Um you don't after go back to the the branching team. Once something has been um you'll only be doing two. >> Okay. >> Because because then they go back to archive. >> There's a There's a very short time where there'll be three, right before the branch >> Oh, wait. That's way too >> So, even if there's three though, I mean, why not just have a button to There's a scripted way to then >> Let me ask you this. Is there a button in Fedora to do that? >> No. But but but there's >> [laughter] >> But there's a very big difference between Fedora releases where everything changes and >> And all that I see what you're saying. >> minor version bumps. You are You are correct. My answer is we're brand new at doing minor versions and we're actively working on how do we make this better and that's one of the things I want to talk about at the Hackfest. Come to the Hackfest with us. It's a great question. Not enough time to get into all the answers. >> thought. I mean, I don't think it'd be too much work. >> The other thing I want to comment on you're not going to have the 10 branch We're not going to have the 10 branches simultaneously whenever uh whenever like Rail 10.1 comes out, the 10.0 branch is going to go to the archive. So, it'll be ideally generally two branches at a time and then a very short period uh where you'll have three. Similar to Fedora where like, you know, F43 will get branched and then we'll have 42, 43, and 44 and then just 43 and 44. >> What about LTS Rail 5? Because you >> You mean EUS? >> Yeah. >> So, yeah. >> We we >> At least in the past it was like even numbers. >> Yeah, we we explicitly left that out of this initially, but it is set up where at any point we could make EUS branches opt in for maintainers and that's also something I want to talk about at the Hackfest. >> Yeah. >> Yeah, right now there is no EUS Apple at all. >> That would make it a lot easier. >> We Yep. Uh we are out of time. Thank you for the questions because yes, those are things we need to bring up. And we'll talk to you next year. Well, at the Hackfest in the next year. >> Saturday.