What is EPEL | Fedora Podcast 057

What if there were some changes coming to the extra software that you get in Fedora? Carl George will join us. We'll ask him some questions about Apple. [music] >> [music] >> Welcome to episode 57 of the Fedora podcast. My name is Noah Chala. I am your host. Delighted to be here. And joining me my co-host, Mr. Eric Hendris. Eric the IT guy. Welcome in, sir. >> What's up, Noah? It's been a bit. Glad we finally got our schedule sorted. It >> has been awesome and it's going to be a great episode. We have the opportunity to talk to Carl George. He joins us from Red Hat. We're going to talk about Apple and some ideas that Carl has to make Apple a little bit cleaner, a little bit neater, tighter around the edges. >> How are y'all doing? >> Good. Carl, welcome into the program. Can you tell me a little bit for those who have maybe living under a rock and aren't familiar with you, can you tell us a little bit about who you are, what you do at Red Hat, and what your involvement with Apple is? >> Sure. So, my name is Carl George. Uh, I currently work at Red Hat. Uh, I've been involved in the Fedora project in Apple longer than that. Uh, got my start doing RPM packaging at my previous employer, Rackspace, in 2014. Um, part of that job involved maintaining a few Apple packages and that that was kind of my on ramp into the Fedora project broader. Um, since then, since getting started with Apple, I've done a lot more things in Fedora than just Apple. And um I get that question sometimes about how do how do I get started with Apple. I wouldn't recommend doing it the way I did. Get started in Fedora first. [laughter] That's much smoother path. Um learn from my mistakes. But yeah, uh when I started at Red Hat, I was working on the CentOS team. Uh now I actually am the team lead for the Apple team at Red Hat inside the community Linux engineering group. Uh, so luckily I get to focus my entire day on what makes Apple tick and how to improve things and make it better, which is part of uh, part of what y'all looked me in here for today to talk about. >> For sure. And and fun fact, Carl and I actually went to new hire training the same week at Red Hat. Uh, that's how I met Carl in person and went, "Oh, wait. You're you're that guy? Okay, cool." >> Yeah, we were uh, a couple of Kevin Bacon degrees apart. We knew a lot of the same people and uh, yeah, got to talking. Can you tell us a little bit about your day job, uh, Carl? What does that look like and what did it look like transitioning from a volunteer role into doing this on a day-to-day basis? >> So, it wasn't a wasn't a direct transition. I mentioned getting hired to work on CentOS. Um, it's a little bit of a funny story. Uh, the guy that hired me at Red Hat, Jim Parin, uh, I had told him for years, when are you going to hire me to come work on Apple? and he would say nobody gets paid to work on Apple. But then he got this other wreck and I went to go work on that for a little while. Um but then after doing uh working on CentOS for a few years, uh my VP approached me and asked uh hey how would you like to start a team working on just focused on Apple exclusively and I said that's kind of funny because that's what I asked Jim for in the first place and he said it wouldn't happen [laughter] and then it eventually did happen. So things can change. Uh the transition was not direct. Um, but more broadly to your point, um, a lot of times I'll get asked questions like, "How do I get how do I get hired to work on open source or how do I get hired at Apple or at Red Hat specifically?" Um, it is not by by any means an instant thing, but participating in open source is a great way to get a job working on open source. Sounds kind of obvious when you say it just in one sentence, but um, [laughter] some people want to get hired and then start doing open source things, but it usually doesn't work like that. usually become a participant or even a leader in a lot of these open source spaces and you get your name out there and you're in the right place at the right time whenever some company whether it's Redhead or otherwise is able to say we have a headcount we want to put a resource on working on this thing specifically and contribute back to open source and you can't just run up to that with no experience or no familiarity and say yeah yeah I'll take that and I'll do that now you kind of have to already be in that role doing that volunteer as much as you're able to in your volunteer time and then then the company makes that observation and says this is this person is the natural fit to fill this. We want to hire this person to sustain this part of open source. You've used the term Apple. I use the term Apple as we introduce the episode. And there's at least one person listening to this episode, Carl, that's saying to themselves, what in the world is [laughter] this Apple thing they're talking about? So for somebody who isn't familiar, explain it to me like I am five. What is Apple? Sure. Apple is an acronym for extra packages for enterprise Linux. Um, sometimes I'll get uh with my hick accent, people think I'm saying Apple and they'll thinking they're thinking of the uh the phone company, but no. Uh, extra packages for Enterprise Linux. The way the the packaging ecosystem works and the components of these distributions work. Uh, Fedora is the parent distribution. CentOS is based on Fedora and then Red Hat Enterprise Linux the Red Hat product is based on CentOS. um rail and cents are very closely related. It's CentOS is almost like the major version branch of rail. So they're very tightly coupled. Um but creating that product in the end there's way more packages in Fedora than Red Hat wants to support for their customers for a long-term basis. Uh so the way it works out is only about 10% of Fedora package packages actually make their way into CentOS and then into Fedor into Rail. every everything else that other 90% of packages they're not they didn't make the cut because they're bad software or not useful it's just things that Red Hat didn't want to commit to maintaining in the product but all of those things Fedora makes it easy to consume those things on CentOS and Rail by rebuilding them from the Fedora package to be compatible with CentOS and Rail and that's what Apple is it's that extra it's not all of that extra 90% all at once automatically but that 90% is the pool to that packagers can take and build in the Apple repo and have additional communitymaintained packages that they can use with the enterprise product or with the community edition CentOS. >> So as a former Linux systems admin uh I I used to use Apple a lot there there was some cool monitoring tools some extra utilities things that weren't in the base operating system whether that was Cent Linux back in my time sent to stream nowadays or Red Hat Enterprise Linux or some of the offshoots. So, as a CIS admin, I was used to installing Apple just kind of as a default and installing things like HTOP, which gave you kind of a graphical u I guess a a terminal based uh output of your system resources. Uh but that's not how it works with Fedora. So, how and and that can trip people up. You go go on your Fedora machine, you try and install Apple and it it doesn't uh doesn't quite work out. So, how how is Fedora and Apple kind of different? Apple is a subset of Fedora kind of like Redhead Enterprise the whole enterprise Linux package set is a subset of Fedora. So uh we actually have a conflict in the what we call Apple release the package you used to enable the repository where it shouldn't even install on Fedora if you just pulled out the repo files and did it manually to force it then it still wouldn't make a diff it wouldn't really make a difference because the packages in Apple are already in Fedora. So you're not gaining anything extra. Uh it's a little bit of a pet peeve of mine when I see people recommend to add Apple to Fedora because I'm like why? Everything's already there. You don't need to. Also, it's not going to work because like gibb is going to be different and this library is going to be different and this and that. So most some of the uh the non-architecture specific packages probably would install and might work but it would be purely accidental because those packages are targeting a different operating system entirely. >> [snorts] >> Can you tell me a little bit about uh the this golden rule that I understand that exists in Apple with regard to replacing a package that's in the base OS? >> Yes. Uh it's kind of built into the name, right? Extra packages. Uh the golden rule, the guiding principle is that um you know add having an add-on repo for rail is not like a new concept. you know, thirdparty repos have been a thing on every distribution for a long time. Um, what people have experienced sometimes is that if you have a add-on repository that replaces packages in your operating system, that can cause dependency problems or other issues later on down the road or even right there on the spot. Uh, so what Apple wanted from the very beginning was we are not going to replace any software in real. We're only going to provide additional software. So in theory, you can always add this repo and do everything you would that's, you know, anything you would do with just rail by itself will work exactly the same because Apple shouldn't be interfering with any of that. Um, that guiding principle I think is why Apple has the reputation it does. Like Eric mentioned that he just would add it without even thinking, not not even, you know, it was just kind of automatic to have that added. Apple has a really good reputation for that because it is very reliable by being only an additional set of packages. Um, of course, you know, like anything built by humans, mistakes have been made in the past. And, you know, we'll find if we find something like that that somehow missed the automatic checks to get into Apple, usually it'll be more like it was in Apple first and then re added it and it's like, oh wait, we need to remove it from Apple now so it doesn't conflict to stick with our guiding principle. So, it's not always 100% perfect, but that is the what we strive for all the time. [snorts] >> So, you you mentioned a package installation. And is that is that all it takes to get started with Apple is just install that uh that DNF package? >> Yes, with the caveat we have one extra step on our website which is uh since real 8 there's a repository called code ready builder. Um in theory that repository was supposed to be just like devil packages and other things that you wouldn't need for like runtime for an application. In practice, Apple packages do often depend on packages from the code ready builder repo or CRB. So, our first step is you turn on that CRB repo, which is available by default, but it's just disabled by default. Um, you enable that to make sure you have all the available dependencies and then you install the Apple release package. That'll set up your system with the repository configuration and the GPG key and then you'll be able to install packages. So, just a two-step process. Technically, you could just install the release package and I don't know, threequarters of the stuff will install just fine, but the rest of it might will have dependency uh dependencies in the CRB repo and then you'll have errors that don't make sense until you realize you missed a step. >> Talk to me a little bit about the dayto-day. How are decisions made? Who governs? Who maintains? Who makes Apple a re reality? >> Sure. the the governing body for Apple is the Apple Steering Committee uh which I'm also a member of. That is technically a uh not a subset uh delegated authority from the Fedora Engineering and Steering Committee, Fesco. Um basically Fesco didn't want to also have to govern all of the things around Apple. So they created the Apple Steering Committee just as a subcommittee sort of things. Um originally Apple steering committee was appointed. It was just people that basically the people that were around creating Apple made themselves the steering committee and then um over time I think it was about two years ago we actually implemented elections so that way it's not just people that have been sitting there and don't want to give up their seat. It's actually uh we have terms and you have to run for election and get elected. And so we've had a I don't want to say churn but we've had a few new faces show up which is great. It's always good to have new voices uh get on the steering committee in the last couple of years since we started doing elections. um and also a lot of people that have been been around doing it for a long time. So, uh that is the governing body. Um EP as a whole, who controls EP and who does things in EP? It's really any Fedora packager that wants to participate. Um I'll get that question sometimes where someone tells me, hey, your job is EP. Can you go fix this package or do this thing or add this package? And I say, whoa, whoa, whoa. I'm not a maintainer of that package. I don't have any say over that. Um, it's still very much like Fedora itself in that individual maintainers have a lot of autonomy over the packages they're the maintainers of. Um, they're not the owners of the package, but they're the current steward of the package. Uh, if they were to disappear for whatever reason, get busy with life or something, you know, hopefully not worse happens, another maintainer can pick it up. So they're still Fedora packages, but the individual package maintainers for the Apple packages have the autonomy to decide which goes in which Apple branch, what which versions they're on to a large degree within the updates policy. Uh and they they make most of the things happen for Apple. The steering committee is, you know, just steering the ship. Any kind of adjustments to the rules, usually helping with the documentation and policy, making sure that's clear for maintainers, uh, looking at workflow improvements, uh, looking at automation improvements, things like that. Uh, keep Apple moving in the right general direction so that way the individual packagers can just worry about their one package and then the rest of the stuff is right where they need it to be uh, to get their job done. >> That's so open source of you. >> [laughter] >> Um, so with with the steering committee, I assume that on major policies and decisions, changing of of scope, that kind of thing, there is a voting process. You want to walk us through what that looks like? >> Yes, we have a it's basic majority voting. Um, most of our votes take place during our weekly meeting, uh, which is in it's on the Fedora calendar. It takes place in Matrix, uh, the Fedora meeting one channel if you're already in Matrix and familiar with it. And uh most of the things that come up for like proposal we'll vote on during that meeting. Um we do have a fast track similar to FESCO. We have a fasttrack process where if something's urgent uh we can kind of ping all the steering committee members and do a ticket vote to get it done quicker. But generally we'll vote once a week on anything any pending business. >> Do you mind if I sidebar for just a second and ask about the decision to go with matrix? I feel like I have seen a uh pattern that spans a number of different companies, a number of different organizations, and it seems like they're all kind of landing on on a shared platform. And the federation aspect of that is interesting to me because it means you, Carl, who is working on Apple for Fedora, can talk to somebody who's working at Canonicle or else. Um, and so it's anytime I see multiple different entities all doing separate things but all sharing a similar open source product, it gets me curious how did that decision land and from your perspective how has it been converging on matrix? >> Uh, like any change it hasn't been uh completely smooth or without growing pains. Um, I generally like matrix. I'm rooting for for it to succeed overall in other open source projects. Um, but that doesn't mean I think it's perfect and it doesn't mean I don't have complaints about it. Um, I don't I won't get into that too much. Like there's like anything else, there's pros and cons. Um, some people think that, you know, we traded one set of problems with IRC for another set of problems with Matrix, but I do think it was overall an improvement. uh you know not having to run your own bouncer being a little bit more approachable for people getting started with the chat I think is a good thing although there there is a s still a significant I think that's probably my biggest problem with matrix I said I wouldn't get into it too much but the the onboarding around it and getting the key stuff set up uh I've helped several new contributors get started with that and it almost always trips them up and I wish it was a little bit smoother so I'm hoping that improves more in the future uh like I said I am rooting for Matrix I like it um but Yeah, it I think it's a good thing like you mentioned the the kind of the federation aspect a little bit. Um where because I'm on Fedora's matrix, anyone else can get my Fedora Matrix handle and they don't have to be on the Fedora Matrix server. They can message me and get a hold of me. Um and usually like in my conference presentations, I'll say that that's the I'll have my email up on the screen, but also my Matrix handle and I'll tell people you can email me, but I'm not going to make any promises uh about getting back to email. If you send me a message on Matrix, I I guarantee you I will reply eventually. Um this is actually really great. So with big change comes some problems and you got to fight your way through. About a year and a half ago you launched Apple 10 and as part of that um that was kind of like a soft launch and we're now at a place where um there are increased packages. You cross that 10,000 package mark. You're now past 25,000. Can you talk to me a little bit hindsight being 2020? Um what has been really well? Where do you think there's some opportunities for growth? And um from the original plan of how CentOS and Rail systems would request repo, where did it break down? >> Yeah. Um so Appleton was a big ambitious thing that uh you say the last year and a half, which feels like it way underells it to me because we were planning it quite a bit before that. I know that wasn't you weren't trying to minimize it at all. It has it did launch about a year and a half ago. Um but the story actually goes back a bit further. uh if you if you'll allow me to um >> please. >> So back before the Apple team got started uh I mentioned I was on the CentOS team and I knew the importance of Apple coming from my background and using it um and we were seeing that um in the in the olden days when CentOS was a rebuild CentOS Linux when it was a rebuild of rail and about a month behind rail apple packages would be built against rail and generally used on both. There would be small periods of time that that month rebuild where maintainers would often have a problem where if a library changed in rail which doesn't happen often but it does happen they would have be faced between the choice of they could rebuild the package against the new library in rail and it would break it was already broken for rail users until they rebuilt it. So they were like okay I I need to rebuild it but it was not broken for CentOsh users or any other rebuild users like Scientific Linux and if they rebuilt it against RA then they would break it for those rebuild users. So there was no way to solve both and maintainers did both things. Some of them would wait until Cynthos or another or their preferred rebuild would catch up. Other maintainers would rebuild it right away so that rail it would work correctly on rail and everyone else it was up to them to get caught up. Uh, and it wasn't really great and it everyone just kind of looked the other way because it was only about a month or so time frame where that would happen and it didn't happen that terribly often. Um, RE is a bit different now than it used to be. Back then, uh, library changes in RE were more rare. Now they happen every minor version. Uh, in real 8, they introduced the concept Red Hat did of rolling application streams. uh the operating system is not rolling but you have things like Golang and Rust that are designated as basically no compatibility guarantees and so every every new minor version there of the operating system there's a new a new major version of those language runtimes and a few libraries too like LLVM that causes uh knock-on effects for things that rebuild against it against rail u build their packages against it uh third party repos in particular particular the operating system itself they can make sure that that minor version is all built together and compliant all at one time and release it. Um but as that was happening in real 8 we noticed that they we had launched cent stream and that was now about six months ahead of rail instead of a month behind rail. So now now instead of it being just a month gap of difference and weirdness now we had this up to six month gap where something would be broken on CentOS until Rail caught up to it and that made it the problem harder to ignore. What I proposed at the time was a kind of a bolted on solution for Apple 8 called Apple Next and the idea was that we would have an additional repo that was built against CentOS instead of RE. So that way maintainers could build against those new libraries earlier then not have to wait for them to land in rail itself. And the idea was that rail could just use Apple and CentOS could use Apple and Apple next together to get the base of Apple and also a couple of package overrides that needed to be overridden in a newer a newer build. Um that was that was the basic idea. It most it mostly worked. It solved the problem. Um but some of the key pro key key drawbacks to that approach was that if a maintainer built a package in Apple next that meant not just building it then but they also had to build it in EP six months later because we didn't have any way because of the bolted on nature we didn't have a way to inherit builds between the two. Um that made it a little bit little bit confusing and difficult to work with. the whole concept of uh an add-on repo on top of an add-on repo confused some maintainers and made it a little more difficult to work with. Um so it was a good first first try uh and it kind of worked. Uh in Apple 9 we we kept the same Apple next model but the key thing that we did different there was we launched Apple 9 about six months before real 9. Um we wanted to give maintainers more time to add their packages. At the time, Apple 8 was lagging really far behind Apple 7 as far as available packages and we were getting a lot of complaints about that. Um, that was right at the right at the time whenever we whenever my VP asked me to start the Apple team inside Red Hat. Uh, because basically there were a lot of customer complaints about the lack of packages in Apple 8. Um, it's no one person's problem there or fault. There was a lot of things that led to that. Um, modularity was introduced in real 8. that was a a complicated thing that made a lot of things harder to get done. Um there was also devil packages that weren't shipped automatically. So we had to work through some of those problems. A lot of and then also the CentOS changes. A lot of contributing factors that made eight kind of a kind of a cursed release in a lot of ways. I've heard people make that joke that eight is a number they should just skip like with Windows 8 and RE 8. >> Just cursed all around. >> Yes. So getting back on track. Um [laughter] we had we started the Apple team. We were like okay what can we do to improve and the first thing we did was we started Apple 9 early. Um and that gave maintainers more time to get their packages added before the rail rail 9 launch. Um after we got that up and running we started looking at how Apple next was working. And we decided this isn't really great. We should probably have do something different here. And that was when I started planning out um not just me uh I started the idea and started working with the other steering committee members about how we could roll it out uh to actually implement minor versions in Apple uh which is the big change that I was leading up to and alluded to for Apple 10. Uh, what we realized in hindsight from Apple Next was that what we we called it like an add-on repo on top of the add-on repo, but really it was a way to target two minor versions of rail at the same time because when when changes would show up in CentOS stream, that was what was planned for the next real minor version. So we would say be targeting 8.4 and 8.5 simultaneously. Um >> once you shift the viewpoint to that, not just targeting two oss, but two minor versions of what really is the same OS, then you realize how the the minor versions we wanted to do in Apple 10 make a lot more sense. >> Just building an Apple 10.0 that at first works on CentOS and then works on Rail 10.0 and then when the time comes, start building an Apple 10.1 that initially works on CentOS and then later works on Rail 10.1. Um and that's where we're at with Apple 10 right now. uh it has been a very big success. It's the most most ambitious thing and most ambitious change that Apple has ever done um introducing that. Interestingly, I have seen older videos from Apple steering committee members presenting at conferences about how they could improve Apple and several times doing minor versions of the repo came up as ideas. Uh at the time there was no Apple team. Nobody had time to dedicate to it fully. So even though people thought, "Yeah, this could be a good idea," nobody had the time and bandwidth to actually go through and implement it. Um, so that's what finally was different was my team got started. By that point, it was me and uh me and three uh two other maintainers, so a team of three. We got to actually spend our day focused on making that a reality. Uh it involved lots of changes across the infrastructure and the build pipeline and the tooling uh to get that working. Um but we were able to get it done and the general feedback has been it has been a huge success. Um maintainers the the Apple next problem of doing two rebuilds for the same library change doesn't exist anymore. You just build it in the minor repo uh in the minor version repo when that library changes and then that gets carried forward whenever it's used in the transition from CentOS to rail it gets carried forward and re used the same way from the same build. Um, so yes, that means that uh rail customers are using Apple packages that were built against CentOS and never built against Rail and it works fine because they're really this different minor versions of the same OS. But I digress on that point. >> So this has made it a lot easier for maintainers to make sure that the right versions of the right packages are in the right repositories. Has this added any sort of uh any sort of burden to the end user? uh much like run uh upgrading from 10.1 to 10.2 on re is is pretty well seamless uh usually just involves some package updates and a reboot uh is is it just as transparent with uh with with Apple. >> So that ties into two parts of Noah's question that I didn't actually finish answering. There's a long it's like a three-part question you gave me. >> That's why there's two of us. Um, [laughter] so yes, uh, for users it, uh, for for your regular user, we strive really hard to keep the instructions the same. You install the upper release package and you're done. Um, of course, you also have to enable the CRB repo first, but those two steps are the same, and you don't really have to think about it too much. Um, we do still see users that don't actually understand that Appleton has minor versions, which I consider a success because it has been a transparent process for them. Um, the caveat to that, some of the downsides, uh, growing pains that Noah asked about, um, where we've started to see some cracks in it is people that mirror the Apple repo for their own usage where they, uh, not public mirrors that everyone's pulling from, but like they create a private mirror inside their inside their own infrastructure. Um the historical pattern has been you sync from the pub apple major version number path and the way we have it set up for Apple 10. That major version path is actually a sim link to the latest minor version and the uh so 10 will be a sim link to right now 10.3 and 10 is actually a sim link to 10 point uh sorry 10z is a sim link to 10.2 the one that rail uses. Uh the the reason we have that Z name that actually came about from one of the things that kind of caught us off guard in the implementation and the rollout was that the original idea we had for Apple 10 was that the uh systems could just request the minor version that they were. So a a CentOS uh 10 system doesn't actually have a minor version. So it would just ask for Apple 10 and it would get Apple 10 that's actually the latest minor version. Uh, and then a rail system that knows that it's 10.0 could ask for 10.0 and get the right repo. Um, and that all made sense to us in all the implementation as we were going through it and we were on track to do that. Um, but then I forget exactly who brought it up first, but it the we realized eventually that that was going to cause problems uh upgrading between minor versions because when you start if you're on rail 10.0 Z and you start your upgrade transaction to go to 10.1, you're still on 10.0 at the start of that transaction, your system doesn't know it's 10.1 until it gets that Red Hat release package installed and updated. So with that, uh, if you have any packages in Apple that were built against 10.1, you're not going to see those until the next transaction. Uh, you could work around that, you know, CIS admin style and just do DNF update Red Hat release first and then DNF update for the rest of the system. But that two-part DNF update, there's no way to encode that into DNF automatic for automatic upgrades. So, we realized we we sort of constructed this impossible upgrade path that wasn't going to work for a lot a large portion of our users that rely on DNF automatic. Um >> based on that that's when we kind of pivoted a little bit and created that uh what we call like the Z structure where uh 10 by itself is just the bare name that uh requests the latest repo but 10z would request the repo that we have configured for for re um and it's the same it's we have sim links on the mirror itself if you're using base urls if you're using the default metal links from the Apple release package uh we have repo redirects in mirror manager the software you use to control that um so we pivoted and redesigned around that, got it working and launched it that way and it does work and it works pretty well um until you're trying to go in and explicitly mirror an exact URL for use on your rail systems for a private mirror. That is the scenario that we we didn't give enough consideration I guess or didn't think fully through how that would work. Uh so people users that are doing that that haven't paid any attention at all to what we were doing uh will go and sync in Apple10 and then they get 10 point like right now they're getting 10.3 packages there and then reporting bugs that these Apple 103 packages don't install on rail 10.2 um which is uh not not the user experience we want to have. Um, now I could just lean on, you know, oh, you didn't go read the docs or you didn't see that conference presentation I gave about this or you didn't [laughter] see that announcement on the mailing list about this >> RDM. >> Yeah. [laughter] >> Yeah. >> I I've been doing I've been doing open source enough now that I have uh I've kind of come to the stage of grief of just accepting that documentation doesn't exist for people to read in advance. It exists for you to point at to say I told you you should have read that ahead of time. [laughter] >> [gasps] >> So I do we do we do try to get the docsuh improved from time to time especially when stuff comes up. Uh but it just feels like a never- ending thing of like oh we didn't think about describing this scenario or describing this scenario. Uh so documentation is always a work in progress. If documentation is your passion please reach out to me because I have ideas and could use some help. Um [laughter] there's never enough people that are passionate about docs. Always a lot of good ideas. Um I have lots of good ideas and I'm terrible at writing. So if I could combine forces with anyone that likes writing documentation, I'd love to. So yeah, the we're trying right now for EP 11 leading up to that is we want to figure out a way to solve that uh solve that use case also and smooth things out. Um the general idea is that we're going to keep it as a transparent process for people that install Apple release. You just install Apple release, you don't have to think about anything. You just install it and go. But for the for the infrastructure side invert the um the additional character thing that we did with the Z. Um so on for 10 what we have set up is that if you're if you have the DNF variable release for minor defined you get a - Z appended on your request and that's how you get the repositories that correspond to rail. Uh the name comes from the fact that inside like rail products they actually call the different minor versions zstreams. Um You might see that if you if you any of the satellite admins out there might have seen like a like an 8.6 point Z type naming in some of the products channels. That's where that comes from Zstream. >> That's not immediately intuitive even if you have run satellite that you need to do the same thing for Apple. Um so we're thinking about how we can if it would it be better to flip that on its head and if you don't have release ver minor defined add additional characters. Um the leading thing we're talking about now is S corresponding to to CentOS stream. Uh but we could also go with C something else some other character distinguish it. That way the default Apple10 experience if you mirror that down is going to be the repo compatible with re and if you ask for or sorry 11 uh and if you if you ask for 11s then you'll get the repo that's one minor version ahead corresponding to CentOS. Um it generally has good general support. Most people think this is a good idea. Uh the more maybe contentious thing is one which variable we key off of and two if we if we're doing it for 11 should we change 10 to match even though 10 is already in production and in use. It's very easy to say yes we'll do it for 11 from the start because it h it doesn't exist yet. We can just start it that way. Sure. >> Um, >> but there's also a very uh very valid perspective that it is painful for maintainers and for tool owners if 10 and 11 are doing something different. Rip the band-aid off. Get it get both 10 and 11 on the better better structured system. Um, we're also somewhat limited by what DNF can do with the variables. I mentioned keen off of the release ver minor uh, sorry. Yeah, release ver minor variable. uh DNF has a way to say if the variables defined insert additional characters. It actually doesn't have a way to do if the variables not defined insert other characters. Um so we're talking about is the right way to do this approach the DNF team and ask for a code change to support that logic or alternatively what we could do is just use the CentOS systems actually have the stream variable defined uh for legacy reasons but it's still there. we could key off that variable being defined to add the extra characters. Um, there are some people that have mixed feelings about that. Uh, one thing that we're stuck on with that right now is using that variable. Some of the rebuilds that are out there, not all of them, actually did define the stream variable. even though they're not based on CentOS uh based on CentOS stream, they define that variable so they could consume content from CentOS special interest groups which are other thirdparty repos that have different rules than Apple. Um so right now if we keyed off that variable, we would have rail rebuilds that want the content that matches rail, but they'd be getting directed to content for CentOS that's a minor version ahead and would cause some confusion and mixups and problems. Um, I think probably the most likely path forward is those rebuilds treat that as a compatibility issue because rail doesn't define that. So, they should undefine it and that'll make everything smoother going forward, but it doesn't help people that already have it. And so, now we're talking about transition path and then also l lends credence to the idea that maybe we leave 10 alone and only do this for 11 going forward. >> Talk to me a little bit about 11. uh you want to keep the minor version design as the foundation but clean up the redirect and assemblink later. What's kind of the core idea um what can be cleaned up in in 11. So that was the the part that I was talking about the uh it's mostly infrastructure changes u maintainers wouldn't from the package maintenance side the workflow would be the same from the the Apple release user side it'll be the same they just install Apple release and go we'll have that configured to do what we want it to um it would almost entirely be a change on the on the infrastructure side both the the Apple infrastructure itself that's part you know shared Fedora infrastructure but also people doing mirroring Apple into their own infrastructure that use case of the private mirror where they're getting they're assuming what the URL is, not reading any any documentation or announcements and getting the wrong content. Um, that's what that change would look like. >> Is it accurate to say that this is not anything set in stone? This is more of an opening set of ideas for discussion and you're looking and welcoming feedback and thoughts from the community. >> Absolutely. And that's why I'm here today. >> Honestly, that that this this this is not one decision but two. the the two decisions are how do we fix this in Apple 11 and the second decision is should we change Apple 10 to match and I mean statistics have have shown that re 10.2 two is now active. It's it's been out for a couple of months now at the time of recording. So, people are just now starting to adopt rail 10. So, I I I think the the question is this sounds like a good idea for Apple 11. Let's put it to a vote. Let's let's decide. And then then the follow-up uh decision is do we make Apple 10 match now before uh before a lot of uh a lot of rail users start using the new and shiny thing as as a as a marketer or as a solutions architect or as a home laber. I I immediately start adopting the newest versions as soon as they're available sometimes in beta. Uh but if if I'm running big enterprise, not so much. Letting letting Rail cook for a year uh is is a really good idea. And with every with six month releases, that puts us right around 10.2. So all all of this this this whole episode was was conceived through the fact that this this may not impact Fedora, but it definitely affects downstream of Fedora and how packages are taken from Fedora and put into the enterprise Linux stream. And so the question, oh hey, pun intended, I guess. Uh but uh >> and technically Apple is part of Fedora. So >> well, there you go. And uh so I actually came across a Fedora discourse uh thread that you started Carl. Um and that will be in the show notes by the way. So check the episode description uh for that link. But uh you you actually put out a request for feedback. You wanted you wanted to get people's thoughts and opinions and try and catch some of those use cases. Uh to to date, what uh what has been kind of the feedback? What has been the sentiment in in that thread? >> Overwhelmingly positive. I honestly thought that there was going to be more dissenters uh and more concerns around it. Um nobody seems to be outright, at least so far, the feedback that's been shared, it's still open for discussion um for a little while. Nobody's been outright opposed to it. Several people have said, "Yes, I I wanted it to be like this all along. Uh why didn't you, you know, have a crystal ball and set it up this way from the start?" >> [laughter] >> Nobody said that literally, but that was the that was the under >> telling travel was kind of implied. >> Yes, [laughter] the benefit of hindsight is wonderful. Um, >> well, if you would really like a dententer, I could jump in that thread and really just complain. [laughter] >> All joking aside, if there's to tie back to what you were saying about, you know, to a degree our documentation is is there. It's maybe not always perfect, but it would be ideal if people read. what should somebody be aware of or what what resources should somebody um navigate and consult before coming to the table and saying I have an opinion. I think I know what should be done. >> Yeah. Taking a read over the initial uh over the documentation we have so far. Um the URL is kind of long and y'all can put it in the show notes for the it's docs.farpro.org and then some stuff. Um, we actually have a cool redirect set up. If you type epel.io, that'll redirect you to the Apple documentation page. Um, and in there, um, aside from this proposal, what I want everyone to take away from this is that, uh, the most common question is, how do I get this? This package is in Apple 9, but not Apple 10. How do I get it there? Uh, we have a how to request Apple packages guide as the very first thing in our table of contents. Um, so I highly recommend people read that. Um, just in general for for the Apple 10 stuff, uh, under our guidelines and policies, we have a branches page. Uh, and that has that's our most complete, it's very incomplete, but it's our most complete so far documentation around uh, how these Apple 10 things work and uh, the different minor versions. Um, it's got a good bit of content in there. It also on the same page it has stuff describing Apple 9 and Apple 8 which are much less interesting because it's less complicate complicated. It also has it touches on Apple 9 next in there and how those branches work. Uh it talks about which which base each of those are built against. And so that would be the the best starting point to understand the basics of this. Um, in that EP 11 thread that y'all are going to link, there's a few other reference links in the in the very first post, things that I felt were relevant. Um, I think I linked to the original Apple 10 proposal and I linked to the the issue where we figured out how we needed to change up and pivot and do the the ZSIM links to get that work the upgrade path working. >> Um, that part that discussion is in there that would be useful to read through. Um, and a lot of it is just, you know, here's all these little weird things that happened along the way and how we got here now. Uh, read those first and you can have the same benefit of hindsight that I wish I had, you know, two years ago. That's awesome. Uh, and and this is a great opportunity to jump in if you're newer to Fedora and see how the process works. The Apple Steering Committee is one of the most wellestablished groups within Fedora uh, as far as making decisions that not only impact Fedora but impact the entire open source community. Could you talk a little bit specifically within Fedora where is a good place to to say hey I want to dip my toes in where's the best place for me to get started es part particularly actually you might be the best guy to answer this if my goal is to wind up on the enterprise side >> right um so because so Fedora is a very broad project there's a lot of ways people can get involved and things I can do uh there's a really cool website called uh what can I do for fedora.org or um it was inspired by another open source project that had something similar where you go there and you kind of can shuffle through different suggestions about ways to get involved. Then it'll link you to like uh different SIG onboarding paths and things like that like I'm interested in Python programming or I'm interested in documentation or I'm interested in RPM packaging. Um that is it's a really useful website. I see Eric's face. I think he just pulled it up and he's going through >> I didve I've never seen this thing. [laughter] Okay, that's going in the show. >> Well, and it literally walks you through is like, do you want to do pixels? No. Okay, the next thing. Do you want to hack the Gibson? I This is great. This is whoever thought of this. Brilliant. It's a million dollar idea right here. >> I believe it was Ralph Bean. And I'm sorry if I I'm not giving credit to someone else that was that came up with it, but I think he he was at least involved in that. And I think he it was his initial idea, but yes, it's some of it's a little dated. About a little while back, I sent a pull request. I mentioned modularity earlier. There was still modularity in there. And I saw it one day when I was showing it to a new user and I was like, "Oh, that's wrong. That's go that's already retired in Fedora and it's not in real 10 at all. So, uh, yeah, we shouldn't be pointing people to modularity anymore." So, I sent a pull request to the to the website to take that down. Um, the whole thing could probably use a little bit of a a content refresh, but it's still broadly applicable. Um, a few other deadends I might mention. Uh some of it will depend on some of the responsiveness you might get out of that will depend on the individual group you get directed to. Uh I have I have seen that before where a new contributor tries to get started they follow the instructions from that website and they end up with a with a special photo special interest group that is less active for not casting any shape but it just happens sometimes with life and jobs and everything like that. Uh, and if they don't get a response right away, they say, "Oh, this this SIG must be dead and I this website's terrible because it pointed me in a to a dead end." And I'll point out like, "Oh, yeah, you know, that person's doing this, and oh, let me reach out to this person directly. They might have missed your ping." And, uh, Fedor is a community at the end of the day. It's a group of people. Um, we're never going to have everything be perfectly smooth, but the solution generally is talking to people and figuring something out. So >> the next time you see >> groups maintain making their keeping their own documentation up to date for how to get involved with that special interest group is the way that that gets improved collectively. I think >> for sure >> you tell Ralph the next time you see him. Noah wants to give you an uncomfortable manhug for that idea. [laughter] >> Awesome. >> He might be a listener. He he uh he still uses Fedora as far as I know. So >> nice. So, I I've I've picked this up from from another podcast that I follow. Uh I stole it and uh you know, I'm I'm going to continue to steal it for for both this show and for the IT guy show, but was was there anything about Apple that we didn't cover today? Uh this was kind of a crash course and kind of a current events look at look at Apple and how it relates to the Fedora community. Was there anything that you feel we should touch on before we wrap up today? >> Sure. Couple of alibis. Uh, one thing that I didn't I didn't get into too much that I'm surprised I didn't was that um by the nature of the pro project um Apple is very packaging focused. Um, a lot of the contributions to Apple are going to be package m RPM package maintenance. That's less true for Fedora. There's still a lot of package maintenance that happens in Fedora, but you can you can walk into Fedora and say, I want to fundamentally change, you know, the design of this or or work on literal design assets for this, things like that. There's a lot more aspects and avenues for contribution to Fedora as a whole because it is a completely community controlled uh project. With Apple being an add-on repo to an enterprise product, there's a lot a lot of things that are out of our control. Like you may approach Apple and say, "I think it's wrong that re does this and this and this." And we're going to say, "Cool. That's a product decision. That's those guys over there." Like some of us may get a paycheck from Red Hat to work on this, but that doesn't mean we have authority over that product over there in a whole different segment of the business. >> So, EPA in a way, it's keep in mind that it is very much an add-on to that product. And so, there are things that are just out of our control that we just have to adjust for. And that is the reality. Um if that's not appealing for you, um something that you know a distribute a distribution that is entirely community controlled uh like Debian might be more appealing where Debian has control over everything in Debian. There's no company that has can make product decisions to make parts of Debbian different things. Uh whereas Apple, we are very much living uh it's you know it's Red Hat show and Apple is just being a useful add-on for that. So that is useful context for how that works. That naturally means most of the contributions for Apple are going to be RPM packaging work to add on additional packages. Apple doesn't really do much else. Fedora does a lot more than just packages. Apple not so much. It's pretty much just packages. >> Yeah. And that's that's kind of the the key differentiator there to to save Carl getting a lot of hate mail was what what what Carl was talking about relates specifically to Apple, those enterprise decisions. Uh whereas Fedora is much broader. uh while while Fedora and uh re sent to us there in in between tend to make decisions that uh you know one one decision impacts the other three usually positively but there are times uh like butterfs where Fedora as a community as a project has said this is something we're going to do and Red Hat says yeah that's awesome keep doing that your community distribution but for our customers we think XFS would be a better file system so I I just wanted to differentiate the fact that while is very very much enterprisedriven. Fedora is is much more uh aligned with with kind of a Debian model where um where we have key stakeholders like Red Hat, but Fedora is broadly a communitydriven project. Um so I wanted to save you a little bit of hate mail there, Carl. [laughter] >> Followup. Uh so the RPM packaging aspect and the control aspect. Yeah. Um, if anyone is specifically interested in the RPM packaging, they might be a good fit to get more involved in Apple. Um, Fedora has great documentation on becoming a an RPM packager. And then from there, once you're a Fedora RPM packager, Eple is just more branches. So, we have what we call discit, which is the package sources, and instead of like a main or master branch, we have the rawhide branch corresponding to Fedora rawhide. And then from there, we create like our F44 and F43 branches. Apple 10's just another branch from that. Apple 9 is just another branch from that. Um, and for the for the minor version stuff we're doing, we have Apple 10 and then from Apple 10, we create Apple 10.2, Apple 10.3 and so on. Uh, so it's all branches within the Fedora Fedora system. Uh, that's why the Fedora is the the best place to start with that. Specifically around the RPM packaging though, I will ask you'all to put a link in the show notes to an RPM packaging guide uh that I I didn't start it, but I've been maintaining it for the last few years. and uh adding lessons to it and things like that. It's a good sometimes I give it as a live workshop at conferences. If you ever see me on the schedule, uh you might see an RPM packaging workshop at one of your uh future conferences you're attending. Uh if you're interested, come on by. >> Which I'm so glad you mentioned the contributor's guide because of all Fedora's documentation, that is one of the best. Uh, which speaking of Noah, I feel like that's a great segue because here in two weeks, our next episode, we're actually going to be talking about how do you actually get involved with Fedora. So, if you're a longtime contributor, a long time listener, right? If if you're a longtime contributor, a longtime listener, uh, definitely grab that that that episode link when it comes out because, uh, we'll be talking about the matrix group. We'll be talking about some of the different documentation, some of the different teams, some of the ways you get involved uh, like the joint sig. So, uh, thank you Carl for that beautiful segue. But, uh, yeah, so in in two weeks, make sure to tune in to this podcast because we will definitely be talking about ways to get involved in the Fedora community, uh, and the Fedora project as a whole. Um, you mentioned conferences. Carl won that that part about getting involved. Um, I cannot stress enough how important that is. If you like Fedora, if you like Apple, if you like any open source project distributions or repos or software or anything, uh there's nothing more important that more impactful than you can do than getting involved. Whether that's spreading the word about it, whether that's reporting bugs, whether that's triaging bugs, whether it's resolving bugs or adding features or redesigning the structure of something, getting involved is so critical for the health of open source. Uh many many projects struggle with how do we keep our keep our longtime contributors and how do we get new contributors. Um generally just with no other changes the longtime contributors that falls down over time and decreases because people people retire people die people get new jobs where they can't spend as much time on open source. Um so without any new blood the longtime contributor base shrinks just naturally. We have to replenish that. And the only way we do that is by getting new people involved. If you think you're not good enough or not smart enough, you're wrong. I don't want to come here and tell people they're wrong, but you are. You are good enough. I had a uh I had a very, very, very smart friend of mine, someone I consider a mentor, tell me recently that they weren't smart enough to do uh Fedora packaging. And I wanted to reach through the camera and strangle them because I was like, you absolutely can. >> Yeah, lovingly. [laughter] I don't want to call him out. I'll tell tell y'all afterwards and y'all are going to be like, "What? He said that?" [laughter] >> That guy. >> Um, do we do we need to get security in here? He's he's threatening the audience here. >> No, he's not. He's He's trying to tell people that it's not as hard as you think it is and that you're smarter than you think you are and that [clears throat] it's a community project. There's a whole bunch of people willing to come alongside you and willing to help you out. Is that an accurate summation of what you're trying to say, George? Carl, >> absolutely. >> Awesome, Carl. Well, thank you so much for agreeing to jump in here and for for educating our audience a little bit about Apple and uh and some of the changes that could be coming to the Apple program in the in the coming months. >> Thanks for giving me the chance to highlight the uh the Apple 11 thread. >> That brings us to the end of episode 57. Make sure to check out the show notes and stay connected with the Fedora community. On behalf of Carl George, our guest, Eric Hendrickx, my co-host, and the entire Fedora community, thanks for joining us. We'll see you next time.