Flock 2026 Two Years In: Accelerating Microsoft Contribution to Fedora

Hey. Hello everybody. Welcome. Um I'm Jeremy Cline. Uh with me are Brian and Ruben. Uh sadly Sandeep couldn't be here today. Um so you'll have to make do with three of us. Um and today we're going to talk about what Microsoft has been up to in the last couple years in the Fedora community. >> I'm going to ask you to tell us what team you're on or how you're here or why. >> Okay. >> Excuse me for my turn off because we're streaming. >> Yeah. >> You want to speak there? >> And I'm on the uh community Linux engineering team at Microsoft. We're all on different teams. So we'll all introduce ourselves. >> I'm Ruben Olinsky. I work on the Azure Linux team at Microsoft. >> We lost our slides. >> Oh. >> Oh, I probably hit spacebar or something. >> I'll get it back. Why don't you introduce yourself? >> Sure. I'm Brian Exelbierd. Um so I officially I'm a product manager, which is weird as a title because my real job is to annoy engineers who contribute on behalf of Microsoft upstream Linux communities including the Fedora community, the Debian community, and Flatcar Linux. Um I've done jobs like that a lot. A lot of you know me from doing that. We have slides again. Okay, cool. >> Why is it weird that product managers annoy engineers? >> Uh it's not weird that we annoy engineers. I don't work on any product. Um but we have a small number of titles and so that's the one I got. So people are always like, "Oh, you do the thing." and I'm like, "No, I don't do the thing. I don't even know how to use the thing." Um so two years ago we created the community Linux engineering team and last year I had the privilege of coming here and presenting a lot of this man's work uh about what we had done in Fedora and you all liked it and you came to the talk so we did it again. Um want to talk a little bit about what we've learned and what we've done in Fedora here in year two, but there's some other cool things that have been going on at Microsoft that we wanted to talk about as well, because they have intersection and impact with Fedora. So, stop me if you've heard me say these exact words before. Um Microsoft is not monolithic. There is no single thought, there is no hive mind. And And the reason I asked my friends here to make a point of saying what they do at Microsoft was to point out like we're on three different parts of different organizations. Although, I think actually in the last reorg we all became one organization. But, we're still figuring that out. Um and so, a key thing is Fedora's going to see a lot of participation from Microsoft that like you'll be like, "Oh, these people showed up and did that thing. I'll go ask Becks about that." And I have no idea. Because it worked. They came in and they participated and they did their thing. And so, people are going to show up from different paths with different thoughts and ideas. So, take this presentation within that context. I don't know why we keep losing our slides. That said, take this presentation in that context. >> You're running Windows. >> Probably. It happens. Um take this presentation in that context. We're not trying to teach you our org chart, but we're trying to help you understand there's a lot of things going on in the background. Jeremy. >> This is me. Um so, one of the things that's happened in the last year, um so 2 years ago we started uploading Fedora Cloud variant images to Azure. Um and in the last year or so, we've started uh automated testing of those images. Um yes, I can see Adam cheering. Um however, those tests aren't really reported anywhere externally yet or they're not wired into the Wiki workflow that Fedora uses. So, over the summer I do plan to finish that work. So, 45 should have automated results for the Azure images in the Wiki, so we can look at those during the uh the blocker meetings and final review. So, that should hopefully be exciting. Um and this is this is great not just for Azure, um but for everyone because a lot of the issues we do hit aren't Azure specific. They're not about Hyper-V. Um and it's a a nice um not second or third, but additional way to test uh Fedora outputs. Um as an example, there was a um a G lib C bug that Kevin Fenzi reported and he thought maybe it was his um interesting AArch64 laptop, but it turned out that it wasn't and um I was able to go back in our test history and spot the day that it broke and I had a very small diff of packages. So, that made it much easier to go find the root cause. Um so, that's pretty great and I'm excited about getting that tied into more official results. Um so, that's what we've been up to in the cloud space. I think it's back to you. >> We we didn't plan for a podium. Um so, Jeremy [clears throat] talked yesterday about signing infrastructure and a lot of the work that he's done there uh and I'm not going to repeat it all for you. Please go watch that recording. It was absolutely amazing. Uh what I did want to kind of talk about was some other work that the CLA, our Community Linux Engineering Team, does at Microsoft that aligns very closely with Fedora. A big goal around here is to increase the number of contributors. And part of the work of our Community Linux Engineering Team is to serve as an onboarding and ambassador style group inside of Microsoft because we don't want to proxy all of Microsoft's work through that team. We want the upstream in our side, upstream engineering teams, to be coming directly to Fedora to work as the upstream for the things that we're going to do. We want all of those folks showing up. We don't want them taking shortcuts. We want them doing it the right way, the Fedora way. And the best way is for us to help guide them to the right people in the community to talk to, the right documentation, the right onboarding, and those pieces. I'm going to kind of change tones here. Our colleague Sandeep couldn't be here, so I'm going to try and cover some of his material and try and do it justice. This is some information about Fedora WSL. We began producing official images for Fedora on WSL, Windows Subsystem for Linux, with Fedora 42. This is a graph of active users over that time. You'll notice that it peaks close to about 30,000 monthly active users, which is fantastic. That shows us that people like this, they're using it. It's clearly being used. We're using it some internally. Like it's it's a good thing. You'll notice this drop. That's cuz I don't have Fedora 43 data. We really do believe, and we're waiting on it, that this represents as well a healthy transition from the release of 43 of people actually upgrading their WSL instances. And that's what we want to see. It's what as a community with a 13-month release cycle we want to see our users doing. And so we're really glad that that's there. My colleague Sandeep specifically works on something called the Azure Community Gallery, and we launched that in 2024, and it's our answer to how does a project put an image on Azure when they're not like trying to sell it in the Azure Marketplace. And so his team launched that in '24, and Fedora was a launch partner with it, and they are super excited about it. This is a chart of usage of Fedora on Azure with some huge asterisks that I want to talk about. This is only direct launches of the Fedora published market or Azure Gallery image. So, if someone brought in their own copy, if a third-party publisher published it, they converted some other system, you're not going to see that in this chart. This is literally people who went and clicked on the Fedora published image. The other thing to keep in mind is that this is the image they clicked on. So, if those Fedora 40 folks have upgraded to 43, we don't know that. But, I really like this chart, and I hope I'm not going too off script, Sandeep, when you review the recording. I like this chart because what it tells me is people have used the Fedora images and are continuing to use the Fedora images based on that 40, but with 43 being second, they're also continuing to launch new ones. We can debate the number in terms of the raw units. Like, what does it mean? Why would you use Fedora in public cloud? Like, those are all great questions, but this shows that the community's output is valued. Couple other things I actually looking at my notes from Sandeep here. Uh couple things to talk about as well are that uh we're continuing to use what we've learned from working with Fedora in community gallery to improve the community gallery experience. So, we're working on some new ways to help publishers like Fedora optimize the way that they're publishing and to communicate better to users who are using the images, and for those users to be able to better tell things like when an image has gone end-of-life or is going to be deprecated so that they can manage their life cycles better as well. Ruben. >> All right. Thanks. So, Max hit a a number of the the high points here, but you know, why why does Fedora matter to Microsoft across our different teams. You know, as we definitely have heard in the the sessions here, right? Fedora's foundational to enterprise Linux. That's definitely true for for Microsoft as well. Uh I'll be talking in the next few slides about Azure Linux, which uh we just sort of formally more fully launched about a month ago at uh Open Source Summit in in North America. Uh and it is based uh very formally on on Fedora as it's upstream. Definitely, Fedora gives us upstream signal, you know, via Rawhide and future releases as to changes that will be coming uh to our future versions and changes that will be working through how to absorb, integrate, uh and make work for our customers. We, you know, would like to believe that there's some some things we've learned from from our work and what we do at Microsoft and with the Azure cloud that we can bring back, we can contribute in in appropriate ways that, you know, we hope to to benefit the broader community. Um and something else Bex hit on a little bit is, you know, what's what's our intended working model. Um I can speak also for specifically for for Azure Linux. You know, we we know we we need to come here uh to meet Fedora and the upstream. That's why I'm here. That's why why we're here. Um and, you know, we know we need to show up with with the contributions and and uh find ways that uh the things we're doing we can we can benefit the broader community. So, I'll give a couple slides. This will be just a sort of very high-level what is Azure Linux. Um it was mentioned a couple times I think yesterday morning. Um it's what I primarily work on and have been working on for quite a while uh even though it was only just more formally announced. So, it's a Microsoft maintained Linux distro. It's built for the Azure cloud primarily. Uh that means for hosting VMs, for uh Kubernetes clusters, for containers. And I'll also uh we will be shortly publishing an Azure Linux specific Windows Subsystem for Linux WSL image um that I use mostly as my daily driver right now for for things when I have to be on on my Windows laptop. Uh we are formally for 4.0 downstream of Fedora 43. Uh it is rebuilt from the specs. So, we're we're, you know, we're taking sources and and input um and and building the the binaries and packages uh for our distro. What are the main differences? That's definitely a question get asked a bunch. It's intended to be configured, tuned, optimized for our customers, for the Azure cloud, uh for our use cases. It's now in public preview, so it's available. It's there. Um if you're interested in looking at it, the sources for it are all on or at least whatever sources we have uh on top of of Fedora are are present on GitHub. I'm not expecting that you go look there, but if you're interested, uh it is available and and they're in the open. Something else I'd like to also introduce, we sort of the other coin, the other half of of of some of what we do, um uh was announced also about a month ago, Azure Container Linux. So, that's, you know, if you think of Azure Linux uh 4.0 as our more traditional RPM-based based distro, Azure Container Linux is our more image mode. It's the hardened, immutable, secured container host OS. Um it's got an interesting derivation. Um it actually takes its design, its image definition, its SDK from Flatcar. But, um once we intersect it with our 4.0 releases, um it's going to take its packages, it's our uh uh as inputs to that image build process from Azure Linux downstream from Fedora, and kind of putting the those two together. Uh the 3.0 version of Azure Container Linux is generally available. Um also, the sources for it all all on GitHub uh and available in in the Azure cloud for for those who might be interested. We'll get into a just a little bit of how do we build it? What's what's sort of inside and how do we interface with kind of infra and and sources? Uh One of the goals we set out with uh for with with Azure Linux 4.0 is we want to really highlight where we're deviating from upstream. We want to highlight it, and we want to be declarative about it. For ourselves, and also for for, you know, auditability and inspectability. We want to know uh you know, we we considered do we do a hard fork? You know, what do we do? We We've had various experiences in the past cuz we have had previous versions of our distro more for internal use within Microsoft. Uh and we realized of course with you know, risk of a hard fork is you start making changes, it's hard to know where the changes come from, how do you rebase? Once you also have a sources you can very easily edit, you start making changes that maybe you didn't need to make, but they were really easy to do so. We really trying to trying to avoid that through construction. We have a mono repo on GitHub. It's got a set of declarative we use TOML-based uh metadata. It includes a sort of a selection of what's the subset of Fedora packages that we include and we build for our distro. Uh and specifically what are the very specific upstream version of those upstream sources and specs that we're taking from. Broadly, we do have a configuration that says take from Fedora 43 as of a certain snapshot, but we also have the capability of saying no for this package, we actually want this disk get commit for this package and and pull that forward so we can we can pull individual targeted changes. We have specs that we've added. Some very obvious ones Azure Linux release, Azure Linux icons, things like that where it's about branding. Also additional packages that we would like to be in the process of of upstreaming and trying to get sponsored for introducing it to Fedora, but maybe aren't there yet. We apply per package build overrides where thankfully there's actually a bunch of useful if def macros already in the specs to be able to do feature disablement or enablement uh or to be able to let's say override vendor macros, things like that. Um concrete example here, you know, for for DNF5 uh the Red Hat subscription manager plugin to uh to it is not relevant for our product. So, that's something we can easily just specify a without flag and the right thing will get sent to RPM build. The other thing that's a little bit more custom that we've done is identifying places where there might not already be those easy toggles and we do need to make some targeted adjustments to the specs. We wanted to make these as declarative as possible. So instead of going with kind of patches, we ended up going with kind of taking inspiration of more semantic patches where we can say, "Hey, we want to remove this build require to remove a dependency or we'd like to replace this argument to CMAKE_BUILD uh so that we can disable a feature." You know, here again an example is setting, you know, RPM vendor macro in rpm.spec. Now, even as I say that I realize probably that might be something that could be factored out also as an as a a spec macro in the future. Maybe that's that's an upstream change that can that we could try to contribute. What else is in the repo? Um we do take that metadata as well as the selected upstream specs and we have a an open source tool. It's elsewhere on our GitHub uh called AZL dev that applies them together and produces the final set of of spec and and disk it sources that are going to be sent to standard build tools. So we can send to RPM build, to mock, to koji uh to be able to produce. And those are sort of a cache of those that are checked in for for transparency and to make sure we've got exactly the same version of those across builds. We've got our base image definitions that are all there across WSL containers, uh images. We have a a live ISO for installer. Uh those were those were a bit more forked. We're heavily inspired by Fedora Kiwi descriptions which we went and we we took as a starting point and then we customized those for our for our product, for our use cases. Um but uh are using using Kiwi and trying to at least stay using the same tech stack. There's additional test metadata. Um some of this may include referencing upstream tests that we're on boarding and and trying to run within our infrastructure. They also include uh product or or uh image-specific tests that we're bringing online ourselves. Uh as well as disposition metadata. That's not really a great name for it. Didn't know how else to summarize it is we want to basically have accounting for all of the changes that we're making. I mentioned these overlays, these customizations to the specs. Want to have tagging for these of which of these we are really deeming as non-upstreamable branding changes or configuration options. Which of these are things that are, you know, I sort of see as debt as uh for for our project. These are changes we'd like to to bring upstream and see if there's openness to uh to accepting. And I want to have some way to account and see how we're doing with that. I'll go really quick since since we're getting a little bit uh shorter on time. Um so our infra, we've stood up in a deployment of Koji running in the Azure Kubernetes service. So uh we're able to benefit from a bunch of community stuff we'd seen uh containerizing uh all the different services. We've got that running and integrated with uh you know, we use uh NFS-mounted Azure-hosted blob storage and an Azure-hosted Postgres. And we use all the standard Kubernetes stuff, you know, cert-manager for being able to uh manage a CA and get the right certs we need for for the different services of of Koji to interact. Uh we do have a distro-specific build and release service. That's really that's the piece that's more integrating with our own sort of corporate uh kind of back-end stuff and our our our more business kind of business-driven requirements. Um it's connected to our GitHub workflows and PR workflows and so forth. Orchestrates Koji builds, manages uh we have a lookaside cache that we maintain uh ourselves um that, you know, has a fair bit of content mirrored from upstream Fedora as well as, you know, things we're adding or or or changing versions on. And of course, publishing to to all the the locations where we have stuff. And that's the super short version of Azure Linux. If people are interested, I'll certainly happy to talk in the hallway or online later um where there's interest and I hope you'll be seeing more of me and some of my colleagues uh Matrix and all all the usual locations. >> Yeah, this Sorry, we're all recording. Um this is literally a summary slide. We'd rather have your questions. >> So, thank you. >> [applause] >> Yeah, it's one. Um so, you were mentioning the life cycle for the Azure Linux and I have a question about the versioning. So, now you are running Azure Linux 4, which is based on Fedora 43. Um how long are you going to keep this? And if you're going to exceed the life of Fedora 43, are you going to be backporting patches or just straight away move to Fedora 44 or Fedora 45? And then how this going to reflect in the versioning of Azure image itself? Like Azure 5, Azure 6, or >> Yeah, that's a great question. Um I don't have all the I'm not the right person to have all all the answers on some of that, but what I can say is it will uh outlast, you know, uh live past the the the supported life cycle of of uh Fedora 43. Um we do anticipate, you know, we have a mechanism in place where we can pull updates as, you know, Fedora 43 evolves or we can move certain certain packages or sets of packages forward uh to later versions. Obviously, you know, when we do that, we start deviating uh from kind of coherent set of things that had been been tested upstream. Uh so, it's it's a place where we have options, but we will be doing backporting and obviously keeping things, you know, in in the current season we're in CVE patched and um and appropriate for our customers. There will be down the road, you know, a 5.0, a 6.0, a 7.0, um you know, sort of traditional enterprise Linux style. And uh you You at those times we'll be finding the right places to to intersect future Fedora releases. Part of it is we're we're learning. We're in preview right now. So, we want to get feedback from our customers around, you know, what they're looking for. Obviously, lots of trade-offs between freshness and and stability. I think we as we heard about in some previous talks. So, trying to find the right the right fit there, but uh hopefully definitely look to to share more as we as we solidify that. Thank you. >> Uh you spoke about people upgrading their WSL images from 42 or 43 and so on. So, do you have any kind of prompt in that WSL environment, the CLI environment, that encourages people to upgrade or do they just like uh figure it out on their own or just >> Yeah, I can probably answer that, which is I believe no, but that is a really good idea. And uh we could do that. >> [clears throat] >> So, >> Well, I mean, I'm I'm maintaining a project against I'm maintaining this project called toolbox, which has like this CLI environment, and uh yeah, I have a feature request for this for a long time, and I was just doing a market research. >> Okay. Well, we don't do it, but I like the idea. So, maybe we can talk and figure that out and share it. >> Okay. Okay. >> Cool. >> You have 2 more minutes. Any questions? Okay. Thank you. >> Thank you all very much. >> [applause] >> Thank you. Thank you. >> [cheering] >> And now we'll never speak again. No, I'm just kidding.