Flock 2026 What Is A Hummingbird And Where Is Its Fedora?

When I hit the my hum my poor hummingbird needs space. >> Hummingbird takes a lot of space. No, it doesn't. No, thank you, Shawn. Ah, it's fine. Leave these ones. >> [snorts] >> All right. Okay, cool. It's awesome to be here at Flock. I'm so happy that Flock is full, overflowing with people, contributors. I mean, this is this is great. I was here last year, and um it's super cool to have it here again. I like this format in the big room. I like seeing everyone. It's just It's a wonderful conference. And today, as you may have guessed, we are here to talk about Hummingbird. My name is Stef Walter. I've been at Red Hat for quite a while, and I lead Linux engineering. Um I'm senior director at Red Hat. And so, over time I've worked with so many different parts of of our our Linux stack and the teams that contribute to it. So, you you may have heard about Hummingbird, and I'm going to go through the why, some of the the what we deliver, and so on. Yesterday, there was a workshop led by Valentin, the mastermind of Hummingbird, where we went into some of the how can we work more closely with Fedora. And so, unfortunately, you have to go back in time to talk about that. But, there will be plenty of opportunities to go into details. There's a lot in the talk today. And um so, I'm going to breeze through things. First of all, why are we doing this? I love following um Daniel Stenberg's um comments about open source, about the the curl project, one of the most used projects, open source projects, in the world. And he has been tracking very consistently this this change in the pace of vulnerabilities being reported to the project. One of these one of these notes that he had recently was that the AI slop is gone and we're getting an ever-increasing amount of really good security reports, almost all done with the help of AI. They're submitted at a never-before-seen frequency and put us under serious load. He's since posted other things about this load, very interesting comments following the state of affairs. This is from the Firefox project where they saw the number of security bug fixes that landed in Firefox um over time. The team fixed more security bugs in April than in the past 15 months combined. In the organization that I lead at Red Hat, we've seen a 6x increase compared to last year in the last 3 months of the in the last 3 months. So, the last 3 months since March, April, May, when you compare them to March, April, May last year or any 3 months last year, there's a 6x increase. I know not everyone is experiencing this, but broadly different teams have different amounts of experience of this tsunami. The And And we know that users are demanding something different from their distribution. How do we know that? Because new distributions are popping up people are paying a lot of money for in order to get um solve the problem of the CVE volume coming that's overwhelming them, the compliance pressure of dealing with all of these CVEs, and the fact that dealing with all of this slows down their innovation, their work on the things they actually care about. So, users are saying, "Solve this for me. I want to pay someone else." And tons of different offerings um from different companies are popping up. This is one of the ways that we know this is real. So, I also want to compare to Red Hat Enterprise Linux, one of the the biggest downstream of of uh of Fedora, has prided itself on stability, API, ABI stability, on long life cycles, on things on on judicious and very careful backporting of things so that we do not break existing workloads, on of course having a good security story, on compliance with common criteria. The lack of change in RHEL is very highly valued. It's one of the big values. On the other hand, the project I'm here to talk about turns that on its head. What gives? These users value compliance, near zero vulnerabilities being reported by their scanner, extreme minimization of the attack surface, upstream proximity, staying as close as they can to the latest in order to get all the fixes. Um, and of course compliance with things like FedRamp and FIPS. And there are a large variety of softwares available. RHEL tries to have the parts that you need, the parts we can keep stable for a decade plus. Um, whereas here, users are expecting a large variety of software. So, Hummingbird delivers a broad collection of container images that These are the success metrics. Have as few or smallest number of dependencies in them. That's the number of components and packages in a given container image. The fewest number of CVEs, ideally and often zero, including lows. Small size. Freshness, as close in days to an upstream release as possible. And the least amount of interface differences from the existing well-known container of the same type, so you can swap this in. That's like the ports, the environment variables, and so on. All of those kind of spots. And of course, broadly, the total number of images should be high, total number of deployments should be high, and the total amount of automated updates should be high. There's a theme here. And the theme we call it is, if you have to explain it, you've already lost. Why? You can explain why the 100 low vulnerabilities don't actually affect, or not actually even built into your component. You can explain that probably well for your component. You can explain it to everyone in this room, then they have to go and explain it to the users somehow, who have to explain it to uh the the the person who's holding them up for compliance for, "Okay, you need to you need to meet these goals there." Who have to explain it to their chief uh information security officer, or maybe a partner, or maybe a a a vendor that's using Fedora has to go and explain it to their users. And you can see how it doesn't matter. Once you start explaining, you've already lost. And so the for for uh for for Hummingbird, the goal is no explaining. Just zero. Just small. Just reduce attack surface. As little explaining as possible. So, we take through Fedora, we take a whole bunch of of uh a bunch of projects, bunch of components, turn them into containers. We monitor the upstreams using the the same mechanisms that that you use, that Fedora has. Um we of course correlate it with security events. We scan through what's what's there. We build these things in a hardened way. Um and we have a bunch of automation. I'll talk about that more in a second. That builds packages and images and pushes them out. It all gets built with Conflux. Builds, tests, scans, produces S-bombs, and continuously publishes this out um for people to use. Where can you see this? You can go to images.redhat.com. I should have put a uh uh a URL or a QR code here, but images.redhat.com and you can see what's there today. Um you can look at it. Um you can see which ones are at zero CVEs, for example. You can go and look at the vulnerabilities. You can You can kind of go and touch on all the things that I'm talking about in this talk if you want to go and look at it. Red Hat Hardened Images is the Red Hat product name for um Hummingbird. There's more to Hummingbird than just what's on here, more different kinds of images built in different ways against different bases. Um but this gives you a good taste if you just want to go and see what's there. Um so what do we have? We have 50 Today we have 51 different big, you know, big marquee name images, Node.js or Golang or Python or so on. Um and those have different versions of course, different supported, you know, major versions of uh or the minor versions of Python, for example, minor versions of of Node.js and so on. Plus there's different variants of these images. Um the first is the default one, where we minimize as much as possible, where they are distroless. That means there's no package manager. In In many cases, there's no shell. There's as few dependencies in these images as possible. Um they're purpose-built for the use case. If you get the Nginx uh image, it's meant for you to just go and deploy it. In addition, for all of them, there's a builder image that allows you to layer stuff on top of it. So this has a package manager, so you can pull additional content. This has a shell. This has other things like that. And there's a pattern where you can layer stuff on it and create your own distroless minimal image. Um distroless is not a not a term that we invented. It is a term that other people are using. And uh it's a bit confusing because we pull content from the distro, of course, but the deployment doesn't have the characteristics of the distro. You move as remove as much as possible from it. So, you're only actually deploying the things that you need and not a full distribution or not the ability to continue to maintain it like a distribution. And me- most of them have a FIPS variants as well, where we allow compliance with FedRAMP and the and the FIPS um uh criteria the compliance criteria, and uh you can you can of course use these in in situations where that's required. So, in total then, we end up with I checked this morning, 868 different tags, once you multiply all of that up together. And uh and this content set should grow, as I said earlier, it's important for people to have as much content in this way um rather than as as small amount, and it should continue to grow from there. So, how are these images different than your container images you may have seen? Minimalism, we already talked about that. They're hardened by default, so of course they're compiled with uh um modern security flags, but they're also scanned with things like SCAP. Um there's many other steps that we take to to and we'll continue to add more things to make sure that they are that when there's a security practice, we'll bring it into Hummingbird. There's, like I said, no DNF or RPM command. Um they're verifiable, so they have an S-bomb. All of them have an S-bomb, which is exactly what content is delivered in there. And we prioritize freshness over API stability. And um yeah, we didn't of course we're here talking about it. We didn't invent all of this. Where does it come from? It comes from the spec files of Fedora broadly. And then we're talking about a high percentage over 90% is the same spec files that are in Rawhide. Um it's built against a Fedora build route. And then of course we test it against the available tests that are available for such images. Now of course not all the tests, we don't test that these work on a full system, a virtual machine, or something like that. But we test that these work as containers. And in addition, there's a CVE feed for all of these for the for the for the images in general. So security scanners can report information about them. Um there's a link there. I'll share the slides later where you can see the the the CVE vex feed. And I want to call out one of the changes that uh Fabio has uh has proposed in Fedora for adding PRL data to all of the RPMs. This is a really uh important step for Fedora to be able to understand where what is involved in each RPM, where all the pieces came from, and which upstream projects they relate to. And right now we have to kind of cobble this together in an odd way. Um this will make a big difference in making sure it's authoritative. We know exactly the content that has gone into the RPMs and therefore into the S-bombs and therefore into the CVE feed. So, we have we're we we went uh we we launched this um in May when we had previews and none of it was done in in secret or in private. People have been able to see and uh watch our progress the whole time. But we launched this in in uh May for for customers to actually use of Red Hat. And uh and the goals this gives you an idea of the kind of pace that we want to have. Is generally there's a lot of things around a 24-hour lead time. So, a a CVE fix that's available upstream should be delivered in Hummingbird in less than 24 hours. That's the goal that we have. And so, a lot of it follows from that. Okay, what would we need to do to to accomplish that? All CVEs that show up in Hummingbird's feed should roughly be within 24 hours of another source's publication. So, we don't want to blog and say, "Oh, we don't know about this." And therefore, you know, perhaps make the first metric look better. But we want to be honest with our reporting. And of course, making sure that we have accuracy, making sure that all of the specially high and critical level CVEs found by other sources show up in in Hummingbird's feeds as well. And to to accomplish that, you can imagine this is really hard. And so, there's a high level of automation. Um the project started after use of AI tooling in in your workflow as a developer was commonplace. And so, a lot of the development is done with the help of Claude or with Cursor or with other tools. Um we have a principle of responsible AI where all agentic code is human reviewed. However, the code that then is perhaps written by a human or generated by AI may run autonomously. So, we push out sometimes up to a thousand I've seen I haven't seen us go over this too many times, but about a thousand autonomous merge requests per week. Those are Those are merge requests, let's say an update to a new version of a of a package, something we pulled, let's say, from Fedora or something like that, they show up as merge requests in in Hummingbird, without a human looking at it, and if passes all the tests, passes all the scans, passes all the builds, it'll go out to production. And that is one of the ways that Hummingbird meets that 24-hour goal. So, we, of course, review, as as we've heard many people recommend here, the the agentic generated code. Once it's reviewed, that code can run repeatedly and do this work. So, how do how does source content flow? Um as I mentioned, it comes from Fedora. So, Hummingbird pulls spec files from Fedora and pulls things from the lookaside cache into its own lookaside cache, so it's there ready to build. We use a mono repo where this stuff is imported into. We have an RPMs mono repo and a containers mono repo. And that's the first place we'll look is Rawhide, pulling content in from Rawhide. We'll track for each update, each vulnerability, we have trackers that show where a fix is available, is it available in Rawhide, is it available upstream, is it not available, and so on. And of course, we'll pull from Fedora, we'll build that, and we'll deliver it within those 24 hours. There's a There's also places where we have to diverge. Um perhaps Fedora hasn't updated or I mean hasn't updated their package regularly enough. Um or maybe maybe they're on they're on vacation, fair enough. That happens, right? So, we'll pull uh an update, update the spec file using some of the libraries that that Packet has implemented, update the spec file to pull a um uh version from upstream, and then deliver put way. And often times, it's our goal to automatically create pull requests in Fedora um in such a way that it's not annoying, but essentially say, "Here, here you go. This seems to work." Um and and propose it to the maintainer as an as an update. Um right now that's happening manually. We have a big tracker of them, and some of them have have been have been uh proposed back to Fedora. But uh but hopefully we'll get to the point where that's automated. Um and yeah, we could use Packet to do that as an example. And then, of course, as Fedora updates, we will we we just switch back to that and keep the keep the diff as small as possible. There's certain things that we also pull from from RHEL. For example, the FIPS module, which has been has been certified by NIST. We'll pull that we pull that from RHEL. And then there's the the build root that we build against is a uh stable Fedora release. And I think we pre- we currently do Fedora 43, but we're going to switch to Fedora 44. Um so, that gives you an idea of how this builds off of Fedora and how it's how it's somewhat downstream of Fedora, somewhat next to it. There's give and take there, and we really want these arrows to flow in multiple directions, and not just in one direction. Recently, um the Fedora Council approved use of the Fedora brand for Hummingbird. And so, you're going to hear more about Fedora Hummingbird Linux. And what is that? Right now, we've talked about just the containers um for the applications, but many people want to see this for the whole operating system, for the kernel. The kernel CNA issues many, many, many different uh um CVE IDs, um and outpaces tons of other projects. Obviously, it's a big project, but they have a very aggressive approach there. And so, many people are worried, "Okay, I see a thousand low vulnerabilities in my kernel. What can I do about that?" Well, it turns out the only way thing you can do is use an updated kernel. So, working with with Justin and the the not you Justin, the other Justin. Um, where's Justin? There you are. Um, and for a long time the automation that that Justin and and others have built automatically updates to various Linux kernel, either the stable branch or Linus's tree kernel versions. That's something that is very valuable work and one of the things we'd like to also make part of this ZRCV story. So, you can actually go and try that out. This is a demo. This is a proof of concept that was that was put together. You can go and try it out at that URL. It's a boot C build of the operating system in this way. The idea is to update it as regularly as possible and actually also make this available to people to use depending on when they really want that compliance, when they want that low amount of CVEs. So, and coming back to one of the things that that Jeff mentioned earlier, I think this is really key is Hummingbird is not here to force and say, "Here here's a here's absolutely the better way to do things." There's many things we haven't even completed trying out yet. I mentioned the amount of content that we have, 51 different big versions. We need a lot more than that. There's a lot of things that need to be proven out. Um, [snorts] so I'd like to see and I think we're proposing as this goes forward that Hummingbird goes into the Fedora Innovation Lifecycle Sandbox. And we have that room to experiment and try a different approach. There's a different approach with regards to automation. There's a different approach with regards to a mono repo. There's a different approach with regards to, "Okay, what content comes via a maintainer? When do we need to pull something directly from upstream? There's there's all sorts of different ideas here and we need to actually see this be successful before saying, "Okay, everyone's workflow needs to change." Right? And that is why this is this is a really cool idea with the sandbox. And I've been I've been open with a lot of the goals that Red Hat has for Hummingbird as well. Why? Because I think it's important that as we look if you look at the sandbox proposal, there's various criteria for going from stage to stage. It's very important that we be genuine, that I be genuine, that others be genuine about, "Hey, this is this is what it means to have Hummingbird be successful and adopted." Some of those are business goals, some of those are user goals, some of those are technology goals, right? And we even if you're not you're not supporting one of them, you understand them and you know them and we're open about them. And so to go to the curation stage, which is where we then figure out, "Okay, how do what good parts do we bring into Fedora or what Fedora policies apply to Hummingbird and vice versa?" That would happen after we are successful. Or or not, right? Or actually do this experiment of do these things work. So, in I hope I've given you a a tour of the why of Hummingbird, how it started. Um there will be many more opportunities for discussion. Many places where you can participate in some aspect or perhaps there's parts of work that you're doing in Fedora that really benefits and vice versa the project and um I'm sorry I wasn't able to go too deeply into all the technical details. There was a workshop yesterday, but I hope I've whet your appetite for the space, the reasons for it and uh and give you the understanding for participating um in the future. Do you want to do applause first or questions first? >> Applause. >> [applause] >> So, we have about 5 minutes for questions. >> Okay, I'll try to be fast. Um so, I'm curious specifically about Fedora Hummingbird Linux. You used the kernel as an example, but the kernel in your hide grabs every day, Justin. So, is Hummingbird Linux ahead of Rawhide in other packages? Is it ever behind Rawhide? How does that relationship work exactly? >> So, I think I understood. The question is are we ahead of Rawhide in some cases? >> Are you behind in some cases? Is it >> Right. Um so, we'll be We are We are ahead in some cases. Like I said, there's a tracker in in Red Hat's Jira for all of those cases. And I could pull it up and we can see exactly what they are. And why why would we be behind Rawhide in some cases? Well, if there hasn't been a vulnerability or a reason to update, we don't always update everything all the time. So, um but but you know, if there's a reason to update, it's very easy for the automation to then pick it up and kick off the new build and and go for it. Does that make sense? Give you an idea of what we >> Sure, yeah. So, it's it's triggered specifically by CVEs automatically. And if you wanted to update for another reason, you might have to feed that in manually or >> Right. And typically, there's enough CVEs that boy, oh boy, like I said, we have These last week was 860, but there's been weeks with up 1,000 updates on if you think about it, 51 images. That's a lot of updates going, right? >> Cool, thanks. >> So, you triggering off the CVEs is is fine for a lot of packages, I'm sure, but the way that the kernel CNA works is they don't issue CVEs by default until it's already in a branch kernel. So, basically, every kernel update is a security update, but you're not going to get the CVEs on that for 1 to 2 weeks after. Right? Um so if you're following Greg's tree, I would recommend, you know, which is what the Fedora stable kernels do. I would recommend pulling every single one because they're all going to have CVEs. We just don't know what they are yet. Right. >> That's a good recommendation. And yeah, definitely we'll we'll think about that. It's common for and has been somewhat annoying for our product security folks, but now we've come to grips with it that often times we will have a fix delivered for something before the the CVE even comes in and we have to kind of identify in reverse. So, um but yeah, for the kernel we can we can probably consider consuming every single one. Um we'll see. >> Uh so, Hummingbird can be summarized uh build pipeline that delivers new uh outputs. And the outputs are great, but the pipeline is also interesting by itself. And I think that um all or maybe most of the components of the pipeline separately would be interesting for uh Fedora packagers >> Yeah. >> already. And independently of Hummingbird graduating out of the sandbox, is it possible to integrate some of those tools like the scanning into the existing infrastructure so that that we can take advantage of it? >> Yes, absolutely. So, and that was the workshop that Boutin uh led yesterday. Obviously couldn't touch on every possible angle of that, but that was the goal of it and we talked about it for um in the morning. Um so, there's multiple ways. One is of course all of this is open source. So, we and we can work with people to take a piece out and generalize it if perhaps it's in a make sure it's in a place that more people can contribute to and reuse. That's one step. The other step is um we would understand how to bring participation in so that perhaps for other content we can do uh the build for it in Hummingbird. We need to understand how to do that. It's a mono repo, but other projects have been able to work with a mono repo in that way with participation and ownership. So, let's figure that out. We don't know yet, but we're willing to try. Then the third way is we build all of these images not only for um the the Hummingbird builds, the Hummingbird RPMs, but we also build them. Actually, this answers your question partly, Adam. Against rawhide packages, just straight up. Like the packages are already built and we'll build a container against rawhide. So, there's a third way of participating, which is participating in that. It checking, we can figure out is that a good way to just have the Fedora images be built. So, there's all of those all of those ways of doing things. The pipeline itself is built in Conflux and we've done a lot of work to get Conflux stable and working. Um we sometimes we feel like we solved a bunch of the issues that other people had, which is good because maybe then others who come after have less uh less parts of that to solve. Um but the scale at which we build was is the big win there. So, there's another win there that for some of these things we have to those thousand merge requests per week will generate 20,000 Conflux pipelines a week. And the ability to scale into the cloud using Tekton and Kubernetes and all of that allows us to achieve that. So, there's there's another win there that we could that we could benefit from as a as Fedora. So, yeah, lots of lots of areas like that. >> Do Do you have time for one more? Uh I don't know if you have time to answer this, but and you probably considered it, but there's a lot of supply chain attacks these days where something gets compromised and you know, a a compromised version of a package gets out. Um trying to stick Hummingbird to like the very latest, is there a problem or a concern there with it you're going to pull in some kind of compromised thing that only gets noticed, you know, like a day or two later? >> So, that's a really good question. If we go so fast, what about the wait time? What about the sit time? And yes, I agree. That is a challenge. Actually, so I want to appeal to all of you. I am actually interested in working with someone, perhaps hiring them, to attack the Hummingbird pipeline. To do exactly what you described in a in a sane way, perhaps a couple of guardrails so we don't actually cause customer damage or something, but actually attack it. And then we we have a whole bunch of people who are really smart, including some of you who've put in security uh practices, but I don't think they're enough. For the reason you mentioned, but also the reason we're using new technology, we're using agents, we're using all sorts of things. There's new attack vectors. I need someone to participate and make it their Well, you can try it out and see if you like it, but then, ideally, make it your full-time job to literally attack it, and we'll find exactly those problems. All right, thank you all. And have a great flock. It's Oh, Jeff wants to say something. >> Yeah, it's more of a comment than a question. Uh two things two things real quick. What he just said about being able to attack it is exactly why I want this in a sandbox. Yeah. Because we can really beat on like a new idea and make it really great before we are asked to depend on it. I think that's fantastic. The other thing talking about can we pick up things? That's the other great thing about the life cycle proposal is like if there's a piece of this big big initiative, this big idea that's ready to go, we can just do a change proposal and get that in >> Right. >> well ahead 50% of it just doesn't pan out, but 50% does, and it's a valuable change, we can make it a change proposal. It doesn't replace the change proposal process, it gives us a way to do big things, and then, when it's ready, pieces can be brought in and integrated. >> Oh, one more question. Who wants to wear the Hummingbird? Come on. Come on. Raise your hands. Who wants to wear the hummingbird? >> Steph, you should nominate the best question. >> [laughter] >> Jeff wants to wear Okay, Jeff's going to wear the hummingbird. Yeah, all right. >> Speaking of pizza, just really quickly, lunch is available for everyone today regarding regardless of ticket type. So, sorry about that. >> No, great. Enjoy your lunch. >> [applause]