Flock 2026 Fedora BOOTC: Building And Operating Image Based Systems

Hello guys. Can you guys hear me? Okay. Uh my name is Grzegorz Siwik and uh I work for team Bootc at Red Hat. Uh software engineer for the team Bootc and today I'm going to talk about Bootc um image mode um and run a quick demo, uh talk a bit about BCVK. Uh I'm going to talk about what new things we have done in the Bootc environment. And yeah, let's kind of dive right into it. So, kind of talk I'll start with what we did in package mode. Uh what we did in there was we uh installed packages, we tweaked configs, we applied quick fixes, and uh over time the uh system would drift and at some point the changes were not bad, but what we do is we just lose track of those changes. And at one point two machines that should be identical were not identical anymore. And uh we would end up with inconsistent environment. Debugging {slash} reproducing bugs would get really hard. And uh there would also be this increased uh security risk with the vulnerabilities cuz there were changes you probably don't know were there. And uh let's kind of see what happened. This This is kind of like similar problem that was in applications. And uh containerization literally changed the way uh we handled that problem. So, it literally changed the way we built, shipped the applications. Uh gave us this common portable format that uh gave us this clear version artifacts for every application that we had. And this obviously came with a lot of big ecosystem of tooling, automation along with it. And so, it was more about like what if you applied the same system to the operating system image in general? And this is where OCI-based uh image mode comes. Now, instead of installing, modifying, uh and drifting away from your initial state, you would build, deploy, and replace the whole image if you need a change. So, your image would be this version artifact. And let's say if you need a new package, you would create a new container file, put in these new files in there, a new configuration, new fixes, whatever you want, your packages, and you would end up with a new image after that. And so, yeah, instead of fixing systems, we would just change the whole narrative and just start replacing the systems. And that was the that's just a fundamental shift on how we think of system life cycle. So, yeah, this is pretty much it. So, instead of application container, we ended up with a bootable container. Um so, only changes like we just have a kernel, bootloader, initramfs, whatever you need to make it bootable in there. So, same container format and bootable end product. So, uh we kind of talk about talked about how and what was the process to get there, but like why do we really need the bootable containers themselves? Uh I'll start with the AI ML stacks. Uh I haven't really personally used it, but AI is cool. And uh so, AI and the machine learning workloads have really deep dependencies. And if you kind of mess one thing up, it could really mess up the whole system. Uh with BootC, uh you can version the entire system, uh kernel, and video driver, CUDA toolkits, ML framework as a single image. And you can test it once and deploy it anywhere uh with same bits from kernel to frameworks. And uh the iterations are really fast and reliable. Fast as in like if you want a new change, you can just put it in your container file, get a new image, and security-wise you get you this operating system is immutable cuz it is based on uh BootC, which is further using OS tree the hood. Um so yeah, the immutability is pretty similar to like CoreOS um and it helps with experimentation. If you want uh some new change, put it in a container file, get a new one, yeah. And if you don't like it, if if it just messed up the whole system, it's pretty easy. You can just roll back the whole image. And uh that just allows experimentation really easy, which is one of the core components of Fedora in general. And uh yeah, third one being deploy anywhere. Um could be edge devices, could be cloud. Um so yeah, single container images uh entire fleet actually. You can manage the OS uh and app as one client uh using familiar registries and pipelines you use for applications. And obviously familiar container workflows, which is something we already kind of like talked about. If you have seen a container file and if you can write one, you're pretty much like 90% there and you can just like customize it and you'll end up with a with the product. This is uh uh basic uh workflow of uh complete BootC life cycle. So yeah, you start with a container file, get a BootC OCI uh image there. And from there you can convert it to um QCOW2, whatever you need uh using like BootC image builder or yeah, or you if you have an existing BootC deployed system, you can push it to the registry and use BootC upgrade or BootC switch. And uh from there like you can just manage your whole life cycle pretty easily. Um so let's have a look at the tooling involved. I know I kind of like talked about BootC, this and that, which comes from bootable container, but it's also a CLI tool. So yeah, this is the heart of bootable containers. Uh this is how we brought image mode to many distros, Fedora being one of them. Think of it as a bridge between container images and your running system. So, what exactly is this tool? It's just a CLI tool. And think of it as a client-side tool for your operating system. So, instead of DNF update pulling in individual packages, you would just do bootc pull which would pull in a complete new image from your container registries. Could be Quay, Docker Hub, GSC, whatever you used to. And uh Yeah, so that's kind of like the gist of it. And so, the core commands of bootc I'll just run through this pretty quick is bootc switch. We So, this is something you would use if you want to switch to a totally different image. Bootc upgrade if you want to upgrade your current image and your if your quay.io has this latest image with your latest tag, all you got to do is like bootc upgrade. It's pretty similar to like when we do get pull. And bootc status, similar to like git status, it gives you the status of your image. And bootc rollback just reverts. So, like if you upgraded from V1 to V2, all you got to do is bootc rollback cuz you didn't like V2 and you'll go back to your previous image. Easy peasy. And before I kind of like demo this, let me talk about this really cool tool that our team worked on in last year, I guess. It's called BCVK, bootc virtualization kit. So, this is a production-ready tool for converting bootc images into local VMs for testing and on Linux testing for production. What it does is it's a very convenient wrapper for around libvirt and QEMU designed specifically for your bootc workflows. It It does all the work that's kind of like underneath there converting a container image into the disk, creating libvirt XML configs, setting up networking, SSH, managing VM life cycle. It solves a very simple problem that you just built this bootable image with your container file. How do you test it as an actual operating system before you push it to production if you So yeah, this just really makes this whole workflow of testing bootable images easy. Um so this good thing about this is like this all happens on your laptop. You don't need any cloud resources. No infrastructure setup needed at all for this one. So there's three modes for BCVK. Ephemeral is for quick testing. It's Podman-based. Um no disk creation needed at all. Libvirt is when you need some persistent VM and you want to like start, stop, reboot, and test those changes kind of like really into No knowing all these workflows in there. And two disk is when you really want to generate the actual disk image, the QCOW2 files. You deploy it. You can deploy anywhere. So from Podman build use your container file to running VMs in seconds, BCVK just makes it really easy. Okay. So let's run into a quick demo. Oh my god. Where is my terminal? >> You have it on the other workspace. I think you need to go back. >> Like here? Oh my god. Go back. Where did you go? No. Yeah. Oh yeah, there we go. Okay. So I have two container files. Um me wish I could see the screen better. Okay. Yeah, so one container file is this. Pretty basic. We're using a Fedora boot C44 base image and we are installing a bunch of uh packages of vim, tmux, and uh we're going to display a welcome message showing that this is version one and uh Here we go. I'm going to Oh, shoot. You mean this, right? >> Thank you very much. >> Okay, there we go. So, yeah. This is uh second container file. Uh we're adding two more packages on top of it, htop and git, and obviously changing that uh welcome message there. And uh yeah. Okay. And what I've done before this demo, sorry, before this talk is I did a podman build using both the container files just to save our time. So, oops. And demo. Oh my god. There we go. Um so, I have two images there, V1, V2, and I also pushed uh the V2 to quay.io so I can pull it in when I run a VM. So, pretty easy workflow so far. And uh for the actual workflow of um running the VM, I just recorded it cuz it just takes more than a few minutes, pulling in everything. Okay. Should look better than that, actually. But, the actual command is just uh libvirt run dash dash name. You just put any name, demo V1. I'm using disk space file system XFS and using the image that's available locally. And this quickly creates a VM for me. Like, that's it. It's one command. That's it. And then uh you can manage whatever uh images you have using bc vklibvirt list. There we go. We have it ready, to go. And basically with SSH, you're ready, you go in, you see the welcome message. Uh it shows that this is the version one. And it's booted from the boot C Fedora demo, sorry, Fedora boot C demo V1, which is a real boot C deployment. So, pretty easy and you're like up in there like in in a few minutes actually. So, yeah, Vim is in there. Git's not in there cuz it's obviously wasn't included in V1. It would be in V2. So, yeah. For the sake of the demo, I also pulled in the quay.io image into my VM, and we'll use that to um switch into uh that image after this. So, yeah, boot C switch um So, this is Yeah, I pulled into This should be like relatively pretty fast. Um So, what it's doing is creating this file system tree, setting up boot loader entry. It's running system completely untouched, which is um how I talk uh how we describe the upgrades, which is A to B model. When you're upgrading to the B image, uh you don't touch A at all. So, there's no like um anything like in the middle there. It's either A or B. So, there we go. Uh so, we did boot C status now. Uh so, the booted the booted image is still V1. The staged uh image is V2 now. Now, this is not applied yet. As I said, up upgrades are like transactional. So, it's either A or B. So, all we got to do is reboot cuz now the V2 is staged. And uh you just wait for like 30 seconds or whatever, go back in it, and it should be booted to V2. There we go. Now, you have htop, Git, everything in there. And so, this is uh yeah, it shows that Git's in there. And uh so, now I want to show you how rollbacks work. Now, for some reason you're like, "Oh, I don't really like it. Uh I don't like this version at all. I want to go back to V1." So, all you got to do is boot your rollback, and it'll just go back to your initial version. Like, no fuss. So, this just makes the whole experimentation really easy. You can just mess around with packages, mess around with configs, whatever you want, and it just lets you go back to your initial state in a minute. Yeah, you go back in, and you'll be back on Yeah, it gets another anymore cuz you don't like it. And there we go. Yeah, that's uh very basic demo. I hope it kind of gives you an idea of what BootC is, what BCBK is, and how BCBK really makes the testing easier for the BootC images. Okay. Back to slides. Oh my god. Better. Okay. All right. So, what we say Yeah, quick recap. We uh instead of adding packages into the V1, all we did is that we rebuilt a new image with new packages. Uh to update, we didn't really do any kind of update. We just switched to a totally new image. To rollback, we didn't do any backup or whatever. We didn't need to worry about it. All you got to do is boot your rollback, and it just takes care of it. Um bit about file system layout. This is pretty similar to CoreOS actually cuz we're still using OS 3 underneath. Um So, all everything is writable during your build time. User, Etsy, war, everything is uh um writable in the build time. You can just put whatever in there, new packages, whatever you want. Uh but there I know we're going to talk about im- immutab- immutability and how this image is immutable. But the thing is we have two exceptions, Etsy and war. Uh war is a read-write for persistent data, which is logs, databases, application state. And so yeah, updates don't touch war. And Etsy is also read-write, but it's it's a three-way merge on an upgrade, just like in CoreOS. Uh your host name, static IP configs, local customizations, everything, those are preserved when you update, uh but you can also pick up new defaults from the image. They all kind of get merged in together. And uh this kind of like is good enough for 80% of the cases, but for for 20%, if you want like high security, like BootC does offer a powerful configuration options. You can make Etsy chanzian. And if you want what it does it when you it just wipes on the reboot and it always reverts to the image defaults. And it's kind of nice for high security environments where you want zero configuration drift. So what's new in last year or so? Um BootC became a sandbox project for um CNCF, which is kind of cool, I guess. Um and there is another thing was um system reinstall BootC. This makes the conversion from package mode to image mode really easy. This command will take care of everything for you without reinstalling the whole system. Uh which is big practical win. Um third one is integration with the system D soft reboots. Um you can apply updates without a full hardware reboot. So earlier, if you wanted this to be implemented, the change would take about 30 to 60 seconds, uh depending on your system. This would just take care of it like 5 to 15 seconds, depending on your system. Uh this is makes the whole development and testing really fast. The OS switches to the new image by restarting system D and not rebooting the whole machine. Obviously, if you want kernel a new kernel, whatever, you would you got to do the whole reboot and this wouldn't work, but it still helps in testing. Uh then you also have bootc upgrade that says download only. Uh earlier, bootc upgrade would just download the whole thing and stage it for you, but now you can just download it and it kind of gives you explicit uh permissions, sorry, explicit um access to when you want to deploy and when you want to stage it. So, that was another thing and then there's bootc install factory reset. This is a CLI option uh to reset an image mode system to its original state by wiping local customization uh in Etsy and var. So, it basically lets you perform non-destructive factory reset on a live machine. So, yeah, these were like some of the things that happened in past little bit. And uh so, what's new? Uh composefs integrity sealed image. Uh sealed images uh they use composefs uh with fs-verity UKI. And uh it it creates a verified boot chain from uh firmware to runtime and file system. So, fs-verity continuously uh checks that there's no changes in your system. It double check that they're accessed at runtime, sorry, it just keeps checking that at runtime and refuses to run if anything's been tampered with. And right now we have a sealed um a bootable container available to test for Fedora Atomic Desktop. And uh but yeah, if you want more information, Colin's giving a talk at the DevConf um on Thursday, 1:15 p.m. And uh obviously wider distri- um cross-distribution, uh we have bunch of um distros now using bootc. Fedora bootc, uh RHEL bootc, CentOS bootc. So, this is We got a bunch of options now, so it's expanding and it will keep expanding, I guess. And uh third one is the bootc operator. This is a pretty new tool, I guess, which came in last few weeks. It's been worked by Jonathan Lebon and Alice, I guess. So, it lets you manage OS updates for your bootc based cluster nodes, just like any Kubernetes resource. So, instead of SSHing into each and every node to update the OS, all you got to do is you got to declare what container image you want in a YAML file, and it just the operator handles out rolling it across your whole fleet. So, it's pretty handy if you want to if you're managing a big fleet on Kubernetes. And yeah, this is pretty much in the next and experimental. And yeah, these are the links for BCVK, bootc, and examples. Feel free to chime in. There's a lot of exciting stuff um working in working around these projects. Feel free to experiment with the tools and uh Yeah, thank you and any questions? >> [applause] >> I'm just wondering like we are using boot image builder to build the QCOW2s. Uh will I get the same results with BCVK to disk? Or >> I think yeah, it It would be same if you're using two disk. Yeah, you get a Yeah, there's no difference, right? Yeah. >> Thank you. >> Um hi. I'm a member of the Super SIG and a number of us are interested in bootc images and we were wondering, do we come to you? Do you come to us? How do How do we work together? >> How do we work together? I feel like if you want to do that, you can just experiment on your own to start with. Like I feel like our repository has all the examples on how to like I get started with. But obviously, if you want any help, feel free to chat us on these channels and we'll be happy to help with that one. So. Yeah. >> Thank you. >> Hi, I'm >> [clears throat] >> I'm interested in using this technology for like a hardened standardized developer workstation at my company. Do you have any experience or do you know another company because I don't think I will reinvent the wheel here who is already using this on a scale? >> Any Do you mean are there any company that's already using BootC technology? >> BootC technology for an immutable workstation for developers and normal people? >> I wouldn't know the answer unless you know any, Colin. >> Yeah, we have the Atomic desktops we referenced earlier, right? And it's kind of the same infrastructure and we're working to converge like the Atomic desktops deriving from the space image, but you can derive from the Atomic desktop container and yeah, so it's it's absolutely a first-class supported thing. I mean, yeah, in my number of our engineers' desktops at home like my Linux workstation at home is a custom BootC based desktop and it's it's absolutely designed for that. There's there's an important customer Red Hat who they you know, they do it at servers, they don't do Linux desktops, but they do some of them do for the same reason consume it as a desktop workstation. May not always their primary like I think one of them has a Mac too, but like it's a good way to like live the live the experience and see what comes next. So yeah, I mean absolutely supported and you know at scale it is just basically rolling out container updates, right? So Yeah, >> [laughter] >> absolutely. Um The the problem is it's hard or we're not always allowed to to share that publicly, but one company who was who was very vocal and public about this already last year was for instance Visa. I would I would say Visa is a pretty big shop with a lot of servers out there and they love it. I've seen image mode or boot C being deployed on satellites. I've seen it being deployed in nuclear submarines, banks, um and sometimes also just smaller deployments in some regions in the southeast of France where there is one, you know, a very competent sysadmin who says like, "Well, well, holy moly, now I can automate everything with one one pipeline and the updates are atomic and I don't have to manage the drift that was mentioned at the at the beginning. So I it really depends on the use case. If you have one sysadmin, suddenly you can manage thousands of machines without a problem and um the immutability is certainly awesome at the edge. So satellites or submarines are certainly very edgy. >> administrative workstations. So we are trying to minimize the attack vectors on on the work stations which which we administrate servers and systems. And what they are doing is they clamp down Windows 11 and call it privileged access work station. And I am saying, "Hey, we can do that better with an hardened compliant immutable work station based on for example Fedora Atomic." But I need life examples. Like I I for example I talked to Siemens. They are using things like that. And but but I need a list of examples to go to my bosses and say, "People, look, others are doing it and they like it." So, if anybody has any tips for me, I would be grateful. Thank you. >> Yeah, just a quick one also like Fedora I know I I kept saying Fedora boot C throughout the talk, but it's called Fedora Atomic now. But a lot of our repositories are saying Fedora boot C, so I had to do it for the sake of the talk. But it'll be called Fedora Atomic going forward. Yeah. >> Thank you very much. >> Thank you, guys. >> [applause]