Flock 2025 Bootable Containers: Moving From Concept To Implementation

Sorry guys. Um well, we don't have a lot of time. So, he's going to try to figure it out and I'm just going to kind of start. We'll see where it goes. Um so, first of all, howdy. My name is Sean Throko. I am a software engineer uh at a company called HEB. I have been a software engineer for about 15 years. If you do not find me online or helping out, then I will be scuba diving. So, if you're interested in that, please come talk to me. Um I am a father of three. I My first time at Flock. Uh so, thank you all for having me. It's been actually a very memorable and very welcoming experience. I appreciate everybody. He removed it. Um and we're here to talk about bootable containers today. Uh you guys have heard a lot about what bootable containers in Timothy's talk and Dusty's talk and stuff and I'm here to kind of talk to you not about the conceptual aspect of it, about what we could do, where we can be, but where we're at today and my actual implementations of bootable container systems. Any luck? >> I don't see it. This just doesn't recognize the >> Can I like email it to you? >> Yeah. I can't Oh, definitely. >> Or a USB drive or something? >> Yeah. I'll bring it over. >> All right. So, anyways, uh so, bootable containers, what does that mean for us? It means that our whole entire OS is defined in a container file, right? It means it uses the OCI image format. It means that all our upgrades are atomic. They're atomic units, so that way you can treat your That works. Cool. Cool. Cool. There you go. >> Thank you. >> No problem. Thank you. >> Yeah. All right, while he goes through that. Um so, atomic units, meaning that you can treat your computer knowing that it's going to boot up that next time cuz no more failed upgrades, no more bad packaging. It means when I go from one update to another, whether that's a minor version to minor version or minor to major or major to major, it's all an atomic unit. It's all either going to roll forward or roll back. And this combined with the very interesting security aspect means you can have a full end-to-end signed and and verifiable secure system between something like UKI boot, between your signed container images, and between OS Verity and the read-only file system and compose FS that will be coming. Here we go. Cool. That's me. Screw it. I'm in. >> [laughter] >> All right, so I just talked about all that slide. I want to highlight something here. There's a lot of icons up here. Because bootable containers are just containers, it means that we can use all of the existing technologies that are already in place for containers but on our OS. And that's a really powerful paradigm. These aren't even all the things, these are just the things that I personally use on my bootable containers. So, this is my hard won knowledge. These are all my adventures that we've gone through. So, first of all, I have a lot of QR codes. This is because I bought the receipts and I want to make sure that everyone is able to quickly get to that, whether you you know, cuz you're not going to have the link if I just put it here, right? So, everyone can look at it and if you're online, you should be able to look at it too. I'll make sure that I give people proper credit for what they've done. Um this was a blog post by Mr. Ryan Daniels. He wrote about his experiences installing bootable containers and things that didn't work. Um the first thing that he found is that RPMs don't always install nicely here. Going to bring up my container notes, all right. Think here. All right. Da da da da So, if you are trying to level in your RPM package, trying to add a user or a group or looking for a certain UID or GID you expect to be there, it might not be there. This is due to UID and GID drift that can occur between two different images. Even though you built it the same way using the same container file, the IDs might drift. So, in order to mitigate this, we recommend you use systemd dynamic users because then there are automatic provisions on every time the image loads and you don't have to worry about UID or GID drift as part of systemd units. The next thing you might happen if you're writing to /var as part of your RPMs, that will not work. Any kind of script that you have is just going to fail when you do the RPM build process. Sorry, when you install an RPM through the build process. I recommend instead that you copy the files that you need to go into {slash} var as part of the image build process. So, keep them as part of your repo and then manually just copy them in your container file. The next thing, systemd does not come up when you do your build image process. Therefore, if you are trying to look for something or rely on something in systemd, it will not work. Important gotchas. All right, the next thing. Uh Yes, the Etsy safety dance. Uh Etsy gets treated a little differently and has some gotchas you should be aware about. Number one is that the Etsy hostname file in the build time is ephemeral. It's actually bind mount from whatever you're build running this on and that prevents the file from getting persisted. Well, actually, it prevents it from being changed and it does get persisted into your actual container image, which can be a problem if you're trying to define your hostname as part of your, you know, day zero operation of deploying your image. There are a couple ways around this. Number one is use a builder pattern where you have a multi-stage build and then you can uh write your file in the builder and then you can just copy it in from the builder as part of your next stage. That will get around uh that problem. You can also run the uh no hostname flag as part of your Podman or Docker build and then that will prevent uh the hostname file from ever being written and then you can write in whatever you want and it should persist into your image. Uh the next thing, RPM packages, scripts, bash scripts, whatever you're going to run as part of your build image process are going to look for certain may look for certain files like Etsy local time, Etsy time zone, and those don't exist during the build image process. So, if you're doing anything that looks to that, it's going to fail. Finally, there is a well-documented three-way merge. That's what this QR code here is for. I highly recommend you look at it. Um it dictates that local files will get the preference. This means that if you have a file that is already in your file system that you have created, and then you say, "I like that. I'm going to add that to my container image." And then you make a container image, you publish it somewhere, and then, you know, you're you do a bootc update. You're not going to get that update. Because the local file that you wrote that first time has preference, meaning anything I do ever in my CI/CD pipelines, whatever, down the road, or any upgrades that are part of my image will not get affected on that system. That's an important note because it means you really have to use one pattern. You have to either do everything locally and use your local files, or you have to use everything in CI/CD and not and resist the urge to change things locally. There are ways around that. There are like temp uh transient tempfs stuff you can do to kind of play with it without writing over-writing what you already have, but it's an important gotcha that has bit people many times. Uh yeah. My personal opinion, we are not ready for a local-only usage of bootable containers. This is a link to something that helps. It's a tool called dnf bootc by Eric Curdin. Um but right now, if you try to just do local builds only, not trying to do bootable containers in a CI/CD pipeline, it's very slow. It's a slow process, and it takes a lot of space because you have to build your image First, you have to pull the image, whatever your base image is you're you're getting. Bring that into local storage. Then you have to do the OS build or the bootable container build. And then you have to import that OCI artifact onto the OS tree storage currently. That only takes a lot of time and duplicates the space, and we're talking gigs here, right? That's how big these images are. So, it takes up a lot of space potentially on a uh storage hampered system. This will be solved though when we have compos FS because then it would be a unified storage, and we won't have to have this deduplicate or this duplication of our OCI images. Oh my gosh, guys. Bootable containers are amazing for major upgrades. I've never had to worry about the I did I did two. I did 40 to 41 and then 41 to 42, and I didn't have to worry about it at all. There's an article here by Mr. Guitar, uh a man named Ben, who talks about his experience on using uh on his personal experience using bootable containers and how he did his major upgrades. Uh the long story short of it is I recommend that you use, just like Dusty said, tags and streams [snorts] to make sure that you're doling out your changes slowly across the appropriate systems rather than in all at once kind of method. Um and this is why I think Universal Blue did something really right. They have a very flexible system where they're able to uh as the pre-release builds of like the next major version come out, they will start building them themselves as part of the CICD process. They'll add a tag for testing, and then it's really easy to just switch tags on whatever, you know, branch you're trying to test off of and use. And so, they'll have a couple weeks where they'll let testers or any user just kind of go that testing branch, try Fedora 42 or try whatever next major version is, report any bugs out, they'll fix them all up, and then by the time 40 42 or the next major version is released, it's good to go. They've ironed out all the kinks, and because of how flexible tags are as like, you know, to an image. You can make it as easy as you want. They have um they have a latest if you want the bleeding edge. They have 41 if you're dedicated just having 41. They have a 42 if you just want that major version. They have uh the long-term support model, which is always like minus the latest minus one. Whatever experience you want to give to your customers, your clients, your partners, you can do that. And I really think that's a great thing, but the problem is not everything installs super well. Uh NVIDIA is still a major pain in our butt, as we all know, and things like ZFS support or whatever extra thing you might need to install not always super easy, but still doable. >> The slides don't change. >> Oh, they're not changing. I'm sorry. That's awful. I'm sorry. That is the uh Mr. Mr. Guitar's blog post if you want that. I'm sorry about that. Thank you for calling on that out. All right. Testing. Testing, testing, testing. Testing is hard, guys. Um I went too far. Sorry. Testing is hard. Um testing a container is really easy, actually, because you usually have a very small scope. Uh it's usually like just your application, and you can look for, you know, my files are there, my thing behaves the way I expect, you know, I pull down this thing and it's there. Awesome, no problem. Testing bootable containers, though, is much more complex. You're not just testing I have my one application in my container. You're testing a whole bunch of services are all there. And because of what I stated earlier, where like things like systemd aren't present during the image build process, you can't just in your CI/CD process say, "Great, spin up my image and let's test against it." Because not everything is there. So, in order to do this, you really have to spin up like a whole VM dedicated to this as part of like your CI/CD This is an article by or actually it's a PR. It's a PR by Martin Pitt from the Cockpit project. He wrote a PR about uh actually taking a BootC image, spinning up using QEMU in a GitHub Actions pipeline, I think. Um and then testing against it there. And it's a start, but it's it's it's tough. Um Yeah, so what I'm going to >> It'll get easy. >> It's not easy, yeah. >> It will get easy. >> Perfect, yeah. There's there are ways to do it. We just don't have it right now, but it's going to scale out better. And yes, testing farm might help with that. I'm very excited to learn more about that. >> Why can't you test systemd inside of a container? >> Because whenever you run your image um as part of your CI/CD process, you're spinning it up as an a container image, right? You're spinning it up using Podman, and not all your things are there. Like I said earlier, you're missing files like Etsy host name, Etsy uh local time. Systemd is not up during that time cuz it's using >> No, I understand that you built it. But you ran it in Cockpit and systemd inside of a container network will not create all those files >> Yeah, you're right. You're right. But not everything is there. That That's what I'm trying to illustrate, that there are just differences between running a BootC container as an image versus running it as a VM. >> No, no, I I wrote a blog um so I mean, the bottom line is you can test. You should be able to test all of user space as a container before you actually install it >> Hopefully. That's That's the goal. Okay, we'll talk more about that. Um cuz we're running out of time here. So, next thing. Um what I recommend is if you do want to test your pods uh your images, I recommend adding a separate stage where you just add on the extra layers that you need onto the layer you're trying to test. And that will allow you to make sure you have things like Pytest or Ansible or whatever else you need to run your tests versus, you know, having that cruft in your actual image itself. Uh this has proven to work out for me pretty well. All right. Yes, how far can we go with bootable containers? Uh I think we've already kind of proven that a server is definitely within reach. Uh developer workstations, absolutely within reach. I mean, that's the whole entire thing like the universal Bluefin DX does, right? Absolutely within reach. I've been using it now for years or 2 years. Awesome. Um Kubernetes host, can you use this as like a Kubernetes like agent node or master node? You absolutely can. You absolutely can. Not a problem. I've even done it using system extensions as a systemd extensions as well as just baking K3s into my image using the scripts that they provide. Not a problem. Can you make it a NAS? Yes. Yes, you can. You can make it a NAS. You can have your SEFS tab be part of your repo. You can uh install ZFS. I've got NFS and Samba both working as part of a bootable container. That gets you pretty far. That's a pretty good. I cockpit on top of that, you're set. That's going to be a good setup. A hypervisor. Ooh. This was my goal. I wanted to know how far could we push bootable container technology. What's ready today? And I said, what's the most complicated thing I can think of? And I came up with an OpenStack hypervisor. I was at Rackspace for a long time and that was like the bread and the bread and butter, right? That was the thing they really wanted. So, I tried to make an OpenStack hypervisor. Can you make one? No, you can't. It's really, really tough, guys. Um OpenStack is really complex and the only way to really deploy it is using tools like Cola, Koala, uh and TripleO and stuff. These are scripts that run and just because how bootable containers image process build image process not everything is there. It's missing files. It just doesn't play right. And I didn't have the time to go and start debugging everything. So too many cuts couldn't make an OpenStack hypervisor, but a KVM one, sure. That could work. All right. Some of my thoughts. Um I really like Dusty's image from yesterday of how there's a spectrum from bring bring it yourself bring do your own to uh consume from a deliverable somehow. Um there's a lot of it's a spectrum and there's a lot of overlap. So things like cloud and it kick start, you know, ignition and like boot C have a lot of overlap. You can you can do a lot of things as day one, day zero, and day two operations and it's it's not really there's not really a good rule for like where to put things. It's all up to what you want, which is both a blessing and a curse cuz it means we have we have containers agent technology that's flexible and allows you to make what you need, but also it's kind of difficult to get started if you don't know really what you're doing you don't have good fundamental practices. Um and then again like I said, local usage not there yet. We're going to get there. Um but people there's a need out there in the community for people want to just do this locally only. They don't want to have to spin up their own CICD pipeline. They don't want to spin up or have to use a tool like Koza to get all the things they want. They just want it to boot and then for it to run and then just be atomic and have all those benefits. That was embarrassing. All right. Um Let's see which one is that one. Tips and tricks? Yes, tips and tricks. All right. So tips and tricks. These are my recommendations for if you want to get started, what should you be aware of? Um I would say learn how to write container files effectively. Um, there's a lot of caching that can happen between the different layers of an OCI image, and I want to make sure that you guys are caching as much of that as possible. This QR code here is my example repo that I have with a bunch of different uh bootable container bootable container images um, and CI/CD and all that stuff. You're free to copy and take that. It also has like boot C image builder built in, so you can just say, you know, what stage you want and just hit go, and you'll just get an ISO artifact out of it. Um, it also has, oh my gosh, it has This was a cool one. I use Sift to create an S-bomb of my container images. So, it's not just my application, but my whole entire OS. And then I was able to use Grype to do vulnerability scanning against all those, and then I was able to upload that to GitHub Security Center. And that means whenever one of my container images gets built, which is weekly, I get a set of new alerts for every single image saying, "Here are the vulnerabilities for these things at this time, where it came from." And it tells you the CVE number and everything and, you know, gives it a score. And that's the kind of like visibility for a whole entire operating system that's not so easily achieved today as part of, you know, just a CI/CD system update. That's really powerful. So, that brings me to my next point, use all the existing technologies. Anything that's out there for container, you can use for your OS. Um, like I said, use tags and streams. Make sure that you're doling things out. Um, a good example I heard yesterday was like a university who wants to roll a change out, right? Maybe you change start with a change in, you know, a computer lab, right? Test it out there, make sure it works, then you can dole it out to your library, then you can dole it out, right? Um, use tags and streams for the best flexible experience that your your customers, your clients, you are looking for. Um, and then a personal recommendation is I have a lot of different types of hosts. Uh, I have like K3s master, I have K3OS node, I have my NAS. Um, switch it up. Use different container files and then bringing your image your base image there or have different stages for each different type of host that you're trying to do. So, with that said, I think that's it for questions. Thank you, guys. Yes? Oh, wait, do you want a microphone? Do we Are we in that anymore? Is this on? >> Um, so how close are you to have the systems bootable from network devices? >> From network devices? >> Like with iPXE or PXE boot. >> Oh, yeah, you can do it already cuz that's that's just a disk image that you're putting onto like, you know, PXE boot, right? That's just a disk image. So, if you get a bootable container, you can use uh, there's I think three different ways right now to create that into a disk image and then you just put that onto whatever system, you know, you want to boot from and that's it. So, I did that for like net netboot.xyz. I just took a base image, put it in there, all my things netboot from it, and then I can switch to whatever image, right, I want. >> Okay, and then in this case, which initrd you are going to pass to the iPXE, any one or the one that they have within the boot C? >> It depends on how you go about it. You can do one per Yeah, if you have one have like five ISOs, right, then you can totally do that and you can boot off that or QEMU do whatever you need. Um, or you can just have one base one and then you can have like Ansible or something go in and switch out uh, to whatever image you want to look at. >> Thank you. >> Yeah. >> Are you Uh, [clears throat] are you aware of any clouds out there integrating OCI images directly like in a way that I would build a golden image using Podman build, pushed that to a registry, and told the cloud, "Hey, please start a VM from this image." >> Yeah, I know what you're talking about. Um I'm not aware, but again, I work uh I don't work at any of these vendors. I'm very much community member. I don't work for Red Hat, so I wouldn't know. But if you want to use like OS builds uh bootc image builder to create a AMI to create a What is it? GVE GK the Google what version of it, right? The Azure one, you can totally do that. Upload them, not a problem. But no, I don't know of anybody doing it. >> Uh another question maybe. Uh is there I mean, is it is the OS 3 thing a mandatory thing for for these bootable containers? Can't I have a normal golden image without OS 3 thing? >> Without OS 3? >> Is there a some technological requirement between these two? >> Between OS 3 and bootable containers? Yes, so currently the bootable container uh base image, like the one for Fedora, comes with OS 3 in it. Um that'll get replaced by compose FS eventually, and then that will be the default and what everyone will use. But right now, yeah, there's a a dependency on that. >> Thank you. >> Questions? This man seems important back here. >> Uh have you done anything with Podman bootc? >> Podman bootc? >> Yeah. >> Absolutely. Yeah. Um I'm I don't want to tell everybody like here's a secret, but yeah, if you just use like Podman desktop to create a Podman machine and then use that, it's a really really streamlined process. Yeah. >> Right on time, guys. Thank you for your time. >> [applause]