Flock 2026 From Compliance to Containers: Managing Fedora Fleets in the Atomic Era

Welcome. >> Um We're uh Adam and Jonathan. Adam's from Fleet, I'm from Red Hat. Um this talk's going to be in two halves. Uh first we're going to talk a little bit about uh NIST controls and um uh the sort of things that auditors will be looking at. Um and uh how we use uh use OSquery on Fedora um as a kind of transparent SQL engine to uh audit systems. Um then we're going to move on to atomic hosts like Silverblue, uh Bluefin, CoreOS, where user is a uh slash user is a read-only file system. And rpm -q only respect reflects the currently booted operating system, but not might might not reflect any uh staged changes. Um and then uh we're going to show how that uh we can reflect the visibility in OSquery. Uh and then the whole stack is open source, so uh want to talk about it? >> Yeah, of course. Thank you. I happen to work Uh there we go, it's come back. Good stuff. So um yeah, what we'll be presenting um essentially. >> [snorts] >> So I I work at Fleet as a customer solutions architect. So I work mainly on macOS, Windows, and the next fleets. Um we're going to be talking a bit about the OSquery, um how we can use it, and that visibility. Um I think here we'll just go kick off, and we'll talk about the compliance challenge. >> Yep. >> Cheers, Jonathan. >> Uh so really the core problem is that Linux uh often fails audit because uh because not because it's insecure, but because it's difficult for it for you to prove to the auditors that it's secure. Um there's not really a whole lot of MDM software for Linux. Quite often uh you rely on things like shell scripts and um you know bespoke uh scripts that you will generate, you know, CSV files or YAML or something like that that you can hand to the auditors. Um really they want the evidence. They don't want you to just say that it's secure. Um and we really want to have some sort of open source visibility into the process. Um every check is queryable, forkable, and on GitHub. Um and I guess the next the thesis for the next 15 minutes is going to be about that. So, what is Fleet? Thank you, Joe. So, yeah. Um so, Fleet, what is it? It's an open source self-hostable platform. Um you can manage, secure macOS, Windows, Linux from one place. Um you run it, so the data stays yours. You can self-host that. It's built on OSQuery. I'm not sure if everyone's familiar with OSQuery. It was built at Meta a while back. The whole point is like a operating system query. Um >> [clears throat] >> so, what it does, it's an agent that exposes the operating system as SQL tables. A query is just a SQL question over those tables. Um a policy is a query with a pass or fail rule that we kind of use for automation. So, one or more rows means um that it's a pass. Um [snorts] for example, disk encryption. Uh you can kind of just simply check is the device encrypted or not? And if it's not, either have a user message, or have a auto remediation, whether that's a script or a package for a software. Um GitOps, that I will talk about a little bit and I'll show you some examples, is the kind of how whole fleets are configured. It's how we define a fleet report, a policy, and all those controls live in Git as YAML. So, it's super auditable. Um changes are pull request, it's reviewed, it's version controlled, and then the CI applies them with our binary. Um and yeah, that will be most of the bits we'll go over. And over to you for the next Yes. >> So, one of the things that we're often uh looking at are to uh show to the auditors that we are following the NIST guidelines and looking at partic- and we're going to look at a couple particular NIST controls that we have for the OSQuery. Um for example, uh this is how we would be uh installing deploying Fleet uh on a regular Fedora system, you can just create a package and install it. On Silverblue, it's a little bit more complicated because you can't just RPM install, you have to do uh RPM-OSTree, and of course our package has some issues with that just because of the directories that you use, but it does get installed, and we are actually have several systems already in Fleet that are running uh atomic distributions, so um here's one of the policies that we can look at. Here, it's a really simple I don't know, can people see this? But, there's a you just basically select one from disk encryption, which is a table in the OSQuery, uh where the name is slash dev slash dm wildcard, and it's encrypted, and that will return paths if it's uh if it has an encrypted disk and it will fail if it doesn't. And the action when we do this with Fleet you can actually have actions where a script can run to to remediate it, but when you've got an encrypted disk that really means you have to go and talk to the the person who owns the system, make sure it gets reinstalled. But we could definitely do a do things with Ansible or something like that to fix it. Um Here's another set of policies that are that's really useful for checking. You can look at a variety of checks for the SSH authorized keys files on the system. And like for the first query up there that looks for really old certificates that it's over a year old. Um and I can go and poke it over to the Fleet page to show you what these look like. Um the middle one it looks for keys where there isn't actually a user on the box. And the third one is looking for keys that are using a non-compliant algorithm. Let's see if I can pop it over to the Let's see where's the console. Policies. >> Yeah. Yeah. >> So here's an example. Do you want to walk through that? Thank you. >> So [snorts] essentially here all we've done is as per the slides, we just have a policy for each one. And we can see that the two hosts that we have in there are passing. And here I'll just show you what the query actually just looks like. As per the slides. Nothing too complicating. We're just using a a join with the user table as well so we can actually get the right information. And what we can also see is as this is a policy and I have it all managed in GitHubs, what we have is just this policies. We can list a few of them together. So, I've actually got multiple that I'm calling out. And then when I need to make a change, we review this, merge it, and then this will be in fact in the GUI ready to report on live status um for your policies. Cheers. Done. Oh. >> Uh how does it work with the uh centralized user How does How does it work with the uh centralized user management? For example, these old file authorized keys files are missing there, so you would not see them and still users might have outdated keys in the centralized store. >> So, the question is how we handle it on it for a centralized user. So, it's like LDAP, Active Directory, something like that. Well, in those sorts of cases, you'd be able to audit it from the data uh from the you'd be actually looking at the uh directory to look at those sorts of things. Um this is on a per host system, so you'd be looking at stuff on the local host. Uh and this actually looks at only local users. If your users are actually being managed in SSSD with uh Active Directory or you know, at Red Hat we use Red Hat Directory Server. Um uh we'd have to be using something else to do that uh because those things are stored in LDAP. So, you But those are definitely things that you can audit. It's just this is for uh looking at individual workstations, things like that. >> And in theory, you can go crazy with this. So, this is what our default tables do, but if there's a way for you to generate an artifact on the device, you can then use other tables like JSON files table or just files table to actually just extract that information. So, if it's maybe too difficult, the teams don't communicate too well, it's hard to get that information from the central team, um there's always ways. And later on, we're going to show some of the stuff we've built with extensions that allows you to kind of do that. >> Yes. So, actually one of the things I do whenever I need to look at local users table and things are in the directory, uh there's a last table which populates records from the last DB on your computer. And so, I just look at people who have logged into the system and then query using those users to look at their home directories and things like that. So, it's it's very possible. It's just this is kind of generic how you would do it. Oh, actually. Yeah, just >> So, yeah, we were talking about this yesterday and we were saying there's some other queries I built out I think about 3 4 months ago for Open Claw. So, the beauty obviously with those queries, you can do this for Linux, Mac, Windows. As long as it's got OS query on there, there's a table and we can detect that. So, what we kind of did is we have four score indicators and we can do scoring in policies. It doesn't just need to be a yes or no. So, with this one, it's if a host has three or more, um that will fail. If it's a single weak signal, we don't want to necessarily raise an alarm. So, we can score it that way. So, separate policies, they'll catch system D, launch deeper systems, uh that's our agents that will kind of survive a reboot. Um we've got reports that back it up. So, an investigation snapshot for the current state of the device. We've got threat huntings which will show differentials. So, with some of the reporting we can send, depending on how you want to adjust it and you'll see them, you can have a full snapshot, which will have everything. Maybe you want it less noisy, and you can then use it just a differential. So, when something does switch, that will actually create the alert. Um >> [snorts] >> And then, yeah, it runs, like I said, on Linux, macOS from just one config. And later on, I'll do a bit of a demo and show you some of those slides and how that works. >> This one's up to you. Cheers. So, we also want to be able to do vulnerability scanning. Um In Fleet, it's all fully uh self-hostable. It uh that doesn't There's no telemetry leaving your network. Uh osquery reports on RPM packages and the OS version needs check-in. And then, Fleet evaluates each of these against Red Hat Oval. Um since we're talking about Fedora, we actually kind of pair those up with particular rail releases, since we're using rail's oval data. Um it it does sync from rail uh from the rail oval uh database. Um but then, it kind of fills in things from the uh the NVD CPE, as well. Um and There's basically here we've got all the various exports that you can do. And then, that's useful for uh doing sort of audit compliances, as well as just looking to see what your fleet of systems have installed and whether you have any issues that need to be resolved. So, how do you go from these uh policies to what the actual auditors want? Um you can report basically a point-in-time snapshot of what you have. Uh you can uh have these queries return results. Basically, uh, in the same answer. There's, uh, when you're using something like get ops, you can look at the get history of every policy, see who made the changes, when they went into effect. And, uh, you can, uh, export all this information into a JSON or CSV file to hand to your auditors. And here on the side, if here's just an example of what the, uh, policy looks like from the API, you can pull the information right there. It shows the that that one there is basically the query that you get the policy for what the, uh, disk encryption, whether it's systems that does You can see that there's 142 passing hosts and two failing. And it even shows the update time, uh, last time you updated that policy. So, now we're going to talk a little bit about, uh, Atomic Linux and how to manage those things. We're going to shift, uh, from the mutable Fedora to Atomic. Um, we are just going to kind of go over the basics of what we've discovered and some of the issues that we did and some of the things we've done to address them. Um, so on DNF, as we talked about, you know, it's really is DNF update, uh, RP, uh, RPM OSTree has, uh, a variety of ways of being able to get that information. Actually, there's three truths that you get. You have what it was, what it is now, and what it will be in the future. Um, this These all these three states um, will coexist at the same time on a running system. You have like the commit that you were booted into, the the commit you're booted into right now, and the commit you will be next. Um, don't get anything like that on Fedora uh standard because you just see what you have. You might be able to look at the get history or I'm sorry, the DNF history and see what was packaged. But, you have these three states and here the talking about here. So, we have the next boot, what's going to be the next boot, what's right now, and what it was. And uh we also want to be able to look at stuff that like what's pinned and things like that. Um, these assumptions that we've made kind of break break down when we're looking at atomic systems. We uh us- assume things like that we have a uh that the Etsy lives forever, um, that DNF is the way you install something. There's lots of But, you you understand that it's a lot different than what it used to be. And that, you know, {slash} user is not writable. A lot of a lot of things assumes they can drop files in {slash} user. And uh yeah, that RPM QA is going to be the source of truth. Doesn't really help whenever your next boot's going to be a different thing. Also, there you can have things that are layered on top of that commit. Thank you. So, here what we're going to look at is how Touched it and I've broken it. There we go. is how we do it with fleet considering that all of this has changed. So, what we uh what we ended up doing is I'll give a bit of background about what the query is doing. But, osquery is not a database, right? Each table is from essentially a virtual table um that's being generated. So, SQLite calls the generator at query time, whether that's a policy or a report. Um the generator collects the live state. So, we've got some examples over here for like processes, listening ports. This is essentially what it's just kind of getting the output from. Um once it's collected all of that, SQLite runs the join and where over those kind of fresh rows just in memory. Nothing persists between the runs. So, rerunning returns current truths. It's not like a cached uh result. And that's what makes it really good for audit evidence. So, the reports push SQL to the fleet D agent and orbit over a web socket. Um scheduled queries and reports, they kind of sit in the config just waiting for an interval. And then um when we look at this, a generator, key point, is a read-only collector, right? So, everything it's doing isn't actually changing anything on the device. It's just getting a read from it. So, it just observes. It doesn't change the host. Um and this is going to set us up for the extension that we've built out. If you can see every table, RPM osquery deployment doesn't exist. So, what we'll talk about is how we've essentially built that. So, with with osquery, there's the ability to just read files, like I was saying from files table, JSONs. But, you'd have to get the fresh output. So, that's like multiple steps, maybe not ideal. So, what you can do is with osquery, just build an extension for something you're interested in. And we were really interested in um being able to get RPM OS tree info. So, it's a couple lines of go, I think like 200 lines. Probably be less if I could code properly. Um I'm a CSA, so I normally just do helping customers after. Our engineers probably a million times better, but um it just shows you the ability and how simple it is. And what we've uh done essentially with this it's 200 lines, like I said. It just looks at the RPM OS tree status and then JSON output. We get one row per deployment. Um it's built and signed through GitHub. Um and then the fleet policy can then utilize that extension. Um if it doesn't detect it, it'll run a script to actually install it on the hosts, so we can deploy it across everyone. And then after that table, it's queryable. It's in the fleet UI. Um columns include stuff like version, booted, staged, container image reference, any layered packages we can start to see as well. And uh [snorts] yeah, we'll show you around, I guess. So, let's see it running in the fleet. And I think it might be easier at this point if I swap from this view over to shared view. Let me just change that. To the entire screen. Super duper. So, first thing we'll look at is the extension. So, we'll look at our hosts. We've got a bluefin and we've got a silverblue, right? So, this one you can see is offline because it's in John's bag. This top one is just running on UTM on my device. So, what we'll we at is first of all the policy we've written. Right? Actually, no. Let's go to the report >> [snorts] >> and we'll look at this one. So, it's atomic deployment inventory RPM OS tree. And what we're doing there's a lot of red and red is what we do to help me mainly for spell checking because I spell things wrong. But, it's because these tables and columns don't exist. They're not native to fleet. It's something that we've added, so that's why they're showing red. So, we'll do a live report. We'll select it to the Linux hosts. We'll run this. Obviously, it's going to go terribly wrong cuz it's live, but we'll [snorts] we'll see. Can take about 20 to 30 seconds to get the query back from the device. >> [snorts] >> It did show my device was online, so that should be should be good. Do some elevator music. >> You know, I do this all the time because I use fleet at work, and so I'm always doing queries like this. You're always like, will it work? >> Will it, yeah, yeah. In this moment, definitely won't work, right? Um so, here we can see my device there. We can see which one is booted, right, from the value. Um we can see the SHA value for it. We can see when it deployed. We can see the layered packages, so you can see that I've layered fleet OS query onto that. So, it's on both images. And we can see which one is staged as well. So then, because we have all of this information in columns, we can now create a policy that says, "Okay, we want it to be on this version. If it's not, the policy will fail." And then, essentially, just have a little script to kick off and get them to the latest. Um I think there was some other examples that we had put in and I'll just show you this one essentially what it is. You You don't need to write these reports every time, right? Cuz SQL's hard, I guess, for most people. >> [laughter] >> Um So, what you have is you just have them here installed. So, I have a report here in my YAML. I give it a simple name, a description. This is just uh so you can kind of see that. A description of the the meaning as well. You can be as verbose as you want. And then I have an interval at the rate that it kind of runs. So, I can have that constantly happening. Sending a snapshot if I want it for logging. >> [snorts] >> And there here I just put in the actual SQL block. We need to make a change, we'll edit that, prove it, merge it, and it's in the OS. So, that was one of them that I was going to show. And I think we had some other examples that we were going to talk about which were some of the ones that we have for for example open claw. And it just shows you the power of SQL and with these tables what we can do. So, you know, I'm looking at for example, let me filter it so we can only look at what's relevant to here, Linux today. So, we can look at MDNS auto discovery. We can look at if we see some memory poisoning. We can see some world mutable secrets. And all it is super like query. Uh maybe not this one. Again, doing a join with the users. But again, these purple tables are all native fleet tables. And then with these columns, we can then start looking for anything that's open claw. Um and any other variants. Again, for memory poisoning. Slightly different one. We're looking at some of the directories, looking for any modifications to the sole file and memory files. Readable secrets. What we're looking at for is again here through the files, looking at certain directories like motd as well, file names looking at that and seeing if anything's visible in that. So, what other examples? I think that was the main bit. I'm going to swap the screen back into this extended display. And we should be good to look at a bit of this. Thank you. >> So, what's the road ahead here? Uh we've had a couple problems getting the fleet package installed on Bluefin and Silverblue. It crashed. Yes. Actually, we spent quite a few hours this morning trying to figure some things out. So, yes. Um for example, it tries to open uh opt orbit and um it was an a read-only system until uh we fixed that. Um it Uh it will try to start the file system or it will it tries to start the service as part of the package, which is not what Fedora packages are supposed to do, but it's what the fleet package does. Um so, we need to fix that, too. Uh you can't you can't start a service in a overlaid package because that's not actually running on your system yet. Um we needed to kind of point at different directories like var lib fleetd instead of /user. Um so, some of the things we fixed. We already uh it has a parameter the the orbit root there. We just need to switch it over. Um and there's an Etsy configuration file in Etsy default. Um There's obviously this uh the uh OS tree RPM OS tree uh deployments table or extension rather uh that doesn't yet exist upstream, but it lives in your Git repository right now. Um it's not that hard to add. Um And some of the things we want to do I want to also track flat packs because people install flat packs on atomic distros and it would be good to be able to track them and we need to both track system level flat packs and each user's flat packs that are installed. Um Oh, yes. Also, one of the things I was at the Boot C uh BoF yesterday and I heard a lot of things about how a lot of the a lot of these assumptions that we're making about RPM OS tree might not be the might not be the case in the future. So, we'll have to be able to handle those in the future. Um and then, you know, we have the uh the root directory will definitely need to be moved for image mode systems. >> Yep. So. Yeah. So, thank you very much. The the code is uh public. There's his Git repository as well. Um here's some information about how to contact us. Um Yeah. Does anyone have any questions? >> [applause] >> Um maybe it's like a stupid question now, but uh is anybody except uh Red Hat interested in Boot C or uh do you have other folks interested from other companies? Like there are some folks here from other companies who are super interested in having managed workstation and so on so on. I'm just wondering if more people are interested in it except Red Hat. >> I think that segues pretty well to um to uh thank you very much. To all of our kind of product is in GitHub. All of our issues uh there. And so what customers can just do is just open feature requests. Um so far I think before this talk I looked there was some questions not in our repo but in the OSQuery repo. Um so that's why I thought it would be cool to to put this together. Um but but definitely if there's if there's interest let us know. Post it and I mean feel free also to write an extension. Um we we take community uh contributions all the time. >> So thanks for the presentation. I have two questions. One uh maybe I missed something but can it distribute software? And the second one is um is there any kind of like or are you thinking about any kind of portal where users could update their upload their custom policy for the benefit of the community? >> Brilliant. Thank you. So to answer your first question um we talked about policies and what they do is automation here is just like a script but what we do for macOS, Windows, Linux, all of them um well apart from the mobiles is is we can distribute and attach the software. So a simple policy we've added something called auto patch. It's not for Linux, though. It's macOS and Windows. So, we use Homebrew for Windows ingestion and for macOS ingestion and Winget for um for Windows. And essentially, we maintain those apps. We look at when there's a new one, update the SQL query itself so that it's on the latest. And if you're not on the latest, it'll fail and automatically remediate. Um so, there's a lot of automations. That's one of them. There's also instead of just having software or a script, you can send a webhook. And if you emit the webhook with whatever orchestration system you have, you can kind of take actions that way as well. Um for your second one on queries, so we do have a page on fleets. And my links obviously don't work, right? Because that would be useful. Okay. Let's not go to pricing. That is definitely not what I wanted to do. And under solutions, ooh, more docs, sorry. We have a full list of reports that have been pre-created. And everything on our page can be just contributed to. So, if you want to add some more, share them. Some people put them in like GitHub issues for a specific workaround. But um yeah, we try to maintain that. So, thank you. Anyone else? >> Hey, thanks. Um had anyone ever looked at containerizing the Orbit or Fleet the client? Like instead of having it come as an RPM, basically. >> So, the problem whenever you're going to containerize the Orbit, it needs to still be able to reach out and look at a lot of things on the file system. It needs to be able to look at devices. It needs to be able to look at running processes. And so we really don't get a whole lot of benefit out of running it in a container. Uh the sandboxing and and namespace Uh you would you would basically be doing the the least amount of containerization in a container you could possibly do. So Um it is something that it would be nice if we could do it that kind of distribution. Uh it would be a lot easier on Atomic systems, obviously. Um but in this I mean I I I was at the at the Atomic buff and I was like maybe I should make a system D sys 6. >> [laughter] >> But uh yeah, there's I I actually wanted to go back to the previous one with this. I wanted to mention that we actually used the Chrome extension query and we used that to find um people who had installed uh malicious extensions at our business. So it's actually we used this to actually find some pretty useful stuff. So this is this is really powerful if you weren't aware of it. >> Yeah, yeah, super simple query, right? So if we just look at it, it's just looking at the Chrome extensions table. That's certainly not a table. We adjust all of that and then you can display it up. So then again, you can even report on it or automate it into a fix. >> Yeah, we actually look for specific IDs. We don't want to collect We're for you know, we don't want to look at what everyone has installed. We just look for specific IDs just to make sure that they're not a list of known malicious uh Chrome and uh Firefox extensions. >> There's there's known lists around on the web where they've got kind of the whole whole list of identifiers that you can just pop into a report and policy. And then look at it through that. >> Yeah, so Infosec comes to us and says, "Hey, does anyone have this extension installed?" And we're like, "Hmm, I don't know. Let's see." >> Um in the deployment so, and definitely relating to BootC, I think one thing architecturally, right? Like so, Windows and macOS are proprietary OSs, right? So, like you get you get forced into an architecture where the OS vendor provides the OS update service, and you can only build stuff alongside, right? Like you But, um I kind of think about BootC that's definitely different in emphasis from our PMOS tree, right? Is this concept that you can just make your own derivative OS that just derives from a base image from the uh your Linux OS vendor, right? And then so, the concretely, this creates a lot of different architectural things. Like in this specific case, like did anyone look at installing Fleet or Orbit? And this is obviously a different thing than the containerization, but just installing it in a base image, and I guess backing all the way up, let me just ask a couple questions. Maybe we'll try and track it. So, less back back and forth, but um how many use cases do you see for managed systems where you can really discourage the Linux user in particular from being root or pseudo at all? And you know, that kind of combines, right? Like if you um build in this uh like the management agent and all that stuff into your base OS, and kind of like try and get away from the mutability at least at the OS level, um you know, it can reduce the need for some of this drift query, right? Because like you know you're booted into uh into what you expect, right? >> So, um yes, I have looked at uh doing a container file that builds on top of the uh I guess the uh the uh boot C workstation base container puts uh the fleet package. Actually, this is for my job, so I also tried to get some things like Splunk working in it as well. Um, and of course, neither of them worked out of the box because they make a bunch of assumptions that this is a running system trying to run the actual processes in the post scripts. So, half time I'm just extracting the contents and then doing an RPM dash blah blah blah no scripts. Um and it works. I I I actually had that working easier than doing the layering in RPM OS tree just because in a container file, you can do whatever you want. >> [laughter] >> Um, and whenever I was thinking about this uh because my users at Red Hat are very interested in running whatever they need on their system. And they definitely will need root. Um, how will we be able to have visibility into the fact that it is running uh a container that contains the tools that it needs to. Does it have uh Does it have a layer on top of it that has malicious data? Did they use uh another source that might have had been compromised or something like that when it was layered? Um, these are all good questions and are very difficult to solve at this point because the visibility of it is hard to get to. And on top of that we have toolboxes. What if someone has a malicious uh container in their toolbox? Well, you don't see that if you're looking just at the OS. The toolboxes live in the user's home directory. Now, there are uh infosec tools that will scan containers that we can use to scan those sorts of things. So, um we don't use it, but that's something that we can use to do that. Um and you want to talk about We're good? We're good? Okay. Did we answer the question you were looking at? Okay. Last chance? Okay. Thank you very much.