Flock 2026 Extending The Reach Of Agentic AI With Customized Fedora Cloud Edition And BTRFS

Thanks everybody for being here. I'm really happy. I'm excited to see who's in the room, right? Actually, like I look around and the uh the the quantity of knowledge in this room is super exciting for me. Um so I really appreciate you being here and um I'm super excited to talk about this. And uh I know I'm going to mildly offend some people in here while I talk, but uh only only for only in uh in the sense that um I want you to think about it. This is not saying I'm not saying that I have uh all the answers just just one particular experiment or group of experiments and uh so hopefully these lead to something more interesting uh later. But I had two ideas that I wanted to uh that I was thinking about that are very specifically uh related to the work that we all do. Um one is the agentic the agentic work, right? Obviously. And then um uh well and then the other one is uh how we get these LLM models uh to the next place. Okay. Thanks. So, and then now my question is how do I get these slides to to move? Um a little bit about me, so uh when you saw those descending numbers on the on the slide and Jeff said I don't know why these are going down. Um that is the cloud sig. I'm a member of that cloud sig and I spent a lot of time on that um looking forward to talking to Michael Winters more about the the depth of that data and so we can look at what was going on there. Uh but it's really important to me. I work on a team at Amazon uh that is responsible for operating system partners. So, my job is to make it possible for uh distributions like Red Red to run especially Red Hat uh to run on top of AWS. Uh So, all the stuff that works on Amazon EC2, I spent a whole lot of time on it. I was the Red Hat I was the AWS architect who was responsible for identifying how our infrastructure would work under Rosa, if that tells you something. All right. So, the first problem that I was looking at was model distribution. This kind of happened as a result of me doing some experiments, in fact, inference experiments with Fedora to replicate things that I had heard about at Red Hat Summit. And I wanted to know more about what it was that I was actually getting across the wire. One of the things that I ran into is that models can be big. Who knew? >> [clears throat] >> Um and that led me to a space where um the first the first system that I stood up had too small of a base volume. And as a result, I had pulled down a container with a with a very large model, and that model just basically ran me out of space. I had to start over. Um re- created that, re-created a volume, and then started to think about how this is a big problem. Like, what if I was doing this for a very large number of systems, right? So, how does this scale? Uh in fact, what do I do when I start my inference practice and I am training these models on a fairly frequent basis, and I need to look at how much of that model I want to to transition. So, this is effectively what I was looking at is like how am I going to do this? Am I going to do this with rsync? Am I going to do this with S3? Is there Is there some sort of like what what is the the layer cake here that makes that uh uh granular practice um easy for me to to transition. Well, guess what? I work on a uh an addition, the cloud edition, which also includes butterfs right there in the in the volumes. And the butterfs uh file system offers a significant amount of tools that are utilities, I guess I should say that make it possible for me to share a lot of this volume uh a lot of the information that I have on a single volume. So, I looked at this from that perspective and said, "What if I just use butterfs send to actually send uh a snapshot?" And then I could look at this from uh from the perspective of how much the volume had changed. Okay? So, that sounds like and this sounds great. And I know that there are like probably a few of you who are very in very involved in AI inference and training who understand what I'm actually saying here right now uh sounds like a myth. Um but uh effectively, I wanted an atomic uh an atomic configuration, right? That would allow me to have one single system, and then this system would allow me to use the butterfs send across um the mux of my choice, right? So, the multiplexing of my choice uh onto uh an inference fleet, right? So, now I've got one system, that's my training node, it's the largest node that I have, then I have these smaller systems, they have an inference volume, that inference volume can be on uh you know, whatever the physical arrangement is, it's on one butterfs subvolume. Right? >> [cough and clears throat] >> So, I set it up. Oops. I set it up so that it would uh so that I would be able to replicate this and I did this on top of uh surprise, Amazon EC2. And and I did this using Fedora 43 to create this configuration and used a volume specifically added for the the testing. Um So, look at what I had to do just on the you know, as a as a basic snapshot, then used the well, I set it up so that it was a so that it had a file. I created the snapshot and then created a a slight modification and then did the butter FS send to look at what what happened there. Um The couple things that I ran into um uh about writable snapshots. So, just want to make sure that if you if you are um if you're doing this work, you you do have some caveats. There are some things that you need to make sure are are set uh during your configuration so that your snapshot will um will actually send. Um So, then looking at this from the perspective of what it was that I was actually expecting, um you can see that I'm looking for a result that is there in the middle. Um which is 2% of the model being shipped, right? Instead of a full send. Anything you do is going to create a glob, right? So, regardless of what your what what support you have in terms of the of the system, when you Python save, you are making a very large model, right? You are making a very large glob. And so, in this example, I'm effectively showing you a volume of different types, a full send, which is exactly what you get when you do when you build an inference or large language model with the um with the standard Python or PyTorch model methodologies. And then the rest of it effectively shows you the variation and differences in the way that that happens. So, >> [clears throat] >> um again, more of the metrics here for this. Uh the torch save, obviously, is slower than a full send. So, that means that regardless of what like the first and central argument that I'm expecting almost anyone to make in this room is the one around around uh dispensing a model by OCI container. Um so, uh you're going to run into the same problem because it's still just a glob. So, that is my elephant, right? Giant elephant in the room is it takes a long time for me to uh to send a volume unless you do some some different kinds of methodologies for for making the uh making the large language model um available. So, your your model should be either sharded or it should be uh created with using an mmap methodology. And that makes it possible for us to have a smaller send. Uh that incremental send is fully supported in that in the butter of FS subvolumes. And so, that was the the goal here, right? So, what I want is for people who are looking at this process to consider seriously consider how it is that they're building their models and look at how this process can be uh can be modified in order to make a much smaller volume set. So, here are some comparisons of the uh you know, effectively the same thing. All right? And um this this uh digital twin concept or something that I like to consider as a part of an essential part of engineering with these AI models is the rapid access from multiple systems. So, if you're looking at this from the perspective of like how am I going to get all of these to the same container ship, this is how you're going to do it. Okay. So, thank you for listening to me about Btrfs and Btrfs send. Uh that is really cool, but there's another thing that that came on to my mind in a very strict like very strong way uh as a result of some of the things that I'm I was looking around here. And that is that we spend a lot of time putting agents on one user and putting one MCP model together on this. And this came from me having a discussion with uh Michelle Send. Uh Celine, how does he say his name now? >> Michelle Landers. >> Michelle Land. Thank you. Uh so um Michelle and I were having a conversation around the AWS uh CLI. The CLI is like no, so as you may know, the CLI is a part of what we ship um and and for the both Fedora and Rail. And uh and the uh the conversation we were having was around an a beta uh option for the CLI. And I and Michelle asked me, he said, he said, "So, how do I get this?" And I said, "Oh, you just put it, you know, you'll just we'll just use this services.json file. It'll extend that and it'll be no problem." And he said, "Can I just put that in Etsy?" And And "No. You cannot. You'll have to modify it for every user on the system." And uh and I thought that's that's terrible. Like, we should fix that, [laughter] right? So, I filed a bug on it. Um and where it went from there, I'm not really sure, but but the but the SDK team heard me out. It was it was a pleasant conversation. And uh and then I thought about it and I thought, we do the same thing with agents and same thing with MCP services. So, what are we talking about? Why are we not just doing this as a service account user? Why don't we have the same problems with uh an AI agent that we have with any system user, right? Why can't Why can't all of its information and skills live in a directory service? Why can't I have all of this? And a lot of these things kind of came together for me um in in terms of uh Kubernetes and where we're orchestrating all these things to run, they're all doing this in in the concept, you know, with these sort of core concepts of overlays, right? I uh And the thing that led me here was a conversation about a Raspberry Pi and how to deploy CoreOS on a Raspberry Pi, which was and they were the question was, well, how do I get this user You know, how do you get this user configuration onto this Raspberry Pi? And of course, the the response was uh well, you could use ignition or you could just build an overlay and just drop the overlay on there and it kind of it kind of took my breath away, right? Just thinking about it from the perspective of like, my overlay was just as important or just more important than my than my actual ignition file. So, um this led me to an understanding. So, CloudInit doesn't have ignition, right? But, we do have I'm sorry. The cloud edition does not have ignition. We have CloudInit. And so, then I started to realize that I can do all of this through EC2 user data. I can create a cloud config, use that in the in the context of uh declarative cloud like a declarative cloud cloud config.yaml. And then do the same thing with container files or I can scare Alexandra with a helm chart. Um And then just create effectively agents. So having agents on the system that build this out. And then looking at that from the perspective of the user the user has its own MCP configure the agent has its own MCP configuration. Why am I deploying these MCP servers as a multiple times with multiple agents? Like this seems totally ridiculous to me. So I believe that we should probably look at this from the perspective of having you know using the same system the same same sockets that we have available here using the same group configuration that we have for for this. So here's kind of an example of what I'm looking at from the perspective of cloud in it that we need an agent provisioner some way to do this that gives us an agent configuration and an agent shell. Um that we would do this you know we can configure a lot of the connection and access to the MCP services through systemd with the life cycle orchestrator. Sorry, I'll go back so you can continue that he's taking a picture. Um and then we can do the life cycle. So then we end up with an opportunity to create full configuration that is deployed as a root user but then gives us the ability to drop privileges like we do in Open SSH. Um so that anyone who like not every agent user has access to this. Um And so, effectively, you have an Etsy agent config, and that Etsy agent config gives you what you need. Uh on top of this, uh you you need to have a short-term and long-term memory. I'm not a fan of providing that over to Open AI or Anthropic or any other AI uh group that uh is um is willing to take it on for me. Um but I am willing to look at this from the perspective there's a I stole shamelessly stole this from uh uh a um uh a researcher at the University of Texas at Austin. Uh this is a uh a configuration called Memory's DB, and the whole concept here is uh sometimes referred to as like an open brain. Um and uh and that is effectively looking at it from the perspective of like where does this stuff live? Well, I've already told you that I can send these Butter FS volumes. So, short-term memory, Butter FS subvolume easily translate translates into an engineering practice that produces a support across multiple systems, and makes it very easy to uh provide immediate access regardless of the architecture that you're using. Like, whichever whatever your public infrastructure or private infrastructure, you still have that that uh ability. So, then the agents are the cattle, and the memory is pets, right? Um I'm going to push through this because I don't have enough time, but this is effectively what I'm looking at from as part of what needs to be something that we can uh like a like an addressable configuration space that we can put together either through directory services or through Butter FS sand. So, just one primitive for everything. Um Um I'm going to say one word about this. Like, I'm going to let this slide be up here, but I'm going to say like, we've been working on users for 50 years. And why are we just forsaking 50 years of technology and then just saying that this all belongs in a container, which is not, in fact, any more solid of a uh a security boundary uh as the user itself. So, multi-host coordination, horizontal scaling, cross-host identity, the concepts around secret rotation, all of this is here. So, this is where it lives. And I'll let it be there. I've got about 3 minutes for questions, I'd say. >> [laughter] >> Five. >> [applause] >> Yes. >> Hey, [clears throat] um this is this is very impressive. Um I just wanted to say that the agent coordination, gossip, and registry that may be solved by A2A >> Uh-huh. >> at least in part. Like, it's like it's supposed to be the universal, you know, protocol for this sort of thing. >> you get the OTA off on immediately. It's it's a It's >> Yeah, it seems like it could work maybe with some extension. Um another thing is that um one of the reasons I think that we don't that often see agents or hear about agents as users, even though it makes complete sense, is that the most common use case for software agents these days is when people launch them in their terminal under their own user to edit their own code live or to perform actions as them, which may be a bit unfortunate, but it's I think it kind of explains this sort of weird like gap uh we're seeing. Because yes, it makes sense to have them as users in many different ways. >> Yeah, I I mean I see that and I but I also see a separation of duties. And so in a in a strange sense like I can regardless of what the agent is, that agent is going to spawn sub-agents, right? Like that's [clears throat] that that seems reasonable to me. But then to isolate and separate like specific workloads, specific tasks, task-oriented uh structures, that kind of makes sense to me. And I do think that like uh the data engineer is being mildly held captive by uh the data researchers, you know, the data scientist models, right? So So the research and and scientific models I think are are uh in a strange way leaking into the way that we expect to do large-scale large-scale um uh deployments. >> Okay, thank you. I have getting back to the BTRFS. So we are sending over network some delta and something breaks. Network is this it can break. How how much I I understand it's instant cuz previous copy on write part we have, but how manual how manual manual interruptions uh they are needed if something breaks during the upload of the layer. >> So I mean obviously that is a an implementation detail that I have not worked out in this in this in the context of this this this discussion. But I do think that that's a that's a very important question. The BTRFS subvolumes themselves have the the checksums that are required to make to make a decision on whether or not I have the whole volume. So, it's not It's not too It wouldn't be too surprising to say if you didn't get it the first time, ask for it again. Right? >> So, re- re-transfer is possible and finally it will work. >> Yeah. And then to have a sync Yeah, synchronization at that point. But, I mean obviously the size of the volume and in this case I was thinking about it considering from the perspective of a digital twin where you know, I can say shadow push this to a shadow device and then that shadow device is the is the end result. Um how I get to this point. And maybe I'm not answering your question and I should probably ask ask that. >> I was asking [clears throat] about resumable sends. >> Yeah, I I mean resumable sends would would be something that would need to be uh would would need to be uh engineered. That is That is not >> And remount is is not necessary. So, I do don't need to remount the file system when it's it breaks. >> For For ButterFS, no. >> Okay. >> Yeah. >> So, for what it's worth, resumable sends aren't yet a thing. >> Yeah. >> So, for what it's worth, resumable sends aren't yet a thing, but I think this is mostly because nobody has asked for it. Um at this level of usage, it's probably worth maybe David you might want to just ask the ButterFS people upstream about, "Hey, could you add resumable sends to this? Because like it with this kind of data throughput, you if an interruption is pretty costly if you have to start over. So, >> It It is something that I have thought about and and uh it just happens to be separate. Right now, it happens to be separate from the from the >> There are ways to manually do resumption, but like having it just built into ButterFS on receive to do it automatically would be >> It would be awesome. >> Yeah. >> Yeah. It's It's actually in my slides as something that needs to be done. >> Okay. There you go. >> I glossed over it. Yeah. >> Um but I also had a uh uh a question / note for you. Like you So you mentioned no checksum in the stand, but like as you just said now, like checksum is built into the >> It's built into the volume. Yeah. >> So did you just mean that like it was just no extra checksum in the file load? >> Yeah, that's right. Yeah, yeah. >> Cuz it your slide read a little confusing. >> I'm only going to I'm only going to find out about it when I try to restore it. Yeah. >> Okay, so any other question? I don't see any, so thanks. >> Cool. Thank you. Thanks for being here. >> [applause]