Flock 2025 Ramalama

Okay, >> introducing remarkable paper. >> Now we're going to run an ad. So, how many people have ever heard that song before? No, that's the wrong one. Anybody ever hear that song before? All right, I can walk away. That's right. I marked. Uh, no one in this room's ever heard that song before. We don't have enough old people here. That's the problem. That was a song from the 1950s u for uh basically open-source projects sort of adopting the name Llama. Llama comes from Meta uh developed llama 4. They have sort of an open uh AI model and you'll see lots of lots of projects are using the name Llama. So um about a year ago now a guy named Eric Curten came to me and he had tried to uh open up some pull requests with Olama. Oama is probably the most popular uh command line tool for running AI models locally on your laptops. and he had tried to open up a bunch of pull requests for them and they basically ignored them or rejected all pull requests and he went out and looked at a lot of stuff and they basically were just rejecting. So they're they're a quote quote open tool um except they don't accept community contributions. So they're not really an open source and and they've started to do some stuff that is in my opinion is is anti-open source. Um, so Eric investigated what Olama was doing and they were using a very open project called Llama CPP. We'll be talking about that during the presentation. Llama CPP actually does all the heavy lifting of making the AI up model actually work on the systems. Um, so Eric quickly wrote a bunch of bash scripts to uh basically fire up Llama CPP, pull down models, curl down models from different websites and things. uh he built a bash script and uh Eric worked I I was I used to work for Rivos where the redhead and vehicle operating system the um basically edge computing um and Eric also worked in that group so that's how I knew Eric and he came to me and showed me the bash grips I was moving into redhead AI at the time uh and I said that's really cool I said but you know we may open up a project on this it can't be written in bash so we act I said we have to write it Python um just so that it's easy for contributors to come in and contribute. Um and he called it he was calling it Podman LLM at the time. I said, "Well, first of all, we can't use the name Podman because the the upstream Podman people hate it when people, you know, abuse their name." And since I've been railing against the name Docker being used everywhere, I didn't want Podman to become a new adjective. Um so, uh I said, "Well, everybody names things after Llama." And I knew the song Ramal Lama Ding-Dong. So we called it Ramal. That's the history. Okay. Um so let's talk about uh uh we're going to talk about Ramal Lama, but we're going to be talking a lot about running AI AI models uh locally. Um so uh the interesting thing here is that it's really really easy to run an AI model on your local system on your laptops. And I think that's critical especially in open source development and as you know in the future very soon in the future there's going to be lots and lots of reasons to running AI models locally and we're going to cover a lot of that here but as we get to edge devices and things like that um so there are a lot of tools out there right now for running AI uh AI locally on your system um and these are a few of them is a one that Red Hat's heavily invested in There's Oama is the um the one next to it. You see Rama Lama. We're going to be talking a little bit about Podman Desktop. Um Podman Desktop has a a plugin called uh a AI lab recipes and that's why you see the the Podman robot up there. So, it's the sale as a robot. Um and then there's a bunch of web UI type projects that also uh run AI models locally. Um, where do you get AI model? Where do you get open AI model? You know, any AI model that you want to download? Um, hugging face is probably the most popular place to get them right now. Almost everybody that builds an AI model, the first thing to do is get it into HuggingFace. Um, and uh, Olama also has a website where lots and lots of people get their uh, AI models. And I think this is a big problem going forward and I'll cover that during the presentation. Um the last thing we'll be talking a little bit about one of the goals when you work with an AI model is actually to build an application that interacts with the AI model. Right? How many people in this room are data scientists? Zero. Okay. I know nothing about AI or at least I I feel like um you know the the emperor with no clothes when I talk about AI. What I do know is about running software and AI is just in my opinion is just another form of software. So as developers want to interact with AI AI provides APIs, REST APIs that you can build. So if you want to build your own chatbot, say you have an AI model that's been trained on your data, uh you want to build a web interface that basically has a enter a human query here and then that's the type of thing that developers are going to be developing and we'll talk a little bit about that. Um, so why would why is it important to run uh AI models locally? Um, so I'm going to look first at the developers point of view and then we're going to look at your organizations, the companies that you work for. Why would they would care? Um, so the first one is convenience for a developer is convenience in right I can just run the thing locally, right? You hear about AI models suddenly showing up um, you know, online podcast whatever, right? The day that uh Deep Seek came out um actually Ola where um Ramal Lama was pulling Deep Seek from and Ola were pulling it. The service melted because so many people wanted to go out and say what what is all this information about deep you know all a sudden Deepseek it's all over the press everybody wants to try it out. Um so just being able to run it locally not have to go and pay someone to to interact with a model or even go to a website. I just wanted to pull it down locally. The next one is ease of integration. So if I'm going to be building an app, it'd be really convenient to have, you know, the AI model running locally. So I can just build my app to to integrate with it. I can figure out how I would use this AI model and I can easily swap out AI models. Which one's better? Which one works better for my solution? And lastly, customization and control. These are all sort of, you know, I can control that AI model. I can, you know, put wrappers around. I can make sure that it doesn't do any bad. Um, so from my point of view, you know, again, I like to do things on my laptop, so it makes a lot of sense. For the organization's point of view, there's things like data privacy. Okay? So, a lot of companies, you know, you interact with an AI model, you ask it questions, and it it knows nothing. You know, you quickly figure out that it's stupid, right? It doesn't know anything about internal, you know, you ask it a question about inside your company and obviously hasn't been trained on that data. or if it has been trained in the data, you got a problem because you know your data is being leaked outside the company. Um uh so you but do you want to take your private information inside your company and upload it to a cloud AI? I don't think you know most companies right now are very apprehensive about doing that. Um then you have cost control. Anybody that's used a cloud AI for a serious business quickly figures out that you're going to get a huge bill at the end of the month. So AI online is very very expensive. Um some I was at a different the reason I missed this talk yesterday. I was at another conference yesterday and I was on a panel where they were talking about um AI and and basically the costs. But yeah, the way I look at cost of AI tends to be, you know, like buying a car versus leasing a car. um you know, you can get probably newer stuff quicker if you're leasing, but you're going to pay a lot more money for it. And I think the cost right now of using AI online is just prohibitively expensive. Um so just something to be warned about. Um and lastly is regulatory compliance. So a lot there's lots of laws. There's lots of countries that don't want their data, you know, being used on computers that are outside of their country. there's uh at least in the US and I'm sure there's other countries and I know Europe has lots of control uh data controls uh things like in the US there's a thing called HIPPO which basically means my medical records can't um you know go out to the cloud you know so that they can't be shared um so there's lots and lots of regulatory rules about how your data but you really need your data and in order to make these AIs useful you know you got to give them access to data so there's a there's a conundrum but if you run everything in house then you don't have those issues. So Rama Lama we talked about uh where it comes from. Basically it's a command line tool for running AI locally. Um but um the container I didn't introduce myself but hopefully a lot of you know who I am. I'm the guy that created Podman. Been working at container technologies. I say long before Docker existed. Um so I I have a hammer. AI looks like a nail. I want to I want to stick it into a container. Okay. And there's lots and lots of reasons for that. But based fundamentally, Ramalam is a wrapper around container engines for running AI models inside of containers. Um, and running it locally on your laptop. When Deep Sea happened back in February, I wrote an article talking about Ramal at the time and the importance of running AI models inside of containers. Um, from a security point of view, you are taking a massive unknown object, I like to think of it as just as a big database that is going to influence a piece of software that you may or may not trust. And that AI model is going to get to that software to do stuff on your system. You don't know what it's going to do. Matter of fact, data scientists don't know what it's going to do. And it could be it could influence the software to go out and do random things on your desk. So why not stick that in a container, shut down its network and just give it the data you wanted to interact with. You don't want it interacting with your SSH keys in your home directory. You don't want to run it as root on your system. So there's lots and lots of, you know, everybody's running into AI. It's like, oh, this is cool. I'm bringing in this engine that no one knows what it's going to do and running my system. Don't let me get into MCPs, which is basically saying, I'm gonna give this AI that I have no idea what it's going to do, access to not only everything on my system, but everything in the world, you know, and we've had movies about that. Um, [laughter] so Ramlama fundamentally is built on container engines, uses containers. Um, it um it works with both Docker and Podman. What I will hesitate, we actually opened up discussions back in the fall with Docker um to try to get them to participate in the Ramal Lama um and we met with them two or three times. Um I'm not their favorite person, but um actually most of the guys that I you went against Docker in the beginning have left Docker at this point. So they they worked with us fairly well but of course they went their own way and Docker introduced uh something I think around beginning of the year to run AI models locally. Funny thing is Docker is now running their models inside of containers and they're doing that for performance and I'll talk a little bit about that later. But Podman is a lot more than just Docker like we used to say if you want to run move from Docker to Podman you do alias do Podman but Podman has added lots and lots of features. So more of the advanced features of Ramal Lama now require things like pods and different functionality that only podman has. So you can do the basic functionality with docker but if you want to get into the advanced functionality of ramalami you need to use podman. Uh ramalama also will run without containers altogether and some people do that on max right now because you can get a little bit better performance. Um, so when you run ramalama, the first thing it does is it analyzes your system and says, "What kind of GPU do you have access to on your system? Is it a cooter? Is it a Mac? Is it a AMD system?" And then goes out and the the weird thing about the AI world is all the software has to be custom compiled for your GPU. It doesn't really work well in a distribution mode and that you know the co software cannot adjust itself. it has to be recompiled for each individual GPU. So in the container world, what we did is we created a one that's compiled for CUDA, one that's compiled for Intel GPUs, one that's compiled for Mac GPUs, uh, and for CPU. Okay? And what I'm going to be demonstrating today is a CPU. So a lot of you people sitting out there saying, "My Mac doesn't have a GPU that can be used." You can use it with CPU. It looks a little like a ticotate going across, but it actually works well enough if you're just developing AI, developing software to interact with the AI, it works well enough that it's not horrendous. Um, so you can start to play with it. Uh, so when Ramalama discovers which GPU you have on it, it goes out to a container registry qu.io IOR ramalama and pulls one of these images down for your system uh with all the software pre-ompiled for u that specific GPU. Then it goes out and pulls down a image a model to run the system and then finally launches a container with a model mounted into the container to run it locally. One of the things I really want I've been fighting for with this tooling is that we need to make sure that AI models AI content all works as container images. I don't I don't believe customers are going to have hugging face or lama inside of their private networks. the customers most users in large enterprises right now have bought into OCI which container registries so I believe models have to be stored in container registry people going to want to control which models you're going to run inside of your company if you know for Red Hat's point of view if Red Hat goes in and sells some of our solutions to it and we have to come in with Olam or a hugging face to install their enterprise version of their web services that store AI models, that's a big hindrance to to sales. If you go in and say, "Oh, you already have Artifactory, you already have Docker Hub, you already have, you know, one of the 8,000 different types of OCI content." We want to be able to store it. So, one of the things Rama Lama can do is convert models from hugging face or O Lama into OCI content. So, we want to be able to push that and then be able to use that. that also makes it available for Kubernetes. So when you want to run an AI model inside of Kubernetes right now most people are just using S3 files. Well, if Kubernetes doesn't know is anybody in this system using an S3 file or they're pulling from you know a lot of people come up and say I'm going to put OAM inside of a container and then run a Kubernetes. So you fire up OAM it's going out and randomly pulling down a model from somewhere on the internet and again Kubernetes doesn't know about it. If I have two containers pulling down the same model, can I share it? No. So, there's lots and lots of issues uh around running it. I like to tell people Ramal Lama should never run in production. It's a developer tool. It develops containers to be able to run in production that use the AI models. So, we talked about GPU acceleration in the previous slide, but this just shows you some of the uh uh different platforms. Uh, Macs are tremendous and I'm the anti-Mac guy here. Uh, Macs have tremendous GPUs on them now running locally. So, if you have an M4 Mac running uh with Podman machine, um, it just it really is a nice experience. And at Devcon next week, my Eric Kurt, the guy that kicked off this project, will be bringing his Mac. I I'm not allowed to touch a Mac contractually. Uh but um anyways uh AMD has some really nice GPUs, Nvidia, Intel, Vulcan. Um these the tools run on top of Windows, but fundamentally containers mean Linux. Okay, so we're running VMs on top of Windows on Macs to do it. Max can actually run the run the AI models locally with d-n no container if you want but um and then obviously on top of Linux you can run this as well and CPUs are always also supported so one of the interesting things uh has anybody ever gone out to hugging face and looked for a model yeah and what did you find >> you find 20 you find 20 you know if you look for deep sea you look for granite or you look for anything, you're going to find 20 to 50 different models of it. Yeah. And so the the problem is, you know, just like if you go back to when Docker first started, Docker came up with a great concept, which is I I call short names. And basically, if you said, I in Docker, I want to run engine X, you just say Docker run engine X. You didn't go out to docker hub and do a search for engine X. You just ran it. So with Oama stole the idea of short names from them and when Oama came out they just did Oama run peep seek or a lama run um whatever grid it um so Ramal Lama built on that and what Ramal Lama has is is the concept of short names guess what it is it's a file name name value appears you say deepseek we think you want this one you can specify the full URLs to pull down um you can specy speify anything else you can customize short names to do it. It's a simple, you know, name value pier for that. We also fall back when you're pulling from. But the cool thing here is now we have open source community coming in and people open up poll requests to shortnames.com inside of Ramlama and say, "Hey, Gemma 3 just came out. Why don't you pull down this model?" and and again for normal humans non-data scientists who don't know what the difference in quantization and all those different you know 70b 30b 50b um you know so that's the goal I would like to see eventually ramal lama grow smarts about we know the GPU we know the amount of memory you have in it'd be really nice for short names to say uh instead of pulling down that deepsek let me you have a lot more memory so I can pull down the bigger deepseek and and and and start to do that that type of thing. But I need people smarter than me to come in and give give me a database that says, you know, for this amount of memory and this amount of this, you probably pull down a quantized version of this with, you know, four bit. I don't know. I I don't even understand what they're talking about when I talk about I'm too old to learn. Um, so anyways, that's that's the concept of short names. Um, and I don't know what this performance made easily. Let's just say I'll skip that right now. Uh so we talked earlier about running everything in a container. Um so basically you you inject into the container the model the models should not be installed inside of containers. So you don't want to tie the model to the source code that runs the model. They should be volume mounted in another thing. I'm standing on my high horse. Models are nothing like just databases. You wouldn't install a database with Postourress SQL in the same container, right? You would keep them separate. I believe the same thing about models. We're going to be talking about rag data and data that you want the model to be able to interpret. So again, you'd stick that in the model. Keep everything else on your host system isolated from the model or imagine in a Kubernetes world where you have hundreds of these processes running and stuff like that. You definitely want your models containerized so they can't interact with the rest of the system. Um, uh, Ramlama out of the box was built to be open source, right? We want to be open. So, we basically say wherever your model is stored, we will pull it from it. So, Ramal Lama will pull from OAM, pull from HuggyFace, it'll pull from OCI, it'll pull from HTTP website, files on disk. we will pull from from anywhere um to the system. Right now we only push to OCI. Okay. Again, we want we want people to think about getting it out of wherever the storage locally, the ones you want to use and then putting it into OCI when you want to push it. Uh we also support any inference engines. Okay, if you think about way Docker and Podman are created, they're we I call them container engines and there's a container runtime run C run live k run um kada there's lots and lots of different run times. So the in in in the container world the engine's the thing that pulls down the software you're going to run and then you can switch out the runtimes underneath. So in Ramal Lama in the AI world the runtimes are things called like llama CPP which is probably the most popular for low-end devices VLM which is the most popular for high-end devices. Okay. >> Isn't it great when AI is listening to you and breaks it? >> I don't know how that I don't even know why that does that. Okay. >> Uh Google off. This was not planned. >> This suggests a component responsible for application life cycle management. It would handle tasks like discovery, finding a >> comp [sighs and gasps] when when you let the AIs out of containers. Um, [laughter] [applause] so anyways, so the basic idea is that Ramal Lama you can swap out the runtimes. So there's Llama CPP, which is probably the the primary one that that's the default one that we use. Whisper CPP is a recompiled version of Llama CPP that uh does things like voice to songs to text. I don't know. I listen to a song, I have no idea what they're saying. you can actually pipe through the song and it'll print out the actual uh uh what's the word for >> lyrics that's the word um and uh so the bottom line is going forward I think we also support open veno lots and lots of people are coming to us and saying you know I uh you know we have different container engines uh and the interesting thing is because we're going to use container images if you want to build a container image for your specific um uh runtime then we can you know just take one of our existing images add your software to it build your own and and do it. Um, we're also fully open. Again, we have lots and lots of people coming to us, pull requests. Sometimes I think we're more popular with contributors than we are with actual users, which um, so our our contributors uh, we're, you know, we've been around for basically since, let's say, September, and we're approaching 100 contributors because everybody that's contributing has their own special project, you know, their own special GPUs, their own special tools, or their own special runtimes. Um when you fundamentally ramama has two commands ramalama run which gets you a chatbot interaction with the service and then we have ramal lama serve. So when you run ramal lama serve the the default uh in the AI world right now is to run a rest API based on open AI. Okay. So all the tools VLM llama CPP everything I showed in the previous thing are always providing a rest API that's based on open AI. So the bottom line is if you build software that can interact with open AI same software be able to interact with locally. So Ramalama serve will by default right well by default will always provide an open AI rest API. Um we just got in this last week although it's broken. Uh, by the way, we release every Monday. So, this thing is moving as fast as as humanly possible. Um, new images, everything. Uh, we just merged in a thing called Llama Stack. I'm going to cover that a little bit later, but Llama Stack is developed by Meta. Um, IBM and Red Hat are both contributing heavily to Llama Stack. And what Llama Stack does is it's it builds a pipeline to communicate. So most people interact with Llama CPP or VLM and that's running the model. But now you want to connect rag databases, you know, your own data to it. That's a different service. So you need to wire those together. When these things called MCPs, which is different services running on on computers, you need to wire that together. Llama stack is the tool right now that we're buying into is the tool for for connecting all these different services together. Um just basically lamstack provides something that the a way for the human to interact ask the AI something a question and tell the AI that it can get some data over here or it can interact with this service over here. Again I talked earlier about Ramalan not being used in production. Ramal is a tool for building these using these containers pulling these containers pulling these models and wiring it all together. What ramal armor also does is it allow generates either kubernetes yaml for running your application in production with the AI model and all the software or generates a quadlet. Anybody here know what a quadlet is? Okay, about 50% maybe more. Quadlets is the way podman runs containers underneath systemd. So quadlets look like a systemd unit file that you can run basically containerized services underneath it. Um there's also in in the uh Kubernetes world now there's a thing called Kerve. So we're experimenting with that but Kerve right now is not in my opinion is not really ready for it's built on older versions of of uh AI not necessarily LLM and it's kind of tied to BLM at this point. So I'd like to I I go back and forth with the K serve people, but basically the idea is that we will look at what you're running and we will generate you a template that you can then customize to run stuff in production. Building your own AI. So again the the goal here since most of you guys aren't data scientists or nobody in here is a data scientist not necessarily you're not going to be building AI models necessarily, but what you are going to build is applications that need to interact with AI models. So you might want to customize and build code. So Podman desktop has a thing called AI lab recipes and these are different tools and different example programs and I think they have them in Python, they have them in um uh J uh JavaScript and a few other uh language and again it's it's contributions and this is how you build a chatbot that interacts with an AI model. this is how you de develop a tool to take uh s you know PDFs and have them go through or this is how you take a document and upload it and have a summarizer so they have lots and lots of examples so um the there's a longer version of this talk that I'm not going to do um that talks about podman desktop and shows all those podman desktop and AI podman desktop's AI lab recipes and ramal lama are slowly merging together okay or ramala is become becoming the fun functional underneath of what podman desktop AI lab recipes is. So we have an AI engine. We have the ability to pull the model. What about your data? Again, when you interact with an AI, if you ask it about um something that's internal to your company or private to you, um it doesn't know. It gives you bad answers. Um, so in the AI world, the way this is handled now is called retrieval augmented generation. Okay? And I always have to read that because I can never remember what the hell, but everybody in the AI world calls it rag. So what rag is is basically taking your PDFs, taking your documents, compiling it up into a format that AI can interact with. It basically builds JSON and then recompiles it into the same types of matrix mathematics matatics that has worked you. So basically going to take your documents and somehow get it into it. So Rama Lama has a command called Rama Lama rag and what Ramal Lama rag will do is take your your documents PDFs and markdown languages HTTP pages u text files and we'll recompile it into it. It's taking advantage of an awesome tool from IBM called Dockling. So, Dockling is running inside of a container and actually will take and look at your PDFs. I think it says doc yeah doc x. So, uh windows documents uh pp I don't just about anything and then they convert it into some kind of format that can be interpreted by the AI. So, what ramalama rag does is takes those documents and outputs an OCI image. Eventually these will be OCI artifacts. Then you can simply use ramal lama- rag serve- rag and it'll pull down your AI data I mean your your compiled rag data and hook it up with the with the model. So in this case we're basically saying take local PDF take a markdown file from a website compile it through docklink output OCI registry I can push that to an OCI registry and now I can pull you know Ramlama will pull it from the OCI registry run it locally and again when we generate quadlets of Kubernetes YAML we basically encode that into Kubernetes YAML on how it would interact. So you can imagine that you could use tooling like this to build um a online chain where you're constantly updating it with new information about going on inside of your company, push it to your private registry and then have lots and lots of services running um and automatically pulling it down. The interesting thing be by using OCI for storing models and using it for storing rag data basically allows you to do things like scanning birds boom birds of bills of materials um allows you to version it okay so I can have my rag database constantly updated I might want to lock in at a specific version right so all these type of tooling if you don't do that you're basically putting this stuff in files somewhere and trying to handle it but if we just concentrate on doing everything OCI, we get to take advantage of everything that OCI provides. So, Agentic, I always want to put up the Arnold uh Terminator. So, this is Skynet and this is probably the most popular thing in AI world right now. It's called you'll hear things like agentic or agents. Uh the protocol is called MCP, model context protocol. So they've defined a protocol for it that that you can hand to an AI. You can think say say you could say something like list all of the pods I have running inside my cluster and open shift and kubernetes right now are building MCP servers. So the AI will figure that out will talk to the rest API provided by an MCP server actually talk you know use your credentials to talk to something like open shift and basically get back that type of thing you don't have to do coup uh kist all this stuff that you're seeing about developers um using AI to build code and interact you know take a look at my code things like that that's all MCP so this is basically taking something that is no one understands handing it to a and then allowing it to interact with the file system or allowing it to interact with websites or allowing it to you know so that's what MCP is all about um and you know it's there's a guy named Michael Clifford who is a he's a real AI engineer for working for Red Hat he actually understands what things like quantization things like that mean and I talked to him back in January and I said he was working on uh aentic in MCP and he said, "Oh, this is all crap." He said, "It's just just really bad." And I saw him about three weeks ago at Red Hat Summit and I said, "You said this was all bad. Now it's everywhere." And he said, "It's been unbelievable how much has improved in four months." So, it's just this stuff is coming so fast that I frankly it's it's a little bit scary. It's incredibly impressive, but you know, who knows? So, we talked a little bit about Llama Stack. Um, so this is uh, by the way, Rama Lama's logo. It's a llama dressed up like Fonza. A very American thing, but the Fonza was from the 1950s based on the song. So that's the that's with the think of a 1950s tough guy motorcyclist. Um, and Podman Desktop with AI lab recipes. They both are moving towards Llama. I'm still not sure if that's really what Llama Stack's logo is, but it's so cute. Llama llama stack llama sitting on a stack of pancakes. Um so uh llama stack is going to be integrated into podman AI lab recipes. I said it just went in this past week and it's totally broken. Um but eventually when you run rama llama serve we will launch a llama. Right now you can do a d-api llama stack and it will launch basically two containers. It will launch a pod. one container is running llama stack, one's running the uh engine and then we wire it all together. The next step is to add rag database, wire that together. Then you're going to add MCPS. So we will build you a whole set set of containers um that basically have you know you'll talk to lamstack um and lamstack will reroute your data to all the different services that you're going to be plugging into it. And um so Llama Stack basically helps you build applications fast. This is sort of their sales pitch. But bottom line, just think about how would you wire together something that's serving a rag database, something that's serving an MCP and something that's serving a data. And that's what really what Llama stack's all about. If you come to Devcon next week, we will be showing a thing called multimodal. You will hear the terms multimod model and multimodal. Multimodal means using one model with multiple different ways of interacting with it. So I talked about you know sound sound to text. Um we have a demo that I won't be showing mainly because I don't have it set up and it really works well on a Mac that takes video stream and hands it to an AI. So it'll take a video stream and then you can ask questions to the video stream. So it's basically interpreting the video stream and interpreting the questions. That means you're interacting with the AI model in two different ways. Multimodal interacting it and it works incredibly well. And if you get to see the demo at some point, it's you know we did it at Red Hat Summit. Um and it was really really cool. The problem we were asking the AI model what it sees and I walked in front of it one time and it said I see an elderly man walking in front of it. So I'm demanding that they fix the the sensitivity level to not call me an elderly gentleman. Um but uh we also you could have go the other question we were asking is what super what um superstar or whatever what star does this person look like? What celebrity does this person look like? and it got stuck on like uh Tom Hanks uh Justin Bieber and women were walking in front of it and I think depending on your skin color they would say Vanessa Williams and uh you know these but it was you know it's it's really interesting and this was running locally on a Mac with a very small model and it did an incredibly good job of of doing it. So you can imagine this getting better and better. Um, and really what you want to do with multimodal is things like, you know, it' be really nice to have a camera that's watching the street and tells you if the FedEx guy is walking up to the door with a package or if a guy with a gun is walking up to the house, right? So, you you probably want to interact with those two pieces of information differently. multimodal also you know that again thinking about this video stream it makes you realize that you really want some edge deployments of AI you would not want to have 10,000 video streams coming back to a centralized core system for the AI to interpret and then the message get back right that video stream has to be interpreted locally so um you know I believe that edge deployments of AI are going to become more and more critical. Eventually, we're going to drive up to a McDonald's in any language in the world and say, "Give me the two cheeseburgers and the fries." And the human is not going to be interpreting what I've said. It's going to be an AI model. So, those AI models again not going to be running in central headquarters. They're going to be running inside of the McDonald's. So, running AI models locally and fully functional for specific tasks is is going to be a huge thing in the future. Anybody want to see a demo? Okay. So, is this uh so Ramal Lama version uh we just released this past week. Uh this is a live demo, so everybody don't use the internet because they're going to be pulling models. Um and it looks good up there. Okay. Uh so, Ramlma version, we've just released 0.9. [snorts] Uh we're releasing on a weekly basis. It's not a 1.0 O product that's for sure. Um but um you know it's usable now for people playing around and developing software. Um but you know there are some instability issues once in a while. So I'm running it locally. Ramal lama info. This is mainly something we ask you for when you report an issue. We'll ask you for Ramal Lama info. And this does things like what container engine you're running. So I'm running podman um underneath the covers. and it just takes the podman info, combines it with ramal info or docker info and combines it all up. So it gives us some information, but you could also look at what it's interpreting. This will actually show you what GPUs you're using. You know, what Ramlama is seeing is the GPUs that you're using on your system. So there's there just decent amount of information here um that you can interpret. Uh [sighs] so what I'm going to do right now is uh this is a a small model. Now small model when I say a small model it's you know only 135 megabytes. Uh but models can get really really huge. Um, and so, um, this is actually pulling down. So, I'm doing ramal lama pull. If I did a ramal lama run, it does a ramal lama pull behind the scenes. And so, I removed the model and now it's pulling it down and it pulled it down because we have a decent internet here and a few seconds. Uh, but if you're pulling, you know, a llama CPP 7 gig 70 gigabytes, it's going to take a while. itself. Um, now I'm going to show you in the local storage. Okay. So, it just basically shows that it pulled down the model and it tells you that was pulled from a lama. Um, and if you pulled it from hugging face, it would show show where it was pulled for. So, um, and if you pulled it from OCI, it would also show that. And these are all the images that are currently built. This my machine is where we're currently building these. We want to get this automated at some point. Um, and we're going to work with Conflux to get all these different images. Um, but these are all the different images that have been pulled down. But basically, it's just doing a podman images to show you the images that are on the system. Um, let me shrink this a little bit. That better? Yeah, you still see it. So here you can see uh Ramalama has a command called dry run and what dry run does is basically say don't do the operation but show me the command that you're going to execute to run the run the command. So you can see here that we're using podman uh for the engine to do our containers. Now I'm going to do a different dry run and this just blew up. So it should have showed you that I was using ramalama ramala. I don't know why that didn't work. Oh, that's a bug. My I don't have the latest code. Um, so should we don't use the latest image. We use 090. Um, so that should have showed you, but believe it or not, it's there. Um, and again, when we run containers, I'm just showing you different parts of the Ramalama command to get you focused on it. Here you can see that we're from a security point of view, we're turning off all privileges. So, we'll make sure that the container is locked down as tightly as possible. And then the last one I'm going to show you is when you do a RAM malma run command, we're shutting the network off. So from the container point of view, even if it wasn't able to attack your system, um it would u uh basically not be able to communicate back to some some other site to tell it about. Uh we also have a kind of a cute thing. So if you're running Podman, it'll show a little the prompt that I'm about to show, you'll see the little tiny sail. And if you're running with Docker, it'll show a whale. I think that's cool. So here I'm actually running a model and um this model is actually running uh granite which is a fairly large model on the system. Um and basically it comes up with a prompt just like you see a chat GPT. Um I'm actually running this thing with just CPU. Okay. Uh, I have Nvidia GPU inside of my laptop and they haven't updated. Nvidia has not updated this off of Fedora 42. So, my CPU hasn't my GPU has not worked properly since Fedora 41. So, so this is going to show you if you don't have GPUs how fast it is. Sorry about the So, it's now this is a fairly powerful laptop, but as you can see, it's not horrendous. Um, asking it questions u about that. Um, so anyways, that's a AI running locally. So, this is going to take that model that I downloaded and convert it into an OCI. So basically what it's doing behind the covers it's doing a podman build. So it takes the model sticks it into a format and then allows you now you can take that small LM and push it up to a container registry and then Ramlama can use it from that point forward. So now you've taken that model from a public registry and potentially stored it on a private registry. And here we are showing it you know it's exist podman images at this point. Um so now I'm going to serve I'm going to take a model and I'm g instead of serving it directly I want to generate a kubernetes. So basically I've taken the model I just created and I'm telling it to generate a kubernetes yaml file for running that in production. And you'd see here basically a Kubernetes YAML file that basically uses it. And you will not see Rama Lama command being used inside of this. You'll see here Llama server which is a wrapper around llama CPP. So that's running Llama server inside of a container. But basically in production you would run this container with Llama CPP inside of it or BLM inside of it and run that inside your Kubernetes cluster. You wouldn't use it. You'd also because the model's stored in OCI, Kubernetes knows how to pull OCI images. It knows how to maintain OCI images. So quit similarly here we're going to generate quadlets and then quadlets you know it's quadlets tend to be multiple files. So we're going to show that we're going to pull down a image. Uh this is a volume file for the images. Um here's the image that we pulled down for running it on the system. Um here is a quadlet. So this is the actual quadlet the user. It takes you know DRI mouse them into it tell us what image. Again sorry about the video jumping but you can see that that image is a ramalama image. So the default image that everybody pulls if you don't have a GPU is to pull a ramal lama. If this GPU on my machine was working, we would have pulled the CUDA image, you know, with the Nvidia GPU. We'll do quick editing here. Sorry about not not being prepared for this second. I just fix that. Yep. Okay, so this is a serve part of it actually. So now I've actually served a model. So I'm actually showing it serving a model and the tool will actually show you what the REST API the the endpoint to connect to. Um we can you can switch around the ports and all that stuff. Um underneath the covers now you'll see that it's running a container on the system. When I do a Rama llama run, it actually does the serve in the background and then wires set it up to together. Um, so you can see that that launched a container on the system and then um, uh, similar you can see it is a container inside of Podman PS. Um, and now I'm going to run Firefox and hopefully this part will work. Um, and now you can see, uh, so this is just a web interface that's built into Lama CPP that you can actually do a chatbot directly with. So, who is Dan Walsh? I'm always happy that it knows who I am. And so, this is basically doing the same thing with the granite model. And it's going to come out in Dan Walsh, and we'll see what it interprets that as. Dan Walsh is a renowned figure in the field of cyber security. Oh boy, I'm pretty good. I'm a director of strategy. I didn't get that. Just got that promotion. And a CTO is recognized significant contribute open source SE Linux. That's correct. Join in 2010. I joined in 2001. Uh did work for IBM. I've never worked for IBM in my life. >> So yeah. Yeah. So uh so you know I included pretty prestigious NSA stuff. I don't anyways uh so you can see that AI is knows a little bit about me but hallucinates a little bit. Uh so what really good is I could tag rag database and when I played with the rag database before I've taken my book and and asked it my book's dedicated to my mother and you could ask questions like who is the book dedicated in the rag? really cool stuff, but I don't have that part of the demo right now. Anyways, um so I'm going to stop the model from running. Um now I'm going to run it with Llama Stack. And you can see that now we have two API endpoints. So it's using uh when you run with Llama Stack, you can talk directly to the llama API. um to talk directly to the open AI. Basically what we just did, you would have to change it to use B1. Uh you also notice that I'm actually running a pod behind the covers because I have llama stack running in one container. I have llama CPP server running in a different container. Um and this next command hopefully works because it was broken yesterday. Yeah, it seemed to kill it and now the containers are exiting. Um, so that uh I already did this part. I think I don't know if there's anything else to show. But at this point, I think I'm running out of time anyways. We already did this. Oh, thank you. So this point we go back to the slides. And now at this point we do questions. Um and we have a lot of AI generated versions of uh basically Ramal Lama uh talking to Podman and actually even demonstrates my book. But anyways, these are all the links. Anybody want to take a photograph of where you can get more information about all the different projects going on and any questions? >> Uh, you mentioned Intel as a vendor that supports hardware acceleration. Does that include Intel Alchemist and Battle Mage DGPUs? Because >> See, this is when you realize I have no clothes on right now, >> right? >> Um, so I don't know. Uh again that's something we allow open source people to come in and and add so there a lot of people are using Intel hardware sometimes ancient Intel hardware um so my best advice to you would be to try it out okay uh but it supports you know it probably supports anything anybody who has that piece of hardware has opened up a contribution to the upstream Uh you said that uh the model should not be in the same container as uh the AI. uh is uh are any recommendations uh regarding a I a collaboration with IBM how to use it uh on Fedora or other Linux distributions because when you AI accelerator and when you see hardware requires AI software >> yeah um well first of all we're working with closely with IBM you know, they help pay my paycheck. Um, but um, you know, so IBM right now is working on Z series. They're contributing Zer support. They're contributing power PC support. Um, >> yeah. Yeah. Z system. >> Well, yeah. Basically, we want we want IBM to build container files specific for Z series and specific for them, but they have to contribute to Llama CPP to get Llama CPP to able to interact with their GPU. And again, we're open to anybody. Some of the largest contributors to Ramal Lama right now are actually Chinese companies. So, China is because China is being sort of cut off from CUDA and Nvidia, they're off developing their own. So we have a thing called Musa and another thing called Khan which are container images contributed by Chinese companies. Um and so what we're seeing is exactly that is like we you know we want to play in this game um of contributing open source and again because of the power of open source there there are basically two guys at Red Hat right now who are working on Ramalama project but we have now a lot of Red Hat contributors but we have a lot of external contributors and I think it's it's approaching 90 90 different contributors in nine months to the project. The project right now has about 1 point uh 1,700 stars. So that's a a rating rating. So I'd like everybody to go on to Ramal Lama and give it a star and say a like or whatever the hell this thing is just because those numbers are important and to continue to get Red Hat to continue to support building this project. Anybody else got questions? The output generated by Rama Lama for Kubernetes is that just running a single instance of the model in Kubernetes or does it actually support distributed? >> So it build it builds a deployment with a replica of one and you're going to quickly know I really don't know what I'm talking about in Kubernetes world either. But um so the the main main idea is is here's the containers, here's a Kubernetes jaml that you understand with the deployment and if you wanted to run it on two now fundamentally it's going to match what you're running locally but if you you know if you're running on a Mac it's going to say use the ramalama pack. Well if you want to run in in CUDA you'd have to go and change the container image to say CUDA instead of >> Ramalama and then the tools will be smart enough to handle that. Um, and then it becomes your responsibility when you go to deployment to make sure that the machines that you're running it on actually have support for the GPU that you running on. So, we're not we're not trying to handle that. But again, it's just like here's containers. Here's how you mount an an OCI image into into the container. Here's how you know all that stuff is being taken care of by people like me who understand how to do this. >> Probably have time just for for one more last question. uh I'm interested into actually playing with the models and then few slides back you showed that you could just get a PDF file and then uh feed it to some sort of a registry and I missed the part there was in in the middle there was a MD file from some random server what was that about I get it like you can get a PDF and feed it to the registry but then >> so so so that was that's Ramalama rag so Ramalama rag will take your documents and compile it into format. Actually, right now we're using a thing called Quadrant underneath the covers. Um, so there's different there's Milvvis you might have heard of. There's different ways that you can compile data into a AI database and there's these all these companies generating and quadrant two of them are Quadrant and Milbus. I think Llama stack has their own and right now we have a hardcoded output a quadrant database inside of a OCI. What we want to do is allow you to specify I don't want quadrant, I want milvis. Again, because a lot of this stuff's evolving. It's not, you know, we want to allow you to pick which kind of database you want to interact with. And data scientists would tell you for this type of thing, mil works better for this type of thing. Qu I I I don't understand what the differences are, but that's not AI models. That's AI data. So you're you're basically taking your data, compiling it into a format that now you could query. You know, like when I asked about Dan Walsh, I could have put my resume in there and it would have used my resume as the primary source rather than just random crap that I downloaded off the internet about who I am. >> Right. >> Right. Yeah. Yeah. And continue to evolve whether or not again you might want to look at what we're doing inside of that container because I'm not sure this is going to scale to you. Everybody's going to go in and do it. Oh, that's really cool. Well, now let me give it 10,000 documents. I'm not sure our tooling is going to do it, but it's simple Python code that we're calling into Docklink to do it. Yes. I thought I had a full hour. >> Yeah, but I >> Okay. All right. So, anyways, thanks for coming and uh I'll be around for I'm also giving a container buff tomorrow if you want to come in and talk about containers and we can do some other stuff. So, anyways, thanks for coming. [applause and cheering]