From Laptop to Cluster: Running AI Workloads Seamlessly from Podman... Ashley Cui & Urvashi Mohnani

This talk focuses on running AI workloads seamlessly from Podman to Kubernetes, presented by Roishi Manani and Ashley from Red Hat. The speakers explain the rise of AI technologies throughout the 2020s, notably the shift toward agentic AI, which allows AI systems to autonomously execute tasks. They introduce Ramal Lama, a tool that leverages container technologies to help users effectively manage and deploy custom-tuned AI models as OCI artifacts. Additionally, they discuss Podman, a rootless container manager that emphasizes security and offers unique features like pods for running multiple containers that interact with each other. The session also covers practical workflows, including migrating AI workloads from local environments to Kubernetes, using Quadlet to run containers as systemd services, and enabling automatic updates and restarts. Throughout the talk, the speakers demonstrate how these tools can simplify the deployment and management of AI applications in production environments.

Hello everyone. Welcome to our talk today. Um, thank you for showing up at 5:00 PM. I know we're standing in between you and CubeCrawl right now. Um, but my name is Roishi Manani. I am a principal software engineer at Red Hat working on container technologies and open shift. >> Um, hi everyone. Uh, my name is Ashley. I am a software engineer at Red Hat and I am

working on um, container tools such as Podman. Um so our talk is going to be uh from laptop to cluster running AI workloads seamlessly from podman to kubernetes. Um so I'm going to start with just a brief overview of the timeline of uh kind of the boom in AI in the 2020s. Um starting I sure you guys know this already but starting 2022 chat GPT is released

and this is kind of the start of a new era where people saw this crazy new technology and were uh generated a huge amount of interest. In 2023, um we kind of saw the AI race where uh different companies wanted to get into uh the AI uh industry and so other LLMs including Llama, Claude, Gemini enter the scene. In 2024, we saw kind of industry adoption. Um

so anti AI integration started showing up everywhere. Everybody wanted to see how they could use AI in their own businesses. And recently in 2025, we seen a shift toward agentic AI. So uh in 2022 and 2023 we saw kind of the chatbot model of AI where we would ask the uh AI questions and it would respond. But uh more recently uh we've seen the power in letting

the AI actually do the work for you um in being able to execute things um and also uh you know I we've probably all uh worked with generating code um using aentic AI. Um so I seeing this shift uh go from chat bots uh into everybody being like this is a really cool technology that we want to use um into something that people use on a daily

basis. Uh how how do we do this like in a way that's easy and secure. Um so there's a lot of increasing usage and adoption of AI. Uh and uh one of the things that people really like to do uh with their LLMs is to uh to custom tune them um depending on their own needs. Um so uh what would be a good way uh to be

able to kind of share uh your uh custom tuned AI needs. Um and then also the current trends toward uh moving towards agentic workflows. Um why consult an AI when it actually can execute and do the work for you? Um, of course, these agentic AIs still need an LLM in the back to do the thinking and then it and then it processes things to uh actually do

the work. Um, the thing with agentic AI is with power uh comes greater risk. Um, so in the past uh you would ask GPT a question and then you would actually do the thing that uh you asked it to do. Um but with agentic AI where you're giving it access uh to your system to to maybe run things um to create things to deploy things. Um and

and so uh we need to make sure that we can do it in a way that's kind of safe. So the grand question here is how can we make deploying AI applications easy and secure, right? Um and so our talk is uh meeting the family of tools that makes this easy and secure to do. Um so I'm going to introduce a few tools to you. The first

one of this is Ramal Lama and Ramal Lama uh tries to make AI easy um and and boring even. Uh so um the idea behind Ramal Lama was that it uses common container commands that we all know and love. Uh so if you worked in the container world before, you probably know things like pulling an image uh running a container. Um and so Ramal Lama seeks uh

to provide that experience when running with AI. Um so Rama Lama is able to pull AI models as OCI artifacts. Um so the idea behind this is we have a great way of you know sharing containers and making sure the container that you run on your system is the same as the container you run on a different system right wouldn't it be great if you could take

this system that we built for containers and say you would have a fine-tuned LLM and you want to share it with your co-orker because it's trained on data that uh you have in your company. uh we already have the system to push and pull containers uh from registries and run them in different places. Why can't we do this with your customtuned AI model? Um Ramal Lama makes

this easy to do. Um Ramal Lama also has the ability to run the AI the LLM in a container environment and also it uses container images uh for GPU discovery and on your host system. So Ramalma is able to take advantage of your GPU on your systems. So the workflow kind of here is you have an AI model um and then you shove that into an OCI

artifact so that you can push and pull it into container registries. These aren't container images so they don't run on their own but the container registries are able to understand uh what an OCI artifact is and is able to uh push and pull them from the registry. Once you've pulled them down, you can run them in a container. And uh Ramal Lama also has commands that allow

you to uh add like more data to your LLM, uh fine-tune it, and then push it back up to the registry uh with your changes. Um so in this instance, uh this is a great tool because um you can kind of package up your LLMs in a way that makes sense. Say if you have, you know, sensitive data uh that uh or you're working on something that

is very proprietary, you don't have to shove your uh your data into um somebody else's LLM running somewhere else, right? Uh you can run it yourself uh with everything that you need. Another one of these tools uh that we're going to introduce today is Podman. Um so hopefully you've heard of Podman before. Uh but a brief im uh brief overview of podman is that podman runs and

manages containers loop uh locally and what's special about podman is that it is rootless, dumalless and multiplatform. Uh so uh podman there's a big emphasis on security with podman right and um podman is great uh for running things in a secure way and what's better uh so so ram malama actually does use podman under the hood um to run uh containerized LLMs but podman can run uh

any like any application um in a containerized fashion so why not put you know your agentic AI which can has the possibility of like you deleting your home directory for fun. Um why don't we run that inside of a container? Um so this makes sense uh as uh because we've solved this problem before, right? Like we wanted to run regular applications in a in a secure and

lockdown fashion. We can run our a AI agents in this fashion as well, right? Um so uh Podman also is really cool because uh there's the feature of pods. you're able to run pods locally, hence the name pod man. Um, and this is kind of like the exact use case for pods, right? You have an LLM and then you also have an AI agent and they need

to communicate to each other to actually run, right? And what better way to then to shove them into uh pods. Um, so you can use uh the same OCI image locally um in local podman and in production. And then we also have Quadlet with which runs containers as system v services. Um and uh podman also is really great because it's able to bridge from local development um

on your local pods in your system into uh something uh like a kubernetes pod. And also the cool thing about podminant is we have been a CNCF uh sandbox project since 2024. So what would this look like right? Um we would have something on we'd have your host system and then podban would uh exec out or uh exec um to a container. Uh so inside of the

pod we have our LLM container and then our AI agent and they're able to talk to each other because of the container networking uh that happens um in in a pod. Uh so these LLMs and these agents are isolated uh from the host. So they're not you know again like deleting your home directory. They're not doing that. And if it messes up your system it's only messing

up the container in uh not things outside of the system. So this is great if you know you're you know developing locally uh if you're asking it uh your LLM to generate code um you have an LLM that is trained on data uh that is specific to your use case um and then uh so you're not you know leaking secrets to uh some mysterious um LLM service

uh that's hosted somewhere else. Um and then you also have your containerized agent um which can't uh do any harm to your whole system. So, we have this concept of a pod and your LLM and your AI agent inside of the pod and it's running locally. Um, this is great because this is kind of what Kubernetes look like. So, instead of just running it on Podban on

your host, we can just throw it into a node and it and it works. Um, so we were able to containerize AI and from a local development system, we're able to uh push it up into production very very easily. And uh that's where we arrived to Kubernetes. And because you are at Koopon, I assume you know what Kubernetes is. But uh that's kind of our end goal

uh in deploying going from podman and kubernetes. Um so that's kind of the overview of uh the tools and kind of the workflow uh that we're looking at. Um so we're going to do a demo of this workflow. Um so what we have here is uh our demo. Um the first thing that we're going to do is we're going to use Ramal Lama uh to pull an

AI model uh an LLM uh if you see like Ramal Lama the CLI is probably something that you're very used to if you use Podman um before. Uh so Ramalama is able to pull this uh LLM from a uh registry. uh right now in this case we're pulling it from hugging face but because of the uh compatibility or uh with uh L or because of uh the

existence of OCI artifacts we're also able to pull AI models um from regular uh container artifact or container registries. Um so Ramaloma also has the ability to run the AI model. Um, but in this case, because we want to shove the uh the LLM into a pod, um, we're going to use the Ramal Lama dry run function and it tells you uh everything that you need to

know to run this uh this um this LLM as uh in a containerized fashion. Um the thing to take note here is uh we're not the we have the model and then the model actually runs inside of uh let's see the Ramal lama container that uh the Ramal llama team does manage. Um so uh you're able to run things very smoothly. Um so now that we have

the the container command um we're going to create a podman pod. Um, and we're gonna name it and we're gonna open up some ports. Um, then we're going to start the Llama 3 server in the pod. So, this is your LLM. Think of uh like it's kind of the brains behind the operation. Um, and we're going to make sure we uh have that inside of the pod.

So, we're going to wait for it to start. Um, let's see. And then, so now uh we have the LLM running. Um, this is an LLM. It's not agentic yet. Um, in order to um make it agentic, we need to use an agentic platform. In this case, we're using goose. And so, we're going to uh we have we configure goose um to uh use the LLM that

we just uh uh started. Um and uh we can say the LLM is listening on localhost 8080. Um and it's an OLAM. It's uh it's via Olama. Um so now we're going to tell uh so this is in the goose image so it's baked in um that that we built earlier. Um and then so we're going to do a podmin run. Um so now we're starting the

goose container. So this is your agentic container. Um we're now we can see that we have uh both the containers in the same pod and because they are in the same pod. So you have your llama 3 server and then you have your goose web container that's using llama 3 uh the llama 3 uh service in the other container. Um and now we can see that we

have our this is your llama 3 uh chatbot. Um and then behind the scenes what we're what goose is using is that llama 3 instance. Um, let's see if it connects. And then this is your agentic AI containerized so it can't break out. And we can ask it what a container is in one sentence and it'll give us uh and then it'll give us uh the response

and you can ask it to do things. You can ask it to mess around with your files, generate files, run code, but uh no matter what it does, it doesn't affect your host system because it is uh running inside of a container. So there you go. We have a container is a standard unit of software design. So um that's the so this is the local use case.

This is all on podband on your local um uh device. >> Yep. Thanks Ashley. Um so now the other part of the talk is how we migrate this workload from our local podman use case to a Kubernetes based environment. Um so the first thing we need to do is we need to convert this model to either an OCR artifact or a type car image where this model

is stored in there. Um so you can do that with ramal lama convert and right now here I'm uh converting it to a car image. What this does is creates an image where um we store the model inside the models directory in the image. You can also specify type artifact and it will store it as an OCI artifact instead. Um for the purposes of this demo I'm

going to use a car image um with Kubernetes. Now you can mount an image that has an OCI artifact and directly grab the the model from there as a volume that's now beta in 135. Um, but for the purposes of this demo, I am going to be using um the car image and I'll show you what I do further along with it. Why is this not moving

along? It's a recorded demo. What is going on? Sorry. One second. It's stuck over there. Okay. Let's see if that moves on. >> Okay. Okay. Sorry about that. Um but yeah, so that created the car image for me. Um I've called a llama 3 car. Um so now when we want to deploy to Kubernetes we need to write the cube YAML for that right. Um now when

we want to configure our AI model our AI our model container our applic AI application container that's a lot of configurations that's going to need to happen. Um you even saw that huge podman run command that Ramal lama does under the hood. Uh so the thing with podman is that we have this command called podman generate cube which when you pass it a container ID or a

pod id or multiple container ids that are running with podman it creates the cube yaml for you that you can usually just like pick and do a cube apply of that yaml and it will work in kubernetes. Um so that's the command I'm doing right now over here. Uh but for not for all cases does it work like a simple like you know plugin play into kubernetes.

Um sometimes there are few tweaks that you have to make which is what I need to do for this case. So this is the um YAML that it generated for me. As you can see here, it's put the model uh parts uh as volume mounted into the YAML here. And that's not going to be something that's available in our Kubernetes environment. So we need to go ahead

and remove that um and update our cube YAML to be able to use the car image that we just created. So what I'm doing here is um as an init container, I am going to um use that converted car image. I'm going to copy the model and then mount that into the containers that need access to that model. So those are the uh three edits that I

had to do over there. So now I can go back here and I can um first I'm going to delete the pod that we had started um that Ashley had started and I'm going to be using mini cube as my Kubernetes based environment right now. Um, as we can see, my mini cube is running and we're going to load the images um into mini cube so it

has access to it since um everything is kind of running still locally on my system but it is a Kubernetes based um Um so let's do that. Yeah. So after the images are loaded, we can go ahead and do cubectl apply of that um deployment. And I decided to generate the service and make it a deployment kind. So when it's um deployed in Kubernetes, it ends up

as a deployment. And we can see here the service and the deployment have been created. Um we wait for the pot to be ready. Um it takes a few seconds uh for that to happen. We have kind of sped this video up a bit also um so you're not sitting around waiting for things to pull, to deploy, to push, etc. Um but still seems like it's taking

a while. >> The unset version. >> Yeah. Yep. So, nope, I went too far ahead. Okay. So, we can get the service uh that's uh running with Kubert uh uh in our mini cube. And we can see that it'll be at that URL. So we're going to copy that go into our browser now uh paste that over there and we will see the similar goose session running

over there. We can ask it the same question. What is a container in one sentence? Um now one thing you'll notice is that uh since this is running um in mini cube that's on my laptop. I don't have a GPU enabled laptop. I don't have all the amazing resources needed to be running all these crazy AI models. Um, so you'll see that this actually takes quite a

while to respond to me because mini cube is essentially running inside a podman container which then I'm deploying this pod inside. Um, so there's a lot that's going on under the hood over there. Um, and once it responds, um, we'll see that it takes about it actually took about two and a half seconds for me, two and a half minutes for me to get this response. Um

so not the best experience but to it is to show you that this is kind of possible on any like you know local system and any like Kubernetes Um looks like I should forward that again. Oh I think I played the wrong video. That's why it's the unedited video. My bad. All right. Um so yeah so that gives us uh uh the answer that we're looking for

what a container is in one sentence. If anything by the end of this talk you will know what a container is nothing more. Um yep so we'll go ahead and clean that up. Uh one thing that I want to mention here is um we mentioned um running agents um and models right the future of agents is multiple agents working with each other where each agent is kind

of working on a specialized thing. So they'll probably have specialized models as well for each task that it's focused on. Um and this is kind of a demo of like how you can containerize that right your agents working with different smaller uh focused LLMs. Um so so far we have been showing we showed you how you can um run this locally with podman how you can deploy

this in a kubernetes based environment. uh but we don't always want to like deploy these models in those environments right we have single node use case we have um just you want something that runs automatically on you know just a pre like a non-cubernetes environment and that is kind of where quadlet comes in so with quadlet you can run your container as systemd services so basically it

starts at boot time it does automatic updates restarts etc um and it's all managed by um systemd for you so to be able To do this uh you need a systemd service file and um as we already saw that it's there's a lot of configurations that goes into spec uh for your model container your agentic AI application container um that will be a a whole other thing

that you will need to write but with the quadlet feature under podman um you can just point it to the kubernetes yaml that you generated and fixed for your uh the minor tweaks that you needed to do um over here in this like cube file that you will create under container systemd. Um for this use case, we need to publish these two ports so we can access

the application and you can configure the service um for whatever else like you need it to do. Um and then you can go ahead and just use system um ctl to control the service now. So we can use that to um start the service uh enable the service and you know it it will start automatically at boot time etc. That's basically how you configure it. Um so

and this uses podman under the hood. So as you can see here with podman ps we see these containers has started up um right now. And one thing we can also see is that um since this is running locally on my laptop it will be exposed at localhost port 787681. Um we are able to access the goo session again and we ask it the same question. Um

so yeah so we kind of showed like how you can uh do your AI workloads locally with podman. How you can easily generate a cubey deploy it into kubernetes. How you can use the same kubernetes yaml to use uh deploy it using padlets for like edge edge devices um single node devices and also you can use the same kubern kubernetes yaml and do podman cube play and

it will go through the yaml and create the containers for you in podman as well. It's a kind of a standardized um definition for your containers. Uh I think that's all. I'll not bore you with the unedited version of this. Uh that's all we have and we have some resources here. The link to the scripts um and the files that we used for the demo is up

there and just like links to the Podman and Rama Lama projects. That's all. Thank you.