Make GenAI Production-Ready With Kubernetes Patterns - Roland Huss, Red Hat & Bilgin Ibryam, Diagrid

This talk focuses on the intersection of Kubernetes patterns and generative AI (GenAI) workloads. The speakers, Roland Huß and Bilgin Ibram, both from Red Hat and experienced in AI frameworks, discuss how Kubernetes can be leveraged to run large language models (LLMs) effectively. They introduce fundamental Kubernetes patterns, such as the controller pattern, init container pattern, and stateful service pattern, and illustrate how these can be adapted to handle complex GenAI applications. They also explore emerging concepts like the model data initialization pattern and prefix-aware routing, which enhance the way requests are directed to specific pod instances based on their current loads and other states. Furthermore, the session emphasizes the significance of using KServe, a project that simplifies the deployment of models by managing inference services and scaling, thereby showcasing how existing Kubernetes primitives can be harnessed in the realm of generative AI.

So, welcome everybody to this session about how Kubernetes patterns meet GenAI. Um yeah, first of all, let us introduce ourselves for a second. You go first. Yeah, but my name is Roland Huß. I'm um software architect, distinguished engineer working for Red Hat, working since for Red Hat since 10 years nowadays, and uh yeah, and also on the field of AI. And actually, we have written also some

books together with Bilgin, but also with with Daniele, so we can tell a little bit more about that. Um yeah, but Bilgin. Yeah, hello everybody. My name is Bilgin Ibram. I am principal product manager at Dapr, where we are the primary maintainers of the Dapr project. So, that spells d a p r. It's a one of the CNCF graduated projects that lets you create durable resilient agents.

Yeah. Okay, so so back to these two books. So, the both books are available nowadays, so you see the QR codes already, so they are also available for free for download, so if you if you like. And the talk here is about really about a a blend of both books actually. So, actually we actually we we've just finished. So, when I say "we" in this case, Daniele

and and I, so maybe we can see the the next ones. I think this this will clarify it a little bit. So, actually we could stay here with three persons, so Daniele as well. But okay, we we created this book on how to run generative AI on Kubernetes, and but also have this previous book about Kubernetes patterns, which describes a set of repeatable solutions for common problems,

and we thought about it's a good idea to present how you can apply those common patterns to the field of Um and this is what we are trying to do in this 30 minutes. as I said, you can of course buy this this new book on generative AI on Kubernetes, but you can also download it for for free. But before we we start and go into details

about generative AI, quick recap what we mean with patterns, what what they are, and um how they are used these days. Uh right. Um so this will be a bit of high-level, but um probably most of you here are either running uh LLMs on Kubernetes or actually interested in doing that. So before we dive into that, we want to start with the traditional Kubernetes pattern. So here

I have a typical web application that you would run on Kubernetes. Probably many of you have run this. Um it has a bunch of patterns if we if we zoom in. For example, you see the controller pattern. Uh the job of the controller pattern is to look at the desired state, which is your deployment manifest, and makes that uh happen. In this case, it's starting two replicas

of an application. Right. If you look carefully, you'll see there is a init container pattern, which in this case could be something like um maybe initializing the database, copying some uh um into a database. Then on the right-hand side, we have the stateful service pattern. Uh maybe you're running a Postgres on Kubernetes with multiple replica. So stateful service gives us multiple things, right? It gives stable identity,

it gives us stable storage, stable scaling behavior, etc. Uh further down, we have the batch job uh pattern. And the idea of this uh this pattern was uh ability to run a fin uh finite unit of work until completion. In this case, this can be generating some reports, right? And at the bottom, we have a demon service pattern, which allows you to run specific pods on every

node. For example, things such as node monitoring and uh scraping the logs, right? And so these are basically six patterns that you can quickly see, but if we if we spend some more time, I'm sure we can identify dozen patterns. Um before we map into a full LLM application, let's look into, you know, a model. So, how why is a model different than a web app, for

example? Uh, a model represents, uh, typically, you know, 10 to 100 GB of read-only trained data. And one of the first challenges when you want to run this as a workload is how do you get that that data on the node? And how do you load it into memory? That typically takes, uh, significant amount of time. Then, you cannot run it on any kind of node. You

need a specific, uh, GPU node, right? And you need specific, uh, a runtime. It doesn't run on a web server, but it you need a model server. So, these are all operational challenges that, uh, typical deployment probably cannot do with a lot of, uh, without a lot of hand-holding. And a good way to do that is the K Serve project. K Serve is incubating a CNCF project

that basically implements, uh, the controller pattern. Uh, you declare your, um, inference uh, service. You tell what model you want to run, where to get it from, uh, on what kind of runtime, like vLLM, GTI, etc. And it does all of that. Behind the scene, it will create the deployments, the services, the scaling rules, the routing, uh, policies, etc. and run that. Basically, that's your controller pattern

applied to, uh, model service, uh, as a workload. And if I map the previous application, how it looks like, for example, in a gen AI world, we kind of have the same patterns, uh, you can see, but with different workloads. For example, we have, uh, the controller pattern in this case uh, performed by K Serve. On the stateless services side, in this case, we have the, uh,

the LLMs uh, that are running. And you'll see that instead of, uh, in addition to request and memory, they are also requesting, uh, uh, GPU nodes. They have a maybe a longer startup time. They still can use init pattern, sidecar pattern to do some initialization. Uh in terms of stateful services, rather running a Postgres, you could have for example a vector database that's Instead of a reporting

service, a batch job would be a job that maybe processes documents, chunks them, gets embedding and loads them into this vector database. Similarly for the demo services, you on each node you would have demo services in this case that perform labeling the nodes. This would be GPU nodes and exporting metrics. Right? So, you kind of have the same patterns applied the same way, but at a much

larger scale. And just to bit of summarize what's happening with the familiar Kubernetes patterns and primitives. We've seen the top four, we've seen how controller pattern changes with KServe, how the stateless service, stateful service, demo service, they all basically can be applied almost like as it is in a gen AI world. And there are a few other patterns that you can apply, right? For example, the predictable

predictable demand pattern. The idea of that pattern was when you are running a workload on Kubernetes you have to declare what infrastructure demands it has, CPU and memory, right? Typically. In this case, you also have to declare as a minimum what kind of you know, how much GPU you need. And you can do that different ways. Um for example, you can do that by requesting exclusive access

to the GPU with your request, but that might not be a good idea. There are more flexible ways. For example, you can use you can use affinities to select how much GPU you need or even DRA, dynamic resource allocation, which which allows you more flexible template ties way of requesting resources from the from Kubernetes. Right, this is from the workload point of view. If you go down,

there is the automated placement pattern. And the idea of this pattern is how do you control what what workloads get placed on different nodes. Here, now there are new concerns. For example, you have heterogeneous cluster with regular nodes and GPU nodes. So, as minimum you have to think how do you protect these GPU nodes from other workloads being placed on them. For example, as a minimum you

can paint them and prevent other applications to be placed on it. And other other patterns as well, such as the sidecar pattern with adapter. The idea of adapter was to offer a unified interface for accessing the workload. In case of LLM, that is accessing the LLM. So, you can use sidecar such as Dapr or such as the LLM stack that allows you a unified API for accessing

the LLM. And the common API that is emerging in this space is the OpenAI responses API. So, you can do those. Right. And one last summary slide. The previous patterns were almost applicable the way they are without many modifications. But these patterns here, they do slightly change. Because we're changing the parameters, we're changing the scale. So, the way these patterns apply do change a bit. So, if

we start with the init container, the idea init container typically has been responsible for decoupling the initialization logic from the application logic. Here in the GNAI world, it kind of moves a bit more into data movement. So, using init container sidecar containers, you to the model weights or you bring them to the containers. Immutable configuration pattern, the idea was to bring configurations to your application. Now it's

the scale changes significantly and it moves towards bringing model weights. Moving down, help help probe. Typically help probe was in the range of couple seconds up to 30 seconds. Here it moves into multiple minutes and in addition to starting up the model server, it's also it indicates whether the model weights have been loaded. In some cases, it also includes warming up, for example, whether the the graph

has been computed, whether the key value cache has been allocated into the memory. So the responsibility of the help probe also grows. Moving to the last two patterns, declarative deployment. If you want to do declarative deployment for an LLM, you have to consider that there might be requests that are running for much longer, so you may have to wait for them before shutting down. You have to

consider that startup can be, you know, up to 10 minutes or more. And things such as rolling deployment could be more challenging because you have less resources typically and it might be better just to do fixed deployment where with a bit of downtime, right? And lastly, batch job. If you are using that, for example, for something like fine-tuning, you may you may consider some of the new

features coming into Kubernetes such as gang scheduling, which means you run the batch job only if you can run all all the task at once if there is sufficient infrastructure, And these are some basically example of existing patterns that you can apply today on gen AI workloads, but there are also brand new patterns that are emerging. Right. So actually now that we have have seen that uh

uh that how we can apply the existing patterns to generative AI workloads, of course, it's not a surprise that these patterns are also valid for for generative AI workloads, but as Bill Jim already mentioned, on a different scale. But, there are also new kind of patterns emerging specifically only that only relevant for GenAI or for for the size of GenAI workloads. And one of those is what

I'm want to present here is the model data initialization pattern with some subpatterns. Which means one of the big challenges to run models within a cluster is really the sheer size of the model data itself. So, if you have to download 50 GB of model data for every pod start, of course, this is does not scale, right? It it it and uh affects your startup times or

your uh uh volume your size constraints and whatnot. And for that, there are several techniques that you can apply for softening the constraints. Here, you see a typical phase of a of a startup of a pod in the life of a pod, and you can uh if the model data is really huge, then this is kind of the dominant time slice that is needed here, the way.

So, what can we do to improve uh here? Actually, um there are several things. So, the the most trivial thing is just to apply the init container pattern, which means you have your your model runtime running, and the init container downloads the model from somewhere else. This could be either, for example, an S3 bucket somewhere, or it could be also part of the init container's image itself.

So, the init container could be already contain the model data, and it but it needs to copy over this model data to some shared volume, so that the main container, which has the runtime, can access this data. So, this is the classical approach. Therefore, you typically use an empty dir volume in between. But um but if you if you use the approach with the baking the model

into an OCI image here, then of course you only need to pay the the penalty only once by when the image is pulled to the node. But you still need to copy over this empty dir. This still takes a quite some time. But if you use a direct download from some somewhere else from external storage, you have to pay this penalty every time. And of course this

is not really useful if you if you think about it. But here you see also the an example how you would do this within your pod specification. The good thing is of course this works everywhere and it always worked like But now let's look what what's the alternative is of course and the next step is to use a persistent volume. And you can pre-populate the model data

on the persistent volume if you like and then people people pods can share this persistent volume with the data so you have to pay only once for initialization, but you still have now your model data on a remote file system usually. So persistent volumes are the distributed file system. So this is also something which affects your latency. But actually to be honest this is the or the

most common way how you deploy models in production. But there are better solutions that you can can use and what I'm showing here is also as well how you specify this in KServe. So we already briefly mentioned KServe is a controller on top of Kubernetes that it uses own CRDs and own own platform for for running um for running models bringing runtimes together with the model data.

Oops, sorry. Okay. Yeah, so but but as I said we can even even do better. Now there are come really some new interesting patterns that might be not so well known. Therefore, I'm I'm trying to explain it quickly. So, one is called a a model car. So, this is something kind of you you can say it's a little bit hackish, to be honest, because the the problem

is if you say you have the model already baked into an OCI image, the question always is why can't I access the data directly? Why do I have to copy this data over into an empty dir volume so that our main container can use it. And there's not really a technical reason. So, Docker had this already from the very beginning since 2014. And actually, if you look

into the Kubernetes issue tracker, there was one issue which was I think it was older than 10 years at the end at the end, which requested this kind of image volume type, which means that you can directly mount an OCI image as a volume within a pod and can access the data directly. But this has not been implemented since for quite some time, and so people went

creative and they found out so one comment on this very lengthy issue about image volumes is um some guy who found out, yeah, we can switch on share process namespace, which means that the processes in a pod in the different containers can see each other. Which is of course you you soften a little bit the isolation, but then you can access the root file system of other

containers via the proc file system. And this is what is shown here is that you can just navigate to on on your own proc file system and reach the other containers root file system, which means which contain the model data. And the trick with model cars is that you have So, you have no init container like in the first uh approach, but here you have a a

model car that starts up as a sidecar, not as init container, as a sidecar, which stays alive during the whole lifetime of the pod, which is just a sleep infinity, so it does nothing really. It's just to be kept alive, so so that the the main container can access the model data via this uh proc file system. And to make this easier, a symbolic link is created

during startup of the model car, puts the the proc the model data under a fixed path everywhere. So, this is a kind of a neat trick, of course, on the expense of loosening a bit of security because now every sidecar sees also the main containers, which might be or or might be or might be not an issue. But luckily, this is only a temporary solution. So, if

you if you're right now in production, if you're not running Kubernetes 1.35 or something earlier, then this might be still a good solution to a good escape hatch for for getting to model data within a container image. But the the final solution we are I'm super happy that this nowadays has manifested and is enabled by default in image volume types. And these image volume types work like

any other volume in in a pod, which means to declare the volume with a specific type. The type here is indicated by this image field. This image field had a has a reference to an OCI image, and you can uh add some pull policies or some configuration like there. And then you can just mount this the root file system of this image into your container itself with

with a volume mount here. You can also specify a sub path, which means you can also go into a directory which is deeper within this uh this model. As I said, it's it has a a long history, but now it's still officially declared as beta beta, but it's enabled by default. So, you can use it these days, but of course, only if uh if you are on

a very modern Kubernetes version. So, to summarize, actually the the the ways that you have for getting your model data, your model weights into to your runtime, here's a quick summary. I don't go over them. You you can see what what you can choose where, but just remember that image volumes is really the end game and actually if you are able to use image volumes, please use

that. Okay. this was now a pattern which is very specific for for really huge data to to mount there and of course Jenny is one of those. But now let's switch to the next pattern which is about also very specific to LLM and yeah, please tell us more. Um yeah, so Roland showed us how to get the model data into the pods, right? And before that we've

seen how we can run that model with different model servers. And I think the next thing to figure out is how do we direct the traffic to the right pod instance? And Kubernetes actually has good ways to do that today. For example, we can use Q proxy to randomly distribute traffic between the replicas. We can keep the traffic within the same zone, right? With zone of affinity.

We can use um gateway API to do path HTTP based routing, but actually all of these approaches they they they assume that your request have equal cost and they evenly distribute the request across the replicas of a pod. And that doesn't work with really large language models where where every request looks the same. So they all go to the same path such as completion and um a

request that probably produces couple hundred tokens might take you know few hundred milliseconds, looks exactly identical to another request that may produce thousands of tokens and takes seconds or minutes. The same with the thinking models that can take you know many minutes to execute. Basically, the traditional routing approach doesn't work and we need a way to route the request not based on the request data but on

the status of the LLM. Um and this is what the uh inference extension in the Gateway API project have come up with. And the whole idea is that we are separating the the routing logic or instead of doing it blindly, uh instead it's delegated to specific components. And this solution has three three elements. The first one is the um uh the Gateway API. Typically, this is done

by Envoy proxy and it faces the incoming request. Then you have a pool of um uh language models that can handle that request. And the new component is the endpoint picker. The The job of the endpoint picker is to identify which is the best replica to handle this incoming request. And the way it it does that is it continuously scrapes the endpoint of the la- language servers

and identifies which one, for example, has uh least amount of in-flight requests, has the shortest queue, has the best uh key key value caches to handle the request. And based on that, it's able to uh to direct the traffic. Probably, one of the most um advanced implementation here is from LMD that lets you also do additional things such as route traffic based on the priority of the

request. If you have certain requests that have higher priority, it can route request based on the capabilities of the model. If a model is fine-tuned, right, it can um route request based on the load on the model and uh etc. etc. So, this is already a quite good approach, but it can be uh better, right? If you look into um how LLMs work, they are stateless, so

they have no prior knowledge of the previous request. So, each request is kind of brand new. That means in a in different kind of applications such as chat-based application, every time you send a message, you actually have to send the full history of that conversation for the LLM to have the full context. If you have an agentic application, every message to the LLM needs to have the

full history of the tool calls, of the system prompt, etc. And if you're using rug, every again, every message to the LLM has to contain the full populated context, right? And and you can see that with multiple messages, they kind of have a growing common part that the LLM has to handle again and again, and very a new segment at the end. The way LLMs handle that

is once they process a request, they do store the the the attention tokens in the in the something called key value cache. So, any future request can use that. But, in order to do routing, your router has to be aware of that so that it can direct the traffic to the same instance. And the endpoint picker approach works with that as well, cuz every time the LLM

computes a new cache key or evicts a cache key, it sends a message to the LLM picker, and the LLM picker has a mapping of all the cache prefixes and the and the replica. So, it is able to identify for any incoming request where it has which is the best instance to handle And that's basically prefix aware routing. It turns out this can be improved even more,

right? So far, we said LLM is handling the whole request at once, but even at very high level, the processing can be split into two phases. The prefill and decode. So, the prefill is the the phase where it processes your incoming prompt, and decode is the generation of the response. And it turns out the prefill can be parallelized, so it's actually quite compute intensive. Whereas, the decoding

phase is sequential, so you can only produce one token at a time. So, it cannot be parallelized, it's slower, it's it's memory intensive, which means you can actually split these two phases into different group of pods. So, you can split the processing and if your router is aware of this split, it can direct the request to the to the right pool. Now, this is a simplified diagram

that the the routing probably doesn't happen directly at the endpoint picker level. The pools can direct the traffic among themselves, but the TLDR is that this request routing moves from the request into the LLM state. So, it's based on the LLM state and you have multiple levers here. You can use, you know, the capabilities based on the LLM. You can check the which which LLM has which

LLM instance has the best cache prefixes, make it aware of the load or even custom policy. And the takeaway really is this is I find it quite fascinating and it's still an emerging area routing request to the right LLM. Roland. Okay, now quickly to the to the last pattern that we have and actually this is a good example how you have higher level patterns that are based

on other patterns like here. So, I'm talking here very quickly in 2 minutes about rag composition. Rag is retrieval augmented generation, which means it's a way how you can include domain knowledge into your queries. And this has two phases. It has an ingestion phase and it's a query phase. Ingestion means prepare your vector databases with your documents and the query phase in short use these your your

queries look for similar documents and add those documents that you have identified into your prompt at the end and so that you have a richer context which can be used by the LLM for generating answers like here. Um but as is as we said there are we have seen this already in the first slides by by Biljana that that you that any of those sub components for

rag composition pipeline is still a Kubernetes workload. A usual Kubernetes workload that you can that we describe also in the book with different patterns like like jobs and and stateless and stateful services. Of course, the database is a stateful service like And but but you see that they are that this very specific subcomponents of a REC pipeline really maps also to very well to existing Kubernetes workloads.

So, I would argue that a REC is very well suited for to be run within Kubernetes because we have already the primitives that support everything of those. And um And yeah, and another quick example, you can everything by the way everything what we talk here described in the those books and even much more about GPUs and and and and whatnot. But here's interesting to be how you

make this embeddings. Unfortunately, I have no time to go into details what an embedding is actually, but but you can run embeddings alongside with your with the user of the embedding that I like the orchestrator directly within the same process. You can have it as a sidecar, or you can even go further and have it as a separate pod so that you can scale independently between those

services. So, yeah, but as I said, sorry, this was a very very short overview, but but you the gist here is really that this is it a composed pattern which is based on other Kubernetes primitives and you can easily build up more complex stuff like like these as well. With that, we are at the end. So, here the link to the book. There's another third book actually.

Do you to say something to No, it's not this is a surprising book. So, just scan that and and look into that what what it is about. And now we have maybe 2 minutes for for questions if there are any. Thank you very much.