Evolving KServe: The Unified Model Inference Platform for Both Predictive and... F. Spolti & J. Lee

This talk covers KServe, a Kubernetes-native platform designed to streamline the deployment and management of machine learning models, especially for AI workloads. The speakers, Philip and Julie from Red Hat, explain how KServe serves as a crucial layer over Kubernetes, facilitating the orchestration of both predictive and generative AI applications. They discuss the framework's evolution from Kubeflow and its recent acceptance as an incubating project in the Cloud Native Computing Foundation (CNCF). KServe integrates with tools like KNative for auto-scaling and Envoy for networking, enhancing performance for serving large language models (LLMs). The session also touches on the importance of infrastructure reliability and the ability to abstract complexities for data scientists, allowing efficient model deployment and inference management.

All right, so um Hello folks. Um I [clears throat] am Philip. Um I'm a senior software engineer at Red Hat. We work on the inference team. Basically, KServe is our main thing nowadays um on the tool we work, right? Want to talk something, Faris? He's up. >> Hello. My name's Julie. I'm principal uh software engineer at Red Hat. Uh we are same team. We are working on

KServe as a maintainers. So, I'm really excited to share about the KServe roadmap and what is KServe today. >> Go ahead. >> So, um in this presentation, you can start work uh talking about say welcome to Amsterdam, right? But, we are on the last day already, so um hopefully you guys have some energy energy left to stay here for 1 1 hour and a half. No? >>

Only 30 minutes. >> Okay. So, um just kidding. Yeah, all right. Okay, so uh let's get it started, all right? So, what is what is KServe based from, right? So, KServe is kind of the main orchestrator for any model workload in AI workload you might have to to run on Kubernetes. So, uh in the in the pillar, we have the hardware. So, basically, we can we can

have a GPU, CPUs, and I think TPUs are uh as well as I guess it's kind of supported on the Kubernetes. Uh Google has put a lot of effort on that, right? So, why not? And in the top layer, we have Kubernetes or orchestrating everything for us. And what KServe really is, it's a top layer on top of Kubernetes that handles everything we need to get our

models ready running in a production grade uh level, right? So, uh cuz nowadays, what is the most important thing is to have a reliable and trustable infrastructure where you can have your workloads running without disruption, right? So uh you could KServe instead of um trying to reinventing stuff to adapt to to our needs, we kind adapt the KServe to use what we have the best already in

the market, right? So the most uh useful tools that we have nowadays are already on CNCF, are already graduated. These are now close friends, right? I'll get there, right right there. So for auto scaling we start with KNative, but we are now moving to Knative. And for networking our main guy is the Envoy Envoy AI Gateway. Uh it has some some good stuff that you will present

later. Uh and the top of this All right, LLM Z is the main guy that everyone is just talking about nowadays, right? And how Why is that so important? Uh it's because it makes your inference so fast that you can even compare your large language language model being served serving served locally with the same power than the ChatGPT, right? The same the same speed. well, we just

not prepare for predictive, right? We are prepared for predictive and getting a way better for generic generic nowadays because generic generic AI is our the main thing. And if you do not get generic generic AI well served, well, what what's the point to move forward, right? So when it began, the KServe journey starts as a Kubeflow uh project. It was initially developed by Google, IBM, Bloomberg, Nvidia,

and a few and a few others. Uh and its first name was KFServing. At least myself, when I started working with KServe, it was already named as KServe. It was like two or three three years ago. And and yeah, it has been evolving so fast. >> [snorts] >> So uh in 2021, there was a big moving forward. Um KServe was kind of rebranded. So uh why it

was needed, right? KServe is starting to get so much popular and that that arrived and needed to, you know, why not have our own brand instead being um just a part of the uh Kubeflow components, right? And the next step was uh KServe joined the Linux Foundation and the AI and the data Foundation. and lately, and one of the most important step was um KServe was accepted

as an incubating project on CNCF. Uh you know, being a member of the CNCF is not like upgrading logos or putting in in the website that we're a member of it. It's more like um having the governance, the the production ready grade, uh the security stuff, and you know, we need to be trustable. And what uh CNCF tells to to the world we is like um okay,

it was accepted in CNCF. It means that CNCF has a very How can I say that? It's a very It's very difficult to be to join, right? And when something joined to it, means that okay, uh it is a incubating project. It has a lot of good good stuff to to provide. And most important, it can be trusted, >> Okay, so what is KServe? easily speaking, KServe

is a the guy that will make everything easy for the the data scientist. The data scientist essentially does not have too much knowledge about Kubernetes, right? Kubernetes can be very very very hard to to maintain and to configure. And what it does, it kind of abstract out the out the complexity of configuring network, configuring security, configuring ingress, egress, and etc. It makes a >> Oh. He's nervous.

>> [laughter] >> I did I did all right. >> All right, so um I'm going to start with KServe. It's a you know, I've Here is today is the last day, but the KubeCon is this KubeCon is first thing for us. I present so many CNCF, but it's a new thing for KubeCon. So I also nervous, but stay with me. I'm going to get used to it.

So KServe is um I'm not sure. Have you been to KServe booth or not? We are talking a lot of model serving. I always say, this is powerful tool to deploy a model. We predictive AI, but we are moving to generative AI, too. And as he say, AMD big things for us, and we've been we are moving together very quickly. So they are release on last year

May, and they are already 0.6. And we are pushing it into KServe as they release it. So we are using 0.6, and uh version will be same as their latest version. So, we are, you know, working together very closely. So, KServe pretty clearly, we provide control plane. So, as uh users, you probably have some of this kind of question, how we scale, request patching, security, distribute, those

are we provided. So, why you need usually, where is your model? That's it. We are using yours, pulling the model, and then pushing somewhere in the pod, deploy. After that, you can control with our CRD. The name is inference service. And then you can change the scale out, request patching, those are. as I say, we start with predictive, and we are moving on generative AI traffic, as

well. So, typical situation, predictive, classifier, regressions. So, you can use SQL loan, XGBoost, LightGBM, whatever you want. There is runtime. Even if there is no runtime for you guys, you can create your own. Create it, custom runtime, use it. The other control plane is the same as before. So, there is no limit to use it. And generative uh generative AI, we are we have new CRD is

integrated almost 6 months ago. And from that point, we supported LLMD. So, LLMD has a lot of dependencies, also uh so many objects to manage it. If you are doing it uh by hands, it's pretty um I want to I don't want to say complex, take time to do it one by one. But, we provide same unified API, the CRD, then you can use it. The other

the rest of them will be created by a KServe controller. So, we have KServe controller and inference controller right now. So, you can choose if you are using predictive, you can install KServe controller only. If you are using the only, then you can con- uh install the specific controller only. So, this is blocks we providing. So, scale efficiency, all of them is about uh efficiency. So, we

maximize all the uh LM metric space auto scaling, those are is the into our generative AI. So, KV cache is the the thing we are keep talking about because GPU is not cheap. It's a It's really difficult to get it even though in the company wise, it's when I request it, normally, it take 1 day, 2 day to take it. If I take it, I don't It's

difficult to get it. So, so we are talking about all the time scale and how how to make it maximize. So, we provide all the piece of them into it. So, it's still incubating project, but it's already accepted many uh uh companies. Recently, we added more and more. So, I would say it's still um mature enough, so you can try it. So, this is a generative architectures.

Uh it's looks very complex. But, um it's pretty uh, specific. It's very similar to L and D inference. So, K 0 controller, KIDA, L and auto scaler, those are is, uh, kind of So, K 0 controller is ours, but the other is almost them is dependencies. But, dependencies, you know, we need it like L and D, uh, pre-built decode, those are. And also, we are using gateway

API. That's why we needed Envoy gateway, Envoy AI gateway. So, very similar names, but, um, it is the, kind of trending. We are moving in uh, from ingress to gateway API. So, L and D using latest version of all the component. But, um, we provided the upstream K 0 provided all the installation for, um, this kind of dependencies. So, as a user, I want to play with

L and D, don't worry about it. Just one script, one file script is executing on your kind of mini cube, uh, and then play with it. Even pre-built decode, it's working fine. If you don't use, uh, big model. Right. >> All right. So, I'm answe- answering these questions. Do you guys have any idea why concurrency fails on LLMs? The answer is in the slide. All right. So,

basically, uh, when we send a request to a GPU, so, a GPU does not have have concurrency, right? It It process what it receives and returns, So, um, that is the the gap that, uh, we Sorry. Uh, that we are kind addressing with LLM Z, right? So, instead of have one user sending sending in inference and that same user waiting for response we have segregated by the

pre-fill cache and the decode cache so we can have like a lot of users at the same time getting response very quick because as the key key key key cache for the LM is very efficient in in sharing between all all the nodes. I mean you do not need to compile everything every time right? So if the cache is ready so just return. All right um So

to efficiently scale on VLLM uh we are basically using KEDA because KEDA is a very more flexible in in terms of creating the query. It accepts any Prometheus query accepts Prometheus query language so we can define the query once. You can use multiple queries. You can aggregate define your your target metric. So the most basic one and one of the most useful is the KV cache. Because

uh depending on the size of the GPU and depend of the the size of the prompt it can full your VRAM very quick right? So that is a efficient way but it's not a there is an another gap that needs to be filled that will be discussed a little bit further in the slides. I think this is a >> Yeah we need to >> just switching because

we not do not know every component. >> Do you hear me by the way? this is Envoy Gateway. So there is two things right? Gateway API and the other is Gateway API extension. So this is for inferencing. So, it is uh gateway API extension. And we are using Envoy by default, but you know, that is abstraction you can change it on the line. So, whenever the gateway,

so traditionally we are using ingress, but we are moving on gateway API with LMD. So, LMD require this gateway API plus GI. We call it GI. It's a gateway So, it give a more flexibility. We can control it everything inside it. So, at the end of uh uh request, we can give for customize inside it like a rate limit or uh we can send it to some

scheduler. Scheduler calculate is the score. We can reset We can find the right node to get it. So, this architecture is typical uh diagram that using um gateway API. So, but um when we go to our LMD, it's more details. as I say um LMD is a big thing for K server as well. So, this is the one so many big company uh spend some time efforts

to create it. It's a new architecture because they have their own issues like a vLM. It solve most of KV cache maximize utilization. But still, they need uh so many large-scale LMD. That means they need more than just KV cache maximize. So, mainly they are focusing on predictive you know, prefill and decode the mode because prefill need computation power the other decode creating the token using the

KV cache. So, they have their own rule. So, they don't need to lose any utility utilizations. So, this is the one 1 years old, but they already joined to CNCF the sandbox. I'm not sure you already joined that the speaker note. They are yeah, pretty fast and there's so many people join it. This is a LMD um uh LMD architectures. So, invert inference gateway is there is

inference gate is gateway API. But, inside it is the request coming in is sent to inference scheduler. That is just pod. They are doing what they are doing is calculated the score based on metrics. The VLM have a bunch of uh metrics. It will send to those are to inference scheduler. Like uh prefill. What they are doing is creating KV cache. So, you are questioning something like

uh who are you? Who are you is they make it KV cache. And when they create a KV cache they send it, "Hey, I'm I'm I'm doing my job. You should know it." And then scheduler, "Okay, I got it." And if somebody send the same question, definitely KV cache is there. So, they don't need to create a KV cache anymore. And then decode, they will create it

uh response the token the expensive one, right? So, create a token and then predict to predict the next token. So, until the end. So, at the end they are doing uh their own job and then they share KV cache. So, there is a Nixel. Nixel is uh developed by uh Nvidia. It's a abstraction. So, each vendor can uh use their own, but it's like a gateway API.

So, LLM inference uh workload, we provided single node, multi node, and prefill decode. So, previously when we have the predictive we only provided single node with a deployment, but now we accept uh LWS. So, LWS is now core Kubernetes object to create a state pool set. State pool set itself is we already familiar with, but they have master. The master can control each group of state pool

set. That's why it's really good fit this kind of multi node multi GPU case. uh disaggregated prefill also using So, our API is like that. Why What I trying to say with this one is pretty flexible. If you set it up, we create it based on what you provided. uh we already provide default one, but you don't need to use it. Whenever you want, you can override

it. uh how what composes the LLM inference service, So, uh basically, uh what has been inherited from the the usual one is the the basic most information, uh where is your model, uh what is the name, and how critical is it? And if you are using Laura adapters, for example, and uh the way this the the the models are fetched, uh it's completely independent. You you can

use OCI, you can use HPS directly, S3 buckets. But uh because, you know, the uh very large size of these models, uh one of the most efficient way, I would say, that you can store in PV and reuse it. Uh KServe supports PVC or as well you can build a OCI image and store inside the um inside your network of of an internal S3 or you know,

it's a totally customer choice. So, the spec router, um the important guy for the LLM Z, right? It is the kind of the EPP, the endpoint uh uh picker selector for the for the very efficient way to, you know, di- direct the the inference request to the right pod. So, here you you you you will define where your your gateway is, the rules of your of your

route, which rules of the for the end point end point picker will be used. There are a few, uh if I'm not mistaken, there are three uh that is being used today. >> And lately, uh the last one is the the LLM LLM Z stuff. There are basically two pods. One is the router, another is the the perf view pod, and the other one is the the

code pod. Yeah, the the code pod. And these three together provides you very fast and very reliable, you know, uh inference uh and most important, saving computation time. >> All right. So, um this is a very simplistic um flow. Uh on the right side is basically the the most simplistic uh that we call the the case of raw mode deployment mode. It basically does not have anything.

Uh it's just a service that goes directly directly to the VLM port, does not care about nothing. It just send the the request and expects the the output, But uh on the left side we have uh one of the you know, the best world I'd say now nowadays for for serving and and and LLM these uh LLMs internally. Uh which is um the first guy which will

be the Envoy gateway, right? The the gateway the ingress entry point. The eight the HTTP route that will be managed by the LLMD endpoint uh endpoint picker. And based on the on the these rows we can um we we direct to the user to the port that to the port that has most of his content are already pre-filled and decoded. also uh using even using uh LLMD,

you can have everything running in the same node. Uh I mean, you can have LLMD in the single node, but you not have um features like auto scaling for example. >> We have 5 minutes. >> Yes. So uh as I said before, uh just KEDA is is might be not not enough. So um we um not not we, but the DWA uh I think it's come from

Kubernetes director, right? It's a native, yeah. So um what is is exactly? Um it stands for work workload variant um auto scaler. Which means that it's does not care only about, you know, uh cache utilization, CPU, GPU, memory. It goes a a little bit deeper. And can auto scale uh very more efficiently than just KEDA working um as it is. So, the activator is pluggable. Um nowadays,

it supports KEDA and and the HPA as it is. And all the intelligence does not is not kept by the uh deactivator. It's exte- uh it's maintained centralized it by the WAA, which means that with only one CRD, you can manage everything and you can manage how will be the behavior of the um of the auto scaling part. >> So, here we need to >> Okay, uh

now we have a little demo. I hopefully the audio will be working. Uh let's see. Play. >> Today, we are going to look into detail on how KServe can be used with LLMD and VLLM in complex deployment scenarios for inference on Kubernetes. We're going to look at how this integrates with the Kubernetes Gateway API and the Gateway Inference Extension, GIE, to bring high-performance inference to your Kubernetes

cluster. LLMD is a Kubernetes native high-performance distributed LLM inference framework. It orchestrates VLLM in a variety of ways to bring state-of-the-art inferencing performance through several optimizations. LLMD has several well-lit paths for configuring these optimizations, and we will look at two in particular today. Precise prefix cache-aware routing and prefilled decode disaggregation. The usual route for trying out these configurations is through the helm file supplied in the guides.

We can get a picture of what these produce with the helm file command, so we can preview them. It consists of two deployments and a gateway configuration, The endpoint picker and one or more workers running vLLM. The Kubernetes gateway is implemented in this case by Istio, but others are possible such as Envoy, K Gateway, etc. There are other support objects here like RBAC and config maps, too.

The important point is that the endpoint picker object implements the gateway inference extension. This is really the heart of LLMD. This endpoint picker is doing the intelligent routing of requests to the vLLM pods. The open source KServe project gives a higher level abstraction of all of this, the LLM inference service CRD. Rather than configuring objects with helm, KServe has written an operator that will do all this

work for you. You just have to give it the configuration and it will go off and run everything for you. For convenience, there is a template LLM inference service config CRD for this configuration, so that it can be shared as a base ref across many instances. We add the bulk of the configuration into this template and reuse it when deploying a model for inference. There are actually

a whole library of them installed as part of the project. Here is an example configuration we have created for the precise prefix cache aware routing well-lit path deployment. The router section is for the endpoint picker configuration, while the template section is for the vLLM inference pod configuration. In KServe, we have configured Envoy to implement the Kubernetes gateway. And here the EPP router will implement the gateway inference

extension. Each of the vLLM pods will need access to the model. This can be a big download and duplicating it in each pod is inefficient. It's better to download this in advance and save it to a persistent volume. We do this through a job and then refer to that PV in the configuration. The well-lit paths outlined in the LLMD docs are available here in KServe too. For

instance, prefill decode disaggregation that we saw in the well-lit paths is an advanced deployment method that can reduce inter-token latency in responses and hence improving interactivity. In this kind of deployment, we can see we have VLLM pods dedicated to prefill phase and to the decoding phase of inference. The communication between these pods can be over NVLink when GPUs are on the same node or over high-speed networking

like RoCE across nodes, but both are abstracted by the NIXXL library. The scheduler here has an important role in deciding whether to disaggregate a request or not depending on the context. In our example, we've a 20 billion GPTO SS model for demo purposes, although generally the method gives the most advantage on bigger models. Here we've installed a new deployment alongside our others in a new namespace. You

can see that we want two replicas of prefill pod on GPU MIG partitions because we're sharing this cluster with other users. As you can see there are two prefill VLLM pods and one decode VLLM pod with its sidecar. We also have the scheduler pod hosting the endpoint picker orchestrating the requests. The service gives us the internal endpoint on port 8000. To test the deployment, we have the

open web UI installed as a front end and we can connect it up to our endpoint. Now when we start a new chat, we can see the performance and content that our deployment produces. Here I've entered a request previously and now I'm going to follow on with another question. To track the performance, we have Prometheus and Grafana installed and have linked up these pods through a service

monitor. We can see the results when we look at the KServe dashboards. These show the endpoint picker metrics and the VLLM metrics in great detail. Also, the NVIDIA DCGM metrics reflect the split workload on the GPUs between prefill and decode pods. We have added a guide around this to the upstream KServe project that describes all the configuration. It focuses on GPTO SS 20B model, but it could

be applied to any similar model. The guide covers the well-lit paths of prefix caching and prefill decode disaggregation in a multi-GPU setup. Thanks for watching. >> Okay. Sorry about it. We a little bit slow, so yeah, stay with us. So, we almost done. So, one two slice. So, everything So, it's our roadmap, but um yeah, just uh let me close it out a little bit. So, we

are uh open source. we are really open it. It is there so many contributor join it recently and send it a PR like a 10 times more than if you are interested, want to contribute it, please join the KServe and upstream uh github.com KServe KServe. And also, we have a website and also provide quick installation. So, yeah, thank you for coming here. Um I really want to

get a question, but we cannot do it because of time. So, after this one, yeah, if you have question, just come to us and then ask it, okay? Thank you. >> Thank you very much. Oh, before you guys leaving, can we take a selfie with all of you? >> Huh? >> A big selfie. >> Oh, big