Cloud Native Theater | Istio Day: Running State of the Art Inference... Jackie Maertens and Nili Guy

In this talk, Nili and Jackie discuss the integration of AI workloads with Istio and the Kubernetes-based LMD platform. They highlight the significant shift in AI, where inference now dominates the workload landscape, comprising almost 70% of it, necessitating a different approach than traditional microservices. The speakers explain the inefficiencies of standard load balancing in handling non-uniform and resource-heavy inference requests, emphasizing the importance of optimizing performance and cost. They introduce concepts such as time to first token, KV cache management, and the need for disaggregated serving. Additionally, they outline how LMD operates as a distributed inference platform and the role of the inference scheduler and gateway in efficiently routing requests to model servers. The conversation culminates with insights into the evolving landscape of AI inference and the collaborative efforts to enhance Istio's capabilities for this purpose.

So, hello everyone. I'm Nili. I'm from a IBM. And I'm Jackie, software engineer from Microsoft, Istio security maintainer and co-lead of our product security working group. So, today we'll be talking about running state-of-the-art inference with Istio and LMD. I'm from the from the LMD team. Yeah, please go ahead. So, why Istio needs to care about AI? I don't know if any of you were today at the

keynote, but I think that to the people that were, you understand that AI workload is becoming the next major workload. And not only that, um when talking about AI workloads, there are inference and there is tuning. And what you can see that the inference is taking more and more bigger part, almost 70% of the AI workloads are inference. So, what's the difference? Meaning we we had microservices

and HTTP requests, and now we have AI workloads. Why do we need to do anything different? If we're looking at HTTP requests, first of all, the hardware is cheaper. Using CPU and GPU is something that makes a lot of difference. We need to be more efficient, more cost-effective. requests are faster, are uniform, whereas LLM requests, inference requests, are slow. They're non-uniform. Some requests can have long prompts,

long response, others can be faster. And as we said, this is like 6 to 9 more expensive requests. So, we need to do something different. If we are trying to do regular round-robin load balancing with those requests, we'll get we will get very bad performance in aspect of the performance and the cost. So, we need to adjust. So, the the the industry challenge of scaling difference has

a few aspects. As we said, they are variable, resource-heavy. We need to ensure SLO. What is SLO? We have throughput, but we have a new notion of time to first token. Time to first token, it means you're you're talking with ChatGPT. You're sending a question. When will you get a response? Where will the model server start generating you the the first token? And there is latency. So,

we need to optimize the utilization to to reduce this complexity. The hardware is heterogeneous. Sometimes where the cluster has GPUs with that are more expensive, more expensive accelerators, cheaper. We need to do We need to understand how we are distributing the in accordance to the the request and the required SLO, so we'll get the the best performance. And KV cache management, we'll touch more on that and

explain why KV cache is one of the major things that are important in and how do we optimize for KV cache. Um so, as I said, these are the challenges. We need to identify the model server, like vLLM, SGLang, what are the situation points? We need to understand how to reuse pre-computation. This is the prefix tracking. This means that if we sent a request and we are

sending a similar request, we can utilize some of the computation that was done before. Lower requests scheduling. The models have tuned weights, and we need to be aware which has these LoRA, these weights already loaded. Try to optimize the request by that and disaggregated serving. And again, we'll explain more what is disaggregated serving and why do we care about it. So, this is the overall picture of

LMD. LMD is a Kubernetes native distributed inference platform. Today it was announced that LMD is now CNCF sandbox project. And looking, you know, at the higher-level view, this is the inference gateway getting an Open AI inference request. For instance, to Envoy, to Envoy. And now we have the the inference scheduler that needs to choose which of the model servers to send the request to. Once the the

model server is chosen, the request is sent back to Envoy, which routes the request to the chosen model server based on load, based on KV cache reuse, and then the response is returned to the user. It It allows flexibility. We have valid paths that explain how to configure the system based on the workload and how to utilize and how to do distributed inference at scale. So, drilling

down to the details and explaining what we actually did in LMD. Anyone of you is familiar with the Kubernetes inference gateway? Basically, the Kubernetes inference gateway allows us to do programmable routing. It's Kubernetes native and it extends the Kubernetes gateway with new APIs. It allows extensibility, optimizing request scheduling, and a clear path to integrate with with any cloud provider. So, we took the Kubernetes inference gateway and

extended it. So, if we have the gateway, the HTTP route, and the service, we added these two CRDs, one an inference pool, basically label selectors that allows us to choose which of the pods are the ones that can serve the request, and an inference objective, which allows us to give more information on how this request should be treated, the SLO based on on the content, on the

header of the And I will let Jackie now explain. Yes, I'll talk about how the inference objectives, or the one of the CRDs that Nili just mentioned, and also the inference pool, how those both influence the configuration of all of the components in the inference gateway architecture. So, the gateway controller, which could be Istio, it watches the inference pool CRD. It uses that information, that resource, to

configure the inference gateway or the L7 proxy. The gateway utilizes this information to determine, you know, when it receives an inferencing request, what what workloads belong to the inferent or selected inference pool, and I'll go into that more. But also, which EPP or which external process is actually going to perform the endpoint selection. So, the endpoint picker service, which is LMD, which is Nili already briefly touched

on, it watches and consumes the inference objective to know the policy and priority to apply when picking endpoints. And then also, it consumes the inference pool API so it knows which um which model server instances belong to which inferent inference pool. So, the endpoint service, once it selects the optimal endpoint, again, Nili will go into how that happens a bit later. Um but once it selects the

optimal endpoint, it sends that back to the gateway. The gateway then proceeds with routing the request or the inference request to the appropriate model server instance. Okay, so what does it look like when you actually bring in Istio as your inference gateway controller? So, in this diagram, you have like the purple dash box is LMD, what it what it controls. And the blue dash box, that's like

the Istio control plane. That's what Istio is responsible for. So, the client, first you'd you'd create a gateway resource. This is a gateway traditional gateway API resource. The user would specify the Istio gateway class. Um the Istio control plane is watching for the creation of this resource. Based on that resource and its specification, it'll configure a inference gateway. Then the user would apply a inference pool resource.

So, this is the GAIE resource. And you can see in this example of an inference pool resource, you actually have multiple target ports. You have the selector, which is what selects the model server instances to be that are a part of that inference pool. And then it has the endpoint picker ref. So, for that specific inference pool, what endpoint picker service is going to be responsible for

selecting for selecting the endpoints. Yeah, and that resource is consumed as I said by both the by both LMD and Istio and therefore the inference scheduler and the inference gateway. All right, and lastly you'd apply like an HTTP route. So these are the routing rules that apply to the inference request. This is what attaches the host name to the backend ref. And in this case, you might

have traditionally seen service as your backend ref, but in this case it's going to be an inference pool. So this this attaches that host name to that inference pool to then select an endpoint picker service. All right, so that's the configuration that a user would apply as an Istio user, how you would set up your Okay, so what is this again at a step back view, what

is what does this flow look like? So the user makes an inferencing The inference gateway selects the inference pool and therefore the based on the HTTP route rules. The inference gateway makes makes the external proc or call to the external process to get the endpoint for the The inference scheduler using metrics collected from the running model server instances and a few other things that Nilly will get

into, makes those routing decisions to determine the optimal endpoint. Then it returns this optimal endpoint to the inference gateway and the inference gateway continues to route the request to the specified Okay, so this slide it's meant to put LMD and Istio into the context of the AI landscape, particularly the inferencing landscape. I've broken it up into several categories. First you have like the routing, so what falls

under actually handling the request and moving the request, the selection, determining which inference pool or endpoint is going to be ultimately selected for routing, the execution, so the actual execution of the model, and operate. This is like the higher level abstraction of how you can operate your inferencing workflows. Okay, and then I've broken this down into control plane components and then the data plane components that they

configure. Okay, so first you have the inference gateway controller. I've talked about this. It's translating your gateway API inference extension resources, your custom resources, your gateway API resources, and it's translating this into configuration for the data plane. And this could be either K gateway or agent gateway now or Istio as your And there may be others as there and there are others as well. in the data

plane, this is your actual inference gateway. So this is the network entry point for the inferencing requests and routing them to the actual model backends. So you could have a Envoy based implementation or agent gateway, which is a Rust based implementation. Okay, and these all fall under the control of Istio if you're using Istio as your gateway controller. So that's what the blue dash box means. Okay,

so now we're moving into the selection of endpoint or model server. Um you have the inference scheduler Um this is also consuming the gateway custom resources depending on implementation, and then it's going to send this to the inference scheduler data plane. And this LMD or Nvidia Dynamo. Those are just a few examples, there's others. you have your actual inference schedulers. So this is the actual runtime systems

that are performing the scheduling decisions or making and based on the current state of those servers and their KB caches. And these this all falls under the LMD's controller. This is all within their control. And then you have a body based router. This is an additional component that you can optionally add that it inspects the body of the HTTP request. So this is not just the path

or the host or the headers, this is what's actually in the body. So this could be like the actual model name, which it might be within the And this can be used to influence routing. So again, it falls under the selection category. And then an example of this is LMD. They have an implementation of a body based router or agent gateway actually does this in path. So

yeah, this kind of falls under both categories. If you're using LMD, this could be a separate process, which is another external proc call from the inference scheduler. Or no, sorry, from the inference And then or you could be using agent gateway, which actually within the gateway itself it does the body based routing. So it can actually inspect the body of the request in the gateway itself. For

execution, these are actual inference engines. So they're actually responsible for executing the model Yeah, optimizing for computational efficiency. So just some examples like VL which is integrates really well with LMD. And lastly under operate, I wanted to include this. I'm not going to touch on it too much here, but you can go talk to folks at their booth to learn more about it, but just the high

level control plane orchestrators. Like if you're thinking this sounds complicated and you want a declarative way to configure all this, that's one option is to have a higher level orchestrator. An example like that is KServe. All right, so I'm going to hand it off to Nilly who's going to go deeper into the endpoint picker and how it actually selects the optimal endpoint. I can handle still doing

this if you want. That's okay. That's okay, thank you. So as we told you, the basically the inference scheduler, the endpoint picker is an external process. So the request comes to the endpoint and it needs to choose which of the pods this inference pool is in charge of through label selector is the one that it will tell the gateway to route the request to. How are we

doing it? We have a very pluggable architectures of filters and scores. First goes the filters. You can write your own filter. The filters have a view the entire scheduler has a view of the pods and again I will explain how we're doing it. We are scraping metrics. So we have a view of the world and the filter can filter out pods for instance based on load. We

don't want to choose pods that have the VL or the the model server has more than some threshold of load. And then once we have a set of pods that were left, we need to give them some score and choose the best one. What kind of score? Again, this is pluggable. You can write your own score. We have load scorer, we have KB cache scorer. The KB

cache scorer basically looks at the requests and look at the state of each model server or VL and chooses the one that has the the biggest match of prefixes, the pre-calculated prefix that can be reused. So we are giving score to the pods and then choosing one. I want to touch what is KB cache because KB cache is very very central and I want to explain how

briefly how VL or any model server generates a response. Again, when we're when we're talking for instance with ChatGPT or Claude, you're asking a question. The question is a prompt. The model server is stateless. It means that when you send a prompt, you are asking a question, you are getting a response. Now if you are asking ChatGPT, are you sure? You will need to send the entire

session from the beginning. Even if it generated you a huge response, you need to resend everything and this goes on and on in multi chat. You are sending each time the entire prompt from the beginning. The model server itself cannot generate any word, any token without doing this pre-calculation on the prompt. This pre-calculation is called KB cache. It's not really cache, but it's this calculation is very

very important and it takes its compute bound. If you are recomputing each time the prompt, it will take a long time, but if you have it already cached in GPU or CPU, it can also be cached by the way in in storage, you the the response time will be much higher. So this is why it is important to try to route the request to the model server

that already has a lot of the prompt cached pre-calculated. This stage of calculation of KB cache it's called prefill and the stage of generating the response to tokens is called decode. So this is why KB cache is very very important. As I told you, basically the the model server is doing the work in two phases. One is the prefill, calculating some something that the KB cache on

the prompt and the second one is the decode generating the tokens and these two stages are very different by nature. The first one is compute bound while the other is memory bound. And what it means that we can disaggregate these two calculation to move to two model servers. One can do the pre-calculation while the other is working on other requests. So this is disaggregation of the inference

and this is something that we are also doing in LLMD. It means what it means let's move to the next slide. Basically it means that the scheduler will need to choose not one pod but two pods. One that will do the prefill and the other since Envoy doesn't allow us basically to do some orchestration or to do two calls, we are using a sidecar. We are writing

the request to the decode sidecar which calls the prefill pod asking it to do the pre-calculation and then once the pre-calculation is done, we can call the decode and tell it okay, take the KV cache values. We're Mixell to transfer the KV cache from one model server to the other. This way we are disaggregating the calculation and making it much more efficient. The graph on the right

basically shows you assuming we have four pods each of them with four GPUs like I have in total 16 GPUs. The red line is without disaggregation meaning that all the pods are identical and doing decode and the green one is taking two decode decode pods and and making four prefillers each one of them with two GPUs. It means I have eight eight GPUs for prefill. I have

eight GPUs for decode. This shows us basically how the throughput is we are increasing the throughput by using disaggregation. So again, this is something that's done in LLMD. When talking about I don't know if any of you heard about DeepSeek. These are mixture of experts. These are the state of the art models and LLMD allows us to deploy this in a very efficient way. basically vLLM gives

another level of distributed inference. Instead assume that we have eight GPUs instead of running them as one vLLM, we are able to run on one container eight pods each of them basically they all work together as one vLLM and what we did in LLMD is basically utilize the fact that the scheduler is able to schedule based on load or KV cache by exposing each of them exposing

a port for each one of them and scraping matrices for each one of them. So that the vLLM the scheduler is not only able to say okay, I want to send a request to specific vLLM but also choosing the specific DP it's data parallelism choosing the specific DP rank and it means that we extended basically the scheduler to to be able to look at several ports and

this is something this is an example that I'm showing you because we needed Istio to support it. In the inference pool we extended it to allow us to have several ports for each of the endpoints. This is showing the complexity we are able to do this also with with calculation. It means that the prefiller and the decode each of them are running in this mode. So Okay,

I'm just this is a This slide just summarizes some of updates um and similar to like what Nilly just talked about. Istio's committed to remaining conformant with the gateway API inference extension. So when inference pool when there were multiple originally only supported a single target port but when it was updated to support multiple target ports a community member contributed or maintainer contributed support for multiple target ports

within Istio so it the multiple target ports. Previously it would have failed if uh intermittently it would have failed and only selected the first port. >> Yes, basically when there there was an array of ports and the endpoint picker choose a port, it only looked at the first one and it failed because the others were not recognized. So we needed Istio to extend the support for multiple

ports. So And then another kind of related work to like Istio in the inference workspace, we recently or I recently have been working on adding agent gateway support to Istio so Istio can function as the controller for agent as a as it's like AI specific gateway. And then yeah, exciting today as Nilly has already touched on LLMD was added as a sandbox CNCF project. So congrats to

Thank you. >> We are inviting you all to contribute. It's open source and we're happy to get contribution from the open source community. Yeah, we're we're both both projects are on slack and these are our GitHubs. So if you have any if you notice you want support for additional features or you're interested in contributing fixes or features, yeah, please check out our GitHubs. Thank you. Thank you.