Tutorial: KV-Cache Wins You Can Feel: Building AI-Aware... Tyler S, Kay Y, Vita B, Nili G & Maroon A

This talk dives into the advanced aspects of LMD, specifically focusing on KV cash management and intelligent inference scheduling using Kubernetes. The speaker provides a comprehensive overview of how LMD enhances the inference process for large language models (LLMs) by optimizing KV cache utilization during model serving. Key concepts such as transformers, attention mechanisms, and the impact of effective cache management on performance are discussed. The hands-on tutorial engages participants in deploying LMD in a local environment to observe the differences in performance metrics when utilizing naive versus intelligent scheduling techniques. Through this practical example, the session emphasizes the importance of proper orchestration in maximizing hardware utilization and achieving optimal LLM inference efficiency.

Okay. Uh, hi all. Can you guys get here? We're from uh, LMD. We'll be presenting uh, we'll be doing like a 25 30 minutes presentation on LMD deep dive on uh, KV cache and uh, what we're doing there. Then we'll be running a tutorial a hands-on tutorial on uh Kubernetes naive routing for or or inference routing for LM inference and how LMD improves it. So I'm Maruna

Yub. I'm a LMD maintainer. I lead the KV disagregation SIG from IBM research. Tyler. >> Yeah. Hey, I'm Tyler. Uh I am a VLM core maintainer. Um I also work on LLMD and I I lead the VLM large scale serving SIG as well as the uh co-lead the uh LLMD PD disagregation SIG. >> I'm K from do uh and we work for Kubernetes and VM and build

the token factory for with the MD technology. >> Yes. So originally had two more co-speakers who couldn't make it. uh Danny Harik a co-lead also of KV Mikall Mik Mala uh a manager in IBM research for LMD replaced by Vita Bortnikov and Neilie Guy also Neilie both in KV uh sorry in inference scheduling SIG also managing in IBM for LMD also didn't make it so you're left

with us here but I think you'll find the content very valuable So before we start with the presentation, let's all if you have your computer, head to the LMD repo, LMD organization, LMD repo. And on the top right, if you see a star or start section not highlighted with yellow, click it to to unlock the tutorial. And from there we'll go to pull requests KV cash wins

you can feel coupon hands-on guide. So once you're there you'll see a link in the middle for my uh LMD fork. Am I going too fast? So again LMD repo pull requests the first pull request there KV cache wins you can feel middle of the pull request a link to my fork and then you'll get here so while presenting hopefully you guys will be able to install

the prerequisites and uh set up what's needed so we'll need uh kind coup control uh helm helm file yq jq docker and the lmd deps. So since these will uh take some network bandwidth hopefully you'll be successful in getting them ready while we present and then we can begin with the hands-on part. So again we'll be running an LMD simulated deployment on kind locally. So the LMD

simulator will replace actual or real VM deployments and we'll use we'll first use a Kubernetes service for load balancing. You'll see how it breaks, how it doesn't fit inference workloads. Then we'll run the LMD inferenceuler for intelligent inference scheduling which you'll understand throughout the presentation and see how LMD achieves or maximizes your hardware utilization in this case your simulator uh capabilities. Okay. So with that we can

uh start. Is this clear? Like, uh, did you all get to the to the tutorial and set up? Show of hands. What year were you born? Show of hand fans. No, I'm just sorry. Okay, let me change uh to extended display. Okay. So, we'll start with a joke that uh encapsulates the essence of what we're doing here. Two LMs walk into a bar. The first one orders

a beer. The bartender goes, "That will be three bucks." The second LM says, "Beer, but I'm paying cash." And that's pretty much the essence of KV cash management. You'll be transforming your cash bills into cash hits. And we'll understand how that works uh in a bit where Tyler will explains K will explain KV cache and uh transformers and so on. So you also see this in mainstream

a AI providers such as uh OpenAI, Antropic, Gemini and others or Google and others where you pay about 10x less for cash tokens and with this talk or with this tutorial understand where that comes from and how that works. All right. Right. So, I'm going to talk a bit about transformers and attention. Um, mostly I want to kind of uh get a sense of the costs of

LLM inference. Um, especially the the ones that you'll see um as you deploy LLMs and the the ones that you really have to think about uh as you use LLMD. Um, and the ones that like really like integrate with with the optimizations we have in place there. Um, so first of all, are people familiar with transformers here? Like does this look familiar to folks? Yes, kind of.

Okay. Um, okay. So, so good. Um, okay. So, in a transformer, um, you know, so we get some text, we tokenize it, uh, we pass the tokenizers to, uh, an embedding layer. uh these turn into hidden states and then we have a bunch of uh layers and then and then we at the end we have an LM head and then that turns into tokens and then that

ends up getting turned into text at the end. So each layer of a transformer and this this is from GBT1 uh where there's 12 layers each layer has two major components uh attention followed by feed forward. Um so feed forward processes each token independently. It's like matrix multiplications and nonlinearities. um kind of I mean it's not constant as you increase the number of tokens but um it's

it's like not that uh interesting uh computationally. Um what you will want to learn about is attention. Uh so in attention uh this is like the area of the neural network where we mix tokens. So in a decoder every token attends to uh the hidden state of all of the tokens that came before. Um because of that it's stateful and the cost grows quadratically uh with the

sequence length. Um and so we have here we have a a diagram of this. So what KV cache is really doing is it's encoding the state of the previous tokens. Um and it's really like kind of like like lower than table stakes. uh if you don't have KB cache every token you have you have to recomputee everything uh previously so it would really be n cubed if

you didn't have KV cache um and then each token we we uh process uh we insert a vector into our KB cache um the other thing is there's a special thing about decoders where we have this attention mask uh you can see the gray squares ers in the upper uh left hand box. Um that lets us generate tokens. So if we didn't have that, we would have

an encoder tokens depend on the uh future tokens. Uh so you wouldn't be able to generate in that case. Okay. So um I work on VLM and VLM introduced a concept known as page detention. So these stateful KV caches uh we manage them like virtual memory. So each uh block so we'll have like a block uh default block size is like 16 uh recent recently we use

uh 128 a lot. Um but each block even though it's logically uh next to the subsequent block will be somewhere else in uh physical uh KV cache memory. So you know VLM is inspired by you know operating systems and so the analogy V uh in VLM is is an analogy it doesn't mean virtual memory it's not an acronym but it is an analogy to uh virtual memory

uh in an OS. Um another concept that you should know uh is automatic prefix caching. So, um, especially in a multi-turn conversation, uh, we're appending prompts, uh, as we go forward in time. Um, and what VLM does is it automatically reuses, uh, KV cache blocks from previous requests or like previous turns in the conversation. Um we do this simply by hashing the blocks um along with like

the we we hash the basically the prefix of the the tokens. Um so then we can automatically reuse across requests. Um and like as the KV cache hit rate the time to first token dramatically decreases. Okay. One last concept uh that will be very important for LLMD is the concept of prefill and decode. Um so we've basically got two phases of computation in uh in LLM inference.

So prefill is prompt processing uh and decode is token generation. So in you have a prompt you know from the user uh we know all of the prompts ahead of time. So we can in one forward pass of the neural network uh populate the KV caches is essentially what we're doing during prompt processing. It's really compute intensive. Um one we have we're just working with a lot

of tokens. Um and so the linear layers become matrix matrix multiplies and then attention also is composed of matrix matrix multiplies. This makes it comput inensive. Um I'm not going to walk you through the roofline plot but it does push us into a computebound regime rather than a bandwidth limited one. In contrast in decode since we're uh computing with each each token needs to be a separate

forward pass. Um attention is uh like matrix vector multiplies. We have to read the KV cache state for requests every forward pass. So it's really uh bandwidth limited. So takeaway here is these are just really different uh kind of like compute regimes. Um and so it makes sense to treat them differently when you optimize. Okay, so that's all of that is VLM. Now I'm going to talk

about uh LLMD. I'm going to cover uh pre-filled decode disagregation and then wide expert parallelism and then I'm going to hand it over to Maroon. Um okay well first here's an architecture diagram of uh of LLMD. Um so requests come into the inference gateway. Um they we do a gRPC call to the endpoint picker. Endpoint picker is scraping metrics uh from the vm pods every uh 20

milliseconds 200 milliseconds. It is configurable. Yeah. Um so based on that we can do loadare routing. they actually know what's in the KV caches. So we can do other optimizations as well. Um and then we route to the VLM pods that we've selected. Okay. So in LLMD we have these like concepts known as like welllit paths. So these are not endto-end solutions. These aren't like it's not

really installable at this point but it is uh the case where you can follow an example from start to end have a working LLMD deployment and then you can take that well-lit path and modify it for your needs. So one of the well-lit paths uh is pre-filled decode disagregation where uh like I I mentioned before we've got these two phases of LLM inference. we can separate them

into different uh VLM deployments and then we can specialize and typically what we see is we do few uh few or we have many small pre-fill deployments and then maybe one large decode. um and we find that this is best for kind of like medium server loads. Um uh you know really high throughput or really like interactive uh uh SLOs's. We find that aggregated uh really works

better than disagregated. But there's this like really sweet spot in a lot of like practical uh regimes where uh disagregated inference is much faster. Um and this is all all coordinated by the endpoint picker. We have a sidecar on the decode instance. Um and we dynamically decide if a request is going to use PD or not. And it we dynamically decide which uh VLM instances will be

involved in a request. Um and then the next one very briefly is Kubernetes makes it really easy to do um you know largecale multi-node deployments which makes sense fore model inference a lot. Um, so if you heard of like white expert parallelism, um, this particularly makes sense on like the Grace Blackwell uh, systems where you know we deploy with like 32 GPUs in a decode instance and

then some on like we'll have like several smaller pre-fill instances as well. Um and then we showed in the like LMD0.3 blog um the uh you know we we so we can push it up to like uh 2.2 uh K tokens per second per H200 GPU uh even up to like uh 96 GPUs in a decoder. So it like really scales really well. Um the benefit to

this is you can spread the weights uh across a large number of GPUs and then you have a ton of KV cache space. Um and it's really good for like high throughput scenarios especially like uh reinforcement learning workloads. Okay. So with that I'm going to hand it over to Maroon who's going to talk about uh intelligent inference scheduling. Yeah. So this part so far has been AI

or MLheavy and that's super important if you're optimizing uh for inference and we're trying to push for this uh intersection of you know maybe PyTorch ecosystem that is MLheavy and Kubernetes that is the fun part and now we're getting to the uh Kubernetes uh like the part where Kubernetes makes a lot of sense or and and we heavily use it. So before we get there, I want

to show you a simple example of why standard load balancing fails, why AI inference workloads are different from the standard cloud workloads we had uh so far. So Tyler explained a lot on KV cache and why uh orchestrating it efficiently is crucial for effective inference otherwise and we'll see the other ways now you'll be burning through your compute. So with a single VM which Tyler covered uh

so far everything is great. you have VLM managing its KV cache, a VLM managing uh the requests, but in production, you're often going to be with multiple instances, so multiple replicas uh to scale out and you know for full tolerance and that's pretty much the production scene and in there this once unified KV cache is now disagregated. And if you're not aware of that and if you're

not building on that, you'll be just destroying all these optimizations. So a simple example here with naive uh roundroin uh balancing. So say you have a uh multi-turn chat workload or an agentic workload where in every step you're growing the context. So we've covered the state in LMS but it's actually uh like inference is stateless. So every time you send a request to clone or to chat

GPT, you're sending the entire uh context. So say you have a couple of steps in a conversation. You send the first one. Your naiveuler decided it should go to pod A. Now you have your KV cache state in pod A. You took some time to think whatever and sent your second question or followup. And now theuler selects, you know, we're moving in round robin. schedule selects pod

B. Pod B now has to recomputee the entire state. You missed uh the the cache in pod A. You're going to uh experience increased TTFT. Your cluster now has duplication of state on GPU memory which is the scarce resource and you're pretty much uh destroying the optimizations Tyler covered. And this is what you'll see happening like this is the standard scenario with naive load balancing and you'll

see that in the tutorial on the hands-on part hopefully. So just to go slightly beyond attention. So Tyler covered that KV cache for attention but it goes it's very natural for the AI workloads. So in a multi-turn chat again on every turn you're extending the context and you're sending it every time. So it's important to get KV cache hits on your context otherwise you're burning compute. We

also see this in uh agents which is you know the new uh big and important workload. So in linear agents the simple ones multi-turn tool use it's sim similar to multi-turn chat with a few uh extras but again on every step you're building on uh large context and it's important to get a KV cache hit in advanced agentic patterns such as uh tree of thought or parallel

bl branching or uh orchestrating multiple agents. You also have uh a reuse pattern where each agent or each sub agent has shares context with others and orchestrating them accordingly is important for uh cache efficiency and you know cost efficiency. So one well path in LMD intelligent inference scheduling it's uh it's pretty much a no-brainer to have this and then uh it's super essential for efficient inferencing. It

builds or it it balances two routing uh paradigms. One being prefix cache aware. So your router or your scheduler is aware of prefix cache states that are distributed in your cluster and makes decisions accordingly. So if it knows a certain pod has the state, it will try to route there and this creates a stickier routing. On the other hand, you have load aware routing which looks at

uh uh waiting cues, running requests, KV cache or GPU utilization and so on. And this family or this uh u family of of routing algorithms prefer distributing requests so that you're not hotspotting. And LMD finds the balance between the two where you're minimizing your effective compute but not creating hotspots to maximize utilization of your hardware pretty much. And this path builds heavily on the uh gateway API

inference extension. So it's uh it's an established Kubernetes project pushed by Google and IBM under that uh the design there or the architecture is pretty pluggable. That's a main principle there. So you have the concept of scorers and filters. Scorers uh score requests given a certain a certain metric or whatever like prefix cache scorers or Q scorers. filters filter out pods based on like uh you don't

want to even consider pods at a GPU utilization or so on and request control for uh advanced features like uh SLOs's queuing and and so on and in a in a simple benchmark on 16 H100 GPUs in a multi-tenant uh scenario workload that on paper only demands 73% % of your distributed KV cache. So on paper your system should be uh sufficient to have even headroom for

serving uh the KV cache demand. You'll see that naive or only load aware scheduling destroys your throughput and you know you'll get cash thrashing. You'll have duplication of state. You'll be generating much more KV cache than you need to and your system won't be able to uh serve it. And with prefix cache aware routing you see that uh these two lines two techniques which I won't get

into but both of them should uh at some point deliver optimal uh performance. You see that it achieves the maximal or the theoretical uh limit and efficiently manages the disagregated KV cache pool as one. And you see that uh two graphs here effective cache throughput how many tokens you're serving from your cache directly. This is the LMD run. These two are uh well this one is the

naive or random scheduling or round robin scheduling and you see that your cash is pretty effective. In here you see VLM waiting requests where your waiting cues are building up due to this inefficiency while on the left you have minimal queuing and uh good utilization of your hardware. A second important well path is KV cache management. So in this one we h we manage or we show

uh north south KV cache management that is KV cache offloading. So you have your GPU KV cache uh HBM or or GPU memory is pretty limited in size and the most expensive resource pretty much you'll be uh working with. So we utilize uh the available available CPU memory and storage memory to offload KV cache and use it to expand the KV cache pool vertically. And then we

have east west KV cache management which is which tries to implement compute once reuse anywhere. So if you uh computed KV cache on one pod or one uh node, you'd be able to share them across nodes or pods. And we see in a in a in a saturation benchmark uh you see on the top here the GPU cache saturation point the point in which we filled the

GPU cache memory which after performance or throughput drops significantly. If you have CPU cache enabled or CPU cache offloading enabled, you have a larger pool that you can use. So it hangs on for a bit longer, but then also once it's full and can't take anymore, uh you get eviction cycles and it also drops significantly and you have storage which is the largest of them but also

slowest in in I/IO. So you get to a like middle state and depending on how fast your storage you can move this up and down but you get sustained uh uh performance across growing number of users and number of requests. I'll skip through the so this work is mostly in VM. We're pushing it in VLM through a KV offloading connector. We have a CPU implementation and the

storage implementation and soon we'll have uh efficient tearing between the two and so on. Uh I'll skip through the um backends but again another offloading uh benchmark which runs about 2.5 token inputs uh 200 token outputs shows that with storage you get a better sustained throughput as your uh demand grows and that's without adding uh hardware. That's what we're trying to achieve with And with that, I'll

hand it to Cayan to uh discuss how Dcloud is using LMD. Hello. I want to show you some uh really use case of the AMD. I'm from Dcloud and we build a lot of onremise token factory. What is a token factory? factory is we build a hardware and software together and and to output the as much as token as it can. Uh for a token factory there

are some real world scenarios. There are if we want to build a talk factory there are many uh models we need to support such as deepseek and queen for reasoning model and ki and minimax for the agent model and the workload is also very different. Some uh application is chat application and now recently agent is more and more popular. the workload is very different if the agentic

um uh workload can hit more cavy catch and and the latest thing is the hardware is different if in one promise environment so our customers uh has different uh GPUs for example the Nvidia high-end GPU or some other GPU so why is the scenario is so hard because the on is the things are moving too fast the motors are improve very fast in recent years. Um for

example for deepseek uh V3 use the MA and the deepseek the V32 introduce a DSA there so different there are many things uh we need to do uh and another thing is the cost is very credential because the GPU is very expensive more tokens means more magic and the second thing is the SSL is very complex in the left in the uh right side of the slide

is one of the example of S target for example uh the output tokens and the key matrix. Another thing is we need uh when we named good output goodput is uh the uh when uh user can get the first token from the service ser uh server engine and the TPOT is a token uh the time between token and token and we need to ensure the SA. So

uh we this I think is out of balance between the cost as and the SRA and uh the throughput. So our approach with the SLD there are there are four layers. The top of the layer is the inference orration. It can do the gateway scheduling and autoscalings. It's very good at uh AMD and the second layer is the inference engine. we can use the VRM and the

third is a kernel and communication. The kernel such as flash attention or the communication means the the nixel and other things to make the communication faster. It's also very important for inference and the last thing last layer is the hardware adaption. The kubernetes can give a very uh good at it. for example the uh Nvidia GPU operator and something Sentinel and something and the AMD can do

another very good thing is the benchmark. It can simulate the real world traffic to benchmark the data and we need I also need a planner to plan how to uh config it. So our look at our work uh workflow is that when we get uh when we try to build a token factory first we should plan uh we watch which which model we need to serve and

which which uh GPU we have and we can decide uh the KV cache the architecture the PD radio and then the VM settings and then we should turn because uh the VM and the model are improving fast so we can apply the latest technology to it to make it faster and then we need to benchmark with the WMD benchmark so we can find the paral front layer

and get the f the the good out uh goodput out throughput and the next the finally we should we can deploy it and we can uh set up the monitoring and capture the traffic patterns and it can be feedback for the next opting cycle. So this is the is the benchmark result. We can see for the uh if the us if more user using it the total

uh the total to uh token throughput can increase but uh the the one user's latency may be slow down. So we need to uh uh do the uh loop and find the best uh the the line the green line the AMD can make the loop work uh in the router and traffic control layer. the MD intelligence uh inference scheduleuler and the KV catch can work as the

gateway and uh the root uh routing set bar can do the communication between the P prof and decode and next thing is schedule and deployment. We can use the AMD mode service leverage HM to uh run the VM in the on in the Kubernetes and we can also use a leader work set and the Q to make uh it run better and finally the benchmark and best

practice. The MD benchmark provide a a very good benchmark infrastructure and it also introduced the lead pass as the best practice. Uh this is uh our architecture for two profer one decode uh for deepseek TP uh 16 uh 8 GPU per node and uh and uh when uh uh when a user can access these two in way on the top and then go to the ghetto API

then go to the uh AMD uh scheduleuler and finally go to uh using the KV catch aware routting to the per and the perf and decode use the nixo to uh exchange the kv cache information. So uh the the token factory also improving very fast. So we need to we need to to uh next we will try to uh offload the cavy catch from the memory to

the storage and also do mix the training and inference together to make more make the hardware more effective and to do the workload driven autoscaling things. Thank you. >> Okay, last couple of slides. So so LMD now joins the CNCF organization. We're very much looking forward to new contributions and there's place for there's pretty much work all across the stack and places to get into LMD. I'll

present just one road map here or one domain we'll be heavily pushing on in LMD in the coming months which is agentic inference making inference uh better suited for agentic workloads. So in the last year LMD and others kind of got to a very good state on request ccentric inference where uh you generally and generically serve requests as independent uh workloads and now we'll be focusing on

workloads. One of them being agentic inference. We have a we have several domains we want to develop in LMD and establish in the inference ecosystem being some examples here. Uh program awareness on the scheduleuler level. So being aware that there's an agentic workflow running and not just individual sets of requests uh context aware KV cache management. So not naive LRU which can miss or or be not

provide optimal orchestration for agentic workloads and others. Semantic KV cache proactive state management optimizations for agentic workload and others to push on the next order of magnitude in inference So again we we heavily and rely and really welcome new contributors and contributions all across the LMD ecosystem. What you saw here is mostly in LMD KV cache LMD inferenceuler. Uh there's our GitHub uh repos. You can join

Slack, SIG, KV disagregation channel, inference scheduler and on VLM we also have work there on PD YDP and others. We have a SIG PD disagregation I think and uh weekly community calls for each SIG and LMD community call uh actually bi-weekly uh for the LMD community call and others. we very much want to uh see you there. So with that we'll move back to the hands-on section

and we're with the hands-on tutorial we're giving we're experimenting with cloud code to see if it fits these kinds of use cases. So together we'll find out uh if it works well. It should work well but uh you know we'll find out. Okay, let me switch to the uh did did you guys progress on the setup and prerequisites? Does anyone need any help setting something? We have

uh okay Martin here uh Martin Hickeyi from IBM also an an LMD contributor also contributes to LM Cash. He'll be helping us uh with the technicals. So going back to the tutorial screen. Well, we can do uh questions actually on the content and then continue. >> Can we put back up the just the the link to the presentation? >> Hands up. Anyone that didn't get the link

initially, they might have come in a little bit late. Okay, that's great. We're just going to put it back up there. So, if you have any problems, please reach out. Don't don't feel that you can't ask or whatever and we'll come around and try and give you help. I know it's a lot to take at the moment, but even if we get you off the ground, get

you going, I know you can go away afterwards and have a look at it in your own time or come out and ask questions out in the LLMD community if you afterwards. So again to get to the tutorial or to the hands-on guide LMD repo in the LMD organization start it pull requests the first PR and it's in the link there to my fork. >> Okay. Anyone

else got a question outside of the the link to getting set up? Okay. >> Or generally any question on what you saw. Okay. So assuming you went through the prerequisites and you have a kind you have a kind cluster with one control plane three workers. You pulled the images the LMD inference sim UD tokenizer routing sidecar and others that were listed here. There's a prerequisite for uh

GPUs and everything else. Is that separate to the tutorial here? >> No GPUs, just only for GPU. So >> yes. So so this this tutorial won't be actually using GPUs or a real cluster since we assumed you don't have access uh to real clusters. So what we'll be running again is in in a kind environment. will run the LMD infer LMD simulator that simulates VLM and the

rest of the stack will be real LMD. So instead of VM running on real GPUs serving an actual model, we'll run the simulator which uh has basic simulation of latencies but KV cache enablement. So if you have a prefix cache hit, you'll actually see the simulator output uh or latency decreasing. Okay. So, at this stage, >> did everyone go through part zero? Like, did you did you

get >> Yeah. Yes, sir. >> By the way, if you're using an agent, there's an agents.mmd that has all the context needed for this tutorial and you can use you can have an agent walk you through and make sure the parts are met. So, assuming you all have the name space, we'll start with part one. We'll use Kubernetes for load balancing. So with that we're deploying eight

pods of VM but actually simulated VLM >> with a KV cache or a simulated KV cache >> 300,000 tokens. So it can hold the state for this much token tokens and it can't it can't uh like if you get to a state where your KB cache is full you can't serve more requests your request will be I should probably also run it Okay, I I think uh

the best value here is to quickly walk through the exam like the the flow and then we'll just go around and see everyone doing it. Does that make sense? >> Okay. So again the first part we'll be uh uh u eight mistral pods or mistral model deployments with a standard will run a benchmark. >> You have a look and see which one. >> So you'll also have

observability dashboards showing uh uh standard like TTFT throughput and so on. We'll be sending traffic to this deployment. We'll send we'll run the LMD benchmark tools. >> We'll be >> Oh, sure. Sorry. >> But also like Uh, any other questions? >> Let me hide my clothes of shame here. >> Yes. So, with the first part, we'll be running the Mistral deployment or simulated deployment. Then we'll run

the LMD benchmark to simulate a workload of 6,000 token system prompts, 150 users that each ask 1,000 tokens unique question. >> anyone else with a question? >> 1,000 token output and we'll see uh the standard Kubernetes service destroying performance and you'll just at some point see TTFT rising to the timeout point actually and you'll see uh waiting cues all over. Then in the second part, we'll remove

the service and deploy the LMD inferenceuler and rerun the same benchmark. So now I guess one thing in the benchmark is they're requesting like a bunch of RAM and CPUs. Um so I think it would be good to give people instructions for how to dial that down. Uh for people who don't have like 16 GPUs are 32 GB of RAMs available >> Oh, we do kind of

need >> anyone else in the benchmark part. >> Okay, so for the benchmarking if you're not uh so the work >> depends up if you still have a question. Okay, and we'll get around here. >> The workload takes I think uh 12 gigs of memory. So an alternative there is running a simple benchmarking script that runs Uh, lightweight. >> To show you an example. Like this isn't

the one to one output you'll get but what you'll be seeing. You'll be seeing latency of this sort the one you saw in the slide. So your naive or your Kubernetes service will behaving similarly to this with the LMD intelligent scheduling deployment. perfecting uh utilization >> throughputs again you'll be the the Kubernetes naive deployment will be capping at like 40,000 >> total tokens processed uh and the

LMD deployment will be double you'll see queuing times shooting up KV cache usage being more than needed than just flying above 100 and queuing. >> People have got to ask people if they moved along to that stage. >> I think we can like should I run it here or like we go around and see people running it? >> Run it up here actually and see. But where

would you expect them to be now in the tutorial? >> They should be in part one. >> Part one. Show me what part one is. >> Deploying VLM baselines. Uh opening observability. Okay. >> All right. >> And sending traffic. K. >> I have some pictures. All right. I have a good picture of the big road. So I send a message out there. Sorry it was a bit

late. Something came up. >> That's okay. Thank you. >> Okay. Has I I'll just come down to you in a minute. Uh hands up. Who's got as far as the the observability? Okay, we're making progress. Um that's good. I always hate that word observability. There's too many uh syllables in it. Um hands up to the person. There was someone down here a minute that had a Oh

yeah. Is anyone still stuck with bringing images down or anything like that here? Okay, we have a couple of issues with images. Um Kellen or um or Tyler, you might come over to the gentleman over here. There's a few uh issues with images. >> Thanks. >> Okay. Um I'll be right there. Um, just one note, if you uh reload the page to the the PR on LLMD

with the tutorial instructions, I added a link to the presentation at the bottom there. >> Okay. Do we is there issues with the kind load? But um we'll say you can pull it down with Docker manually. Okay, we're having images with that. >> All right, one second. >> That is not the >> Are you guys running it manually or using an agent to go through the >>

sections? Who's using an agent? You guys are pretty hardcore in today's day and age to run things I think there's something. >> yeah. >> So either they go here. This is the actual tutorial or they >> Yeah, this. >> So this So I think there's like a manifest missing in this one. And this just needs to be go to your branch. >> Yeah, this should be Yeah.

Okay. Yeah. So here >> this was this uh we missed updating this. If you're on this page, you should be on my branch on my fork. So just continue from my branch and don't work from LMD main. Oops. Just make sure as well that you're logged into to Docker and your CLI, even if you're doing a kind load. hood cuz it's going to use it underneath the

hood. Oh, it's You're right in the back so they can't see what you're doing. Yeah, I'm trying to Where would you expect people to be now? at the end of stage one like running the benchmarks running the benchmarks is it? Yeah. >> Hands up anyone that's got it past the images problem and got their pods up and run it. Yeah. >> Okay. Uh, is there anyone still

caught in the image the images issue? >> We still got a few. >> PRI. >> If you're waiting on Grafana and Prometheus, it might take a little bit of time to get off the ground. Okay, >> just uh give it give it a couple of minutes. >> What do you want to update? >> I came over and I told you. >> Oh, just the link or the

>> I invite you to the Give me Make me a collaborator. there we go. should have it in the top. Just just get on the top. There we go. Wonderful. Let's seems to be find Find some of the layers of the image. Uh Maroon we have a question here from >> hi um so I was just wondering that uh you have shown that you are using NY

proxy uh and you have you implemented yourself uh I mean the gateway API or you are or you are using something under >> Yeah good question. So the gateway API originally had nothing for inference. So in there you had you could contribute HTTP routes to route traffic given an endpoint or a criteria or something to a service or to an ingress. The gateway adds the concept of

inference pools as a backend. So now you have an inference pool which selects inference pods such as VLM, SG lang and others by labels pretty much. So with a back end of once uh when envoy or a gateway wants to go through an inference pool or select a target to an inference pool, it will be tied to an endpoint picker which implements the envoy external process. uh

protocol. So, Envoy gets the request, it goes to the EP which is the LMD The inferenceuler selects a pod based on whatever you have configured like prefix cache, affinity and whatever. Once the pod is selected, it gives it back to envoy and envoy uh talks to the pod directly and serves the user. So what we implemented is the external uh process or XT prog protocol which the

Kubernetes implements and LMD inferenceuler builds on top of or extends. >> thank you. >> Like it sees doesn't see like subsequent I don't know how to debug this. Take a look. Yeah, we had the You had the >> No question. >> Is there something you wanted to show people? >> You want to show you? >> The one thing seems to be coming back is I'm saying it

to Tyler seems to be one of the images, the very first image that you have to kind It seems to be missing a layer in the image. I don't know why >> missing a So yeah, everybody, if you are seeing uh if if you get to a point where everything works but you can't see anything in Graphana, we are missing some pod monitors on the uh VLM

inference sim. Um, I'm going to I'm having Claude work on a PR uh to get that fixed and uh hopefully it's done uh momentarily and I will put up a PR against Maroon's branch and uh let's try it again. >> Yeah, but the big thing I suppose is that you'll be able to, you know, take this tutorial with you and work on it afterwards uh and play

around with it. That's usually where you get the most value out of. And if you have any questions, uh, people can come out into the LM MD community, can't they run? >> Yes. So, putting aside the graphana, once you run the LMD benchmark, you'll get a rich set of results that show uh the important metrics and provide a couple of graphs that you that you saw on

the slides. So, if you're running L&D you can proceed in all cases. And it's another If you still have a problem with the image, uh it might be worth remove that image that's giving you a problem and pull it down again with Docker just in case it missed the layer coming down because we're only seeing it intermittently. We're not seeing it with everyone at the moment. By

the way, beyond the tutorial for local benchmarking or testing, we have a simulated accelerators guide which deploys uh which runs a deployment with the LMD simulator. From there the LMD benchmarking guide all the tutorial uh can move it forward. Given that we have 10 minutes left, I don't think uh we'll be making it to the end of the tutorial where you compare the Kubernetes native run and

the LMD So what we can do is you can continue attempting to run or follow the tutorial or or I can present. So again, the results you'll be seeing are pretty much what we presented in the slides for the intelligent uh guide where I can go over it again if if that makes sense. Time wise. But I'm also seeing are completely empty. So I'm not sure >>

empty >> in the dashboard. >> That's because there's no dashboard. >> Yeah, but there's like there's like a baseline manifest So before we wrap up, uh so if you have a real cluster like this was supposed to be a like a tasting on on local machines is good for development but that's pretty much it. Once you go home and you have access to all your GPUs and

you want to run LMD. So again, we have the LMD guides in LMD/Guides each guide. So I'll be going through the inference scheduling one. Each guide has an overview of uh what you'll be seeing uh the hardware it was tested on uh a very short very tested and profiled set of instructions that uh deploy the guide or the specific deployment there. So for example the default in

inference scheduling is you can find it in the uh MS or model service inference scheduling directory. So if we go to the values file, you'll be deploying eight replicas of Quen 32B each with two GPUs or accelerators of what you have. We tested with Nvidia H100's. And once you go through the setup of the guide and you run the uh test curl request and see it working,

you can see a benchmarking section that has that runs the LMD benchmark with the workloads we verified and tested. So you set up the run only script similarly to what you have in the tutorial. You export what's needed and you run the benchmark. And here is an example run that was run in Google cloud I think uh that compares uh a Kubernetes service and the LMD deployment

you get out of running this guide. So you see very similar uh patterns. The Kubernetes service just blows up as the benchmark goes. LMD maintains uh the potential li or the like the maximum potential of the hardware and workload provided. So again the workload shouldn't be using the entire KV cache or the GPU memory you have if managed efficiently and that's what you see with the LMD

deployment. Uh there's also just a summary table of Kubernetes standard Kubernetes performance and LMD performance and the deltas and gains you see with the LMD deployment. So again this is the guides we saw the inference in inference scheduling guide intelligent inference scheduling we have a bunch more. So if you want to run a PD disagregation deployment, we have a guide for GPTUSS 120B tested out and verified

on uh 8 H200s with Infiniband or 88 H200's uh with Rocky and uh in here actually we put some tips. the same layout for every guide, how to deploy it, how to verify it's working, and then a benchmarking section. So that's PD. There's also YDP, which is the most advanced guide maybe uh verified on on DeepSeek R1, but this one is pretty demanding hardware wise. So once

you're home, you want to run LMD on actual clusters on and actual accelerators. Check out the guides. It's very recommended to start with the inference scheduling guide. Uh it's the most lightweight. It's the most fundamental/basic but essential in every uh inference deployment or inference serving deployment. And from there you can move on to uh advanced ones like YDP. And I do have a PR up for the

pod monitor fix alleged fix. Um untested at this point but if you if you are brave you can try it out. >> Yes. So, so again for running locally, running locally uh is only suitable for development and even if you have access to a cluster like you can also develop on a cluster in the LMD inferenceuler repo. You also have guides there and how to easily get

started so you can start contributing. We'll be filling up the organization with good first issues. Currently, we're lacking some. We'll be filling them up throughout the week and we're pretty much looking forward to contributions. We're on time, >> I think. Yeah, we're on time. So, any questions or anything before we close? >> There's a question there. >> Okay, thank you. So my question is if you want

to have an intelligent router, how that will fit into this picture that you have uh shown us in terms of picking up different model types based on the requests. You know what I mean? >> You're talking about a router that sits above what we showed, right? Like you have you want to pick a you have a request, you want to pick the most suitable model and move

there. So the closest we have right now in a PR to a guide is uh multimodel serving. So you have multiple models uh but you specify you do specify the model name in the request. It knows how to go to the correct pool and serve it there. We'll be updating guides and we'll be updating guides with uh modeluler or like semantic router and so on integration with

LMD how to optimize model selection for inference given a pool of models. >> Yeah. But like typically like a semantic router lives in a layer above above LLMD. So LLMD's job is optimizing instances of VLM in a cluster. But like if you're trying to decide like if I want to route to like this model or this other model based on the request like there are projects like

the VLM semantic router that like have an encoder model that classifies requests like as difficult or easy or like it should use this model or that model and routes based on that. But that's really like outside of LLMD's responsibility. if you have such >> tal can you repeat the question? >> Yeah. So the question is like if if we have like have we measured the performance like

how does LLMD behave in such a situation? I think like my we h I haven't tested it. My take is like it's kind of like orthogonal to LMD's responsibilities. >> Yeah. But but again, we'll be uh we're actually working on something right now and we'll have a a new LMD repo or sub project at some point that shows these integrations and what you can do with intelligent

model selection. >> Uh we have another question here. So if I understand correctly, LMD kind of shine when you have long context because you can reuse the the cache uh with the uh the KV cache around around your nodes. And uh I think I glanced by your graph that you also benchmark it when you have a mix of uh short and long context kind of queries where

um you have a bingpack kind of problem because you have to kind of maximize the use of all the GPUs trying to understand where to have your um I mean how to allocate your VLM for the particular user. So does VLM actually does this uh this work of intelligently try to fit all the nodes with the maximum number of uh requests based on the length of the

context uh associated to the query. >> Yes. So LMD shines so if you have if one VLM instance can serve your workload you're good with just VM. If you have if your workload is uniform, you're good with just random routing or roundrobining routing. But useful or or or common production workloads are heterogeneous in in nature. So it's not uh the same input or output ratios or lengths

and you have multiple VM instances. So that's where LMD comes in. It makes sure you're utilizing your distributed VLM deployments to their best and you're um you can handle the heterogeneousness of the workloads. Does that answer your question? Do you want to add >> it? it it kind of we there's something in being worked on in LMD um called like the SLOware router that has like an

SG XG boost based like uh model that gets like trained uh like at inference time like during as it's serving requests in the cluster um that's like I think the feature like the routing strategy that we have in LLMD that like most closely matches what I think you're talking about uh would be like something like that. Um the other writing strategies we have are based on like

you know Q length and number of connected requests and like uh how full the KV caches are kind of like more like huristical uh kinds of things. Um things that make it hard to like kind of like precisely know like where what the right BLM instance to route to would be like you don't know the number of output tokens ahead of time. Um because like the LLM

is the thing that's deciding that. Um does that >> Yeah. Yeah. I mean this answer this answer my question. Okay. >> Yeah. Thanks. >> Okay. So maybe we'll we'll pull it in, will we? Yeah. >> Uh so thanks to everyone that turned up today and a round of applause to all the guys here that helped presented and provide this tutorial. >> Thank you guys. >> Thanks everybody.