Tour De Force: LLM Inference Optimization From Simple To Sophisticated - Christin Pohl, Microsoft

This talk covers LLM inference optimization with a focus on self-hosting models and the considerations involved. The speaker, Christine, discusses the scenarios where self-hosting may benefit users, especially enterprises with specific compliance requirements or research needs. She emphasizes the importance of clearly defined optimization goals and explores key metrics such as time to first token and throughput. Christine outlines the inference stack, detailing the infrastructure, framework, and runtime layers, and presents various optimization strategies. Techniques such as continuous batching, KV cache management, speculative decoding, and disaggregated inference are shared to enhance performance. Throughout the session, she encourages attendees to weigh the importance of different optimization techniques tailored to their needs.

Super excited to have you all here. I'm Christine. I'm a global black belt solution engineer focusing on AI infra. And over the next 25 minutes, I'm going to take you through a tour of how to do LLM inference optimization. The goal is here not to dive into one topic in particular, but rather give like an overview of the field for you to pick and choose the topics

you're going to then dive deeper into. So, the first question when we think about LLM inference is actually should you self-host? Should you self-host a model on your own GPUs? And the honest default is most enterprise customers out there probably shouldn't. Uh the APIs are highly optimized. They are easy to use. And self-hosting does contain a certain setup effort. We will see that once we look into

the stack. So, if you're just getting started, you need to figure out your business case, stuff like that, focus on that. Focus on the agentic stuff and not on running your model optimally on on the GPUs. But, there are three good reasons why you still might decide to self-host. The first reason is you are the provider, meaning inference is actually your business. Yeah? So, whatever you do

to your inference stack pulls directly through to your competitive advantage. The second reason, and I think this will apply to many here in this room, is you're part of a research community, right? You need zero-day access, you need to control the full stack to run your experiments, etc. And the third reason can be also for traditional enterprises if they have specific needs. Those needs could be you

fine-tuned models that you now want to run, or you've got compliance requirements where you need to own your full inference stack, you're in an air-gapped environment where you cannot use an API that is running somewhere online, or doing inference at very, very massive scales, then at certain point in time there may be a payoff to self-hosting. Now, let's say you made the decision, you decided, "Okay, for

whatever reason, I want to self-host." The next, second most important step is to think about what are my goals. And here it really is about being as precise as possible. What do you want to optimize for in terms of metrics? And with that, not only like targeted mean for a metric, but really a list of this is the metric I care about, and this is also a

metric I do not care about. This is the 99% percentile I'm optimizing for. This is the 90% percentile I'm optimizing for to give consistent user experience. The more goals and narrower defined your goals are, the higher the chance you can reach it by applying the right optimizations. The most important metrics when looking into inference optimizations are really time to first token. This is the phase once the

request hits the GPU until the first token appears. This phase is heavily compute-bound because we have to calculate the whole KV values and the attention scores and so forth. And then the inter-token latency, which is then memory-bound, we're reading the whole pre-calculated values to then iteratively, auto-regressively predict the next token. Another important metric is the throughput. The throughput is really about tokens per seconds across all requests.

So this is competitive to inter token latency and time to first token. So if you really care about throughput, you would usually have a batch scenario where for instance the time to first token would not be as important. So it's important to keep those trade-offs in mind. And of course you want to look into cost, cost per token on like the naked GPU, but also total total

cost of ownership because you will need people to deploy your inference engine and so forth, which will also add to your costs. So moving on, this is how I put together what I see currently as the inference stack. I'm sure you've got components to add, but looking into it from three different layers. At the very bottom, I see the infrastructure layer. This is all about managing your

GPUs, managing the interconnect between those GPUs, managing things like routing, load balancing, or auto scaling. Then one layer further up, we've got the framework section. The framework section is super nice for us because it makes our lives easier, and we've got the three major inference frameworks here, vLLM, SGLang, and TensorRT-LLM. And beneath that, we've got the CUDA and PyTorch ecosystem where we might want to refer to

for lower-level kernel And then, we've got the final layer on the top, the runtime layer. And the runtime layer is all about making a single replica run most efficiently. So this is about caching, batching, quantization, speculation, distribution, and disaggregation. Okay, today's talk is going to follow the following optimization scheme. We're going to start at level one. This is like the easy game, low effort, high return on

investment. It's all about model choices. Then we're going to look into how those libraries, like VLLM for instance, can actually help us automatically, more or less, make our inference more efficient, and then we're going to look into custom strategies. So, let's start with model choices. And this is super obvious. I think everyone in this room knows it, but in practice, I often see it for being forgotten

a bit. Use a model that is just smart enough, and not a tiny, tiny bit smarter. Just smart enough. That's really the goal you should aim for. The way I do this is, of course, as everyone else, I go and check leaderboards. And then, as a second step, I go to my favorite AI research tool and pull all the different papers and their appendices. Especially if I

have a use case, which is in like a more niche European language, like Dutch or Finnish, stuff like that. The leaderboards don't really get that. They abstract that knowledge away, right? So, you want to dig into the papers before you do like vibe code it a vibe checking and then custom evaluations on your golden data set to really get hardcore facts. But the second step, especially for

European languages, there's a paper called Euro LLM, which has a lot of helpful benchmarks on all the European languages we have, and also comparison data set. So, that's worth checking out if you're looking into like a niche European language. Next up, I think this is also de facto standard now is to use quantized model, right? What we get out of quantization reduced the numerical precision by a

factor of two, basically, to get a throughput speed of 1.3 to 1.7 per each step. Now, this is one of the few optimization that speeds up both prefill and decode. You can quantize different things within your model. You can quantize the weights, the activations, the KV cache, and the attention. Now, it's mostly safe or nearly always safe to quantize the weights, and we'll find weights and activations

quantized on Hugging Face already, so you can just use it basically. KV cache, we see huge improvements in in that space. If you're if you're thinking about stuff like TurboQuant paper and so forth. But, you have to be a bit careful here for longer context window. And attention can be a bit riskier to to quantize. So, I would be careful with that. When you use a quantized

model, it's important to keep in mind which GPU, and I just put Nvidia GPUs here for simplicity, but we could also obviously add other vendors. Which GPU you're having to which is the lowest native precision level this GPU supports. For instance, like the Blackwell generation, it will support all the FP4 formats. But, if you're having like a H100 a H100 a H200, it will support FP8 format

only for the actual computation. You can still store on the GPU your weights in FP4, but and during the actual computation will be upscaled basically from a precision level. Of course, not giving you the full format or the full support as if it would be supporting FP4 natively. Just to keep in mind. Next up, this is very practical from real-world experience. Use smart routing. Many use cases

do not require a single model, but like one very smart model and one not so smart model basically. So you can either do a classifier based routing which will just add a few milliseconds latency on top of the request or you can try in batch scenarios with the less smarter model first and then if that's not successful with the smarter model. And then one thing I found

really helpful in like very practical practical and user-facing scenarios is to hardcode trivial intense. So if you are like an airline chatbot or building an airline chatbot, you would be surprised to see the amount of request just being like hi, help, what can you do? You know, like those top 20 hardcoded. Like hardcode the answer, you don't need your LLM for that. You don't need to waste

tokens. Basically. Moving on, this is the level one recap. So everything about model choices should give you like a good 50% optimizations easily. Like finding the right right model, quantizing the model, and routing smartly. Moving on to level two. Library level optimization. Now here it's really about what do those modern frameworks offer us that we simply need to apply. And the first thing which is actually very

nice and it's already there so you do not have need to worry about it at all is that those frameworks will give us continuous batching. Meaning normally when we do inference on an LLM, we wait until a batch is full and then we send off this batch to our model {slash} the GPU. Now of course those batches will finish up at different points in time meaning that

the GPU sits idle as you can see by the black boxes on the left. Now those framework take frameworks take care of actually continuously batching those requests together even though they take different points in time until they finish. They are at different lengths, they've got different KV cache, and so forth. So, they get to do a lot of nice work for us in the background. Second thing

what those frameworks do for us is the whole KV cache management, right? So, as we all know, the KV cache thing is like very huge and very massive part also of your VRAM requirements, as it adds up as it's needed for each request separately. And those frameworks basically achieve that they allocate just the right amount of storage blocks at the time it's needed, and not any more.

So, they do not allocate like a complete the complete storage block at the beginning of the request, but rather really see iteratively how much storage is needed. And they also take care that if two requests share the same prefix from a prompt perspective, the KV cache gets reused, obviously. Now, one thing we might be needing if we're actually looking into this from a practitioner perspective is in

a production deployment on like Kubernetes or stuff like that, you will not simply have one replica, but you will likely have like at least three replicas. And what you want to do is normally in a cloud-native world, you you will route to those replicas by load. Now, in this scenario, you might actually want to route on a mixture of of metric between load and the KV cache.

So, you look how's the load on my replica, and how much of the KV cache for this particular request is already present there, and you route based on that scenario. Things like NVIDIA Dynamo, which is another open-source framework that can then be combined with things like vLLM, can be used for such things. Next up, also about KV cache is KV cache offloading. This is something we see

more and more in practice. So, originally our KV cache was only stored in the VRAM, and if the VRAM is full and we need to evict, we just evict what was ever what was not needed uh basically anymore. Now, the challenge with that is obviously our VRAM in the GPU is still very much limited. So, the idea is to offload the KV cache or the KV calculations

we made to lower-level storage tiers, like the host memory, the local SSD, or even some network storage. Now, you might say, "Okay, what is the practical use case of that?" Imagine like me being like the average vibe coder and actually having a lunch break. And during that lunch break, I'm I mean, I see more and more people like uh coding vibe coding still during the lunch, but

let's say I'm actually having a lunch break, right? Now, the KV cache can be offloaded into the host memory from my long uh coding session with many, many tokens, and then can be fetched again after I return from lunch break, while other people can use the VRAM uh for their KV caches basically. Moving on to parallelism strategies. Now, the bigger the model, the less likely it is

it will fit on a single GPU, obviously. So, in that case, you need to distribute it across GPUs. The easiest way, of course, is to get like a big node out of eight GPUs that are all NVLink and NVSwitch connected, and you can simply uh deploy the model in that way. However, there may be certain cases where you do not have such a big node, and you

just have smaller nodes, like a node with two H100s and another node with two H100s. So, a colleague of mine looked into that example to find out what would be um parallelization strategies that work. So, within a node, you can use tensor parallelism to really split up each layer across multiple GPUs. However, tensor parallelism requires like a high bandwidth interconnect because of the uh communication uh that

is required basically, like NVLink and we switch band basically. However, what you can do if you just have two smaller Ethernet connected nodes, but you still want to search like serve like one very big model is use pipeline parallelism because then you can split the layers sequentially across nodes. So, in this case, the first uh 40 layers are on the on the on the left node and

the other layers are then on the on the second on the right node. And at that point in time, you only need to communicate once uh over over the Ethernet making this a viable solution that then we can deploy models like a quant three 235 billion also on such a setup where the the load or the nodes are really distributed. Moving on to summarize section two, I

think this is really table stakes if you want to go into production. Uh continuous batching is a given. Uh you need to think about proper KV cache management. Um most of that is a given, but there are certain things you can adjust like the quantization level and then also ensure that if you're really going into production, the K the cache aware routing and the KV offloading. And

then you can think about if you've got larger models, how do I distribute them more efficiently with like uh the different parallelism strategies. Coming to our final level, custom implementations. Now, this is really if you've exhausted the first two, if you're really advanced, you really want to get your hands into into the actual like engineering stuff. The first technique I want to talk about is speculative decoding.

Now, we know that normally our LLMs are working in an auto-regressive manner, meaning per iteration, they just predict one token that is next. And this, of course, is uh not as nice because we know that our inter-token latency is mostly memory-bound because we need to read all the KV. Um but we actually have a lot of idle compute, yeah? So, the idea is why not use a

small draft model or a draft head to propose multiple tokens ahead auto-regressively at the time. And this small draft head or small draft model can really be just a half a billion to a billion parameters. So, the idea is that our draft head will predict something like So, the input is Paris is a and then it will predict beautiful and historic European. And then what will actually

happen is our target model, our original model, will verify those tokens in one pass. So, this is not auto-regressive. Verification can really happen in one pass and say "Beautiful and is correct, but historic I wouldn't say historic is correct, basically." And then our target model will generate another token. So, it will say "Paris is a beautiful and vibrant, basically." as the output phrase we generate. So, we

get three tokens within one pass, Now, the nice thing here is that uh it's very much a very efficient technique if you get a high token acceptance rate between the draft head and uh the target model. Also, uh the mathematical distribution between the the output of using speculative decoding or not using it will be the same. So, you can get the same output distribution, which is very

nice as well. The most common technique here uh in the market currently is Eagle 3. If you want to check that out, uh co-invented by a guy at uh Microsoft, which uh basically trains such a draft hat on several input sequence you put through your original uh model and then extract out of the hidden layers certain features to build up this draft hat and then predict uh

those tokens. Moving on, another technique uh you might want to look at if you're if you're serving at huger scales is disaggregated inference. So, the challenge is that as we know, the prefill phase is compute-bound. The decode phase is memory-bound. So, neither phase, because they're are like bound by different things, is fully optimized in their natural state and can be competing for the same GPU resources. The

solution here and the idea here is really to use separate GPU pools, where each pool is sized and tuned independently. And the idea is that, for instance, in this case, we will have three prefill workers, and once a prefill worker is done, it will send over the KV cache once to a decode worker, and the decode worker will take care of generating the remaining tokens. There are

ways to also scale prefill and decode workers on the fly, basically, depending on the point in traffic or the point in the the traffic at that particular point in time. This is a use case that is especially worth considering if you've got very high-volume traffic, if you're using large models above 100 billion and if you have pre-fill heavy workloads like agenda coding for instance where you have

to look into long input uh sequences. The final technique I want to talk about is about fine-tuning smaller models. So this is obviously where the world of inference and the world of training or fine-tuning uh come together. So in this case you will have your original knowledge basically into a a larger teacher model and then use that model to generate synthetic data. Uh a use case I've

seen in practice for this is text to SQL basically. Like SQL generation is not trivial but it has a very limited vocabulary basically. So a small model might not be able to do like complex joint statements by default but it can be fine-tuned using outputs of a more capable model to basically be a distilled model that then can be deployed as inference and you can imagine if

you're going down from like a 100 billion model to a 14 billion model, this will save you tremendous inference uh cost and basically uh enabled much faster inference as well. Of course, if you do something like that you will also need to take care of once you switch models to retrain, to monitor the training and so forth. So this opens up a whole new world. So bringing

it all together to summarize, I really think first of all you need to find out is self-hosting something for me. Then you need to find out which level of self-hosting do I want to be at? So in this case, you know, uh you can be on a level where you just deploy a model on a GPU and you know what GPU you're getting, but the whole endpoint

and routing and so forth is all taken care of you by like in on Azure like Azure AI Foundry basically. Or you can be on a way where you're saying, "Hey, I want to go the whole Kubernetes route." And then you're just using like the best practices. Like we've published a repo on self-hosting best practices on Kubernetes, and you do the whole stack yourself, right? So, there's

a a way to it. And then the second thing is really going over those different levels and finding out which configurations and which things you want to consider that makes sense for you, and to know your metrics, what you actually want to optimize. Is it in your case the time to first token? Is it the inter-token latency, or is it the throughput? Starting from the goals and

then working back, I think is the best way to success. Thank you so much.