Tour De Force: LLM Inference Optimization From Simple To Sophisticated - Christin Pohl, Microsoft

This talk by Christin Pohl from Microsoft focuses on optimizing large language model (LLM) inference, exploring a spectrum of techniques from foundational to advanced strategies. The session delves into model choices such as selection, quantization, and smart routing, followed by library-level improvements utilizing PyTorch-based frameworks like vLLM, SGLang, and TensorRT-LLM, emphasizing continuous batching, KV-cache management, and tensor parallelism. Additionally, the speaker discusses custom implementations including speculative decoding, disaggregated inference, and fine-tuning smaller models. The session highlights practical trade-offs and essential metrics like time to first token, inter-token latency, and cost per token, offering valuable insights for both novices and experienced practitioners looking to enhance their deployment strategies.