Cloud Native Theater | Istio Day: Running State of the Art Inference... Jackie Maertens and Nili Guy

This talk covers the challenges of efficiently running large language model workloads, highlighting the issues posed by limited GPU memory and shifting traffic patterns. The speakers, Jackie Maertens from Microsoft and Nili Guy from IBM, introduce LLM-D, a distributed inference system designed to address these problems using the Gateway API Inference Extension. They explain several innovative techniques such as cache-aware request routing, prefill/decode split execution, locality-aware scheduling, and dynamic worker scoring, all aimed at enhancing throughput and reducing latency for real-world inference traffic. Attendees will learn how to leverage these advancements with their existing Istio installations to optimize inference serving.