Route, Serve, Adapt, Repeat: Adaptive Routing for AI Inference Workl... Nir Rozenbaum & Kellen Swain

This talk covers adaptive routing strategies for AI inference workloads in Kubernetes, presented by Nir Rozenbaum from Red Hat and Kellen Swain from Google. The speakers discuss the limitations of traditional static routing methods, such as traffic splitting and session stickiness, which fail to account for the dynamic nature of inference requests and evolving cluster conditions. They introduce the K8s Gateway API Inference Extension, highlighting how it utilizes real-time data like queue length and cache utilization to optimize routing. Attendees will gain insights into the impact of these adaptive strategies on latency, efficiency, and cost-effectiveness in managing Kubernetes inference workloads.