Lightning Talk: KV-Cache Centric Inference: Building a State-Aware... Maroon Ayoub & Martin Hickey

This talk focuses on KV-cache centric inference, highlighting the significance of state management in LLM inference, particularly the KV-cache which is crucial for reducing latency in production environments. The speakers, Maroon Ayoub and Martin Hickey from IBM Research, discuss how traditional optimizations around compute have become less effective as the bottleneck shifts to state management. They introduce a novel approach where cache serves as the central organizing principle of the serving platform, encompassing tiered memory management, cross-replica visibility, and cache-aware scheduling. The session details how the tools llm-d and vLLM are utilized to implement these concepts, illustrating practical applications through benchmarks, deployment patterns, and insights from their experience building a robust KV-cache centric platform.