Why WideEP Inference Needs Data-Parallel-Aware Scheduling - Maroon Ayoub & Tyler Michael Smith

This talk, presented by Maroon Ayoub from IBM and Tyler Michael Smith from Red Hat, explores the challenges of WideEP inference in large language model (LLM) serving using PyTorch and vLLM. The speaker explains how traditional routing methods overlook the placement of key-value (KV) caches across data-parallel replicas, leading to inefficiencies and increased latency. The session details a proposed solution that modifies routing to be data-parallel aware, leveraging existing signals from vLLM to enhance cache locality and load-awareness. Attendees will gain insight into how these routing adjustments not only improve throughput but also optimize the use of available resources in WideEP inference scenarios.