Keynote: Rules of the Road for Shared GPUs: AI Inference Scheduling at Wa... M. Muralikrishnan (ASL)

This keynote features Mukund Muralikrishnan, Staff Engineer at Wayve, who discusses the complexities of AI inference scheduling in large, multi-tenant Kubernetes clusters. As the demand for AI inference workloads grows, the need for predictable access to GPUs becomes crucial, especially when managing diverse workloads that compete for GPU resources. The speaker explains the limitations of default Kubernetes scheduling and presents Kueue, a Kubernetes-native queuing and admission control solution designed to enhance scheduling efficiency in shared GPU environments. This approach not only allows for predictable GPU allocations but also improves cluster utilization and minimizes operational challenges. The session concludes with insights on how frameworks like Ray integrate into this model, supporting the scaling of Wayve's AI Driver platform.