No Pain No Drain: Lessons From Node Drains at Scale - Ryan Hallisey & Natalie Bandel, NVIDIA

This talk features Ryan Hallisey and Natalie Bandel from NVIDIA, focusing on the challenges of draining nodes at scale in Kubernetes environments. The speakers share insights from their large-scale operations, emphasizing a workload-aware approach to automated and safe node drainage. They discuss an algorithm that selects nodes based on workload distribution, GPU utilization, and overall cluster capacity while managing both planned and unplanned maintenance to prevent overlaps. Attendees will gain valuable knowledge on detecting and recovering from stuck or incomplete drains, ensuring safety and enhancing operations for those managing GPU workloads, whether in cloud or on-prem environments.