Collisions in the Dark: Illuminating the 95% of Kubeflow You Can't... Amine Lahouel & Laura Llinares

This talk, led by Amine Lahouel and Laura Llinares from CERN, explores the critical need for effective monitoring within Kubeflow, especially as the use of their machine learning platform scales. The speakers share their experience in developing a comprehensive observability stack designed to provide actionable insights into performance, resource consumption, and overall system health. They demonstrate how they utilized established monitoring tools and created essential components to enhance visibility into the workload metrics, catering to both researchers who need detailed data on GPU utilization and Platform Managers who prefer broader usage trends and cost analysis.