Lightning Talk: The $100K GPU Mystery: Why Your AI Training Dies at 99% - Michael Ifeanyi, Google

This lightning talk by Michael Ifeanyi from Google focuses on the critical issue of GPU memory fragmentation that causes distributed AI training processes to fail at 99% completion, incurring significant costs. The speaker explains how traditional monitoring fails to detect this issue and provides insights on how to identify fragmentation before it disrupts jobs. Participants will learn effective configuration strategies to mitigate failures in AI training workflows. The session draws from actual production incidents and emphasizes practical tools to enhance the efficiency of AI training lifecycles.