Lightning Talk: Debugging the Undebuggable: Introducing Torch.distributed.debug - Tristan Rice

This talk covers the complexities of debugging in distributed training using PyTorch and introduces the new PyTorch Distributed Debug Server. The speaker, Tristan Rice from Meta, highlights the challenges of identifying issues in large jobs that involve thousands of ranks. He provides an overview of the server's architecture and features, explaining how it facilitates stack trace inspections, performance analysis, and hang diagnostics across multiple workers. Attendees will discover how this innovative tool can significantly reduce debugging time, enhancing the reliability of large-scale training jobs.