Lightning Talk: Step-Aligned Telemetry for Distributed PyTorch Training (Time... Abhinav Srivastav

This talk covers the challenges of distributed PyTorch training, highlighting how misalignment between time-sampled telemetry and step-based training can lead to issues like throughput degradation and GPU idleness. The speaker, Abhinav Srivastav from TraceOpt, breaks down common failure modes in Distributed Data Parallel (DDP) training that are often overlooked by standard metrics, such as rank stragglers and step-time variance. This session demonstrates how step-aligned, rank-aware telemetry aggregation enhances debugging by providing clearer insights into performance issues and relating time and memory usage directly to training semantics without the need for complex profilers.