Lightning Talk: Why Logging Isn’t Enough: Making PyTorch Training Regressions Vi... Sahana Venkatesh

This talk presents an innovative approach to identifying training regressions in PyTorch, emphasizing that traditional logging is not enough. The speaker shares a practical pattern developed at Wayve that transforms PyTorch training metrics into effective operational guardrails for large model training. By implementing scheduled short and long training runs and using standardized performance and stability metrics, this method enables automatic regression detection through alerts. The focus extends to the selection of key monitoring metrics and the associated challenges, including false positives and alert fatigue, ultimately aiming to provide a reusable pattern for other PyTorch teams to improve retraining predictability.