PyTorch Symmetric Memory + NCCL Device APIs: A New Path Towards Multi-GP... Ke Wen & Sylvain Jeaugey

This talk covers the advancements in multi-GPU kernel development using PyTorch Symmetric Memory and NCCL device APIs by speakers Ke Wen and Sylvain Jeaugey from NVIDIA. It addresses the scaling challenges posed by large models and Mixture-of-Experts architectures, focusing on the need for efficient, fused multi-GPU kernels. The presentation introduces PyTorch Symmetric Memory, which facilitates direct GPU-to-GPU memory access, enabling dynamic data exchanges while eliminating CPU overhead. Additionally, it demonstrates how NCCL device APIs enhance performance through in-kernel primitives, featuring practical examples of compute-communication fusion and novel customized patterns, showcasing the evolution of the PyTorch and NCCL ecosystems for high-performance distributed programming.