DualPipe from Scratch: Implementing DeepSeek's 5D Parallelism in PyTorch - Dev Jadhav, ING Bank

This talk covers the implementation of DeepSeek's 5D parallelism and DualPipe in PyTorch, addressing gaps left in the DeepSeek-V3 paper. The speaker presents an open-source reference implementation that has been verified against the original architecture, emphasizing the learning and extensibility aspects. Key topics include the sharing of K_pe across heads in decoupled RoPE, the timing of bias updates for auxiliary-loss-free load balancing, and the warmup formula that minimizes bubble overhead. Attendees will gain insights into how to build a bidirectional scheduler that achieves significant throughput gains and explores hierarchical communication methods to enhance efficiency in large-scale GPU setups.