Lightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond - Zain Huda, Meta

This talk covers the advancements in recommendation system scalability at Meta, focusing on the implementation of 2D sparse parallelism. The speaker explains how this technology enables the scaling of sparse recommendation embedding tables from 1,000 to 8,000 GPUs, facilitating the largest Ads model training runs in a production environment. The session also delves into the technical challenges associated with handling sparse operations and shares insights on optimizing these systems to enhance performance while reducing memory usage at scale. Attendees will gain valuable lessons from designing high-performance systems that leverage extensive GPU resources effectively.