Lightning Talk: Bringing Google’s Colossus to PyTorch: Rapid Stor... Ankita Luthra & Trinadh Kotturu

This talk examines the transition from compute to storage bottlenecks in PyTorch models as they scale to billions of parameters. The speakers introduce Rapid Storage, which implements Google’s Colossus stateful protocol to enhance data access through persistent gRPC streams, effectively bypassing REST APIs. They discuss how this architectural shift results in less than 1ms random read/write latency, 20 times faster data access, and a remarkable 6 TB/s aggregate throughput, significantly reducing tail latency for random I/O. The session also explores the integration of Rapid Storage with gcsfs and the broader fsspec ecosystem, ensuring high-performance I/O across various data frameworks like Dask and Ray, ultimately helping users maximize GPU efficiency and achieve linear scaling in cloud environments.