Sponsored Keynote: From One Node to Distributed Training and Inference. How the PyTo... Ramine Roane

This talk covers the evolution of PyTorch from simplifying single GPU programming to becoming the primary production tool for distributed inference and training. The speaker explains how the open source ecosystem plays a critical role in managing the challenges associated with scaling AI models, which have dramatically increased in size. As models now exceed 500 billion parameters, PyTorch continues to be widely utilized in research, with a significant number of papers employing its framework. The session discusses various solutions for model scaling, including GPU parallelism, rapid inter-GPU communication, and orchestration using tools like Kubernetes and Slurm. The speaker highlights AMD's contributions to open sourcing their stack, including ROCm and Flight DSL, which enhance PyTorch's capabilities in a diverse hardware environment. The core message emphasizes the shift towards distributed-first approaches in AI development.

Good morning, everyone. Uh, I'm Ramin Iran. I live in the US, but I have a French accent because I grew up here. And, uh, I'm the VP of, uh, AI software applications and product management at AMD. And, I just wanted to spend the next 5 minutes to tell a story that everyone in this room helped write, and that's how PyTorch went from making single GPU programming easy

to becoming the de facto production tool for distributed inference and distributed, uh, training. And, uh, how the open source ecosystem is basically essential for the next step. So, uh, PyTorch really went, uh, from making accelerated computing easy, uh, Pythonic, and, uh, abstracting the hardware complexity. And, that's really caused a rapid adoption. And, with rapid adoption, uh, comes problem. One of them being scale. Uh, AI models became,

uh, bigger and bigger, but, uh, the open ecosystem, uh, is coming to the rescue for that and has been for the past, uh, few years. So, about scale models, open models are now now easily over, uh, 500 billion. I mean, Deep Seek was 671, uh, MyMovie 2 Pro from Xiaomi is 1 trillion, and that's, uh, probably the top two models right now on Open Router, if you

check. Um, so, uh, single GPU inference is not a thing anymore. Even single node, uh, is is really not a thing, even for for inference. Uh, and compute, that's the good news, is growing about 5x a year. And but this model are running 24/7 in production. So, um how do we deal with that with just PyTorch? Uh the good news is that PyTorch uh is, as you

heard, still used in research and training quite a lot. Nine out of 10 papers are done with PyTorch. uh that simplicity that caused scale is also coming with solutions. model scaling basically in involve uh things like parallelism, GPU parallelism, uh rapid communication between the GPUs, between the nodes, between the racks, and production orchestration. And it also has to handle hardware diversity. So, how does the ecosystem come

to the rescue all this complexity? And uh that's basically things like uh Monarch, Slurm, Kubernetes for And then with scaling, there PyTorch, uh there's Torch Titan and RL, uh Torch FT for fault tolerance. And uh that caused uh huge explosion of other ecosystem as well, like Open AI Triton, like uh you heard about Helion, but AMD also open sourced all of its stack, like uh ROCm and

Flight DSL, which is the Pythonic uh DSL for writing kernels, which are vendor independent. And that uh takes me to the key takeaway. Basically, uh PyTorch made uh AI programming easy on single GPU, but uh uh it's uh also now support with a huge ecosystem uh distributed first uh thinking. So, it's really it really moved away from this single uh GPU And uh that open ecosystem and

uh hardware diversity is really key to the next era, and that's what AMD is really uh focused on and working with the Linux Foundation and the PyTorch org and the whole uh ecosystem working on AI. Thank you very much. All right.