Keynote: vLLM & Ray Updates - Tyler Michael Smith & Artur Niederfahrenhorst

This talk covers advancements in vLLM, spearheaded by Tyler Smith, the chief architect of inference engineering at Red Hat. He discusses Model Runner V2, which improves efficiency and performance, particularly through a more modularized codebase and asynchronous design. The speaker highlights the broad model and hardware support of vLLM, including its capabilities for multi-modal inputs and outputs, and its advancements in parallelism for large mixture of experts models. Following this, Arthur from Anyscale presents on Ray, a distributed compute engine tailored for AI workloads. He explains how Ray has become essential for managing complex AI data pipelines and scaling reinforcement learning applications, emphasizing its capabilities in optimizing GPU utilization for various tasks.

I'm Tyler Smith, chief architect of inference engineering at Red Hat, and I'm a core maintainer of vLLM. I worked on vLLM for about 2 years, and in that time it's grown immensely. So, from about like 10K GitHub stars when we started working on it at Neural Magic up to about 75,000 now. And we've got thousands of contributors, over 50 companies contributing to it. We've got really broad

model support, over 100 models, different architectures rather. And really wide hardware support both in tree and out of tree. So, now I'm going to talk about a few things that we are working on now and continue to work on. So, first is Model Runner V2. So, this is kind of following along the you know, vLLM V1 that we introduced last year. So, this is a re-implementation of

the GPU Model Runner to make it more efficient. You know, as a user of it of vLLM, you'll see you know, better stability and better performance. But, it's really important for developers to simplify and modularize the Model Runner going you know, we're reducing the Model Runner code from like a 6,000 line file to several files that only like 1,000 lines. So, one of the things we do

is we're decoupling the state, the per request persistent state from the input tensors that store them. And we're moving input preparation to and moving to like an async first design. So, there's no synchronous Model Runner In an really extreme case, you'll see up to 50% performance improvement at high concurrency on a small model for a large GPU. Last year we introduced vLLM Omni. So, this is Omni

modal input and output. So, basically decomposing and orchestrating input encoders, text, image, audio, video. Using vLLM for auto regressive inference. adding diffusion inference for multi modal output, text, audio. vLLM now supports wide expert parallelism on both H200 and GB200. So, this is multi node MOE inference for for large mixture of experts models. The reason you really want to do this is to spread the weights as thinly

as possible across many GPUs in order to maximize KV cache space. This lets you run more requests at large context and higher concurrencies. Um Really important especially for NVL72 systems to you know, scale performance and really feed the beast in the really like concurrency hungry MOE layers. Next up, I'll talk here's here's a draft of our Q2 roadmap. I'll highlight a few things we're working on. So,

one We're going to be hardening the Model Runner V2, making it the default in some cases. In Sig CI, one of the things we've done over the past few months is start a regular release cadence every 2 weeks we cut and then release like pretty close closely afterwards. We're going to continue that. Um Although yeah, like 0.20 will be 3 weeks because we're revving PyTorch and also

CUDA. We are adding disaggregated large scale serving in vLLM Omni using RDMA to transfer input tensors. We're expanding and refactoring online quantization. We're adding elastic and fault tolerant expert parallel. And just improving performance for YDP in general. And we're introducing vLLM IR, which is you can think of as an improved version of vLLM's custom ops. And thank you. Now, I'll hand it over to Arthur. >> [applause]

>> Can you guys hear me? Oh, yeah. Nice. All right. Good morning, everyone. Thanks for joining me. I'm Arthur and I work at Anyscale, the the company behind Ray. And I'm excited to give you an update on the Ray open source project. So, for those of you who are new to Ray, Ray is a Pythonic distributed compute engine built for modern AI workloads, and we recently joined

the PyTorch Foundation. So, I want to start by talking about where the project is today first. Ray has come a long way, and after a slow start 9 years ago, the adoption has exploded. So, last year alone, we saw a 7x growth in downloads. And why is that? It's because Ray was to scale compute intensive AI workloads like reinforcement learning back when these were were like the

exception and not the norm. But, with the rise of multi modal data and [snorts] agentic applications, Ray has become more relevant than ever. So, today thousands of companies are using Ray from AI natives to digital natives to enterprises, and they they all need a foundational layer to support the distributed systems challenges that come with some of the largest and most complex AI workloads. So, Ray serves Riot

Games in gaming and Recursion in drug discovery and AI natives like XAI and Thinking Machines. Let's take a step back. So, since this is the first time we're speaking at a PyTorch conference in Europe, I'm going to quickly walk you through what Ray does conceptually. So, Ray has one really simple primitive for distributed computing, which is the actor. And we'll use that now to build a distributed

reinforcement learning loop that scales across your entire Ray cluster. >> [snorts] >> So, we've got two classes typical for RL. We've got the trainer that optimize the model, and we got the generator that produces data using the same model. And we need to synchronize weights between them. So, we got two methods for that, too. >> [sighs] >> So, Ray is an actor framework, now we can we

can decorate our two classes to turn them into actors. And by doing so, we tell Ray that they require an H100 and B200 respectively. So, now you usually instantiate those classes in your local Python process, but by calling trainer.remote and generator.remote, Ray creates a process somewhere in your Ray cluster with the appropriate resources. So, you can create one of them like we're we're doing here, or you

can create thousands in a in a for loop. And they can talk to each other by calling each other's methods. So, now we can start iterating. We can train our model, we can synchronize the weights, and we can generate new data to train on. So, we've combined training and inference and CPUs and GPUs in a single distributed application. And to better support large models, we announced native

RDMA support last November, and with it large tensors can be moved between GPUs across your cluster. So, with Ray's actor API alone, you can [snorts] create very interesting distributed applications. And with this in mind, it doesn't come as a surprise that Ray dominates in the world of LLM reinforcement learning frameworks. So, as you know, RL provides reasoning capabilities to LLMs, and these open source frameworks here do

this by using libraries like SGLang and vLLM. But, all of them need to orchestrate, and therefore they use Ray. So, companies like ByteDance, Nvidia, and Ant Group choose to build with Ray because it's uniquely suited to distribute complex AI applications. Let's switch gears again and look at an AI data pipeline. So, AI data pipelines have some challenges to them. Multi modal data is unstructured, some stages require

GPUs, and other are more CPU heavy, and they're often pretty large. So, so optimization is key. Let's look at an AI pipeline application that Nvidia built, the NeMo Curator. So, as you all know, model quality comes down to getting the data right, and there's an enormous amount of effort that goes into curating training data. So, you have to clean multi modal data, you have to deduplicate it,

and you have to filter it based on quality. And each of those stages comes with its own set of distributed systems challenges. >> So, for deduplication, you need some sort of awareness over the entire data set, and when you go from text to images audio to video, you go from small models to very large vLLMs that are quite challenging to run. And then, you also want the

pipeline to be efficient. So, you want the stages to be scalable independently of each other to eliminate bottlenecks. So, you never want those GPUs to just sit there underutilized. So, Nvidia saw these challenges when building NeMo Curator, and they turned to Ray. So, with Ray, they can operate on large heterogeneous clusters, and they can stream hundreds of petabytes of data and overlap operations like tokenization and inference.

And they can scale pipeline stages independently of each other to remove bottlenecks. So, last year they reported 99.5% GPU utilization in steady execution state. So, Nvidia is not alone. As we speak, there is um people around the globe uh building with Ray in a thriving ecosystem. And we've seen an explosion of new open-source projects from uh AirBrick from ByteDance to sorts of AI workloads. So, we have

RL uh data workloads, genetic workloads. So, Ray isn't a framework only. There is an entire ecosystem that comes with it, and we are very excited to build that ecosystem with all of you. And with that, I'd like to encourage you to, you know, go go to your laptops and and install Ray and build something that scales nicely. Thank you.