Keynote: From Inference to Agents: Where Op... Jonathan Bryce, Brian Stevens, Mark Collier & Lin Sun

This panel discussion focuses on the future of artificial intelligence and its implications within the open source community. The speakers highlight the importance of robust AI inference models and the development of open source technologies that support these models. Brian Stevens discusses the evolution of inference serving stacks, particularly through the vLLM engine, which is now widely used in PyTorch. Lynn emphasizes the challenges of deploying AI agents at scale and the need for governance and security in cloud-native environments. Mark stresses the collaborative efforts across various open source projects and communities to address complex AI problems and enhance the interoperability of tools like PyTorch and Kubernetes.

So, I'm excited to have this panel of folks across the AI spectrum here to chat a little bit about where where we think AI is heading. And could you just each briefly introduce yourself? Start with you, Brian. Yeah, Brian Stevens, recent boomerang back to Red Hat through the acquisition of a company called Neural Magic focused on inference, but probably more importantly boomerang back to KubeCon and CNCF

cuz when I left Red Hat 11 years ago to join Google, one of the first things I did was set Kubernetes free, which created this, but I didn't expect this. This is crazy. Mark. Uh yeah, Mark Collier. I started a project called OpenStack a long time ago, but these days I'm focused on open source AI as the executive director of the PyTorch Foundation as well as the

LF AI and Data Foundation. And Lynn. Hi, good morning again. Head of open source at Solo.io and your KubeCon co-chair. Well, maybe we start with you, Brian. You know, you you mentioned that that you you were back at Red Hat after working at and running Neural Magic. One of the earliest people that actually, you know, convinced me how important inference is. Can you tell us like your

view on the market and and how it's driving what we're seeing in open source right now? >> I always put it in like three kind of camps. One is like we need like really robust models and you saw earlier from Rob like and we want them in open source as well. Like the advanced reasoning models are now like unlocking new use cases. I always put the second

track as agentic. I don't think we knew we needed agentic a year ago, but now we do. But but you've all seen it with like like uh you know, uh cloud code and other things like how the agentic loop matters. But the focus that we've had is just like with Kubernetes is but what's the stack? What do you run inference on, right? That's the third track and

the way the world could have gone is that every accelerator is building their own stack on how you actually operationalize AI inference or all the big model providers could be building the stack on how you actually operate operationalize AI inference. We took a different view and said that needs to be public domain, a new community inference stack um from the ground up. And um the cornerstone of

that has been the vLLM, which was just born 2 and 1/2 years ago. Um already is now the de facto most used uh inference engine uh part of PyTorch. Yep. And then as you heard earlier this morning, the next layer up, that's great. That's how you run a single node, but then nobody runs just a single node for inference. You got to run cluster scale. The algorithms

all change. This is how frontier model servers do it. And so that was NRLMD and advanced gateway capabilities. Um and then when you have an AI-centric mindset, you think, well, let's just go build this in our own HPC-style way, but we want to make sure it's good for not just frontier models, right? We want to make sure that it's good for enterprise as well. And that meant

Kubernetes. And that was why it was such it was so important to bring this new serving technology and new serving stack into Kubernetes. And so where it's at is, like I said, already the core part of it has been de facto. It's used by all the new frontier models. And then now we're already seeing the early signs of LMD capabilities you heard about earlier today being used

by the same frontier models. Some of those algorithms are really important. So I I think we're solid. I think we're solid that we got an independent open source, really robust inference serving stack. Yeah. Um you mentioned agentic loops and you know, not to spoil anything, but you're you're going to you actually have another aircraft over there on stage and some agentic demos you're going to do in

a minute. But Lynn, can you tell us where you think the key open source momentum is in the at the agentic layer right now? Yeah, definitely. I think a lot of us probably are using uh live coding tools, cloud code, cursor, uh Visual Studio Code. So we're all running AI agents on our laptop, uh but the biggest challenge right now is how we actually deploy and run

AI agents at scale, at security, at the governance we feel comfortable. Inside of the cloud native community, what's emerging is uh the networking layer becomes super interesting with agentic. Uh there's Kubernetes agentic networking working group working on defining the next API for agentic networking so that projects like Istio and other meshes could potentially implement, provide us security, confidence, uh governance layer on top. Um the other challenge

with AI agents, I think, is how do we actually can trust the agent and make sure the performance, the accuracy we need and have human only approve on the critical steps. So I'm excited to see some of the movement on that in the open source area as well. A bunch of open source projects popping up uh in the cloud native and outside of cloud native in the

AI F Foundation. Very, very exciting. Yeah, I think that that's one of the things that I mentioned earlier and that that I want want to keep working on. We have obviously an incredible set of technology right here in in the CNCF, but we also have really critical projects like you mentioned vLLM that's in in the PyTorch Foundation. And Mark, as you've taken over the PyTorch Foundation, you've

been involved in the infrastructure world as well. How do you look at at the, you know, not just kind of the CNCF scope or the PyTorch Foundation scope, but the overall open source movement that we need to bring together with things like the new agentic AI Foundation that that Lynn just mentioned as well? Yeah, I mean, I think if you if you look at the problems that

we need to solve, these very deep technical problems, that there're just a lot of different domains that we need to have experts um really leading the those efforts. And those experts are doing that work in a a number of different communities supported by a number of different foundations. And so it's really incumbent on all of us to not be stuck in our siloed thinking and just really

look at what the users actually need and what we need to deliver for the industry to make sure we have open source AI for training inference and agents and all the places we want to go next. And I think as as a concrete example, we had Uber just spoke, you know, a few minutes ago. And they talked about how they have thousands of models they're training and

serving. And to do that, they're using PyTorch, they're using Ray, they're using vLLM. So these are all projects that are supported, hosted by the PyTorch Foundation, but then of course they're running it all on Kubernetes and in all kinds of other tools that are part of CNCF and and the community is building here. So we we see those examples time and time again and we just have

to make sure we're thinking about what everybody needs and and kind of in in a sense, if you if you look at LMD, which was announced as a new CNCF project, it's very tightly integrated with vLLM, which is, you know, PyTorch Foundation project. And and really the way that I think we should approach this is think almost like we're co-designing or co-evolving all these open source tools

rather than, hey, let's make a bunch of pieces and throw them out there for users to assemble. You know, that's not going to get us the speed and the innovation that we need. We need to think across communities. You know, we have people from 100 countries here. So think globally, think across foundations and communities. And and that's how we can actually like move at the pace we

need to in the open. >> And that's already been working. Like like the developers behind LMD aren't in a silo. They're also contributing to vLLM to do these advanced patterns. They've been working with the K gateway team and the K serve team. All of those things had to change in order to achieve this. Nobody So there were no walls anywhere. It was like developers on developer, you

know, doing this work together. Yeah. Yeah, it's amazing. So it's already actually happening, but we just want to keep encouraging that kind of collaboration because, you know, and everybody sees this pace is just unheard of that what's happening right now in AI. We want to make sure again across training, inference, and agents that we open source is really competitive and delivering, you know, options there for everybody

in the market with specialized models and all these things that you've been talking about today. Yeah, and collaboration I think really is the secret weapon of open source and it's why we see open source as such a powerhouse in every technology shift that that comes along. And and I think this is the biggest one yet. And and and so it's going to be really critical to keep

that going. Now, Brian, you know, you helped us launch OpenStack 16 years ago. You you helped launch Kubernetes in the CNCF that we grown You know, you helped bring LMD and what what do you think to close this out here? What do you think is the biggest opportunity for all of us to be thinking about right now? Cuz you've got a pretty good track record there. Well,

I thought it would be done by now and I thought I'd be >> [laughter] >> doing something else, but I'm here because this is really important and it's fun. Like this is like kind of like anybody that tells you they know what's going to happen a year from now, don't believe them. Like cuz we're learning a lot as as we go. But I say you the but

it's super exciting the capabilities that AI will bring. Hard computer science problem, but what excites me probably the most is that this is I know this sounds bad. This is one of the first times I've seen open source truly leading this wave. Right? Like I'm a little worried about like the open source models' capabilities, to be honest, but from an infrastructure and platform perspective, there's no apologies

for what's being built here, right? It's leading. It's leading proprietary. It's leading any single vendor. So that's a really exciting thing to be a part of. And so any way I can help amplify that thing, Yeah. We're all in. Great. Well, thank you all for the for the conversation and we're going to let Lynn take it away from here and and get some more demo action going.

All right. Let's give a round of applause to all of our panelists. >> [cheering] >> Great job.