Keynote: From Inference to Agents: Where Open Source AI Is Headed - Panel (ASL)

This talk features a panel discussion on the evolving landscape of artificial intelligence and open source, with key insights from experts like Brian Stevens, Mark Collier, and Lynn. The panelists delve into the importance of AI inference and the need for robust, open-source models. Brian emphasizes the development of a community inference stack rooted in public domain principles, which serves as a foundation for advancing AI technologies. The discussion highlights the significance of agentic loops and the challenges of deploying AI agents at scale. Lynn outlines the emergence of new networking standards in the cloud-native community, while Mark calls for collaboration across different open-source domains to solve complex technical problems. The conversation concludes with a look at the future opportunities in open source AI.

I like that I get to sit by the glider here. Oh, as long as the collar doesn't come out. We're actually going to have a I won't tell you who you may be able to figure it out, but it's someone who's a governing board member, a TAB member, and a TOC member get inside the glider in just a few minutes. So, make your guesses now about who

it is and then stick around to see if you're right. >> But, this is the fun part. Yeah. I'm excited to have this panel of of folks across the AI spectrum here to chat a little bit about where where we think AI is heading. And could you just each briefly introduce yourself? Start with you, Brian. >> Yeah. Brian Stevens, recent boomerang back to Red Hat through the

acquisition of a company called Neural Magic, focused on inference. Um but, probably more importantly, boomerang back to KubeCon and CNCF cuz when I left Red Hat 11 years ago to join Google, one of the first things I did was set Kubernetes free, which created this. But, I didn't expect this. This is crazy. Mark? Uh yeah, Mark Collier. I started a project called OpenStack a long time ago,

but these days I'm focused on open source AI as the executive director of the PyTorch Foundation as well as the LF AI Data Foundation. And Lynn? Hi, good morning again. Head of open source at Solo.io and your KubeCon co-chair. Well, maybe we start with you, Brian. You know, you you mentioned that that you you were back at Red Hat after working at and running Neural Magic. One

of the earliest people that actually, you know, convinced me how important inference is. Can you tell us like your view on the market and and how it's driving what we're seeing in open source right now? >> I I always put it in like three kind of camps. One is like we need like really robust models. And you saw earlier from Rob like and we want them in

open source as well. Like the advanced reasoning models are now like unlocking new use cases. I always put the second track as agentic. I don't think we knew we needed agentic a year ago, but now we do. But but you've all seen it with like like uh you know cloud code and other things like how the agentic loop matters. But the focus that we've had is just

like with Kubernetes is but what's the stack? What do you run inference on, right? That's the third track and the way the world could have gone is that every accelerator is building their own stack on how you actually operationalize AI inference or all the big model providers could be building the stack on how you actually operate operationalize AI inference. We took a different view and said that

needs to be public domain, a new community inference stack um from the ground up. And um the cornerstone of that has been the VLM which was just born 2 and 1/2 years ago. Um already is now the de facto most used uh inference engine um part of PyTorch. Yep. And then as you heard earlier this morning, the next layer up that's great. That's how you run a

single node. But then nobody runs just a single node for inference. You got to run cluster scale. The algorithms all change. This is how frontier models servers do it. And so that was NRLMD and advanced gateway capabilities. Um when you have an AI-centric mindset, you think well let's just go build this in our own HPC style way, but we want to make sure it's good for not

just frontier models, right? We want to make sure that it's good for enterprise as well. And that meant Kubernetes. And that was why it was such it was so important to bring this new serving technology and new serving stack into Kubernetes. And so where it's at is like I said, already the core part of it has been de facto. It's used by all the new frontier models.

And then now we're already seeing the early signs of LMD capabilities you heard about earlier today being used by the same frontier models. Some of those algorithms are really important. So I I think we're solid. I think we're solid that we got an independent open source really robust inference serving stack. Yeah. Um you mentioned agentic loops and uh you know not to spoil anything, but you're going

to you actually have another aircraft over there on stage and some Agentry demos that you're going to do in a minute. But, Lynn, can you tell us where you think the key open source momentum is in the at the Agentry layer right now? Yeah, definitely. I think a lot of us probably are using uh live coding tools, cloud code cursor, uh Visual Studio Code. So, we're all

running AI agents on a laptop. Uh but, the biggest challenge right now is how we actually deploy and run AI agents at scale, at security, at the governance we feel comfortable. Inside of the cloud-native community, what's emerging is uh the networking layer becomes super interesting with Agentry. Uh there's Kubernetes Agentry networking working group working on defining the next API for Agentry networking so that projects like Istio

and other meshes could potentially implement, provide us security, confidence, uh governance layer on top. Um the other challenge with AI agents, I think, is how do we actually can trust the agent and make sure the performance, the accuracy we need, and have human only approve on the critical steps. So, I'm excited to see some of the movement on that in the open source area as well. A

bunch of open source projects popping up uh in the cloud-native and outside of cloud-native in the AI for foundation. Very, very exciting. Yeah, I think that that's one of the things that uh I mentioned earlier and that that I uh I want want to keep working on. And we have, obviously, an incredible set of technology right here in in the CNCF. Uh but, we also have really

critical projects, like you mentioned, VL and that's in in the PyTorch Foundation. And Mark, as you've taken over the PyTorch Foundation, you've been involved in the infrastructure world as well. How do you look at at the you know, not just kind of the CNCF scope or the PyTorch Foundation scope, but the overall open source movement that we need to bring together with things like the new Agent-K

Foundation that that Lynn just mentioned as well. Yeah, I mean, I think if you if you look at the problems that we need to solve, these very deep technical problems, that they're just a lot of different domains that we need to have experts um really leading the those efforts and those uh experts are doing that work in a number of different communities supported by a number of

different foundations. And so, it's really incumbent on all of us to not be stuck in our siloed thinking and just really look at what the users actually need and what we need to deliver for the industry to make sure we have open source AI for training and for instance agents and all the places we want to go next. And I think as as a concrete example, we

had Uber just spoke, you know, a few minutes ago. And they talked about how they have thousands of models they're training and serving. And to do that, they're using PyTorch, they're using Ray, they're using vLLM. So, these are all uh projects that are supported hosted by the PyTorch Foundation, but then of course they're running it all on Kubernetes and and all kinds of other tools that are

part of CNCF um and and the community is building here. So, we we see those examples time and time again, and we just have to make sure we're thinking about what everybody needs and and kind of and and in a sense uh if you if you look at LLMD, which was announced as a new CNCF project, it's very tightly integrated >> Yeah. with vLLM, which is, you

know, PyTorch Foundation project. And and really the way that I think we should approach this is think almost like we're co-designing or co-evolving all these open source tools rather than, "Hey, let's make a bunch of pieces and throw them out there for users to assemble." You know, that's not going to get us the speed and the innovation that we need. We need to think across communities, you

know, we have people from 100 countries here, so think globally, think across foundations and communities, and and that's how we can actually like move at the pace we need to in the open. And that's already been working. Like like the developers behind LMD aren't in a silo. They're also contributing to VLM to do these advanced patterns. They've been working with the K Gateway team and the K

Server team. All of those things had to change in order to achieve this. Nobody So there were no walls anywhere. It was like developers on developer, you know, doing this work together. Yeah. Yeah, it's amazing. >> So it's already actually happening, but we just want to keep encouraging that kind of collaboration because, you know, the and everybody sees this pace is just unheard of that what's happening

right now in AI, and we want to make sure again the cost training, inference, and agents that we open source is really competitive and delivering, you know, options there for everybody in the market with specialized models and all these things that you you've been talking about today. Yeah, and collaboration I think really is the secret weapon of open source, and it's why we see open source as

such a powerhouse in every technology shift that that comes along. And uh and I think this is the biggest one yet. And and uh and so it's going to be really critical to keep that going. Now, Brian, uh you know, you helped us launch OpenStack uh 16 years ago. You uh you you helped launch Kubernetes in the CNCF with that that we've grown here. Um you know,

you helped uh bring LLMD, and uh what what do you think, to close this out here, what do you think is the biggest opportunity for all of us to be thinking about right now cuz you've got a pretty good track record there. Well, I thought I would be done by now and I would say I'd be doing something else, but I'm here because this is really important

and it's fun. Like this is like kind of like anybody that tells you they know what's going to happen a year from now, don't believe them. Like cuz we're learning a lot as as we go. But I say you the um but it's super exciting. The capabilities that AI will bring, um hard computer science problem, but what excites me probably the most is that this is I

know that I know this sounds bad, but this is one of the first times I've seen open source truly leading this wave. Right? Like and I'm a little worried about like the open source models capabilities to be honest, but from an infrastructure and platform perspective, there's no apologies for what's being built here, right? It's leading. It's leading proprietary. It's leading any single vendor. So, that's a really

exciting thing to be a part of and so any way I can help amplify that thing. Yeah, we're all in. Great. Well, thank you all for the for the conversation and we're going to let Lynn take it away from here and and get to get some more demo action going. All right. Let's give a round of applause to all of our panelists. Great job.