Sponsored Keynote: From Pilot to Production: Scaling and Optimizing Agen... Idit Levine & Keith Babo

This talk addresses the complexities of deploying AI agents in production environments, focusing on the K agent project which enhances Kubernetes by treating agents, skills, and tools as first-class entities. The speaker discusses the creation of Agent Gateway, designed to provide essential features for developers working with AI agents while easing the burdens on platform teams. Additionally, the Agent Registry project is introduced as a solution for managing visibility and organization of deployed agents and tools within various environments. The talk also highlights the importance of community contributions and the integration of OpenTelemetry for metrics and tracing, underscoring the momentum in these open-source initiatives within the AI ecosystem.

So, with the crazy pace of innovation that happened today in the AI ecosystem, the developer, the AI developer in your organization, would love to run everything. And they would love to run it everywhere. Right. Okay, so creating with anything, deploying anywhere is great for pilots and POCs, right? But like not production. Right. And that's exactly the That's exactly what Solo is aiming to solve. All right. So,

filling this production gap, let's start with everybody's favorite runtime, Kubernetes, right? Um unlike the other agentic runtimes you see in this picture, Kubernetes lacks a direct representation of agents. And that's exactly why we created the K agent project to treat agents, skills, and tools as first-class entities in the platform. K agent acts as a general-purpose agent harness like Open Claw with declarative agents, but also supports deploying

uh agents you develop in other frameworks like LangChain and ADK and CrewAI. Agent Gateway helps these developers that are targeting Kubernetes with agents with features they need like memory management, session optimization, human-in-the-loop controls, but it also takes away all the undifferentiated heavy lifting that platform teams need to host production-grade agentic infrastructure. So, next, let's talk about networking and security. When you're looking for a gateway, what are

you looking for? So, the first thing you need, it need to be a good gateway, right? Needs to have all the features that you want to run today, but even more important, you need to [clears throat] be able to run the features that we will need in the future. We built Agent Gateway architected for the down up to basically catch up with the AI ecosystem. Second of

all, it need to be efficient. You need to run at scale without taking crazy amount of your infrastructure resources. We publish a public benchmark that you will be able to take it yourself and run it in your infrastructure and assess how efficient Agent Gateway is. Then, you want community. And even better, an open government community. So, Agent Gateway is a project under the Linux Foundation. There's tons

of companies and users that are donating it and contributing it already right now. And last but not least, you need users. You need users that will run at scale with a real-life use case. And hopefully, those users are happy. So, the next problem we see everywhere is visibility when teams are hosting agentic platforms, visibility into what's actually deployed, right? This is a major problem in the industry.

We created the Agent Registry project to solve this by providing a single catalog and pane of glass for all of the agents and tools, skills, prompts, evaluators that you're going to use across your environments in agentic applications. But it's much more than a static catalog. It supports imports and federation from multiple private and public registries with a curation workflow to make sure that you're only exposing approved

agentic artifacts to your developers. It even supports being exposed as an MCP tool server, so if you want to hook it up to like a general-purpose agent framework like Cloud Code or K agent, you can do that with semantic search. And right after you will have a central registry, you will want to hook it up to one of the runtime. Probably more than one runtime. With Agent

Registry, you can actually register agentic runtime like K agent, Vertex, Agent Core, AI Foundry, and even Nemo Cloud from uh from Nvidia. Once you will register it, Agent Agent Registry will be continuously discover and drift detect drift of your environment. >> [snorts] >> That will help you a lot with your shadow IT. Next, we created a central and unified deployment that you will be able to deploy

all your agentic workflow to those platform. >> We also going to aggregate all your metrics and and tracing with OpenTelemetry. The rain is nice. With OpenTelemetry and and and a hotel pipeline all the way to your favorite dashboard. And integrated with Agent Gateway, you're basically getting a central and unified way place to run all your to enforce all your security as well as your access policy. Okay,

so with Agent Gateway in the Linux Foundation, K agent in the CNCF, over the last year we've seen tremendous momentum in these communities of use and contribution. It's just been incredible to watch over the last year. So, we were just talking about registry and how that is this critical gap in the industry. That should probably be in an open-source foundation as well, right? Yeah, so let's fix

it right now. So, let's donate Agent Registry to the CNCF, right? Now.