Least-Privilege for AI: Authorizing Agents and MCP Tools with A... Luc Chmielowski & Nina Polshakova

This talk covers the security of AI agents through the use of the MCP protocol, focusing on the integration of the agent gateway and Kyverno. The speakers, Nina Pushkova and Luke Miroski, discuss the challenges of authorizing agents in production environments, where poor deployment can lead to significant issues. They introduce MCP, a stateful JSON RPC protocol designed for AI workloads, and explain the importance of applying IAM principles to it. The session includes a demonstration of the agent gateway, a data plane written in Rust, which integrates with external authorization using Kyverno. The speakers highlight the necessity of a dedicated AI gateway to manage authorization for various agents and workloads, while demonstrating how policies can be effectively applied to secure these agents.

Uh welcome everyone uh to our talk lease privilege for AI authorizing agents and MCP tools with agent gateway and Kyverno. Um I'm Nina Pushkova. I'm a software engineer at Solo. I was also the Kubernetes 133 release lead. And uh most recently I'm working on a project called agent gateway where I'm a maintainer and it's an AI gateway that integrates with Kyivero. >> And hi everyone, my name

is Luke Miroski. I'm a namatar engineer um and I a maintainer of Keano. Um, and I've been working for the past few months on a tool called Kevano authorization that we'll see later in this demo. Um, today we're going to talk about how to secure basically AI agents um via securing MC the MCP protocol. Uh, we'll see like the challenges of it and how we got a

solution for it. Um recently in the news like if you watch the tech news there is a lot of news like that where a company deploys an agent in production the agent does something really bad and the company has a lot of problems. Um just a quick show of hands who deploys agents in production at the moment no one. Oh yes some hands. Okay. Um yeah so

this is like quite scary for a lot of companies and a lot of people. Um and this is where all of our like our talk started. But first before like getting into how to secure it we need to understand how they discuss with the world right. Um before uh having protocols like MCP and A2A agents had to talk the same language as us like use SQL use

cubes or whatever to discuss with the world. uh it wasn't like really streamlined. So people researched and created those two languages and in fact now it's like a communication standards. Um everyone that is using uh cursor or whatever you you are using MCP servers to do stuff you know. Um so we're going to see why IM usual IM principles apply to MCP. If you look uh into

the OASP just a bit, you see that there are a lot of things about authorization and overall like this is one of the main thing you do when you start creating an app, you work on the authorization, right? So why apply AM to MCP? So MCP is basically a JSON RPC based protocol uh with some key differences. So it's not just design RPC. It also has like

a semantic layer for the LLM so that the LLM knows like kind of what to do with it like every tool and stuff like that. Um it is also a stateful protocol and we'll see later what it's important and uh it has also stuff like capability negotiation uh for between the agent and the MCP server. So right now we are we are at the stage where if

you want authoriz authorization um it's like baked in the MCP protocol sorry and um you have to implement it pretty much by yourself. Uh you have to create all the authorization stuff in your MCP server and do it by yourself. It means that depending on who created the the MCP server, for example, if you use third party servers, um you might have you might even not even

have authorizations. On top of that, if you are a platform engineer trying to handle authorization at scale, um you going to have like audit trails that are in lots of different places and duplication of the authorization logic. So you might want to centralize everything into like a single point of authorization, right? Um and this is where kind of Kano comes in. Um for those who don't know

Keano um it's a declarative um like it's a policy as code engine based on YAML and cell. So cell is like common expression um and it allows you to do like usually for Kubernetes run policies on your cluster so that nothing bad happens. But recently uh we've been working on with other maintainers at making the kano rules apply at edge and um the idea was to use

the same validating policies that we have for Kubernetes and to apply them um on native like HTTP or envoy servers and um I'm giving like the second part of this is going to be like you have to put it on a gateway and this is where a genway comes in. >> Yeah. So, um, in addition to Kyerno, we're going to show agent gateway in action. And, uh,

what is agent gateway? It's a high performance data plane written in Rust. I feel like I have to mention the Rust part first for all the Rust fans out there. Um, it but it's not just an AI gateway. It does route all traffic. So, it supports the traditional um, gateway, you know, system. So, it supports Kubernetes gateway API, the latest version of that 1.5. Um, it's also

an inference gateway. So, it supports inference extensions on top of gateway API. So it integrates with things like LLMD. Um it routes all traffic uh with a unified API. So you can uh use you know open AI and agent gateway will translate that API into anthropic. So you can have one AP one unified API for all of your uh providers. Uh it supports ATA natively and it

also supports MCP which is kind of the focus of our talk. Um it also runs anywhere so it doesn't require Kubernetes. It also runs in standalone mode or in Kubernetes. So there's a a Kate's controller that you can use with the Kubernetes gateway API or you can run it uh standalone just right like as a proxy on your laptop. Um and most importantly for this talk it

supports external authorizations like Kyivero. It actually supports the same uh authorization spec as uh Envoy. So that's why uh Kyerno will work with agent gateway out of the box. Um, but before we get into the inner workings of agent gateway, you might be asking yourself, why do I even need an AI gateway? Like we have plenty of great gateways out there. And uh, I was kind of

also a skeptic a couple years ago. Um, I didn't think you needed a dedicated AI gateway for egress traffic. But um, as the field evolved, I realized there are a kind of a couple new uh, you know, things that gateways need to support for AI workloads. So the first one is LM serving. Um, so your uh inference you need like an inference gate g g g g

g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g gateway to route to self-hosted generative AI workloads. That's pretty unique. Um second, the LM consumption feels like just egress traffic, but um it's actually kind of different than traditional traffic because it's high latency

and you have a high cost associated for each call you're making to the LM provider. Um also your traffic is operating on tokens, not requests. So you're like rate limiting based on tokens, not how many requests you're sending. And then finally um there's body based routing that you have to consider. So you might route to a different model based on the model field set in your body

not based on a route that you've defined. Um and uh in the last couple years there's also been a lot of new protocols that have emerged to support agentic workloads. So MCP and ATA are net new protocols that you know traditional gateways don't natively support. So um you know focusing on M MCP uh why can't we use a traditional proxy for MCP specifically. So um as Luke

mentioned uh MCP is u like stateful. So traditional proxies are stateless. They have rest style interactions. The um you know requests that you're sending are often short-lived. You're dealing with microser traffic. On the other hand MCB proxies uh use a stateful uh protocol built on top of JSON RBC. those requests are scoped to a persistent uh session context and uh those sessions are often longived. So uh

another thing that uh the MCP spec supports is ooth 2.0. So when you have your agents, they need uh you know secure user approved access to your tools and your data. So um and those permissions need to be delegated, scoped and revocable. Um and you don't really want to implement this at the MCP server like for each server. um it would be nice to have a you

know something else to do the work for you. Um and agent gateway actually does implement the MCP OOTH uh 2.0 spec. Um so it translates that OOTH info from your MCP client to what the IDP expects and kind of handles that for you. So there's a couple benefits to this. Uh you need to authenticate only once. So you don't need to repeatedly reauthenticate per tool call. There's

no midsession interruptions. Um it gives you a centralized way to um you know uh have offend and offz at the gateway level and uh it works with uh existing IDPs so you don't need any MCP specific support or changes on your server side. So with that let's get into our demo scenario. So this is kind of a silly demo um that I think everyone's probably familiar with.

Um so in this example we're going to use K agent as our um like the K agent comes with a bunch of different MCP tools. So we're going to use that as our example because I think everyone's familiar with you know how coupooptail works. Um so in our example uh you know you opened up your your agent of choice uh like claude cursor open code um you

you know connect to your cluster um you ask your LM provider nicely like please apply this random manifest or like please generate a manifest for me and apply it to this namespace the production namespace. Uh the LM provider is very nice and gives you the tool to call. Um you have direct access to all of those MCP tools. There's no off in place and the manifest gets

applied and prod goes down. So this is not what we want to happen. And you might be thinking to yourself like oh but I'll have like clusterwide permissions in place. It'll be great. Um but again this is just a demo example to demonstrate how to protect the MCP flow specifically. Um you should have additional security on top of this. But for the example, this is what we're

going to to use. Um, so let's not break prod and uh add agent gateway and kybero to the mix. So same scenario as before. Instead of directly accessing our uh MCP tools, we're going to go through agent gateway. So I, you know, hop in, I log into cursor. I, uh, instead of going to my MCP tool, I'm going to u, you know, have agent gateway front that

uh, MCP tool or MCB server. And agent gateway is going to tell me to use the OOTH 2.0. 0 flow to login. Um so then uh it's going to make me log in through Keycloak. I log in as Bob. Um Keycloak issues me my token. So the OOTH 2.0 flow is complete. Um and then I have access to view and list all of the cinjun tools that

are available to me. So once I have access to all my Cinjun tools, I can see that there's a useful tool to apply manifests to my cluster. So I I decide to try that. um my LM provider tells me which one to to apply. Um but before I actually run that tool, K agent, sorry, agent gateway reaches out to Kyivero to uh check that I actually have

the correct permissions to do that call. Um and Kyverno checks its rules and says, "Nope, you can't do that." And production is saved. So um we don't uh you know, take down prod um you know, the user can still use MCP tools, but only the ones that we allow them to. So with that, let's actually see it in action. We have a live demo. So hopefully it

will uh you know work. Demo gods, please. Um so I have a kind cluster. I'll first do a walkthrough of my environment and then uh show the ooth2 flow and then show how cursor interacts with uh agent gateway and kyerno. So in my uh cluster I have my agent gateway controller running. Um and this is configuring two gateways. One for the keyloak ingress just to expose keycloak

and the other one for the gateway proxy. So this is my MCP gateway that's going to front my my um cinjun tools. Uh these are the kinjun tools that I have running. Let me actually make it bigger for the people in the back. Um so we have uh keycloak u we have the kinjun tools. Um uh those are the uh that's agent gateway controller proxies and then

uh we also have kybero running. So the kybero offz server is also in my cluster. Um like I mentioned agent gateway uses the gateway API. So all the uh ingress uh for exposing keycloak and for our MCP proxy are both uh defined via the gateway API. Um so this is on port 8080. And then uh the gateway API also has this uh resource called an HTP route.

So I define my HTTP route to point to my um my agent gateway backend. That's my MCP backend. And that will use the path prefix /mcp to to route traffic. Um that backend looks like this. So it's a simple selector. Um, so you can actually have like a virtualized backend with agent gateway. So I could actually have multiple targets here. But for this demo, we're just going

to show one. So all I'm doing here is selecting the cinjun tools that I had running in my cluster um in this uh agent gateway backend resource. And then uh for the policy side, I have a um a uh policy to set up my MCP authentication. So since I'm using keycloak um I have to configure agent gateway to know how to use um keycloak for mcp offn.

So I have a policy here to configure that. Um this configures just the audiences uh all like the um you know job key paths all all of that stuff uh for the flow. Um and then the last policy I wanted to show is I also have the external authentication uh authorization policy set. So this is what's pointing uh is attached to my HTTP route. So all traffic

that's going to my MCP server is going to first hit Kyverno and um that's going to run the the policy uh there. All right. So let's see it in action. I think I have my MCP inspector running. So um I've uh uh this is the MCP gateway that I have. I just over it wrote Etsy host to make it a little nicer. Um this is uh the

port we were looking at. So that's the listener and the gateway and this is our path. And if I try to connect now, it's not going to let me because I haven't done the ooth 2 flow. So, um I'm going to go into my authentication settings and we'll just walk through the the flow process to kind of understand what's happening. Um the first step is the metadata

discovery. So, um the policy that we had in agent gateway uh tells us to reach uh out uh to Keycloak. So, we do the metadata discovery and it tells us this is our our token endpoint. Um so, then we're going to register our client. Um, so we we register our client as the MCP inspector. So this is going to be different when you're using our IDE. Um,

we're going to uh prepare the authorization. So this just gives us the authorization URL. So I follow that and then I have to sign in. So um, I can sign in as the unauthorized user, but then I won't have any permissions. So to show uh it actually working, I'll sign in as Alice. Um, that gives me the uh uh authorization code to copy. So this is also

you can automatically do this with the quick flow but we're walking through it step by step to kind of understand what the different steps are. So one once I enter the authorization code I finally get my token request. So when I click token request it'll uh request a token from um my my gateway and then authentication is complete. So now we should be able to connect. So

before we were getting the connection error. Um if you looked at the MCP inspector logs, you get like um you know 403s. But when I connect uh we can now finally connect to our K agent server as Alice. So um now when I list the tools I have access to a bunch of tools. Um uh so these are all the ones that are bundled automatically with K

agent but I think the ones we care about are um let's see like uh the Kubes ones because they're a little easier to understand. So let's look at like uh get uh get resources is a good one. So in all namespaces let's get like the pods. Um if we run that tool uh you can see the the result uh results in something that similar to what we

saw in K9's. So all of the pods in our clusters across all namespaces. So that was the MCP inspector part. So there's no agents yet. Now let's see the same thing in uh cursor with uh you know a more agentic flow. So um if you're using cursor or any IDE you can uh configure your MCP server um with however the IDE lets you do that. So in

this case I have agent gateway defined here. This is the same URL we were using with the MCP inspector. Um so when I click connect it's behind the scenes it's going to do the same flow um just a little faster. So I can log in as Alice again. Um I can grant my uh access to cursor. So this is a different client as you can see. And

now I have access to all of the tools. So this is the same list tools command that we saw earlier in the inspector. Um again a lot of tools. Um but now we can start actually chatting with our agent. So um we can ask so instead of like form like following the same tool spec that we did in MCP inspector, we can use like natural language. Uh

so what name spaces are in my cluster? I can't spell. this should run. Hopefully the Wi-Fi No. Let me see if it uh extension host. Oh no. Uh, >> nice one. >> Nice. Uh, well, we have a video of this working, so I can switch to that instead then. Um, because it it's just to show the flow. Uh, so let me go into slideshow mode. Not skip

that. Okay. Um, so we'll skip to Oh. Um, okay. So uh yeah, so we'll skip to the agent part. Um so when we uh in the chat we can ask it to you know a name the name spaces in my cluster. Um it'll uh connect to the MCP server run that and then um you know list all the the namespaces. We see there's an interesting production namespace

here. Um so after that we can try to apply like a example manifest to production and um it uh should deny us. So let's see if it works in the video. So it says the user wants to apply this manifest. I tell it to yes you can apply it. So that's the URL. Um but every time it tries the MCP call will fail because our authorization rules

are in place. So it's preventing us from doing that. So now I'm going to switch it back to to Luke to uh show the policies in place that actually protect us from from this >> Yeah. So we've seen all the setup with the gateway and how we can communicate with the agents. Now we need like the authorization part, right? And this is where Kano shines. Um so

for this demo what we called in the like the MCP tool we called is a tool called create resource from URL. So basically you send him a YAML URL and it's going to apply it to your cluster right which can be really dangerous if you're trying to authorize everything because you have the YAML you're not able to like check before maybe if it's something bad or anything

but this is where Kano comes in. So we have, you can see here we have a lot of policies. Some of them are some of some of them are wider than the others. I'm just going to like given the time and stuff just do one. Um but basically you might want to have a lot of different policies that go like narrower and narrower. So it's really hard

to say for French person. Um and yeah, so what it looks like is sorry I >> can close. Yeah. >> Yeah. Every time >> over here just wait. Uh >> sorry. Sorry. >> Oh wait. Ah. Okay. Wait. Maybe. No. Cursor is not happy with us. >> Okay. There we go. >> Yeah. Sorry. Uh okay so here you can see like for those who are who are familiar

with like the new Keano policies uh we have here validating policy which is like the exact same resource that we are using uh in Keano for Kubernetes. Um this is important basically this is how we tell okay this is for like the authorization server and not for Kerno. So you can have living at the same time in your cluster policy that policies that are for Kano and

for the authorization server. Uh you see here that we are using envoy as a mode. Uh we have two modes HTTP and envoy. Uh we are using envoy because agent gateway natively like handles the envoy external external authorization um protocol. And then uh in a policy you usually have three big parts like variables that you're going to be able to use in the rest of your policy.

Um a match condition uh that is going to like filter what where to apply this policy and then the validation part. So let's look a bit at the variables. So you have um basically what we have here is we have a lot of um MCP parsing. Um we in Kano added an MCP cell library so that it easier for you to pass a request maybe say okay

the method is a tool call for example um and the tool call name is like the tool call we called in the in and also you can because like you need to pass uh the arguments with certain types in cell um we added a lot of functions like that in the MCP cell So basically this is um the first part of our uh variables. We just want

to get everything from the MCP request to make sure we we are able to to treat it later in the validation. Then we have here a part about GWT. Um here what we are doing basically is we are um trying to do basic a bit like what agent gateway does and fetch the fetch the token from key look because we'll need it for authorization right uh this

is what we are doing we are decoding it uh so this is going to be a variable that's going to handle the decoded JSON the GWT sorry and um now come the interesting part so as I told you before, we are trying to apply an um external YAML. So we might probably want to fetch it first, analyze it and then run the request. This is what this

is doing. Um basically it checks that um it unmarshals the YAML uh from the URL um and then we in the validation we are going to check that um before um it goes into the the MCP server. So we are getting like the resource kind uh API version, API group and whatnot. And this is like one of the most like this is where the real um authorization

happens. We are not trying to um recreate the full authorization flow, right? Kubernetes already does that really well and we don't want to do something on the side. So what we are doing here is creating a s subject access review in Kubernetes which is a like a resource that allows you to check uh for access to a resource and we are filling it with all the variables

we got from like the user the group also the resource that we got from the URL right and here is the like last part of it where we just post that uh to Kubernetes it's going to give us back a result which is like allow out the denied by for example given on the like airback you have user groups you have in your cluster and whatnot and

um in the end like the validation is pretty simple we just look at the s if it's allowed we just send a response and if it's not we send a 43 basically and here you have the match condition basically we are trying to narrow down um which uh tool is called and what method is called so like we don't want to apply this specific policy for every

single tool. You might want to have maybe one policy per tool if you have uh like really really specific things. And yeah, that's pretty much it for um the cavern part and the authorization. where are we? So yeah, with this demo, we try to show you that um native MCP oats might not be enough especially for enterprise and this is where agent gateway can bridge the gap

and bring a unified um authorization layer to your AI infrastructure. On top of that, uh you can protect your agents using tools you already have such as Caverno or any policy engine. and uh whether it's like this method for IM or tools we are not reinventing the wheel we are using tools and processes that exist since like years you know and this doesn't only apply to IM

you can do that with a lot of things for example for model verification uh cosign does a lot of things uh isolation and whatnot so yeah you have already a lot of tool in your clusters like cloud native tools you can use them actually to secure um AI workloads And it's really important to reuse them and not try to reinvent the wheel before doing something else. And

this is all for us. Thank you everyone for listening. Uh you can find the demo here and if you have any questions uh feel free.