Enterprise Challenges with MCP Adoption - Christian Posta, Solo.io

This talk focuses on the challenges and approaches to adopting Model Context Protocol (MCP) in enterprise environments. The speaker, Christian Posta, discusses the journey from pilot projects to production implementations, highlighting issues surrounding security, observability, and compliance. He introduces MCP as a unified protocol designed to facilitate communication between AI models and external data and service endpoints without the need for customized coding. The session emphasizes the importance of security practices in enterprise settings and critiques the initial lack of enterprise-focused features in the MCP specification, including necessary authentication and authorization methods. Practical solutions for integrating MCP into existing infrastructure, such as leveraging agent gateways and enhancing cross-identity token exchange, are explored. Through demonstrations, Posta illustrates how to implement these concepts in real-world scenarios, encouraging a collaborative approach to refining MCP for better enterprise readiness.

First of all, thank you all so much. The KubeCon organizers for having me. Thank you all for showing up in the middle of the afternoon here. Um I know it's been a a long conference, maybe a long day. Uh I'll try to make it worth your while. Some of you may be here to try to find a nice quiet place to take a nap. I appreciate that

also. Um we're going to be talking about MCP adoption in the organizations that I've been working with uh over the last year or so. And uh and some of the challenges that they've experienced going from pilot and and POC into how do we get this into production and enable more of the organization to adopt this type of technology to build AI agent uh type applications. Uh and

and do this do this safely and do it at scale. So, uh let's get going. My name is is Christian Posta. I'm the global field CTO at a company called solo.io. Uh we work on cloud networking, API gateways, service mesh. We're creators of Istio. You may have heard of that. Um and then more recently over the last year, year and a half, uh K agent, agent gateway.

Uh this morning we announced our donation to the CNCF of our open source agent registry. So, everything we build around AI um um networking and security uh for platform engineers is rooted in open source. We are uh very heavy contributors to the open source communities. All right. So, we'll be talking about MCP or model context protocol. Some of you may have uh heard of it or may

know of it, but it is a protocol to allow our AI models to talk to and use uh data and functionality that live outside of the uh of the AI model. And the idea is instead of custom writing code to talk to particular API or particular database or a a SAS uh set of endpoints or data, let's unify the protocol layer so that we don't have to

have this custom code and we can just plug and play any of these adapters uh as we need. And uh and eventually the model could potentially discover these on the fly and and use data as it as it needs. Now, just like anything in AI these days, we've seen MCP hit the uh you know, the the hype wave. And um you know, starting from it it's released

in November 24 up through, you know, even even recently. I would say in in my own experience, around the middle of uh like let's say the summer of 2025 is when we saw enterprises start to go, you know, headfirst into MCP adoption trying to figure out what it looks like inside their enterprises. Um by then is around when the spec kind of introduced more things around security.

Even though they probably aren't targeted, the uh the you know, the spec uh what it says about security and off isn't targeted primarily at enterprise. So, there was a number of open questions and uh on they were unsure about how enterprise would adopt this. And uh and a number of other, you know, challenges that became evident, became clear. Uh things around security for sure. Things around tenancy.

Um things around observability and logging and uh compliance. How are we going to live in this agentic world uh where we're dynamically maybe calling MCP servers and and how do we stay compliant, especially for our regulated industries or enterprises? the none of this was lost on the MCP community. Uh actually not that long ago, a couple weeks ago, they uh they wrote their future-looking road map uh

blog where they cover four different pillars about what they're going to focus on here. One of those was enterprise uh readiness. And they admit that yeah, up until now enterprises have run into some uh these predictable problems. And the way they acknowledged it was "We're not really sure what to do. For those people in the enterprise, please come help because, you know, we want your input and

and that's how we want to move forward." The more unsettling part of uh of that part of the blog was uh "Well, we don't want to burden the core spec with enterprise needs. What we want to do is add them as extensions." Which, like I said, is a little unsettling for me. Uh it's almost like MCP trying to build an airplane and saying that "Uh we'll figure

out the doors later. You know, as you're boarding, bring your favorite door. We'll figure it out before we take off." Um and I don't think that's the right way to uh treat this and to uh approach this. Um and luckily, we've had quite a bit of experience adopting and uh scaling APIs. Uh and uh you know, 10, 15 years of uh of security practices around things like

microservices and clouds and platforms that we can leverage and uh and use as we adopt MCP. What we want to be careful is looking at the spec as this is the way to do it. And uh the spec says we have to do this thing that we never would have otherwise done in an enterprise. So, let's just let's just do that. Um we we we should uh

allow some of the existing practices to uh to take precedent. As I said, you know, I've been seeing this uh this protocol is being adopted and people are moving forward with it. And uh the few things that I'm going to talk about today are things like discovery and onboarding. We'll talk about the MCP off part of the spec. Um and then we'll look at uh some more

complex things that that come up like trying to call out to SAS MCP servers or uh APIs. So, let's get going. I made a comment earlier about how the spec really wasn't originally designed with enterprise needs in in mind. And one of the main things that we see or saw earlier was uh the uh the idea that for MCP, what you would do is take a standard

in uh a SCDIO server, deploy it locally, deploy it on your machine with your agents, um and then use that to connect up to uh file system or APIs or whatever it wanted to call. for POCs and proofs of of uh you know, pilots, that kind of stuff, innovation trying to get moving quickly, this was okay. Um but as the enterprise looked at how do we scale

this out, what they saw um was something they didn't really like, which was if these MCP servers need to communicate out over the network and call APIs or call databases or SAS endpoints, uh we need to put credentials into these uh these servers. API keys, personal access tokens, these long-lived bearer credentials that it's very easy to lose track of these. Um if anybody gets a hold of

them now, they can use them. They're fairly coarse-grained, uh long-lived. And so, um you know, jamming API keys into everyone's desktop and and and the proliferation of this across desktops is not what the enterprises want. Um and and never mind that what's happening here also is and in many times they were calling out to these SAS MCP servers directly. They had no idea what was being called,

with what data, whether they were compliant. You know, if something happens, can they show what happened? They had no idea or they're they were having no idea with uh with this approach. So, uh the first main step is if you're going to adopt MCP capabilities that call out to remote endpoints, those MCP servers should be remote. Right? And the the spec does allow for the uh the

HTTP transport. Now, it didn't start off. It was a little bumpy uh when it uh introduced. It started off with this SSE thing where you had to open up a couple connections to make this work. Now, they've kind of figured some some of it out with the transport or the streamable HTTP Um and uh and and so now we can offer MCP servers over over remote. Now,

this gets This is the first first step. Um and uh but but this alone doesn't solve the problem. What we need to do is a familiar pattern that we adopted with APIs is we need to enforce consistent authentication, authorization. Um we need to track telemetry, access logs, um you know, traces, distributed tracing, and uh and enforce policies about who's connecting to what uh what back-end services. And

uh MCP is, like I've been alluding to, so familiar to what we did with APIs. It's not exactly the same because with APIs, we could just uh you know, the request comes in, you get one request, one response, right? And we can just look at the headers and quickly route it. MCP is very It's in the body. All of it's in Uh one request could generate multiple

responses. So, the the gateway technology, the proxies need to evolve a little bit here. Um and so that's why you probably see uh MCP gateways. We've built agent gateway at uh at Solo. And uh you know, connecting to these things is one thing, but then defining an approved catalog of these remote MCP servers and then from there deriving the policies for how you expose these MCP servers

and how you make them available to the enterprise, um you know, that's that's still an an open question where a number of uh folks are looking at using things like a um and a gateway together, which I'll demo here in a second. You know, like I mentioned earlier today at the keynote, we did uh donate our agent registry open source project to the CNCF. So, let me

quickly walk you through a a simple demo. Now, I know a lot of the stuff I'm talking about here we could probably go each one of these topics we can go an hour or longer in. Please find me afterward. I'll be around either around here at the Solo booth if you want to talk in in in depth. Or if you if you disagree with some of the

things that I saw, I'd love to hear your perspective also. All right, so um our for our first demo, what we're going to take a look at is uh we have our agent registry right here. Hopefully, you can see that. Hopefully, that's we'll get it a little bit bigger there. Um and so, a registry.ai is open source project. It's a registry for MCP servers, agents, skills, a

number of artifacts that you might use in your agentive view. deployments. What we're going to do is we're going to we're going to say we have this public MCP server, this server everything. If you're familiar with the server everything sample MCP server, we've made it remote. Um and what we want to do is we want to take this server everything and we want to expose it into

our enterprise. And so, we'll we'll have some process of doing that, but ultimately getting it into into the registry. So, if I click add server here, I'll give it a name. Uh Give it a version. And then we'll add this as a remote MCP server into our into our catalog. And you can see we're going to be If I zoom in, can you all see that? Yeah,

sweet. We will add this as a streamable HTTP endpoint. We'll add the the MCP URL here, and then we'll create the server. So, now we have it in our catalog, we have it in our registry. Yeah, if I come here, we can take a look. We can do things We'll see the overview or description. We can add more stuff to that. We can see that this is

a remote MCP server. This follows the the JSON format that you see from the model context protocol official registry format. We can add things like scores. So, pulling in information from GitHub, pulling in OpenSSF scoring, and give and give things a score. And then lastly, what we can do is generate the required configs to deploy this into Kubernetes. So, now this gives us a way from the

registry to deploy into Kubernetes and expose this on our agent gateway, for example. So, if we take a look at demo one let's go ahead and run that. Can you all Can you see that all right? Go ahead and let's close this up. All right, so now we'll I'll show you if we call the server everything directly and we filter its response we can see the tools

and the and the descriptions what we should see is a response. You can see what tools are available in in server everything if you go directly. All right, but we and hopefully the internet cooperates here. All right. And so, you can see the tools here. But we don't want to go directly. We we want to go through the agent gateway and we've already added that. We use

the CLI to inspect the agent the registry, we can see what we have there. And then we can get the the manifest, the Kubernetes manifests. So, this is what exposes that register that MCP server on the agent gateway. And and now we can actually apply apply it. So, we'll do a cube apply here. And now that's running or should hopefully. Internet's quite slow. Um to the uh

to the gateway. And now if we try to call it through our gateway, we can see that this is this is available through the gateway. Now, this is just the first step. This doesn't solve all the off and other things I'll talk about in a second, but this is just the first step. Get it into the registry and have a controlled way, probably through CICD or GitOps

or something to get these these MCP servers exposed through the gateway. So, hopefully this returns. It does. We're going through the gateway now and that's great. Now, let's come back to the next part of of the talk here, which is around MCP The MCP off was not introduced initially with the spec. But it took until June until we kind of updated some things and and I at

least gave an acceptable answer. Previous previous versions were completely out of line with what enterprises would expect. But in June, they kind of figured it out. And that's probably right around like I said, in the summer of '25 when we saw enterprises starting to make their way down the the path of of MCP. The problem with this is it was it's heavily focused around things enterprises don't

really do. Things like anonymous dynamic client registration. That was probably the biggest thing that stood out. The second uh big thing was it was heavily oriented around OAuth specifically the authorization code flow or grant from OAuth which I would argue, you know, is built for the public internet. Where a user maybe my, you know, Google Gmail or calendar or whatever, you know, I want to give some

client delegated access, scoped delegated access to my data. All right. In enterprises, that's not really the way they look at it. Inside inside the enterprise, they don't typically do three-legged authorization code flow OAuth. The enterprise knows who the employees are. The enterprise owns the data. The the employee doesn't own the data. The enterprise typically knows about the APIs and services and databases and so on inside the

enterprise. What they care about is who can access what, when, and where. And enforce that policy and show that that policy is enforced. Um so, a number of the components of the MCP off spec apply to public SaaS or public internet type use cases, but a number of the RFCs that they mentioned you know, aren't consistently implemented by providers. Things like dynamic client registration or DCR, they

are not anonymous DCR is not used in the enterprise and won't be used in Some providers like Azure don't even support it. Um and and and like I said, OAuth authorization code flows, yeah, you might use it with your business partners or your customers might use it, but inside the enterprise that's not it's not typically typically used. A big part that was left out is how do

we Okay, you get access to an MCP server, but an MCP server might have 90 tools. Uh if you look at the GitHub Copilot MCP server, there's a lot of tools that it exposes. So, MCP off doesn't say anything about how to filter this tool list to any particular user. It doesn't say anything about how to block access to any specific tools. Uh so again, coming back

to a familiar pattern that we can use where instead of tying our off to OAuth and you know, what what's appropriate for public APIs, what we want to do, at least in the initial cases, is tie access to our internal identity providers and user single sign-on. So, what that means is a user comes and installs an MCP server on VS Code. When they sign into something, they

don't sign into GitHub or, you know, what whatever, you know, Figma, they sign into Okta. And or Azure, whatever the identity provider is for the enterprise, and the enterprise then knows who the user is. Now, when the user makes a call to this MCP server the gateway can apply policy. It can filter tools. It can block specific tool access based on who this user is. Now, more

advanced use cases where the user might not be sitting with the agent, we will want agent identity. And you know, that that should be part of the policies that get enforced here. Um but why don't I show again a quick demo? We'll continue with the Um we'll go here, exit that. Make this take up the whole Oop. Yeah, whatever. All right. So, the first thing that we're

going to do is I'm I'm going to walk you through what the OAuth spec says to do. And if we try to call the GitHub Copilot MCP server, you'll see that it fails because I don't have any off attached to the request. But if I try to call it I want to see the headers. I want to see the the response. The spec says that if you

can't uh proceed, you can then throw 401, but then also return a the WWW-Authenticate an attribute that points to additional metadata that the client can use to go find the off server. So far, this is Use this public internet or inside the enterprise, that's fine. Um now, if we go take a look at the GitHub off server uh what we We see or or at this protected

resource metadata. What we can see is that it says find the off server here and these are the scopes that you want for accessing this this this API. So far again it's probably this is this is this will this will fit within the enterprise. Now if we actually look at the GitHub metadata the API the off server metadata we can see we have an authorization We have

a token endpoint. We don't have anything for dynamic client registration. GitHub's IDP does not support dynamic client registration. So what do we do? We need to have a static client. Now if you look at GitLab they take a different approach. They do want to support the off spec completely and so they do offer dynamic client registration endpoint. That won't be applicable in our enterprise and if you

do run into solutions that are saying well we can do DCR inside your enterprise or just go ahead and do it. You should pause for a second and think about that. What you really should do is tie into your Octa or your Entra and use client IDs from So what we're going to do is using the agent registry CLI again what we're going to do is we're

going to that went a little fast. We're going to call with a with a flag that will output our configuration that enables the MCP off so MCP clients can see it or connect to it but specifically pin it to our single sign-on provider uh oops sorry internal provider. And we're only going to request uh um OIDC for for single sign-on. Right? So now if I go ahead

and apply this to our server everything now when I try to call this MCP server I will be challenged following the MCP off spec up to the point where we try to do dynamic client registration or a authorization code grant. So if we try to call it through our internet allowing we should see that it will fail. Right? Because we don't have the off we don't have

the token applied. so if we look at the headers we should see that we return now agent gateway automatically returns the right resource metadata URL here. And then now if I were to go to a um a client ooh that I was not expecting hold on. Port forwarding shut down when I ended the the demo there. Let's see if I can hurry this up. come on. We'll

go to our agent gateway. And we'll port forward it real quick. All right. So now we come back to here we come to our our client here and we connect into the agent gateway for server everything. We open up our off settings we'll do this step by step. Uh first step is discover the the metadata after that 401. Crossing fingers. Right? We want to use a uh

a configured client ID here so that client registration goes quick. And now when we prepare our off and try to log into the the off it will point us so if I follow this uh this URL you can see it's going to our internal identity provider. So if I open this up this will ask me to sign in. Well I guess I'm already signed in. Um we'll

give the app permission here. Now we got to copy so this is this is for illustration purposes only uh exchange happens automatically in uh in off. and so if we follow that step we'll get the token request. And we get our our token we can see we get our access token and ID token and this is the token from the identity And so now if we continue

to connect we can hopefully it'll connect. maybe. Yes all right. So now we can see we're we're connected here. If we list our tools now we can see the tools of server everything which again managed by the registry deployed agent gateway now controlled through off from our IDP. Now from here we can apply policies that limit tool tool tool visibility depending on the user or or claims

in their token or groups or entitlements that they're part of. But so agent gateway gives you a lot of flexibility um here to do that. Now I am kind of running short on time. I do want to cover this part before we get going. Um and that is now calling across identity domains. So we talked a lot about internally in the enterprise but enterprises do you know

these SAS Atlassian GitLab ServiceNow um Data Bricks they're offering hosted MCP servers and we want to make them available in the enterprise but we don't want to allow our users or agents to call and connect directly to the to these SAS MCP alternatively if we build our own MCP server internally we might want to connect to the rest APIs that these providers offer. Again not directly want

to go through the gateway but there's a couple of challenges here. The first is on the left hand side of the gateway which we've talked about we can tie this into single sign-on. We can tie this into Octa Entra so on and identify the user and apply policies now right? But once we get to the the going up to the the SAS provider they require a scoped

delegated off access token. By the time the call gets to the gateway the user the browser all that stuff is out of the picture. So now the gateway has to figure out how to do off with the with the provider. doing things like federated single sign-on maybe you can go to Data Bricks directly and instead of using Data Bricks as IDP it uses Octa to use your

internal IDP. So federated single sign-on. That doesn't solve this Because SSO identifies the user so it authenticates the user but getting to Data Bricks and using scoped access tokens for Data Bricks is authorization and those are not the same thing. And not every provider has a way to map those identities to scoped access tokens. Now there is an initiative in you know in the communities or in

the industry in off specifically for cross app or cross identity app access. And the the approach the idea here is internally within the enterprise we have a single location where we can manage the policies about whether an app is allowed to communicate out to these upstream SAS providers. And so what happens here is we can take the identity token or identity assertion exchange it if allowed on

the identity provider for a jot assertion grant token a jag token that can then be taken and presented to the upstream identity provider which can do the swap for the access token. That's the idea with this flow. I like this flow it would be amazing if we could do this but I don't know if you've been paying attention to off or how long it takes to get

specs into actual production could take years. The last part here taking the jag doing the token exchange with the you know the Data Bricks identity provider GitHub Atlassian any of these Google whatever that doesn't exist today. So we can't really do this today. The the alternative approach is something that we do in we do it in agent gateway enterprise but I just want to be clear this

not exclusive. I mean we can you know agent core has something similar and off zero has a token vault and the whole process you can build around agent gateway open source as well. But what we do here is when that user logs in to into the MCP server managed by agent gateway we can allow the SSO off load to begin. That'll come back to agent gateway and

then if agent gateway allows the policy if for this user to call out to Data Bricks in this in this case we'll also orchestrate the Data Bricks sign-in and we'll store the token in this token vault. What gets returned to the user is the SSO token. All right so now anytime the client's making a call whether it's internal or external MCP access going through the gateway managed

by policy we can decide whether or not to let the call go out and if we do go out to Data Bricks the gateway can automatically inject the token. The client never sees that. Um and I do have a demo for that but I do not have time for that demo. I will be at the Solo booth after this so I can I can show you there.

Uh as I'm running out of time I do want to thank you all so much for coming out to this talk. Hopefully it was useful. If you liked things I said and want to continue the conversation please do reach out. Uh yeah I did I have the contact information there. If you do not like what I said and you disagree with some of the things that I've

said, I would love to hear it. I would love to hear your your perspective and understand, you know, what you're seeing. Uh but again, thank you so much and enjoy the rest of the conference.