Project Lightning Talk: MCP Routing In Linkerd - Flynn, Technical Evangelist

This talk covers MCP routing in Linkerd, a service mesh that enhances security, reliability, and observability for applications. The speaker discusses the significance of MCP as a stateful protocol, emphasizing how it facilitates communication between agents and external tools through Linkerd proxies. He explains the role of JSON RPC over HTTP in MCP requests and highlights the challenges of monitoring observability, particularly the need to analyze response bodies to identify errors. The speaker also touches on the adaptive routing capabilities of Linkerd, which include stateful routing and tool-based authentication, introducing a new resource called MCP route to manage these functionalities effectively.

I'm Flynn. I'm a technical evangelist for the Linkerd project. I'm going to talk to you about MCP routing in Linkerd. If you are not familiar with Linkerd, we are a service mesh, which is to say we're an infrastructure component. We worry about security, reliability, observability underneath your application so that your application has to worry less about it. If you are unfamiliar with MCP, we are the first

mesh that announced support for it back in Atlanta. MCP is a protocol that you've probably heard far too much about. The idea is that if we have an agent that is doing things on behalf of a user, MCP is a protocol that will allow the agent access to tools outside the agent so it can do things that are more interesting than just stringing together text. If we

drill into this a little bit, what we're actually doing is speaking MCP from the agents to an MCP server, which in turn controls the tool. If we drill into it a little bit further in a Linkerd installation, all of these things are running in pods. Every pod gets its own Linkerd proxy. So, the agent is actually speaking making MTP requests to its Linkerd proxy. Wow. Which has

to decide where to route that to another Linkerd proxy to get access to an MCP server to get access to a tool. The first place that this gets is that MCP is a stateful protocol. So, if we have an agent that is doing things and it has just made a request that gets routed to the top MCP server, it is an error to route a second request

in that session to a different MCP server. In itself, this is not a big deal if you're an infrastructure element like we are. Stateful routing is a thing that we've had to deal with for decades. It's mostly just a thing we have to remember to do. Where it gets weirder is if we look down into the requests themselves. MCP uses JSON RPC inside HTTP, so we have

all the usual HTTP goodness up front. But if you look a little closer, you find that the interesting things about this are buried in the body, not in the headers. Even more interesting, if something goes wrong, remember we wanted to observability, we want to let you know when things go wrong. If we get an error response from one of these tools, you're still going to see an

HTTP 200, everything is fine, and you have to again look into the body to find out what went wrong or even that anything went wrong. So an infrastructure system that's only doing HTTP observability will tell you everything is great even though your tool is failing 100% of the time. In other words, as soon as you do MCP, you must do payload processing. This is true of lots

and lots of things in the AI world, but this is a place where it gets annoying. What's really fascinating about this is that the headers are there for a reason. They come up front, you know where they're going to be. They're They might not be small, but you at least can put a bound on them. If you look at this particular JSON body, the interesting thing that

we want to look at for the method is down at the end of the body. Because you can do that. There's no particular request, no particular requirement for ordering. If the body is something short like this, and we just want a summary of this short phrase, that's not a big deal. But the body doesn't have to be short. And so if we land in this situation, life

gets interesting. We have to figure out how to read past all of this stuff before we find the things that are really critical for us to look at. Alternately, maybe the body shows up like this and then just stops. So, how long do we wait for this before we decide to give up on this request? Anybody? I mean, it's an open question. We don't really know. It's

a little bit more interesting in Linkerd because we tend to view a connection as always being a stream. So for us, a single unary request is a degenerate stream, not an atomic building block. in order to add this without breaking the rest of our world, we first had to teach Linkerd how to parse JSON RPC, which in turn required us to do all of the payload processing

which I was just talking about, which in turn required us to answer all of those questions I just asked about how do you deal with degenerate bodies safely. Once we did all of that, we have to remember to do stateful routing as opposed to just your normal load balancing. Once you do all of that, then we got to teach Linkerd how to do things like route based

on tools and do authentication based on tools. So, you can say things like Flynn gets to go and use the I don't know, word summary thing, but not the Bitcoin mining service. And these are all important. Since you don't get to do that all the time, we also added a new resource for this called MCP route. And if you want to know more about how we did

all this, come over to the Linkerd kiosk and talk to me. Um we also have stuff about Linkerd on EKS, and hopefully you will all come through and check out the escape room party on Wednesday evening. Thank you. All right, thanks Flynn. I appreciate