Beyond JSON-RPC: Scaling Model Context Protocols With gRPC in the Py... Ashesh Vidyut & Madhav Bissa

This talk explores the integration of GRPC with the Model Context Protocol (MCP) within the PyTorch ecosystem to enhance AI application interoperability. The speakers explain how MCP decouples AI applications from data sources, addressing the challenges of needing to rewrite integration code when switching frameworks. They discuss the limitations of using JSON-RPC over HTTP, particularly regarding latency and payload size, and propose GRPC as a more efficient alternative. With GRPC's support for protocol buffers, AI applications can maintain high performance and reliability, while ensuring secure communications through mutual TLS in cloud environments. The presentation outlines a roadmap for implementing GRPC across multiple languages, highlighting its role in improving scalability and performance for AI-driven workflows.

Hey everyone, welcome to our talk beyond JSON RPC scaling model context protocols with GRPC in PyTorch ecosystem. The mic might look funny. It is for recording. It will not any sound. So, hi. I am Madhav and this is Ashish and we are from the GRPC team. To those who are not familiar with GRPC, it's an open source network library and that allows different applications to communicate with

each other in an efficient and fast way. And it is built on top of HTTP as against REST which is built on HTTP. So, I'm going to talk about the agentic world, specifically how it is being shaped by MCP and what are some of the problems that it is facing and how we intend to solve them with the help of GRPC. Post to dive into some of

the lower level implementation details. let's talk about how the world was before MCP. So, every AI tool required custom tightly coupled code to talk to every data source, you know, like GitHub, Postgres, Jira. So, this like when you're switching AI frameworks, it meant you would have to rewrite all your integrations from scratch. So, for example, imagine if there is no MCP and you have built an AI

assistant that needs to access the ticketing system and talk to your database and respond to your customers. Now, after some 6 months later, there is a newer model that is more, you know, advanced in customer reasoning and you want to switch your AI model or you want to change your ticketing system or you want to change the internal database that it is that is being accessed. You

would have to basically write the entire scaffolding again with the new API that the new model or the tools that are being used by the model. This is basically the end-to-end integration problem. Now, the solution to this was model context protocol. It has standardized the architecture. Now, you have a universal client-server protocol that decouples AI applications from their data. And the key components are hosts, clients, and

server. So, hosts are basically the UI that you're using, you know, it could be VS Code, Cloud Desktop uh Cloud Desktop, uh terminal CLI apps, etc. And the clients are basically the internal connector which sit inside the host. And the servers are independent programs that act external tools or APIs. Now, it provides MCP basically provides three capabilities, three core reading the resources, uh executing the tools, and

uh templates for prompts. Now, reading is like, you know, in a very crude analogy, it's for the AI model. So, it basically discovers what all resources are available. So, that is how it reads the resources. And then, you know, executing the tools are like hands, like you want to push a commit, you want to run a SQL, things like that. And templates for prompts are basically preset

instructions like task. So, that provides the rule of rules of engagement for the AI model. What needs to be, you know, the in the context, what needs to be the rules of engagement, how uh the resources are supposed to be used. let's look at the current state. Uh to get us off the HTTP transport has already proven to be highly successful. Uh they are the workhorses for

local scripts and simple agent implementations we are seeing today. It provides a reliable foundation for developers to start building immediately. Now, the real magic happens through standardized JSON-RPC messages. By using the By using the MCP protocol, you move into build once and integrate everywhere era. Now, instead of rewriting your integration code for every new AI framework or IDE that comes along, you build one MCP server, and

it's instantly compatible across the board. We're already seeing a strong ex- quantum. Like with native support in Claude, high-performance IDEs like custom cursor, and rapidly expanding library of community connectors, MCP is now quickly becoming the industry's connective tissue. It's no longer a question of if your tools will talk to each other, but how quickly can you plug them in? Now, look Let's look at the performance wall.

Now, while JSON-RPC over HTTP is fantastic for getting prototype off the ground, we run into some physics problems. Like when we try to scale this for production loads. First, there is the serialization and deserialization overhead. JSON is human-readable, which is great for us, but it is incredibly bulky for a machine. Every time we pass those large JSON strings, we are burning valuable CPU cycles and adding latency.

Now, in a high-performance environment, we spend on parsing text is a millisecond that we are not spending on actual model inference. Secondly, we have the challenge of moving large context windows. Now, if you're transmitting massive context or complex tensor meta metadata in raw text format, you're essentially trying to squeeze a firehose through a straw. For example, giving entire code bases into context, right? So, the size of

context window has increased like hugely. And to give you some idea, a context window of 8 MB, it bloats up to like 15 to 20 MB if you're using JSON. This is a nightmare in transport pipelines. Whether uh that if you switch to like Protobuf, uh which is uh supported by GRPC, you would be uh using only 8 MB. And if it is more numerical data, you

would even shrink it to 4 MB or something like that. So, it's it's much uh you know, uh lighter on the transport. finally, there is also the chatty nature. Like modern AI agents don't just make one request. They engage in rapid uh back and forth. When you're using a protocol that isn't designed for this level of frequency, those tiny overhead, you know, milliseconds start to stack up.

And a negligible delay in a test script eventually turns into like a sluggish lag in apps. To move forward, we have to look beyond standard HTTP and find a transport layer that can keep up with the speed of this uh of the models. the core idea that we are proposing here is actually quite simple. We want to bring GRPC and protocol buffers into the official MCP pluggable

transport layer. Now, think of it like an engine swap. The We are not changing the car. The MCP car still has its resources, prompts, and tools. The API remains the same. What we are just swapping out is the uh hype is the standard engine like with a high-performance binary-powered one. By making this pluggable, stick to, you know, simple STDIO for quick debugging, but plug in GRPC the

moment they need to scale. This brings us to our ultimate goal, uh which is minimal latency, maximum throughput, and production-grade reliability specifically optimized for So, I'll hand over uh to take you further. Thanks, Madhav. Hi, everyone. I'm Ashesh. I work in the GRPC Python team. And continuing with the presentation, so to to solve the bottlenecks we just discussed, we are looking for a structural upgrade MCP over

GRPC. As we can see the architecture diagram, the fundamental relation remains intact. We still have our MCP client acting as the model or agent communicating with the MCP server. However, we have replaced the generic text over HTTP pipe with the high-performance GRPC interface. The real technical breakthrough here is how we maintain the integrity of the Reinventing MCP, we are simply translating it in into a faster language.

We map MCP existing primitives, resources, prompts, tools directly into GRPC service definitions. By doing this, a tool call that used to be bulky, loosely typed JSON message becomes strictly typed binary encoded This allows us to keep build once, integrate everywhere philosophy of the MCP spec while gaining production grade speed, streaming capabilities, and type safety that only GRPC provides. It's still flexible protocol just learning on a much

faster engine. Moving on to the next slide, uh we we'll not talk about the auth and security for the MCP servers and clients that that will be built using GRPC. So, the critical operation foundation that keeps our system running are authentication and security. One of the strongest arguments for GRPC isn't just the speed, it's the natural fit for the infra are already managing. We aren't asking them

to support a bespoke experimental protocol. We are leveraging the same backbone that powers modern microservices across the industry. In today's zero trust environments, security is non-negotiable requirement. GRPC provides native support right now, and this ensures every exchange of data between the agent and and the server is secured by the mutual TLS, the private and the mutual TLS uh the the mutual TLS feature, right? So, beyond the

handshake G a citizen in the cloud-native world, it integrates seamlessly into the service meshes like Istio and Linkerd. this gives the industrial strength load balancing and health checks as a standard feature. When an AI agent is making mission-critical tool calls, we need to know in the real time if the tool is healthy, connection is secure. By moving MCP to gRPC, we stop treating AI as an isolated

experiment and start treating it like a production-grade service it needs to be. now looking at the migration from HTTP to gRPC on the server side. So, if primary architecture goal is to ensure that developer experience remains consistent regardless of the underlying The most important takeaway from this code comparison is the principle of similarity. If you look on the middle of the both snippets, the core logic of

of our PyTorch tensor tool is exactly We managed to minimal changes by decoupling what from the how. The domain logic doesn't care if it has been uh called by a REST API or high-performance gRPC stream. It only cares about model context protocol. We didn't want gRPC flavor of MCP that required total rewrite. You simply swap the Uvicorn entry that is there in the left side of the

slide to a gRPC function. That serve gRPC function. On the left side, you'll see how we create a simple tool using PyTorch. To do that, we first create a fast MCP object visible after the imports. decorator on the fast MCP object to define a tool with the definition. This is just a basic tool for demonstration purposes. Once we have the tool definition, we use Uvicorn to run

the app. On the right side, we see how we can use gRPC transport with the same fast MCP object. We just need to import a function from the library called serve gRPC. It needs fast MCP object to run the MCP server with gRPC transport. In the slide, we have highlighted the code which is addition to enable gRPC transport for MCP servers with green color. On the right

side, you will see the introduction of the gRPC transport settings all the gRPC configurations to the underlying gRPC server. Ultimately, this approach allows you to scale your infrastructure, improve performance without ever touching the actual tools your AI models depend upon. So, moving to the client-side changes of the migration. So, when we look at the client-side migration, the philosophy remain interface stability over implementation details. On the left

side, HTTP and streamable Uh on the left side, HTTP clients require streams to initialize sessions. On the right side, notice how gRPC client streamlines this. By using our pluggable gRPC client session, we encapsulate the complexity. get address and session handles the communication automatically. The core principle here is to minimal changes to enter in the to the interaction Once the session is established, the code used to list

tool or call tool is identical. Wherever you are requesting a simple or a complex PyTorch tensor, the level API remains consistent. This ensures that a client developer doesn't have to relearn how to interact with MCP server because the network protocol changed from web-based stream to high-performance gRPC channels. It's about keeping the developer focused on the capabilities of tool rather than mechanics of the connection. Just the APIs

that you are that that you see here are the part of active development and may evolve as the transport spec is finalized. Now talking about key advantages of using gRPC in MCP server client Uh gRPC is a faster transport protocol which we just discussed in eliminates the needs of separate proxies by handling security and other functions natively reducing complexity. It avoids the operational complexity of side cars

through proxyless mesh and prevents the need to rewrite existing tools to work with model agent protocols like A2A and MCP. The gRPC transport is designed to easily scale load balance deployment which is challenge with existing MCP streamable HTTP transports. The the gRPC proxyless agent mesh integrates with AI safety and security directly along with observability tailored for agent driven workflows. As a polyglot framework, gRPC allows agents and

tools to communicate in language, clouds, and runtimes. So, yeah, uh let's talk about what next we have in the road map in ecosystem. This project is a collaborative effort. We are currently in active pluggable transport discussion with MCP SDK maintainers. Our goal is to ensure gRPC is seamless integrated option within the broader MCP ecosystem. We are currently developing a new MCP C++ SDK with the features gRPC

transport out of the box. Following C++, we are moving to go and Java. These languages are the backbone of many enterprise ecosystem environments and having transport there will allow for massive scale and high performance agent communication. For those who want why the why behind this move, the performance benchmarks and architectural vision, we have published a blog post on the Google Cloud blog. It covers why we believe

GRPC is the right choice for native MCP And moving These are the report repository links that we we want you to star or bookmark. The GRPC transport MCP proto repo, that is the first point, is live. This contains the foundational proto files for the any MCP SDK. We will use this proto as the underlying consistency across every implementation we build. As we know that AI community lives

in Python, the GRPC transport pluggable repo for Python is marked as coming soon. It is designed to be pluggable library, meaning we can drop it into existing MCP implementation to upgrade And yeah, as we move move from the development to alpha releases, these links will be your primary help for documentation and sample code. Let's see a demo on how GRPC will be how much GRPC will be

faster as compared to This is a short I've created a MCP server using HTTP and a MCP server using GRPC. Uh On the left side, you will see uh PyTorch Yeah, MCP server using HTTP and the same function, which is actually the tool call, uh which is called create PyTorch tensor tensor, is there on the right side using And this is the same that we have in

the slides. And regarding the client, if you see, so th- this this tool doesn't do much, it just returns a torch.random after taking dimensions as the input. yeah, on the right side, this is the same faster uh same fast MCP But in the end you will see this has been served using GRPC transport and here it has been served using a Uvicorn app, right? And on the

client side we have the similar changes that we Uh the APIs that you will see we are calling this call tool multiple times. The N will be taken as an input in the demo. And on the GRPC API, which is the call tool, and N will be again taken as an input in the demo. I'll just quickly start the server. I'll start the GRPC MC server. And

now coming to the client side, So, now if you run the MC client first, and we do N 1,000 iterations of the call total time taken by the request response cycle is around 2 seconds. While now if you run it on uh client, the same 1,000 iterations, it will take around 0.7 seconds. So, this is like a small demo which shows the latency comparison between GRPC transport

and the HTTP transport. But all other comparisons are there and benchmarking are there online available for us for us to see. Yeah, uh that's and we can have questions right now. Thank you.