Cloud Native Theater | EnvoyCon: External Processing, Internal Leverage: MCP Tool Calls... Jens Kat

In this talk, Jens Kat, a senior software engineer at ING, discusses the integration of Envoy to support AI agentic use cases within the bank's existing service mesh. He explains the role of Envoy in connecting over 2,000 services built with various technologies such as Spring Boot, Go, C#, and Python. The focus is on the Model Context Protocol (MCP), which facilitates communication between AI systems and ING's data. By implementing MCP through Envoy's external processing capabilities, the team is able to streamline data access for AI agents while ensuring security and reliability. Kat emphasizes the importance of utilizing Envoy's extension points to offload specific business logic, allowing for faster evolution without burdening development teams with complex integrations. Additionally, he highlights the intention to open-source their MCP implementation to benefit the wider community.

Hello, EnvoyCon, KubeCon. Thanks for having me. Today, I will talk about how we use Envoy to support AI agentic AI use cases in ING. So, this won't be a talk about AI, but about enabling AI. Um My name is Jens Kat. I'm a senior software engineer at ING, uh in the service mesh team. And ING is a bank. Uh we service around 40 million customers uh worldwide.

We have about 60,000 people, of which a third are working in tech. And my team, service mesh team, is uh providing support and uh making sure everything connects uh reliably, secure in production for over 2,000 services. Uh indeed, as uh Erica said, last virtual EnvoyCon uh in October, we uh me and uh colleague Stathis explained how we started switching to Envoy for all our production workloads. Um

today, so I won't be talking about uh using AI or LLMs, but I will sketch why we are using Envoy and how we are using Envoy to support agentic AI use cases using the external processing feature. ING has a mature service mesh. We have all kinds of services connecting together. Built with Spring Boot, built with Go, C#, Python, uh what not. And uh the connectivity is made

possible by Envoy. Now, since last year, the AI agents and the use cases for AI agents agents came along. And those AI agents to be uh to be doing something uh meaningful, need access to the world, to our ING data. So, the question was, how can we connect the AI agent to the uh ING specific data that we already have? This is where the molecular context Protocol

could help us out. So, MCP, Model Context Protocol, is a standard uh to connect AI systems, AI agents, with external systems. It's an open standard. It was um invented or proposed publicly uh over a bit over a year ago, end of 2024. It's now adopted by uh many more besides Anthropic, Google, uh and others. It's already 16 months old or young. Well, uh time flies in the

in the AI world. And basically, it's a a JSON RPC protocol where you have a client, an MCP host, that talks to an MCP server where tools, resources, prompts are being hosted. Um and this would unlock us and enable us to do agentic AI together with our Now, the question was, where do we enable this kind of MCP implementation? Where do we put that? And what does

that mean for the teams, more than a few hundred teams that we have running in production? Would we burden them with implementing MCP servers in in Python, with fast MCP, uh in Go, in whatever language, and have basically the the the begin state as we had with microservices, issues with discoverability, security, reliability. Well, we already have solved that problem. We have a mature service mesh. So, we

propose to put the MCP servers next to the AI agents and have those MCP servers configured with dedicated and specific tool uh definitions, and reuse the fact that we already have Open API documents uh in our production service mesh. We want to bridge this JSON RPC MCP protocol to HTTP and do that only in one place. Now, we had two options to um deliver this product or

bridge this uh this gap. And one was of course look at off-the-shelf products. What is out there we can build or build upon a dedicated library. We can uh leverage open source products like uh for example the Envoy AI gateway. We filed a ticket and having this MCP to rest uh translation was closed as not planned because out of scope at that time at least. Uh and

of course the complexity or the uncertainty with stack is that you will encounter new failure modes. So on top of that we would have to build observability, resiliency, and security or do that in a sidecar fashion. Now we also looked at Envoy. We already know that technology. We have it running in production. There are many extension points in Envoy to build this kind of business logic. And

we already saw that MCP support is in progress in the community uh by for example implementing the MCP filter that is now available in the latest uh Envoy release. So we chose to put Envoy or the logic to translate MCP uh tools and specification towards open API or the HTTP service mesh in Envoy. Now for extending Envoy we have a couple of options. Uh the basic uh

well the first one is implementing a filter. Uh you can do that in a plethora of languages go, wasm, uh Lua, C++, or as a dynamic module in uh also several SDKs that are out there. Um and all of them have more or less access to uh requests and and responses. Or you can use external processing. And this external processing is basically offloading to a separate service

on the same machine or somewhere else. And this uh external processing also supports uh yeah, transformations of all the properties of a request and response. In ING, we chose to implement this as an external processor. Now, to give you a short overview of what external processing is, it is an gRPC interface, and this gRPC gives hints to Envoy on how to transform specific parts of a request

or response. It has access to the headers, the body, and trailers. And in this sequence diagram, you can see that Envoy calls out to the external processor, and in the external processor, we give transformations back to Envoy, and Envoy applies the transformation that we wanted to apply. And the gRPC interface, or the external processor, can also send direct responses, so you can abort a call chain, and

it also has access to dynamic metadata. So, it's a very feature-rich um extension. It has been stable since 1.29, so over 2 years ago. And uh the gRPC server has state. So, Envoy opens a connection or stream to the gRPC service per uh request, and that's why you see on in the sequence diagram the bar or the activation state of the external processor keeps uh active while

Envoy is sending to upstream, for example, and only closes when Envoy uh stops the filter chain and has responded to the client. Um Envoy interacts with HTTP client uh to the external processor. Statistics come out of the box. Uh trace parent forwarding is supported, so you can pick up and leverage uh of telemetry in your external processor. And as a yeah, um extra optimization, you can use

Unix domain sockets if you run on the same host to prevent serialization of uh TCP stack. Now, how do we implement MCP with external processing? First of all, there are a couple of yeah, initialize and list calls in the MCP spec. And those were easily handled by giving back a direct response. The direct response just says, "Okay, I'm initialized." Or it says, "I know which tools are

now available in this external processor in my server." And give back this list to the AI agent. Then for the more dynamic part, so part two, which is actually calling tools, we use the tools call and implement intercept that in the external processor, give back the request transformations, Envoy applies the transformation and sends it upstream. And on the way back, the HP response is being sent to

the external processor. We give hints to the to Envoy to wrap it back in JSON-RPC. sends it back downstream. Now, on the wire, what does it look like? So here on the left, you will see a JSON-RPC payload. JSON-RPC by definition is always a post request. And it has a couple of fields in it. One is the method. So which RPC method is being invoked? And then

it has some parameters, which is part of uh of the MCP spec. in this specific case, it has the find pets by status uh uh tool call. And we know in the external processor how that should map to which Open API endpoint. So on the right, you see that it should be a get request, and it should point to swagger.io with the specific host. Now, on the

way back, Envoy gets a response from the upstream, uh in this case a JSON array, And the external processor or the MCP implementation has to wrap it wrap this JSON response into the JSON RPC payload so that the AI agent understands and can continue with its processing. So in here you see the ID is equal to the the one that was sent so that means that the

statefulness of the MCP request is kept in state in its memory during the request life cycle and we just package or give hints to Envoy to package the payload and make it document. Now how do you actually configure Envoy to start doing this? Envoy HP XProc is a filter is an HP filter so you need to have an HP filter chain. So it has to sit before

the router. You have to specify where the GRPC service is living. The GRPC service can be on the Unix Unix domain socket or somewhere in on the VM or outside the VM. For processing you have to tell it which life cycle hooks it needs to forward to the external processor so that you can act on the specific parts of the request or the or the body. In

this case it sends the request headers but it skips the response headers because we don't care in this example. And very important for our use case the JSON RPC protocol was is post and your HP endpoints can be any HP method and Envoy has some guardrails in the external processor filter so it doesn't allow any mutation to happen unless you opt in. And for this specific example

we had to rewrite the host header and rewrite the the method of the request so that meant that we have to pass allow all routing with the value true. So it's an opt-in feature to allow this kind of mutation. Now, for observability, as I mentioned, the trace parent is being forwarded to all those uh or along those requests. So, if we want to pick this up in

our uh gRPC server, we can. And in this specific example, we have added hooks or created spans for every uh mutation that we do. So, you see here uh the request coming into Envoy, the request process is being called. It has some microsecond uh transformations, request body, and then you see it goes to upstream, and we enable this by using the spawn upstream span uh property. And

on the way back, you see that the response body external processor is also being called. So, that means the re-wrapping into this MCP payload. All working out of the box. Envoy, this extension point of Envoy turns out to be ideal for our use case. So, wrapping and transforming responses is business logic. Uh and we can put that business logic in external process. For us, it meant that

MCP has not to has not to be solved by specific teams. So, the use cases are not blocked by backlogs of uh we don't have the priority to uh work or implement MCP for this use case. The um the use case owner can implement or configure this MCP server uh for themselves and leverage the service mesh that we already know and love. Um we have seen that

the business logic, uh transforming, adding spans, uh stuff like that, can evolve faster than Envoy. So, now Envoy uh community with MCP filters are catching up with what we have been doing for over half a year now. And this meant this means that we can scale down our custom efforts and go back to more um mainline Envoy. On top of that, it has full integration with observability.

Now, anything I told here was not ING specific except for the contacts. So, we are proposing this to open source at the GitHub ING Bank so that other companies, users can use this approach as well. It runs in a Docker or wherever your Envoy runs. So, you could also use this locally for your VS Code integration, for example. But, we're still in process of going through all

the gates. Stay tuned for that. Now, to wrap up, AI agents want to use data, want to interact with the world, and MCP is the way we implement that in ING. We already have an mature and existing service mesh, and we want to enable those AI agents to do this in a secure, reliable, and observable manner. And Envoy and its extension points allows us to implement the

specific use case. Envoy has a lot of extension points, and that means that you can build whatever you want, actually. We have seen that external processing has the capability to do anything or allow you to do anything with the request, but it has guardrails. So, take care of that, and you can enable or disable them at your liking. Just to note, this specific solution is running in

production in ING serving AI agents, and we are proposing to to open source this Open API MCP X proxy server. Naming things is hard, but I think it it catches the the meaning of of what it does. Thank you for listening to this. I hope you have a nice Envoy call, and uh I think there's some time for questions. Thank you. Does anyone have questions for Jens?

>> Could you please widen the context by explaining the use case? Like from the start, how do you like trigger the agent? Or it lives by by itself or Um how do we trigger the agent? Yeah, that is use case specific. So, you can either have a trigger, like a batch and like offline processing mode. So, trigger hits an agent and the agent says, "Oh, I need

to call out to whatever data that I need to fulfill my use case." Or you could be online, say, "Okay, I'm now triggering either through website or whatever channel you have." Go to the agent and the agent needs to fulfill data or need to interact with the world and uses the tools that the use case has enabled. So, it's mostly autonomous, but people can also interact with

this Yeah, so both and indeed. Yeah. And in this specific case, the and the tools the listed tools are for that use case only. So, we don't have the mega MCP server with 10,000 endpoints. We Or the use case can decide we only need five endpoints or one. So, we don't pollute the context of the of the AI agent. Thanks. >> Thanks. Hello. So, as you mentioned

that you had like thousands of these services, right? And your agents are talking to them. And then in a bank, like you could have like several services who are doing similar stuff. So, how does your agent actually like map and then how do you keep those? Because, you know, similar computations or whatever could be happening across different lines of businesses. >> Yeah, true. So, um We have

our API marketplace, so now we go beyond. So, API marketplace MCP marketplace where a team can say this endpoint or this this open API specification is MCP ready. Because for the AI agent to understand the context it needs a proper description, operation ID, and summary of of what it does. So, by having a kind of governance in place, we can tune or the the use case owner

can say, "I need this operation to happen and not have crosstalk." And the crosstalk or the subscription model that we have in our service mesh already prevents let's say unwanted traffic, but for the AI agents we leverage the same and there is yeah, some extra governance in place to uh make sure we don't do the wrong thing. Yeah. We have time for one more question and I

think Rohit, you can call on stage if you're around and start prepping. Hello. Thanks for this talk. Um uh I understand that the the container that meant that map to HTTP reads open API specification in order to provide a list of tools to the agent. Mhm. Um do you support the dynamic new tools discovery? No. So, for This is a good question. So, in the MSP spec,

there are methods, I think about 19 methods that can be implemented. We implement a handful, which are for our use case relevant and dynamic tool discovery is is not one of them. So, that's uh a priori we know which tools should be made available. And if a team or in a use case needs more tools, then yeah, they have a config change or redeployment of that of

that The agent need to reboot the the connection with the MCP to discover new tools. Sorry, could you The agent need to reset the connection with the MCP to discover the new tool. >> Yes. Okay, and could you describe your feedback loops when there is an an update of the Open API schema? Um feedback loop of the workload that transform that takes the Open API schema to

fit uh the workload. Uh is it live updated when you release a new Open API version of a specific API? yeah, so now we we have solved that problem in our with API marketplace with subscriptions between services. And yeah, versioning should or new versions should not be breaking. That's one of rules we have in our API marketplace. So if you have a new endpoint or an updated

endpoint, it uh break current behavior. So that means that the consumer can opt in to that new behavior. Or it's a complete new endpoint and then yeah, you need to have new subscriptions. So that is already managed and solved in the current service mesh. So it's not an MCP server feature that we uh Okay, great. Thank you. Yeah.