Envoy in the Era of Agentic Workloads - Yan Avlasov, Google & Erica Hughberg, Tetrate

This talk explores the evolution of Envoy in the context of agentic workloads and AI networking. The speakers, Erica Huberg and Yan, discuss their roles as maintainers of the Envoy AI gateway and the challenges posed by the increasing complexity of AI traffic. They highlight the necessity for new mechanisms to handle inference calls, authorization, and traffic management. The conversation delves into the pitfalls of traditional API design and how the emergence of dynamic modules enhances Envoy's extensibility. They emphasize the importance of observability and security in managing agent identities and ensuring compliance within AI systems. Overall, the session illustrates how Envoy is adapting to meet the demands of modern AI infrastructures and encourages community involvement to refine these advancements.

Well, I guess I I guess I can start talking because you probably think you're in the wrong room at this point because on the agenda, this is not the title of the talk, is it? Did you pay attention to detail people? The title of the talk in the agenda says envoy in the era of agentic workloads, but we shooed it up a little. So now it says

envoy, the future ready foundation for agentic AI networking. That's what it says on there. For those who weren't able to read, I thought I'll read it out loud. But I'm going to do something very unusual that I don't normally do at the beginning of a presentation because at this point I do think it brings context of why Yarn and I are here. So my name is Erica

Huberg and I'm one of the Envoy AI gateway maintainers which uh helps make it easier for people hopefully to use Envoy to handle AI traffic. And here we got Yarn. You're also a community executive I would say for the uh Envoya system. So if you you know if you want to get into the community like she's the person to talk to and I'm Yan. I'm Envoy senior

maintainer. I've been with the project for eight years and I worked at Google. >> Yeah. He he's he's been around for a long time. So if you want to you want to talk to someone who about what Enroy been through over the last few years, >> we have a rule no hard questions. It's 2:30 and it's the end of the conference. >> Yeah, we're running on >>

take it easy on us. >> Running on very little sleep unfortunately. But so the world changed again and I think it's very fun actually. Probably as we know created a lot of work for all of us and as we know AI agents don't just call APIs. If they did it would have been easier wouldn't it? We already had all that infrastructure in place. Fortunately things got innovated.

So now we have to accommodate for this new era of uh network and traffic handling. We have tools, we have models and you know what models call it like inference calls. is probably one of the most interesting things especially for me because I used to work in fintech and people used to tell me that I had designed the APIs wrong when they could run for very long

times and it was the same API endpoint like a portfolio analysis one and if you had a big portfolio very complicated one it would take long time to process then was very compute intensive if you're a little one it would be fast I think that's a bit similar to what we're seeing with inference traffic yon >> it's a bit similar But we also as the title says

uh agent take actions. >> Yes. >> Is it a problem? >> Because now they have to be like authorized and we need to know what they're allowed to do which I think is quite tricky and those actions going to affect our systems and even how we express that. They need to know what tools are available to them. >> Gets complicated as well. >> Yes. Uh the funny

part is that uh the humanuilt application is going to take the same actions all the time. agents plan using LLMs and that presents a problem is that the some of the actions that take might not exactly be what you want. >> Well, just be clear. Yan thinks the humans are very good at taking the same actions over and over again. I mean, he's here with me and

I don't say the same thing twice on on the slide. So, there we go. I'm like an LLM >> a bit. >> A bit. Thank you. So we yeah we talk about that old world of yeah people people like to tell me that all my APIs had to be stateless was one of the things I was told I had to do >> rest we all like restful

restful >> and uh the uh and we had a lot we did most of it over HP right I mean >> yes >> gpc is y >> and I think this last point though is like we had the proxy for handling all our traffic and routing it and making some authorization But it was easier when it was less. There's so much activity now with these agents doing

things. >> True. >> Quite chatty, if you will. But we have this opportunity now at the proxy layer to enforce and enable even more performant inference systems. Anything else you see in this agentic world that's really changing? uh a lot of things but I think this is the probably the main pain points is that the traffic has changed and interesting parts are in the bodies of the

requests and proxies don't like to read anything from the bodies it takes a lot of resources and a few years ago we were optimizing for milliseconds and kilobytes >> that world is gone >> and you know what that was probably one of my pet peeves about the open AI API back. Was it the model was buried? >> That's right. >> The body. >> That's right. And it

can be at the end of that JSON message. >> Think about that. >> Think about that. We've got something to think about. >> So then we have all of these new standards that are coming and they're evolving so fast as well. MCP and ADA. It's it's a quite a bit of information overload at times and trying to focus as as you know people like us who trying

to build tools to enable people to adopt these things. Sometimes it feels like it's hard to keep up >> and that's just a small snapshot. We uh this is the most probably popular protocols. uh but it's almost weekly that some other CP protocol comes up that does something better than MCP. Uh there is various variations of OpenAI. Uh the protocols change maybe not daily but weekly or

monthly. >> Yeah. And then it's like they change and then you still see a tale of migrations, right? So it's still a lot of people using the chat completions API, right? But now a lot of people are moving over to responses and responses isn't even new. that's been around for a bit and then trying to handle how you do inference calls across the different providers. I mean

like GCP and Gemini have got some things that OpenAI does not provide. So it definitely gives you some challenges just to enable. So that is one of the challenges we deal with on onway AI gateway. we have had this like connectivity connecting from A to B that in large >> we're done >> basic >> we've done that that was easy we say but it isn't actually to

be fair I think that's one of the things that people often underestimate I've seen enough handcrafted AI gateways at this point that fall over under load and aren't able to properly route traffic. I mean it works really well when people are running it on their machine or at small scales and I think that is what really you know makes envoy proxy give us a wonderful foundation and

we are now at this control part that has is the new challenge to do even better than we've done before. So security for example looks a bit different. We have new identities to deal with. I mean you can say we deal with workload identities and service identities for a long time but a agents are a little bit of a different beast. >> They have two identities split

split brain. So one identity is the user that granted it authorization and the other is identity of the agent workload. How should you make your authorization policy based on that >> and how how much is being delegated? I think what's really interesting though for those who are aware there are lots of work happening to really extend that uh token exchange from the open ID connect to that

uh anthropic and octa are looking at but things are evolving and as those are evolving we need to keep evolving how we are able to enforce those things at the network layer auditability probably is actually one of the nicest things about one though because we get a lot of observability >> uh we certainly do but that's actually not simple either because open telemetry came up with the

new semantic conventions and they were not very unway friendly. So there was a lot of work for us. It still is to actually export the things that people have standardized in >> and policy enforcement guard rails like didn't I have my wonderful CISO and our Paul isn't here but one day I was trying to do something at work and then he said no and I'm like I'm

not doing anything bad just making a little request to Claude uh you know Claude Opus 4.6 six via goose and somehow that got blocked. Someone put some guardrails in and that's using we personally at work use onway proxy to help enforce those guard rails. >> That's true. >> I still don't know what I did wrong. I'm innocent. I promise it wasn't naughty and then observability as well.

What I don't know how you guys are doing it at Google with how you're dealing with your you know guard rails. Is there anything you want to share? Have you >> guardrails? Uh there's yes, there's a lot of guard rails. Uh I use Gemini to organize my inbox. Some mornings I wish it would just delete everything, but it has guard rails. >> Oh, it is not allowed

to do that. >> No, no, it puts the high priority emails on top and then they have to deal with >> That I think that's a bit rude, isn't >> It's >> It should. >> Talking about autonomy. Sometimes I wish the autonomy they can just have that. >> You'll be like, "Yeah, please go ahead. Answer my emails. Deal with the problems." Yes. And again auditability or observability

are related as Yan said but it's also observability as you said like it is changing a lot of new standards are coming out. So what is really nice though as the onward community because it are so many different people from different companies all sharing the same problems. We are able to collaborate to drive these things forward which I think >> observability is the number one request that

we hear from our our users. They want to know what agents do. Like everybody is paranoid at this point and this is where observability comes into play. They want to see the actual protocol values that have been exchanged. >> Uh so they can either diagnose problems or decide if that's a good agent or bad agent. >> And as you probably have gathered at this point, a big

takeaway from this point of the presentation is that Envoy is built for this moment. Because you already trust envoy, right? >> I do. I mean, there's a question. Why do we need to drag envoy into this new era? Why why teach old dogs new tricks? What do you think? >> I I think that first, you know, the first time, so for those who don't know, yeah, I

I said earlier, but I am one of the maintainers of one way AI gateway. I will admit that the very first couple of months of that was a reluctant experience because my first reaction to when Dan from Bloomberg came and said I have this wonderful idea. No, he didn't say it with that excitement but he did say we need an AI gateway for envoy. I was like

but it's just traffic isn't it? I'm already doing that as my first reaction. By the way I'm now an enthusiastic maintainer of oneway gateway but I started to realize that yes a lot of our foundation was there. >> And it made sense to start building these features because even though there are new tricks to be learned, a lot of the old tricks are still needed. Yeah, >>

that's true. I think while you know MCP all these new fancy protocols uh are different, they still underneath as a plain old good HTTP and we all uh everything that we built for HTTP is still necessary. flow control, memory management, security. Uh, a lot of the same actions that apply to the old HTTP traffic still apply to the agentic >> and also some when we're getting some

more enhancements to the load balancing invoicing, >> that's right, load balancing should really change. Yes, >> it's quite exciting and I think that even the benefits that we are building into envoy now is not just enabling better handling of AI traffic, it actually benefits all of our network traffic as well, which also makes it a very valuable investment to invest in these enhancements that we are continuously

doing. >> Your hair all the time. >> My goodness, my hair. My hair is a problem. I'm so sorry. I don't know if I don't know if this one's on. Is it on? Okay, great. Otherwise, I can speak really loud and um does envoy really understand agent traffic. Yarn >> that's the new trick that we had to teach the old dog. Uh it doesn't understand now. uh

there are still some corner cases where uh interesting things are not implemented yet like as a cenifications uh you can track the progress uh on uh in the envoy repository in the issues uh but I would say that 95% or even the most important use cases that we saw are fully covered at this point. >> Yeah. So like I mean what I work I work at Tetrate

by the way I didn't mention that to anyone but for example we have a SAS hosted AI gateway using envoy. So all our tool calls and all our inference calls are going in production via this uh infrastructure. So there's and a lot of things are happening very fast as well in the envoy ecosystem to support. could we say that we are enforcing what actually matters? We are

addressing key aspects in our >> key aspects for sure. Yes, key aspects for sure. Uh we can enforce agent identity using uh spiffy mtls. I mean this assumes that it runs within the kubernetes otherwise we wouldn't be here talking about this. Uh we can use the arbback policies. This is the same uh extension that we had for a long time. we we adopted it to to the

new protocols uh external processing for authorization or uh if you need body modification uh examining each uh MCP request this is all now fully supported >> I think also a really interesting thing to highlight is the introduction of dynamic modules >> absolutely >> which is if you don't know about this and if you're not very familiar with onway proxy extending onway proxy up until last year sometime

required you to have something running externally like and access that via external authorization or via external processes. But now with dynamic modules, you're able to build a filter and have that run inside of the same workload and you don't have to fork on Voy and rebuild it, which is nice. So the >> this is great. I think it really expands and uh gives a new sort of

dimension to envoys extensibility. I think before dynamic modules extending envoy was a bit like going to a dentist appointment. >> that fun. Did you have that fun extending it? No, I'm joking. >> No, no. Uh maybe sometimes even going for a root canal. Uh but we we have we have improved that. We expanded the um the number of languages that can be used. before we had to

ask developers to know C++ and I always felt sad when a you know then somebody who wanted to contribute to envoy had to walk away because they didn't have the mastery of C++ and right now we support go and rust and uh I I feel like that's a great way to expand onway community and the contributions that can come in >> and I think that general mention

of rust is quite an interesting one at this time of the era as well because there are a lot of efforts when it comes to enabling more of our traffic handling in Rust and Envoy as well is supportive of that journey and really dynamic modules are making that so much easier. >> Yes. And also just highlighting here the project build on envoy uh where this is actually

development environment is organized very nicely and for the most part you can actually focus on developing the module rather than thinking about where do I download all the different tools. How do I set up my tool chain? >> How do I deal with basil? >> Yes, basil >> which by the way is not fun. That that is worse than going to the dentist. You think so? >>

Basic. >> Yeah. Me me me me me me me me me me me me me me me me me me me me me me me me me me me me and I tried to build on my computer this little guy. >> Oh, I I guess I I got used to it. >> Yeah, desensitized. And here's just a little bit of an example obviously with tool registration being

able to use that. And for those who do want to read more of the details, Yan and I and a few others actually published a article on the envoy LinkedIn that has a lot of this in a lot more detail. Yes, I think you know just to give a quick explanation here uh this is not a production example. This was from a um from my prototyping and

I wanted to build an agent that reads a issue from GitHub uh scores it based on the sort of Google private data and then posts a bug into our private bug system if it's an important issue on GitHub. >> Basically, if yarn has to worry about working on it or not. Yes, >> he's trying to make sure that he does as little work as possible. >> No,

I don't want to look at the issues. Yes. So, triage it for me, please. But note that the agent actually has two MCP servers that do the same thing. They deal with the bugs. Which one do you think the agent will pick to create an issue with proprietary Google data? >> The public one. Of course, >> you're correct. And if you gave a different answer, you are

correct, too. This is a rare moment. The room, the everybody in the room is correct. Uh it was very bewildering to figure out how this worked but it was biased definitely for GitHub and we would want to avoid posting an issue on GitHub with Google problems because Google doesn't have any >> Of course it doesn't. It's perfect. >> No no problem no issues at all. >> No

problem. No no problems at all. Uh and so this is a policy that could have solved that. I didn't have that at a time where uh I could have restricted the agent from being able to push proprietary data and create an issue on GitHub if it for some reason picked the wrong MCP server. >> Stop it from being naughty. And something that is also important is how

we are dealing with the will be for a long time. I know we are talking about stateless MCP but to be fair we are in this stateful world of MCP and we needed to make sure that that >> really well >> right and the stateful is definitely a challenging problem it brought a lot of it's very complex if you have a fleet of proxies and you have

to transfer that state between proxies uh it is uh it took a lot of work to solve we have a solution uh that generally works and we have a backup solution if that one wouldn't work for some reason we we're looking for user feedback um if you need to use if if you have MCP servers with that supports uh states try >> yeah and I do want

to highlight that as that we get to sure that that is working really well we are going to migrate the envoy AI gateway MCP session handling to use the envoy proxy native one because when we first launched MCP support gateway in onway air gateway. We actually had to do it ourselves. >> It was an X proc. Uh I would say this was a great approach because it

allows you to iterate very fast without having to wait for envoy releases uh and sort of obey the envoy cadence. Um extending through XRO is a great way to have a great velocity >> and that applies to anyone who's building on top of Envoy themselves. like don't be afraid to extend it like use dynamic modules, use external processes, whatever is most comfortable for you. >> Correct. >>

And A2A is very exciting. >> Very exciting. Yes. That's uh that's a way to actually compose agents. Now I I think we are not there at a huge scale. It's very the examples are still pretty thin for agents communicating with other agents. But I think this is where it will definitely go because we we have to start composing agents. >> Yeah. Like I I already have like

trying to get my agents to talk to each other properly, >> right? We don't want monoliths coming >> Oh, but we miss them so much. >> We do. >> So I have mentioned several times here about Envoy AI gateway which is really a showcase of the extensibility of both Envoy proxy and envoy gateway. So if you're using Envoy AI gateway, you install envoy gateway which manages envoy

proxy for you and envoy AI gateway is literally an extension on both of them and it's a control plane Kubernetes native but you can also run it in standalone mode. So you can run it on your computer without Kubernetes and it gives you a Kubernetes native CLD based configuration. And what it really came to originally was that we needed to platform like AI platform engineers to expose

the very first we first announced in Salt Lake City first um few CubeCons ago >> CubeCon. >> Yeah. Time flies. It feels like yesterday, but it's like almost two years ago. So what we needed to do then was as product engineers and developers were starting to use inference endpoints. We were on chat completions land, a lot of those AI platform engineers wanted to standardize how that was

accessed. So they could have those control points but also simplify it. So they could have one unified API whether they were calling open AI anthropic or calling AWS bedrock. So they wanted to have a unified way of doing that. So >> right I just >> show years away Kubernetes just catching up to this. If you didn't hear the talk on AI gateway that's like they're just starting

like you were you pretty much done with what uh AI gateway uh Sag wants to do right now. Oh, they should just use Sunware Air Gateway then. >> I I think so. Yes, we should go into their community call and tell them, "Hey, >> we can do this together." Yeah, we say that to them, right? And we're like, "Hey, be part of a party." >> But so,

just a little bit of a >> this is a little demo for those who might have been to my talk earlier today. We we did show this as well. Uh this is actually using Open FJ and Envoy together. So, it's extending Envoy proxy. We are dynamic module. But just show you a quick one here. So here for example we have the code assistant agent who is not

allowed to be able to call use the create issue tool. But what was nice about this is that with combining with open FJ, this is not an open FJ talk, but thanks to the dynamic module extension, I can actually So this may be confusing. I'm granting engineering right access because this agent is part of the engineering team. So it's actually in this scenario getting all of the

permissions of the engineering team, which I think is kind of fun. Uh if you want to see that longer story about what this was about, you can always watch that talk back. Then you can get into details. But this is actually running on Voy AI gateway. We're going to do check some uh user examples here as well. So when you run on way gateway example, you will

get something called some cool self-hosted model that is basically a mock back end. But what's happening here is thanks to how we are able to process that that onway air gateway is attaching model from the body so that we can make this authorization decision calling the open FGA server. So just want to showcase the power of extensibility and ability to integrate with other things and authorize how

you access tools etc. >> That's right. This demo actually uses all of the all of the uh features that we talked about. This uh this is envoy that's been extended with open fga. Uh it stands for fine grained access control. Envoy natively doesn't support it yet. So Erica wrote a go extension to actually add this functionality. How long did it take you? Well, because it was really

nice and AC thankfully, uh, I was able to use their built on envoy CLI and a bit of clever prompting with Claude code. Yes. >> Okay. No, no, no problem there. >> No, it worked out. Do you know what the hardest part of this entire demo is? >> What >> is is that display of that graph? And it's still confusing. I couldn't even figure out to draw

it by hand. It's it's all right, but it could have been better. I still don't know how. But you know that was the hardest part but making it functionally work and as you can see I think this is a lovely example of the some of the uh you know latency this is quite complex graph traversal authorization decisions happening. So >> shows envoy extracting the protocol properties uh

that allows envoy to or open FGA extension to block traffic based on the uh which tool is calling or which model is being called. >> So it's to me this is uh something that I've really like the ability because to extend things because I do think that most of the time we don't have a proxy we don't nor a gateway of any sort that does everything we

want it to do. So let's talk about what we're up to right now, Yan. And maybe there are people in the room that want to join us on this journey of building envoy, >> right? And I don't think this is a even a full list. uh we would say you know we already talked about agentic protocols and some of the things that we want to wrap up

in MCP uh open telemetry using uh the semantic conversion conventions that open telemetry defined guard rails guardrails integration today there isn't really an easy standard way to integrate with guardrail services a lot of it is bespoke uh and that presents a challenge I think that's one of the area where we need to standardize and improve uh fully implementation for A2A. I think most of this is there

supporting agent card. Uh but I I think it doesn't support all the transports yet that uh A2A is defined for. >> So basic if people need these features and they're valuable to you like please join the conversations, please join trying things out. Sometimes contributing to open source is asking the questions. >> Correct. just uh stating your need and uh a lot of that is very new very

new to us very new to you and we need to know your use cases we need to know what works what doesn't for you what problems you need to solve uh probably a lot of people have the same problems let's solve them >> exactly and even though we are as said many people from many different companies I think even in terms of the entire envoy maintainer ecosystem

is somewhere between 40 and 50 people do we even cover all of the use cases that they exist. We do not. And uh we don't want it to be a little echo chamber where we all like, "Yeah, we're doing great. So, please join us in whatever capacity works for you." And I think one of the things that I really do want to leave people with here is

that Envoy is a safe bet for AI infrastructure. I mean, you guys at Google, they're using it loads as well. >> Yes. Yes. uh all of our infrastructure including agentic uh support for agentic networking is all envoy based within >> and as we this is probably loads of things you feel like it's now repeated it should feel like we've repeated this multiple times throughout this conversation where

you have the is production majority it's been around for a long time it's been along for longer than yan has been in the envoy community not longer than yan has been around though Jan's been around longer than 10 years he he he is older than pen, but there is, as I said, really loads of opportunities for you to extend it and make it work for you. And

we are investing so much in all these protocols and we continue to stay up to date with them and allowing you to have that security and forcibility as well. And often if you cannot do it directly, you can do so by extensions and the ecosystem alignment. And especially I think we see a lot of people in the community world traversing into different open source projects as well.

So we have that opportunity and I think something you say like it's still early. >> It's still very early. Yes. >> Even though it feels like now we've been around for a while doing agentic >> It still feels like a beginning for >> So how how do you get involved? Obviously this has been a bit hard to write down. I know. So, I thought I'll help you.

If you go to the Envoy AI gateway site, and I'll tell you why this is easier than going to some of the other ones, is because if you go here and you scroll down, well, this is hard. This is a tutorial on how to get into the Slack in Envoy because we're not on the CNCF Slack. We are on a special little corner of the internet. You

have to go here and you have to click this button, join us on Slack. And then you get to this page where you put your email in and then you can be part of our group. >> I think we need to make it easier. >> I think so. >> Yes. I've been thinking about that a lot. >> So we don't need to do a tutorial like this

where to click. >> Yes. But at least now there is a recorded tutorial. >> Okay. That's true. Yes. If you can refer to YouTube. >> Yes. But you you can find us and we want you to come and be friends. Something I did remember that I I was like, why did I put this site here on a tab? There was a purpose, by the way, because you're

probably wondering what on earth. Well, I thought I wanted to show you there's a little secret thing here. See here? Uh, use MCP. It's very little because I have a big screen and and you're sitting very far away from it. Um, but if we if we do this, look, we have something that is powered by Kappa. We're very grateful for Kappa who is providing this to open

source projects. But this we have Kappa that is ingesting all of the documentation and all of the GitHub discussions and code so you can install it into your favorite little coding agent and you will be an envoy expert and Yan can do less work. >> True. It's uh it's almost as good as me. >> Almost. We tested. Yan said it it was almost as good as him.

So, thank you for spending time with us today and hope to see you in the community.