Cloud Native Theater | EnvoyCon: The End of a De... Erica H, Kateryna N, Yan A, Rohit A & Leonardo M

This talk explores the evolution and future of Envoy, a prominent open-source edge and service proxy. The speakers share their journeys into the Envoy community and discuss how Envoy's capabilities have expanded beyond traditional HTTP proxying to encompass dynamic modules for improved flexibility and extensibility. They emphasize the importance of community contributions and the growing support for protocols in networking, especially within the realms of cloud-native architectures and data-driven applications. Each speaker reflects on their experiences and the challenges faced during their engagements with Envoy, highlighting its impact on modern application architecture and the integration of AI workloads. Overall, the discussion reveals the excitement surrounding Envoy's continued development and its role in shaping the future of networking technology.

We are here to talk about the past, the present and the future of Envoy. And uh if you do know do not know what where it comes from the end of a decade, the start of an age, we obviously know that Taylor Swift sang this song dressed in a purple dress. And clearly she did that for Envoy because why else would you pick a purple dress? So

wi with that I uh want to start by asking both myself and my wonderful friends here what really brought them to envoy in the first place and what kept them there. So Katarina what really made you come and hang out in the envoy community and contributed and become a maintainer? >> Um okay my story started quite long time ago. I was just searching on GitHub what project

to contribute to as a hobby and then I also studied C++ at university and I thought okay now it's good time to revive the ancient knowledge and then I found this super hot popular project called anway no idea what that was at the time and I just started contributing because it was uh interesting to me networking domain and yeah that's how I started and then I was

working at Spotify at the time and then I moved to the team that was adopting anway >> so you got to really do it at work Yeah. >> So, you got paid? >> Well, I got paid for doing open source, right? >> That's amazing, isn't it? She's very lucky. And uh Yan, what what how did you and and I think you're the veteran on this stage. >>

Uh I think so. Yes. Almost eight years. And I ended up in envoy purely by chance. I walked into the presentations to which I wasn't invited. uh and the presentation was by a principal engineer about the future of data plane at my company and she mentioned envoy. There was a lot of questions people were asking. Envoy was a newcomer at this time and uh skepticism was uh

you know understandable people asking why not EngineX and her answer was I don't want we don't need a proxy we need a platform for our future. And I thought to myself, I don't know what it means, but it sounds interesting. That's that's the team that I want to join. >> Good marketing. We don't need a proxy. We need a platform. >> Exactly. Yes. >> Inspiring. I'm on

board. Rowit, do you have an equally inspiring thing that brought you to this space? >> I don't know if it's inspiring, but I used to be a fullstack developer before I became a networking engineer. So when I joined data bricks early 2019 I was put on this networking team where I had no idea what network is. Um back then they already made the decision that they're going

to go with envoy and I think there was a lot of thought that put into future use cases and then the requirements that we had uh in terms of traffic. So I made my first contribution in envoy to change the response code because data bricks had a lot of clients that were retrying on a specific JSON field and then envoy back then did not support like the

customization of the response code. So that was my first contribution to envoy and then I think yeah it just the community the use cases grew internally and I contributed I started contributing more and more >> and then you became a maintainer >> then I became a maintainer and I'm very grateful to people here maintainer. >> Senior maintainer soon hopefully. >> Yeah. Well, so you you do a

lot of work in the open source community and rowit and I will be hosting their next virtual environ which is in May. So please uh you know put that on your calendar to participate. We want to see you there. >> You're going to see some exciting new content. So please if you are around join us for the virtual environ as well. >> Yeah. If if you like

rower he's quite nice so you can see some more of it. Leo, you you are I mean you're a pure end user here. >> Yes. Uh I'm I work for Spotify. I'm the new Katina there. So uh we are I just uh heritage the the whole like envoy architecture. Uh I'm part of the team obviously and yes I mean what brought me to envoy like I don't

know I wanted to contribute more be part of amazing open source product and I saw this inside and said I want to focus on that I'm I'm really going forward and that's that's the thing that got my attention. >> So I think it's really interesting to so I was an envoy fan girl for a really long time. Yan had no idea who I was until I showed

up, but I I ended up running an API platform team and because I really wanted to expose functionality and data and then obviously you need to expose that. But also what happened was well it was the era of breaking monoliths down into microservices was very hot in 2017. You see very popular activity. So that's what brought me to install Envoy Proxy the first time ever and uh

that's how we ended up here on a stage together in Amsterdam for different reasons and we stuck around because people are lovely. Well, I think so at least >> and uh we I think we have to before we look into the future because the future is bright and exciting and envoy may be 10 years old but it is I think we all believe opportunity to shape the

future of networking but looking back I'll start with you Yan why was envoy needed was my use case the reason and uh what was and How how did Envoy impact you beyond you know just becoming part of building it >> right I I think that statement that uh Envoy is more than a proxy that it's a platform it's true and uh we've built so many solutions on

envoy that not that are way way way outside of sort of traditional proxying of HTTP requests serving HTTP workloads we run envoy in our popsub uh serving stack you know uh multiplexing messages uh envoy is found in so many many different places because of that vision of it being a platform and it's very easy to change the business logic it's very easy well I wouldn't say very

easy you do have to have some uh some expertise there we're making that a lot a lot easier with the technology that Rohit was talking about uh but at the time compared to all the other proxies. It's actually very easy and uh giving the product teams the power that they need to implement the kind of changes that they need in a data plane through either extensions or

xrock. I think all of that combined if I look back that's what's really unlocked the power envoy for our company for a lot of other companies and that is uh was unparalleled compared to other to other solutions in the you know in the space. I think was also really exciting about envoy and like its purpose of being open sourced at the very beginning was to really it

wasn't such a common issue many companies had and we all had to route traffic and we all wanted to >> inject our own magic sparkle into the proxy right as we're talking about those extensions to really tailor it to your use case. So Karina when when you started adopting it were you extending it were you doing was that something that was needed for you or what what

did you see in the industry? Okay I can talk that time when I was working at Spotify and I joined the team that was starting doing the migration at the time. Sorry Leo I'm speaking now for our systems that I no longer maintain but our perimeter was consisting of multiple it was quite fragmented. There were multiple network hops and there were multiple technologies like engineext hroxy. Uh

they were serving different types of traffic. The observability stack was not great. Uh then you couldn't dynamically configure at runtime your proxies. Uh and then we got to know about the shiny new CNCF proxy and we started exploring it. We saw that the uh amount of metrics that it gives you it's a great granularity uh performance-wise. But when to your question about is was it easy to

extend? No. I think at the time I was the only person who was doing some envoy patches. The rest of the team uh they weren't experienced. Over the time we did uh grow more contributors but I would say it was not easy to extend during the time. Uh >> do you feel it's different today? >> Absolutely it's different thanks to the work that Rohit is doing and

then also we did uh extend use some Golang filter uh and we used to have Lua filter. >> We still have Lua filter. It's impressive. That's not great. But >> well, look, we're >> I think you left it though in some safe hands. >> Yes. For to inherit. >> I mean, it's not just my work. There's some colleagues over there like more people. Yeah. Yeah. Definitely. No,

no, no. I'm not alone. I'm alone here. Not in uh in the journey. But uh I I helped I I joined when we were doing another migration from uh engine X uh to Mboy. And this was like a limited use case, but we could really see the the opportunity for putting everything together on the same um like the same piece of software that was solving like a

was meant to be like a proxy but was extended to have multiple things and multiple use cases. So this is really really helped us building this uh this uh platform to be honest. >> Roit do you do you feel like you resonate with what other people are saying here? Do you >> or do you have some other hot take on uh what was happening? >> Yeah, I

would say definitely I resonate a lot because when I first joined data bricks it was a very small team. It was like three people who were trying to do what we do with envoy today. Uh it's a big scala shop. So we had a monolith written in Scala that was doing all the request processing uh rate limiting and that sort of stuff. It's very hard to do

and maintain protocol layers and some of the things that we are doing with envoy and I'm very lucky like we did not go to engine X and then had to do the migration. Envoy was the choice and it also shaped my career and I'm very grateful for it. I think that's a really interesting specter but something like even with Katrina said like we all need this traffic

proxy in some capacity and coming together across many different companies and solving those problems together is quite fascinating and with the sort of cloud native era that was really kicking off at the beginning of Envoy like it was a perfect fit to solve a lot of problems that people were getting into and with that let's move into the present day what's been going on what are we

doing and row it we have heard so many talks today about mentioning dynamic modules so does that make you feel a little bit proud >> I love it absolutely love it we have envoys everywhere we have I think 25 envoys that are being deployed at different architecture different places at data bricks there are a lot of people who are trying to customize envoy um I I would

say barrier before was that you have to really understand the architecture. You should know how to write C++. But then now with dynamic modules, it is so easy. People who are experts in Rust, they are writing more modules in Rust. People who understand Golang. So we are no longer the bottleneck. It's a very small team of people. And then now with this dynamic module thing, they don't

have to recompile the binary. They can just do things on the fly. We have set some good abstractions. And yeah, we are seeing a lot more use cases and a lot more adoption internally at data bricks as well. And I love it. I absolutely love >> And talking about the extension point, so obviously I work at a company called Tetray, right? But we have this we for

me I keep saying that that oh sorry that was a static shot that uh like Karina I keep saying she's a very smart lady but I ended up with wanting to yeah also tried to write a dynamic module extension and I have to admit a bit of help from Claude I succeeded. So me me and my friend Andress are actually doing a talk tomorrow. I'm going to

show it. I'm going to show it off. And even I was able to do it, row, I was able to use effectively dynamic modules. I'm very proud of myself. Uh, thank you. So, so I'm also very excited about the possibilities with this. It makes me feel like I can do stuff, too. But Yan, as a senior maintainer of Onway Proxy, he's been around in the community for

eight years. You wouldn't think it. He doesn't look like he looks quite fresh for being around for eight years. But um what are you what you know you you're still here so you're clearly invested in this what what makes you excited about the current envoy you know setup >> yes uh I think extensibility making extensibility easy I I I'd say that it's actually going to be a

very big impact it expands the community uh it's very very difficult to find C++ engineers like how many people here can write C++ there is some how many people can write a lot more so Rust so this is like this is the way that the community is going to expand and I feel like even a couple of years ago would have people to come and say I'd

like to write this extension and the first question is do you know C++ and it was a big bummer to a lot of people um I think of course aentic We we all like here talk about Agentic. Uh Envoy added a lot of support for a lot of protocols that are relevant today in the um in the Agentic networking. MCP, A2A, uh OpenAI. We're covering all of

the major use cases for uh for agents that are trying to call to MCP tools, A2A agents, uh integration with observability. the uh we we want to make it as sort of native as possible to open telemetry. Uh so all of these things like it's both it's a challenge uh because the protocols are very different and the thinking is very different compared to like a traditional u

HTTP proxying but it's also you know it unlock it's unlocking so much value for people for for anyone who's running >> I think that I mean I'm maintainer runway AI gateway so like the whole AI space is very exciting for me I was maybe a bit reluctantly interested in networking and AI origin And then I really started to understand that there was a purpose to think a

bit differently. So I appreciate all the work we are doing together as a community there. And to make sure we have time for some audience questions if you have any. I wanted to move on to talking a bit about the future because we feel like we touched on the future there yan. We sort of got into the future of AI. So Leo, how are you feeling about

the future of Enway? Are you feeling positive? But you're running away. >> I mean, uh, Envoy Envoice started, uh, with a purpose, right? It was built, uh, to be this reverse proxy that could deal with the traffic and then it started to become the standard, right? And now what I see on the community is actually the opportunity for it to dictate uh, not dictate but tell the

direction that we should follow, right? uh choosing the the what to to give back and allowing people to contribute like the the whole uh dynamic modules is a example of that allowing people to uh build their own uh part of it and and the infrastructure and I think this is the greatest opportunity that we have. >> I think that's a really nice reflection there like looking how

envoy came as a solution to a problem. Yeah. expanded into an industry standard when it came to network handling and now has this fantastic opportunity to work with the industry with the community to not just implement solutions to problems but actually frontr run it and pave that path. Yeah. >> So Katrina how do you obviously over there he's an end user right? So as a maintainer how

how does those statements land with you? How are you are you feeling excited or what do you see for the future? Uh yeah, apart from the industry trends of AI and ease of extensibility dynamic modules, I'm personally excited. I'm slightly biased about our recent like future improvements in the load balancing in reliability space. Uh so we do uh introduce improvements of how Anway reacts to memory spikes

and uh CPU spikes by introducing uh more type of overload actions in overload manager. And we also are looking to improve our load balancing suite. We want to introduce more algorithms. Apart from that, we already have Orca which is client side load balancing. And we are also looking to improve the envoy queuing strategy where we introduce Codell and adaptive LEO. Basically, depending on latency, certain uh traffic

will be more preferred than other. This work is in progress. So that's something I'm personally very excited about. >> You know what? That is a very exciting thing. And I think it's one of those problems that you don't even know you need a solution to until you run into it. Like I used, as I said, I used to run API platform teams and the amount of times

I've got to that point and you have really complex requirements for your load balancing strategy. So >> y I agree with you. It's like it's also more like this hidden gem of continuous improvements into envoy and hopefully now it will be there like it will actually make life a lot easier for people. I obviously have loads of questions for these people like questions I haven't asked but

I will let other people ask questions too. So are any questions for me or the other people here? We can talk about envoy gateway AI or envoy in practice. Go a hand. >> Just a question on uh dynamic modules. You've touched on all the benefits that this can bring in terms of flexibility, but my question is around are there risks in terms of uh people making envoy

unstable or less secure with introducing uh components that maybe have less rigorous testing and how do you see that impacting the broader ecosystem? >> Okay. Yeah, I can take that. So yes like because uh the dynamic module share the same memory space you could potentially do something in your module that can crash envoy. Uh but internally what we are doing is we are creating good wrappers. So

if there are panics or anything we are trying to implement rust uh rust based modules. So we have wrappers and then every module sort of just like goes in that wrapper and that's how we are making sure that you're not allowed to do something which is unsafe. Uh but yeah that's a totally valid concern. Any any other questions? Oh, Dennis. >> Uh, thanks Erica. Uh, so my

question is um we see a shift right now uh because let's start with preamble. So for example, EngineX as a uh tool was uh as a server was released in 2004 and for example NY then was released in 2016 and it was a shift in the paradigm how we do stuff and it was like a separation for data plane and so on and now we see another

shift where we also need to do the um to route the LLM traffic. So do you think uh the we can extend envoy uh and use it for these workloads or do you think it can it will need a deeper uh transformation maybe on networking level not only like maybe we came up with some other stuff uh and we can scrub the TCP UDP quick and whatever

and we will need for the LLM workloads a new more um lower level uh protocols or and way can handle it uh with the current set of protocols. >> Yeah, that's a fantastic question. I'm actually going to throw that right at Yan. >> Oh, ah and I are going to talk about on somebody else pick it up. Uh >> I actually think like it's not the first

time that people were asking question is TCP good enough for X? Uh I was involved in a project many years ago that tried that that had to do with routing. Um and I think it is just so difficult to do right TCP is like at the core of our networking stock and I think it is much easier to adopt application layer to the new protocols like LLM

serving and I think envoy is perfectly cap capable of that. Even very difficult cases like disagregated serving in LLMs are completely possible in envoy. Um it's uh it's just a matter of time I think that that uh the implementation will be there for you know for for example for disagregated serving. So you guys done a lot like you and your colleagues have done a lot contributing. >>

Yes. With the integration with LMD specifically and uh we we actually had a design spec for for that for that part specifically in envoy. Uh so where I am the way I see it right now is I don't think we need a revolution just kind of throw away the networking stack and start back from where you know DARPA was what is 40 years ago. So I don't

think we are yet at this point. Maybe it's maybe still in the future. I think right now we can do many many things that we need to do in an application layer. >> Anything I mean other people are involved in AI for on >> not AI specific but uh you can think that HTTP2 has been around for many many many years and only recently has been become

the standard right. So I think we're moving towards uh using better what we we researched uh and uh you have the example of gRPC like it's it works on top of HTTP2 and you have uh I would say you have platforms now you don't have just the products themselves right you don't have this the solution themselves I would say >> yeah that's a great example of H2

when I was looking at the breakdown in our in our traffic for for load balancer H2 was like maybe two years ago was a sad 10% uh like we did all this work building this wonderful protocol and only 10% of people actually using it right now it's it's increased uh but it's still not even half right it like takes just such a long time to change things

at the lower levels >> any any more questions from the audience I'm happy to run around there's also a wonderful lady there with a microphone if someone needs it only if no one else has a question. But um just thinking on what you were saying there with gRPC and the threats to the future of Envoy um with increasingly uh internal microservices using uh gRPC to do proxyless

service mesh getting uh rid of envoy as as as a proxy. uh do you see that as a trend that uh will uh yeah I guess a threat uh to to envoy >> uh I I can share some of my obsolations that I've seen in the community a lot and it was something that maybe surprised me a little but I think it's quite interesting to see how

people some people are embracing psyllium for the service mesh and I see a lot of those adopters choosing envoy based ingress proxies for that I believe that even uh is I surveillance the the proxy of the ingress is an envoy proxy. >> Yeah, we use envoy for L7 uh in silum. >> Yeah. So you you see envoy is in many even those who don't even use that

one they use other envoy solutions together with selium and even like envoy gateway together with selium we've seen a lot of examples of in the community. So, it's well paired as far as I can tell because traffic still has to go north to south and not just east to west. But I I'm a big fan of their envoy plus like selium combination personally. But any any other

thoughts from other people on this? >> Uh I I I think it's a fair question like it's specifically about proxilis gRPC. I don't see it uh I I I see it as a complimentary uh technology rather than a competitor. Uh I think there it's a great fit for a lot of applications and it's also a poor fit for a lot of applications. Um especially if you need

to have a custom business logic in your data plane or your clients have to be very lightweight. uh or I mean if you need if you want to run proxy gRPC you need a control plane for it there is uh there are a lot of complexities that are associated with that that are you know might be non-trivial for a lot of people to resolve and it's much

much much easier to have an intermediary like envoy so I think we we'll definitely see a lot of applications migrating to proxil gpc is it going to be just completely 100% proxis gpc I think that's highly unlikely. >> Yeah, I think I'll add to it. So, if you can do it and if you can do it properly at scale, it's amazing, right? Like you can get the

last bit of performance out of it. But then a lot of people cannot. Um, if you can do proxyless, all you have to do like you have custom load balancing policies in Envoy, if you can do it in your gRPC client and if you can teach your clients how to do load balancing the same way that Envoy is doing, it's amazing. Then you can remove a hop.

Uh but yeah, like I would say a lot of clients and a lot of people don't know how to do it properly. >> So yeah, I think like just generally I don't we're an open source community of people from loads of different companies who are together solving our shared problems so we don't have to write the same solution like loads of times over and over. And I

think that so from a you you asked if it's a threat I think as long as we are solving problems together for the the situations that we have it's a community that is thriving and growing and this AI traffic handling which is a part of ingress egress is loads of different things that I feel envoy is more popular than it's ever been before and I think that's

crazy to say about a 10-year-old project uh that I feel like we're innovating. I think Yon you told me that was it you told me the other day that the the speed of new features and change has actually >> increased. >> Yes, >> we were actually hoping it was decreased because it will be less work for our security team but it actually went all the other in

the other direction which is amazing. I think it's great. It's uh >> and we have a hu I think and I counted all of the at one point all of the maintainers within the envoy ecosystem because we have different solutions in there. It's like I think we're near 50 maintainers 50 from various different companies in the industry which is a huge group but also shows the dedication

and the care for the project. >> Yeah. And it's not a static group as well. It's not just old farts like me, you know, who's been there for 10 years. We have new people coming in, uh, new companies joining and uh I I think like the dynamics are are amazing. >> I do think we are running out of time. Am I correct? I'm I'm looking over the

people at the back there. Are we running out of time? I don't know when things are supposed to end. Three Three minutes. Okay. Just wanted to check because if you have been here throughout the session, you know I can talk for forever. So, you know, look, they're laughing at that. >> That's true. I agree. Sorry, Erica. >> Don't get on the phone with me. Dennis has done

that. He knows what it's like. Any any questions anyone want to ask from the audience? Do not ask me why I can talk for so long. Okay. people then I will keep asking questions. I said I scrapped a bunch of them to try and make time for other Row it you know with the dynamic modules and I my success mine and Claude's success I will say with

writing a go dynamic module. One thing that I've been wondering about because it seems like you cannot have many Go dynamic modules. You have to bundle all of your special Go things into one thing. No, you can have as many as you want. >> Yeah, but but like is that a good idea? >> I think it depends on the strategy of the company like how you're building

these modules. Are they using the same dependencies or not? I've seen both models. Some people are just like bundling everything in into a giant module and then just like plugging that in everywhere. And the strategy that we are taking we are building smaller modules and then yeah we are plugging it like HTTP and other extension points >> or and do did you know Ro maybe I know

something you don't know is that Envoy Gateway will support plugging in dynamic modules in its next release. >> I remember seeing a PR but yeah I'm very excited like >> I'm very excited about that because again for those who do not know Envoy proxy can be a little bit complicated. So Envoy gateway makes that configuration a lot easier. So people like me can find it accessible. So

I'm quite excited about them. I think that will really make dynamic modules even more accessible than they currently are. So I think that and people are using dynamic modules in production already which I think I've heard some people ask about that like oh is it can I use in production but you know many people are. lastly let's let's maybe say goodbye because I think as I said

I talk a lot and maybe at this point we can go through and you can actually tell people who you really are, what you do for work and where people can find you elsewhere than on this stage. So, Katarina, who are you really? >> Okay, I'm Katarina ones doic at GitHub. I have been a longtime envoy maintainer also looking forward at some point become senior maintainer. I

work currently at isovalent uh working on selium and owning envoy at isalent. >> Yan. >> I am Yan. I work at Google uh right now senior maintainer. I've been in on enab. >> So if you are using GCP you may be using envoy without knowing it. >> Not maybe you are certainly using envoy. >> So if you didn't know you're an envoy user. If you use GCP

you are who really are you? >> Yeah so my name is Rohit. I work as a software engineer at data bricks. Um I lead the traffic platform team there. And yes my team is all about the proxies ingress egress. We have proxies everywhere and it's envoy. >> And Leo, we know you work at Spotify and you inherited all of these things from Katarina. Yeah. But >> yeah,

my name is Leonardo Damata and I've been working with Envoy for three years and helping out in with the virtual envoy. Uh so it's I'm really excited about this. >> I'm really appreciative of Leo's contributions to help with that. And my name is Erica Hubberg and I work at Tetrate. I'm an Envoy AI gateway maintainer and been involved in the Envoy Gateway project for a while and

I host the end user sessions. But I do want to say if you're ever scared of getting involved with the community, do not be. You can be a fan girl, fanboy and become part of the group and then you get to sit on stage with these people. Am I not lucky? So, thank you so much guys for being so welcoming. Thank you everyone. Thank you everyone.