Cloud Native Theater | EnvoyCon: The Next Generation of Envoy Extensibility: Dynami... Rohit Agrawal

This talk covers the topic of Envoy extensibility, particularly focusing on dynamic modules which have been developed recently. The speaker, Rohit Agarwal, a software engineer at Databricks, discusses his experiences with Envoy and the evolution of its extensibility features, from Lua scripting to WebAssembly (Wasm) and now dynamic modules. Dynamic modules allow the creation of filters and other constructs in Rust and Go while running in the same process as Envoy, effectively eliminating latency issues associated with serialization found in Wasm. Rohit shares specific use cases from Databricks that demonstrate performance improvements, such as reducing logging latency and simplifying RBAC policy management. The session wraps up with a call for feedback and suggestions for future dynamic module features.

Uh, so today I'm going to talk about Envoy extensibility. It's something that I've been deeply working and deeply involved for the last couple of quarters. I know I promised a live demo, but that was before my talk got slashed in half. So, unfortunately, no live demos today, but I promise I'll do it maybe even a better demo next time in the EnvoyCon. So, today I'll talk to

you I'll talk through the dynamic modules. I'll walk you through some of the use cases that we have internally at Databricks, which we are using dynamic modules for. So, yeah, let's let's get started. So, a quick intro. My name is Rohit Agarwal. I work as a software engineer at Databricks at the traffic platform team. I'm a long-time Envoy user, contributor, and more recently a maintainer. I've been

contributing to Envoy for several years now. You might have seen some patches with matchers API, like more recently dynamic modules, stuff like that. There is a QR code on top of the screen that will take you to my LinkedIn profile. If you want to get connected after the talk, please feel free. So, before we start, just with a quick show of hands, how many of you are

familiar with dynamic modules here? All right, okay, a couple of people. So, let's talk a little bit about Envoy extensibility and how it changed over the years. So, as you see, like early 2017, we had something called Lua. It was good, a scripting language. You can do small stuff, like manipulate HTTP headers, like basic stuff. Uh, very lightweight scripting, it will do the job. Around mid-2019, we

had Wasm, and then the entire entire picture of the extensibility changed. It was a big deal. It gave us a sandboxed multi-language runtime. You could write Wasm filters in Rust, Go, C++, you can compile them to Wasm, and Envoy would run them in a VM. Um, it supported stat sink, HTTP filters, network filters, access loggers. The scope was much broader. Uh, but still, like one of the

important parts is there is always a cost. You're running uh, WASM as a VM, uh, which means there there's going to be serialization overhead when you pass the data across the boundary. You can't zero copy access to headers and data. Uh, and the sandbox, it's great for security, but it adds latency. Then, if you look at it, like Jan 2023, we introduced Golang HTTP filters. It became

slightly more easier. Now, you can write Go code, and you can write HTTP filters using that. Uh, we started to get demands on, "Can you add a network Golang filter? Can you add a listener Golang filter?" I think there were some experiments, but it not materialized. And then, in June 2024 was when we started to work on dynamic modules. So, the first thing that we did, we

added an HTTP dynamic module filter, and then you'll see how it is different from the attempts or the things that we had uh, until now. so before we dive deep into dynamic modules, what it is, like what is so special about these modules? So, if you look at it, at the very bottom, it's a C++ core, uh, and on top of that, we define a stable C

ABI. The ABI is more like a contract. It defines the function signature for Envoy, which will call into your module, and the functions your module can call back into Envoy. Now, you could implement against the ABI directly in C, but that would require deep knowledge of Envoy expertise. So, instead, we provide official SDK in Rust and Golang for you to be able to write these modules. Uh,

one of the key insight here is module run in process. Like, it's in the same process as Envoy, uh, same memory space. Um, you can say, "Here are the request headers." It's handing you a pointer. It's not copying the header. It's not copying the data. It's just handing you uh, a pointer. That's why it's it's so latency, uh, it's so insensitive to latency. You're not like incurring

serialization, deserialization that you had in WASM. Like all that goes away. So, what's the trade-off? There is no sandbox. So, the modules have same privilege as the Envoy itself. You need to trust your modules because if your modules malfunction, you can literally crash Envoy. So, you have to like build whatever abstractions you have to build to not be able to do that. So, moving forward, let's talk

about the scope. If you look at it in last version of Envoy, the last major version that we released 1.37, we now have support for modules for almost everything. If you start at the top, we have UDP listener filters. If you want to do like you want to write your own DNS servers, you can do it using module. We have support for TCP listener filters. We have

support for network filters. HTTP filters got much more enhanced. We have upstream bridges, input matchers, access loggers, bootstrap. If you want to load your data and you want Envoy to stop receiving the traffic until until something is ready and then you want it to like start receiving the traffic again, you can write bootstrap extensions. We have support for load balancing policies. A lot of companies try to

do their own load balancing, a customized load balancing. So, you can now write GoLang or Rust code to do to be able to do that. We have support for cluster extension. And if anything is not here, it's coming soon. So, anything you want to do tracing or anything which is not covered yet, it's on the road map. Uh now let me walk you through two use cases

internally, like how these dynamic modules are helping the teams. So, if you look at it, we had this uh requirement where we had to record audit logging from Envoy. So, Envoy was filtering the traffic and all these audit logs was being written on the disk as JSON because that's what we can do with Envoy today. We had a GoLang process which was reading these JSON logs, converting

it to a proprietary proto, and sending it on to the Kafka um Kafka streams. And this was very latency prone. We were seeing like 30 seconds of latency for it to be first written to the disk and then this process reading it periodically, materializing it converting it to the proto, sending it on to the stream. This is the world with dynamic module. All that latency is gone.

We added a custom access logger written in Rust. We could which we don't even have to write the JSON. It would just take the log. It will convert it to the proto and it will directly send it on to the Kafka stream. So yeah, like all that latency and everything went away. The process like the CPU memory, it all got sorted out with the dynamic modules. This

is one more use case where we had some R back policies. These policies were coming from a from a control plane like some service that wanted that wanted R back rules to change dynamically as users are configuring these R back policies. And then the old flow was we had a Go lang server that would read these policies materialize the LDS and RDS updates for Envoy and then

send these to Envoy. But then these configs were massive. We are talking about like hundreds of megabytes of config because there were so many users and I'm not sure how many people are familiar with the matter API. It's very verbose. If you look at it like one rule would get translated into 25 30 lines of config. So it again, like there was a lot of latency. We

would commit like maybe 60 seconds or 2 minutes for all this to happen. Like a customer entering a single policy whether they want to deny traffic or accept traffic and then we doing this massive thing and updating our Envoys. This is what it looks like with dynamic modules. We have a Rust module which does polling. It will directly get the policies. It will update the R back

in real time. and yeah, all the complexity of having our back and then you're using matches API, it all went away. I'll share some results like how it made life better at for us internally for developers as well as for our customers. Um this is another use case that we are exploring. We have a lot of sidecars. We have x.z, we have xproc and we do all

sort of stuff using our sidecars. We try to get the IP and then we try to add a stateful session filter so that Envoy could use that IP for load balancing and there are like a lot of use cases that are happening through the sidecars. Um and this is what it looks like with modules. There are no sidecars anymore. If you need a load balancing policy, if

you need an access logger, if you need a network filter, you can just all do it within the process. You don't have to like make an external call out. So all the latency, all the we use Kubernetes, so all those like sidecar containers These are some of the results from the experiments we had so far. Our config used to be like say 30 megabytes, 50 megabytes in

some cases. We saw a 50x reduction because we don't have to copy these configs. It doesn't translate into 100 lines of JSON config for a single policy. And we also see like 5x reduction in the startup time. Um it went down from like roughly 2 3 seconds to like literally milliseconds. So yeah, like I think there were some results on CPU and memory reductions as well, but

then we have to do a lot more load testing to share those numbers, but these are the concrete numbers from config size and startup time being reduced. Um so what's next? We want to do dynamic modules everywhere. We want to do it for tracing, we want to do it for like any extension that is uncovered. We want to see people using dynamic modules to extend Envoy at

every single point in the life cycle. And before I go, these are it's not a it's not a effort of a single person. We have a work group and I have uh Baiping who is part of the work group. He is a software engineer at Tetrate. We have Takeshi who is part of Netflix and he's also the core member of this work group. And yeah, if you

have more use cases for dynamic modules, if there is a feature that is missing, you want us to implement or there is a new extension point that you want us to cover, please come talk to us and we'd be very excited to help you start on this journey. Uh with that, um I'll open it up for questions and thank you so much for joining. Any questions for

Rohit? Okay. Erica, do you have a question? I can always have a question. So, Rohit, I am still not very comfortable with Rust. So, Go for me is a little bit more comfortable. But, I also do you think I should learn Rust? Is it better for dynamic modules or should I stick with Go? Yeah, personally I don't have that much experience writing Go. Internally, we are writing

a lot more Rust because it's memory safe and then uh there are some guarantees that you can get writing Rust and then yeah, like I don't know how it looks like in Golang. I'm pretty sure like given the the choice of language the dynamic modules offer, you can write any language. But, yeah, like a lot more LLMs and and I think I heard are great at Rust.

So, Yeah, that's literally my like because of that I can venture into writing in Rust. Yeah, not not that I have to learn it all, but because the coding agents have gone to study. Yeah, and it's a fun language to learn. Did you have fun learning it? I had a little bit of fun. All right, more questions? >> Any more questions? I think I have a question

then. So, with the evolution of dynamic modules going forward, you think we will still accept C++ extensions in Envoy proxy repository or will we have like a separate repository where people will commit these community-wide dynamic module extensions instead? Yeah, I think that's a great question because we have a contributions model as well. And then I feel like in future rather than having a contributions model and people

people submitting the C++ extensions, we would point them to dynamic modules and then they can write the extensions in Go, Rust. We might also have plans to natively build some Rust extensions within the Envoy core. So, that's also being explored. But yeah, we might also accept the C++ extensions in the main code if a lot of maintainers agree that it's a it's a good contribution, it's widely

used. One more question from Erica. Yeah, now you made me think. Will Do you believe that we will have official like Envoy proxy dynamic modules published like, you know, quality stamped by the Envoy maintainers? I think it depends on how much more work maintainers are going to take. It'd be nice to have and I personally would fund it if possible. But yes, like if people around us,

they think like we can have a repository where maintainers are reviewing the extensions and then if we have more expertise or more experts in Rust, we can definitely do it.