Cloud Native Theater | EnvoyCon: Zone Aware Routing With Per-Locality Load Awareness- Isaac Wilson

In this talk, Isaac discusses zone aware routing within Envoy and the associated challenges experienced in his company's infrastructure, which runs on a combination of on-premises and Kubernetes environments. He explains the architecture, focusing on how Envoy Gateway facilitates traffic routing across multiple zones. The speaker walks through issues related to uneven load distribution and presents solutions aimed at achieving true load awareness per zone while maintaining local preference for routing. He elaborates on using the Load Reporting Service (LRS) for traffic analysis and mentions developing a locality-aware load balancing policy to optimize backend routing. Isaac emphasizes the significance of dynamic modules in enhancing Envoy's capabilities and shares insights gained while working on these solutions.

Yes, thanks for the intro, Erica. Yeah, so I'm Isaac. I work at a company called The Trade Desk and yeah, today I'm going to talk to you about a little bit about zone aware routing within Envoy and some issues that we How do I mute this? Some issues that we had with trying to run that with our environment and some changes that I'm trying to get into

Envoy more recently to hopefully solve it for everybody. Um Yeah, so to just quickly maybe go over our architecture. At the the last KubeCon I did a talk about our migration story from HAProxy to Envoy, but this is a basic overview of what one of our data centers look like. We run a lot of our infrastructure on premise and within one of these data centers we might

have say like I don't know 100 to 200 racks within it and then with with in each rack we have a number of nodes and then on top of that we we run Kubernetes. Uh that's a a bit quick but hopefully this graph is starting to make sense and we use Envoy Gateway as the service to facilitate traffic routing. I'm also an Envoy Gateway maintainer and what

we do currently is on every single rack we we we use a least request routing algorithm and then we enable zone aware routing and we try to schedule our workloads roughly equal per rack. So if you look at the diagram here you can see that we have about three back end pods and what that ends up meaning is that Envoy is going to route 100% of the

traffic locally. If you're familiar with zone aware it effectively takes account of how many Envoy pods there are per zone as well as the number of upstream hosts there are per zone and then will route traffic accordingly. And for for most of the time this works amazing, but if you maybe have unequal amounts of incoming traffic or maybe different clients sending different amounts of traffic per zone

this could create challenges. If any of you went to the Spotify talk yesterday that was actually an amazing talk and kind of solves a very similar challenge to this one. Yeah, that's a quick overview. Hopefully that makes sense. And so jumping into maybe like what what are some of the the issues with zone aware routing that that we experience. the way that we uh facilitate like different

Envoy pods within the the data center getting traffic is we use what's called BGP anycast. What that just means is every single node that's running Envoy is advertising a certain IP. They're all advertising the same IP and you can mostly get a a roughly equal amount of network flows per host that that is exposing Envoy. Like I said overall that works great, it's not necessarily guaranteed. You

could have some some like ECMP bursts or just issues with how the routing works or the different amounts of client traffic like like I mentioned before. Um and so with zone aware routing we sometimes have issues like what this screen depicts where if zone A just happens to be getting 50% of the load the back ends there end up becoming overloaded because Envoy is not aware of

the Envoy hosts in zone B and C and so this is just a a poor distribution of load and that's something that I'd really like to fix. And so the next question is well how can we solve this? The the goal is to get true load awareness per zone without sacrificing that local preference. So we'd prefer to route traffic to the local zone as much as possible

because we get reduced network cost, we get faster latency and it's just more efficient whether you're running in cloud or in on premise data centers. But we don't want to do that at the expense of request distribution. Like the the overarching goal is to to achieve an equal distribution of load amongst your back ends. And so there there's a couple ways that you can do this. Rohit

was just talking about dynamic modules and and how much that has sort of uh like just becoming more and more stable and getting more and more features. Well that that was a actually a great idea. Just in the in the past couple weeks I I started like taking a look at what this would look like. Um and this is one great option that anyone would be able

to take on and implement themselves without any changes to Envoy. So there's a few different ways that that you could take a look at this, but the idea is that you have a central load service and then all of your your Envoy hosts can send what's called it can use LRS which is load reporting service to send those statistics to your central service. Your central service can

then aggregate that data and then you can also with dynamic modules have what's called like a bootstrap extension. So Envoy can periodically uh check in quote unquote with that central load service, get an idea of what the load looks like per zone and then your by by using the dynamic module LB policy you can effectively replace the entry load balancing logic with whatever you want. And so

what I've done here is effectively reimplement the client side weighted round robin which first checks whether or not like checks what the distribution of load should be then in real time understands what that distribution should be and will then uh route traffic accordingly to your back ends. And if you don't want to use Orca a different way to do this you could you could instead route based

on something like QPS or whatever uh Orca just has been getting a lot more exposure in the community, but I I I do acknowledge that it's not not necessarily easy to ask all of your your users to make changes to to add that support, but yeah, this is one way that works and it's I I I was just trying it last night and before this and hopefully

the demo gods will be with me to to show you some of that now, then the other thing that I've been working on is trying to add something in tree. So there's already there's already client periodically takes Orca reports from your back ends and then figures out what the weight should be per host. The only challenge with it is that it doesn't have any awareness of localities

or that that local preference that that I that I was talking about earlier. And so what I've what I've been trying to work on adding the ability to add a locality level picking policy within Envoy. That way you could you can do that. So you could swap the the child policy which which is the like client side weighted round robin or whatever you're using to pick an

endpoint. Let's say it's least request. It could be regular round robin or whatever implementation you have. But at the locality level what I'd like to do is is something that takes actual load per locality. So it so in this case Orca metrics and can then route accordingly. And so the the way it works is that very similar to to the way that Orca works today it adds

a way for those reports to also get sent to this locality picker. It figures out then it hands off whatever the whatever the the the host choice is to your your desired picking policy. And yeah, feel like I raced through that way too quickly, but um How much more time do I have, Erica? Can Okay. Um well yeah, so after this I'm I'm going to jump into

a demo, but yeah, so the one of the learnings that I had is the dynamic module LB policy is amazing. Thank you everyone who worked on it. It's been super helpful and uh the the this this unlocks the ability to add any custom business logic that you have to do any routing that you want. Some of the missing features in Orca though out of band reporting if

anyone's familiar. So instead of doing in band which would be in response headers from the back end which then your load balancing policy sort of like has to keep track of separately. Uh one of the challenges there is that there's no reporting from zones that you haven't sent traffic to. So if I jump back to the example here if I have an Envoy in zone A that's

sending 100% of traffic locally it has no awareness of the load of the the back ends in zone B. So the way that I've solved this in in this policy is to send a configurable amount of probe traffic. I've just picked a arbitrary default of 3% so some traffic always goes cross zone to sort of discover what that load and then everything else can can stay local.

But that's something that if we can avoid it it'd be ideal and that's something that I hope to add soon with out of band reporting inside of Envoy. But typically from that if anyone has else feels strongly about zone aware routing please take a look at PR and yeah, let me know if you have feedback or ideas around this. Yeah, with that let's see if a demo

will work. All right, there. Oh. No. Oh, shoot. There. Again? Okay. So, Yeah, what what this is doing is it's spinning up two Envoy hosts, one in zone B, one in zone A, as well as four backends that are evenly distributed against it. And then separately, it sends load in an uh uneven fashion to zones A and B. Uh and the expectation is that we get uh

a result that the uh the the overloaded zone should spill some traffic to the underloaded zone. And let's see how this went. So, yeah, with this we can see that zone A is receiving Okay, I think something is wrong here. All right. Um well, yeah, so the the the the demo might not be working as well as I would have hoped, but uh yeah, that that's um

that's the policy that I'm I've been working on. And uh yeah, with that uh Uh thank you, everybody. Yeah. So, that was also a use of dynamic modules, if I understood correctly. Yes. Yeah. Which is very popular. I will say we've heard about that a few times today. Does anyone have questions for Isaac? There we go, there's one. Put your hands up and you have to wave,

because you know, it is hard to now figure this out. But here you go. Yeah, thanks for the talk. Actually, I'm curious about the dynamic module usage. We're actually doing I'm from Uber, and we're doing something very similar to this. We can probably chat later. But I'm wondering like what has been your experience with dynamic modules? Because we've started off with like a bare-bones C++ implementation for

this. Uh but we already have some Go line code and Go line expertise that we might make use of. So, wondering what's your experience. Sure. Um for for for us, like it's it's mostly just been about trade-offs over maybe what you have experience with. Like you said, you have a lot of Go code already. Um or what whether or not you want to build your own Envoy

binary. Uh There's a lot of different ways to extend dynamic modules. Like I know the last what, two or three presenters were probably talking about, but uh yeah, dynamic modules is super powerful, because it it you don't have to uh maintain your own build environment for Envoy. Uh and yeah, like you said, you can you can write Go code if that's what you prefer. Uh the only

challenge might be that some things may or may not be supported in the the the API already. Uh but I know there there's tons of work, and I feel like every week there's a a a new PR at adding more support. So, if it's not there today, it might be there next week, or start a GitHub issue to get it. But uh yeah, the experience so far

has been very positive, because then we don't have to build our own Envoy binaries and manage shipping them around everywhere. We can uh just ship what comes directly from the Envoy proxy community, and then uh continue running that. Does it Does that answer your question? We have another question over here. Yeah. So, I I looked at the presentation EDB. They are produce the Cloud Native PG. Maybe

you heard of that product? I'm familiar with it, but >> they also have some extension mechanism of custom resources. Uh where you for example can define yourself which what kind of backup utility you want to use to make a backup of your Postgres database. So, they're also extending the the the the way the custom resources work in Kubernetes. And I'm not sure whether this is the same,

but maybe it's it it has the same target, or or do you want to aim the same things? Yeah, I I I think in a similar sense, yeah. So, like uh I don't know all the history of how a dynamic module itself came to be, but um yeah, kind of sort of what you're you're talking about of like wanting to give users the ability to extend the

software without needing to package everything on their own. Um and I think that's what dynamic modules provides. Uh yeah, gives you a cleaner interface to add whatever custom business logic you have. Uh like maybe jumping back to uh the the other person's question, we have a a different dynamic module to inject custom logic during the uh request filtering phase, where if we want to uh like like

like say we get a bit request, and we want to tweak some of the data that gets fed to our backends, we can do that uh in real time without needing to add a bunch of customizations on top. Cool, we have one more questions over here. And I'm just going to tell you, Isaac, can you unplug your computer while you answer this question, and so Shaolin can

go and plug his in. So, you're also going to be miked up, Isaac. Uh but I'm going to pass the microphone over question. Uh thanks. And you mentioned the Spotify talk yesterday about the zone-aware routing, and I was wondering if there was anything about their solution that was like new to you, or like especially interesting? Yeah, um yeah, so so they're they're doing like a a proxy-less

version of this. Uh so, they're they're not necessarily using dynamic modules, they're just using like like uh like an XDS gRPC client. Uh but yeah, the the same uh I guess architecture can be implemented within Envoy. And that that's what what I'm trying to do here. We have just a couple of minutes left. Is there any Yes, there's a question. We Yeah, Isaac, you got friends. Did

you plan all of these questions? Uh thanks. Uh sorry if I'm maybe I'm hallucinating. Uh you said you uh migrated from HAProxy. Uh yes? Yes. Uh and so, I just have a question like how was your why? Uh how was the migration going? Did you kind of encounter any problems? And how was the performance and overall like your experience after migration? I don't know. Because we also

want to do the same. I don't know, so Yeah. Yeah, um I I I did a talk at KubeCon a few months ago, actually, uh about exactly that. So, I I'd say go go check that out. But uh yeah, overall, the the why was just we wanted to get off of uh like we wanted to make our load balancing Kubernetes native. We were previously rolling HAProxy on

on bare-metal servers. Uh and then we were also having some some challenges with the way that we did routing. Uh zone-aware routing allows us to get some of that fault tolerance while still getting the um good enough equal or like good enough uh that it solved a lot of things for us. But um yeah, happy to chat afterwards if if you have questions.