Cloud Native Theater | EnvoyCon: Inside Spotify’s Envoy Architecture: What We Le... Leonardo da Mata

In this talk, Leonardo da Mata, an engineer at Spotify, discusses the implementation and optimization of Envoy in production. He explains the architecture, detailing how requests are processed through a custom version of the Envoy proxy that decorates requests with Spotify authentication and metadata. He outlines the challenges faced during the migration to a new architecture that aimed to improve request processing efficiency and lower latency. The speaker emphasizes the importance of community involvement and adherence to open source principles, sharing lessons learned including the impact of filter order on performance and the need for thorough testing during migration phases. Ultimately, the changes led to a significant increase in request handling capacity, showcasing the benefits of utilizing Envoy effectively.

Yeah, so uh I'm Leonardo da Mata uh and I'm a engineer at Spotify. I'm going to share you to you um some learnings that we had using Envoy on production. And maybe you can learn from us our mistakes. So, oh my god, how do I going to pass the slide? There you go. Yeah, so um here's a summary of my presentation. I'm going to tell you why

uh we had to what is architecture, uh why we did to had to improve it, the plan that we chose to improve it, uh the problems that we faced, and uh if it's worth it to run Envoy in production. Spoiler, yes. Um yeah, in case you don't know, uh Spotify is the largest uh digital audio streaming, and one in 11 people in the world uses our services.

But, you I don't know if you know the us. And the perimeter, what is it? Like, I mean, very simplified version, but you can see what it is. Um but every request goes through GCO G GCOB, and then reached our version of Envoy proxy, uh where we make some uh decoration on it, and sends to to our back-end uh services with this internal information that is used

by Envoy. You can think like a motorcycle with the sidecar with cool features that carries the request through our infrastructure. And it's the front door, right? We are handling 10 million requests per second, actually 11 million. Uh I got someone updated the number yesterday on the presentation. Thanks, Yannick and Anya. and it is powered by this custom custom version of Envoy, and it also has uh and

on the side used to have this side car which is a Java process that we used to call using X out Z. Uh, do to do our decoration. So, it means like adding Spotify authentication information, some metadata that we needed, and so on, right? Historically, this is uh, every request were going to this path and we were calling this JVM process to in every in each request

increasing our latency. Yeah, so as I mentioned we wanted to be more involved to I didn't mention that yet. So, we wanted to be more involved in the Envoy community and also we wanted to make more standardized. We are actually abusing this uh, decorating the requests and we wanted to be more close to the architecture itself where we have the filters in the chain to change the

requests. So, we decided to we saw opportunities here to change your things. one of the things that we learned throughout this process is that we were just doing 500 requests per second per CPU. Uh, whereas the community had the benchmark of a 10,000 requests per second per CPU core. So, we were way below that. And also we had we were actually fighting the X out Z um,

and the the async model of Envoy uh, with this process. So, we found opportunities here. We decided that we could move bunch of these uh, Java features away from this our decorator. And we we wanted to move as close as possible to the open source version. So, everything that was already ported or could be ported uh, to the open source version would uh, go. And we also

found some features that we had on the decorator that could be moved away from from the this Java process into Envoy filters. So, we started a plan. Okay, now we we already had Envoy running and this has been discussed on multiple times and everything. But, we had this sidecar that was a pain point for us. Uh, so we needed to find a plan to migrate this. So,

I mean everyone sees this like if they're doing a migration deployment or anything like this on scale. You you you put a plan and you decide on Okay, I'm going to put on one region, check how it is, put on canaries, check how it goes, and then follow a process. So, we we actually established this this Okay, we're going to enable, disable, compare, see how it goes.

And because I'm going to quote that, replacing a sidecar is a behavior migration like a it's not a code migration. It is it wasn't simply swapping a thing. We had to try it out and see what was going on. And we decide I mean what we did to to to actually I mean you could think it's simple, but what we did like to compare both approaches. We

had two different The request was going to through two different paths duplicated. One with the old version and one with the new version with the filters. And we're simply comparing metrics, counting the difference between the headers that we are adding. a very simple architecture uh, that uh for comparing. Not like huge uh planning or anything crazy. We're simply counting the differences in on the comparison. And this

actually helped us to understand, okay, things were not working as expected and what else we would do. So, we had the plan, and we started the deployment, we did the comparisons, and uh one of the things that happened is that we had to revert multiple times. The reason why we had to revert multiple times is because people going to use your APIs the way they want. They

are not following the standards, they are not following the the good practices. They They need to solve a problem, they're going to use it. So, this is something that you should be aware and should uh take care. and it's part it's part of our business or deal as a engineers to know that it's okay to make mistakes and and go back and and try it again. Uh

this is part of the process. One thing that we didn't expect, oh, actually, uh uh this is I swapped the order, but uh one interesting prob- problem that we had is that we had the talking token prefetching that has to happen on every uh startup of our processes of this sidecar. And we tried to build this internally with Envoy filters. We spent a long time implementing and

trying to deploy and reverting. Uh actually, uh we found out that okay, we can just get this piece of Java and do as any I need container. Uh use the Kubernetes directives for for initialization and be available to listen to requests. And then this idea that wasn't the the perfect solution for us because we were planning to do it for for completely on filters. Actually worked better

for us. So maybe not everything needs to be perfect when you're doing such migrations or such changes. What we didn't expect was a big incident that we caused. There is a there is a blog post that I'm going to share the link for it. But basically when we deployed everything the order of processing the request changed. So a filter order changed and this actually triggered some bug

on one of the filters that we have for rate limiting. And this caused Envoy to crash because Envoy was crashing more requests were being retried and then this caused a crash loop oom in the causing out of memory for the So we had to basically drain some region and move everything to another region revert everything and try it again but we eventually figured out that what was

causing this oom which is a filter order. So make sure to know that like filter order really matters. Also some other things that we learned like during this process. We had Java before so there's no Java a garbage collection to save you in C++. like trading that the thread model for for Envoy. Don't don't fight with it use it. So mutex for us so saved a problem

that we had with caching. Also try to stick as much as you can with the community the open source and everything. We had like a complex way of building the image. When we simplified that and moved to a more close to the the what we have on the mainstream, it was much easier for us to to add uh put our custom filters and and make the changes.

And also contributed back. I mean, we what what changed? Like what it was it worth it? Uh we had an improvement from 500 to 2,000 requests per second per core just doing the swap. Uh and more recently we achieved 4.5 thousand K requests per second per core. Is that a typo involvement? That's might be a typo. Um we I mean I'd say I at least I am

more involved, but uh we are trying to be more close to the community. I see I see so many people inside the Spotify using Envoy and evolving and and uh we have AI gateway um that is also big of a deal as well. And you can see from my motorcycle now it's much modern. It still has the same features. You see the the nice dog over there.

And doing the same like the same things, right? Uh so, yeah. I mean, I've already spoke to those, but make sure to formalize all undocumented decisions. Like in production things going to use everything in different ways. Um Uh for us doing a side-by-side comparison was really helpful. So, don't be shy to just okay create something that can duplicate the and and send to some some place so

you can compare. And yeah, I think that's that's it. Uh here's a QR code with the all the links, but you can take a pictures pictures if you want. And I'm happy to talk about this outside. Yeah? Thank you. I'm Yeah, thank you, Leo. Thank you so much for sharing your journey. We do have a couple of minutes to ask questions. And Jens, while we do that,

you can come up here so we can get your computer hooked up. But why we do it while we do that, we have the opportunity to ask Leo some questions. Do you got Anyone got a question? I got my microphone here. Fantastic. Hey, Leo. Thank you for the session. Just wanted to know from you like I know you said you already did this improvements and have got

to 4.5k. What is it that still you have some ideas on what is still not causing it to meet the 10 10k? >> Yeah. So, because because we're using the X out Z, that's mainly proposed wasn't to decorate the whole request. And some of the the things that we do, we could fail fast the request or or stop the the processing or even give a return to

user. We're actually We're actually having this round trip to the Java process for doing that. And the Java process was waiting for everything to to go to answer the request. So, with the filters, we're actually using better the the whole architecture of Envoy, the way it's meant to be. So, I think this is the main reason. We haven't explored like why we just saw okay, doesn't make

sense. Like you can have 10,000 10,000 requests per second and we were just processing 500. So, there was a bottle bottleneck there. I mean, it was not just okay, we have internally Java. We are very happy about it. The backends is not the Java is not a problem here, right? But I think the Envoy expects things to be processed. The whole like processor per CPU and and

the idea of a sync that it has internally. With the X out Z, it's an external call. So, maybe maybe that's that was the reason. Any any other questions for Leo? There we go. Yeah, so my question is you had Java sidecar and Envoy is C++ sidecar. How did you between making your own C++ or Rust sidecar which could uh So, we did we we actually removed

the sidecar. The sidecar was a container running alongside on the same pod as the the Envoy. We actually removed that and put everything inside of Envoy the Envoy process itself. We decided to write in C++ because at the time when we did that, we didn't have all the languages supported yet to to write everything. Now, we can do a Rust. We can We have ways of doing

with the extensions. Yeah, so actually those who are interested in how you can inject your own logic into Envoy proxy in the runtime, Ro Head is going to speak after Yiannis. So, those who like to customize things for their needs, definitely you know, come and listen to that. I think that's that's it. Okay. >> Well, well we have we could have time for one more question. Okay,

okay. But Yiannis, can you put your computer up here? >> I'm going to remove that. Yeah, remove that it because we can let Yiannis play around with the cable which is we know one of the hardest things to do as a conference speaker is that part. But any Yeah, thanks for the talk. You mentioned that you've actually using Envoy only on the edge part, but yesterday I

attended a talk where actually Envoy is not used inside the mesh. So, can you talk more about We have Envoy running as on the service discovery. We have Envoy running. There's a talk the last KubeCon KubeCon about running it as on the what we call web enrichment platform. To because we have more complex features. So, Envoy is not used only on the perimeter. I'm saying that the

perimeter has Envoy as the our entrance port. But it's not like just that.