From NLB Sprawl To Mesh Efficiency: How Skyscanner Handles 60M Reque... John Clark & Steven Thwaites

This talk details how Skyscanner has enhanced its service mesh efficiency by evolving from load balancers to a multi-cluster setup and finally to an ambient mesh architecture. The speakers, John, a senior software engineer, and Steven, a solutions architect, outline how Skyscanner operates across four production regions with six clusters per region on AWS. The transition to Istio 1.4 was pivotal as it provided mutual TLS and streamlined service communication. They discuss the complexities and trade-offs of moving to a more advanced multi-cluster setup, highlighting how they optimized costs and performance while maintaining failover capabilities and observability. The introduction of the ambient mesh entails replacing traditional sidecars with a centralized z tunnel for managing service traffic, which significantly reduces resource consumption while still ensuring robust security and telemetry.

Good morning. Um, today we're going to be talking about how Sky Scanner have made their mesh more efficient by moving from load balancers through to multicluster and then through to ambient mesh. A little bit more about me. My name is John. I'm a senior software engineer at Skyscanner with a focus on Kubernetes and STO. Hi everyone, I'm Steven. I'm a solutions architect at Solo. um and I

work with my team and with Skyscanner on their ISTO feature needs and designs. >> So uh a bit more about Skyscanner. We run everything across four production regions. Each region gets six um clusters that we run on AWS. We split these across our availability zones. So each a gets two clusters that we keep isolated into the a we fully on spot with Carpenter. We connect everything together

across the globe with ISTO and um we yeah we serve about 600 services at any given time 35,000 pods and about 60 million requests a minute. So going back to the origins of how Skyscanner set up their mesh um in 2019 we have a project globe which is essentially to modernize what our mixed cluster ECS mix was. We've decided to split large clusters into smaller single A

clusters to reduce our blast radius, allow us to do phased upgrades without risking breaking a whole region. One of the challenges with doing something like that with splitting out clusters is how do you connect everything together? So at the time we chose ESTO and STO 1.4 which gave us MTLS. It gives us all the DNS abstraction that we want and telemetry out the box. So Steven's going

to talk a bit more about itself. Yeah. So many of you will already be familiar with STTO. So I'll keep this really brief and just tie it into Sky Scanner story. Soto sits your in your infrastructure layer and it handles all of your servicetoervice traffic transparently. So you your application doesn't really know it's there. You don't need to add any libraries or any SDKs and no code

changes. A sidecar proxy is injected into each pod in the mesh. Um and that from that point forward every bite benefits from having uh being secured with mutual TLS. Um it emits telemetry so everything is observable. Um and is controllable as well. You can control how the map traffic moves. The control plane SDOD watches for changes in the Kubernetes API server and then it generates envoy uh

proxy configs and certificates and then pushes them down to each of those proxies over XDS. So main reasons people will adopt will be around you know connecting services securing them controlling their behavior and getting observab observability benefits and most what we see at solo is that people will adopt for one of those reasons probably security or observability and that the other ones will become loadbearing over time

but for a mature um customer an advanced user like Skyscanner all of those were important from the outset. >> [clears throat] >> Thank you. Um, so at Skyscanner, we try to provide a core capability to all of our developers, which is that they can reach other services by simply calling service name. Skyscanner.io. So we use STO to abstract all of that away behind the DNS and they

don't need to worry about how to get a request to another cluster, how it's going to go to another region. They get MTLS out of the box. They get metrics out of the box from their sidecars. um they get automatic failover so they can if the request hits a cluster that's having problems it will automatically bounce over to another cluster and this all happens in in a

single request behind Sky Scanner IO to go into a bit more of the technical detail about how we do this um each cluster essentially has one IP it's one IP behind a gateway um we then use things like virtual services to basically well we use service entries to point at those IP PS and say that this is cluster A, this is cluster B. With a virtual service,

we can then say that you're going to get eight retries. And with the destination rule, we can say you're going to fail over or you're going to outlaw detect in three failures, which basically means that in a single request, you'll get nine retries and you're always guaranteed to bounce over to another region. The way I kind of look at this is that clients load balance clusters and

then the cluster load balances the pods that they go to. our first generation mesh in 2019. We built this. We built multicluster before multicluster was a thing. We have six clusters in a single region. How do we connect them together? We deploy something that we call a default gateway which is just a common ingress path into that cluster and each default gateway gets an NLB and we

just call that NOB with an IP and it's easy. IP address of an NOB has a locality. We can turn that locality off. We can train it. we can outlive detect it. This works this works very well. It's standard. It's easy to understand. But the costs really started to go up. When we first started, the costs are no nothing. But when more and more services start to

use this architecture, those costs quickly started to go up until they cost 10% of our entire cloud bill. So one of the problems that we have is we've built a a very custom platform based on multicluster from six seven years ago. How do we actually move that to what is known as multicluster today without breaking everything? So Steven's going to explain a bit more about multicluster. >>

Sure. Yeah. So by the time multicluster kind of arrived in in uh open source ISTTO around reaching maturity around sort of 2020 and 2021 uh this was the architecture. These are the architectures that pe people would go for. So um on your left there is kind of the happy path where you have multiple clusters within one network. Um and that's quite straightforward. All the pods can just

see and route directly to pods in other clusters. We have the uh sod reading from each uh partner clusters um API server and watching for changes in the remote cluster and otherwise handling things in a similar way. when we have a in in production and in larger customers, the reality is more often to have clusters in different networks. So we need to be able to route across

uh those networks in a similar way to the way that John was describing. So we um in this case here we will have um we'll have east west gateways which are published as uh network load balancers and then the uh the remote pods will be published and advertised on that east west gateway and forwarded to the uh remote pods in the in the opposite um in in

in the secondary uh cluster. Now what's important here to understand is that the default behavior of outlier detection. So for example, if a request comes from cluster A and is going to cluster B and the pod is misbehaving in some way, you will get the that specific pod um ejected from the pool of endpoints and so it will fail over to another pod within that same cluster.

That's not the desired behavior at Skyscanner where they wanted to basically have three retries on an endpoint and then the entire cluster is ejected. So everything that John is going to talk about now is really about making sure that um that pattern is applied um in modernist. >> Thank you. So um multicluster we have a fixed requirement essentially we need to be able to call a cluster

with a single IP multicluster out the box it's just it's not the way it's built so same cluster in our original design even for same cluster traffic we still went through a load balancer to come back into our own gateway that's the easy change we just switch that to the IP of your service cluster local address um within the cluster make it well known and that's easy

but multicluster is the What we did was we used multicluster. We connect to all the other clusters in the same region. We then only collect the IPs of this gateway that we transit everything through for each cluster using a um something called cluster local in the steel settings and then we deploy an egress gateway into every cluster that we then call as a single IP and that

egress gateway then chooses a gateway pod in the other cluster. So it's a bit of an odd setup, but it replicates what we have that is a TCP load balancer. We're essentially putting a TCP low balancer in the cluster for each other cluster in the region. These um egress gateways, they're layer 4. Um they don't do any termination. We don't do any processing with it. They just

quickly pass through and the flow continues just like it did before. And we're able to then just update all of our IPs, change it from IPs of load balancers to IPs of um egress gateway services in the cluster. This gives us a second generation mesh which we built in 2024. Um, we managed to basically kill off all the NOBS across the cluster. As I say, it's 10%

of our entire cloud build. So, it's a big win. We repoint the service entry IPs at the new pass through gateways. We do this as a platform team. We don't ask anything on our developers. They don't even know about it. Um, and it all just works. We have some trade-offs. So obviously STLD when it uses multicluster is it repeats all the work it's done in its own

cluster with every other cluster that's monitoring. So our STLD scales up by 10 times. But this is a small price to pay when we see the savings that we get from multicluster. But there's always things to be doing. There's always advancements coming out in this space and um one of the things that we still are always want to do is simplify things for our users and set

sidecars are still a problem today. So we started to look about how we can possibly move to ambient mesh with such a interesting architecture which Stephen's going to talk about now. >> Cool. Yeah. Thanks. So it's it's worth spending a second to contrast um sidecar STTO which is what everyone has kind of known and loved over the years um with with ambient which is now sort of

has reached maturity recent in over recent years um as well. So bear in mind that even when you get rid of those NLB costs, you have a sidecar in every single pod that is participating in the mesh. So with 35,000 pods, that contributes a substantial amount to the CPU and memory running in your cube So Ambient handles this a bit differently. We do away with the sidecar.

Instead, we have a a pod, a demon set which is running on each node in the cluster. That's a uh just handling layer 4 only. So we're separating the concerns of handling layer four and layer 7. So this uh layer 4 only zunnel um is handling all of the uh MTLS the uh layer 4 uh policies um you know authorization identity um and then optionally should we

need it we can deploy an envoy waypoint that can also itself be shared. So instead of again having one side car per pod, we can now have a Z tunnel which is as a demon set across your entire data plane. Um and then we can have a waypoint proxy if we need it. Um you know per service per name space. Um so we're talking about reducing the

infrastructure cost and also the complexity of management uh drastically. Another another reason people enjoy uh moving to ambient is also because you know if you needed to upgrade a envoy proxy um like a sidecar pod you would um uh you would have to actually restart your application as well. Your whole pod would need to be restarted. So this does away with all of that and the ambient

mesh sort of becomes part of the background. Okay. And just this also works with multicluster as well. So just looking at the sort of the data path in a normal cluster we'll just have a client working and hitting the Z tunn be rooted to the Z tunnel um because it handles IP routting as well and then that will be um sent to uh the other Z tunnel

in in the partner cluster and then optionally at the waypoint closest to the destination any L7 policies will be handled. Similar again on um when you have a different network um architecture uh we have east and west east west gateways as well that also can use NLBs um and the in in open source the method of sharing um secrets and of sharing state um is the same

as in open source um as same as with sidecar solo's distribution of STO is slightly different in that we can actually do XDS peering between the um STOD instances is in each cluster and that means that there's kind of um less that has to be exposed and there's just more lightweight um sharing of information only what's needed between each of the STOD instances. Okay. And then just

to double click on the actual resources which are being used um beneath the sort of in infrastructure layer we're going to make use of um so a service entry kind of declares that STO is in control of a uh of a service within the mesh and then that is selecting um selecting different workload entries which describe where pods are um in local and remote clusters and they

will pick up the endpoint IPs and those will be labeled as well with topology hints as well. Soto is aware of what's nearby and what's remote. Um the actual policy of what you want how you the behavior you want is encoded in the destination rule. So that allows us to configure where the outlier detection um how much outlier detection we want to tolerate. Um and then also

uh where the traffic should be rooted when outlier detection is is found. Um so these are the kind of primitives that Skyscanner has to work with. Um and then when when you're able to actually work with those level components, you can achieve um very different uh behaviors and you're able to sort of retrofit into into what Skyscan has been doing. John, thank you. So um if you're

following our complex architecture, um for anyone that's tried to move to ambient with something with an existing setup, it's difficult. Um we have very complex architecture. We have a lot of challenges with moving to ambient. We've got to move all of our sidecar logic to centralized D tunnel or waypoint decisions. We have to think about a new multicluster design. Um we have to got to replace all

the telemetry that gets generated by sidecars today. And we have to choose where to we can use the primitives that Steven's just discussed or kind of roll our own. So another complexity um we decided to fix multicluster or fix multicluster for us by taking what solo provide their peering setup through XDS and using the workload entries that solo build the STO builds from pod IPs and other

clusters and then we sync those workload entries to a Kubernetes service endpoint slice and this gives us our our requirement of still having to call a whole other cluster with a single IP. how it's a very difficult thing to do in multicluster. So now that we have a single Kubernetes service with virtual endpoints behind it controlled by our multicluster controller working with Solo's multicluster controller um we

can now call that single IP and um we just go straight to those pods in the other cluster. We no longer have to use egress gateways. We no longer have sod scraping f six other clusters. It's just a lot more lightweight. One of our other biggest challenges is um telemetry. So this is just an example of um one of our dashboards and we build this with um

spans that come out of sidecars. So every request that is sent from a pod generates a client span and every request that's received by a pod generates a server spam. We then pipe this into open telemetry, convert it into metrics, send it to our dashboard or send it to our observability provider and we build on top of this. We build dashboards, we build alerts, and we build

SLOs's. We have um other dependencies on this type of telemetry. So how do we use this without sidecars? Um just to give another example of what that looks like just now. Yeah, sidecar on the server side emits a span. span turned into a metric we call HTTP server duration or gRPC or so on and the same thing applies for a client and this gives us the duration

of those requests as well as status codes um response codes all the data that we need for um our observability. So the solution with ambient is a little bit unorthodox. Uh we to get this data we need layer 7 and layer 7 no longer happens at z tunnel. The first point it happens at is waypoint. So we've moved clientside telemetry to be generated from the waypoint instead

and we essentially take the identity of the pod that called it that goes through D tunnel and we update the identity of that spam. Likewise on the server side we no longer have a server sidecar. So how do we generate that server spam? And this is where having this default gateway that we built seven years ago has actually come in useful. We can take the span that

comes out of that do the same set the identity to what the pod has been called and we get the same data. We obviously have some downsides with this. Our latencies are slightly changed. It's no longer as close to the source as it was but we find with cunnel that this is almost negligible under a millisecond and this this is generally indicative latencies [clears throat] and people

can set up their own telemetry in their services anyway. So an overview of our third generation mesh. Um all of our Skyscanner IO traffic is now going through a waypoint instead of scikar. Um we have zero development changes. We've not asked anything on our service owners. Um we've managed to still retain all of our failover um all of our detection. We've got certain header based routing. We

support githops um subset rollouts. This all still works in ambient. We've ditched the multicluster egress gateways and we've scaled this right back down since it's no longer having to repeat all the work it was doing before. And just some learnings. Um it's been a tough road to be honest. We've been looking at ambient for years now and um we've now been we're in the middle of our

migration to production clusters in this. Um ambient is great for green field. If you don't have STO already, you can deploy ambient very easily. If you do have VTO and you use anything that is in some way complex, it's difficult. Um, take your time. We are migrating new, we're running new clusters with Ambi instead of trying to migrate existing clusters. Um, which we have found a lot

easier. Obviously, it depends on your architecture. And I would say don't be afraid to build some custom glue if you need to make it work. Um, we've built a platform and custom glue so we can't have the expectation that we're going to be able to get rid of all of it. Um, but we find that the trade-off is most definitely worth it. And that's it. Thank you.

[applause] We have time for questions. Maybe we have time for questions. Nobody said we don't the mic. >> Hi, thank you. Uh, I actually have two questions. So, first any thoughts on how you're going to handle the Z tunnel upgrades because that's a total shift moving forward from restarting the sidecars to having to basically take off uh out the node. And the second question uh do you

ever have Lua in your gateways and how will you replace that assuming that ambient does not support Lua if I'm not mistaken. So on your first question, um, we haven't really had any problems with this. Again, we're still in we're still rolling out to production, but we with node restarts and so on, we tend to cover this quite easily. I wouldn't have anything specifically to tell you

that we encountered. Um, on the second topic, we use envoy filters in a lot of places. >> but we only really use them on ingress gateways and ingress gateways aren't going away. So if we were using them on sidecars is a different story because then you need to clone it to >> Z waypoint. But um for anything that we do with our ingress gateways that hasn't changed

and it's still fully supported. >> Hello. Thanks for the presentation. Very nice. I'm curious first question uh about these 2D performance. You mentioned that it scale 10x in terms of what uh memory CPU usage >> CPU. >> Yeah. Does it scale horizontally or vertically? >> Yeah. So, so we have as an example before we did multicluster we'd have three sod pods in any cluster and after going

to multicluster we had 30 for horizontal scaling. >> Oh, okay. Makes sense. Uh the second part I'm curious about uh do you uh integrate with works outside of Kubernetes uh like I know staple applications and so on and how you handle L4 level MTS and so on. Uh and this amant mode supports it. >> Yep. So um we we have a let's say an AWS account that's

got a lot of our stateful stuff databases data store. Um and we actually run other clusters in that account that we route everything through. So you get MTLS from our production clusters through to that cluster and then that then routts onto our um data store and >> it runs uh the destination cluster runs eastd some kind of a service right to handle the termination. >> It it

runs a single gateway. So it runs yeah it's it's it's a big overkill cluster but it runs a single gateway that has uh essentially a virtual service behind it to route to the correct destination. That's all it is. It's a >> it doesn't works. I mean because it has to be terminated. I'm curious about that part. >> We do we technical details. I think for the that

routting people they still call it over HTTPS. So I still we still call Postgre over HTTPS but then we wrap around that between our sidecar or our waypoint and that gateway and then it gets unwrapped and then sent on still as >> I get >> and that's unchanged with um Ambient. Mhm. >> Um >> the last one, how you handle Easter upgrades in general since you're using

uh gateways, right, and envoy filters a lot, >> it tends to change, the contract tends to change when you upgrade it. Uh how you handle that? Uh >> as in like the envoy like the config changing behind the scenes if we do an upgrade. >> Yeah, exactly. >> Um so these things are always to be expected because we have 24 production clusters. We do the same with

dev. We split it all out. We have channelneled rollouts. So we will roll and upgrade to one of those 24 clusters as an example and we stop all the traffic coming to the cluster and we slowly reintroduce it. If it breaks we roll back and then we fix. >> Thank you very much. >> Hey um thanks for the talk. So you mentioned the need to introduce um

the way points to get increased and more um better telemetry, >> right? >> Is that per service? >> No off. Yeah, that's probably something I should have called out. So then it would be just replacing that with that >> in STO 127 or something they started supporting cross namespace waypoints. So we after weighing up the options we decided to deploy one waypoint per cluster instead. So we

have a single waypoint in the STO system that everything uses um which has its trade-offs but because of the way that we've built and we have all these different clusters we can we we feel like we can take that risk if we break it we'll only break one of them. Um so yeah okay but thanks thank you >> thank you for the talk very insightful uh I

wanted to ask you when you were in the process of migration to z tunnel and ambient mash have you seen a significant impact on latency inside the network and if so like how did how did it measure how did it compare against the existing >> so again still mid migration but we haven't found anything um our latencies on these metrics are slightly different but actual raw latency

if you time the request from a client from the source we haven't found anything notable at all um we right now with sidecar we essentially have three layer 7 hops sidecar gateway sidecar but we're trading that now for a layer 4 Z tunnel and two layer 7 hops and then a layer 4 Z tunnel >> my thinking was along the lines of Z tunnel basically becoming a

point of contestion this is a contested resource that is used by multiple other resources And the question then is is it noticeable in CPU usage in resources in any way on the change of >> it's super performant really very performant. Yeah >> not yet for for SK not yet but yeah like it's built from the ground up in in Rust I think and is um from a

resource perspective we haven't found anything of it having problems yet. >> Okay thank you. >> Yeah being a keyboard. Thank you. >> Are there any more questions? >> Hey, thank you for running Sky. Amazing. Uh the results are amazing. But question, how do or um orchestrate the rollouts for example like you know devstage broad or like between clusters and multiclusters. So how does it work? >> As

in for like uh upgrading STO or something like that. >> Um yeah so we as I say we just have we have what we call channels. So we have a development channel. We have a alpha channel which is three of those 24 clusters. We have a beta channel which is eight of them. And then we have another channel which is all of them. So we just roll

everything AC in channels. Um we we basically don't have any exceptions to that. If we're upgrading any software or any tooling um it will always go through these channels and we have when we do the first one we stop traffic and then we reintroduce traffic slowly and then we don't do that for the rest. My main more question was about the tool set that you use for

that. >> Okay. Argo CD. >> Oh, so yes. So um we yeah Argo CD um we we build everything in manifests and everything's a chart and we use Argo CD to sync it all and on top of Argo we don't just autosync everything. We we're actually moving to Argo workflows now but we basically use a pipeline to call components in order as it goes rather than just

autosync from the resources. So we can call it, we can sync everything in the correct order.