Building the Next Generation of Multi-Cluster with Gateway API - Mike Morris & Alan Grosskurth

This talk focuses on the evolution of services in Kubernetes, specifically through the lens of the Gateway API and multi-cluster services. The speakers discuss the limitations of the old service model and share lessons learned over the years regarding service management and API design. They introduce concepts like separating service front ends from back ends to enhance flexibility, and they detail experimental designs including the Cluster IP Gateway and a new Composite Backend resource for multi-cluster environments. The presenters emphasize the need for a more adaptable architecture to accommodate modern technological needs, particularly in the context of AI workloads and resource optimization. They also outline their plans for prototyping innovative concepts that could improve inter-cluster communication and traffic management within Kubernetes.

All right. Uh welcome everyone, and thank you for sticking around for the last talk of the day, I believe. >> [laughter] >> All right. Uh hi. I'm Mike. I'm a product manager at Microsoft working on upstream open source networking stuff, Gateway API and Istio. And >> Hi, I'm Alan. I'm a software engineer at Google working on service networking. And so today we'd like [clears throat] to talk

to you a little bit about Gateway API and uh the data model and multi-cluster services. So, the agenda today we're going to go through some background and lessons learned about the history of services and how services have been used in the past. Um some of the lessons learned trying to do different things with services. Uh then we're going to talk about the desirable properties of what we

want to have out of a solution for multi-cluster involving Gateway API. Uh we're going to go through some ideas for solutions, um talk about some experimental implementations and prototypes, and then leave with some closing thoughts. All right. So, who here has ever regretted an API design decision they have made? And who has had to live with it because your users expected it to not change? So, we've

had that same problem in Kubernetes. Uh service is one of the oldest APIs. It is very much a load-bearing API. It is used by almost every user of Kubernetes, and it was designed 10 years ago. So, things have changed and evolved, and we have new constraints and new design patterns and new workflows that we're trying to enable, and we've kind of gotten to the point where we

can stop just like patching things on top of service. Uh the maintainers of Kubernetes are also at a point where it's not really sustainable to even accept those kind of contributions to to keep just like duct taping additional things on top of it. Um so, we need to start rethinking what we do with this uh and how we can kind of modernize some of the stuff under

the hood uh to really allow the flexibility that we need for some of these modern demands that we're facing. So, one of the things that we kind of discovered at service uh in through Gateway API when we were working on Gamma, so the Gateway API for mesh uh initiative, is that service really kind of has two facets to it. So, there's a front end, which is how

you might be familiar with interacting with the service. It's a human-readable name, there's a FQDN hostname, the like .cluster.local address that you're using it to call that service from all of your other services that depend on or all the other workloads that depend on in the cluster. And then there's a cluster IP address. Um and if you've used a service mesh at all, uh it can be

desirable to hijack traffic at this point to redirect it to a different back end. Maybe that's for a canary deployment, maybe that's to seamlessly reroute it for multi-cluster needs. Uh there's a handful of different reasons why it's useful to do that, where you're not uh where you're kind of like skipping a up uh the default back end part of the Uh and then if you're using Gateway

API though, even when you're using like an HTTP route attached to a gateway and you have a back end ref service, you're not actually using any of that front end part because the gateway is serving as your front end to bring traffic in from the public internet uh in the ingress use case. What you're using from service is the back end part, which is really just a

bucket of end points. Uh it's often pods that uh are they each have their own end point IP's, and the gateway is just a means of routing to those pods. So, these concepts were introduced in Gateway API GEP uh and kind of explained in more detail there. Uh but we found it a really useful model for thinking about the different parts of a service. So, service does

both of these, which can make it cumbersome, uh heavy implementation detail, and you don't always want both of them. Sometimes you only want one of these components. typically when we want to consume a service uh as a dependency for some other workload that you have, the front end concept is you want that to be present where the service is consumed, but especially multi-cluster, back end workloads are

often spread across many different clusters and sometimes even heterogeneously. So, you might have a dozen end points in one cluster and only two in another one depending on the size and scale that you're operating at. Uh and this gets even worse with AI and GPU availability, as we'll see in a minute. So, service both does too much and it's still been too limited. So, it's limited to

L4 only, it's not extensible, and there's a lot more that we want to do uh with new protocols and methods uh now. So, who here has heard of the multi-cluster services API, commonly MCS API? So, MCS API was one of the earliest experiments with doing an out-of-tree API, so outside of Kubernetes/Kubernetes. Uh it's a set of CRDs. The common ones you may be familiar with are service

import and service export. Uh and this basic model makes a lot of sense, but we've come to find some constraints uh as folks have tried to adopt it or implementations have looked at uh this and made determinations of whether it's the best fit for their needs or not. I've been involved with MCS API for I think four or five years now. Um and there's definitely been some

lessons learned uh also working with projects that have implemented it. So, one of the things that we've heard again and again from users is that needing to update code to switch to a .cluster.set.local uh address instead of your .cluster.local is just it's too heavy a burden for adoption, especially in large organizations where that requires updating code across hundreds and hundreds of workloads. namespace sameness, uh that's basically

the idea that a service by given the name foo in one namespace uh on one cluster is logically expected to be equivalent to a service by that name in that same the namespace of the same name on a different cluster within a cluster set, so with within a group of clusters. This is a really important concept to avoid things like squatting or having your traffic redirected something

that is a completely different web service or database service than the one that you're expecting you're directing it to in multi-cluster. Um but it's also difficult to retroactively apply to legacy environments or especially if it works across large organizations where cluster ownership might be distributed. Maybe uh you've had a proliferation of clusters because they were so easy to create that you have many different teams each managing

their own clusters and each have different namespaces and different conventions for what those things mean. MCS API, it was designed to be flexible enough to to accommodate different network topologies, and that's one of the things that we definitely want to preserve when we're looking at uh new concepts uh what with without requiring kind of flat or IP spaces uh across clusters. We we want to um also

allow overlap uh and intermediated by gateways, but not require gateways because sometimes that's an extra hop that you don't need. Um and then yeah, the the cluster set concept and cluster properties can be useful mechanisms, but it's not always been well understood how some of these other APIs intersect with MCS API scope. And then there's also implementation details. So, uh if you're a project considering implementing MCS

API, one of the things is that even though you're creating the service import resource, you the only way to get a cluster IP currently or or historically has been to create a basically shadow service underneath it, and the need to do that ends up leaking service details up into some of the newer iterations of the MCS API. So, that's kind of like one of the unfortunate complexities

of this dependency on a really heavy resource. Um also expecting control over watch resources in one cluster and write into another can sometimes have implementation, security, or responsibility concerns, uh particularly if you have multiple implementations, uh you have to make sure that you're not trying to have competing reconcilers uh doing this. Um and then while distri- distributed uh or centralized managed models are both possible, so distributed

models like Cilium has does not have any kind of central controller for MC for their MCS implementation, uh and then Azure and GKE fleets uh both have like a centralized managed model. Well, both of those are possible, but there's no reference implementation and not really much guidance to help uh projects figure out how to design and build something like that. So, MCS API, it's been stable for

many years now, and there's a handful of projects that implement it, and it works pretty well. Um but it's also no longer enough for some of the modern demands that we're seeing. Uh one of the things that was not an issue at all when it was designed is flexibility. Service was kind of just the way to do things in Kubernetes. Uh but in recent years we now

have gateways uh in other clusters that we might want uh have cross-cluster traffic between. Uh we also have inference pool in like the Gateway API inference extension recently, and that was one of the kind of things that kind of prompted rethinking some of this architecture and thinking about how do we do something that is going to be more future-proof, that isn't directly dependent on service, and gives

us more of the flexibility that we need and also allows for some of the other advanced capabilities that we're looking for. So, like spillover is one of the things that particularly with AI workloads, there's an increasing need to be able to have excess capacity off the cluster. Um maybe you have a limited number of GPUs within the current region that you're in and you want to failover

to other regions or to failover to a managed provider that uh you're also depending on the model that you're using, uh you may want to kind of optimize the routing of that. Um and then efficiency is kind of one of the other challenges. So, with microservices and commodity hardware, you typically have a pretty even distribution, uh especially for some of the historical simpler uh geo-redundancy cases. Um

and round-robin endpoint selection may have been more or less good enough. Um but with sparse or regular or highly differentiated GPUs, where if you have like your GB200s, but you can only get a handful of them in one specific region, what they can their capacity to serve is completely different than uh the bargain-basement GPUs that you may be able to get uh in other regions. um there's

been some preliminary work investigating Envoy's load reporting service and applying that to other applications uh such as Cilium to think about how to take some of that capacity to communicate um the actual load and availability um of your endpoints uh so that you can do more intelligent load balancing. And that's one of the things that just MCS API had it has no capacity uh for communicating any

of that information currently. uh inference pool import is basically a short-term hack. Um hopefully, when we actually build something that's more fully future-proof, this shouldn't need to exist anymore. Uh and we want to kind of get ahead of the creating a pattern of X import for the next half-dozen resources that we want to figure out how to expose across So, there's a handful of properties that we've

determined would be desirable in a potential future solution here. Uh one is the ability to separate front end and back end, but still connect them together in a flexible way. Um another uh that we've discussed in Gateway API for a while is the opportunity for cluster IP gateways. So, in including a DNS hostname and I'll just go to get into an example of that in a little

bit. Um we also want to be able to change routing details. So, to route to multiple clusters, add L7 changes, or canary deployments without changing how consumers address these services. Uh we want new types of backends that are not Uh we want to avoid requiring specific network topologies or gateway deployment models. Um and uh it with all these these network models have different scalability, performance, and security

concerns. So, um we want to make sure that uh architects make informed decisions about how they're building this and have the flexibility necessary to do so. So, turn over to Alan, who's going to go through some of the kind of prototyping work that we've done as far as solution ideas here. Okay. Thank you, Mike. Um so so these are I'm going to go through some ideas that

are, you know, not in a final form, but um you know, very much under development, under discussion. So, so very interested in in talking to people about it. Uh so so the first one that Mike mentioned is called cluster IP gateway. Um and you and so this is actually a single cluster uh you know, specific uh data model. And so so you might be wondering, what does

it have to do with multi-cluster? So, we'll get there in a minute, but um you know, sort of the main idea is is you have these service facets that Mike was talking about with a gateway um you know, representing the front end. And so so historically, you know, the cluster local DNS domain in a cluster has been, you know, sort of special and not extensible. Um but

with some recent work in open source Kubernetes, uh it's now possible to actually reserve VIPs um from the service cider range and actually program new types of domains. So so the idea with with a cluster IP gateway is you have a front end gateway, um but instead of mapping to a load balancer in cluster or off cluster, it's a virtual service and you get a DNS name

under gw.cluster.local instead of svc.cluster.local. Um and you do get a a service IP that that um resolves to that's just like, you know, with a service. Um and then TCP route is the sort of building block. Uh so so this is the thing that connects um a front end and a back end. And so we're taking the example of a TCP service here. And and then an

endpoint selector, this is the new idea. So, um you know, there was a a very very interesting um talk in 2024 um by Pooja Trivedi and this is about uh goes into a lot of detail about about endpoint selector, but the idea is it's just the back end selection part of a service extracted out into its own resource. Uh and so this provides the sort of automatic

endpoint management based on a a selector. Uh and then so so we have this sort of clean separation um between uh you know, front end and back end. And if we look, you know, I'm going to show a slide of densely packed YAML here and this is this is showing sort of a uh and that's got a new gateway class, you know, called cluster IP GW. Um

it's a listener on port 80 allowing TCP routes and then a TCP route that's parented by the gateway uh with an endpoint selector as a back end. And so you can see it's doing the port translation from you know, port 80 on the front end to port 8080 on the back end. And the endpoint selector um maps to pods with a specific label. So, this is sort

of a cluster IP decomposed into the Gateway API data model. Uh okay. So, once we have that, uh we uh go into more um advanced things. So, so we want to bring multi-cluster into this and we want to figure out how how can we do it in a way where we can, you know, satisfy all these criteria that Mike mentioned before. So, so this is um similar

to the cluster IP uh gateway that I talked about before. Uh this the main idea here is you you can insert a new resource in between the route and the back end. Uh so so the route can be, you know, any type of Gateway API uh route. The back end can be various things and and this new resource um we're calling it right now composite back end

uh is something that um you know, in a single cluster, it it's sort of a no-op. It doesn't it doesn't do anything. But when you have multiple clusters, it provides um automatic back end aggregation across clusters. So, you you can end up sort of uh you know, merging back ends across different clusters into the same front end. So, this is this is where the, you know, separation

between front end and uh you know, really helps to, you know, basically model things. So, you can even have a front end in one cluster or multiple clusters and the back end can also be in one cluster or multiple Uh so there's a short doc about this from 2025. And if we look at the example here, you know, the the main changes from the last one are

highlighted in red here. And so we have TCP route pointing to a composite back end and composite back end pointing to the endpoint selector. Um if you look at what the topology looks like, you could pick another cluster that's in your set of by having a composite back end with namespace sameness, so the same name and namespace, uh you know, the whatever controller and gateway implementation you

have can essentially merge all those back ends and provide uh you know, load balancing across all of them. And so something interesting here coming back to the example of inference gateway, how would this look potentially with inference pool? Uh so inference pool gets the uh Hot Wheels, you know, red hot flame there cuz it's hot right now. And um so we we change from TCP route to

HTTP route and we have composite back end pointing to inference pool and we're able to aggregate across uh different um clusters that way. So, I have one more example of YAML. This is uh taken from a real art piece on display here in Amsterdam and I'll just leave it open to interpretation. Uh okay. Now talking about some experimental implementation. So, so in order to see if if

these ideas can can work, um you know, what we want to do is is try to just you know, prototype them in a way that's sort of completely out of the Kubernetes core and, you know, just just in a in a in a flexible way and see see what we can do with them. So, uh one thing we have here, uh this is a prototype of uh

experimental implementation of cluster IP gateway. Uh so just pushed onto GitHub this morning and, you know, the idea here is you have um CoreDNS with a new plugin and this can watch gateways and look at their status.addresses and synth synth synthesize gateway.cluster.local DNS records. Uh then you have a controller that watches gateways and TCP routes and endpoint selectors, creates these IP address resources, so this is a

cluster-scoped resource uh that allocates a service VIP, and then creates endpoint slices. And then you have an agent which, very much like kube-proxy, is watching uh instead of services and endpoint slices, it's watching gateways and endpoint slices and programming NF tables uh similar to what kube-proxy does, modern kube-proxy. But you could imagine this could be like any like a CNI like like Cilium also doing this type

of behavior. So so with this you can you know create a that you know type of cluster IP gateway and you know have that nice in cluster load balancing without any services. Uh and next one is how would we extend this to multi-cluster? And so the main idea here is we want to disintermediate the Kubernetes API. So why do we want to do that? We like Kubernetes,

we like the You know, the main reason is the Kubernetes API is is focused on a single cluster and we want to basically bring in multiple multi-cluster information and we don't want to have to materialize all of that information into objects in every cluster because that's difficult to scale. So so basically the controller can coming back to you know XDS. This is a protocol for subscribing and

synchronizing resources different control plane and data plane. And so you can have essentially the controller watching the same information it was before but instead of creating objects like endpoint slices it's it's synthesizing XDS and serving it. And so you have the Coredns plugin receiving XDS and the the agent receiving XDS and program you know doing their jobs based on that. And then the interesting thing here is

if you have multiple clusters you can use sort of you know some sort of hierarchy potentially of XDS relays you know exchange information and propagate across clusters. So it's very very early idea but we're working on some you know just sort of playground examples of how this could work. So I'll turn it back over to Mike for some closing thoughts. Yeah. All right. So one of the

things in taking on this problem space one of what is building a new island. We don't want to build something that like kind of makes sense for this case but is so far separate from all of the rest of Kubernetes that it could get ignored. We want to make sure this is like seamlessly integrated parts of Kubernetes and gateway API that people are using today and are

really embracing. So how do we do that? One of the things that we're looking at is taking existing functionality from service that works that is fine but making it more atomic and composable and less monolithic. So breaking service apart and some of the parts that we want to reuse making it possible to reuse Antonio's work on extracting IP address like the multi-cluster or multi-cider or something or

other cap is one of the first parts of that. We also want to solve high priority problems quickly. So this is one of the things where a lot of the AI efforts and buzz in the in the community is helping fund things that we have wanted to do for years but have never really been enough of priority. So we want we need to bring some of these

solutions to users quickly and gateway API is going to be the way to do Changes in the core Kubernetes while they're going to be high impact it also can take substantially longer. So the gateway API experimental channel is going to give us a venue for getting these things and these concepts into users hands quicker. We also do still want to though start on some of the longer

term decomposition by working in Kubernetes core too. So kicking off that work and getting it started even though we know it's kind of a longer time horizon. And then putting these pieces together we want to use these components to create patterns for addressing multiple intersecting problems and not just one-off solutions. So what does that look like? I just mentioned IP address came out of but a lot

of the work for figuring out some of this new stuff like this backend resource is actually coming from AI use cases. It's not just multi-cluster use cases. There's work happening on an egress gateway proposal that basically came to a very similar conclusion about wanting a backend resource and an endpoint and we're looking at how we can join these efforts together to use some of these resources in

a way that is more functional across more use cases to solve multiple problems at once really because that's how we know these things are going to stick and land and then they're not just going to be a one-off thing that six months from now when the AI needs change gets discarded. We want to make sure that we're building things that will have a lasting impact that are

solving both the new problems as well as the problems that we have known already exist. So the egress gateway has a proposal and prototype being driven by Morgan, Keith and Shane in the working group AI gateway and the backend resource proposal is it's draft right now but Keith is driving this in gateway API. So working group AI gateway is where we've had a lot of discussions of

these AI use cases that are helping to accelerate progress on some of these changes that we've wanted to make for a while. But it doesn't own any code. So that's where we have conversations but ultimately we take those proposals and we find the appropriate venue to put them into a project where they actually can become code. So gateway API is one of the immediate targets for that.

Gateway API meetings are where we'll be discussing some of these proposals that are coming from working group AI gateway. And then also finding intersection with known ingress and egress traffic management needs. And then sig network meetings are where we're ultimately going to need to go to talk about upstreaming the some of these components particularly at the small scoped things that serve other purposes as well like endpoint

selector. That's one of the pieces of this that we think will go upstream into Kubernetes core because it serves other use cases and adds kind of that like flexibility and decomposition that we know that we we want as well. So would love for you to come join us in these working groups and let's build this together. >> [applause]