SIG-Multicluster Intro and Deep Dive - Stephen Kitt, Jeremy Olmsted-Thompson & Laura Lorenz

This talk provides an introduction and deep dive into the Special Interest Group (SIG) Multicluster, led by co-chairs Jeremy Olstead Thompson from Google and Stephen Kit from Red Hat. They discuss the challenges of managing multiple Kubernetes clusters, including the need for application isolation, compliance, and performance optimization. The session covers various APIs being developed within the SIG, such as the Cluster Profile API, Work API, and Placement Decision API, which aim to facilitate the effective management of multicluster applications. The speakers also highlight the importance of understanding user pain points and evolving the tools available for multicluster management. They call for community involvement and feedback to better address the needs of users.

All right. Hey everyone. Uh, thanks for joining us at the SIG Multicluster intro and deep dive. Uh, I'm Jeremy Olstead Thompson, uh, one of the co-chairs for SIG Multicluster. Um, and I work on GK at Google. >> And I'm Stephen Kit, the other co-chair, and I work on Open Shift networking at Red Hat. And Laura, Lawrence, she couldn't be with us today, unfortunately. >> Um, but she

did a ton of work on the deck. Uh, >> yeah. >> So, thanks Laura. Um so today we're going to talk a brief overview about what this SIG is about, what we do uh for anyone who's newcomer. Uh we'll go through uh the projects that we're working on right now and uh how you can uh be a part of the SIG. So kick it off. Uh what

are we about? So of course um we're about multicluster, right? um solving the common challenges that come with uh managing multiple clusters um and the applications that you deploy across those clusters uh and this is you know I'm sure everyone here is interested in this because uh multicluster is everywhere but there's a lot of different reasons uh why you adopt multicluster right you could have footprints in

multiple clouds you could be running across multiple regions uh it could be for isolation between you know various classes of applications could be compliance could be for release management, rolling rollout control, uh having canary clusters. Um there's tons of reasons. Could be for performance, getting clusters close to your customers. Um there are lots of reasons for multicluster and all of these bring different requirements to the SIG.

Um multicluster applications are also increasingly interesting. So this is applications that actually spread across clusters and and leverage different capabilities in different clusters. They can be very powerful, but they are very very hard. Uh and one of the main reasons for that is that Kubernetes was built with this idea. We've been saying this for you know a few years now. The this idea that uh uh the

cluster is the universe. Um you know for most of its life there was not really any concept of outside the cluster. Everything you care about consuming within a Kubernetes cluster was in that cluster. So that meant you know for no nothing else uh you couldn't address the cluster. The cluster didn't really have a a self-reference right. So we've been doing a lot to address that and Stephen's

going to talk about that in detail. Um but uh this kind of has been our focus and so our approach is uh first of all recognizing the diversity of the ecosystem. Like I said there's a lot of different reasons why you might want uh multicluster applications. Um we also see people running all over the place on prem on edge across multiple cloud providers. um you know there's

it's really hard to have one-sizefits-all solutions. Uh so we want to recognize those environmental constraints. Um we also want to recognize the existing ecosystem of tools. Uh when we design APIs as a group, you know, this isn't the first time these problems have been solved. A lot of people have their own solutions. There are projects out there that have built solutions. So we want to build APIs

that work for everybody that aren't, you know, dictating new ways of doing things or or conflicting with what uh projects have already built. Um and we want to kind of lift the existing standards um and uh have consistency with with existing APIs. You know, shifting to multicluster ideally is not starting from scratch and uh and rearchitecting your system. Um along with that we try really hard to

avoid solving uh optional problems which is I know for all of us engineers can be really painful. There's that like really cool problem, there's that whatif problem um can be really fun to solve but it can also be a really huge distraction that leads to things not shipping for way longer than you'd like. So we try to focus on kind of the most critical problems and we'll

talk about some of the ways that we are figuring out what those are uh later uh today. Um and composable building blocks as well. Again recognizing that everybody's got different needs. Um the solutions should be able to be pluggable with your existing solutions uh with each other um and not you know try to dictate an entire stack at once. Uh so with that uh Stephen you want

to talk about uh what we built? >> Yep. Exactly. Thanks. So in the SIG we have a number of projects and I'll go through them all. Um but the to start off with there's this idea that we take a number of clusters and we're going to manage them all together. So we talk about a cluster set uh and that's some that would uh group all three clusters

there and uh there's this central cluster that's yellow. Uh that's something like a management cluster. So our APIs try not to be too prescript prescriptive. Uh you don't necessarily have to have uh an explicit management cluster as such. In some cases, the cluster set will be a group of clusters, but it's the underlying infrastructure that manages everything. Uh, in other cases, the the management cluster is one

of the workload clusters. Um, but anyway, it's helpful to understand how the APIs all fit together. And so, the first one is cluster profile API. Uh, and so this is one of the APIs that addresses what Jeremy was saying about the clusters not knowing about each other. Because if you only have a single cluster, then it's the universe. It doesn't have to know about itself. Uh cluster

profile helps fix that. And it's a profile um uh an API that allows the management cluster to store information about all the other clusters. So things like uh their names, how to access them, credentials to access them, uh whether they have GPUs or not, that sort of thing. And we'll see how that's useful later on. The next API is the is similar to cluster profile but inside

each cluster and this is the API that tells each individual cluster about itself. So that's how the cluster will find out what its name is uh whether it's part of a cluster set, what that cluster set is and so on. Um then we have the work API and that's where you actually start using this to build things. uh and so it it lives in the management cluster

and that allows you to create resources in the management cluster that get pushed out to the member clusters. Uh then we have the placement decision API and that's complementaryary to some extent to the work API but that's an API that allows you to store decisions about where things should go and I'll go into detail about that later on. Then we have the MCS API. Um, and that

again is a building block to build things on top of all this all these clusters. And that allows you to take a service in this cluster here and make it available there. And so workloads running in this cluster can transparently access the service over here without you having to create any gateways or anything like that in between the clusters. Uh, and last but not least, we have

something that's not an API, but a library multicluster runtime. Uh, and that's the multicluster equivalent of controller runtime. So, you can use that to build controllers that will watch and react to objects not in a not just in a single cluster, but in all the clusters in your cluster set. And I said I would talk a bit more about placement decision API. Um, so this is something

that was added to the SIG uh last year. I think the ke was merged. Uh, and so like I said, it allows you to should go. And how does it work? It doesn't address all of the problem, but it address part addresses part part of it. And so the the workflow in general is that you'd something on the left here an initiator creates both uh the workload

that you want to put somewhere on your clusters a placement. And this is something that's vendor specific for the time being. We don't have a shared uh API for it yet, but that's where you would say well I have this workload. So for example, a demon set or a job um and I have these constraints that I want to satisfy. So for example, it could be that

it's a job that needs GPUs or it's a job that needs a node with this amount of memory. and then entirely separately. So these two pieces no longer need to know much about each other. Uh you can have a you'd have a placement controller at the top there uh that looks at the placement and the con the requirements that are defined here and chooses one or more

clusters where the workload should go and writes that information into the placement decision and separately an orchestrator that could be a controller or could even both of these could be uh humans theoretically uh or agents as well. Um, and this will read from the placement decision and the workload and go and actually set things up on each individual cluster. And so this is what a placement looks

like uh in YAML terms. The important parts are this placement key here. So that's how you tie uh a placement request to a and a workload to a placement decision. Um and here the results the decisions that were actually made uh so in this case um the controllers decided that the workload that we're talking about here should be placed on cluster one and cluster two inside fleet

one and in both cases it's because they have GPUs available and in a little more detail we've got a couple of use cases uh so the first one is we have some GPUaware uh AI training uh The initiator is a machine learning platform, a pipeline for a specific training job. And we want to place some PyTorch training job that needs eight GPUs. So what does that look

like? Uh we have the pipeline or the platform on the left here. Uh it's created this job. So in practice, it wouldn't actually look like that because you wouldn't have an explicit job on the management cluster. you'd have something like a manifest work using work API or some other representation of the job and a label the placement key that's going to allow all of this to be

matched later on. Uh and it creates the placement request. So this is the thing that's not defined yet. Um but in actual implementations there are implementation specific objects where it's going to write the information that this workloads needs GPUs eight GPUs. Then a separate component theuler is going to read this placement request um and read information about the available clusters through using cluster profile API for example

and it figures out where the workload can go and writes that in a placement decision with the same placement key so that it can be matched here and then the syncer so again this can be a controller or a human act reading this man and doing it manually or some something else. But it doesn't have to know anything about the constraints. It doesn't have to know um

be able to interpret the requirements. All it needs to do is be able to read this placement decision and the job template that was defined match them using the placement key and then it goes and uh sets the job up on the target GPU. So the key point there like I said the scheduleuler understands GPU requirements but the consumer doesn't need to know anything about that. It

just is able to read this and place and the second example is a progressive rollout. So in the first use case it was really a sort of oneshot decision and uh acting on that but placement decisions can be longer lived and can change. So we're sort of joining the process here somewhere along well somewhere in the middle of the process. The idea is that we've got a

workload that we want to u add to more and more clusters as as we do AB testing. Uh and so currently we've got it on the 10% cluster the 25% cluster uh based on this placement decision that says place it on the 10% cluster and 25% cluster. So imagine we've been through the same process as I showed in the previous use case and we've ended up here

but the placement decision lives on and there's a rollout controller that watches how AB testing is progressing and if things go well it's going to want to uh push the the workload out to more and more uh clusters and so it ends up writing this updating the placement decision saying that it should now go to the 75 uh% cluster And now the sinker which still doesn't know

anything about the uh constraints requirements and so on sees that the placement decision has been updated. And so it does this. This is a typical reconciliation scenario. So it sees 10% cluster it's already there. 25% cluster it's already there. 75% cluster goes and sets the drop up uh or the deployment. And here this all works because the placement key stays the same. the placement decision object is

still the same. It just gets updated and the sinker acts on that. And that's how placement uh decision can be useful. And as you see, it's a combination of APIs as well. So this is something that we try to do make sure that everything uh is complimentary and I'll hand it back to Jeremy for multicluster. >> Yeah. So uh thanks Stephen. Um, so how have we been

figuring out what to work on? Uh, well, you know, it used to be just kind of whatever folks who show up at the SIG talk about, but we started last year, um, uh, and big thanks to our community member Priya for driving this. Um, we've started, uh, user experience research to basically understand like what are the major pain points and, uh, uh, for those of you who

are at CubeCon North America last year, we did a kind of readout. I'll do a short recap of that and then talk about kind of where we've taken it since. Um so uh some of the key insights from the surveys that we've sent out uh over the last year um are that the kind of the the main points are first of all documentation gaps and and kind

of everything. I think it's hard uh and and please let us know if this resonates with you if you're if you're an end user um or if you're a vendor if this resonates with your experience with your customers but just knowing where to start and how to put the pieces together is really hard. Discovery is is a big challenge. Um the next one I think observability is

everybody's pain and in multicluster this is uh uh super common right like if especially if you have uh footprints across multiple platforms um you know a platform may have a great solution but it can be hard to get consistency ac across platforms and there are solutions that exist but and there are products that exist and you you can buy them but what are they right how do

you how do you figure out what you should be using also you know it's 2026 everybody's interested in AI um AIdriven automation for multicluster is really cool. Uh this sort of actually builds on uh one and two I think. Um AI uses documentation to figure out how to do things and uses observability tooling to see if it did a good job. Um so if those things are

weak uh you know it could be hard to figure out how to implement uh you know AIdriven automation uh in your multicuster solutions. And then I think the operational complexity again is kind of building on on these as well. Um, it can be hard to know what pain you're going to hit in a multicluster solution from the start. Um, and then lastly, and sort of on its

own here, is if you want to go build multicluster solutions, if you want to go uh build your own controllers and tools, how do you start? Um, you know, uh, Stephen mentioned that we have a uh, like a controller runtime equivalent. Um, discovering that, understanding that that exists. You know, hopefully everybody learned something new today, but uh, you know, not everybody is here. and how do we,

you know, do a better job of getting that out to the community. If you're interested in the detailed research, please check out uh uh this link. We'll post the slides, don't worry. Um and you know, we kind of did a readout at uh CubeCon North America. Uh but in this next phase, we've kind of turned this into a first persona and you know, uh journey, the primary

goals that we're going after and trying to address. Um the persona that we came up with, um we named Alex. Uh Alex is a hybrid engineer. Um we're specifically focusing on um on Alex who deploys and manages um and observes clusters across uh multiple environments. So onrem edge uh cloud A cloud B. Um in a large company uh Alex might be a platform engineer SR in a

small company Alex might be just an application engineer who's wearing multiple hats. Um uh but I think the key here is you know there's lots of multicluster same uh same cloud platform uh use cases too. We know that. But we're kind of focusing on the that multi cloud footprint um because that's where a lot of these challenges kind of get exacerbated. Um Alex's main problems here are

first of all the lack of purpose-built tooling. We've been trying to address that. There are uh tools out there. Um but if you don't know what the tools are or if there's not a tool that solves your specific need, there's just a lot of of toil. um a lot of day-to-day burden um in trying to figure out how to do things on cloud A and cloud B.

It can be hard to hire people who have these skill sets and and know what they're doing or even find vendors um who can really do a good job of supporting uh across multiple clouds. Um the differences between cloud platforms can be you know really challenging as well when we talk about uh uh observability like you know I work at Google you know we have solutions all

of the cloud providers have solutions um but they're all you know potentially a little bit different uh how do you get that unified view um and then when you're dealing with like arbback and permissions and integrations with other parts of the cloud or other parts of your platform um again everybody has good solutions for their local platform um defining things in exactly the same way when you

know people use different names for things or or there's slightly different capabilities uh can be a So what we find is that uh Alex will usually start with a lot of confidence, right? I know Kubernetes. I know what I'm doing. I can probably put these pieces together. I've been doing this a long time. Um, and so you jump in, you think, you know, you you've got the

tools, you've got uh you've got the experience, uh, you have documentation for Kubernetes, which is pretty good, and you can kind of figure out what you're doing. Then you find yourself spending a long time kind of reinventing the wheel in a few places and trying to figure out what those standards are. And so it's like this gradually discovered uh sort of latebinding toil um where the differences

kind of rear their head and you want to actually understand how how you can get a better upfront view of that consistency. And so that's been a big driver for you know this API consistency when we when we focus on those key parts um like Stephen was mentioning with placement decision if we can kind of simplify this like this final stage of plug in your CI/CD pipeline

um there are things that we can do that uh you know can hopefully take some of this away and provide some of those standards. Um and when you find those standards things do get a lot easier is the feedback uh is the feedback that we're seeing. So, uh, first call to action of the day. Um, help us out. So, we did this, we did this first round

via, uh, surveys, but we're really interested in talking to people as well. Um, so if you are an end user, we'd love to hear from you if this resonates. If this doesn't resonate, we need other personas. We need to kind of figure out what's unique to you. That'd be great. If you're a vendor, show off your community involvement and convince your customers to help us out as

well. Um, we would love that. uh you know short interviews uh you know everything will be anonymized but you can help us basically figure out what the next things that this uh SIG will be working on. I think that that would be really helpful. Um and you know we want to make sure what we're doing is is good for the community, good for Kubernetes users everywhere. Um

and then the second uh call to action is uh generally we'd love you to be involved. Um so see some familiar faces here. um you know, thank you. Uh but, uh it's really easy to um be part of the sik. You can come come to our meetings and just tell us what you're working on. Uh tell us what you're building. Uh what isn't working for you, what

your gaps are. Um we have an open agenda in our weekly meetings. Uh anyone can sign up for anything. Um you can just show off a cool toy you built. Um or come help us brainstorm uh some of the problems. um tell us what what your problems are, what your customers are. Um or yeah, if if we're missing something that's unique to you and again, we know

that everybody has unique needs. Um a lot of the times when you come to the community, you'll realize that maybe other people have these same unique needs, but sometimes they won't. And so we still want to help. Uh so, you know, come tell us what you're working on or, you know, join in, help us with our test and conformance uh suites or just help us make our

website a little bit better. We'll share that link in a second. Um, as I mentioned before, uh, the UXR, uh, you can, you know, provide feedback, but you could also help us, uh, talk to people. Um, and, uh, try things out. I think, uh, more than anything, trying out the things that we talked about today and giving us that feedback is is really, really helpful. So, uh,

we have some links here. Um, we've got our homepage where you can find some information on the projects we've been working on. Um, and again, we will post the slides. Uh so uh don't worry if if you don't uh capture it all here. Um we've got a Slack channel uh that's open and a forum. If you join our our mailing list here, you'll get automatic invites to

our uh meetings which are uh every week Tuesdays at 12:30 Eastern, 9:30 Pacific, 4:30 uh p.m. UTC um uh for our America AMIA slot. And then we also have an APAC AMIA slot uh bi-weekly Thursdays uh at 7 a.m. UTC. So, uh, we would love to see you there and get your feedback and, uh, try our stuff and see if it works for Thank you. All right,

we've got time for questions. There's a mic there if people are if well, if you have questions, uh, or there are microphones here that we can pass around maybe. Uh the QR code here points to the schedule entry and you'll find the slides there later on because I haven't uploaded them yet. And there's a feedback form. >> Hello. >> So not necessarily a question but uh mostly

just a uh a little bit of insight. So, I've been hopping on the call for last year. Uh, which, uh, you might think at first like, oh no, what am I going to say? How am I going to help? Um, that's not necessarily always the point. Like half the time I don't say anything. Um, but I just listen in because it means that you also get this

um this option where you might think you have problem A, but it turns out you actually have problem B, but you never heard problem B described, so you didn't know you have it. So even just joining in, reading the docs, reading the caps, um that also will help you. And then maybe 10 meetings in you're like aha the light bulb goes off and you have something where

you can share something that might have a much bigger impact than you would have first expected. So I would say join in. >> Thank you. Yes. >> You said the placement request isn't yet implemented. Does it mean you already have an idea for an abstraction or is it like fully open at the moment? we're actually debating if we do need one but >> yeah. Yeah. So yeah,

one of the ways we come up with APIs is that people bring them to us. So there are implementations that have something similar and work API for example I think came out of that uh placement decision as well. Um and so uh well the people who worked on placement decision I think they would like to have placement as well uh in the SIG but uh we haven't

seen a proposal yet um and that is one of the things that we look at is it actually useful to have something or do we not need it is placement decision enough to come up with generic uh solutions or at least you know some uh vendor agnosticity let's say Um, >> okay. >> Well, and also, so I actually when it first showed up, I asked the same

question. I was I was like, you know, we have a decision. Shouldn't we have a request? And, uh, the thinking is kind of, well, the decision is really like instructions for that syncer component, right? It's for your CI/CD pipeline, figuring out where to put things. Um maybe maybe requests, maybe what you're placing and how you decide is always going to be proprietary or there's lots of other

frameworks out there that can that have some knowledge and can make these decisions and maybe that's something that can remain unique. But then you want a common pipeline to be able to actually deploy stuff. And so um the decision uh you know is basically instructions for where to deploy >> um and not even how but just where. And uh you know then you have free complete freedom

for the how and the all of that logic pipeline can be whatever you want uh or your own thing. Um but yeah if if we have a good idea we definitely can do that. Um uh but yeah right now we're you know we're waiting on that proposal. >> Yeah there might be some other things too that we need to figure out first. So for example in cluster

profile uh cluster profile is a very open spec. It just says how to store data. It doesn't say what data really gets stored apart from well there are a couple of uh defined keys. Uh and but one of the things that we have there is what we call a property ladder. So prop properties people can implementations can add their own properties um inside the cluster profile object.

And if it turns out that they're generally useful, uh we haven't actually ratified this yet, but there should be a process to to make them um official and then ultimately required. And I think we need to spend a bit more time on that on the properties that we see in cluster profile that are generally useful to see if we can actually come up with something on the

placement side that would use those properties to make useful scheduling decisions. And maybe it will turn out that we can't ever all agree on a set of properties that's useful for schedulers. And so in that case, placement might not be particularly useful, at least not a common one. >> Let's see. Thanks for the answer. >> Uh hello. I have some question about uh the design of work

object. Uh is that okay right now? >> Uh yeah. uh I'm using command in my company and as a employment you know commander also has a work object and uh I found the command the over the work object will amplify the right operation to the hub cluster because the changes in the member cluster will be read into the member will be read into the hub cluster too

so it's sometimes it cause a very serious performance issue so I I wonder what you guys think about it. Um I think anything that tries to sync too much like we've generally been discouraging bulk state syncing uh back into the hub cluster because basically you end up in a world where your multicluster solution has uh single cluster scaling limitations and so um like I think with with

work a lot of the thinking has been kind of broadcasting outwards um and minimizing what gets represented in the hub cluster to like relatively static status like if you're trying to do a rollup of like let's take the most pathological example we can come up with like a service with a deployment that's roll if you want all pod statuses represented in that hub cluster you're probably basically

just running one gigantic cluster uh in ter from a scaling perspective so if you can avoid rolling up things that change super frequently or that have a bunch of like percluster state and really keep the hub cluster as more of an abstract representation um things should scale much better. >> Maybe uh later we can discuss it in a GitHub I guess. >> Yeah, >> that'd be great.

But that is a that's a valid general concern as well because for example on cluster profile if you think about it we're trying to represent properties about the cluster and ultimately if they're supposed to be useful for scheduling then you'll want things like uh how much free memory is there or how how many free nodes are there how many available GPUs are there and that's something that

changes all the time and so that sets a limit to how much actually useful information we can get back to the management cluster because if we try to be too um up to the minute and have with lots of information about all the clusters, it's just going to kill the overall performance of the whole thing. Um in one of the talks uh yesterday, uh they were talking

about potential changes to the MCS API based on changes to service and gateway API. Uh I was wondering if uh you guys could share your thoughts on that or how how that's tracking. >> Well, generally we've been trying to just say MCS should match service. So service changes MCS should grow with it. Um cuz you know we we definitely don't want it to get behind but we

also just don't want it to be different right you know how to do a service when you go to multiluster it shouldn't be starting >> and then I guess similar thing uh I haven't really followed it for a little bit but last time uh MCO was the thing has that been renamed or that's changed or uh >> oh multicluster operator that was yeah the Google project >>

yeah it's it's on hold right now I think is the um but if you're really interested interested. Um, you know, that would be the kind of thing. Come come to the SIG and tell us. I'm sorry, we just got a flag. Uh, but we can talk after. If you're interested in progressing these things, come Yeah, come talk to us.