Multi-cluster Orchestration System: Karmad... Hongcai Ren, Tessa Pham, Michas Szacillo & Zongqing Li

This talk discusses the Karmada project, which facilitates the management and orchestration of multiple Kubernetes clusters. The speaker introduces recent features implemented over the past year, including stateful application cluster failover, which allows users to preserve the state of applications during transitions between clusters. Furthermore, federated quota enforcement is presented as a way to manage resource limits at the control plane level, ensuring adequate capacity for workloads amid cluster failures. Other notable features include workload anti-affinity, enabling users to group applications logically, and priority-based scheduling to ensure critical workloads are prioritized during scheduling. The talk also covers real-world use cases from organizations like Bloomberg and Trip.com, which leverage Karmada for efficient resource management and enhanced system resilience in their multi-cluster architectures.

Hello. Hello everyone. Welcome to our session. Uh I'm Hungai from Huawei. I'm one of the maintainer of Kamala project. I'm glad I'm glad to be here with my partners. >> Hello. Uh my name is Mihashilo. Um I work on the streaming platform team at Bloomberg and we're a user of Carmada. >> Hi, I'm Tessa Fam. I'm a senior software engineer at Bloomberg. I'm working on the AI

training and notebooks platform. Hi, I'm Juni from trip.com, a online travel agency based in China. Thank you. >> Okay, here is our agenda today. First, I will give a brief uh introduction to Kamada project and then my partner will dive into our uh new features over last year and some use case. So, at the last I will cover the community status. Okay, before we get started, I

want to know how many of you have have heard of a commada before. Oh, awesome. Thank you. Thank you. Uh, so basically Kamada is a project that help you uh managing multiple communicated cluster and it uh can help you to schedule your workloads uh among different clusters. And the amazing thing is that if you are going to migrate your application to commada, you usually don't need to

change anything to your application. So that's because commada speaks the uh kubernetes kubernetes api and uh another amazing thing is commada can support every kind of uh workloads uh no matter the kubernetes deployment uh jobs and uh any uh workloads like spark flink and pytor uh so uh also your customerized uh resources Okay, I will hand over to my partner for the next part. >> Great. Thank

you. I'll use the laptop. That's okay. Hello everyone. Um, so Karmada is a rapidly evolving project. Uh, we release quarterly. Um, and over the last year we've created a lot of new features um that have been created with the community. Um, we can't talk about all of them today, but there are a few that we're going to be talking about in more detail. So first with stateful

application cluster failover um Carmata already provided a way for users to monitor cluster health and trigger cluster failovers in response to cluster outages. However, at that moment uh cluster failover is still a very disruptive process and it was something that was primarily used for stateless applications. Um, recently we added support for stateful applications and you can actually conserve state when transitioning applications from one cluster to the

next. So starting from version 1.15, the cluster failover feature has added support for this. Um, and it allows you to define the types of state that you may want to conserve for applications during failover. So let's take for instance one cluster decides to crash um and Carmata will reschedule your applications to another cluster. Um in this case let's assume that they're all stateful. Well you can use

the state preservation feature um which just extends the existing API defined in the propagation policy. Um and it's comprised of two fields. The first is a JSON path which is just an expression that targets a specific piece of state that you want to conserve and then the alias label name. Um, so this will just be whatever you want to call your label. Um, and then it'll be

up to you to consume that state once the application is failed over to the next cluster. Now let's say that you have cluster failover enabled in your infrastructure. How do you actually ensure that the fallback cluster has sufficient capacity? Um, so if all clusters are saturated and one cluster goes down, even if you have cluster failover enabled, those workloads won't have space to get failed over and

rescheduled to the next cluster, and it's going to make failover useless. And as a result, you really need to capacity plan smartly. Um, and this is where federated quota enforcement comes in. Uh, so this was added as part of karmmatada v1.14. Um, and it allows us to set resource limits directly on the karmmata control plane level. Um, so you can apply your resources directly to your namespace.

You have a federated resource quota synced to that namespace and then before the Carmata scheduler actually decides to schedule your application, it'll double check that the resource that it's scheduling isn't going to go over your limits that you currently have set in your federated resource quota. So just to give an example, let's assume a user is managing two jobs each using 10 CPU and you have a

federated resource quota that has 20 CPU as the limit. So both those applications can be scheduled no problem. Uh one gets scheduled on cluster uh member one and then one gets scheduled on member two. Um both consuming 10 CPU in the future. Let's say the cluster goes down. Well then we have sufficient capacity on the second cluster for your other application to get rehomed. Um, if we

didn't have this type of federated resource quota, a user could just blindly be applying applications until they saturate all the member clusters and at that point failover won't work. So, moving on to other features, um, some of our users deploy applications that are very sensitive to downtime. Uh, think on the order of seconds and in that case, you may want to schedule multiple applications to different data

centers for high availability. Um this was the main motivation for workload anti-affffinity um which lets users define logical groups that they can associate separate workloads with. So in this example we use a group by label key called pipeline ID. Uh we schedule two separate workloads and we group them based on um the actual value. So in this case indexing job and that way karmmatada can logically group

both of those separate workloads and decide to schedule them to separate clusters. The inverse is also supported. Um let's say you want to collate colllocate a group of applications on a single cluster. You can set the same exact type of label key. In this case, we have some sort of training jobs um that we'd like to collate on one member cluster. Um in that case, Carmata will

automatically detect that they're related and should be scheduled to the same place. And then additionally, uh we've added support for multiple component workload scheduling. Um, I'm assuming most of us here aren't scheduling just Kubernetes native resources. A lot of the time we're using something like a CRD. Um, and then the CRD operator takes care of translating that CRD into Kubernetes native resources. So when it comes to

Karma scheduler, um, it needs to understand the amount of replicas that that CRD is going to have and the replica requirement for each resource. Um now with CRDs that have multiple components this becomes a problem because since karma is assuming that the replo requirement is going to be the same you can actually lead to inaccurate resource quota usage. Um so you can um estimate incorrectly. So let's

say we have a flink deployment here that has two components one being a job manager and one being a task manager. They can have vastly different resource requirements and as a result we can either underestimate or overestimate the total resource usage which can lead to inaccurate scheduling. So now that that's supported um if you apply a CRD um that karmmata can interpret um once it gets applied

the karmmata controller will extract that information into the resource binding um determine the component requirements and then be able to actually schedule that requirement based on the total usage and find a member cluster um that is suitable. And next I'm going to hand off to Tessa who's going to talk a little bit about priority based scheduling and her use >> Okay. >> Right. Thank you, Mihash. So,

last but not least, we have rolled out a new feature in the past year as well. When you have multiple workloads all competing to get scheduled across federated clusters, you need a way to tell Cromata that one workload is more important than than another. And that is what priority based scheduling does. It ensures that critical workloads will get scheduled first. The workflow is pretty simple. You first

will create a standard Kubernetes priority class with a numeric priority value and then in Chromata's propagation policy, you will add a schedule priority sub field that will reference that priority class by name. Now this gets handed off to karma scheduler which will skew the resource bindings based on priority from high to low and if there are ties they will be broken by first in first out order.

If there is no priority class specified the priority will be defaulted to zero and that resource will sit behind any resources that were prioritized explicitly. So in this example on the right we have a deployment called critical engineext um in the default namespace and we're telling kermata that this resource takes high priority and to propagate it to member one and member two clusters. This diagram will demonstrate

the queueing behavior more concretely. So let's say in the priority queue right now we have a couple resources and a user comes in and submits a new resource that resolved to priority of 500. Where will that go? With priority scheduling, it will sit behind the job or the resource with priority a thousand and it will be scheduled before the one with priority 20. And this will guarantee

that critical workloads are not starved behind a flood of low priority submissions. Next, we'll move on to talking about use cases of Chromata in the real world. Starting with Bloomberg. We have been an active adopter of Cromata and we have integrated it in multiple projects in the past. In the past year, we've extended it to more to two more projects. The first one is model cache. We

came up with a CRD uh so that we can cache a model on every node to save startup time. I'm not going into I'm not going to go into the implementation um but if you're curious you can check out the link uh down below where my colleagues have explained it. But we basically build out a model cache API that is a thin layer that sits above Kromada.

So how it works is when we receive a request it will route it to the karma API server which will then propagate the resource to all nodes and clusters and with model cache we want to cache it on every node and cluster. Um on the left hand side you will see this is the response we get back from the model cache API which has the info of

the model but the most important thing is the status. It will indicate what nodes the model fill to cache on and the status of the ready nodes whether it is being downloaded there was an error or it's downloading and we get all this information from kmata API server. Why do we pick for model cache? Simply because the multicluster federation is is exactly what we look for to

cache models at scale. Carmata can natively fan out resources to all clusters which is what we need for model cache. So we don't have to write any custom script to do so per cluster. You also have flexible control over where to place a model cache if you choose to by using propagation policy which can target the caches by cluster profile, GPU availability region or any other labels

that you want to match. And finally on the road map um a future uh item that we can do is we can also feed the inference traffic metrics into the scheduling decisions so that karmmata can prioritize caching the model that has the highest demands and some more features that we also get out of the box with crata is one global status. We only need to query Kromata

API server to for all the statuses of resources on member clusters. Um and how do how do we do that? We specify the logic to aggregate status um in uh yeah in the resource binding. So if you have multiple clusters where a model is cached on, how do we decide that that model is successfully cached? Um you can specify you can customize customize this logic and then

ask kmata what is the status. Carmata can also autodetect and reconcile when there is a failure uh in any cluster and it will reschedu the model cache to other healthy clusters. And finally, if there is a change to to the propagation policy, you only need to make that change in the policy and Chromato will reschedu the model caches accordingly without you having to touch the member clusters

at all. At Bloomberg, we also offer Jupyter notebooks that is packaged in a job. Um, we offer that to our internal users. And the problem is how do we schedule a Jupyter notebook on any cluster that has available resources. So here you see we have a notebook that requires v100 GPUs. How do we schedule that? Um again we come to crata and crata can smartly pick a

cluster that has v 100 um GPUs and also available number of GPUs to schedule this uh jupyter notebook job to. Um again kmata can intelligently place this notebook given the resource requirements. um we don't have to write custom scripts and again we can flexibly uh place this notebook where we want it based on GPU requirements or based on um region if you want to schedule the Jupyter

notebook on the same cluster with any other resources that that it needs. Um and again features out of the box with Carmata is global status. So with Jupyter notebook you don't really need to know what cluster it gets scheduled to. You just ask her what is the status and it'll it'll give you that. Um aggregated status. You can customize what it means for a notebook to be

live and and running. Um Carmada can reschedu the notebook to a healthy cluster if a cluster fails. Um however, you'll lose the job uh that is already in the notebook, but that is not Carmata's fault. And finally, if when it comes to resource contention, um you can use priority scheduling to schedule any notebooks that needs to be urgently um urgently uh used. Um we don't have that

uh we don't have that need at Bloomberg right now, but that is definitely something uh we can do in the future. Um now I will hand off to Zongqing to talk about how Trip.com is using Kromada. >> Thank you, F. Hello everyone. I'd like to share how we use commander in our multicluster architecture to improve the delivery efficiency and system resilience. We use command to quickly distribute

cluster add-ons uh and build a resilience first high availability architecture for stronger for isolation and enable smooth last scale uh crosscluster application migration. In the past when we brought out a new cluster multiple team has to manually manually deploy networking monitoring logging and many other Kubernetes add-ons and other supporting components. This were also ordering dependencies between these uh components. So initializations so initialization often to several states.

Now with commada we can distribute these foundational components in a unified way. We can only need to make one configuration change in the commander propagation policy and all the components can be automatically roll out and made ready. Today this capacity already cover more than 40 clusters. It has reduced the time needed to bring a new cluster online from several days uh to less than four hours. This

has significantly uh improved our delivery efficiency and help us respond much faster to the uh business growth. The second improvement is resilience as a architecture level because a large portion of crypto traffic is a transaction critical and because traffic demands a high elastic we need not only HPA based scaling but also a very high level of availability. So we done the design. So we designed a texture

like Uh we spread status uh application uh applications across two or more member clusters and each clusters runs its own independent HPA component. When a member cluster serving live traffic fails, traffic is automatically shift to other clusters based on the health checks. Will HPA is triggered to uh skill out and complete the selfhealing process. Even if the feder federation control plan is a temporary unavailable, it only

affects application updates and the replic rebalancing. It does not affect HPA based scheduling for running services. In fact, our SLA allows the federation layer to be unavailable for up to one hour. This architecture gives us uh give us core services stronger high availability and disaster recovery capabilities. The third improvement is a smooth crosscluster migration as our business has grown rapidly in recent years. Some of of some

of our clusters once exceed 4,500 nodes. So we often need to split split uh the clust the clusters and migrate applications to the new clusters. In the past we did not we did not really have a an admin admin administrator administrator side migration capability Uh application migration had to be triggered by service owner through new releases. uh this was both time consuming and very costly. Now we

only need to adjust the weight in the command propagation policy and the workload will automatically scale out gradually in the target cluster will scale in in the source cluster enable smooth migration. We have al uh with this mechanism we have already completed large scale migration across more than 10 clusters involving a total of 300,000 post in a stable way. Here are some numbers. Automated uh baseline ro

reduce add-on deployment time from several days to less than four hours. Uh today this compatibility uh covers more than 40 clusters and the valve command control plan manage around five uh 25 clusters. Our smooth our capabilities has already migrated more than 300,000 post and and our distributed HP based resilience and architecture ensures that fault can be isolated to single cluster. Today's capabilities today's capabilities are already running

in production uh across multi multiple regions. Looking ahead we will continue to improve cross crosscluster scheduling and explore more possibilities for large scale multicluster system. Thank you. Next to the hungai. >> Okay. Thank you. >> Okay. Uh uh I will talk something about the uh commander community. Uh commander was open sourced at 2021 and become a sandbox projects uh that year and then move the levels to

incorporating uh in 2023 and now we are in progress of the graduation. Uh we have a uh a lot of uh uh contributors and uh also public adopters. Uh also some uh cloud providers are also using Kamada to uh build a cloud service. uh um and all we always uh uh listen to our end users and we also build a two dedicated team for uh scability. Uh

by the way the scability team is lead by zuning and uh we also uh have another team for the dashboard. Okay, here is our public adopters. Uh all of these are using Kamada in production and uh uh we we have set up a commada adopter group team and to help them uh we always welcome them welcome all of the users join the group. So you can uh

connect to other end users and share your experience and uh looking for support or something. Okay. uh that's our map this year and uh for the last the past for the past two years we are almost focusing on uh AI workloads and uh we have finished the priority based scheduling last year and the next we are going to cover the preparing part and also we want to

have a ber multiclass queue uh also for AI and also uh a lot of our guys are using kamada uh between their uh uh self built data center and the cloud. So we also need to another feature for the uh for people to select the clusters between self data center and the class and then the public cloud and uh also uh priv resource awaren uh I always

are looking forward more feedback for the project. so we can add more to to the projects. Okay, I think that's uh that's all. So I think we still have some time. So if you have any questions, thank you. Thank you guys. >> Uh I have a question. Thank you for the talk and uh does the failover work you said set the quotes and stuff like that. Does

it work with the CL cluster autoscalers for example that you don't need to overprovision the the failover cluster so this workloads would be rescheduled and cluster would auto scale as well. >> Uh if I'm understanding the question correctly you're saying how does federated resource quarter work if there's a cluster that's autoscaling. >> Yeah. Yeah. So if the target cluster if the target cluster yeah is set up

to auto scale yeah it will work as well. >> Uh yeah I think that should work as well. Um I mean ultimately federated resource quota um is just meant to be on the chromatada control plane level. Um >> it doesn't check available resources actually in the cluster at the moment. >> Well checking available resources happening is happening during the scheduling cycle. So like the first portion of

each scheduling cycle, you take a look at each cluster that matches your cluster affinities and then you're taking a look at the amount of available capacity on each cluster. So if your resource that you're trying to schedule to that cluster is going to go over the available capacity on that cluster, it won't get like that cluster will get scheduled out um or filtered out of like the

scheduling candidates. >> That's the question. So if if it doesn't have at the moment, but it would be autoscaled if you would provision it there. >> Oh, I get you now. Yeah. Um I don't know if that's currently supported then because the only thing that we see is the snapshot of that cluster at this given moment. >> Okay. So you need to overprovision kind of. Yeah. >>

Okay. Clear. >> Hi thanks for presentation. Uh I was curious about federated resource quarter it's called right. Is it per namespace? >> uh yeah per namespace. >> Oh okay. So Carmada awarees about each cluster quarters right. Is it CPU, memory and or every every u resource like max and so on? >> Uh every >> all the specs right >> and another part of the question is about

the kmada performance. Um did you faced any issues with it when you are scaling with the amount of clusters as a you know kmada has some kind of control plane components and so on. uh and what kind of issues you faced if any thanks >> I don't know you have opinion on that >> yeah for the performancing uh we have run uh uh performance test uh at

2022 and uh command can support uh about uh one 100 clusters >> and uh so uh we are going to have another another round of the performance as we have made a out a lot of improvements for the the past year. So, uh I think we are going to working on that this year. Thanks,