Scaling the Kubernetes Ecosystem: Uber's Approach to Cluster... Silvio Simunic & Vadim Plakhtinskii

In this talk, Vadim and Silvio from Uber discuss their approach to managing the lifecycle of Kubernetes clusters at scale. They highlight the challenges faced by their team due to the complexity of provisioning and maintaining over 100 production clusters across multiple regions. The speakers elaborate on the previous manual processes which required numerous steps, leading to delays and inefficiencies. They reveal how they leveraged automation and a declarative model, using an operator to handle cluster creation, configuration, and decommissioning more effectively, ultimately reducing the time from weeks to hours. The discussion also touches on safety measures for making changes during production and how they established a seamless integration with Uber's ecosystem to ensure resilience and reliability in deploying cluster configurations.

Hello everyone, my name is Vadim and today with my colleague Silvio we're going to present to you how we actually manage cluster lifecycle at Uber. Yesterday I was thinking why it's actually interesting for you, right? Because Uber have a really big scale and not many company run on this scale. But I think the idea is that we first of all of course want to share you like

what we're doing there and like kind of show you our infrastructure. And also we want to tell you that some problems which you can face can be super simple for you to solve. But if you go on Uber scale, it's actually really hard. And we go through the different ideas. we have a different problems there and we want to share what we figure out. Let's start and

understand what is cluster lifecycle is. So, imagine you want to run different workloads, stateless, stateful, right? You need Kubernetes cluster where you're going to run all all of them. And your cluster during its life go different stages. First, it start to be turned up, then it's active and during active phase you can change your cluster setup differently, right? Maybe you want to apply new controllers or you

want to change node type which you run there. And if your cluster is not really lucky, it can be decommissioned at the end. Uh let's I want to tell you about one story. Imagine we have developer. Let's give him name also Vadim. I like this name. And Vadim really want to run new batch workloads at Uber because he thinks it's really cool and it will give him

really good promo, so why not? And then what happened next, right? He needs create cluster for testing for example, only one cluster to just test his ideas and show the boss that oh looks actually it's really working really well. And many many years ago at Uber how it looked like, I think it's around one year ago actually. Uh Vadim will go to the 15 or maybe even

more steps. First, he need to get capacity for his cluster, right? Because it's like should be like out of nodes and stuff. Then he will deploy control plane there. He needs to take care how it's going to run, where it's going to run. Then he needs to apply configuration his cluster. Of course, somewhere it doesn't work well, and he needs to repeat it again and again. And

super painful, but at the end, after several weeks, he'll get his cluster actually where everything going to run. But Vadim is super unhappy because imagine to just to try some stuff, it's required you like two or three weeks of manual work. And it's actually really, really not good, I would say. let me tell you also a bit about our scale, right? Because I told you like Uber

scale, super big. So, we have around 100 plus clusters in production, which actually handle user workloads there. We run this cluster across 15 plus different zones, different regions. our clusters actually really easy to configurate. So, you can do different different shapes, different configuration for your host, different configuration for your controllers. So, it's like super dynamic. And of course, there is around more than 1,000 engineers who wants

to run and use these clusters. And of course, they want to change everything what they even possible. And another important stuff about Uber clusters, we actually use pretty big clusters. Our nodes count for one cluster from 5K to 7K, for example. So, it's like really, really And it's where our team actually came up because if you like small company, maybe you can just get new cluster in

Google Cloud or something. But at Uber, you can see we run it across on-prem data centers and also cloud, different cloud providers. And our team at Uber, we located between users who runs their operators and foundation. Foundation is team whom we go and say, "Oh, please us give some host because we are need to run our clothes." So, our team is compute and what we do in

Uber, we uh responsible for control plane, right? For all native uh controllers which we run native companies which you run there. How we deploy them, how we scale them. Also, we we responsible how we apply cluster configurations, right? Because you want to configurate again a lot of different parts and we responsible to maintain this. And at the end, we also responsible to run different controllers because our

clusters run stateless batch. We also now going to run stateful workloads there and we have a lot of custom things how to do it properly on our big uh number of scale. So, where we start uh before we do all our incredible incredible work. Uh our cluster provisioning require you to go through the 15 manual steps as a developer. And it doesn't sounds really cool, right? Also,

it's uh gives you a lot of uh weeks of work because it's really really hard. You don't even have single source of truth when you as a developer as a cluster owner can see what's goal state going to be for your cluster. And honestly, it's kind of sad and remember this Vadim who wants to create new batch cluster to just try We have a lot of Vadims

like this who is not happy, but our team tried to make their life much easier and try to uh make them a lot of happy. after we see this all these problems, we came up with ideas how we actually can solve all of this, right? First of all, what we really need to do, we need to create cluster configurations and make it more better for people, so

they can easily change some different settings for their own cluster, but we need to be really careful because our team is responsible to maintain this configuration and apply. So, we really need to be careful how much flexibility we provide to our users. Also, we need to create them automation cluster turn up process, right? So, you want to have the same experience in Google Cloud, just push one

button and your cluster is done in like several minutes, maybe even maybe hours. Then, we need to allow people change their cluster when it's actually active. So, imagine you have a cluster with 5K nodes, it's running workloads, it's serves traffic, but you still want to do some changes, right? You still want to change hosts which running there. You want to still change controllers which you run on

top. uh again, if your cluster not super lucky and it's supposed to be decommissioned, we also want to automate this process for people. So, they also push one button and cluster is disconnected from production and it's doesn't serve any workloads and we can turn capacity back. So, let's go with me to this journey and I will tell you how we do it step by step. As I

told you before, uh we need first provide people configurations for their cluster and how they can work with this. I think the main uh point here we need to define what cluster is at all, right? What dimension it exist and what actually important. So, we have a lot of different configuration for our clusters, but I think the main thing is what class what of type of workloads

you because it's actually mapped to our cluster type we call it. So, you can run status workload, you can run batch workload, and many many workloads there. Yeah, super. After this, uh you also our cluster have different topology. It's can be zone clusters, regional, global. So, it's uh actually another thing which you want to configure it when you create And another main thing for us, it's cluster

phase. As I told you, you go from the turn up phase to decommission phase. And through the steps, different cluster phase enable different uh different point of our stack. For example, if your cluster is not fully active, you don't want to get observability. You don't want to get alerts, right? If something work because it's still turn up process. So, we have this all dimension. And then thinking

how we can model our configuration for our clusters, right? Uh first things which we find out that we actually can split it in two parts. static parts, it's the it's uh cluster configuration which with your cluster through the all its life cycle. So, when you create new cluster, as a as a cluster owner, you want to provide name of the cluster, zone or region where it's going

to run, also type which is our cluster which you going to serve. And this change this configuration don't change at all during cluster life cycle, right? Because it's there with cluster for all all all time. And actually this type of type of configuration super simple for us because we don't really need to care how we going to change it in productions for you. This is one part.

And there is of course second part which is dynamic configuration which you want to change during cluster life cycle a lot. And we need to figure out how to merge them all together and be able to create this cluster configuration for more than 200 clusters which we actually run. Uh and what we figure out, right? Templates is really good. You can template a lot of stuff. And

why we don't use it? And I think also another thing which we figure out is you don't really want to configurate each cluster. You it's not not needed, right? Because, for example, you run status for clothes and usually configuration for your cluster for status for clothes is the same all the time. So, what if we tell people, "Okay, you can configurate your cluster type kind of template

what's going to run there, but you don't really want to configurate anything for cluster specifically." Of course, if you want, you can, but most of the time it's not really needed. And then we provide um layer. We use Starlark for this. Uh it's kind of Python-like language which kind of get your static config our dynamic config and merge them together. And at the end it provide you

for each of cluster which you run in production or like staging etc. It provide you all configuration which which needs to be applied or used. Another important stuff here is actually when we do it this all the stuff, we already have 100 cluster in production which is already running there, right? We cannot just say, "Oh, let's move everything to new configuration type and just it will work."

No, because of course it's not going to work. So, we carefully need to support all the way how you define cluster and how do you configurate them a new way. So, we also provide a layer which you do cast between old type configuration and new type configuration. But at the end, we make Vadim from another university a bit happy that now he actually configurate what he wants

to run and how he wants to run this stuff. Okay, so now when we solve the configuration problem, let's have a look into how we actually use that configuration tool also automate to the cluster provisioning at scale. the project we went with is uh the Kubernetes native, which means we we have a declarative model and that is going to let us to define in form of the

CRD. So, we're going to have the cluster object as our input to the automation and then we have an operator which is going to reconcile that state. This operator is then also going to sync with Git and it's going to talk to external APIs to create this cluster. So, basically this is how we made the shift from manual process to an automated one. We can now describe

the cluster and then the system is going to make it real. Here we can see an example of this cluster object. So, that's going to be an input to the automation and it's how we define the goal state. Uh In the spec we can see some of the things that the user can define. what should be the zone of this cluster? Should it be a stateless or

a batch cluster maybe? Should it be a staging or a production cluster and so on. And then in the status, this is what the operator is going to be updating. We have the conditions and it's going to let us know which steps of the cluster provisioning were done. At the end we can also have the phase that's going to tell us if this cluster is still in

the process of being created or is it even done. So, it's only going to be this thing that the users have defined and everything is going to be handled by our automation. Now let's take a deeper look in how our operator actually works. So, as we said we have a cluster object as an input and then we have our operator which we call the Kubernetes cluster orchestrator

and it's basically a bunch of different the controllers which work together to make this Main control that we have is the cluster controller and it also works at the workflow engine to execute all the steps needed to make this cluster. We are going to hear more about this in a bit. what happened? All right. Let's restart this again. Uh nobody want to show what we did. Should

we apply? I think we need some help here. Yeah, maybe it's going to be I'm going to tell you what we going to do. So, it just stopped working. I just connected again. Why it works? super. Thank you for everything. It is stress, but it's okay. Okay, so we're very Yeah, as I was saying we have the as the main controller and then we also have some

other controllers to be able to delegate some of the more complex steps. So, for example, to make uh ETCD for this cluster, we have a ETCD controller to also validate this to this cluster at the end. We also have validation controller. Here we can see a simplified view of uh all the steps involved in creating the cluster. So, at the top you can see some infra provisioning

steps. Then we have to get the host for our cluster. Then we have to set up the control plane. Set up also the Uber services, some configuration, and then at the end we do validation. So, some of these steps have to be done in parallel. Some of them also depend on each other. And also for some of them, we might want to check before or after the

step if it can be executed. So, the question here we had is how do make this happen in the reconciliation loop? Uh had to write all of this to not make it like a huge mess. So, for that, then we decided to model it as a graph and we came up here with our own framework, which let us do that. As a result, we now have like

an easy way to write these steps and also extend our automation to also support more of the workflows. yeah, so one of the most challenging parts that we had to do is integration with all the Uber internal systems that have to be here involved. So, for example, our automation has to talk to many different the deployment engines. We also have our own way of having managed the

capacity, how we generate the certificates. Also, we have to talk to different APIs to make the cluster. So, since all of these systems are external to the operator itself and can be not reliable this scale. Here we had to apply some patterns for the resilient integrations such as item and calls, rate limits, retries, cool downs, and so So, yeah, this was really important as the failures let's

say that this scale can be normal and we also needed to design for that. Okay. After some stress, we can go another fun part. It's after we know Vadim, right, what we can do. We can give him access how to configure his cluster to run his workloads. And also now we can build his cluster super easily. But, there is next big problem. What if he wants to

change something during cluster serving production workloads? For example, it's already not stated in current production. His his new budget workloads running there. It's running across 100 clusters or like 50 clusters. Still a lot. And he thinks like, "Oh, let's actually deploy new controller to all of these 50 clusters to I don't know somehow do not maintenance for this because it's required, right?" And there is problem. How

to do it actually safely? Because, as you remember, we now can generate for him 200 configurations and we can tell him only for now, "Yeah, it's you can use it, but you have to apply it manually, change your custom manually." As you can do it, but of course, it's not really working well because we get incidents and then Vadim has to receive with in out of zoom

with people from director level and explain him what actually happens there and why. And I don't think it's really good idea. you can think about this problem from different angles. You can use maybe some open source solution like forget ups right to apply config from your git on the Kubernetes cluster. But uh I want to tell you different perspective. So, as you know at Uber we think

a lot about how we deploy stuff safely, how we apply configuration in safe way and there is this too, right? Because class in production and we need to do it super safe. So, for now we have only one system, only one operator which create your Uh but it's not really good way because what if you want to change some code there, right? You just do global blast

and can break everything. So, idea is let's do it a bit more safety and let's use actually what Uber provide you to do it. Like we have really good system when you can deploy your services. You do it like in safe way. It's can go like first uh tier three, tier two and tier one uh environment. It's going to run into end test and check results and

then only after this it's going to go to So, we actually can apply the same logic to our orchestrator system and say, "Okay, we don't want only one operator run and operate but we want to shut down it and say, "Okay, we have one operator which will serve only uh dev clusters, another one which will handle staging and final one which can do production." And then your

configuration delivery problem actually can be simplify and be binary delivery sorry problem. So, let's just connect our binary for our orchestrator with configuration which needs to reapply to our clusters. In this case, after our developer wants to run their new controller or something, he can update list of controllers, just push it to the Git and then CICD for orchestrator system automatically will get this change and apply

to the cluster and do it in really safe way because it's first go to end-to-end environment and wait until end test to pass, see everything works well and then go to staging and then to production. So, at the end we actually don't involve any third-party services. We just use what we already have and make it safe deployment there. Okay, so now when we can make clusters and

we can also change them safely all the time, the cluster is eventually going to reach the end of its life cycle and we're going to have to decommission this let's take a look into how we do that and how we do it safely. So, basically for the whole decommissioning workload, we use the same model as for provisioning. We have the CRDs for the goal state and then

we also have the controllers that are going to execute the steps. Another thing that we have to do here is also clean up all the resources that we create as part of the cluster creation and for that we also use the finalizers. So, in this context, the finalizers are going to let us to block the deletion of all the objects until externally created resources of these objects

have been deleted first. So, only after that operator is going to be able to delete all the objects. And we also store history of everything that happened in the status field so that we can audit it later. And now we get to the most important part for decommissioning and that is safety. This is very important as uh mistakes here can also lead to the maybe outages, some

data loss, and even bring down the whole business. So, to solve this, we implemented safety on the multiple layers. On the first layer, a cluster that has to be first approved by someone else, and we check for that by admission policies. On other layer, which happens in the operator itself, we also have a quarantine period, and that lets us have some time to be able to change

our mind on if we want to stop with the cluster deletion. Another thing that we do in the operator is the concurrency limits, and this is also very important as it allows us to limit of how many clusters do we actually want to be able to delete at once. So, in our case, we want to be able to all the clusters at once because it cause like

big incidents, obviously. And on the third layer, we also do some rate limiting towards other systems at Uber that have to be here involved, and that lets us kind of limit some of the most disruptive actions, such as terminating hosts uh or also down scaling some of the critical services. So, yeah, basically all of these layers set together, it gives us a very high confidence that our

automation is going to be safe. Okay, uh now let's take a step back and look into what we achieved here and some of the key takeaways that we have. So, the impact of this work has been very significant for us. Uh we started with that that was hard to work with and is now in a more regulated place. It's also centralized in Git and we have a

single source of truth. We also started with a manual process of cluster provisioning and also decommissioning to a now fully automated one and it only takes hours instead of weeks. And all of these together also allowed us to have our cluster life cycle fully integrated with the rest of the Uber ecosystem. What are some of the key things that we learned with this? So, first integration was

the hardest part, right? We didn't just have to build like a new system that's going to be isolated. We connect many different systems into a single automation. Second keeping declarative configuration in Git turned out to be a very good decision. So, it also enabled us to do the GitOps and it gave us a single And last and third, we think it's important to start small. So, we

didn't automate everything at once. We first added the support for the cluster creation, then for the reconciliation, then for the decommissioning. Then for even the more cluster types and so on. So, as requirements evolve, we think it's important to also design And overall, this is how we moved from a manual cluster management to a fully Kubernetes native life cycle. That'll be all from us. Uh and thank

you for your time. Thank you. If you have any questions, yeah, we will be happy to answer them. Yes. There is microphone Uh uh Hi. So, do you also handle OS updates and Kubernetes version updates through this process? Uh yes, it's not through this process. It's a bit different process, but yeah, our team also responsible for all version upgrades in our infrastructure across all this cluster, but

it's go a bit in different subsystem which we use for this. Yes, but it's also automated. Okay. And so, you have a specific clusters where you can test. Yes. Yeah, so in general, we have like uh our fleet divide into dev, staging, and 20 production clusters. And if you want to allow everything to production, right? We through this like steps, we first go to end-to-end where we

just apply and run our end-to-end pipelines for all our uh workloads like batch, stateful, stateful. But it is really working well and also we don't have like problem with uh uh uh like scale etc. And then we go to staging and then production. So, it's like fully uh super safety way there. Okay. Thank you. Yeah, of course. Hi. My understanding is that you basically have written from

scratch what your operator actually does. What are your opinions on using uh let's say the cluster API to to implement the the the same thing that you did. Any uh experiences, pro cons, etc. Yeah, so the question is uh basically why didn't we use some existing open source solution, but do we instead build our own, right? Yeah. yeah, we basically looked into using uh open source stuff

like we looked into the cluster API. We also looked into the V the but since we have many different systems at involved in this cluster provisioning, it didn't really match like to have this solutions work, right? So, they have like their own CRDs and so on, and it didn't really Yeah, fit like in our requirements. Basically. Okay, thanks. Yeah, I think you need to think about Uber

that we have a lot of different custom things for almost for everything, right? Because when it was created, open source solution doesn't exist before. Uh but we now try to go a bit to more open source world and see what we can explore here. But for this system, cluster API was really for us big uh thing which we look a lot and see how it's actually do

a lot of stuff. So, we Yeah, try to get the same ideas here and integrate with all Hi. Hi. Uh I have very similar question. Uh have you tried to other tools open source like Kamaji from Flant Analytics I know another Ukrainian guy, he wrote Cube DC uh which does exactly the same which you do. And also we try to do the similar solution. We used to

project Sveltos to deliver those addons. So, have you tried any public solutions? Uh what kind of challenges did you have? uh are you going to open source your solution? Yeah, I think we we we look, of course, on different system which already do the same work, right? Problem for us that you have to do it like for 200 cluster in production. And it's really what's make it

super super difficult because we for us the main thing is uh do it really in safe way because if break something, it would be like disaster, right? You people can't order food and stuff. So, for us it was more easy to create our own stuff than adopt already existing things because a lot of them not support what we want. Also, we have again a lot of custom

stuff, for example, for release pipeline, right? It's a completely different system which already here and it's already provides us all things which we need. So, for example, why we don't use flux CD stuff like this to apply configuration? Because we say, "Oh, yes, we can use it, but then we have third-party service which actually goes completely different way that our binary delivery there." So, what we just

can simplify it and make it okay. We will use what we have already and just make it's it's already safely. So, why not? So, yeah, I I think for us the the main thing do all stuff in safe way. And about open source, uh, we don't have any plans about this. We more curious to maybe work with open source project to make it better. So, this is

what I believe we for sure can do. Thank you. Hi. You mentioned different ties tiers of environments, but does it mean that if I have 5,000 nodes in production, I also need three another clusters of different types previously to test my change? Uh, it's actually not really because, as we have different tiers, right, of our clusters, we also have benchmark clusters. And this cluster actually when you

can test your stuff on loads which you do in production. So, for example, you can simulate like 5,000 nodes here. You can simulate, like, Uber scale which actually hit your production cluster. And it's one of the release workflow which we do. So, when you want to release new Q version or, for example, new service which will, like, kind of core components, you require to go to production

and see how it's going to behave on actually on these loads. So, we don't need to support five like 200 and uh clusters with 5,000 nodes. So, at least one additional tier. Yeah, kind of yes. Yeah. Okay, thank you. Hey. Uh thank you for the talk. Uh I have two questions. One is um you mentioned the foundation team or something like that. Is that the team that

provides like the network base native network that you will need and subnets and all of that or is out of scope? Uh sorry, could you So, if you need to set up network etc., do you do with this project or is done by your team? >> yeah, it's already in our ecosystem. So, uh we use there is a working team to actually handle all networking communication here

and there is person from this team here. You can ask a question if you want. But yeah, it's a foundation layer which actually give us already networking which we can use. So, we don't think about it when we build our clusters there much. Okay. And the second question is about the configuration as code. I think you mentioned the probably the tool that you use or can you

explore a little bit more the that part, the the way you merge the static and the dynamic. So, we at Uber for our configuration layer, we use we use Starlark. Uh it's a kind of Terraform. I think it's like it's Python-like language where you can do whatever you want almost. And uh it's integrated with all our deployment system which we have there a lot. And for example,

also we use it to generate alerts and observability stack for you. So, in this layer, it's like kind of Python code which you write where you can program how you merge these two configurations together. For example, okay, I have five like 50 YAMLs with my cluster information like zone, node, like type of the cluster. And I have templates which says, "Okay, for stateful cluster, I need this

type of controllers to run there. I need to this and all flow schema configuration and this configuration you need to apply to the cluster. And then this star lock what it does it's like consume this input and just implement apply to this templates and give you different how do you say different >> combinations builds for different companies which it needs to use and then you can consume

it for example even for your service deployment so it's now oh if I run on titles cluster I need to do this or if I run for batch I need to do different way but you can consume the same source of truth to generate all of these configurations. Thank you. Yeah. Thank you for the presentation. Uh how long did it take you to build this system and

how many teams were involved? Yeah, I think we started like a year and a half ago uh and it took uh as we said we first started uh with uh you know automating the provisioning then we added also reconciliation of the active clusters then we also did uh the decommissioning and also the other types of the clusters so it's still like an ongoing uh thing as we

get like uh the new types of the cluster that we have to add and so on so the core part I would say of the automation and also configure I think it took us like half a year right? I mean pure shields done I believe like in several months but make it production ready I think it half a year and it was only out one team involved

in this process and it's around four engineers three like three of us who work on this I believe core. Okay. So it's not that much people. So that single team had to develop all the controllers by itself or did it receive any kind of requirements from the rest of the teams like from the network the security? No it's uh we don't have requirements because it's our team

own cluster life cycle and we actually need to solve this for our customers, right? So, we build our controllers and how it's going to look like. And then we just have to go to another team and say, "Okay, let's integrate with networking stack. Let's integrate how we get host and stuff like this." So, yeah, so it's was pushed from our side uh not from top. Right. Thank

you very much. Yeah, Hi, thank you. Thank you for the talk. So, I wanted to ask like where your controllers are hosted. Like I assume it's a different cluster where the control plane of those things are And how is it resilient? Uh so, as you say like yes, we run our controllers to build clusters, but of course we run them on separate uh kind of cluster where

it's actually operates. Uh it's not full Kubernetes cluster because this cluster doesn't have data layer at all. It's only control layer and ETCD. So, it's uh we have special team who provide you platform as a service. So, when you can create different resources and they provide you um K CAPI which you can use, right? To write your controller. So, we actually run on their on their system.

So, on their API server, controller manager, and ETCD. And it's uh What they do is that they globally distributed. So, they don't have zone regions. It's like global clusters. But they do tearing. So, they also split this global cluster in smaller parts. They have separate control plane and separate ETCD there. So, as we say, when we create our system, we say, "Okay, we will have operator which

will connect to tier zero, tier one, and tier three of their clusters where they actually which they operate." So, it's we kind of for free get this uh division of our system. And under the hood is it is still uh for resource provisioning is it still Terraform etc. because like to create cluster, you will have to provision nodes first, right? Sorry. Sorry. Uh under the hood like

when your operators or controllers are scaling up the nodes like provisioning nodes for the new cluster, is it under the hood Terraform or like you are creating the nodes some internal tool is there or like how is it? So, we have capacity team who actually kind of hide from you all all of this layer because we have on prem and we operate on cloud providers. So, for

our user actually they don't really care much where they going to run, right? Because if you run your several servers your service, you don't need want to think about this. So, these systems they kind of hide it from from even from us. We know where we operate on which cloud provider, but there's we they just say, "Oh, we need 2,000 nodes." And then they say, "Oh, where

is your financial approval for this?" And then we say, "Okay, these people have approved 2,000 nodes." And then they kind of provide it to us so we can use it. So, before creating the cluster users or the team had to take the approval first for the provisioning and the their capacity estimation. >> Yeah, so before you want your cluster done, you need to go to this portal

and say, "Okay, I actually think it's going to be 5,000 nodes." And somebody needs to say, "Yes, okay. It's okay to spend money on this." Got it. And for these controllers, are you guys using CubeBuilder, KRO or Yeah, it's it's custom. It's custom tool. Custom. Yeah, cool. Thank you. Thank you so much. Hello. I have a question about the orchestrator. So, you explained that cluster creation is

a a step of a multi a workflow, right? Where like you start with the CD creation and eventually the end-to-end test. Inside the controller itself like with the within the reconciliation loop, how do you reflect these stages and make sure that predictable. Is it like through the CRDs or in what ways? Maybe like some learnings from that part of the controller implementation. Yeah, so basically that was

one of the hardest part uh since like when you make the object on this cluster you still have to have a ETCD cluster somewhere else, right? So Yeah, so we basically use like uh at a periodic monitoring of like this external resource. So Yeah, like it's being checked like every 10 seconds or something like that it still exists and so on and then if it on the

external system anymore we also delete the object. In that case. And so I guess you reflect the state of truth to the CRD and use that as Yeah. >> for the next stages. Okay, thanks. Yeah, I think we need to go but you can still find us somewhere around to ask to ask a question. But yeah, thank you so much.