One Controller to Rule Them All - Taming Multiple Orchestrator... Srikar Paruchuru & Egor Grishechko

In this talk, Sri and Eager from Uber's compute team discuss their experiences in managing multiple scale orchestrators for stateless compute workloads. They detail the architecture of their platform, referred to as UP, which interacts with Kubernetes clusters across various regions and cloud providers. The presentation explores how they developed a new controller solution designed for failover scenarios, addressing the challenges of service scaling while ensuring that critical services remain operational. The speakers examine the importance of a robust status management system and the design considerations necessary for maintaining system integrity during high-stakes situations. Throughout the session, they emphasize the lessons learned and the significant cost savings achieved for Uber.

Evening folks. Uh I'm Sri and this is my colleague Eager. Uh we are from uh Uber's compute team here in Amsterdam. Stateless compute to be specific because well stateful is a different beast entirely altogether. Um today we're going to talk about uh how we tamed multiple scale orchestrators. Uh we have millions of workloads running and what if multiple orchestrators get to claim authority over the scale of

an individual workload. How do you reconcile that? How we reconcile that? how we settled on a solution, the things we broke, fixed, learned, all the while saving tens of millions of dollars for Uber. So, we're going to take you on that journey with us. It's been Yeah. more than a year on this journey. >> All right. Before I do that, let me uh walk you through uh

a brief overview of how the stateless compute platform is set up. Up at the top, we have the platform called UP, which is essentially the face of compute. That's what service owners at stateless service owners at Uber interact with every day to scale deploy their services to scale up scale down their services and what have you. Underneath up we have we have a lot lots of Kubernetes

clusters from different zones spanning multiple regions. We ingest hardware from cloud GCP Oracle onrem and we manage this I mean that's where our team exists. We manage these Kubernetes clusters at scale for Uber. As I mentioned, up is essentially like sort of the face of the compute. Uh but that's an oversimplification really. Uh so if you look at I mean a picture is worth thousand words. So

if you look at this is the UI of up. This is what our platform service owners see every day. You can see how featurerich it is. You can deploy a custom build of your choice. You can uh you can mention the kind of uh failure domain resiliency. I mean zone is a failure domain for us. So you can say oh I want my service to be alive

in like three zones at least or two regions. You can specify that on up. and up serves as a federation layer. It's going to decide which zones to place the workload of the service in. Uh and if you can see from the UI, you see that there are two running instances of the service in something known as DC22 which is the zone and then DC22 stateless 01

which is the cluster. So that's the kind of well that's the kind of experience like service owners at Uber have. That's a reimagining of uh what I've said so far. So you got these three interfaces. At the top we have up the federation layer and then Kubernetes which is essentially what as comput team we manage and then Linux at the end of the day up interacts with

Kubernetes via CD which I'll get into a little bit later. We call deployment, a play on deployment and then Kubernetes interacts with Linux via pod and multiple container runtime components and the way back up is essentially you understand uh how much resources on the node are left. You aggregate that at the Kubernetes level which is then sent up to up which uh understands how much square capacity

is present in a cluster and that lets it federate uh makes smarter decisions in zone placement. All right. So this is where I suppose the Kubernetes part uh we zoom into the Kubernetes part which is essentially what our team does. Uh we have an in-house CRD known as Uber deployment. So when a service owner creates a service on OP the intent I mean there's a lot of

intent in there. Uh they want a specific image a specific replicas a bunch of business requirements SLA thresholds all of that is encapsulated into this CRD which is UB deployment which Kubernetes recognizes. And once this UB deployment is created on the cluster uh we have a dedicated operator for it or controller which we call it which we call as the UB deployment controller. What it does is

it it looks at all this all the spec the current state of the deployment. It computes the gold state which is essentially a bunch of pods at the end of the day. All you want is really like your service to be running. It creates it creates the pods. It monitors the health of the pod and it reports a status object back which up watches over. up sort

of relies on the status object and uh it has an internal state machine. When the status says success, it goes ahead and does the next thing. I also have to mention up also doubles down as a continuous deployment platform for Uber. The internal state machine sort of uh uh explains uh how up works. It it looks at a current deployment. If it's successful, it's it's going to

deploy the next build. I'll get back to this later. Uh this is sort of like a check ops gun. Uh of course, there's there's sort of a question here. Why why CR? Why why not just use a native deploy native kit deployment? As I mentioned, we have a bunch of niche use cases. For instance, we don't always use a native deployment. We use clonesits to support in

place upgrades. I suppose latest versions do have it, but we are a couple of versions behind. And this was a decision that was taken nearly four or five years ago. Right. So, I just wanted to bring back your focus onto the running instances that we just basically consumed from the status field. uh the the justification for this focus on status field you'll know like 10 more slides

from now right so that's basically the primary scale orchestrator uh the service life cycle manager which is up uh this existed in over like for four or five years until now it was great but then a couple of years ago we had we had new use cases new orchestrators like the failover orchestrator which I would talk about and over to you >> clicker please thank you um

yep Here we go to the you know like higher level because if we are speaking about Uber we are speaking about scale and if we are speaking about scale we are speaking about millions thousands I don't know hundreds and etc. So I need to give you a kind of bird's view uh overview on how we operate as the company from the you know platform engineering infrastructure perspective.

So we have two data centers those work in the active active state. So when I mean active active it means like for simplicity let's say we are separating traffic 50%. So zas you know like requests go going to the different data centers and going through them but then you know it's like things they do happen and the emergencies they do happens and sometimes uh a lot of

things can go out and can go off and because of that we have this you know like tiny thing not really tiny but let's say tiny uh that is called fover and I suppose many companies have different foverers and in our case we have our last resort and our last resort is the regional failover when we are saying okay region number one cannot accept any requests because

like there is a problem with the zone A and because of that we are draining all the networking traffic and redirect it to the like second region but because of that we kind of get into interesting problem because let's imagine we have three services in A and when we are in the fover state we kind of need to allocate them in the second data center which I

suppose that's obvious and uh due to that due to historical reasons Uber was like keeping double the capacity of you know like of CPU cores of the memory of all the goodies hardware goodies that you can imagine in the second data center that just to be able to handle this region of fellow and then some like clever folks in the company said is like look we have

a lot of idle CPU cores it's not like it's not great to have them when you are not using resources and they just sitting around so let's figure out how we can re use it for like other other useful purposes I would say and the idea was pretty simple that when we are doing the fover and when we are moving workloads from the region A to region

B we are just you know like evicting low tiers of our clothes and we are putting high tiers of our clothes on their place. You know like putting is a wrong uh wrong word because literally we are scaling with like we have the same set of services in both of the like regions. We don't have like any specific workloads also running only in A or or only

in B. So like kind of it's all about scaling. But that's an interesting problem because uh Uber is huge. Uber is really huge. So it's like you know we had a very hot conversation with our colleagues from compute today at our booth. It's like how many clusters do we have? So we don't know the final answer because different people know the different numbers. uh but Vim will

be giving a presentation tomorrow with Sylvio. They know the exact number because they are doing this. But uh we do have more than 50 clusters running in production and I know for sure that we do one and a half pot launches per day. So I needed some metric because I just gathered them. So I don't know big numbers. I love them and we at least have 10

millions of containers. But this is introducing us to the following problem. So because we introducing this new clever fail over previously we had this UDC way and UDC way was fine it was like managing with replicas messing with replicas saying like okay I need six instances or eight instances or zero whatever but it changes because we are getting the second fover and we are getting this second

orchestrator and this second orchestrator is kind of kind of emerging thing that not happens on each day but we still need like to implement and to roll out into the production and to use it. And during that time, it's like we thought a lot about hybrid autoscalers, you know, it's like when our federation layer literally decides about borders for our workloads and then we were going to

utilize HPA and etc. I suppose talks are going still but in general we needed some kind of the solution that would work for us would would be future proof I would say because when you touching Uber's production you want this like you don't really want to touch it many times you you want to touch it once and possibly like it should be working and everyone should be

sleeping in the night thus it bring us to the second topic >> yeah I mean the most obvious solution right you guys I mean eager are uh presented I mean for failover I mean the needs for failover I mean the obvious choice seems to be the battlelisted controller that we already have UDC uh of course I mean there's a like the first reason the first legacy reason

is that it's pretty tied to a single orchestrator which is up but the deeper reason here is the concentration of power uh UDC is on the hot path for service life cycle like every every second of the day and failover is a niche scenario it happens maybe once every six months and what if there's a regression during a failover And a service owner could cannot scale up

scale back. I mean all the basic services are done all because you wanted to support failover that happens once in 6 months. If you forgive the lord of the rings reference u that's why the council of did not give the ring to barome you don't give more power to the already powerful. >> Yeah and you know with the lord of the rings continue your joke. It's like

when we were preparing for the talk, we thought like oh that's cool. Let's use Lord of the Rings and then I opened the schedule and saw that there are there were like seven talks with the name like to rule something one something to rule him something. I'm like okay we are going to use Star Wars next time or something like that or Warhammer 40k I don't know.

Uh so we decided that uh let's pretend we are young company and let's develop a new solution. literally let's develop a new controller that will solve the problem of scaling for us. So what I am showing for you right now it's extremely simple and you would ask me why I'm standing here and why we are doing a talk out of nothing and I would tell you that

this simple CRD that we are calling service scale it's like we were designing for six weeks of 10 weeks or something like that and it's like there are a lot of folks those were participating in the architectural discussion etc because we had quite a few problems in our minds because you know it's It's not a good thing to go into the database for the controller manager and

it's not a good good thing to introduce a lot of latency into your control plane when you are managing like all the Uber services and you know it's like thus we needed just implement something that should work from the start that is reliable have you know like historical kind of representation of the data that who wants to like literally how many instances which controller wants and another

like I would say very important thing uh yeah it it it it was like about debugging I would say because when you're developing a thing that you want to push to uber production that literally going to scale each workload in in the fleet you want if something goes off and something will go off always it's like that's a rule you want to understand what's going on thus

we you know like we developed this simple CRD with the very simple ideas we just have a map of ins because it's like you can believe me uh we thought about subscotic solutions I don't even remember them but in the end of the day we decided that materialization and when you understand what's going on in your cluster and you know like literally you can go to one

place and to say like okay Uber deployment wants 20 instances but fover wants zero right now and kind of it gets understandable how it behaves why we are not speaking about the status because it's quarter of our talk after that so stay tuned all the fun stories are there it's like we will be speaking about roll out testing and etc so in our new paradigm we got

into the situation when we got a thing that I'm calling it like sidecar CRD I don't know why maybe someone calling it as well I didn't read nothing I just decided that is cool name and uh that's a thing that lays on the side of the Uber deploy deployment and just receive signals as I said it's pretty simple but you know like devil is in the details

because uh for example Uber deployment wants to have like 10 replicas and then fover happens and says like no no no that's f do zero and we just do you know like kind of simple if inside because if is reliable and we can live with it and then you know like behave uh based on this if and like as I said the most important important part by

keeping uh like intention from the Uber deployment known we can do fail back pretty easy because in our case we see oh the previous state was like that we don't need to do external API calls we don't need to go to this database we don't do literally nothing so there are two fields that literally rules the like all the horizontal scale scaling in Uber right now but

it's only the part of the story I would say because That's that's the interesting part and that's why we are here. >> Yeah, like I said, we're going to talk a little bit about statuses in general. So, I want you to look at the the status field for service scale controller and see if any particular field stands out, seems unique, very non-cubernet. >> There you go. The

workflow status. I mean, if you go through the other fields, I mean, they're they're pretty observed generation, desired replicas, total replicas. That's pretty much how a deployment status looks like. That's that's very Kubernetes fundamental. But if you look at workflow status, it says workflow status success. Wow, that looks like a very definitive opinion on the state of the system. It seems pretty nice at the first glance.

Oh, I just know I don't have to look at conditions. I don't have to look if a deployment is progressing. I just know it's either success or or failure. Uh that's great. I mean, that's that's why we designed the field in the first place. But the issue with Kubernetes is there's no final state. Recoin sellers are always going to go on forever. Uh there's no strong consistency

guarantees. It's fundamentally an eventually consistent model and we have I mean our consumer is essentially up which uses an internal state machine. So when we say the status of certain thing is success it's going to take that and proceed to the next state. There's no coming back and the issue here is so our case controller they read objects from informers which are essentially caches. So they can

be stale they can be stale by a few seconds. So if you look at a stale state and if you say that something is successful when it's not and up takes that decision and proceeds to build the next image of the service we have a problem on our hands. So what what what I'm trying to say here if you can avoid it avoid it. Uh but we

couldn't I mean we we were already subscribed to this. So what we did instead is we built our own consistency model. We sort of built a read your own right god rail to to before we do status computation. The idea is pretty gnarly, but I'll I'll I'll try to walk you through an example um to to demonstrate how this works. Uh so you got you got the

I mean just to reiterate uh the Uber deployment controller writes to service scale and the deployment the service scale controller writes to deployment. So when Uber deployment controller writes to a child object which is downstream object which is service scale and deployment it propagates the generation of the Uber deployment as an annotation onto the objects. So when the Uber deployment controller when it's computing status it's going

to read the service scale object it's going to read the deployment object. If it sees this I mean if it sees the current generation in the annotation of the downstream object it knows that it read at least it's right. I want to make a distinction here. I'm not claiming we're reading the latest right. That's that's that's much more strong. Uh all I am saying is each operator

gets the guarantee that it reads its own latest right. So before Uber deployment controller computes a status, it at least knows the changes it made to the downstream objects are reflected in its reads before computing status. That's what we mean by read your right god rail. Like I said, it's a bit gnarly, but but if any of you are uh in similar situations, feel free to reach

out to us at the Uber booth. >> Please do. Maybe there are better ways. >> Yeah, we will convince you not to do >> and then of course. >> All right, the next problem. Maybe some of you saw this already. Uh in the previous architecture in the primary orchestrator model you had a you had I mean the path was pretty linear and simple. You had up ud

UDC goes to deployment or clones it pods great and in the new architecture you had two writers. You had the UDC and the SSC writing to the same deployment which in itself is not a huge deal. I mean Kubernetes supports multiple writers. We have optimistic concurrency control. We have server set apply patches all of which we used in our designs. But even then we realized near simultaneous

rights with the combination of still informers can cause inconsistencies. We saw this when we ran at scale. I mean in theory this should this should be pretty straightforward but it wasn't. We traced the I mean we traced down the root cause. So essentially in the deployment control you got a scale function which is responsible for well you know scaling up scaling down the the native deployment. uh

and it does uh and it when it tries to read objects from caches it can get stale objects and there's a particular shortcut a fast scale operation that it does where it's susceptible to a stale read and an inconsistency. The inconsistency manifests as an annotation spec drift on the replica set and it breaks proportional scaling. Essentially what I'm trying to say is your workload is stuck. You

need somebody to go in man go in and manually uh manually patch the replica set to get to get it back up to get it back up and get it to run. Uh so so we had a huge issue on our hands. I mean we had we'd rolled this out. We had this was in production. We had tier 0 tier one services that were just stuck. Uh

and we had no ways to detect or mitigate I mean mitigate this. All right. So what did we do? Uh I mean once we understood the nature of the issue which which manifests like I said as a drift between annotation and replica set spec. So we built observability uh to detect these things. We have an automation that inspects the state of all of our clusters. It it

is pretty extensible. So we just extended it to detect replica sets with these condition and emit alerts uh sorry metrics alerts on top of it. And and then we built a healing a healing automation that again uses the detection mechanism uh figures out where this happened. It goes ahead and not just the replica set uh it literally patches the replica sets annotation to catch up with the

spec. pretty gnarly stuff but but this is what I mean when something's rolled out on production this is this is what you do the holistic fix which I suppose everybody I mean that's what everybody thinks about is of course fixing the controller manager in which the deployment controller code resides patching this fast scale operation that I talked about we do have an upstream PR in progress uh

but like I said when you have something in production this is not what you want to do you mean you do not want to patch controller manager get to the latest controller manager and face god knows what new issues this is our current state And >> yeah, >> and yeah, with that, >> yeah, I'm going to hand it over to you. >> And I just want to

mention, we forgot to like to tell you that each time you're debugging, you know, like uh Kubernetes when you have many many many many many many everything and one particular failure usually when it's detectable, it's like it's too late to search for logs or it's too late or it's been stuck for two days or something like that and people notice and like oh my god, what to

do? And about this slide, you know, it's like when uh we were doing rehearsal today, we were like, "Oh, maybe we should potentially speak about rollout and integration things, but then you know, we we would spend another 40 minutes uh about speaking about infrastructure, how we do rollouts and etc. So instead I decided to speak about sync that usually saved us and yeah, it's not mistake. I

I thought about name for the slide and was like no, that's my favorite slide. I I love integration test. That's sarcasm." If you like Yeah. So in general uh a cool story when we were rolling out at we were rolling out the solution as Shikar mentioned we didn't have a real outage budget. So we needed to deploy it and it should magically work for each service in

Uber without like without failing. That was a cool task I would say. And the cool task was that we wrote like ton amount of the integration test. So literally integration tests and unit tests was the thing that helped us to not going into the production debris with some you know like some broken something. So literally it was the first piece of software in my life that each

line was covered at least three or four unit tests. I mean >> for example the read your right god that we talked about it's pretty stuff. I mean you you would never able to you would never be able to simulate race condition production. So we used the kind cluster setup and it uh it saved us I would say at least months of hours of dev effort. >>

So in our experience with computing and Uber is like if you're developing custom controllers do integration testing based on the lightweight Kubernetes clusters even you know it's like it's such a pain to support this sometimes because solution tend to drift and etc. If for example you are using kind not not really use it use 3S or something like that and then if you write a lot of

integration tests on the top of it you will save yourself a lot of sleepless nights just because unit tests uh they even with all the fakes that controller runtime providing and etc or even doing some kind of the local testing it's not always catching all these small quer you can have us integration tests despite being my favorite thing ever. I would say they saved us from a

lot of tough conditions and literally it was the thing that help us to deliver projects without downtime at least and users didn't notice it and you know it's like as the remark as the closing remark because we decided to close a bit early on time to answer some questions. uh this project at least showed to us we learned from it that simple solution as it seems on

the you know like on the nutshell because we are speaking about controller we are speaking about CRD with the map inside it takes time and a lot of effort to be developed it takes time to to make it readable and make it understandable and uh it takes time to debug even the simple solution if you are working like this many of something whether it ports whether it's

something and there are always surprises on the way because I was surprised how many caching layers kubernetes has so like four or five I believe so it's like each time we were finding a new cession layer that I didn't know about I was like cool how how we are solving this so thus it was you know like a very fascinating story how like we were trying to

heal and fix and provide better future for our production without destroying what we had we we have a quite fascinating story about about roll out of the thing but if you are curious just catch up us and we will tell you that I would say thank you that's all for