No Pain No Drain: Lessons From Node Drains at Scale - Ryan Hallisey & Natalie Bandel, NVIDIA

This talk features Ryan Haly and Natalie Vanell from Nvidia, who discuss their transition from a manual node drain process to a fully automated solution for managing lifecycle events in their GPU data centers. They highlight the challenges faced with traditional methods, such as inefficient workload management and the need for better coordination during maintenance. The speaker details their multi-controller architecture that utilizes a drain coordinator and a lifecycle orchestrator, which enhance operational efficiency and minimize GPU hour costs. Key concepts including workload awareness, self-tuning drain throttles, and an improved ranking system for node selection are explored, showcasing their commitment to creating a more Kubernetes-native approach to lifecycle management.

Okay. Hi everybody. I'm Ryan Haly and I'm joined by my colleague Natalie Vanell. We're from Nvidia and uh before we dive into the details of our talk, I want to share a little bit about uh what what we experience uh maintaining our infrastructure and our team has experienced. So in a single day we will do about a thousand node drains. a thousand no drains. Three years ago,

we would do almost zero. It was a manual process for us. And we would experience all these life cycle events every single day. Things like device failures, driver failures. These life cycle events would hurt our overall capacity in our fleet and they would hurt our sanity. And so we never really liked experiencing these things. And we wanted to come up with a solution, something that made our

lives better. And so that's what this talk's about. It's a talk about a journey. It's about a story um about all the work that went into going from a manual life cycle management process, a manual train process to a fully automated seamless solution. And uh so we're going to go through that story. Okay, first uh I I love having a picture to kind of kick things off.

I kind of feel for this this person and the the crane operator. This is like I can identify with this this person. This is like what drain's like if you wanted to do drain manually. This is like how I would feel. There's a lot of crates falling everywhere. You kind of need to figure out the how to control the the chaos that that you're experiencing. Okay. So

um the scale what's the context for for all the things that we're doing. So at Nvidia we will uh we run thousands of GPUs uh in our data centers. Our workload is a game streaming workload. It's a latency sensitive workload. It is uh it's ephemeral. It has a an SLA to it meaning like it will terminate after a certain amount of time. has a maximum timeout to

it. Uh we do lots of planned maintenances. We do lots of unplanned maintenances, actually even more unplanned maintenances than we do planned maintenance. And uh drain has a cost to it. Has an operational cost. And that cost is GPU hours. And our goal is how do we minimize that? How do we minimize the cost of of doing drain? ically maximize the amount of GPU hours uh that

we are using uh we're giving to our customers. Okay. The problem so I talked about uh uh drain this so this is and and and sort of go back to the beginning of the story for us is that we started with cube cut drain. This is how it all began. That's probably how everyone's story begins when you are doing when you want to do drain is you

probably start here and um this is great. It's a perfect starting point. It's a great tool. Uh but there are some challenges when you start here and first of all just a context set about cube drain. Essentially this the tool is something that will start by coordinating a node. It will evict workloads and then it's done. So we thought that while that's really useful, there's more state

that's inherent to this process and this state is is much there's many states. There's much more to it than just you know these three steps. And so we figured that okay well what what else is there? And so like what comes to mind immediately is like well if you're doing drain why not think about where to do drain right we're trying to improve the operational cost we're

trying to increase the uptime of our of our GPUs let's think about where to drain does that improve our ability to keep GPUs available and it does it actually does and and so these kinds of things these kinds of things are actually like way to put them is that they're like they're states they're things that would impact our ability to to maintain the zone and so we

have come up with several of them. These are just kind of like high level um states that just um describe the process. So like could schedule right we can schedule drain we can we can do some sort of ranking picking nodes which node we should drain uh we do cord and monitor then we will you know do our our maintenance and then we can return. So it's

a whole process. There's a bunch of states and so quickly you can kind of see how the scope of drain just becomes such a larger concept a whole life cycle management. Um and that's and that's what we've learned. And so as you as you start from that cube cut drain we this is something that you will quickly learn and evolve to. Okay. So next let's talk about

the challenges. What are the difficulties that uh if you start from this point of doing cube cut drain what are the things that you're going to need if you want to build this full life cycle solution. Uh first is u uh velocity control right it's really important that we're mindful of our end user and the capacity that we're making available. So we need to be aware that

we're not taking away too much capacity. Right? We want drain and maintenance to be transparent to the end user. we don't want them to know about it. So we want to be very careful how we control these strains that we do. Um next workload awareness. How do we teach drain about our workloads and their patterns and their characteristics? Um so that's something you have to do. Um

drains get stuck silently. That's a that's another challenge. And to explain this one, we have to go to the next slide to number four, which is that when you do cube cuddle drain, that state lives inside the client, right? You run it on your laptop, you did a cube cut drain and it fails. Well, who knows about that? It's only you, right? So, we really want an

automated system so I so I don't have to go in Slack Natalie and say, "Hey, look, the you know, the drain didn't work, right? It's not really scalable." We want something that is automated. uh and then coordinating across maintenance types. So this is a really complex uh idea. Um so think about um how when we have these these maintenance these unplanned maintenances that where you could have

a failure let's say like a device failure and this device failure um you don't really know what happened. So you you reboot the note. Well you reboot it and it doesn't fix the problem. So what do you do next? Well maybe you rebuild the node. So you can kind of see like there's a bunch of different remediation steps that that you can do and maybe that doesn't

fix it. Maybe you need to RMA the physical device and and the problem is like this this constant struggle between what is actually the state of the device in the physical world versus what can be observed. And so you kind of have to go through this process to actually do discovery about you know what is actually the the real state of of of something in the physical

world. And so you experience this a lot. And so coordinating these things is very difficult in in doing discovery. Okay. And then finally a topology aware um topology awareness during drain. I the term I like to use for this is topology aware maintenance. Um kind of a scary term to think about but I would say if you're doing if you're running a GPU cloud um like us

and and and say you have you know NV switches and you need to upgrade the firmware on on the NV switch. Well, there are times where you need to also upgrade the firmware on the GPUs attached to that NV switch. So, you need to be aware of like what what change that you can make can affect other things. Um, so it's a very advanced concept. And so,

in this talk, we're going to uh explain how we solved one, two, three, and five and partially solved four. Um, and then six we have not solved yet. Okay. So, going beyond uh cube code drain. So, this is a summary slide. So to kind of reiterate this, uh, first you need, um, something that is a declarative way to report maintenance. Um, so a way to share state

is the way I'd put that. Um, a scheduler, right? This is a kind of a cool concept. Like I said, being really intelligent about your drains. You can really schedule them. You can make really good decisions about when to drain, where to drain. Um, and and so building a schedule is really powerful. Um, having observable state enforcement. So having this expectation that you will drain, something will

drain eventually and if it doesn't then we need to move on. We need to do something else to fix the problem. Um and then automatic state drain and detection and recovery. So like I said, we can get to these we need to sort of define what a state of stuck drain is. We need to have an SLA and and then we when we discover this, we need

to fix it. And lastly, uh metrics and measuring. So being, you know, having the system be fully aware of all the different things that are going on, coordinating with our tenant, making sure everyone's aware of of what we're >> Okay. So now Ryan, explain to us the things that uh we have challenges with in cube train and what we want to have from a system that solves

these challenges. I will talk a little bit about how we actually solve that um in Nvidia for our product that is GeForce. Now that's the online gaming platform. And the idea for you is that you might see those concepts and think oh I can use this one in my system. I have partially built solution but I didn't think of that one. or you don't have any solution

at all and you can think and design based on our experience and things that we already built. So we have two controllers, two microservices we where one operates in a context of the entire cluster. It knows what's going on with um nodes with maintenances and drains and uh in the context of the cluster how many are scheduled how many are in drain right now and so on

and that controller basically is the drain coordinator. It decides what and when to drain. That's the the smartuler. The other controller is operating in the context of a single node operation. It knows the state of that node. It uh transitions the um between the states and the uh and uh the entire uh communication between those two controllers is happening using CRD. the maintenance CRD where we save

um the So let's talk about drainator. It decides uh what to drain and when. Uh it receives maintenance requests where the the system or uh the process that requests maintenances wants to send the request and forget about it. not to babysit that request, not to check on did the node um drain what is going on, how the maintenance is going. The idea here is uh to create

a provides you with the ability to launch and forget about that and move on. So the drain coordinator receives those and creates a CRDS for that specific target node. Then it picks which nodes to drain and when. It has a u ranking pipeline and adaptive split speed limiter. I will touch on that in the next slides. And it also monitors the drain process uh on the node.

It knows to watch the um specific workloads, the relevant workloads and to see when the node is ready and drained. The second controller is the life cycle orchestrator. It manages what happens during maintenance h for that specific node and it has three responsibilities. It manages the state transitions in the It enforces SLA and sends a notification to whoever is interested uh to understand what's going on with

that specific node in its life cycle. How those are uh coordinate the actions. So we have as I said before um the maintenance CRD but the real state uh lives on the node itself. uh it is uh uh we can find the annotation that are being used labels uh cordon status and so on. So the maintenance CRD mirrors that state that from labels from annotations and the

cordon status on the CRD. So all the other components are able to uh see that CRD and to have a very clear API how where to see all the status at one place. So the drain coordinator reads the node uh the node status writes it back to the CRD and the orchestrator watches that CRD and drives the transitions and the um the SLA timers and notifications. They

don't communicate directly. All the communications is with uh using that SRD. Let's talk a little bit about the maintenance life cycle. So with a simple node train cube cuddle node train we have uh not really we don't really have a life cycle but uh here we extended it to have uh more states that uh processes can query and understand where uh where we are in this process.

So we have a CRD that is created. It goes into scheduled status. It doesn't mean that uh any process started. It is just scheduled to be drained and to go into maintenance at some point. When the node is cordoned the maintenance the uh basically the maintenance process has started. It means that the node begins the drain process. When drains drain is done, we don't have any workloads

it means that the drain is complete and the actual maintenance can start. When the actual maintenance is done, we have a complete state and it means that we can bring this node back into cluster. It uh went through all the life cycle. In case of failure, there is an incomplete state and it allows uh uh the same controllers do some sort of remediation maybe to recue it

for another um run of maintenance and see if that uh helps. So now let's go a little bit deeper into how the drain coordinator picks the right nodes. Those are the small things that we have added with time that helped us improve uh the throughput of how fast we can do maintenance on the entire Um we have a um drain algorithm where the basic idea is smarter

workload where node selection. We have a composable ranking system where the controller knows how to rank any node based on all kind of things. For example, for uh if the node is already drained, we add a specific amount of points. If this maintenance is an urgent maintenance, it is unplanned and we need to quickly reboot this node for example, we add more points. If we have fewer

workload sessions, we add the also points. And the last one is diversity injection. For this one, I will uh talk about it in the next slide. But to summarize this uh slide is that here we have basically composible way to rank different workloads. And if we decide we need uh another function that will rank other work additional workloads in an additional way we can easily plug it

in. It's basically sort of a plug-in system for ranking. So what is that diversity uh injection? Without it, the system only would drain uh would do maintenance on already draining nodes because they would get the most points and that's the easiest ones. They already don't have any workloads. We will take them into maintenance real quick and after maintenance they go back. But what happens next? Next, we

still have all the other nodes that are non idle. They have workloads. And at this point, it means that we will be starting the drain process for all of them. So it looks productive in the first phase, but the second phase might take very long time. So the solution is to mix a small percentage of nodes where you have workloads and you need to drain them so

that when you go and do maintenance on the drain nodes those already started the drain process. They will get into drain state uh faster and you can uh pick them for the next iteration. Another thing that we have implemented is um node type awareness. Here uh we have uh for different node types different workload profiles. It means that uh our controller doesn't just look on all the

pods on the node and decides if we have zero pods it means that it is drained. We have specific functions with specific uh ways to define drain that are different per different node types. If this is a GPU compute, we are looking for very specific pods. If it is a storage, we check that the volume is amount and so on. If the uh node is control plate

node, we have a totally different set of expectations to define that the node is trained. So each type gets its own detection and uh this is uh how we uh how we make sure that these nodes are actually drained and we set the expectations what it means. is a self-tuning drain throttle. We don't have a a specific number. We don't say not max 10% of all nodes

at a time or not max 100 nodes at a time but instead we have a range and in that range the system could um adapt. If we have one successful maintenance, we increase our uh throttle, we open it up a little bit. For another one, we open it up a little bit more. This way, if everything goes smoothly, it's great. We can uh speed things up. For

failures, we want to tighten the throttle and have less maintenances. For each failure, we close the throttle a little bit so that if everything starts failing, we really fast we get into very slow process and it helps uh people uh check what is going on. It helps us uh understand where the problem is before we go into disaster mode where all the nodes are in failed states.

We have two different timeouts for stuck drains. First time out talks about when the drains has completed but no process started the maintenance. No one takes takes it uh and does the maintenance. This is the first time out. It means that someone sent the request for a maintenance but no one acted when the node is ready for that. The second timeout is when node is drained and

some process started the maintenance but it never finished. It is stuck in the maintenance and no one knows where what what's going on. We have a timeout for that. And in this both cases uh we can uh have a auto remediation. We can clean things up. We can uh make the node back um as uncordoned and bring it back into production if the node is functioning. So

a little bit more about the self- remediation. We have self-healing recovery mechanisms that allow us to try and get out of all kind of problematic situations automatically. So as I mentioned, we have the drain throttle that we tighten up on many uh we have a mechanism that uh uncordens the node back and brings it back uh into the cluster. We can automatically reue the maintenance and see

if it will go uh and be successful the next time. And with all that we have timestamps on the maintenance CRD and that allows us to understand what's going on and when. So the system recovers without human intervention and we have the audit of what's what has happened and when. And now a few points that are not that interesting, a little bit boring, but will save a

will save you from the disaster that always comes in the middle of the night. So cordon ownership we make our uncordon nodes only in case that they are the ones that cord on this node. They don't we don't want a situation where our controller uncordons and not that someone else has cordon for any other reason. We have admission web hooks uh as guards to avoid duplicate maintenance

different uh weird states that we are that are not intended to happen and so on. We reject maintenances on empty selectors. It means that no one even accidentally can request the maintenance on all the nodes for example in the cluster. And we have met metrics that help us uh with our uh with understanding our phase transitions, completion rates, um failure counts and So all these things that

we have built uh we wanted to share with you so you can get uh ideas for systems and from what we've seen and what we know many other companies and systems are building the same things uh a little bit different but pretty much with the same intention. So what we can do to have it as a more kubernetes native solution. Okay. So what are the takeaways? So

at a high level um separate what to from what happens during maintenance. Right? So, kind of the microser model, focus on exactly what we're doing and and exactly what needs to happen. Uh, second, um, make your drain throttle self-tuning. Super important, right? We don't want to cause an incident just because we're trying to do, you know, the an upgrade or something or just because we're trying to

remove one GPU, right? We want to be really careful about what we're doing. We want to be very precise. Um, use configuration scoring and not a single rule. So be expressive with like how you want to uh control your drain. So it's not just we can't just have a sing single bit that we flip. We want to have a policy. Um and then number four have uh

have a stuck drain recovery mechanism. These kinds of things like like longunning events that sort of have it's difficult to find when they end. We has to be a a stopping point. There certainly has to be an agreement that at some point well either this there's a problem on the node or there's you know there's something that's blocking us that should not be and you know that

could just be the application and we need to have some sort of agreement with the application owner that eventually you will benefit from what we're doing and you you probably want the upgrade or the CV fix or the whatever that we're doing. So we need to come to some sort of agreement. Um, and then finally, um, treat maintenance as a first class life cycle. Um, I feel

like this is something that always makes its way into like the product further down the line, right? This is why we call it day two. And, um, but it's super important. It's something that is important to plan for. It's something it's important to budget for and build around. Um it's something that when you get a lot of pages at night that you eventually start to say, "Wow,

we should really fix this." So it is really important and you should absolutely treat it um just as important um as as the other parts of your stack. Um and so one of the thing is uh about like our journey is that um is that we we use a lot of inspiration from the community and and sort of building um our projects and and I think also

there's other projects that use inspiration from other projects and this has kind of like gone on a lot um in the ecosystem and um which is great. I mean I think like that's you know what's great about open source is that you can learn about what other people are doing and and and build on top of that. And I think what's um what's awesome is that um

is really what this highlights is that this is this is a problem and um people are investing in it, but it's also a problem that um that we can probably solve together. Like there's probably some overlaps. Actually, I would say there's definitely some overlaps and how we're all solving these things. And so trying to find a way that we could um build something maybe into into the

community so that so we're not all architecting and building and maintaining all um all all this code. Maybe there's some things that we can you know do together. Um and so there is this uh uh the node life cycle working group um that I'm a lead on. Um and then a few others uh listed here Lucy from Uber and and Red Hat from Flet from Red Hat.

uh the three of us are leading this group um and uh this is one of our goals is we want to we want to look at uh life cycle management in Kubernetes and we're looking at tools methods and and projects ways that we can improve this um uh like we said this is an important space and I I would say like especially for me from the Nvidia

side you know I care about this very much I I can see like all the different things that go into life cycle managing a GPU cloud and the complexity that is there and and this is something that's going to be important to to invest in. And so that's something that I really want to to emphasize and and bring into the community. Um so there are two caps

that I would highlight to everybody. Um that I would encourage you to read. Um the first one um so one of the topics from this talk we talked about workload where drain this first kept uh is I would say covers that area. It's this concept that you can that draining is not just uh you know we're going to remove a workload that it's it's we want our

applications to coordinate you know to figure out how they should drain. It's is more complex than just a cube delete um of a pod. And so the second one um uh so as I should mention the eviction request API this one the ke was merged for 136 and the code we're we're hoping to get it to 137. Okay. Okay, the second cap uh it's called specialized life

cycle management. This is um trying to find a a a native way to bring life cycle management into Kubernetes into entry. Um this one is a cap that we're uh we're discussing and trying to work uh with the community on on designing. And so this one I what would really be great is for people who have experienced this to come share your use cases with us. um

you know we want to solve this problem but would really help us is you know people telling us like what are the experiences that they have and and and what they would like to see you know improved and and and that would help us try and build this in you know get this into tree try and find a certain ways that we can get uh some improvements

um and there's a lot of stakeholder stigs uh and what's important about this is that this life cycle management touches everything I mean literally everything. Um, I could come up with all sorts of different examples of like Damon set rollouts where like you get stuck because cublet is stuck and how does the Damon set controller figure out that the cublet is stuck, right? It's like something is

wrong there. Like things happen in the physical world that can't be reflected in software and the only way that they can be learned is discovering is through discovery like things going wrong essentially. Um, like how do we deal with those corner cases? So it affects everything and and so it's a big project. So that's where I would say like you know you know come help us out.

I it would be really great to see um you know your use cases and and you know share with us uh and we'd like to make you some improvements in the community. Um and then finally like the the Slack group for this working group is here u the mailing list uh we meet Mondays you know come join us and come talk to you come talk to us

we'd love to hear from you and thank you very much. We do have a microphone for questions. I think we might be over a little bit, but if there is one, we over. Okay. Okay. Well, we'll do questions afterwards. Thank you everybody.