Improving Pod Disruption and Node Lifecycle - Filip Křepinský, Lucy Sweet & Ryan Hallisey

This talk focuses on improving pod disruption and node lifecycle management within Kubernetes. The speaker, Lucy, a staff software engineer at Uber, along with Philip from Red Hat and Ryan from Nvidia, discusses their work in the Kubernetes node lifecycle working group. They aim to enhance processes related to node drainage and maintenance, shifting from reactive to more proactive methodologies. Key topics include the introduction of an eviction request API that allows for specialized lifecycle management of workloads, enabling applications to handle pod disruptions more effectively. The presentation also highlights issues related to graceful node shutdown, kubelet amnesia, and ongoing efforts to create standardized APIs for node lifecycle management.

Um, okay, everyone. Welcome to improving pod disruption and node lifecycle. So, my name is Lucy. I'm a staff software engineer at Uber. I'm joined on stage today by Philip from Red Hat and Ryan from Nvidia. We are the leads of the Kubernetes node lifecycle working group. So, our working group focuses on trying to improve the entire node lifecycle experience with Kubernetes. So, that's anything from node drains

and maintenance through through admitting what nodes kept taking nodes out of clusters. All of these things that right now today are pretty reactive and things we're trying to improve these process flows. So, we're trying to do that in a few ways. I talked a bit about in the morning talk if you were there. Who was at the morning talk? Put your hand up if you were at

the morning talk. Hey, nice to see you all. so we're trying to do that in a few ways such as video eviction request API and to close the node maintenance which Philip and Ryan will speak about later. Um, so we work quite closely with a few of the other SIGs around Kubernetes, particularly node. We're very closely tied in with them and also app scheduling auto scaling. Node

is such a tight integration point that we end up having stakeholder SIGs in nearly every part of the K8s project. So, today we're going to talk about three particular projects that we wanted to call out in a more technical level. We're going to talk about eviction request which is if you were at the morning talk, you know is a way to to signal the intent to evict

a workload before you actually evict it which allows you to much more complex logic before evicting the workload. We call that specialized lifecycle management. You may have seen it under the old kept name declarative node maintenance which I think gives a pretty good idea of what's going on there. And we're going to talk about graceful node shutdown and all of the kubelet amnesia bugs around the project.

So, to start, I think Philip you're going to talk about the eviction request project. Yeah. So, when we talk about eviction like we have to look at the pod or pod disruption or termination in general. So, as you know like there are many vectors on of like terminating a pod and it's very hard to like deal with deal with the termination in concise manner especially like if

you don't have a high availability application or you require special handling of your application. So, there are ways today how you can do that. You can just use your SIGTERM handler in your application to handle this, but this might not work in every scenario. So, the ways how initially we decided to do graceful eviction or termination in Kubernetes was by introducing eviction API which is basically imperative

path that you call and give an intent to evict a pod. And in cooperation with eviction, we have PDBs. And this works for a lot of applications, but there are edge cases. So, for example, if you have like a single replica application that you care about, it's a maybe a database or virtual machine. And you you you don't want to run this in high availability. Like PDB

doesn't give you a way of expressing your wishes of graceful termination. And there are ways to work around around this. Basically, you can create admission webhook or and set a PDB to zero. But so it's people are dealing with these scenarios already and but this is not in very observable and we would like to improve this situation. And I can mention other cases where this is useful.

So, for example, your application even if it's high available, it might prefer scaling out first before it scales down. So, you might want to first maybe ensure that you have enough resources to run on and you want to basically postpone the eviction until you are ready like in the in your next step. So, that's why where eviction request API comes in. We basically allow for We took

this imperative eviction and tried to implement this declaratively. So, this is an feature that do and kept merged in 136. We did not make a like the code freeze deadline, but we are pretty close and I'm pretty confident like we will see this feature come to life in 137. So, as I was mentioning, like you have a for example engine X pod, it's just an example. It

can be a database or anything and you would like to evict the So, you put the UID of the pod because that might change during the time this eviction is handled. And as we have seen like there are many vectors of eviction. So, multiple components can actually evict the same pod at the same time and might have different basically would expect different outcomes or even the same

outcome and there like there some of them might withdraw the eviction later. So, we need to track all the all the requesters that request this eviction. So, that's like the main field under the target. the next thing we would like to do as as we have seen there are many ways of basically migrating an application or disrupting scaling out. So, this is not something like we we

can prescribe, but we can allow other actors or we call them responders to basically add a hook to to our eviction request and allow them to run before we even trigger a pod So, we we hope like the in the first iteration of this API, we want this to be like the most graceful. So, the application can actually take care of like for example the migration which

in some cases might take hours or days. And this is not something that is like natively supported in Kubernetes. So, we would like to change So, as I was mentioning, you can actually add these your custom hooks or responders as we call them to the pod API. You write or your ID of the responder and then we have a controller we collaborate on the eviction itself in

the status. So, currently the way how to do this is we go like one by one and select each responder. And the responder has to fulfill a contract. Why? Because for observability reasons like the PDBs as they work today, if you just write a PDB with min unavailable zero, like you don't have a way of telling if this if the graceful eviction if it actually will terminate

or you cannot observe the process of the termination itself and this is good like a feedback for other components that for example care about cluster maintenance about node drain. And so we have this like a communication channel through the API where your responder is expected to implement a heartbeat. So, like currently in the alpha one, we set it to hardcoded 20 minutes. So, unless your application doesn't

respond in 20 minutes, you are given as much time as you want to basically migrate. And if you don't fulfill this contract, go to the next responder and we have a default responder in our in Kubernetes that is added to every pod and that's the original imperative eviction. So, you see we will have a controller for imperative eviction. If there is no responder, we will just fall

back to the imperative eviction and run into PDBs again. So, as you can see a layer above PDBs. Okay, I I I can do the feature ideas later and then I would like to show a demo. Okay, so we we can try these interactions ourselves. So, for example, uh we have a simple pod. And okay, so let's start with a pod. It's running. And I can show

you just the basic default scenario where you don't have any responder and you basically want to evict. So, I need to supply like two names two fields which is pod name. And then I have to also fetch the pod UID. So, let's go. And as you can see on the top, the eviction is progressing and we see the pod is already terminating. And I can give you

also like a wider status. We also have a message from the responder which is the imperative eviction that says we have deleted the pod successfully. This also gives you a way if you have This gives you a way to trigger termination even within your application without deletion. So, you can actually observe this eviction API and your pod can decide to terminate on its own and the pod

object will stay there with the logs and everything? So, we have multiple ways of achieving And then let's try with a PDB. Yeah, it's a simple PDB that should have at least one available So, in this case, let's first delete the eviction request. And okay, so now we are in a scenario where you are where you where you care about your pet application, but your application cannot

be disrupted. Okay, so when we run the eviction now, the pod still keeps running. And when we look at the wider status, we can see that it's not possible because we are violating the PDB budget. And we have an attempt basically have an exponential backoff and we will retry. So, today there are many controllers that do eviction and they have to retry and for example like kubectl

drain. And this gives you a way to just prescribe what you want like declaratively and we will take care about the eviction eventually you even after you have a like a disruption budget allocated. Yeah. Yeah, and just to clarify here as well, in the demo here we're showing use pod disruption budgets, but we have written this code in as an eviction request controller. This could have been

any business logic. This could have been drain out an application's data if it's like a database. This could have been make sure there is a new replica available before deleting the old one. that we want to program. PDBs is what we expressed it as, but you are not constrained to PDBs here. You're constrained to whatever code you want to write. Yeah, I actually have a I can

show that as well. Okay, so we have an next application which is simple replica set and I have implemented a responder that tries to migrate this application. So, this basically has a higher priority than the default eviction. So, let's try to first create the replica set. Okay, we see the pod is running. Okay, we need the pod name again. And now when we create the eviction, Oh,

here. That first we create a new replica set called DB migrated. And then once it's available, the old one can go away. once the original pod terminates, we should Yeah, and we should see the status is evicted now. So, basically as Lucy mentioned, this allows you to add any like logic that you need. How much time do we have to run? We have a description of the

the description of the responder as I said like you you define it in the pod and then the contract is in the eviction request itself. So, your responder only needs an ID and then needs to communicate with the So, basically and say say like give us a heartbeat every at least 20 minutes. It should be probably like every 2 minutes or 3 to not overload overload the

control plane and like tell us what it is doing. And if we have that, then it can fulfill its mission. So, actually I I have the responder here as well. Like if you want to just quickly peek. So, basically you need to just observe the eviction request, then see that the target is valid that you take care about this or care about this target which is like

the replica set. And at the end, so I have a migrate function here which is here which basically creates the new replica set, but this depends on your on your ideas. And once you migrate it, at the end we should always like update the status. we really recommend using server side apply because there are multiple actors So, basically everyone has there is no conflict in in the

updates. So, it's just a simple function but this as I said this depends on your And yeah. Okay, so I can show next demo as well for the cancellation. as I was mentioning we have like multiple requestors. Okay, so let's first try to clean up the state. Okay. Okay, so let's say we want to create the PDB again. And like basic application. Okay, so it is running

and the PDB is down here. Okay. Okay, we have the pod name. And now when you create the eviction, it doesn't progress. As you see like this we say it's progressing, you can look at the status, but we are basically retrying the eviction. So, when we get the report from our default responder imperative eviction that we cannot evict the pod because of the PDB. So, what you

can do if you are if if you basically maybe you are an administrator and you don't know what this application is doing and you not confident that disrupting is is the right way. Maybe you need to consult somebody and so in that scenario, what you would do, you would basically go to and change your intent. And your original intent was eviction and now you can do like

withdraw the intent. And once we do that, we get a failed eviction request because nobody is requesting the eviction at And in in this scenario, this allows you to communicate like between various components like who has the interest in Like we understand that there are scenarios when you are too far in your that canceling might not be an option and these are totally valid scenarios. So, this

but this is the state we we are introducing in alpha one, but later we might add support for like forbidding this cancellation for responders. And so, we are start have to start somewhere. So, this is our current And basically to allow for this cooperative To to allow for this cooperative graceful disruption. And like that since we have an API and as I was mentioning there are many

other vectors of terminating pods. And the problem is that you don't know from which vector your termination is coming in. So, ultimately it would be great if we could like change most of the mechanisms or like make them make this change optional for some of these termination mechanisms to allow us to to basically subscribe to this eviction from different sources like a kubelet or a scheduler and

then your application would would not have to specifically handle like every termination scenarios. obviously there are termination scenarios that you cannot reconcile. If you pull the plug on the machine like sad, but nothing can be done. So, there will still be some termination scenarios it will not be possible to respond to them, but our goal is to basically enable as much as we can. And we have

also other ideas in work. So, currently there is a new pod group and work called API in six scheduling we are not supporting these APIs yet because for the pod group, you might want to disrupt the whole pod group at the same time. So, this API will enable us to do that as well. So, the target will not just be the pod, but it will be a

pod group. And ultimately, it's it's possible to add other types in the future after consultation and uh looking at the use cases. As I and as I was mentioning, we have the hardcoded heartbeat deadline, that's also probably going to change the the cancellation policy, maybe other eviction types, and in the end, uh this API doesn't uh live alone. This This is an API for coordination. So, the

most important part is uh that this API will be useful for everyone if everyone starts using it. So, if you have a controller, for example, that's doing the node maintenance, like a node drain, you would get the benefits only after you implement add it to your uh like uh logic. So, basically, instead of just having an eviction call, you do uh create eviction API eviction request API.

And you can also supplement like delete calls as well. And in the end, like when you have these special scenarios or like uh availability, you care about availability, or you care about resource reservation, or you have just uh like a pet application that you want to run, you can just implement your own responder and uh make sure that all the paths that you care about are using

uh the eviction that's all for the eviction request, and now Ryan with life cycle management. Great. All right. Thanks Philip. Uh so, we've heard from Philip as uh one of the layers of eviction, right? There's there's more. We have the node, right? How do we handle this this case and and actually even not just the node, but everything above it, right? Because there's we've got all these

AI machine learning learning workloads that are entering the ecosystem. Like now we can think about topology, right? So, there's like all these aspects and of layers that are sort of ahead of us. And so, I'm just going to talk about node, and um that's what's the other area of focus for our group right now. So, two things uh improving the node uh life cycle management um two

areas, node maintenance, uh and the second one, uh graceful node shutdown. Those are the two areas that we're going to focus on. Um and one other point of context is that um for uh node maintenance, and uh we there's been this talk uh been this this kept that's been around for a little while, it's called declarative node maintenance. Um probably many people have seen it before. Um

one of the things about this kept is uh it had a an an opinionated view of how we wanted to do maintenance of a node. And so, we kind of got together and we wanted to make some changes and and shift the scope a little bit, narrow some things Um and so, I'm going to talk about what's um has has been the follow-up to this, what's that

what will uh supersede this work. And the next uh the most common way we all drain our nodes, right? It's uh kubectl drain, we do kubectl uh cordon. These are the tools that we use all the time. Um and it actually is a challenge for us is that there's lots of tools. It's actually we can't really agree how we want to do drain and how we want

to do maintenance on our nodes. And so, we have so many tools that exist in the ecosystem, and that is also part of the challenge. Okay. Uh and here they are. Actually, there's more than than this list. Uh but the ecosystem's fragments, right? We can't quite agree how we all want to do drain. And that's okay in the sense that drain is opinionated and how we want

to do maintenance is opinionated. But what the issue is is we're have so many opinions about how we want to do this, and we're all implementing the same things over and over and over again. And it would be great if there's some ways that we can take the the the work and the knowledge that we're all doing uh that we're all doing the same and try and

find some common APIs. It would be great if we had some of these tools that could work together. I think there's value in they all solve different use cases. So, that's one of the things that we're that we're thinking about. Okay, so the approach. Uh what are what are we doing about this? Um so, I break it down into three steps. Um first is observability. Okay, so

to picture this problem space and how we're going to deal with this, I just want you to imagine like if you if I if I was doing kubectl drain on the laptop up here, and I told you in the audience you needed to figure out when that node is drained, right? I have probably so many different opinions from everyone in the audience to know when exactly that

node is drained, right? It's not really clear, right? Because I run kubectl drain on the laptop, the any status, any state is on the laptop, it's on the client side, right? I'd have to tell you when the node is drained, right? You could observe some things happening on the node, maybe it's cordons, maybe pods are being removed, but you don't really know when I'm done. And even

in the case that I I was draining, right? You don't even know that I that I was draining, you just see cordons and pods moving in and out, it doesn't even mean I'm draining, right? We can't really agree about uh these concepts. So, that's that's one of the problems, but even before we do that, uh we can't even agree about, you know, where where I can share

the information about about drain, right? So, observability, the first first challenge. How can I share with you what I'm doing? The the the node state. Um and and the transition that we're going through. Okay, and then is is meaningful state. So, first we need to figure out where we can share this these node states, and then second, we can we want to try and agree on what

they are. So, we can probably all agree that there's a rough meaning of draining, that it's removing something from a node. Specifically, what might differ a little bit, but we all have a rough definition of that. So, if we have next we would want to build these states that are common. And then third, uh building what I call ecosystem adoption, right? This is like the idea that

we we start drive this across the ecosystem. So, I showed in the previous slide the list of projects that have common ways of doing life cycle management. So, driving this adoption across the ecosystem, and really the most impactful thing is inside the Kubernetes API. Like think about the cluster autoscaler or other sort of uh core Kubernetes controllers that could benefit from understanding these different node life cycle

states, and having that integration with other ecosystem projects like node problem detector, or your own custom provider that's life cycle managing your node. Those things are really important. We want to integrate those things, they all should be able to understand life cycle of a node and be able to work together. Okay, so that I said the problem that I'm after is observability, that's the first step. Um

there's a kept I'm working on um in our in our group, and this is called specialized life cycle management. This is the kept that is got uh where we where I mentioned we changed the scope of declarative node maintenance, and and this is what we're uh we're working on. Uh my summary of it is to create a standardized declarative API for coordinating life cycle management. Um and

so, for all you, what we're looking for is uh your use cases, your feedback, um your enthusiasm, whatever you, you know, your comments on the kept that would really greatly appreciate uh we'd really greatly appreciate that, and it will help us um actually drive this these concepts into Um and and that would um you know, help help us bring these ideas uh you know, across the community.

Um so, yeah, please check it out, that would be great, and that would help us a lot in in really getting this and pushing this uh into the ecosystem. All right, hopefully you all got a picture of that. There we go. Last one. Okay. Uh the last slide. So, kubelet amnesia. So, I mentioned uh graceful node shutdown. This is one of the features that uh things that

we care about. So, I said declarative um we want maintenance to enable node maintenance, the other one is uh graceful node shutdown. So, this is a problem that people have probably experienced in many different forms. So, I'm just focus on one aspect of it, which is which is uh appears during And what happens, the way at high level I'd look at this is that if you go

and through graceful node shutdown, um the kubelet goes and tries to drain pods on the node, and while it's draining pods, it might suddenly forget that it's doing graceful It could be interrupted, right? And when that happens, the kubelet goes back to being a normal kubelet. It's it can start taking pods in again, right? And and so, it actually has state, right? There's state. It's it's a

graceful node shutdown kubelet state, you know, whatever you want to call it. It should be doing graceful node shutdown, and it forgets. So, that's why we call it a kubelet amnesia. And so, how do we deal with this problem? Well, what would be useful is like if kubelet could observe that it should be in graceful node shutdown state, that it should be draining pods and working its

way towards shutting down. And so, there's lots of ways that we can think of doing this, and um that's what we're trying to to work on in specialized life cycle management. The exact way that we want to uh solve this And um uh so, that's how we're going to try and tackle this problem. Um there's probably there might be more questions about how we can go into

this in the details of this um so that we can answer in Q&A, but we have 1 minute left, so I'm going to turn it over to If that's that's me. Uh yeah, I just wanted to mention the graceful node shutdown. We are looking at this feature holistically, so there are many other issues that uh we would like to attack in the future. Yeah. Yes. Yes, and

that's the thing we've been doing in the uh working group right now is we've been very tight about prioritization, not because there's not a lot of things we want to do, but because um we do not want to dilute our focus and then not ending up delivering any features to you folks for a very long time because we're working on 10 things at once. But these are

some of the things that we are actively working on and exploring right now um in our working group. So, um yeah, that's a preview of the technical side of what we're looking at over the next year. If you're uh interested in becoming involved, uh there are What is it? 400 people in this room? And uh when I go to the Zoom meetings, there's eight, so I feel

like at least some of you should be turning up. Uh there is a uh QR code to the readme page for the node lifecycle group that has our weekly meeting on it. Um and I we will genuinely listen to you if you turn up and explain your problems with node lifecycle. Uh we also have a Slack Slack channel on the Kubernetes Slack. Uh you can get into

the Kubernetes Slack very easily. Just Google join Kubernetes Slack. There's a web page that will give you an invite. Uh and then feel free to join our Slack channel and chat with us. And yeah, and our next weekly meeting will be uh Monday at 8:00. Uh yeah, thank you so much for watching, folks. And if you have any questions, uh we can take them. There's a mic

here in the front. Do we have time for questions? Give me a nod or shake. I don't know. I think we have 1 minute. If we can have one question, it looks like. Over there. Yes. There's a There's a mic in the front or maybe you can pass it around, I see. Yeah. So, it's about the eviction requests. Like uh on 1.37, we'll have it may hopefully

have it on the API, but kubectl drain will still use I assume the old version and the cloud uh operators that kind of do upgrades of nodes and all that will use So, is there a plan or like maybe a like could like would people just use the I know mutating webhooks to kind of create a shim or will just wait for like what's the plan for

adoption? So, we've had like a preliminary discussions with these integrators, and I think most of them are about aboard. So, they would like to see this feature come to because they are they have to solve these uh like eviction uh misalignment or bugs in their code, and this will basically offload their like problems to us and to the responders. So, it's in everybody's a good interest to

actually implement it. But, we have been discussing this, and yeah, we have we have a plan after we merge it to uh go further in in this direction. So, not not initially, but it's relatively early right now in the API standard. And once we merge this, you'll still be able to do kubectl create eviction requests, even if it's not on the drain in the drain command. That

will be immediately available. And I think that's all the time we have for questions, but just come up and talk to us. Um and we just won't talk to Mike. Thank you. Thank you. Thank you.