The Future of Kubernetes Node Lifecycle - Lucy Sweet, Uber & Dawn Chen, Google

This talk focuses on the evolution of Kubernetes node lifecycle management, emphasizing its importance in supporting AI, machine learning, and high-performance computing workloads. The speaker discusses the changes in Kubernetes from its early days managing simple web front-end workloads to its current capabilities, including standardized GPU orchestration. Key challenges addressed include the need for improved node management, handling resource demands of modern workloads, and enhancing observability and coordination in node operations. The introduction of features such as eviction requests and node readiness indicators aims to provide clearer signals and management capabilities for Kubernetes nodes, reducing inefficiencies and improving application reliability.

Okay. Hello everyone. Welcome back to the Cuba EU. And uh uh last year at the Cubicon uh London uh Lucy and I uh f focus on present of give like the advance of the resource management and talk about how we focus heavily focus on the how we evolve our Kubernetes resource management for those AI machine learning and HPC workotes. And this year we are here taking a

literally logical next steps and talking about how we the future of the of the Kubernetes node life cycle management and uh to we have to to really support of the AI in the year of the AI massive of the AI years and we have to evolve of of nal life cycle management and here we are so to looking to look at what we are First let's look

back of the past what brought us here and the over last decade and use use case kubernetes support has uh dramatic shift back to the 2016 and we are heavily focused on the uh the simple web front end work nodes and by uh 20 uh 2020 we are matured of the handle of the state for site workload and also the complex developer uh environments. Just last year

in 2025, we reached a major milestone uh with of the standardized GPU orchestrations. So and uh and and also last year at this stage and I promised in addition of the DIA and the Guler you you may have heard a lot uh over last couple Koopa accounts and I promised several uh 2025 wins and uh uh are all in the reality today like the features like such

as like the granary of the ple resource and in place p resizing and also workload specific node uh uh swap are now stand uh become the standard in Kubernetes, right? So but uh to really support those kind of so the fundamentally things is the Kubernetes the guiding uh from day one the guiding philosophy for Kubernetes it is Kubernetes should be boring still remain the same. The problem

the here it is a paradox and we want to really want infrastructure uh keep boring and but uh the innovations you guys build on top of the Kubernetes actually should be uh exciting and also the high stakes as possible. Um to really achieve that obviously uh we have a lot of problem and we need to look into especially on the node management. So we need to look

into re-exam how we handle off the node life cycle today. So the reality it is uh we still running the same core kubernetes right. So uh but the uh context we are operating in is shift completely uh last couple years We are not just simply manage ultra street of the um the simple web services. We are actually manage of the expensive um uh uh accelerators such as

TPU, GPU and which require of uh incredibly um hardware uh alignments and we also uh start running uh managed and running the massive of the batch work nodes which needs to be treated as the single unit instead of individual uh parts and also our stateful site right uh has grown so complex so that they have much higher demand uh on the uh higher of persistence and also

the lower latency which is than ever before we we understand right so so to look of those kind of uh things so those challenge actually those workload uh in the 2026 that give us the challenge it is the stand the old standard of kubernetes um uh the defaults uh it is built wasn't built with those special needs of the context what we are facing today right so

we if to solve those problem we need to look into those how we manage those machines how we are going to join and serve and also ret those machines within your uh modern clusters right so by looking at those new work nodes we support right so there are four correct uh character takes stand out and which make us to think about the current node management um uh

is in inadequate it's not enough right so the first one it is those workload uh is not leaving not like of the old of the microservices you can restart in a seconds right so uh LLM uh training jobs may run couple weeks a simple uh maintenance A simple maintenance uh uh event may trigger it's not just our maintenance bleeps like before right so it's where trigger of

the cast chosing problem for your work node and also those work nodes are very brittle right so most of the work work nodes I couldn't tolerate those noise right so a simple of the driver GPU driver TPU driver even CPU driver have the drift and or maybe of the alloys label in of your memory bus will actually to not just downgrade slow down your performance actually could

be uh shuttle entire of your training work nodes right so and also we are also facing off the much much strict right u compared to before um so um at today's modern accelerator cluster any minutes of the idle time means a large financial potential uh drink uh to the to the users. Finally, we have the series of the data grat gravity problem than before. So uh during

uh like running of the inference uh parts and uh you are basically it is take it's not just couple second it is maybe it's 10 minutes of the uh uh black uh period for your uh working nodes. So um now I'm going to hand over to the news to talk about from the uh simple signal and also coordinate the needs talk about our today's node challenge. >>

Thanks D. So good morning everyone. My name is Lucy. Oh can you hear me? Hello. Yes. Yes. My name is Lucy and I'm one of the leads of the No. >> Hello. Okay. I'm just going to use this one. Yes. So thanks D. My name is Lucy and I'm one of the leads of the node life cycle working group. So there are numerous challenges today across node.

It's in no way a perfect picture. Um but when I think about these I like to break them down into almost three groups of uh challenges. We have uh signals um observing uh capabilities on nodes uh discovering features on nodes. Nodes are not homogeneous things. They are they are distinct and separate and they have different capabilities different chipsets different whatever. Uh we have actions um coordination of

actions right now. Sorry, using a running actions right now is not really defining Kubernetes. There's not really uh node maintenance per se as a first class feature. Most people just kind of make it up themselves. They do some sort of drain probably using takes and tolerations. These sorts of things. The same is true of safe replacement and of a batch state for batchware disruption. And then there's

also coordination. So um the observability and visibility into the intent of doing something. So not just this node is disrupted but I intend to disrupt this node in the future. Right? So these are the three uh areas I like to back uh to bucket these into. Um and so a few of the challenges we have right now which I which I've selected as being particularly important. Um

uh one of them is uh maintenance equaling a drain. So the issue here starts at the uh planned action point even for routine issues like um hardware checks or uh necessary updates on an AI inference node. There's no nuanced way to signal a maintenance start. So instead we default to treating this almost like an unexpected node failure. We taint the node which forces a pretty aggressive drain

of it. Um or you might taint it with no schedule. you've already effectively then blocked this node from being used at all until it's completely drained out. Um, and that aggressive draining use means you have a lot of underutilized resources and you unnecessarily are stalling workloads before the hardware is even taking offline which is wasteful and is leading to poor efficiency. Um, another of the uh big

issues I think people have is node readiness. A lot of people I think especially in large companies have noticed that the kublet being ready does not mean it is ready to serve applications. You have things that you want to run on this node before before that. Maybe you have some sort of damon set running some sort of networking thing. Maybe it has connects into your observability stack

somehow. Right now it's just the kublet is ready and therefore everything it's ready for any workload. There is no sort of um ability to say the node is ready for this thing but it's not necessarily ready for this thing. Um and as the complexity of workloads and nodes has increased, this has become an increasing problem and people are having to solve it themselves through all sorts of

uh homebrewed uh concoctions to uh make this work. Um so um so those are two of the problems and then uh I'm going to hand over to Dawn to talk about one of the other ones which is a feature discovery. >> Thanks Lucy. So let's talk about another problem. Look at the information gap what we have today. Right. So uh so when in the kubernetes by design

we used to think about the node is cutle right so you can throw away and so uh so any kubernate we assume they can handle all the work node and uh so yeah so but the situation is changed today um so um so currently future discovery it is inconsistent at best kubernetes right so this introduced obviously. Oh, you can't hear me, right? So, sorry. Sorry. Okay. And

and currently Kubernetes is incredible conservative. >> Just keep going. >> Oh, sorry. The Kubernetes is inc. >> So, Kubernetes incred incredible conservative. It is simply just report about like the uh Ninx and also x86 and it it is blind to all the high value details which is schedule care about and also our two 2020 uh6 work node uh leaded uh to better discussion right so this two

to two make that scheduling can do the okay jobs today and uh so we observe of the label we have the label fragmentations, right? So customer in the production we say that notice a cluster uh a cluster using of the label of the node and with the know one naming convincing another cluster using a totally different of the naming convincing and this make of the right of

the portable and crosscloud uh work node spike a light mile and also complete store of the automation what we are promised make the kubernetes uh boring right and why this matters today right so because um in today's year of the black well and also the um the guardry that most expensive of the GPU, TPU uh accelerator and uh clusters right so the okay scheduling placement decision actually

not just slow down you and and could be contin of the uh shutter of your training jobs uh performance right so you are going to get really really bad push puts right instead achieve there's no way even though you tone off the performance there's no way for your for your achieve of the good puts um so another problem all this needs a previous know the problem is

needs of the the the final problem which it is our application replicle availability is no longer guarantee as before right so so in the mac year of the microservices And we are focused on optimize the fast restart. But today's work node have the data gravity. So uh if you evict off the inference part on a node and you basically actually have to waiting for uh the new

node come up and is ready install off the uh drivers at the same time and you pull off those one maybe the one terabyte of the model uh model weights and also warm off spinner time warm off your cache right so during that windows you basically have like the your replic effectively zero on that note. So this is totally breaking our promise. The root of problem of

this one it is basically it's combination of the uh uh ultra stration latency and also the blindness right so the so the the system kubernetes uh take the system take time to detect of the failure and and when they process that time your service is done and but at the same time and the work node because link of the uh intent and information so node don't know

the eviction being evict why it's being evicted is due to of the right rooting kernel patch or it is just due to uh the accelerator it is unfair right so so they don't know how to uh effectively to recovery they don't know it is supposed to be the checkpoint the work nodes or it is they have to uh shift the traffic right so all those kind of

things um why this is uh this this problem is so difficult at this torment. So it is later after Lucy and I will introduce a solution to make uh to evolve those kind So okay so now we move from the frictions to the fix right. So the year 2026 uh uh fix is not of a a collection of the features actually we try to provide the coordinate

effort by the signal to try to provide uh a coordinate node life cycle and so we are attack this problem from the four uh different uh uh fronts right so first we provide better uh disruption signals and through of the node radates and also part the disruption budget right so and we also second we we are actually end up of the safe maintenance flows through of the

eviction request API which is really complicated AP uh the the features third one we try to like earlier mentioned that we try to get rid of those um enable uh uh uh fragmentation and provide a stronger much stronger of the scheduling signals and to move uh ourself into the managed intent. The last one, last one it is all needs to abide observability, right? And then we are

close this working workload intent gap and to make infrastructure uh uh infrastructure node especially node life cycle visible and observable and predictable and so help our infrastructure uh engineer and have the confidence to rebuild of the automation. That's kind of what our road map and how we are going to move forward. So next I'm going to hand over to Lucy talk about most complex features. >> Yes.

Uh thanks Dawn. So uh one of the things Dawn just mentioned is this idea of moving from almost unobservable and reactive um systems in Kubernetes to understanding intent and eviction request is one of the ways that we're tackling this and it's one of the major deliverables of the working group. So eviction request is a new Kubernetes API that is meant to evolve the original eviction API. The

original eviction API in Kubernetes uses pod disruption budgets and the eviction object. You want to get rid of a pod, you create an eviction object targeting that pod. Either the eviction object gets created and the pod is removed and you get an okay created or you get a 429 too many requests error. Now the problem with that is this is just a point in time unobservable decision.

What happens if you need to go through a process that lasts longer than a second to get rid of a pod? What if you need to uh drain local data on a node to a different node? What if you want to spin up a new replica before spinning down the old replica? In the current system, this isn't really possible. And even and even if you repeatedly request

eviction and do it that way, no one can observe what you're doing. So eviction request is uh our answer to this. Eviction request is a new object uh that will be in Kubernetes. Um it allows you to create an eviction request and target a pod. Uh so uh it allows you to create an eviction request and target a pod. Um that then that eviction request is created

immediately but does not mean the pod is removed immediately. It's starts a process where controllers can observe the eviction request, see that someone does want the pod to go away and react on that and do their own business logic. they can gate that eviction request from completing and therefore do all of their data moves and whatever else they would like to do before the pod is then

finally sent to the uh original API initiated eviction. So to give a very brief uh overview, we're not going into the API fully here. Uh if I have a pod here and I want to get rid of it, maybe the pod has some local data or something, so I don't want to be too disruptive, I can create an eviction request on the cluster targeting the pod. Um,

and then once I've done that, other controllers connected to this cluster can see, oh, look, there's an eviction request and it's targeting this pod, which means that um, some there's some stakeholders who would like this pod to be evicted from the system. Um, I'm going to go off and do my logic. I'm going to uh I'm going to do whatever I need to do. Maybe it will

take 5 seconds, maybe it will take 5 minutes, maybe it will take 5 hours. Um, who knows? Eventually, once that's done, they um, ungate the eviction request. So, they signal this controller is completed on it. Um and then the pod goes through standard API initiated eviction. So this uh feature we uh merged the ke but we did not manage to merge the alpha feature for this release

136. So we're going to target an alpha for this in 137. Uh this is one of the major things that we've been looking at and this finally uh solves at least the issue of not just seeing the uh not just seeing the specification of cubase but the intent what do us not just what the current state of the system is what do users intend the system state

to be in in the future and can we then perform longing actions based on that right um so that is one of the big features we've been uh delivering uh this year um and then ne next uh dor is going to talk through another one which is node de node declared features. Okay, I briefly talk about two more features we are also currently working on and uh

so one it is earlier I mentioned that we are suffer right so with the label uh spaghettias right so two we are in next release and we are introduced of the node declare of the uh feature which is introduced of the node status declare features and we are basically change about the chaos of the label and ging all those kind of things into uh more uh managed

of the uh uh uh uh intend right. So, so then with this features right so you are basically immediate of the three wins and you got you got sorry you got the wor school of the protection and the true portability and also the zero config schedulings right so this is help us to solve those black hole scheduling issues and also another features I briefly mentioned is node

readiness gets right so just earlier um Lucy mentioned that when we talk about the node writing and that's totally no bar for kubernet today right so you just don't know if it's GPU installed and running properly and also even sometimes even you don't know that driver handle your hardware version generation all those kind of things right so all those kind of things actually we have introduced this

fix it is out of tree uh from the rile but you can deploy based on your product environment and this feature It is introduced about new CD called the readiness rules. So you basically can uh customize after what kind of do you mean for your uh production your nodes it is node real ready right. So so this one totally change of some previous think manual admin and

manually tag of the node with the with the labels right. So then they change it to the management and management uh uh intend. So with this kind of things and uh we can before scheduler schedule of the part we are insure of the that ID the node is from the uh simple ID broken ID into of the real name serving mode. Right. So I just want to

mention that we have we only talk about the introduce of the sorry we only introduce of the three features and but we are addressing from the four front ends right so for the signal and action and the coordinations and there are other other features is ongoing and discussing and proactively development. So, but I just want to mention that even with those features, this is still isn't enough,

right? So, there's problem if we really want to make a Kubernetes infrastructure boring and uh we need move from the infrastructure focus and the node focus to the workload intend. Just one example, right? So for the training jobs and for maintenance actually customer really want can you come do the maintenance combine all your maintenance no matter is host level maintenance kernel maintenance and also kubernetes upgrade all

together so then you can reduce after maintenance time like aggregate maintenance time and also they want of all in one or nothing right so they want you finish all those work and return release of the accelerator back to the you training jobs as soon as possible. Right? So that's kind of but for the informance job especially for undying re informance job it's totally different and because they

need uh we need to build up the policy and uh based on the user requirement based on those kind of things and do the ruling maintenance not just logging maintenance and also for the new uh disagregate of the inference the situation even much harder right so node behavior is totally different some node is the for the uh is more compute field uh uh bounded and some face

it is because we deploying of the inference is not at the inference request instant inference faces right so then then some of you require the node it is uh have to be compute bounded and some it is memory uh intensive work node so you need to make sure that different type of the node readiness different um uh declare a capacity describe those node all those kind of

things actually it is uh what we are looking into right so at the same time also we continue enhance our uh our resource management because to really achieve our good put for those work nodes so now I'm going to hand over to the uh Lucy >> yes thank you Dawn um so this then brings me on to this question why do people use GKE autopilot why do

people use uh G EKS autopilot because as much as there are a lot of people in this room who care a lot about nodes. Users just want their applications to run. They don't care about nodes. They just want an application to be deployed successfully and to run successfully, right? Um and so they what they do not want to be in is a situation where nodes are pushed

onto them as an avoidable logical boundary, right? And so that then leads me to this idea of going beyond node of person of moving the abstractions of the abstractions away from uh node pod and onto higher level ones that users would want to reason about more anyway. Um so we we can start to look at that through abstracting um and making nodes an implementation detail rather than

something that you have to care about a lot dayto-day. Um and uh the the overall goal is just that nodes become another detail. It's something that I'm sure a lot of folks in this room will continue to care a lot about. Um but the end users deploying applications don't really have to uh care too much about it. So there's a lot of ongoing uh efforts in this

area right now. Um so we have a workload API from uh sig scheduling mostly. Uh it this is a foundational API which was recently introduced in v1 alpha 2 that moves us away from scheduling individual pods on nodes to scheduling whole workloads as a cohesive unit. So all of the uh collect collection of things needed for example to run one application as one unit. Uh rather than

creating pods at all you create workloads instead. Um, we're extending DRRA beyond just individual devices to use it as a mechanism for modeling complex infrastructure topology and node groups. Um, and we're exploring how to model groups of nodes as abstract resources similar to how we manage pods. So that if one node fails, it can be replaced by another with the same required signature and tooling con constraints.

Um, so that's what we're so that's where we're going and that's what we're thinking right now. And I just want to acknowledge that uh this work is not done by just the two people who are on stage right now. The Kubernetes contributor community has uh grown and grown and grown. Uh this is us on a Sunday over at the stairs um over at that side of the

building. Um and a lot of work has been put into this by a lot of people across both sig node node life cycle working group and other sigs and working groups throughout Kubernetes. I'm not going to name names because I will miss someone if I name if I name names but some of them are here in this audience. So, um, I really like to extend my appreciation

to them for all of the work that they have done to, uh, get us to this point. Um, and with that, thank you. I'm going to take questions. >> I I have a question. >> Uh, I did not understand how you treat notes which are in an unknown status. uh in some undefined status you you have you have some somehow to eliminate them before before you can

go on I don't understand this >> and no status of the node that's the legacy problem what you have most of the and no problem it is of course unstable kubernet kubernate right so couldn't kubernetate in the crash loop and couldn't report the heartbeat and the status back to control plan and control plan haven't heard another problem it is a network partition we are yeah we should

solve this problem and uh but in today's we actually didn't really but that's the the problem we are going to detect enhance of our network especially network improve our reliability and also network partition handling prop yeah >> also if you have stateful sets on the node you have to remove somehow the uh the the volume >> yes it's uh >> we can evolve of the eviction request

API instead not just eviction request actually and we could could do something better right >> in classical clusters this is solved by fencing but I don't think you want to introduce fencing >> um we can revisit but actually we've been talk about the fencing in a previous due to the complexity of the features than we didn't do but just like what I mentioned like in this talk

because we support more work nodes and the situation changed we can re v revisit those kind of things. >> Yeah. Yeah. >> But yes it's um and just to quickly jump in here I'm not promising anything. I would love for eviction requests to evolve to an intentbased system for volumes for example as well so that you can tell okay there is an intent to eject this volume

so I'm going to move the data before I start trying to do that. Um yeah uh >> if if you make any changes to node labeling do do you have plans to communicate those changes to the AI conformance working group? >> yeah Jennet to sit next to me. >> Cool. >> And and so we yeah we work we work >> and yeah I basically work with all

the AI inform infrastructure teams. Yeah, >> thank you for the talk. Um, you mentioned a lot of uh beautiful features, but uh there is still open issue about the restarting of the Kublet itself. Uh, and you mentioned that the problem with the maintenance of the node. Are you going to address it somehow in the future? >> Is this the whole Kublet amnesia bug? >> Huh? >> Is

this the whole Kuba amnesia issue? Yeah, about the ports uh and labels and pots can be rescheduled after the >> and maybe some raises between Kublet and Kube API >> etc etc. So it still opens the old issue that uh don't try to restart Kubllet uh in the runtime and train the node before >> Is it still actual? And >> I I so I know that um

folks in the working group are looking into this issue. There is a talk this afternoon um and Ryan Haly who's on that talk I believe said he's going to talk about this exact problem. >> And graceful node shutdown and how that can help solve for that. >> Um we just didn't have time in this talk to uh catch everything. Yeah, great. So, >> we didn't mention all

the features we are working on. Yeah, >> we're just choosing three features. >> All right. Uh, thank you. >> Thank you. Thank you everyone.