Addressing Non-Deterministic Scheduling: Introducing... Ajay K, Sreeram V, Karthik N & Priyanka S

This talk discusses the challenges of non-deterministic scheduling in Kubernetes and introduces the Node Readiness Controller (NRC), a project initiated by the SIG Node community. The speaker outlines the problems that arise when nodes announce readiness prematurely, leading to application failures due to uninitialized components like CNI plugins. With NRC, a coordination layer is established to ensure that workloads are only scheduled on nodes that are fully prepared by utilizing a Custom Resource Definition (CRD) to extend node conditions. The session covers the design considerations of the NRC, its operationalization in real-world environments, and the controller's monitoring capabilities. Additionally, it addresses how the NRC integrates with existing Kubernetes components and discusses future collaborations and enhancements planned for the project.

Hello everyone. Good morning and welcome to the final day of KubeCon. Thank you for coming to our talk. Uh today we are going to talk about um addressing non-deterministic scheduling and as part of that we'll introduce a new controller, which is a new subproject of SIG node, node readiness controller. Um my name is Priyanka Saggu. I work at SUSE. I'm also participating in some other parts of

the Kubernetes project. I'm a technical lead for SIG ContribX and recently participating in a bunch of um KEPs in SIG node. Yeah, hey everyone. My name is Sriram. I work at Digital Ocean as part of the managed Kubernetes team. I've also been involved in the upstream Kubernetes project for a while now as part of SIG release for the release team and I've also been working on a

couple of KEPs with SIG Hi everyone. I'm Karthik. I work as a cloud engineer at IBM. I'm a contributor to the cluster API and SIG node. Yeah, thank you. And we're supposed to have Ajay as well joining us on the stage today, but he could not travel, so we will have his part recorded and we'll play now. In a moment, but today's agenda. So we are going

to cover our presentation in these parts. We'll start with what the node readiness problem looks like today. Following followed by that we'll introduce the node readiness controller. We'll also touch a bit on the design aspect of the node readiness controller. And then we'll go into how node readiness controller operationalizing in production spaces looks like today. And we'll finally conclude the talk with some future work we are

planning to do and collaboration with the ecosystem and resources to get involved. So we have a part from Ajay, which is recorded. I'll just start with that. Hi everyone. I'm Ajay. I'm a software engineering lead in GKE node team. Today I'm here as the lead maintainer of the node readiness controller project. This project was born out of real production pain we have felt at Google. So we

kept seeing these recurring ghost failures. Where a node would announce, "I'm ready." But in reality the CNI wasn't actually initialized or the driver hadn't bootstrapped fully yet. Causing application failures. So the problem is that Kubernetes nodes have a race condition built into the heart of the bootstrapping process. So kubelet is not aware of different infrastructure requirements for your nodes that change from cluster to cluster. So I

built this coordination mechanism for GKE. But realized this wasn't just a Google problem. We wanted to fill this architectural gap for the entire ecosystem. And that's how NRC became an official SIG node project. Um okay. So to understand the problem, we have to look at the nature of how Kubernetes handles readiness today. So let's look behind the curtain on the node bootstrap process. In Kubernetes, node ready

is a very low bar. When a new node joins the cluster, if the kubelet is up and sees a basic CNI config is present on the desk, it sets the status as true. It would lift the scheduling gate for the workloads to land on it. But ready doesn't really mean usable for the workloads, right? Think about your production stack. Does your CNI plugin actually finished installing the

routes? Um if you are running expensive AML uploads or the GPU device plugins warm and ready to accept the pods? If these are not ready, then when the scheduler sends a pod to the node, you don't get a running application. It just ends in crash loop back off on unpredictable startup. So, coordination is impossible without something like NRC today because Kubelet doesn't know about these external dependencies.

So, this brings us to our fundamental challenge. How do we make node readiness deterministic in Kubernetes? That is exactly why we built the node It is an out-of-band controller that adds a layer of coordination between your infrastructure components and the scheduler. Our vision is deterministic node bootstrapping. We want to move away from a hope-based scheduling to a world where we know things are ready. We want to

ensure that workloads land on the nodes only when the critical foundation is 100% operational. our mission to provide a declarative system for the nodes. So, instead of hard coding readiness onto the Kubelet, we give you a CRD-based framework where components can report their own state and the controller will act as the gatekeeper. Now, let's look at this animation to see idea in action. So, we have a

cluster and NRC is in the control plane. A new node joins. NRC immediately will see it and apply a as a temporary taint. The node is now protected. Now, look at the three daemon sets. Your CNI, your GPU driver, and storage driver. They are still warming up. The scheduler is trying to send pods, but it sees the taint, and it waits. Now, one by one, the components

finish. They turn green and report back report back to the API. Now, the lock opens. The NRC lifts the taint, and now only when things are ready, the pods move on to the node safely. This is what NRC in a nutshell. So, when we designed this, we had three requirements. First, we didn't want to change a kubelet. We used a CRD to extend the fundamental ready condition

for the node. Second, it has to be declarative. You define what ready means for your own specific nodes. And finally, it had to target the critical demons. The stuff that you know ahead of the time, like CNI and drivers. So, that must be there before a single app container can start on the node. Um so, what have we achieved today? NRC is the production-ready out-of-band implementation of

the node readiness gate. By using an external CRD instead of modifying the node spec directly, we actually created something better. It gives you more flexibility. You can use label selectors to target specific node pools, and like you know even use features like dry run to test your rules to safely before enforcing them in production. We use a well-defined prefix for your taints to ensure that we never

conflict with existing taints. Whether you need a simple one-time bootstrap check or continuous monitoring like you know guarantee for the life of your node, it's all there. Um this is ready for production. We are already seeing great results. I invite you to try it on your workloads and share your feedback with us. With that, I'll hand it over to Priyanka to dive into the mechanics of the

controller. Thank you. So um right after this, I'll just quickly show a demo of the controller in action. It's a very quick demo. Um here uh whatever the animation that Ajay talked about, you will see that in action in a Kubernetes kind cluster here. So, if you see here, we have two Wait a moment. Here I have a kind cluster with a control plane and a worker

node, and the worker node starts with a um is registered with the startup taint called readiness.k8s.io network ready. I'll try to see if I can um pose it. So, the the NRC controller is running on the controller plane and the control plane in this case, and the worker plane work worker node is registered with the startup taint. And if you see this taint, it's a very special

taint prefixed with the readiness.k8s.io. That is very important for readiness node readiness controller. Node readiness controller only monitors these taint. It does not um block or it does not um hamper any other taints that that on the system. at this point, if you see the taint is already there on the worker node, but it's not lifted. And now the requirement for the from the node readiness controller

is to actually lift this taint, and that would be the signal from node readiness controller that this particular component, in this case it's um Calico It If the taint will be lifted, that would be a signal for the Calico component is ready. So, what what now we'll do is um we have created a node readiness rule. This is how we are declaratively defining our um requirement, readiness

requirement. We see that node readiness rule has two main parts. The conditions, that is the condition that this node readiness um defines for the controller to look at the node status. So, if this condition will be available on the node status, immediately that would be a signal for node readiness controller to call we are ready, the node is ready to take uh workloads on. Until the condition

is ready, the taint will stay on the node. So, now here we are flipping we are deploying Calico, the condition will be available on the node, and the taint will then be lifted by the node readiness controller. So, the moment we deployed Calico, the taint is now lifted. Yep. this is an example of a node readiness rule. Here we can see um the node readiness controller brings

this only CRD, uh which is called node readiness rule. The major parts are the conditions and the taints. The condition, again I said, this is the condition that every component that we want to be ready has to provide on the node condition status sub resource. Until that condition is satisfied, the node readiness controller will look at this node readiness rule and taint the node so that the

workloads does not get scheduled on that node. We also allow to um define the enforcement mode. So, we have two enforcement mode. We can start with the bootstrap only. That That means we just want to define the readiness until the bootstrap node bootstrap is done. And once the node node is bootstrapped, we stop the monitoring from the node readiness The another one is continuous one. Uh Karthik

will talk more about uh the uh enforcement modes in detail. And finally, the node selector. This node readiness rule is going to select or apply on which nodes that is going to be defined by the node selector field down Now, how does it look on the node side? So, we define that node readiness rule on the node. Once the node readiness rule is created and the NRC

is running on a in inside a cluster, we'll see on the node there will be a taint registered by the node readiness controller. And you see the taint is prefixed. We also see the node right now has the uh standard node ready condition that is not enough to lift that taint. What we need is the special um newer component ready condition. So, this needs to be provided

by the component. We have mechanisms from within the project to provide that condition or the component can itself provide that condition. All we need is the node need to have that condition published on this node condition sub-resource. Once the condition is there, it's ready. The NRC will monitor it and lift that taint from the from the node and it the node will be ready to schedule workloads.

at this point, we saw this taint is still there because the condition is false. So, what does what is needed to lift that taint now is this condition to become true and once the condition become true, the NRC will react to it. It will remove that taint. And another thing which I showed that in this example, I showed enforcement mode is bootstrap only, which means how do

we tell it is the NRC has to stop at bootstrap time. It does not have to continuously keep tainting if Calico, let's say goes down again. So, how do we do that is once the bootstrap is done, we will annotate the node. We'll add annotation to the node with a completion annotation and then NRC will stop monitoring that node So, this is a recap. This will be

available in the handout notes. This is exactly what I explained. Um when we started using NRC and we we started deploying examples, we figured out there are a bunch of things we have to fix before NRC does not block itself or the components or other other components in the ecosystem. So, what we realized is majorly two things. Let's say if you are running cluster auto scaler as

well in your cluster and then node readiness controller, what cluster auto scaler will see is because now we have tainted the node with the special readiness.k8s.io taints, the the workload will not get scheduled on those nodes. But for cluster auto scaler, it will see those unscheduled workload and it will become confused and it will start scaling up down. So, we do not want that to be the

case. Uh and we managed to solve it from the help of cluster auto scaler project and uh people from the community. Recently, I think about a month back or within last few weeks itself, there was a PR to introduce a new flag uh called startup taint prefix uh in the cluster auto scaler binary, which allows us to add this special taint from the annot- see at the

time of um cluster auto scaler uh configuration itself and we do not have to restart cluster auto scaler every time the node is tainted with a new readiness.k8s.io taint. With this uh flag, what it does is every time cluster auto scaler will see that there is a uh node with a readiness.k8s.io taint, it will scan and it will just ignore it. It will not go up. It

will not scale up down the nodes just because of the readiness.k8s.io taints. So, that is one problem we managed to solve and the another one is the daemon set deadlock itself. So, um if we have a component that itself needs to be deployed on the node to actually make that component um readiness available, that won't work if the the the components does not have matching tolerations to

be scheduled. So, there is a in progress. I think it's almost ready to merge PR in the um node readiness controller project, which is utilizing uh mutating admission policy map. This feature is going beta in 1.36, so it's a production ready feature now. Um what it what it will do is it will scan all the node readiness rules that are available in the uh on a particular

node and it will automatically inject matching tolerations on the daemon set only on the daemon set objects. So, the daemon sets then can come up on the node and um help us to configure the infrastructure to actually bring the readiness conditions ready. With that, I'll pass to Karthik. Uh yeah, so let's look at the operationalizing the node readiness. Uh as Priyanka mentioned, uh to ensure that we

apply this rule uh to a cluster seamlessly, uh NRC comes with two modes of uh enforcement. Uh one is called bootstrap only. Uh in this mode, what we do is that whenever a node is uh initialized, we add the taint, and if the conditions are met, we remove the taints, and that's it. So after that, NRC doesn't care about what's happening in the cluster. Uh the main

use case uh for this is for one-time initialization. For example, you are initializing some GPU drivers or pulling very heavy images, right? Um going ahead, if one time is not sufficient, what we should do? So then we have this uh continuous mode. Uh in this mode, NRC will keep on monitoring a node throughout its life cycle uh for the readiness criteria. For example, when a condition is

met, it will remove the taint. If condition is faced later in the day, it will add the taint back again. So for example, to give an use case, say consider you have a security agent running on a node, right? Uh for some reason the demon set crashes, you don't want any new pods to be deployed on that node until the demon set is back. NRC helps with

this continuous mode in that case. Um those two are the enforcement mode we have. Also, to facilitate easy operation and a safeguard, we have a dry run support as well. So when you uh apply a rule in dry run mode, uh what NRC does is it does all the evaluation, but it don't apply the taint on the nodes. Instead, it will show the results in a status

field. So with this, an operator can get easy idea of what actually going to happen if you apply a uh rule in a Moving ahead, uh this By now, we know how NRC is and what it does. Uh, but if you're running in a production cluster, uh, you might have a couple of questions, right? So, what is the latency that NRC is going to add to my

cluster? Or how long my compute-intensive nodes are stuck behind a taint just waiting to be evaluated? So, to answer these questions, we have done some, uh, experiment. So, main objective of this experiment is to show that the controller takes very less minimal time and also, the memory usage is very less under high load as well. Uh, to capture this, we use some metrics and also the ecosystem,

uh, the provided tools. For example, this NRC itself is built using controller runtime, so we get out-of-the-box metrics. On top of it, we instrumented few more custom metrics so that we can capture, uh, more data. here is the experiment results. So, what we have done is that we simulated a scenario where thousands of nodes are trying to join a cluster at once. For those who haven't used,

we used Quark, uh, to simulate this scenario. This Quark is nothing but a Kubernetes without a kubelet. It is yet another sig, uh, project. More details about can be seen in that project. Uh, so Quark helped us to instantly create 1,000 nodes, saves time, also avoids real compute cost. So, in this experiment, we created 1,000 nodes joining a cluster. We created single rule. What we are expecting

NRC to do over here is that, uh, it should taint all the nodes because whatever the uh, condition is not currently satisfied on the node, then we patch the node with the required condition so that it uh, supposed to untaint it. The result is seen, uh, we detect the NRC is able to detect 1,000 nodes, taint it, untaint it, and eventually not even a single event has

been missed over here. Uh, very interesting and a curious part is the peak velocity we achieved. We hit a 500 nodes per second, uh, taint addition. The green line you see over here is the taint addition and the yellow is the taint removal. Uh, the controller is able to maintain high throughput throughout this node life cycle, uh, even for a rule action. The last one what we

see over there is a work you depth. Since the thousand nodes added once we have a high peak, but eventually that got drained off with the as we gone processing. Right. Now we know that our controller is fast. What about the latency? So the first graph shows the latency thing. The green line you see bottom down is actually the evaluation logic taken by the controller that is

less below 35 milliseconds, right? And the yellow one is about we are waiting for the Kubernetes API server to respond. So that's also well around 35 milliseconds. The processing pipeline latency what we have here describes about what is that weight queue weight time. Since we have done this experiment with the default configuration of QPS and burst in the max concurrency limit, we see a bit a high

in the graph, but we wanted to have this in way so that we can show the graph varies in various ways. Finally, we have covered latency. We also covered the processing speed. Now we talk about the infrastructure consumption. With this high load as well, our controller never reached more than 50 MB of usage. Also if you look at the processing unit, it's less than a quarter of

a CPU. So from infrastructure perspective, the controller is almost silent. When it comes to go routines, we almost hit at a 56 and we remain at 56 even after the work completion. This makes sure that we don't have any leak and doesn't causing any memory issues. So the boring but yet interesting are the last two ones where it shows that we have zero panics, zero OOM kills,

are zero failures. So ours is a rock solid binary and ready to be used at a That's That's all we have for today, but what if the scalability for the future? So currently we have a hard limit for ETCD of 1.5 MB. With high scalability, currently we are about consuming 6.5% or 1000 bytes per 100 bytes per node. Um, to remove this for a future use case,

we are trying to evaluate into a new CRD for splitting a CRD into a rule CRD and evaluation CRD. It's under plan. Currently, we are working towards it. Yeah, I'll hand it over to Sriram for next. Uh, hey folks. So, now we wanted to show you what we have done recently to improve the security when you're using node readiness So, as we had discussed earlier, the node

readiness controller lifts the taints on different nodes based on whether the node is ready or not, but the controller itself does not, uh, check if this condition has been satisfied. So, that can be done in three ways, actually. The first, uh, way is the component that you're deploying itself, uh, updates the status of nodes. So, for example, if you have Cilium deployed on your cluster, Cilium itself

would update the status of a node as if you have a custom component or if you have some custom logic based on which you want to update the node status, you can do that in two ways. Uh, the first is the node problem detector. So, this is a project under Kubernetes 6 that has existed even before we, uh, created the node readiness controller, and this can be

used to write custom logic to update the status of nodes as uh, ready, which the node readiness controller would pick up and, uh, lift the taint. We also have a very lightweight binary that we have, uh, shipped along with the node readiness controller called the readiness condition reporter, which does the same thing based on custom health checks and all that you can configure. So, these are the

two different ways with which you can, uh, update the status of nodes. And when you set these up, you have to have, uh, cluster roles that would let these, uh, node problem detector or node readiness uh, deployments update the status of nodes. So, when you create a cluster role, by default, what happens is these deployments gets privileged to update the status of all the nodes in your

cluster. So, if a node is compromised, that can mean that the the reporter component can be used to update the node status of all the nodes in your cluster, and that might not be ideal if a malicious agent gets like unauthorized access to the cluster or something like that. So, we are using Sig Auth kept 548 5284 called constrained impersonation. So, this kept is in beta in

the 1.16 release, and it helps us mitigate this issue. So, we are going to look at how that works. So, this is a regular old cluster role where you like grant access service account that is used by the node problem detector or the node readiness reporter to update or patch the node status. And this would let the service account update the node status across all the nodes.

But, with the new constrained impersonation kept, you can create a cluster role which has a constrained uh access on the nodes. So, the reporter component is only able to update the status of the node that it is running on. So, with this uh it can only like even if the component gets compromised, the blast radius is very less. So, if you create a with Kubernetes 1.16 where

the constrained impersonation feature is enabled by default, you can create a service account like this with the constrained uh impersonation as we saw in the slide, and you can see that the node does not have access to uh sorry, the service account does not have access to update So, this is something that we have uh we are making use of to make uh the node readiness controller

more secure and uh production ready. So, now let's look at all the releases that we've had so far. So, we've had three minor uh releases so far. In the first release, we implemented all of the core APIs. And uh later, we added all observability and all the optimization that uh Priyanka and uh Karthik just talked about. And in the latest uh V0.3.0 release, we have added all

of the security improvements. And we are at the place where uh the node readiness controller would work with all the existing Kubernetes clusters without any dependency on any other component. And this is the uh latest version which we are confident that is a production ready and can be uh started using by uh cluster administrators. And uh yeah, we would like your feedback on this if you uh

want to try this out on your cluster. And yeah, we are already uh like speaking with folks and uh the V3.0 like 0.3.0 version is planned to be used in GKE soon. Yeah, now I'll hand it back to Priyanka. Um with that, just um I want to bring all what we discussed today in the presentation um in one slide here. This is all what we discussed and

part of what is on our agenda uh our road map for uh the future uh coming months. So, the first box, I think we already have hit the mutating admission policy part. And cluster auto scaler integration, we are also looking to uh see if we can include headlamp UI in our project for better um user in ex- experience. Uh we are also working on creating helm charts.

Uh that would be another way to deploy the node Uh we already have many examples in the node readiness controller repository and our website to uh uh, to explain how to use NPD in many scenarios. So, that is also covered and I think partly what uh, Karthik mentioned, we are working on working on making our APIs more efficient. So, the part where we want to uh, split

the node readiness rule and node evaluation API into two different CRDs to lower the uh, uh, pressure we have on net city. Um, yes, and and then finally the security stuff that just Sriram talked about is the bottom box. So, this is what we are currently working on or we are planning to work on in the coming Um, I just want to say these are the key

takeaways uh, from this presentation that uh, Kubernetes not bootstrap definitely involve a lot of initialization um, of dependencies before we can say the node is actually ready to deploy our workload. And node readiness controller today is provides a solution to that deterministic readiness gating problem. Um, it does integrate well with other components and we are working with more and more SIGs within the Kubernetes project to make

sure if we are missing out on anything we we account for those as well. And uh, with that I'll also hand over to Sriram to talk about how to get involved in the Yeah, so uh, this is everything that we've done so far, but the work is not done and we'd like your help. So, if you're interested in getting involved in the node readiness controller project, we

have a couple of resources that we'd like to share. Uh, first of all is the project repository which comes under the Kubernetes SIGs organization. Uh, the repo is called node readiness controller where you can find all the source code and all the examples and the documentation. We also have a Slack channel in the Kubernetes Slack workspace called uh, SIG node readiness controller where we uh, discuss all

of the work and uh, we also have a bi-weekly meeting on every Tuesday. Uh, You can find the details of the meeting and the invites in the Slack channel. And finally, we also have our project website where you can find all the docs and all the examples that we just talked about. The website is at node-readiness-controller.6.k8s.io. And we'd like to hear your feedback. If you are somebody

who is interested to contribute to the project or if you're using the node readiness controller or if you have any uh feature requests or any sort of feedback, we'd like to have you in the meetings and we'd like to hear from you. We also wanted to give a shout-out to all the contributors who have helped us ship the node readiness controller so far and we are expecting

to see your faces in these in the slide in the in the future talks. And yeah, with that we are at the end of the talk and we can take questions. I just want to repeat again if if anybody among the audience who is a component owner, we definitely want your your input to make node readiness controller more and more better because this is to be used

by the component owner. So if you have any specific requirements that you see node readiness controller to be providing, please please provide us feedback. We are more than happy to work with the component owners to make that happen. Yeah, thank