In-place Updates with Cluster API: The Sweet Spot Between Immu... Fabrizio Pandini & Stefan Büringer

This talk explores the complexities and advancements of the Cluster API, particularly focusing on in-place updates for Kubernetes clusters. The speakers, Fabritz Pandini and Stefan Ginger, delve into the principles of immutability and how they impact Kubernetes operations, specifically the creation and deletion of pods and machines. They discuss the challenges presented by immutability and the solutions introduced to improve rollouts without causing disruption to workloads. The introduction of the update extension component allows for in-place updates, enabling smoother transitions between configurations. The session emphasizes the importance of understanding desired states and operational constraints in optimizing cluster management, presenting practical strategies to implement in-place updates effectively.

Welcome everyone. Thank you for being here at the talk about cluster API and in place update. It is overwhelming to see so many people for a project like Luxer Abi. We are not talking about AI in this talk. So okay next >> so I'm Fabritz Pandini I'm principal engineer at Broadcom and also cluster API maintainer. >> I'm Stefan Ginger also principal engineer at Broadcom cluster API and

also controller runtime maintainer. Okay, so in place updating cluster API for people that knows the project you should probably be aware that this has been a journey. This is probably the most complex change that we merged in last few years. It took really a while for the community to figure it out how to make it right. So in order to understand why this was so difficult from

a technical standpoint, let's start setting a little bit of context. So immutability, let's talk about immutability in Kubernetes. When you create a pod, what happens is that a container, a piece of infrastructure gets created on your machine. Apply pod create. Okay. Now what's happen when you edit your pod is that Kubernetes you edit your YAML but Kubernetes is not going to change in place your pod. Kubernetes

is going to deletate the old pod and create a new one. So in in Kubernetes there is no in place updates. There is always create and delete. And cluster API is a project that treats machine like a mutable pod. Okay. So when you create a cluster, you create a new machine. When you scale up, you create. When you scale down, you delay. When you roll out changes,

you first create and then delay. Okay. Why mutability is important? it and because it is the one of the key of the success of Kubernetes and cluster API it is because system based on on immutability have some nice properties and I let me mention mention two of them. So system based on mutability are simple. They are based only on two primitives. They are not complex to implement.

And the fact that that they are simple, it makes those system more robust, less errorprone, less racy when things happen. The second properties that I would like to highlight is that system based on immutability are also more predictable are more reliable because whenever you create a new machine or a new pod you create starting from a trusted image. You create a pod you start from the uh

docker image. You create a machine you start from your machine image. So what does it means? It means that there is no configuration drift. All your pod, all your machine looks the same. And this makes your life when you have to do troubleshooting uh way simpler. Okay. Um but unfortunately the reality is more complex. Um there can be friction at infrastructure level. So for example, it might

take some time simply for VMs to be created. uh you when you when you roll an entire cluster um you might not be able to create like just a bunch of new machines because you maybe don't have the resources for it. Maybe you're running out of GPUs. Who knows? Um there can be also friction at application level um because not every application is designed to handle drains

or like single node um outages basically the node goes away in a nice way because maybe you only have one replica. You shouldn't really but maybe you only have one replica of your application that goes down during a roll out or maybe you have long running jobs and you have to wait for them to complete before you can move ahead with your not train or maybe have

database and just take some time for new replicas to come up because they have to sync data. So one more so is improving uh they recently added a feature like in place port resize but what is cluster API doing? So as we heard cluster API is immutable by design. Uh in the last few years we made significant improvements to mutable rollouts. So for example we now avoiding

unnecessary rollouts if we can. So it is possible to update labels annotation stains for example on nodes without replacing machines. We also reducing um pot shing rollouts. So if you roll through entire cluster it might happen that pots are getting drained and evicted multiple times. But what we're doing is we put taint on old nodes. So ideally they only get um trained once throughout the entire roll.

What we also implemented in 1.12 in our last release is that when we up the cluster by multiple versions um in some cases we're skipping over worker upgrades so that your workload is not getting sorry it's not getting disrupted as much. So but now why are we here? So the last uh major step to improve the whole story here is that we implement in place updates so

that you can update machines in place and that's what we're talking about. Okay. So we now understood why the project the cluster API project uh introduced in place. It is because we want to be more gentle with the uh with your workloads. We want to treat them nicely. We want to avoid the disruption as much as possible and also we want to make cluster PI to work

uh better on every type of infrastructure. So let's now move on and start talking about how we implemented uh in place and and a good way to understand this is to start talking about a few design principle that that were implemented in the proposal. So the first one is that everything start from the desired state. If you think about how you work with cluster pi today, you

define your cluster, you define your yaml, you define your cluster spec and that's all the user has to do because then it is up to the system to make sure that the desired state gets uh uh gets implemented in your infrastructure and this must be this must remain the same also within place. What is the only difference is that now we are introducing a new component which

that we call update extension and so the process will be exactly the same no change for user zero nada. So you apply your YAML cluster API will take over and and make the best choice to make it possible to that your state uh we will be implemented in infrastructure by using or immutable or out or update extension depending on the case. This is also interesting because this

allow everyone to start using in place without changing or eventually disable if they are not not happy. So let's keep moving. So what is what how this choice how does C API decide when to use in place or not? Again what is driving the decision is your spec. So like in uh Kubernetes also in clusti uh we have a couple of fields in in uh in the

specs that allows you to define some rule some constraints that the system has to respect while while it creates or deletes machine. So there is of course the number of replica then there is max surge which is the maximum number of additional machine. So how much infrastructure you can add while performing an action and also the risk which is the most important for this for today discussion

max unavailable which is how much disruption you allow in the system while performing any life cycle operation. Okay. Why why this is really important for in place because the system cluster API assumes that in place update might lead to machine unavailability the operation can fail. So better safety sorry and so cluster API before choosing which one of the option to to use it make sure I I

have enough buffer a viability buffer in my system to introduce a potential dis disruption. If yes, if you have budget, cluster API consider using uh in place uh updates. Otherwise, cluster API fallbacks to performing uh immutable rollouts and scale up and so you increase the viability in your system. Next up, what happen when class API have enough budget to consider doing in place? There is one one

additional step that gets performed. So let's let's explain this by uh using an example. In this example, I'm changing the desired spec of my machine by setting an SSH key SSH key. I want to change how the user login in my machine. But also I'm changing the kernel version of my operative system. So I'm performing two operation in one single change. Okay. What the cluster API does

before considering in place is that a cluster API contact the the update extension and ask hey are you capable to handle both changes if the answer is no it mean that c the update extension is is not the right tool for the job we don't want to do in place and then are allowed to change the kernel version this will be additional disruption for our code we

don't want it. Okay. So in case our update extension does not support all the change being performed, we fall back on roll out and uh please note this is not a constraint or a limitation. This is a feature is intentionally designed in the system because update extension is a new component in the cluster API ecosystem and we assume that it will start small. you will start uh

implementing only a few set of change at at the beginning and over time it will improve and by this design decision we make it possible that uh the cluster API adapts to evolving uh update extension over time without changes. Okay. And the last principle of the implementation. So we are finally at the point that we know we have enough bud disruption budget in the system. The update

extension is the right tool for the job. Finally we engage the name stio. The update extension and update extension take care of performing uh changes on the machine. One key point is to remember about this phase. There is you no human intervention. There must be no human intervention because your machine must move from a well-known state to another well-known state. We don't want to introduce drift in

our system because drift is a problem for operation. So in data update expansion must perform carefully validated and repeatable changes when you implement them. This is this is what you have to keep in mind. uh apart this cluster PI you know cluster PI is a very extensible uh project so we are we the project the core cluster PI project doesn't have opinion on how in place update

are performed you can use SSH command you can run privileged pod with uh that that uh modify your machine you can do AB uh deployment of you can run machine aent everything works for clust API I is good enough. >> Okay. So now let's take a look at how this works uh completely in practice for control plane machines. Um the following is based on cubin control plan

just for people familiar with cluster API. Okay. So first let's take a look at how immutable control plots work today. So in our example we have a control plan with three replicas. Um as explained before we have max search max unavailable in that scenario max search is one and that effectively means that before the roll out we always have to keep three machines available and we basically

have a buffer of creating one additional machine before the roll out we have three machines and then basically yeah we have three machines we need three machines so only thing we can do is we create an additional machine because we cannot go down to two. So first thing we do, we create one additional machine. Then we delete. So then we add four. Then we delete one machine.

We're back down to three. And basically this just repeats until we're done um with the entire roll out. So just delete, create, delete, create, etc. And that's it. So um the alternative configuration is to use max search zero and maximum one. Maybe one important point to mention here is for control planes, we only work machine by machine. So we never try to create like three additional control

plane machines or something like that. uh just because we want to be pretty like safe and robust with our control plane of course. So if you have max zero max unavailable one that means we only have to keep two machines available. So we start with three machines. So in that case we can basically start with deleting one machine because we have as mentioned we have an availability

buffer of one machine. So we delete one machine we go down to two then we create an additional machine we delete another machine create another machine etc etc. So basically the same thing but just uh instead of going up and down between three and four replicas we go up and down between two and three replicas because the will developer allows us to do that. So what about

in place updates? So first of all it's important to note um in which cases we can do an in place update. What is super important is that we have more than one replica because if you would have only one replica and we try to do an imp place up on that one replica and there's actually anything going wrong. we have sealed replicas and we don't have a

control plane anymore. That wouldn't be very great. Um and on top of that in general that's also just an integrant in your system. The control plane has to be healthy because we don't want to start messing with unhealthy control planes. Okay. So sort of same example again we start with max search one and again this means we have to keep free available So if you would now

do an in place update on one of the free replicas, we would go to two available sorry to two available replicas and we would violate basically the rules that that the user gave us. So instead of starting with an in place update, we creating one additional machine with that one machine, we are now at four available replicas and we can start with in place update. So the

second machine gets in place updated. Once that's completed, we in place update the third machine and now we at three machines that are all at the target state and available. So the fourth machine won't be in place updated. We just delete it. So overall we create one additional replicas. We do two in place updates and then we delete the the one additional machine. So again the alternative

scenario we use max z max zero max unable one. So in that scenario we only need uh two available replicas. So what we do in that case is basically we can immediately start with an in place update because we have one buffer. So we impless update the first machine. Once we're done, we in place update the second machine and then we in place update the third machine.

So in that case we we are only doing in place Yeah, that's it for controlling Thank you. It looks simple. It is not. Okay. So >> yeah, it was not easy to implement. >> It's not so easy to implement. Okay. So we talked about control plane but what we really care about is worker machine. So the machine that are running your workload. So let's start talking about

how we do in place for worker machines. So as you know in cluster machine we use machine deployment which are more or less like deployments in Kubernetes. And so like deployments in Kubernetes have a replica set in cluster ABI we have machine set. And as you may might imagine, the first step of any roll out is that the system create a new machine set with a new

spec. Okay. And the and the goal of the roll out is to get rid of the machine of the old machine and and uh create new machine uh on on the right. So as you know now there are two parameter that drives this operation that makes this operation happen. So max availability and max surge. However we have there is a there is a difference. So while in

control plane uh max surge and max unable can be only zero or one and only one of them can be set because on the control plane we want to go slow on machine deployment both of them can be set. So both of them can be greater than zero at the same time and not only they can also be greater than one. If you look at this from

a from another angle, it is like that max variable define the speed at which you are emptying the old machine or getting rid of the old machine and max surge define the speed at which you are filling up the new machine set. Okay, with this in mind, we go back to our example just to make a parallel with the control plane. So we we we talk about

max unavailable zero and and max surge one and uh replica tree and we look at how immutable roll out work. So before the introduction of in place so I think that you you you know it now three replicas max unavailable zero what does it means that we don't have budget for the disruption we cannot introduce the disruption at this point so what is the only option for

cluster API create a new machine okay as soon as the new machine gets ready we now have a budget for Okay. And since we are doing a mutable around it in this case, disruption means delayed. So we start deleting the first machine on the old machine set and so on. As soon as the old machine goes away again, we don't have budget only option scale up. We

create a new machine on on the new machine set and so on and so forth until all all the machine are on the new machine set. Okay. And this should be nothing new should be you should be used to to this. Now let's look at how in place fits into this picture. So same example replicas three max uh max surge one max unavailable zero. Okay we have

machine one two three. What we can do in this situation? What class API can do? Max unavailable zero. No room for for disruption. the only option scale up and we scale up we create machine four on the new machine set exactly like before. Okay, now we finally have room for disruption in the system because we have four and the replica number is three. So we can start

considering in place. Okay, now please pay attention because what is happening is a little bit more complex than starting in place because what happen is is that we start before starting in place we take an existing machine and we move to the new machine set. This this means that we are changing on a ref we are changing the spec and then when the machine is aligned to

what the new machine set define we finally start updating in place. Okay. So it is not only in place but is in place plus move because we have to be conformant with the concept of replica set or machine set as as you wish. So at this point machine one is is updating in place. As soon as it's finished we are now back to aviability 4. That mean

that we are room for another for introducing disruption. So we can start in place updating machine two. Okay. Again not only in place but move plus in place in this case. Finally, we have three replicas at the new state. What we can do with the last one? We got a read. We delete. So when maxability is zero, what the system does? It creates one additional replica. So

it creates room for disruption and then it start start doing in place in a in a safe way and okay so second example let us let's let's assume that now we have maxavability one so does mean that you the user allowed the cluster API to introduce disruption since the beginning of the rollout So what is what is what happened is that we start immediately using in place

move in place immediately as a first action because it is allowed by the user and and the process keep go keep going on. So as soon the first machine is is update is completed we start the second one the system wait that the update completes on the second one and finally the third one. So what happened here is that when you have max anability greater than zero

the system is allowed to introduce disruption. So is allowed to use uh in place since the beginning. And one things that is maybe interesting to notice is even if max surge is greater the system does not use it is not required to use it because when it is possible to use in place the system prefers to use in place to perform the change because this is what

the the user asked for. >> Okay. So we still have a little bit more time. So we have one bonus slide here. Um one feature I briefly mentioned before um was uh skip worker rollouts when upgrading multiple miners which was actually like I would say the the second major feature that we did in in the last release um in December. Um and I would say it's even

basically two features in one a little bit. So in the past in cluster API you could only um upgrade the cluster like in one go basically by one Kubernetes minor version. And what we implemented is something we call chained upgrades which allows you to to go up by multiple versions as a user like you change a cluster version by multiple versions just once and then the last

API takes care of uh takes care um to to basically upgrade a control and worker machines accordingly to get to that target state. So in this example we assume that uh we have 131 as a initial version and a user bumps the version to 134 and what cluster API will do is that basically first we upgrade the control machine then the worker machines control plane worker etc

etc until the entire cluster is at 134. So we go through a bunch of upgrade steps but cluster API will take care of going through all of that and reaching the desired state. Um the the second feature here is that um we realized we can optimize this because um it's not really necessary to to go through all the upgrade steps with worker machines. So um we call

this efficient upgrades which basically chain upgrades just more efficient and I guess less disruptive less friction. So in that case uh we are first upgrading only the control plane and only when necessary we upgrade the working machines. So what you see here is that we basically reach the maximum of the kubernetes policy. So n minus 3 we can only be three versions behind. So control is at

134 because we reach that support skew. Uh now we have to upgrade the worker machines but we only have to upgrade them once and that basically means um 66% less work upgrades. So and hopefully a lot less disruption threat. Okay so sort of pivoting back. Uh so wrapping up um mutability is and always will be the core of cluster API. I think it's still like the the

the most robust and safe way to do an upgrade. Uh we will continue to improve it, make it more efficient, more robust, faster, all of this. But uh now since the last version, you can also do in place updates to existing machines whenever it fits your use case basically. And if you want to start using it, um you can basically imple implement your update extension. We also

added a link to the slides uh where you can find our documentation on how to actually implement one. Um yeah. Okay. Um as a as a last slide we would like to give you some recommendation some recommendation for people that are looking to start with uh in place. So the first recommendation is that even if we we went a lot we tried a lot to help people

to run u workload on cluster BI make them uh make cluster PI less disruptive. You should not use in place up update as an excuse to avoid solving foundational problem on your system. So if you have application that do not support nice nicely drain you should look into it and try to fix it because drain is a foundational primitive in Kubernetes. If you try to avoid drain

is sort that you is sort of like that you are trying to fight Kubernetes and this will not end well for you. Kubernetes will win. So same more or less the same apply for infrastructure. So if you have infrastructure that that is low or you have some problem on your on uh provisioning your machine, you have to look into it and try to optimize it because uh

this Kubernetes is not a mainframe. You cannot avoid the fact that sooner or later you have to create new machine, delete old one. So you should embrace disruption, not fight it. It is at the core of cloud native. So we help you but take take the step to make yourself more comfortable with cloud native technologies. And the last tips is about in place updates. So from our

point of view in place updates are best suited for rolling out change that do not require pod uh machine drain and pod restart. Going back to the example before, it is okay to roll out an SSH a new SSH key is less okay to try to update the kernel because you have to update when you update the kernel you have to drain your pod and so you

if you drain the pod just create a new machine. So you get a new pristine machine equal to your template. you you reset your system. No drift, no risk of drift, no risk of security all it is better for you and it makes your system your cluster to to live longer. And uh with that we thank you very much.