Do You Trust Your PodDisruptionBudgets? You Shouldn’t! - Kārlis Akots Gribulis, Saxo Bank

In this talk, Karlis Sakoss Gribulis discusses pod disruption budgets (PDBs) within a Kubernetes environment, drawing on his experience at Saxo Bank. He describes how he encountered issues with PDBs when performing routine updates, leading to unexpected downtime despite their intended purpose of maintaining application stability. The speaker explains what PDBs are, including their limitations and how they specifically address voluntary disruptions while being ineffective against involuntary ones. He covers scenarios like hardware failures and network issues that Kubernetes cannot prevent, and emphasizes the importance of understanding the nuances of PDB configurations. Through examples and a live demo, Gribulis illustrates how PDBs interact with delete and eviction APIs, and highlights the significance of resource requests and update strategies in mitigating downtime during maintenance activities.

It's nice to see so many people interested about pod disruption budgets, although I thought everyone knew about that. But let's start. About me, I'm Karlis Sakoss Gribulis. I'm from Riga, Latvia, but currently I'm working for Saxo Bank. So I'm living in Denmark. I've previously built Kubernetes clusters for companies such as Accenture and Velux, and I sometimes get bored and collect too many certificates. So quickly about Saxo

Bank, just so you have some context. The company was founded in 1992. It has a 100 billion USD client assets. 1 and 1/2 million clients, 70,000 financial instruments, and it's been designated a SIFI bank, which means some say it's too big to fail. From my point of view, it's I should not fail, otherwise I'm in big trouble. So as a container platform engineer, I I do containers,

I do updates, and as many of you probably make some platform changes, you bump some resources, and some of you have maybe the assumed safety of PDB. That's something that I had before. But then on the regular platform maintenance evening, you know, I'm just doing my bumps, I'm updating ingress, updating other controllers, and then suddenly I get a call from a manager 10 minutes later asking, "Hey,

there's no metrics anymore. What did you do?" I'm like, "I I didn't even touch Prometheus." And I see the pod is down. There's no metrics coming in. So the ingress kind of kicked it out, and then we started to figure out, "Okay, what's going on? Why did it happen? How can we prevent it in the future?" And so on. And at that point PDBs felt like a

nice Swiss cheese. So it's, you know, it's nice, but it has a lot of holes. So many things can go through it, and you have to be aware of all the rules within it. So for those who don't know, as this is a cloud native novice track, what is a PDB? If you would ask you know, our lovely AIs, this is what it would generate for you.

Which, you know, kind of highlights it mostly what it is. Yeah, Protects pods, it has only two settings, which is minimum available or maximum unavailable. Although, I only hope it actually meant that you can select one of those two, not both, because that's You cannot select both at the same time. The PDB is invalid then. And uh yeah, it's meant to keep your applications more stable and

keep them more uh highly available in your environment in a disruptive environment, because Kubernetes by default is a disruptive environment. this is a line I also got from the Kubernetes documentations. I also highlighted it, because we'll come back to it a bit later, but it's meant to prevent simultaneously having too many pods down during uh voluntary disruptions. And that's a important term to uh keep in mind.

And uh when does it apply? Uh it applies on eviction API, on on those calls only, not the delete API. Some of you maybe have seen yourself if you just delete a pod and there's a PDB preventing it, it won't prevent that. So, API delete calls bypass, they don't care. Uh only eviction. And eviction API is very kind of very niche in itself. When is it totally

not respected? It's in involuntary disruption. That's something that basically Kubernetes is not aware of. It it it cannot control that. It uh cannot mitigate that. It cannot yeah, see it in most cases. So, for example, hardware failure, you know, if your CPU is fried or your disk just smashes in half, yeah, there's not much you can do and the pods will eventually uh die. If you have

maybe hypervisor failures, either be it on your Windows machines or other platforms like VM or so on, if there's error on that side, it will also the node and the hypervisor will have issues and the Kubernetes cannot prevent issues uh on that level. Similarly, if a cluster admin who has too much power suddenly deletes a node, the pods will be gone. I'm not saying I have ever

done that myself, but it could happen theoretically. Uh and then a a kernel panic, meaning if there's maybe some mount issues or any other issues that kernel might have, then suddenly it might crash and and not be responsive and then and things might Uh on a similar note, it's uh if there's a network issues like a network partition, for example, if suddenly you have uh nodes in

multiple different zones uh and then suddenly one zone is not reachable. For example, in in Saxo Bank we have these monthly continuity tests where we basically uh network-wise disconnect one data center out and just see what happens. Everything should still work. And in those scenarios we can see that, you know, all the pods are unreachable and then by default Kubernetes could only within 5 minutes actually move

them over and recognize them as as that state. And then if there's any resource issues on the node, be it too many processes running on that host or uh or there's not enough storage or memory, those are called these uh taint-based evictions and that's something uh some of them you can actually control yourself at what point would the scheduler ignore it and and still reschedule it if

it's on a node that has that uh state. And then this is not a separate category, but I still added it because that's something we use in in our bank and I know many other companies use as well. It's something spot nodes. It's kind of combination of previous issues. You can save a lot of money with them in in cloud. And uh but your nodes are are

never your VMs are never guaranteed. You might have it now, but the 1 minute later it's out and then your PDBs will not protect against uh such scenarios. And uh one thing to keep in mind, even though involuntary disruptions can happen, uh they still count against the PDB, so it could go, you know, kind of quote-unquote in depth if there's too many pods out, meaning if there

was supposed to be a voluntary disruption, it would Kubernetes would try its best to not to go further in-depth if it can. And when it is, well, should be respected because that's also what I highlight. There's this uh uh an interesting few points that how Kubernetes look at voluntary As we saw previously, it mentioned that PDB should protect against voluntary disruptions. But voluntary disruption is counted only

when a service account or a user makes that call. Uh and then from application owner's point of view, it would be either you update an image or update increase resource requests and so on. Or if you delete the deployment itself. Or if you delete the controller that manages the pods. Or deleting directly the pod. But this is where it kind of goes against itself. It is a

voluntary disruption, but we previously checked if it's a delete call, it actually ignores the PDB. So, in most cases, it should respect the PDB if it can, but there are many holes even in in that point of view. Uh from cluster admin's point of view, it's for example, you want to update the node or you know, if it has an issue or if you want to do

a Kubernetes version update. Or just delete it and scale down with cluster auto scaler, those are counted as And similarly, if a pod needs to be moved away from the node to make some space, something as I mentioned at the start, something that happened to us a space was made for ingress pod in favor of a Prometheus pod. So, that's something Kubernetes can also do. And that's

still technically counts as voluntary disruption because you know the source that made that call. You know who wants that. Um yeah, for PDBs, even though it's nice in these few cases, we've also seen in our experience where PDBs are not always so good to you, especially if you have PDBs like that. For example, max unavailable zero or minimum available of 100%. I mean, you might have developers

that say, "Yeah, of course, I want 100% uptime. Why not? It's amazing." But in these scenarios, you would have your cluster, for example, stuck in an update cycle. In In most likely, you will see during a a drain phase of a node that it'll time out and and then would not be able to proceed further. So, be careful of such developers or even controllers. I know there's

some projects upstream that also sometimes have PDBs by default that are set like that, and they rely on the controller or the operator being kind of smart enough to figure out, "Hey, a pod needs to be deleted. I'll delete it directly or I'll edit the PDB myself." and so on. Uh but if you make it yourself, also keep in mind if you want to make PDBs like

that that you control it properly. And And if you face these scenarios where PDBs block you, there are ways to uh ignore it. For those who don't know, multiple options, but I'll highlight highlight three of them. One of them is disable eviction. This is for the kubectl uh drain command. There's a flag for disable eviction, meaning it does not go over the eviction API. It goes directly

through delete API, so there's no PDB protection. If there's force flag, which basically you get rid of all the pods that are unmanaged. and grace period, meaning you can set it to a X amount of seconds, and if the hasn't been successfully finished within that timeframe, Kubernetes will ignore the PDBs and will delete the pods directly because that's you wait for that amount of time before you

you stop. Uh one important thing to mention, one one things that also ignores PDBs, I'll show it in a demo a bit later on, but it's a pod priorities. Uh because there's something's called priority classes in Kubernetes. Uh and they indicate, you know, the relative importance of one pod compared to another pod. You can set it up yourself anywhere, you know, there's a big number range anywhere

between the pinkish and the purplish colors. You can set it yourself, but on the blue side, that's something that Kubernetes has reserved for itself. You cannot do that. That's mostly for system critical stuff, for example, API server, scheduler, or etcd and so on. But you can set up the priorities yourself. Uh and then in scenarios where a pod wants to be scheduled and it has a higher

priority than a pod that has taken the last space on the node, it can preempt it, meaning it can make it be terminated to make up the space. And if there is no priority specified, which is the default, if you don't specify, then it's by default zero. So, all pods that don't have it specified are counted as equals inside the cluster. And you might have scenarios where

you uh for some reason, for example, uh ignore it this preemption behavior. You can set it on priority classes specifically to do it for all the class in general, or you could do it for one individual pod as well if you want it to have specific priority, but still not have the power to preempt other pods, you can individually uh set that setting. let's see if I

can show some of these for you in my demo. Let's hope the demo gods are on my side. So, I have a local kind cluster. We have multiple deployment set here on the left the naming kind of suggests what they have. Either they have no priority, no PDBs, or low priority, or high priority So, for the setup uh I have these uh PDBs. All of them by

default have minimum available of 25, and then the low priority has 30, which is against which we will do couple of tests to see how well it behaves. And then we aside from the default two priority classes, we have uh these uh high and low priority that I set myself. That the values don't matter too much as long as one is, you know, higher than the other.

So, if we look at the preemption that one that I mentioned just now, let's see how it would work. So, for example, if we scale uh the no priority no pod disruption budget deployment to 100, then it should hopefully scale to uh around 62. There's just no more space in my nodes. Uh so, that's the the cap it will have. But then if we scale uh low

priority with no pod disruption budget to 30, if you would look at these numbers here, it went from 62 to 57. So, it kind of took out the no priority ones in favor of this low priority one because it has a if you compare them, it has a higher priority, so it killed the previous ones. the same would happen if we do the same with the low

priority and it has a pod disruption budget. It will again remove some of these in the no priority list and just schedule uh these in favor. So, and then let's go through the examples which counts as a as a voluntary disruption, the delete Uh let's see how it that performs. I slightly cheated in here. I have a a file that has a random pod name from the

current ones that I will uh try to uh delete, and that would be over in here. You would see it went for a moment to 29. It has no PDB, so of course it can be deleted. But if we do the same for a deployment that has a pod disruption budget of minimum 30 and it only has 30 pods, the expectation would be that we can delete

it and it did because as as I mentioned before, the delete calls ignore that. But if we look at the eviction calls instead, and I have another random pod. I have made a kubectl proxy command, that's why it errors here, so you can ignore that, but I already had it open. But eviction, it's a it's an actual resource in JSON file that you could apply with a

curl command against your API server. You would need to make sure it has the pod name and the correct name space and then you could for the sake of the demonstration, you run this command against the API server. And it has, yeah, no pod disruption budget and if you run the full command, you would get this response from the API server. It was a bit too fast,

the number didn't go down here, but when I did it before, you could see the number go down, but based on this response, you can also see it could evict the pod, so it was deleted and then it got scheduled back up. We got a 201 response. And if we do the same against a pod that has a pod disruption budget of 30, then the Yeah, some

of you can use couple of seconds to think, would it respect that or not respect that? And then once it succeeds, we can see what sort of response the API server gives, which is, yeah, a status of failure as as we would expect that because it has a PDB and it has a minimum of it even says that in the message, it needs 30 healthy pods to

be available. And it has currently 30 here, so it just gives you 429. Just try again later, maybe new pods will be scheduled back up and so on. what would happen if we restart? It's not technically uh a delete API call and it's not technically a uh eviction API call. if we try to restart uh the deployment that has low priority and no pod disruption budget, you

could monitor this number in here. It went down to 23. It's slowly rolling out and restarting. And uh the question is what would happen pod disruption budget uh with a deployment that has it? Maybe. Who think that PDB will will be respected on a restart scenario? If there's any people, maybe you can raise your hands. Okay, there is there's a few. Let's see. And if we yeah,

run the same command against that, and if you monitor the number in here, yeah, it still goes down. Uh so, pod disruption budget on restarts or if you do an image bump or resource request increase and so on, uh pod disruption budgets are completely ignored. And the reason why it's ignored is because of something called update strategy. Uh you could see it in your deployments or demon

sets. Uh that's a special field that you can control. And then you can set up how many percentages of pods, for example, can be uh rolled out at the same time, what's the max search, and so on. And then the you know, the question is what's the main purpose of PDBs if delete cannot help uh you know, if delete can ignore it or restart can The place

where it actually plays a role is on node drain. That's the the main place where it can protect you. And then if we do the same scenario here for a node drain, we drain one one of the two nodes. Uh you among these messages, you would see that it's evicting pods. And then uh you can also see among them some error messages saying, "Oh yeah, it cannot

evict that uh because of the PDB preventing that." And you could see that this number didn't go down, but the no priority ones and no PDBs did went down because then low prior no PDBs got rescheduled to a second one and then the no prior no PDB pods got taken out in favor to give it more space. And if we do the same command but we use

one of the flags that I mentioned before, which is the disable eviction, the behavior is then different. You could see all of them get deleted directly because yeah, it then bypasses the eviction API and directly deletes it and then the PDB is no longer respected. So, let's then uncordon it so there's more space. Uh one thing for those who've actually read my description and that was my

assumption initially as well, the quality of service, you know, does that play any role in respecting or ignoring pod disruption budget? Um so quality of service basically, you know, in the name it more or less describes uh the service level of one pod compared to another. There's three levels of it. There's a best effort, burstable, and guaranteed. The difference between them is how you request your resources.

Best effort meaning you don't request any resources. You just go by default how it Burstable is if you if they what you request does not match your limits. And guaranteed is if if your requests match your limits and that's across all the containers. If you have one container where the requests don't match the limits, then the quality of service will no longer be guaranteed but it will

be burstable. So, that's also something you have to keep in mind if you have pods with uh multiple containers. So, if we scale these ones, they have a a higher priority, even higher than before. So, they would kick out and if you look here at the bottom, the no prior no PD uh it's getting kicked out first. And while it's slowly scaling up, I can start the

next command. It would be uh with a different quality of service. And then you can just see PDBs in action. So the ones that had no prior or no PDBs was out first. Then compared between low prior and PDB and low prior no PDB, the second one was out first and then it did go in depth of pod disruption budget. So you can see here if you

have multiple priority classes your pod disruption budget can also be ignored in that scenario if if a pod with a higher priority comes in. So it's treated as a higher tier citizen. And then if you scale the last one it should then get rid of the last few pods here at the bottom. It can only fit 16 of them. There's no more space on my machine. But

something that I didn't show in here but something that I did then locally to see if that plays a role I can also kind of spoil it here. Quality of service doesn't play a role in scheduling itself. It plays a role during node resource constraints. So if suddenly node is under memory pressure it would take into account priority class and and then also quality of services if

the priority class is equal the one with higher quality of service would be left inside the node in order to have the last resources shared there. if we go forward from this knowledge that we had in the small demo that I showed different places where it PDBs are respected or not respected then the question is you know there's many places where PDBs are not respected and then

the question is you know why do you need it and how would you deal with it? There's multiple tools in Kubernetes you can use for For example one of the suggestions would you can make sure you request resource you know request that that the most important part because that's the only way under normal circumstances, that's the only way you can guarantee you'll get this amount of resources

if you request that because limit is only best effort in most cases. Only if node has enough free resources and so on. So, on the basic level request is a one way to do it. So, for your day-to-day workload day-to-day loads, you need to tune your requests. You can use pod replication not only just increase the replication count to 50. That might not always help especially if

suddenly all of those 50 pods are on the same node on the same either node or zone. If that is out then suddenly your application is out. So, you can combine this with other fields inside your deployments. topology spread constraint or pod anti-affinity or node affinity meaning make sure the pod the pods are on maybe specific nodes in a specific zones if you want to or make

sure there's not too many of the same pods in the same zone or or node depending on how you want to make sure that it's spread out. And then of course the update strategy because that also you if you have a PDB that and your expectation is that no more than one pod gets down during updates but you suddenly have an update strategy that oh yeah, just

100% can be killed then that also you might be surprised suddenly seeing your application having downtime. So, all of these things can and should be tweaked. The last not the last thing but one of the extra things to keep in mind as I mentioned before is priority classes by default puts you as a higher tier citizen inside the cluster. So, you're already treated better compared to other

pods inside there. Quality of service as I mentioned would only help you during when it's yeah, node resource constraints meaning yeah, there's only no memory then uh quality of service would win if priority class is equal. And then of course still you can use pod disruption budgets that they still work even though maybe it has a few uh tweaks that you didn't expect. And then uh one

of the things you could ask, you know, should you even trust them knowing it's a a nice Swiss cheese, you know, like like with the cheese it's it's nice and good. But uh blindly of course you should not trust it. Um you need to be aware of as I tried to highlight most of the things in here. You need to be aware of all the things that

are taken into account in different scenarios. Uh so you are not surprised like me when I got call from my manager suddenly saying, "Hey, we're blind." Uh well what can you do? If you want to learn a bit more about how, you know, more in-depth, more technical details, how prioritized workloads are being treated or how are they being even calculated inside the the the Kubernetes or inside

the Linux itself, there's a great talk from Alan from KCD Denmark. He goes more in details how CPU weights play a role in that and how you can compare them and So I I do highly uh suggest that. yeah, that's it from my side. Hopefully you learned something new. Thank you. And of course if there's any questions, there's a microphone there I can see. We have 5

minutes left. But yeah, if there's any If not, then thank you.