A Tale of Two KEPs: How the Community is Taming Kubernetes' CrashLoopBackoff - Yang Li, Google

In this talk, Yang discusses the challenges and solutions related to the crash loop backoff behavior in Kubernetes, a system he has been involved with since 2017. He explains how the current default behavior, which was established in 2015, causes unnecessary delays of up to five minutes when a container crashes. Delving into user feedback and requests, Yang outlines the community's struggle to adjust this behavior through Kubernetes Enhancement Proposals (CAPs). He highlights specific use cases that illustrate the negative impact of the backoff delay on modern workloads, including task isolation, transient outages, and critical sidecars. The introduction of CAP 4603 and later CAP 5593 aims to provide more flexible configuration options for container restarts, allowing operators to manage backoff times according to their specific needs. Ultimately, Yang emphasizes the importance of careful change management in open source development and the need for practical solutions to long-standing issues.

So hello everyone. Uh my name is Yang. Uh I have been working with Kubernetes since 2017. Uh over the years I have been an application developer, a cluster administrator and a contributor to the open source project. Um currently I'm a technical solution engineer at Google Cloud helping users scale their infrastructure on GKE. So today I want to tell you a story about crash back loop uh crash

loop back off and caps. So as we know crash loop back off means some container is crashing repeatedly and kubad as increasing delays before restarting it so that it won't waste node resources. That delay can grow up to five minutes and caps are Kubernetes enhancement proposals. When we have a non-trivial change to be done in the project, CAP help us helps us communicate and coordinate the change.

So this is a story about how the comm community community navigate years of debate to finally tame the five minute war. So first of all uh let's do a quick show of hands. How many of you have deploy a port watch it crash and found yourself starting at the arrow? Okay that's majority of the room. Now, keep your hands up if you ever wanted to fix a

Kubernetes behavior but didn't know how to navigate the cap process. Okay, I see you. You are in the right Um, so if you have raised your hand, you are part of a long history. The crash loop back off behavior was introduced way back in 2015. You may wonder why exact five minute? Why an exponential curve? Um when we talked to the original Kubernetes maintainers, the answer was

simple. There was no deep benchmarking. The current behavior was not designed beyond the motivation to throttle misbehaving containers. It was just a quick defensive mechanism to project the node. So the delay doubles every time up to 300 seconds and you are stuck in in that five minute pennet box until your container runs cleanly for 10 minutes. Now people didn't like this. By 2017, the issue 572 91

was opened requesting to make the behavior adjustable. As I just checked this morning, it has over 360 positive reactions and is it is the third most reacted issues in the core repository. So if you read through the over 100 comments, users were actually begging for three different things. First, the success exit. They argued, "My container did it job and exited with code zero. Why is Kublat punishing

it?" Second, early recovery. People dealing with short two second network glitch um felt that waiting five minutes to reconnect was just too long. Third late recovery for parts already start stuck at five minutes users just want to manual reset it. So everyone hates the five minute wait but they all wanted a fix in a different way. Um so the pain was so severe um that users went

to extreme lengths across the layer of the stack. Um at the application layer people wrote bash wrappers scripts to hide the crash from Kubernetes completely. At the cost layer, teams deployed custom reaper operators that constantly pull the API to delete any part entering back off, forcing a fresh restart. At the infrastructure layer, teams were forced to maintain customer folks of Kubernetes just to patch that one uh

hardcoded 300 second constant to compile their own kublet. um when your users are doing all these things to just bypass a simple timer, you know, you have an API gap. So why are we fixing it after seven years? Um because the shape of workloads involved, we looked closely at three modern user stories when where the five minute war breaks down completely. First task isolation. Um, think of

a Q worker or a gaming server. By design, the container wants to exit when it's done to start a fresh new session. If we force them to delete the pod and schedule a completely new one, the startup latency is terrible. They need a fast in place container restarts without the rescheduling overhead. high highly coupled workloads like a IML. Sometimes a pod fails due to a quick recoverable

error like a two-c database blip. The database comes right back, but the pod is now locked in a several minute back off. So the single weight can cause a massive cassette delaying the entire training job for much longer than the original two-c arrow. Third critical side cars. Imagine a pod where the main business logic is running perfectly but its infrastructure site car is killed by the system

like OM kill because the exp potential bit back off applies to the site car. Your healthy main application is trapped in a network black hole for minutes just waiting for the proxy to wake up. Okay. So to solve this specific specific pain points, CAP 4603 was introduced in 2024. The original goal of the CAP was ambitious. We want to fundamentally turn the crash loop back off to

give develop developers exactly what they were asking for. However, changing a core defense mechanism that has existed since 200 uh 2015 isn't just about writing code. It requires a careful balance between developer experience and the note stability. So during the CAP process we explored a few alternatives um but ultimately reject them. First user asked for instant restarts um for container that exit cleanly with code zero. We

had to say no. This comes down to deployments versus jobs. Deployments are designed for long running services and force a restart policy always. So to a deployment any exit is treated as an unexpect crash. If your workload finished a task and exits you should using a job. We couldn't hack the deployment logic for short-lived tasks. Second, we proposed a restart policy rapid. This is the api in

pod spec, but giving developers the power to reduce back offs is too risky for the global node stability. Plus adding a new value creates a version skew nightmare between the AP API server and older cublet. So um there are actually uh other ideas which uh we don't have time to cover today. Now that we agreed the configuration must be handled carefully. The next big question was how

fast is too fast? To answer that we have to analyze the physical limits of the cublet. As you can see on this diagram um restarting a container is heavy life cycle event. Cublet must trigger kill pod to unmount volumes and remove cgroups. Then sync part kicks in to wait for the network attach volumes and pull images. So, we couldn't just guess the new limits. We had to

measure them. We run extreme stress tests um observing a node saturated with 110 instantly crashing single container paths. The data showed us um that even under this stress API traffic only saw initial search to the cublet client side uh rate limit of five QPS. The CPU usage of cublet increased two times. Um but crucially the node survived. So this data this data gives us the exact boundaries

we needed. It proved that enforcing a 1 second minimum limit is safe enough to protect node from So um with that data we started um iterating with alpha features in Kubernetes uh version 1.32 um cap 44603 started providing a knob to cluster um to configure maximum backoff down to minimum 1 second at the node level and in Kubernetes 1.33 um the same cap can change the uh

global initial backup to 1 second and the minimum back off to 1 minute and the breakthrough was using the cubat um instead of a pod API field where any developer could abuse it. We put the knob on the node level. By exposing max container restart period at the node level, we shift the power of the app developer to cluster operator. This is brilliant because it scoped the

blast radius. You can now safely provision a dedicated AML node pool with 1 second or two second limit while keeping your standard web service cluster on the safe 5m minute defaults. So is it solved the version problem is it solve the security problem. Now here is how open source actually work in practice. Changing the global default back off for everyone in the original cap is a massive

change. It requires a lot of soak time in alpha to ensure we don't break the world. But this note level knob that was already ready. um it was safe and operators needed it now. So the community did the programmatic thing. We split the work. This change takes the already implemented and in alpha um cap 4603 and split into two caps with the intention of graduating the smaller

portion defined here to beta. Um it's actually already uh in beta in one uh 35. So the CAP 5593 was created to extract just the node level setting and fast track it. Um so where do we stand today in 13 uh 1.35? 46 03 which aims to reduce the global default backoff delay for everyone continues to be tested as an alpha feature. It needed more time to

soak. Meanwhile, the new CAP 5593, which provides the configurable max container restart period, was introduced to graduate that uh feature to beta and you can use it today. So, um the is great. Um but let's see what this actually means for your workloads. Um to demonstrate this uh I have a multi-node Kubernetes cluster uh configured three nodes worker nodes to show the evolution of the feature a

node with the um one uh the new default behavior uh actually the new configuration that we can configure it to two seconds max back and Second, um we have the alpha uh which changes the global default to 60 seconds and um the legacy behavior 3 second uh 300 seconds. So um we are using these three nodes uh to across the across the user stories in the cap

to show the uh actual effect. Not sure if the video work. Maybe I need to click. Um so the first user story task Um a Q process system or a gaming server where a container process a single item uh exit with code zero relies on the cublet to restart. Now um with the two second configuration the sec uh the container is tearing through tasks. You can see

the restart count um and uh the 60-cond node performing slightly better than than the older behavior but still being throttled. However, um the the legacy 300 second node simply because the container is doing the job too fast, the cublet penalized the the behavior. So it get completely trapped in the uh back off. the two second configuration here is clean clear the winner and for task based work

workflows. Now scenario two fast restart. So um fast restart on transistent outage. Here we have the web applications pinning a database. I'm forcing the hard outage by scaling the database to zero. So all applications are now crashing. Um but crucially uh the outage is is happening. The legacy node back off is climbing rapidly towards 5 minutes and the six second node is climbing too but kept at

one minute and we are about to see the database online. So we can look at the three parts. Now the two second part it detects the recovery almost instantly and heals itself. The six second part will recover within a minute. um demonstrating the improvement um by the first cap. But what about the um Lexi part? It completely obviously to the recovery stuck sleeping um maybe for Oh,

actually this this one this time it actually was lucky and recover earlier. Now the third the third one. So the third one uh was for side car So in this part the main business logic is perfectly stable. Um but it's crucial network proxy set car is suffering from OM kills every 10 seconds. Um and you can see the crash. So on the two second node the site

car revives almost instantly and the pod quickly returns to the healthy two slash2 ready state meaning the main app barely noticed the network issue. And the six second node is better than before um kept to one minute but look at uh laxi node with every crash the back off penalty doubles towards 5 minutes. So the window here um the set card that grows longer and longer the

pod gets start gets stuck in the one slash two state. So during the entire time your main application is running but it's trapped in a network issue because the site car is sleeping. yeah the key takeaways of this um first container restarts are not free. Cublet has real physical limits which is why defensive design and a push back from the feature requests is sometimes necessary. Second, the

two caps finally gives cluster operators a green near safe control over recovery times for their most expensive workloads. And third, open source is a marathon. breaking a m massive ambitious cap into smaller practical pieces is the best way to solve a seven-year-old problem. And uh I want to send huge shout out to the cap authors Laura and Hank and the entire sick node community. And thank you

all for your time. Uh I'd love to take any questions. Maybe someone can pass the microphone. Or or maybe you just shout and I will repeat. >> Sorry, I can't hear. >> Sorry. >> Yeah. Yeah. >> Maybe uh maybe I will go down or or you can come closer. >> Test. Can you hear me? >> Thank you. Thank you. >> It showed the same config flag for

the static two seconds as the um the other one, the 100 or 60 seconds. So, how does it know to always have a static two second restart or uh increasing back off? It was the same config flap you show uh showed on the slide. >> Yeah. Let me go back. Are you talking about this one? >> Yes. So max container restart period is the same for the

two of them. It's the same flag. >> So how does it know that it is a always restarting for every two seconds at the bottom? >> Or does that answer your question? >> Not really because it says two different things. At the top it's increasing at the bottom it's >> sorry let me clarify here. So the two examples here is actually two two configurations. So with the

first one um it will start at 10 seconds and grows to 100 seconds. The second one here will always be 2 second. >> But how does it know to start at 10 seconds here? >> So the 10 second um is a default value. >> It's still the default. It's still the default. Okay. That's is what I wanted to know. >> Yeah. Yeah. Yeah. Okay. >> Um, yeah.

>> And because the bottom one is lower, it will just stick to that. Okay. Thanks. Okay. Um, we still have a few minutes if you have any questions. Um if not uh well this is the first time of this talk prepared being presented and uh I have a very limited time to prepare it. So I'd appreciate you have any feedback. Thank you folks.