Generalizing Kubernetes Controller Sharding: Patterns That Wo... Motohiro Otsuka & Tomoyuki Nakamura

This talk covers the topic of generalizing Kubernetes control plane sharding. The speakers explain that sharding is not merely a feature toggle but rather an architectural decision that affects system scalability. They discuss their experience with a Kubernetes-based internal platform at LY Corporation, detailing the challenges they faced, including synchronization issues and increased reconcile latency, as they scaled to a significant number of pods and namespaces. The speakers introduce KCS, an open-source project aimed at achieving horizontal scalability for Kubernetes controllers through consistent hashing and namespace-level sharding. They detail their architectural evolution, including the initial failures of their design and the eventual pivot towards label-based sharding, which proved to lead to more predictable resource usage and improved performance. By the end of the talk, they present five general principles for sharding architecture in Kubernetes environments.

Good morning. And thank you for being here. My name is Tomoyuki Nakamura from LY Corporation and he is my co-speaker, Motohiro Otsuka. Today we want to talk about generalizing Kubernetes control plane sharding. This is not a story about clever trick. This is not a story about performance hack. Our core message is this. Sharding is not a feature toggle. It is an architectural decision about where you place

your replication boundary. At that boundary determines whether determines whether your system scales or quietly collapses. We walk through five parts. First, why scaling broke for us. Second, what sharding can and cannot solve. Third, why our first architecture failed. Fourth, how we pivot it. And finally, the reasonable decision framework we ended up with. This is not just our story. It is a reasonable model you can apply to

your own controllers. Our team develops and operates a Kubernetes-based internal pass platform at LY Corporation. This diagram shows an overview of the pass environment we provide internally. It consists of a single control control plane Kubernetes cluster and many data plane Kubernetes clusters. Internal users define up resource in the in the control plane cluster and through they manage resource such a pulse running in the data plane clusters.

My team operates at roughly 66,000 pods, 5,000 nodes, 3,000 namespaces, and 500,000 managed objects. At this scale, we started observing pain. Not out of memory, not classes. Instead, through initial through initial synchronization during restart, longer reconcile latency, delayed user people promulgation. The system was not broken, but it was approaching our structural limits, and we knew vertical scaling alone would not protect our however. Let's quickly review how

most Kubernetes controllers are structured. Most Kubernetes controllers follow the same model, shadow informer cache, single active reader, and standby replicas. This design is extremely efficient for vertical scaling, one process, one shared cache, minimal duplication. But, it is not elastic horizontally. Reader election ensures only one replica reconciles. So, when reconcile pressure increases, you cannot just add replicas and expect throughput scale. Controllers are vertically efficient, but not horizontally

elastic. That is a structural constraint. Uh I don't know actually uh how many people are there uh struggling with Kubernetes reconcile delays caused by large number of resources, but we can't really work. So, we choose Kubernetes contra-sharding, a KCS. KCS is an open-source project developed by Team Ever. You can find it at GitHub. It's going to make Kubernetes controllers horizontally scalable. KCS works by computing shard selection

from hash of resource identity. For example, namespace name and parent identity for controlled resources. Using this is to determine shard ownership. Assigning object via labels. Keeping parent and controlled resources on the same shard. Supporting drain during rebalance. Let me know Let me now explain how KCS actually assigned shards. KCS uses a consistent hashing ring. Each shard gets five 100 virtual nodes dis- distributed across the ring. When

a resource is created, we compute its key, hash it with XX hash, and use binary search to find the next token on the ring. This approach minimizes rebalancing when shards are added or removed. The key format determines how we compute the hash. For a main resource, like app, the key includes group, kind, namespace, and name. For controlled resources, like app child A or app child The use

key uh the key uses the parent's instead of the child's own name. This ensures parent and child are always assigned to the same shard. KCS obtains a parent resource name from child's resource owner reference. This is why owner reference with control tool is essential. To design the first attempt, we had to start from one implementation KCS computes shard assignment from a parent-based hash inputs, such as parent

namespace and parent name. A child object can use owner reference to resolve that parent identity. But the grandchild cannot always resolve the grandparent identity needed for the same hash input. So, same shard co-location is natural for parent and child, but it does not automatically extend to deeper chains. Our paths have complex resource dependencies. It seems that making it difficult to neatly decompose them into a simple parent-child

relationships. So, our design hypothesis was manager manager isolation by namespace could cover dependencies beyond simple parent-child closure. That constraint shaped the next hypothesis. Given that limitation, we shifted from object level co-location to namespace level ownership partitioning. At that point, that still looked practical. At that time, app-related resources appear namespace-closed. From dependency graph perspective, links did not cross namespaces. So, namespace looked like a natural shard boundary. Our

hypothesis was simple. If we spread reconcile ownership by we could reduce synchronization pressure while keeping product behavior unchanged. Conceptually, this looked green. It looked generic. It looked reusable. Our first architecture used namespace-oriented sharding. When we decided to adopt namespace-level isolation, we faced two architecture choices. Option A, dedicated pod per namespace. This is a typical pattern you might see. Option B, use KCS to shard by namespace with

one pod handling multiple through Why did we choose option B? We expected that by using KCS namespace sharding, one pod could handle multiple and be cheaper than the pod per namespace approach. But first, why did we need it to fork adopt KCS? Because our internal past dependency graph could not be partitioned only with that parent-based ownership rule. We had dependencies that required deeper chain consistency beyond parent-child.

So, to isolate the full operator domain, we were forced to use namespace-level manager isolation. We forked KCS and we implemented a manager per namespace design. The approach is to start contra runtime manager for each namespace and provide dedicated controllers per For example, controller A, B, and C each watches the namespaces assigned to their respective shards. And if a namespace is assigned to them, they start a controller

controller that maintains a cache scoped only to that namespace. On paper, this looked clean. Our initial expectation, namespace sharding, would be more efficient than running one pod per namespace. In practice, this architecture might have worked if it weren't for the specifics fix of our pods. By maintain- maintaining a fixed number of controller replicas and adjusting them according to changes to the number of namespaces, we could have

achieved horizontally scalability. But the reality was different. The in-process manager instantiation cost turned out to be roughly equivalent to the pod per namespace cost. Namespace sharding multiple multiplied the already multiplied the structure. Practical overhead far exceeded expectations. After all that, the upper beta and that OOM restart rates, go routines increased from 3,500 to over 500,000. We observed CPU memory saturation. At that time, namespace was 3,000 apps,

11,000. This regulation did not exist before sharding, so sharding triggered the amplification that forced to us to stop and measure. We moved to repeat it validation measurements. We measured the people of use multiple times in controlled environments. The simplified model we derived was this formula. The dominant amplifier was namespace correlated growth. Each additional namespace increased managed instances. Each manager instance added its own informer and cache structure.

So, against the namespace count, this was linear growth. The problem was that namespace count was not something operators could practically cap. And at large N, absolute your resource usage became much higher than expected. Remember, our operator was already multi-threaded again. Before sharding, we run base manager plus cluster manager times N. With around 40 data plane clusters, that was 41 manager instances in one process. Before sharding. After

sharding, namespace sharding created a manager for every namespaces. Each of those managers replicated the same structure. So, the math became Before sharding, we had about 40 manager instances, the same number as the our After sharding, 3,000 namespaces times 40 manager per namespace equal 120,000 manager instances. From 40 to 120,000. Amplification factor times 3,000. The structure was already multiplicative. Namespace sharding added another dimension to the product. And

the cause of the mistake? We underestimated the unit cost of each manager instance. This estimate error amplified 3,000 times was catastrophic. Now let's generalize. Kubernetes watch semantic error, single namespace or all namespaces. There is no native arbitrary namespace set watch primitive. This constrained multi-namespace sharding design in general. However, in our architecture, the primary explosion came from cache and manager replication. The API constraint explained the outer boundary,

but the inner explosion came from cache duplication. Two layers, different problems, both models. A quick note about the recent developments. Just 8 days ago, on March 17th, KEP 5866 was merged into Kubernetes. Server-side sharded list and watch. This enables hash-based range sharding for list and watch operations. For selector equals shard range object metadata UID 08. This partitions watch streams by UID hash value ranges. The initial implementation

supports UID-based sharding. Namespace-based sharding is planned for Sorry, planned for future versions. This addresses the or single constraints from us different angle. It's an alpha feature in version 1.36 behind this the sharded list and the watch feature gate. Kubernetes controller sharding in the same sharded behavior relies on parent space hash inputs. Children can follow parents through But dependent senters cannot always derive the required ancestor identity for

the same hash key. So, Kubernetes controller sharding is not a full graph partitioner for an arbitrary dependency depth. Kubernetes controller sharding is a distributed ownership layer. It is not a distributed cache framework. Configuring those areas leads to incorrect architecture. so we pivoted. We split some monosti- monolithic operator into app operator and deploy operator. All resources is reconciled by the deploy operator are assigned to a data plane

cluster. So, we can shard them by data plane However, the remaining resources that are not yet assigned to any data needed to be partitioned using different dimension. And we adopted two design principles. work within Kubernetes API design Our first attempt fought against Kubernetes watch semantics. Namespace watch is all or single. At that time, we validated this There is no arbitrary namespace set So, creating a manager per

namespace meant creating a full informer set per That was fighting the API. In the pivot, we used label-based sharding instead. This is a Kubernetes controller sharding default. Label selectors are native to Kubernetes watch and list. They work within the API, not against Second, keep manager count per process fixed. Manager startup is expensive. Each manager carries informers, caches, go routines. If manager count scale is an external dimension,

cost becomes unpredictable. In our first attempt, manager count scale with namespace count. 3,200 namespaces * 41 managers = 131,000. that architecture, upper operator uses label-based One manager handles all namespaces. Manager count per pod is one. Deploy operator shards by cluster. Each pod manages exactly one data plane Manager count per pod one base plus one cluster manager. For upper operator, we apply the caches only to the app

center domain. App and its child resources, app child A, B, C, and others. This domain fits caches assumptions. Parent-child relationships are clear. Ownership is well-defined. And importantly, cache replication remain bounded because we did not create manager per Now, let's look at the plural operator. The plural operator takes up child resources and creates deployment service and other worker or resources on the top Before the pivot, one operator

pod contains a base manager for someone cluster manager for every data progress data plane cluster. Operator to cluster manager that was one to n. With 40 clusters, that was 40 cluster managers sharing one process. All caches, all informers, all reconciler loops in the one pod. After pivot, each plural operator instances managers exactly one data plane cluster. Operator to cluster manager is now one to one. The dominant

cost driver per pod changes from n clusters to just one. Cache scope is bounded per instance, and failure blast radius is limited to one This made cost predictable. Predictability is a form of scalability. So, did the pivot actually work? Let us look at the resource usage across three architectures. Before sharding, everything ran in single leader pod. Concentrated but bounded. In the first attempt, namespace correlated amplification burst

as to OOM. In the pivoted architecture, we have multiple child Each child carries a base overhead. Manager startup inform a baseline. But, the total resource usage across all child is comparable to pre-sharding levels. No namespace correlated explosion, no OOM regression. And, because Uh sorry. I don't know. Uh because shard fails independently fail failure blast is bounded. A single shard result does not cascade. Reconcile latency improves. The

system has been running a stable release in production since the pivot. The pivot worked and because it worked we could observe new edge cases that only appear at scale under real traffic. Let me walk through two of those. To explain this issue, let me use a familiar analogy. Think about deployment and replica sets. Normally, a deployment creates a replica set. The replica set gets an owner reference

pointing back to the deployment. Under KCS, the child's shard is computed from the parent's identity via that owner So, parent and child runs on the same shard. Reconciliation works as expected. But, what if I user create a replica set directly without a deployment? That replica set has no owner reference. So, no shard is assigned to it. No shard label is set and controller has only reconcile resources

that have a shard assignment. So, that replica set is reconciled by no one. Nothing happens silently. This is exactly what happened in our past. App creates a child resources. Normally, owner reference is set. Same shard, no problem. But, we had a pattern where an app child was pre-created before the app existed. No owner reference, no shard assigned, no controller reconcile that app child. It simply sat there

invisible. to every The fix was adoption logic. But, there's a challenge. An upchild without a child assignment isn't in the cache. So, we had to use API reader to query the API server directly. Once the owner reference lens is set, KCS will assign the object to the correct child. The principle is clear, sharding but reconciliation assumes ownership alignment. If any object can exist without a proper owner

reference, you must design adoption logic before enabling sharding. In practice, this issue This issue stems from the design of KCS. If we are to design our own sharding logic, we should do so in a way that avoids this pitfall. Maybe KEP 5866 solves this problem. Another issue appeared during rolling updates. In KCS, each child is identified by its pod name. With a deployment, pod names are random.

Now, when you do a rolling update, new pods get new random names. The child identity is changed. Every child Every roller changes who each child is. That sounds harmless, but it's not. Every managed object carries a child label. This label This label says which child owns it. When child identity is changed, all of these labels must be rewritten. In our case, over 11,000 objects needed label updates

at the same time. or at once during a single year rollout. This is a red storm hitting the API server. And at our case, that caused API server overload. The fix was simple. With a stateful set, pod names are ordered and best. After rolling updates, the then the name stays the same. Sharded identities are preserved. No longer leader needed. So, additional API server load. We moved up

up operator to stateful set and turned leader elections related settings. Sharded identity must be operationally stable. At scale, this is not optional. We end with five general principles. First, respect API semantics. Work within Kubernetes API design Use label pair selection. Respect watch Second, fix manager count per process. Do not let it scale with external dimension like a name spaces or Third, shard by architecture. Choose your sharding

access based on your architecture dimension ratio. Fourth, stabilize shard identity. Design rollout so shard identity does not fluctuate. And fifth, operational risk comes first. Evaluate operational risk as a first-class concern. This principle generalizes beyond our specific case. Now, let me connect it back. Now, let me connect it back to this our Our conclusion label pairs over name space base hold from one simple fact. Namespace count was

3,000, crystal count was 40. Namespaces dominated. If we had sharded along the namespace axis, major count would have multiplied along the largest dimension. But if your architecture is the opposite, clusters will outnumber namespaces, then namespace level isolation could actually be cheaper. The sharding method is not universal. It depends on what dimension dominates your architecture. It is a choice of replication boundary that works within current API design

principles and keeps per process cost for your architecture dominant dimensions. Thank you.