The Future of Kubernetes Scalability: Challenges of the GigaWatt... Maciek Różacki & Artur Rodrigues

This talk explores the future evolution of Kubernetes scalability, focusing on the challenges it faces in the context of AI computing. The speakers, Arthur Rodriguez from Anthropic and Maciej Krzyżanowski from Google, discuss their experiences operating large-scale clusters and the implications of workload patterns. They emphasize the importance of mutation rates in Kubernetes clusters over just node count, highlighting how existing limitations can hinder scalability, especially under high churn workloads typical in AI training. The speakers propose that the Kubernetes community should address issues such as controller sharding and improving workload scheduling to better support the growing demands of AI workloads. They also advocate for easier customization of Kubernetes and the need for robust processes to translate bespoke innovations back into the upstream project, ensuring that Kubernetes remains relevant as computational demands evolve.

Welcome everyone to the future of Kubernetes scalability talk. Together with Arthur we will give you a glimpse into what we think is the future evolution of Kubernetes and what challenges we think it has to face so that it remains relevant in the gigawatt AI computing era. We obviously do not know the future. So what we're trying to give you folks here are the key points that we

are observing in the field and the implications that steam The implications that steam out based on our own experience building some of the largest clusters that we know of. And My name is Arthur Rodriguez. I'm a member of technical staff at Anthropic. I've been working on Kubernetes clusters there for the past two years. And my name is Maciej Krzyżanowski. I'm a product manager on Google and I've

been working with the problems of operating at a very large scale with Kubernetes for the last eight years. And again a minor disclaimer, of course we're going to share a lot of things that come from our professional experience, but the views and especially the opinions about the future and what the community should do are our own not our company's. And with that, all right. So let's first

let's talk about basics of the terminology and semantics. So usually when we say that Kubernetes has many is is a large scale cluster, we mean how many nodes it have. Maybe nowadays we would say how many accelerators it actually has in one cluster. But we in practice when you look at the realities of how it is to operate very large scale platforms, just the compute power is

really and the number of nodes is not the right parameter or not the right question to ask and workload patterns actually start to matter more. And the better question to ask we find especially lately it's what's the mutation rate that the cluster can sustain. In our own synthetic load tests, uh, we find that, uh, it's in the order of 5 to 16, uh, pod mutations per second.

Obviously, that varies a lot depending on what's the storage layer, uh, behind kind of that that that that that test. Uh, and that's the ceiling on etcd, uh, regardless of the node count. So, at some point, you start growing your clusters and the nodes that you're adding on top of it, they're basically sitting idle because we cannot sustain the those, uh, those pod mutation rates. And that

primarily comes from etcd having a single kind of write ordering, uh, characteristic so that the pod write path, uh, as it's as it's shipped does not shard. Um, and that's the orange area that you kind of see in the graph. So, that's basically capacity that is trended unless you do something. Uh, so, uh, what's then the right mental mental model that we suggest you have? Of course,

you could go into a full analysis of all of the dimensions, which are probably hundreds of those, but really, uh, to start with something mentally simple, uh, you should think, um, about scalability as a combination of the number of nodes that you have in the cluster or the amount of computing power that you have, the level of churn, so activity of of workloads, so activity that is

associated with, uh, scheduling, removing pods, updating them, and the heterogeneities of workloads, so diversity of what's, uh, running within the cluster, and the variety of patterns that might occur when you're mixing training, inference, microservices, databases, all running within the same, uh, data plane. So, just to give an example of, uh, an incident that we actually ran into production, uh, we had this controller that we wrote ourselves

that was writing the same value in an annotation on the same key like over and over again. Um, and this was a such a large pattern that it became the overwhelmingly, uh, right kind of, uh, it took most of the right load in the cluster on that specific one. The fix here was obvious was to check if the value was the same, but it just goes to

show some of the characteristics that you might be dealing with, uh, when you're in the middle of the of an incident. Uh, at at the end, the cluster size remained the same. We just changed that particular controller, and everything kind of went back to normal. So, the framing that we want to kind of take throughout the the rest of the talk here is not necessarily just the

cluster size, but again emphasizing that under some heterogeneous load, uh the mutation rate is actually what kind of matters most. And if we look at the most important representative workloads, which actually are stretching uh the scale limits and the profile of uh Kubernetes uh project and the Kubernetes clusters, we um up until now, the primary ones that were the driving function of of how Kubernetes has been

growing and for a big part of its history were the primary active uh extending uh force for Kubernetes scalability were microservices, especially operated by large digital-native companies that were founded with especially with the internet and cloud-native era. And the plat- these platforms um operate a variety of services that uh um scale horizontally, that are relatively fragmented into multiple microservices, as the name suggests, um and they they

perform operations like uh blue-green upgrades, uh operate with disruption budgets. Now, when we think about the um most problematic and actively stretching workloads when it comes to AI training platforms, we're looking about Well, at the AI platforms, we're looking at the AI training workloads, which conceptually on some aspects are simpler than microservices, and on some aspects introduce new complexity and new challenges. They are simpler because uh

they usually run with low pod density, um they do not require uh significant scheduling activity, but at the same time, um they are actually gang scheduled, which is not part was has not been part of Kubernetes primitives. Uh they um require all or nothing scheduling, so then the recovery process after a failure usually creates a lot of activity in the system. They are are very long, typically

running for weeks. Sometimes longer, sometimes shorter, but the time duration of a workload is very significant. And the scales that we see then with AI trainings really goes into tens of thousands of nodes where platforms operating microservices have are quite settled very well and matured in the domain of 2,000 4,000 node clusters, a little bit maybe now stretching towards 10,000 node clusters for hosting microservices. So, we

are constantly seeing kind of in the headlines kind of the gigawatt numbers. And we've seen lately especially over the past year that kind of its power commitments were actually those at least one one of those axes of scale. Power has been as hard to acquire as accelerators and chips have been. And ultimately you have construction timelines that dictate how fast the software team such as mine have

to move internally to meet those kind of target deadlines. At the end of the day you don't want to have a physical cluster or a physical data center that's up but you cannot make kind of useful use of it. And we've seen that over the past few years that we a path that would have kind of as as much as I was saying move towards like 2,000

to 4,000 perhaps to 5,000 to 10,000 like AI labs have found themselves like going straight to the tens of thousands of nodes like really really fast. And if you have a control plane that's falling behind, you're literally spending millions of dollars per day on those side of accelerators. I want to acknowledge that I represent a very small set of of users that are kind of driving a

disproportionate kind of attention from CSPs and and the community as a whole. And it's only natural that the providers such as Google and Amazon will follow that kind of money spent. That's just a market kind of operating as it should. But I want to argue as well that the that the pressure that we are kind of exerting on those clusters to some extent are not specific to

AI workloads. Um any kind of operator running kind of high high churn workloads or high churn kind of scenarios are going to benefit from the same findings that we do find now and that we kind of solve ultimately. Uh we're finding these ceilings first because we're reaching some of them but they in theory do not exist just for them just for us. Um and speaking of heterogeneity

I at least like to see it at in three different axis. Uh the first one is kind of multi-cloud. Anthropic for the longest time has had investments from both kind of Google and and and Amazon. Most recently we we had investments from Microsoft and Nvidia as well but it just represents kind of a very interesting kind of set of scenarios on on GKE we have mega clusters.

It's a project that Machina and I have been working for the more than a year at this point hitting tens of thousands of node scale. And it backed by Spanner in the back end. On EKS we have ultra clusters which have a different kind of technology behind the scenes. It's SCD that's partitioned using a an Amazon specific journaling kind of technology. But the reality is that these

are different control planes and the mitigations that we use on one don't kind of necessarily translate into the into the other. So we end up having kind of two different set of run books for these clouds. A layer below we also have different architectures for the for the chips themselves. So we have TPUs on Google primarily. We have Nvidia pretty much everywhere and we have Amazon Trainiums.

Each one of those have their own kind of set of topologies and placement rules. And the nodes are largely not interchangeable. So the the the solutions that we come up for one of them do not necessarily translate to the And finally we have the workloads themselves that sit above all of these clusters and and chips. They span from training which is kind of very predictable and large

and is restart averse to inference which kind of looks a little bit more what you would find in a traditional company with You have kind of reinforcement learning which is very bursty and the same with CI it can come like it doesn't have an upper bound literally depending on which day of the week you're in. And all of these have like very different mutation profiles and they're

not kind of isolated from each other necessarily. My job at Anthropic is ultimately to hire to hide this heterogeneity from the rest of the company. They just want to use the clusters. They just want their jobs to kind of run at the end of the day. And Anthropic since it's beginning kind of battered Kubernetes because of the ecosystem because of the API surface. It's a choice we

made but it comes it comes with a cost as kind of that scale increases. So I I I would argue that like the job that we've been doing Maciek and I we we wouldn't we haven't we wouldn't be able to have done it like without each other. On the Anthropic side we kind of have built our custom scheduler over the past couple of years. It needs to

be topology aware exactly because of those how those chips operate like the the fast communication low latency communication that exist there. They need to they need to be gang scheduled so all or nothing. And these were like things that weren't available when we started kind of having those needs. We ended up developing our custom CRD which is close to what job sets looks like but with some

some extensions. Originally we were stateful sets for for for different reasons because we had like implicit ordering between between the pods but also because it allows us to do kind of on delete update so literally control when nodes kind of had kind of their specs change as opposed to just changing the spec and everything getting kind of restarted. Um the other thing that we found with with

stateful sets that kind of moved us towards kind of our own CRD is that it allows us to control the rate by which kind of the pods create. With stateful sets you have a batch and wait sort of behavior and it just goes to show like stateful sets was something that we kind of chose in the beginning because it had some of the right semantics but as

it scaled we outgrew them. Um now going into another example kind of we developed our own CoreDNS plugin that kind of watches pods directly. This is to kind of avoid the interaction through endpoint slices and have been kind of a blast of of synchronizations between informers all over the all over the control plane. And we actually had three different generations of of of kind of read caches

if I could argue. Now operating at that scale of course requires collaboration and as Arthur said our teams on both companies put a lot of efforts into making this happen. And there is a quite unique role that the infrastructure and base services provider needs to do for this to to work and really the largest clusters in the industry are ones that are also managed by hyper scalers.

there are certain changes that are simply required how you think about how you integrate with the underlying infra. Then the most notable customization that really or deviation from the standard way of how you deploy Kubernetes is the fact that we at some stage we decided to change the database storage from etcd into spanner. Most of the rest of the stack is still contributed to to the open

sources is going to be upstream where it makes way more sense to to fix these things. And so all of the innovations that were were detected during testing, validations, preparations or as these huge clusters are operated and we're jointly troubleshooting and fixing what's not working well, that stuff is also then going into the upstream Kubernetes. And I would like to actually propose a hypothesis here which is

that the staffing bar has actually been dropping. If I were to look back at the early stages of the company, I could probably place somewhere around 2023 as being kind of traditional Kubernetes users in a small kind of a 100 to 200 person company. Most of our interactions with Kubernetes was actually through the CSP creating tickets so that they could kind of take a look at what's

going on wrong with our clusters. That shifted at the end of 2023. We're able to kind of hire more infrastructure people, more software engineers. And at some point Claude kind of got good enough that it helped us kind of write software faster. So it was around that time that we had our first version of our kind of a read cache kind of taking some of the load

off of the control plane from burst users. And the first schedule implementation kind of appeared kind of replacing scheduling gates. Again, to solve that scheduling the topology aware topology aware scheduling problem. If you move to 2025 in this year, we have Claude writing writing most of our infrastructure software. So it's writing kind of controllers, that CRD that I just mentioned it went from a design to actually

being in production in a matter of weeks as something that before it would have taken at least a quarter. And we have more controllers, we have proxies, more kind of read read layers, especially one that that's quite interesting that uses that exposes the Kubernetes as a as a sequel kind of interface to the rest of the company. And my claim here is that we're probably going to

have more operators, more folks like you in the wild kind of creating bespoke solutions and fine-tuning Kubernetes more and more and more. And this is the long tail actually building in in the ecosystem. So we need to figure out a way to get that path from bespoke solutions to upstream, that pipeline that kind of produce solutions such as queue and job sets a little bit crisper. So

it matters more and more these days. So we see that the there is a significant trend in the efficiency of how you can build, operate, and customize platforms that use Kubernetes. But there are also two other trends that I wanted to highlight that are very important when it comes to how you how we see the future of Kubernetes and the the environment in which in which it's

going to be evolving. So one important trend that it's quite noticeable, is that for AI, Moore's law is still in effect. Every new generation of accelerators is substantially faster than the previous one, whether it's a 2x difference or a 3x difference. It is a rapid progression. If you look at variants of the Moore's law like computing power per megawatt or other like they all see a very

rapid progression towards increased computational power. You would expect actually that the strength could be directing to Kubernetes towards shrinking when it comes to node content cluster sizes. We do not see that actually happening. The demand for computing power to do research and to run inference is only increasing and it's simply taking consuming any hardware innovation gains that are going into the market. They all are being consumed

by further to fuel further growth. On top of it, we see that the changes of workload patterns have an even bigger impact on how Kubernetes clusters are loaded and they create an even bigger challenge than just the compute power growth and demand for more compute power. Specifically things like the transition from developing models using primarily pre-training into a very significant post-training phase that's using reinforcement learning, changes

in architectures of models where either mixture of experts or reasoning models are changing the balance of memory versus computational load or the sizes of context windows that are constantly growing. All these things are drastically changing how the underlying infrastructure is actually loaded and pushed to its limits. Yeah, so the underlying point here is that the hardware is getting faster. These nodes are getting kind of denser flop

wise, but the reality is that the company that I joined two years ago, it was primarily a research environment with like big training runs, but very much smaller customer footprint that it has now. Then we had this the the the evolving kind of of reasoning models and so on and reinforcement learning become became like a big thing. So, it just goes to show that the the red

bar that you see keeps growing. So, heterogeneity is definitely something that matters more and more for us. Just to stress that point once again. So, if you want to put forward some hypothesis, what we think actually is going to be happening to the in the future of Kubernetes and where we think it will be evolving and what we all should do as a community that's building and

shaping that project, what what we should really consider. So, one is that large large clusters are not going away and actually they are going to be only getting larger, probably reaching scales of a quarter million of nodes or equivalent in chip count. they are going to give a lot of benefits in in the ability of very easily building and defining trade gun schedule training workloads. They will

be easy to operate with smaller teams, simpler upgrades, simpler maintenance procedures at the same time. They are going to come with major risks and challenges associated with the blast radius, with exposure to reliability problems, with challenges how do you manage noisy neighbors in terms of workloads and the load that is going to be exerted on the control plane and the risks that come with that. Then, multi-cluster

architectures already being adopted by AI labs and quite standard also in in the wider industry where they offer a great risk mitigation surface by spreading out the load of the control plane, spreading out the blast radius surface, your failure domains. At the same time, they create a layer they create a significant management overhead for the platform teams. And they also come with the downside of still requiring

a layer of centralized management and routing constructs like how do you do multi-cluster training or how do you route requests and workloads and the whole layer of capabilities that need to still be centralized actually for you to operate a fleet. We will also see then a pattern that Arthur mentioned that it's going to be only easier to customize Kubernetes and build your own best box solutions and

it is going to be a pattern that we'll see. So, on one hand as a community we got to make sure that this is a pattern that can be easily followed that we maintain and cherish the architecture of Kubernetes that is very plug and play and you are able to add and remove pieces of it as you see fit. At the same time, we will need to

strengthen our pipelines and processes of how we push these customizations into the upstream, how we evolve them towards being standards so that still the base Kubernetes product follows the innovators on the edge, standardizes what they are doing, and democratizes access to it. And then last but not least, with the heterogeneity of the way how power is distributed, how the physical world is simply constraining our system's design,

we will see that there will be a pattern of also growing usage and adoption of self-hosted at very large scale. So, today the domain primarily of managed Kubernetes offerings that were highly scalable and were integrated with the infrastructure, some of that work will also have to be translated into self-hosted environments. And at the end of the day, I would argue that companies such as Anthropic should be

investing on all four axes here. As as I shared before, we we're building our own data centers so self-managing is on the table and we would like to see kind of making that process like easier and more accessible not only for us but for the rest of the community. Um, we are definitely interested in having bigger and bigger clusters to the degree that that's possible and as

much as said, some if if not most of those enhancements that kind of power that scale end up going to a upstream Kubernetes and benefiting steady steady state smaller clusters. Um definitely solving multi-cluster is is something that we're interested uh perhaps because not all capacity is within a single region. As a matter of fact, it might be very distant uh from uh both kind of the other

half of that capacity and the control planes in some cases. And uh yeah, we're going to continue customizing. So, the easier it is both to do that at vanilla Kubernetes, but also with uh CSPs such as Google and Amazon and Microsoft, the better for the I wanted to double-click a little bit on that uh customization aspect and um the how um Kubernetes will will also uh these

platforms will include a significant portion of bespoke uh uh components and how we think also it will be as a pattern adopted more widely by users of Kubernetes. Um Kubernetes already it was designed in a way that uh it is modular. Um it follows the principles of separation of concerns between controllers. And um it's it is actually quite easy to to operate it with this plug-and-play mo-

approach that you replace certain pieces of it, pick a la carte what you really need, and replace um what you believe is going to run better. And you actually need to be able to operate in this way also on managed services like GKE, and not only on your self-hosted Kubernetes. Uh but for for in um but we as a community also need to make sure that we

look at uh what is being customized, what is being built bespoke as bespoke solutions by companies that are innovators, and they really need to go fast. Um and and they did cannot care about standardization. Um we also need to take a look at uh how we actually take the learnings from that, uh distill what is really uh necessary uh or really valuable as a shared standardized capability

in core Kubernetes. And we need to make sure we have that pipeline uh back to the upstream Kubernetes or add-ons to to that we believe are should be the primary, let's say, standard add-ons into Kubernetes. On the slide, we have a couple of examples of how that uh pattern was used in practice. Well, for example, Anthropic replaced replaced the cube scheduler for their scheduling of their workloads

and modeling things like topology of a scheduling. While we in the community founded both Q add-on to Kubernetes as the initial way to wrap it in a standardized way. And now the work behind the scheduling is actually being integrated even deeper with the recently announced workload auto scheduler initiative with all of the major partners supported by all of the major partners in the community to standardize the

scheduling construct. Similarly, we see such patterns for workload controllers like with job set or or other like API server innovations for for optimizing workload. Um we wanted as we are coming to an end of the presentation, we wanted to spend a minute on giving you a glimpse of what together with Arthur we've identified as highlights of the most pressing immediate things that we think that the open

source community should look at when it comes to ensuring that can support the scale of AI labs. And number one that we wanted to highlight is the fact that in many platforms the scalability of parameters associated with the level of of the activity of controllers, the limitations, the number of pods that you have in the cluster is actually the the primary active constraint that these platforms are

hitting. So, it's very So, we believe that it's very important that we invest in proper controller sharding and work queue partitioning so that this load can be distributed versus today most controllers are designed with this leader election model where you have the primary driving replica ready of a controller. So, it's in practice these are from scalability perspective single replica controllers. Yeah, and from the API server side

we've seen a few instances where like the cluster basically explodes and you have this pathological case of trying to control everything else that's trying to reconnect to it. So, two kind of suggestions here of things that could be worked on and in fact I on the back off scenario there is active work happening there. Is to have some sort of like back off mechanism. Basically telling clients

to to like they've tried to connect like it didn't work out, wait it out, we're going to tell you like when it is when it would be safe to do so. Especially for list operations. And on APF side, so priority and fairness. We we find ourselves like talking to to other companies and each one is trying to do kind of their own thing and I think there

is a space here for like some high churn scenarios to be mapped out on upstream or by kind of big CSPs. So that we don't have situations where you have like nodes trying to reconnect. And their heartbeats kind of being on the same priority as like some random bulk list requests. Finally, if you go down to the workloads themselves. You have kind of the workload workload aware

scheduling kind of family of improvements that are happening right now on 135 and 136 with some of those becoming beta I think in 137. One in particular that's quite interesting to us and that we have been kind of very happy to to work with Google with is on the preemption side. So right now you have pathological cases where you might have like a large job that's running

at a certain priority. A small job comes in, it can be as small as like one pod with a higher priority it kind of bumps one of the pods and then you have like that kind of huge capacity stranded because it actually needs that kind of final pod to work on. So just to kind of separate those two outs in terms of priority. Our scheduler internally already

does this, but it would be one reason to diverge once that kind of hits higher levels of maturity. Similar on job sets, we would like to see kind of the immutability of the spec and perhaps also the fusing of admission and scheduling. One of the things that a company such as Entropic might find themselves in is the situation where you have a lot more pending pods than

running pods. And the reason for that is because there's no auto scaling that we could use. We have a fixed set amount of capacity and that's it. Uh you actually have an interest in having a bunch of pending pods in the cluster just so in the off chance that some capacity kind of is free for you to use and eat that out. Um so we'd like that

kind of uh uh fusion of admission and scheduling to kind of take place so that you actually don't have pending pods. You just have a job uh a job set spec that kind of tells you how many of those needed need to exist when the time comes. So to summarize then for the final 2 minutes, uh we wanted to highlight that um all in all it's a

very exciting time. Lots of challenges but also really lots of cool innovation and things happening in the community and with Kubernetes. It is being used to really change the IT industry and um all of the products that we're using. Um we think that when it comes to um the patterns that we're going to see of how uh larger scale is going to achieve with Kubernetes, like it

or not we'll have to invest in multiple paths to and multiple um uh ways to to achieve that. Large very large clusters are not going away and they they are going to keep growing to the limits that are really permittable by the constraints of the physical world. Um multi-cluster capabilities are going to be crucial as a risk mitigation and as a way to for blast radius and

as a way to manage the heterogeneity and the complexity of how the capacity infrastructure is actually physically installed across the world. Um and then to fuel the innovation, the velocity that is needed by time-to-market companies um like AI labs, we um they will need very easy and very quick um uh ability to best customize and build best book uh uh controllers in Kubernetes. Which and the bar

for that is actually going to be only going down with the innovations that are happening in with the AI models. Um so, as a community, we'll have to be prepared for that and make sure that we have the right processes to learn from that innovation and build the right standardizations and base solutions so that we we make sure that Kubernetes is relevant for for the industry in

the long run. Yeah, so at the end of the day, I don't see this as a race with a clear winner in terms of like, oh, it's either very large clusters or many small clusters or self-hosted. Uh each one of those is answering a different set of constraints and we don't have to kind of hedge on on a single one for a verdict. Uh what what I

think we would like to see is for Kubernetes to to make the combination so cheap that that we can just follow as the mix shifts. Um and yeah. Great. Thank you. Thank you so much. Yeah, stay around after the session. Stay around and take questions.