SIG API Machinery: SIG Updates and Deep Dive in the AI/ML Era - Stefan Schimanski, NVIDIA

This talk focuses on the advancements and updates in the Kubernetes API machinery, particularly in the context of AI and machine learning workloads. The speaker discusses recent features in Kubernetes versions 1.35 and upcoming 1.36, highlighting improvements such as declarative validation and mutating admission policies. There is an emphasis on how scalability and performance have been enhanced in Kubernetes API servers to cater to large AI clusters, which often manage massive volumes of pods and nodes. Key challenges related to API performance and data consistency are addressed through various strategies, including effective caching and streaming of API requests. Additionally, the speaker introduces concepts like object sharding to optimize resource management within Kubernetes environments.

All right. Welcome to the sick update API machinery. So, we have a special subtitle today, deep dive into the AI ML era. So, I have a larger part today about the work of the recent releases and future which enables AI workloads and everything is of course about API server and APIs, but there's a special part that topic. And so, we're representing the sick here. There are a

couple of more people, of course. There are two more leads. This is our chair. Um but I'm the only one this this this time in Europe who participate, so I'm alone here on the stage. Um many of you have seen this before. So, in every sick update, we have this overview slide basically what we are doing. Like what is the sick? And obviously, it's about API handling

mechanics in the API server. Versioning, serialization, rest mechanics, everything like that. Um everything which is about resource definitions, open API, CRDs, what is kind of what are the features in APIs in the API metadata. Then on the left here, I skip scalability for a second. Extensibility is CRDs. Mainly, it's admission, of course, as well. And we own that. We own everything in the client mechanics and the

core controller mechanics, especially in the uh controller manager. And the last thing here is scalability, but this is kind of a shared owned um topic. There's a sick about scalability as well, and um the sick, of course, has a focus just on the API server, but um it's fluent. Um people work on both sides, of course. What API machinery is not, it's not about like we don't

um do API reviews as an API review team in the project. It's not the machinery itself. And we don't own the APIs. We don't own deployments or anything like that. Of course, we own some APIs like admission, around admission, but not in general. And um same thing, we we don't own many of the controllers, right? We have some, but many about workloads, especially, we don't own. And

kubectl, same same thing. There's a CLI sick and etcd also has its own sick, so But yeah, this is the overview what we are doing, and updates. So, let's uh look on the current version, 1.35 1.35, and uh in April there will likely be 1.36, and um there are a couple of caps being promoted and started. And yeah, you see it here in the last column, and

we will talk about some of them. Uh a lot of the topics have been started. They haven't been promoted, but it doesn't mean there hasn't been work. It's just like um there are still things missing for GA release. But what uh when GA 1.10, declarative validation. This is interesting if you write APIs in the API server in kube itself. Um it's not something externally visible, but it's

uh important nevertheless. Um I have a slide in a in a second about mutating admission policy. This went GA. So, less webhooks, finally everybody will be happy who has experienced trouble pain with webhooks. There's something a mixed with a firm proxy, which also went better. It's also quite an internal thing for managed Kubernetes to um survive updates in a better way. And there's a new thing, and

I also have a slide about that, manifests which are on the disk of the API server for admission. All right. So, this is uh the overview what happened in the last release and the few releases before. That's what I just uh promised. So, mutating admission policy um was better. Now it's GA. For those who haven't seen that, here's an example. Uh It's basically admission like a webhook,

but it's not a webhook. It's just embedded in the resource, so you can either JSON uh patch JSON patches here, or you can write um apply configuration, which can include cells, so you can write simple logic in the um admission policy object. And you can also reference to a config map, for example, for configuration or similar things. So, once uh this went CA uh GA, and um

it's generally available. All right. So, um this topic and also admission, you can have on disk like everybody knows static pods. Maybe some of you have used them. They are since the beginning in kube, we have static pods, so you can put manifest of pods in some directory on the API server on the on the control plane node, and then um the API server will start them,

or the kubelet will actually start them. Similar thing here, so you can put on the file system the admission configuration, which also exists for many years, but there's a new feature, static manifest directory, and then you can put in your own manifests, and they are loaded by the API server. API server doesn't get ready before those are active. And um they cannot be seen in the cluster,

so it's for managed service providers, right? You can restrict certain things, and the user cannot see it. That's the idea. So, alpha uh in the news version being worked on. All right. And um the bigger topic, as I promised, about AI, right? What do we do in sick API machinery to enable AI? And lots of the things here are invisible. You will not have noticed them. Probably,

maybe you have noticed them in in performance, or you saw press announcements by cloud providers running giant clusters, and this is the topic here. yeah, so the question or the topic here So, here's API server. I have two pictures. Um and they are taken So, Marek is here. taken from Marek's talk, I think, a year ago, right? which has much more detail. So, um look on YouTube

for the talk. Uh lots of the things I show here um have been done in the last few years, and they are already shown in in his talk in detail. But um for the general audience here, so um when you ask for pods, like your your controller asks for pods, there are different ways to do it, right? And um one of them uh is the one which

has here the resource version zero, so it's a string of zero. And basically means give me whatever you have in the cache. So, could be outdated. If you're lucky, it's fresh, but you have no guarantees. And then there's the empty resource version, and this is the quorum read list, so you can ask for the latest pods, right? And then there are variants like you can have paging.

You can limit it to 500 elements, and um you get a continuation token eventually back, and then you can iterate and in small requests, basically, list the whole uh set of of pods. Give me whatever you have, but in small chunks. And the big question is like are we on the left side where we use um yeah, the watch cache like the in-memory cache of the API

server, or are we on the right side, and we we have a cache, but it's not used because we go to etcd. And if you have the second case, this uh slows down everything because it's much more expensive. So, the goal here is to um to have the left variant, right? To use the cache. And all the story now in the in the following slides, it's about

the topic. So, yeah, so we have these three variants here. It's actually more complicated, so um here's a small table, and there's a variant. I think Wojtek created one with like three times the size with all the different cases. And you see already the colors, right? A lot of reds, and this is from uh 1.30, I think. Like a lot of requests which list objects go to

etcd. Very bad. And uh the work is about fixing that. And um yeah, it has been done. There's something related. Um Oh, yeah, I didn't say that. From from those three requests actually what informers use, like if you have kubelets, if you have the controller manager, they use um the zero version here, give me the latest you have in cache, not the the quorum variant. Actually, they

try to use um paging, but paging is ignored because for those, it's important that it's cached, and in the moment you go to um if you look at this table we just saw, in the moment you use paging, you lose caching. So, it's very very hard to uh stay on the safe side, to not uh damage performance, and at the same time get enough consistency. And there's

a very old issue like years ago from Clayton, when kubelets use this um the zero version, right? Give me whatever you have in the cache. You can imagine you have a kubelet running a stateful set pod, and of course, stateful set means the pod must be um at most once in the system. And now, if you uh if the kubelet restarts for whatever reason, it restarts because

of an update or because of a crash, it will list again. It will it might hit another API server, has a different cache, and it could be there's a time travel, right? You get suddenly old um pod specs, pod objects. It could be that the old kubelet on the one side still sees the the pod on on what this node, and then the restart happens on another

node, and you have two pods of the same name. Could happen in theory. what he's saying here basically is this this variant of the request kubelet is doing like um listing by zero um is unsafe. And we know it, but um in reasonably big clusters, maybe it hardly happens, so we are lucky, kind of, but the more you grow the clusters, the higher the risk is that

those things happen. So, we should change that, and after all this work here done for AI, we could actually change it, but we haven't yet. Anyway, so um the motivation is basically that. AI clusters are big, and if you look into the sick API uh the sick scalability mission in the documents in the in in GitHub, you will still find those numbers, right? 5,000 nodes, 150,000 pods

and containers. And here you see the comparison with what big clusters in the AI training area like they are 100,000 nodes. Some are bigger and you see the I mean the difference here it's 26 X. Which we test, which we support officially and what you actually can get get out of that. And lots of the features we will see now are critical, are essential to get to

that to that scale. All right, so this is a um the focus and there's a trilogy of changes you have to make. And there are basically three yeah, three problems you have to solve. One is when I ask for quorum read like the the latest state which is in etcd, this must be cached, right? Or should be cached. It's not it wasn't in the old versions. Um

next thing and I mentioned it already, pagination should work with the cache. It didn't. This must be fixed. yeah, most of you will know like there's a 60-second timeout for requests in general. So when a request a list request comes, it must go through priority and fairness might lose a few seconds there waiting and then you maybe you have 50 or 40 seconds left and then you

have to serve the whole request. If you ask for 10,000 objects, yeah, this might just timeout. So you might have a bad time waiting for those. in parallel I prepared some demo here. So we have 133 running on this notebook. Everything is live, so something breaks, we will see. Um you see here at the top right resource consumption, so it's idle. Uh 1 and 1/2 GB and

you see here the the green number actually should be red because it's a it's a thing we want to avoid. So this API server started up and did three lists to etcd. And then we have a green number here. Those are consistency checks. We will learn in a second what it is. Um those are similar requests, but they're innocent, they're not expensive. So we want to avoid

the green ones. And I disabled all the feature gates which matter here in this area and yeah, I have uh 10,000 objects in 10,000 config maps in this server. Um they are spread over 10 namespaces and in the beginning I will just take one namespace of that. So it's 1,000 objects and each has 100 KB of data, JSON data or roughly in this dimension. Um and yeah,

I can kubectl get of course and you get 10,000 or two or something objects, config maps. All right, and you saw the speed already. It's not super fast, right? But maybe one request is not the end of the world. But if you now run 20 workers and you do a quorum read list on on one of the namespaces, 1,000 * 100 KB. 20 doing the same thing.

And I have to make it a bit smaller so it doesn't scroll and you look here on the top right. My MacBook has 24 GB of RAM. Um and it's it fits. It doesn't go over that. So I haven't seen any any it's near, right? It's 18 GB. And just listed objects and not even so much, right? The number in the AI area 100,000 nodes is a

million pods, not like 10,000. Different dimension. And you see it takes like 8 9 seconds. And you see here the number of etcd lists goes up, right? 63 and 39 were actually the the actual ones here for the test, I think. Um yeah, that's what you see. It's bad API for the memory consumption of the trilogy. Let's go into uh the three changes which were made. And

Marek is here, so he has lots of many things of the here were done by him, so you can interview him of course in details about details and look at his talk, so it's very good for more details. So nothing here is rocket science. So you you will understand in a second what happened. Everything is basically pretty simple changes, but if you see them, you say, of

course, why not earlier, right? First thing, if I want to ask for quorum read, I want all pods, but really the latest state, what can I do? I can look into my my cache, but I have no guarantees. What do I do? I ask etcd, what's your last resource version, right? It's 117. Then I check my cache, is it at this level? Like does it have Did

it hit this walk up watermark of 117? If it didn't, I wait until my watch I mean the watch is against etcd, so it's filling up my cache. I wait up to 3 seconds until my cache is up to date and then I serve from the cache. Right? Very simple small loop. And you can do that and you get already some improvement and because those requests don't

go to etcd. So if I so it's restarting now the the API server here with the right feature flags. Now we have consistent lists from cache on. It's starting, so yeah, it's 4 GB at the moment. And when I now list and look on the three, the three number shouldn't increase, right? Let's see. 20 workers listing, memory goes up. It doesn't hit the 18, but it hits

like nine at the moment. But the etcd list call here like this is three, it's constant. That's already great. It's a big improvement and you see it in the numbers like we have 4 5 seconds now. It's already a bit better because it doesn't has to it doesn't have to copy data from etcd and then convert it and everything. And there's also less memory because the buffers

from etcd also take memory, of course. So already a big step. yeah, CPU less CPU latency drops and so on, so much better. And yeah, this is what I talked about and so trilogy, second step. Streaming. The old API server like 1 130 just listed all objects I wanted to have like all pods for example. Put them >> [sighs] >> like listed the the unstructured or the

objects the go objects in memory, called JSON marshal or protobuf marshal and created a buffer like a memory buffer of up to 2 GB and then it was sending that over the network, right? It's also pretty dumb. What you could do and yeah, it's actually not so complicated to do, you could just encode element by element, send it to the network. Throw it away, takes the next

element, encodes and so on. You can do that with some um encoder magic in go and it's improving the situation and if you do that here, so let's start again. Kub API server now with a feature flag. when we are listing you see like yeah, 1 second, how the streaming works, right? It's not a big buffer, but you really see how it progresses over time. And you

know the client could make use of that if it wanted, but on the wire it's the same like it's unchanged. Clients don't have to change anything. And this is a pattern throughout everything we see, right? You don't have to touch your clients. Maybe you have to update client go to participate in those new improvements. Again, etcd lists here at three, so nothing increased there. It's all from

memory and it's yeah, it's similar speed, but um yeah, there's some impact. We will see there's another variant of that which seems even better. So trilogy, step number three. Paging. I said paging sends us directly to etcd. Either we ignore paging which for the informer case the Kub API server has done in the past or we yeah, just bypass the cache and ask etcd for caching for

paging. but we want to fix it and imagine you have a cache and the cache is updated regularly. So you make a list of your first page, right? And in parallel events come in like new objects appear in the cache. Maybe if you now ask for the second page, new objects are already visible, right? Could happen in theory. So during your your long multi-second list, new objects

appear. And then at the list the informer uses a watch to continuously events and then you could have basically a time travel experience. Like the list is updated in place and then watch suddenly see events which are actually from before the objects we have seen already. It's a bad thing, right? And here's an idea which was implemented to uh yeah, I mean we have we have a

B-tree. You might know B-tree from from your computer science class. Um balanced tree, one of the balanced trees and you can update B-tree by going from the root and finding your target element in the in some node. And potentially you have to reshuffle a bit, but basically you only touch everything you have seen while going through the tree, right? It's logarithmic in the in the size of

the tree and the number of elements. Just the height and because it's balanced, the height is logarithmic. So the insight here is for every resource version, when there is an update for this B-tree, I don't touch it in place, but I make a copy of every node in the B-tree I I have to change. And every node I don't have to change, I just point it back

to the old copy. So I clone everything on my path through the tree, logarithmic again. And I get a second tree and this way you can have many many trees B-trees here and they share the majority of the data. then you can imagine like when then a request comes in which wants paging and it has a continuation token, you just choose the right tree in your history

and just go from there and there's no time travel happening because it's really the state from the time the list originally started. You can do that and let's see it in practice. So we are restarting again with right feature gate. So, we have now list from cache snapshot. It's on. This is the snapshotting thing for the B-tree. And now I do a change here. So, here you

see this yellow text. Now, we are listing all 10 namespaces. So, we are going from 1,000 objects to 10,000. So, we go 10x, right? Not a million yet, but 10x. We do that. So, it's again a quorum read and now it has um limit 500. So, we get the 500 and then there are continuation tokens and we get the next 500. And the picture um what happens

here. So, if I do it a bit smaller, you see how the the pages come in, right? And now, as I said, 10x. We are 10x um everything is 10x bigger. And you see the memory consumption there at the top. So, it's not going up so much. Of course, it's it's kind of understandable like we have pages of 500, not 1,000 or 10,000. So, the garbage which

is in the in the heap of Go is much smaller. So, that's why the the impact on the memory is not so high. So, um that's the trilogy. And we have seen um a 10x improvement and uh a lot less memory. A fourth step here, and this is actually what is used in in informers nowadays. It's watch list and um it's just another idea for streaming. If

I want to stream, at the time we didn't want to touch the the JSON and the protobuf marshaler. We thought we have watch, right? Watch is streaming by design. So, the only thing which is missing in watch, we cannot send initial list. So, let's add that, and that's what we did. So, in the in the uh yeah, in old API servers, the only thing you could do,

you could list and then you got this giant blob, right? 10,000 objects potentially. What we do now, we do a watch request and first, I mean, there's a parameter send initial events here, which you have to give to your URL. And then you get a list in the form of events, like normal watch events, and then there's a bookmark event and the first bookmark event um plus

some annotation tells the client Go, "Now, the list is finished and everything now are real watch events." So, we have one request, everything is streaming by design and um yeah, I mean, you you can just take a look what happens and it's basically as expected. All four feature gates are on. We do the send initial events true. And now, um you have streaming, so it's event by

event and the memory on the top right, if you watch that, at the D request, there are none, everything is for memory. And it's basically flat, right? There's no memory consumption, just a few megabytes which goes up, but you can make do many many requests this way and memory is not a problem anymore. And if you do all of those things, you I mean, here's another slide

where you where it's compared, right? We went from the from the big to up to get to 2 GB of JSON blob, first in memory, then over the network, we went to small um uh yeah, streamed um chunks, lines, JSON and then the the watch-based variant, which looks very similar. And when you do that uh in 1.35, uh you are safe basically because all the cases are

covered. if you look back here, why is this the case? So, I showed feature gates, right? Um if you look here on the on the important feature gates, uh streaming encoding is GA, consistent reads from cache was GA as well already. And the watch list is now better in uh it's better anyway, but it's enabled in the server for two releases, so for one releases, 1.34. And

now it's in client Go, it's a default. Like informers will use it by default. If you update your client Go or your controller runtime, um snapshot of the cache uh is also better and it's um it's also default on. So, you don't have to do anything in in 1.35 to get this All right. This is um the part about enabling big scales with the API server. And

yeah, you see it's it's not rocket science, right? Everything is kind intuitive if you see it. Of course, doing that in the API server, changing the things, changing them in the way that um old clients don't break, this is of course um not so easy. There's one more thing for big clusters. we finally get something for sharding. So, there has been approaches for sharding in controller sharding.

Like there were approaches, you label objects and then you use label selectors to get the right um yeah, the right range of objects. And there's an alpha feature uh Jeffrey is working on that. It's proposed and it's implemented. I guess it's merged already. Code freeze has been already. And the idea is that you basically you put a selector here, but a selector on a range, and the

range is 64-bit uh integer. And um there's a hash function applied to every object, which um targets like the the image of the hash function is this 64 bits. It's very fast, but it's it's distributes objects over this key space. And when you do that, you can basically in the invocation of the watch like the request, you can choose I want the first third of keys. And

the second instance of the controller says, "I take the second third." and so on. And that way um the server only has to send the events and the objects to the right replica. So, we save space or not space, but work on the API server and um we also save memory and CPU and so on on every controller. So, it's an improvement and the the eventual goal

is to run also KCMs or the controller manager of kube in a in a sharded way. And there's a discussion at the moment about like what do we want to shard, where is the hash applied to? So, um I think the the initial version is just the UID of the object, but maybe we want to to consider namespaces somehow separate in a different way. And the idea

is to have more keys that you can even specify or by node name, so you can think about multiple ways to uh specify what's the uh to to which values uh the the hash is applied. And when you do that, yeah, you you are splitting the the this value um space here of the hash values uh in three parts in the example. Have three watch shards and

everybody just sees what it has to see. All right. And then there are some more ideas and some are very fresh, so I talked to Marek what is uh what is on the plate and could be done. One insight is um big AI AI clusters, they have also big pod specs. Like those pods have grown a lot. It's not like the 1,000-byte pods, but it's more like

100. That's why config maps 100 KB in my example. So, you could think about what to do with those, right? You should test them. You should have CI against them. That's one one topic. And [snorts] um maybe priority and fairness should be aware of those sizes. So, there's work to do. Another idea, um pod specs of AI workloads like training workloads in the in the pod specs,

there is a lot of overlap which is just the same for every pod, right? And um you could think about having um a pod spec object somehow on the side of the pods. You could think about deduplicating things in memory. And uh I love this picture here uh also from Marek. Um this is a pod in in some big cluster like the memory distribution. And the managed

fields, they cover like half of the pods manifest. So, maybe it's also not the best use of uh CPU and memory. So, there's lots of improvement potential, right? To even go further in the scale. Just to give you some idea for um yeah, what you can do. All right. So, this is the past 2 years or so, maybe um the big changes. final slide. Of course, you

can reach us if I can So, SIG API Machinery, um you can find us on on Slack obviously. Biweekly, there is a meeting every Wednesday, second Wednesday. Mailing list exists and there is the GitHub page where the mission and everything is uh declared. Yeah, and the names, so David, Federico, uh Joe, and myself. So, we are leading the SIG. And of course, uh contributions are welcome. Um

tomorrow uh there is a meeting greets uh over lunch. Uh I will be there, so others maybe as well. Marek hopefully as well. So, if you want to have details, talk to us. And of course, um if you have feedback, please tell us. Um this is the QR code for the feedback. That's all. Thanks. >> [applause] >> As a user of the managed >> please. So, as

a user of managed Kubernetes clusters or my own clusters, what do I need to do to get use of those features? Yeah, a new client Go would be good to get like the the watch list is not on by default if you have an old client Go. Uh and of course, your cluster must must be new, right? Update your kube version, too. Uh I mean, you you

saw the slides, maybe I should show it again. when which feature is on. And obviously, so 1.35 is a good spot because then you get basically this automatically in client Go. But server side, there's no change. But yeah, this is the answer. for more questions. All right, then thanks a lot for listening.