The Latest in GPU, TPU, NIC and Other Device Support - WG Device Ma... John Belamaric & Patrick Ohly

This talk discusses the advancements in dynamic resource allocation (DRA) within Kubernetes, led by key members of the working group including Patrick O'Doherty from Intel and John Belamaric from Google. The focus is on improving the management of hardware accelerators through a new resource slice API and resource claim API, allowing users to specify their hardware requirements more effectively. They elaborate on collaborations with SIG scheduling for optimal pod placement and the development of features related to workload-aware scheduling and hardware compatibility. The speakers also highlight the integration of native resources such as CPUs and memory into the DRA framework and the recent contributions from community members. Furthermore, they discuss ongoing enhancements and the future direction for DRA, including support for handling device failures and integrating innovative scheduling capabilities.

Okay, well, we have a few more seconds. We're getting ready. Doors closing. Last chance. Okay, let's get started. Welcome everyone. This is another update of the working group device management meeting. Also sometimes known as the working group where we do dynamic resource allocation. So, let me introduce your hosts, your co-chairs today. Um my name is Patrick O'Doherty. I'm a principal engineer working for Intel on Kubernetes. I've

been with the arrays since the very beginning uh writing the initial caps. Presenting today with me is John Belamaric from Google. He joined us when we started the working group and is very good at coming up with all kinds of odd ideas where I would have to tell him that well, this is crazy. We can't do that and then we figure out how to do it anyway.

Also another chair is Kevin Glus, distinguished engineer at Nvidia here on the front row. Uh he's covering the driver side to some extent and sig node. So, we all collaborated together on this. And nowadays, we have lots more a lot of lot more people involved than in the beginning. Well, you'll get to see some of that uh as part of this presentation. Yeah, um I'm assuming that

you kind of know what we do, but let's let me repeat it anyway. So, we came together to solve a problem of really how to make it simpler to manage accelerators in Kubernetes. We were unhappy with the device plugin interface and wanted to have a better way to configure hardware to describe hardware, how to adapt to allocate it in a more flexible way. And that's what led

to dynamic resource allocation as a kind of alternative to device plugins. And then sometime later, interest uh grew and we formally founded the working group to host this because it is really a cross-cutting affair. It's not just one sig that can own this. It's a It's a core change equivalent to what it has. Lots of different stakeholders starting with architecture, auto scaling, network, node, and very much

a lot of things are scheduling related because we want the right choice for different pods to really land optimally. So, a lot of lot of things actually have been going on in SIG scheduling around that. We have successfully delivered the array. It is GA for a while now. Um well, okay. Well, that's old news. just to recap, so what we have right now in terms of solving

this problem that I just alluded to, it starts out with the resource slice API describing devices uh available on nodes or for nodes in the cluster. That's the information that gets published by a DRA driver. The users don't need to deal with that. specifically, it's more like something that gets consumed by the scheduler. What the users deal with is the resource claim API. That's where we describe

what kind of GPU, for example, they want. They might ask for a GPU with a certain amount of memory or certain alignment between different devices. And then, it's the job of the scheduler to pick a suitable device and uh allocate it for the pod. That then gets uh sent to the kubelet, and the kubelet interacts with the container runtime through the container device interface to make that

selected device available. And we've also, as an alternative, we've also figured out that we can do something with NRI. DRA net, for example, is a driver that actually doesn't depend that much on CDI at at all. So, there's some flexibility what happens inside the node at that point. Now, this is very core uh structured parameter version of DRA. We basically need to describe devices in enough detail

that we can make those decisions. And going forward, we are extending that. And with that, I hand over to John. Uh sure. Uh thank you, Patrick. Yes, so yeah, what what we we got a lot of a lot of things going on and I'll show you a little more detail in a minute, but at a at a high level, some of the problems we're working to solve

um relate to uh alignment within the node. So, if you have multiple devices, you have your GPU, you have your um NIC, you have your CPU. Uh we want to make sure that within the node, um those things all uh line up and so uh you get optimal performance. So, that's one of the things we've got some of right now, but in order to do that better,

we need to bring in resources that currently aren't under device plugin {slash} DRA driver management like uh CPU and uh memory. So, we call those native resources or node allocatable resources. They're the things that have traditionally been managed out of in your in your pod spec, but when when they're in your pod spec, they're uh they're um you know, you can use topology manager or things like

that that are that that allows kubelet to make the decision, but it could that's a kind of a late binding and you could actually end up landing on a node where it's not possible to get your policy implemented and your pod would fail. So, we're moving that up to the scheduler to make those decisions. So, that's kind of the I kind of went out of order. That's

the second bullet. The um the there's a lot of effort going on in the sig scheduling around workload aware scheduling. If you went to the sig scheduling update earlier, um there was the discussion of that. And so, another big area we're working on is working uh uh integrating with that effort. Um and what that allows you to do is when you're dealing with um multi-node work workloads

that that need all execute at the same time. So typically this is like a multi-node inference or training where having just part of the workload running isn't useful. Um the workload or scheduling group is coming up with a pod group concept that allows you to do gang scheduling and to do topological placement of that. But um what what's often needs to happen in concert with that is

programming of some fabric connecting those nodes. For example, with compute domain in uh GPUs in Nvidia GPUs you need to create a secure subset of the GPUs that all act in concert and to do that you configure iMax. So uh similarly in Google we have something called TPUs and we have uh a way to program a fabric that interconnects individual TPUs. Well, those interconnect configurations are done

not on a per pod basis, but on a per group of pods basis. So that's a long explanation, but but the what that means like in these abstract APIs is that we want the uh life cycle of the resource claim. If we use a resource claim to to ask for that configuration of that fabric, we want it to come and go with pod groups, with leader worker

set replicas, with uh you know, whole groups of pods, not with each individual pod. So that's that's an integration we're working on. Um we're also doing a lot of work to just expand the resource slice API to model more types of devices. So and more use cases. So So, you're familiar with any networking in in networking there's something called SR-IOV. Well, it's a PCI thing, but in

networking it's used where like your physical NIC can be divided at the partitioned at the hardware layer. and we can do that today, but we also want to be able to regulate the bandwidth um constraints on those individual virtualized functions. And we have we can So, we sort of today we can do one or the other. We can regulate bandwidth or we can do the virtualization, but

we can't do both. And so, we're working on a way to combine those. So, there's a These are the sort of like continued execution on the modeling APIs to make them able to cover more scenarios. Hardware's super complex. We don't need to model the hardware exactly. We need to to model kind of the scheduling constraints on that hardware. So, that's what we're working on. Um and then

just just filling out an additional functionality right now. Probably the biggest gap is when you use DRA, those pods aren't preemptible. That's a problem. So, that's something we're hoping to address in 1.37. Uh here's that like detailed view. Um we'll share these slides. I don't think we've uploaded them yet, but we'll put this PDF up on Sched, and then if you're interested, these all link to the

individual uh efforts going on. Um you can see uh 1.36 is the far right column here. Um 1.36 code freeze was last week. Uh we have one outstanding um one with a star there that has not yet merged, and maybe at this point may not make it. So, that may stay in beta for 1.36, but um you know, you can as you can see like there's a

a substantial amount of effort going into into all this. Um so, what are we doing? Those are the things all in flight, but we have more things that we want to do. Um one of the areas that uh I think we've seen a lot of curiosity around or interest in is kind of the day two operations. So, is essentially a you know, you launch it, you schedule

it, and the scheduler doesn't come back into play again. It's kind of a day one thing. And day two is like, well, what happens when one of those GPUs or one of those NICs fails? Um well, how do you evict that pod? We have some functionality. Patrick has implemented some functionality to to help with that. But um there's we think we believe there's a lot more we

can do, and so that's something we're exploring for the next uh few releases. Um there's also a lot of innovative kind of ideas out there because although DRA was built initially to handle these hardware to model hardware, uh it's actually a pretty generalizable API for pub- publishing information about resources, whether they're node local or just node reachable, and uh consuming those those resources when they're limited, uh

and scheduling, you know, the pods that are using those resources. So, people are looking at other ways they can um they they they can use the API. It's perhaps worth adding that we explicitly have parameters that the user can supply when they are asking for That is something that hasn't been possible with device plugins, and I can imagine all kinds of interesting usages for DRA where these

custom parameters basically describe how to set up something, and then the DRA driver does it for for the workload. In in fact, that's going to be used. So, the the Nvidia um Well, actually, I think the next is the announcement. Yes. I'll I'll announce this, and then I can talk about it. So, the announcement you probably already heard, uh it was announced in the keynote, uh Nvidia

is donating their uh GPU DRA driver and their compute domain DRA drivers to the Kubernetes project, um which is awesome, and Google's going to donate uh a TPU driver as well. We We are That was announced on the Google Cloud blog, and I'm announcing it here. We're excited about that. Um yeah, thank you. And And so, you know, we're And another announcement is that the CPU DRA

driver, which I saw Praveen in the crowd there somewhere. yep, there he is. He That's a allows CPUs to be taken under management, which means we can do things like that internal node topology alignment and get optimal performance. But, to Patrick's point, the the NVIDIA uh if you've ever wanted to use like uh multi-instance GPUs or time slicing or MPS sharing, um this this driver it's it's

not part of the G It's not GA in the driver yet, but hopefully maybe next next release. Um I don't know. you'll be able to use those additional parameters that are associated with a resource claim to reconfigure the GPU to actually uh be, you know, dynamically at that time that the pod is actuated to to can be configured with those those type of time sharing parameters. So,

we have concrete use cases for those parameterizations already today. Yeah, we we are Well, we you just saw the table. The table with all the caps that had went into 136. And it's now my pleasure to call out that some of these were actually done by first-time contributors. Nour Eldeen, for example, he's I don't think he's here, but he he did his first contribution, major contribution, in

that space by implementing the resource pool status API, but a feature, a cap in that that long table that allows you to ask about what is the status of my devices, what what is still available, what's in use, and so on. And that is an alpha feature. Uh Shingo is here. He Well, perhaps Well, at the end we can give ask for people to stand up and

give them a shout out. But Shingo, he's it's not his first cap, but he definitely did some major the first contribution to DRA. John Hoon stepped up already. Some of these Some of Some of these people have been around, but some of these uh the first caps and cap is always a heavyweight thing that you If you start working on Kubernetes, you have to design it, you

have to dis- defend your decisions. It's a long process, but these people have all gone through that in the DRA space for the first time. That makes me as a as a maintainer of DRA very proud that we've been able to get some people more engaged and on board. So, let's continue. Um uh Alay is here. He has done something about metadata, so injecting information into the

container that helps, say for example, Kubevirt to work with the Um Pravin was mentioned, I think, for Well, something else in the DRA CPU driver, but he also is has to design something about native resources that makes goes along with it. And Subhasish, that's actually a beta contribution, but he took over. I think he's also in my As far as I know, he's not here. He's also

new to to Kubernetes and did some good work. Um I I like that we've been able to get a much broader contribution base, and perhaps next time next year it come someone else will be standing here and talking. if you feel like this is something that you can help with, that you're interested in, then we definitely have opportunities. Um we have more ideas, enhancement issues where we

don't have an owner right now, right? I'm sure we could find something. There are many many enhancements we want to do at that that yes, don't have owners. Yeah. And we are as a working group uh regularly meeting online. Well, we have a Slack channel. That's a good first starting point. Um all of these are links, so when you get the slides, you can you can follow

the links. Our meetings are a bit spread out right now. We have a bi-weekly US Europe weekly meeting on Tuesdays. Um but we actually in that slot we actually meeting every week. The other other slot is used to discuss native resources because it was such an important topic that really needed lots of insights and face-to-face time in in Zoom. I'm also running a meeting on Wednesdays every

second week where I'm in the early morning my time in Europe meet with people from Asia because we didn't really couldn't really good couldn't give them a good opportunity to participate. These These other meetings are just in the middle of the night, which is impossible. So, we we have are having this other meeting. It's a bit less uh less attendance, but I'm I'm showing up answering questions.

We've had some good discussions about for example consumer consumable capacity in that And I hope to keep that going. Yeah, so we were a little quicker than we expected. So, that leaves us lots of time for Q&A. Um any any questions? We can also go back and talk about some of these features perhaps if if that is of interest. >> happily do that. >> It is we

we we are we are very familiar with all of this stuff. So, it's always a bit tricky to figure out how much time we should spend on talking technically details of these things. So, we we gave you the summary and yeah, okay. Just a small question. partitionable devices was supposed to go GA, Uh partitionable devices uh, went beta. Um, it's been alpha for a long time. Um,

And then April was Some of these things have slipped a little bit. I think partitionable devices might have been planned for GA already earlier. That might be true. Um, I think the the So, in 130 So, partitionable devices we wanted to go beta in 135, but then we decided that we needed um, a change in the API, and so we went and did another alpha instead. Uh,

and that important change because the way the API was structured before, you were limited on the number of um, you basically you The way partitionable devices works is there's sort of a uh, a set of resources that individual you you you sets of resources, and you have devices that draw down on those resources. So, what This is how we implement the NVIDIA MIG piece. So, for example,

if you partition a a GPU, and you take a medium-size partition out of that GPU, the the full GPU is no longer available, and no long and the small ones that overlap with that medium one are no longer available. So, we have a kind of an accounting mechanism for this. And the way that we had structured the API in the 134 release meant that you had to

stuff all of the partitions that shared the same underlying resource in the same API object. And so, we we had a artificial boundary on how many um, on how we could scale it. So, we broke that, and shifted it around. So, now you can kind of scale these things arbitrarily large. So, the the Kubernetes API server is backed typically by etcd, which has size limits on the

objects, which actually distorts our API uh, uh sometimes in ways that are not user-friendly, but are just dictated by the the structure of the API. So, we had to fix that, which we did, and now in 1.36 it's going beta, um which uh should allow us to do whatever we need to. So. Uh I have a bit of a a meta question. Uh I was wondering how

you guys think about balancing offering additional features and and and functionality versus protecting Kubernetes from getting too complex and too coupled to hardware uh and and all of that uh and yeah, how you guys think about that? Yeah, I'll give an opinion. Um so, um I'm So, the resource slice API only really needs to be understood in full detail by the vendors who are using it. Users

don't They need to understand that there's a device abstraction, but they never actually, unless they're they're troubleshooting, they're not going to be digging in the troubleshooting weird scheduler issues. They're not going to be digging deep into that. So, I can accept more complexity in that particular API because it's a small subset of the world that needs to really understand it in depth. Um the the other side,

the resource claim side, it's important to try and keep a little bit simpler. the point of the resource slice API though is to not couple to hardware. So, basically, one of the goals we've had is we we saw all this change happening in hardware, all these workloads that couldn't just the unit you you you know, didn't want to just grab an integer count of something. but but

we didn't want to have to build knowledge of every vendor's hardware into Kubernetes. So, what the aim is is to build a kind of general-purpose modeling API for resources. So, it's essentially runtime programmable, you know, devices. So, there's limits to that. We we we make a modeling API and then we find out, you know, yeah, we can model PFs or VFs, but we can't model that do

consumption, but we can't model the two together, like I was talking about before, right? So, the modeling API is not expressive enough. So, we try to err on the side of expressiveness for the modeling API, the resource slice, and maybe err on the side of simplicity on the resource claim. That's That's how I try to differentiate, anyway. So, the initial structured parameters didn't support, for example, partitionable

devices. We basically designed There's something very simpler, but simple that we knew we could deliver, and then we allowed future extensions how to how to describe devices. But, it The complexity is going up. It's one of those things where we currently to some extent confuse potential implementers of DRA drivers. That's really the first hurdle that they need to get through when they try to design their own

DRA driver. How do I describe my hardware to the scheduler? What are the important attributes? What are the characteristics of what can be used together? This is where they need to understand well, core DRA, uh partitionable devices, consumable capacity is another way of having one device that you can kind of split up, but more arbitrarily. Basically, give out a chunk of, say, certain network bandwidth to one

consumer, rest of a community bandwidth to some other port. you don't always need to use all of those. So, that's the other confusion. People think that, oh, I'm writing a DRA driver, I have to support this this this a long checklist. It's not typically true. If you have something very simple, you just use some subset of the available functionality. But, figuring that out is um potentially a

challenge. I It's That's uh admittedly could be a problem. we are trying to help with that by providing the example driver. We are still kind of catching up with our own caps in the example driver. We we we agreed that we will have the example driver with different profiles. So, one profile might match say even partic- even some particular vendor hardware and behave exactly like that vendor

hardware. That is something that we allow. We haven't done it yet. But, we definitely want to have profiles that showcase consumable capacity, which is a pending PR, but it just hasn't been merged yet. Um in that example driver so that vendors and users can see what it looks like when it's implemented and then figure out whether that's something that matches their hardware. Um there's one more thought

on the whole Are we making Kubernetes too complex uh question? We have done all of these changes because we saw use cases for them. So, we are doing it because it's necessary. And to some extent, the Kubernetes ecosystem, I think, now has to catch up a little bit. We were not not just a for auto scaling, for example, we kind of have solved it for the auto

scaler that is built on the Kubernetes source code. And that's a bit easier for them because they just use the same logic that we built into the scheduler also in the auto scaler. it becomes more tricky for other downstream projects or projects built on top of Kubernetes like Q, carpenter, all of those are still adapting to a world where work more workloads are going to use DRA.

But, there's I think no way around that. we can't avoid doing this because then Kubernetes as a as a project will not all the AI workload needs that we are seeing. Great. Thanks very much. you've mentioned um the effort to integrate what's currently considered native resources into the and maybe my brain's going towards the array abuse, but um is um inclusion of those native resources and their

features potentially also alleviate things like node feature discovery where you're spamming your nodes with like 40,000 labels declaring like I have this CPU feature, this CPU feature, which in turn is again needed for stuff like elastic search for example, where you need to schedule your on a node that is capable of providing a function while it's not really taking that away from a pool of available Right.

It's not It's not a resource that you're consuming, but it's It's It's a feature that's a function of one of the resources. I would say that first of all I think I'm going to take those in two separate Even without native resources, already with GPU for example, you if you if if you had a fully if we were if we had We're creating clusters now, you wouldn't

need to label your nodes with the type of GPU that's in it because the you could just ask for it that type of GPU in your resource claim and we would already exclude any nodes that don't have it cuz we have that metadata about the GPU now. So yes, in the sense of if we were to So that's the first answer is we already can do that

kind of like get rid of node labels that you that are only used for scheduling because you could do it through the through DRA. Yeah, but >> but with native resources like that's not going to be we're not going to do that typically for all nodes. I suspect that I mean we might, who knows, current use cases around native resources in DRA for are for you when

you're using Slurm, um which wants hyper control over which which CPUs get used. Therefore, uh use cases that are sophisticated and you've got to line up your GPU and your NIC and your CPU in the same like socket or see PCIe bus, NUMA node. >> Same NUMA domain. Right, exactly. So, they're geared around those those um but but but I mean we'll see. I think we're it's

early, right? Like you could eventually you you those CPU devices will have those kind of metadata attributes on it and you could say I want a P core or I want an E core or you could say I want, you know, ARM, I want whatever. You you could do all of that in the uh in your your resource claims. So, the answer is kind of like yes,

maybe eventually, but we're still early to know whether we would get to that level of of of broadly deploying the native resource in DRA piece. Okay, thank you. Yeah. And all DRA drivers are optional, so you don't have to have them in the cluster. Hi, thank you. Uh I have a question about about the next slide on the what's the next. So, the D2 I I have

a question, so how would the working group can push forward the D2 operation? Do you have any idea how DRA can help with that, especially the device failure handling? So, two points there. One is that the working group is broader than DRA. So, like the like we may not use DRA in that device degradation thing, but we believe that that the right people are in the working

group. It's the It's about device management, and DRA is just the scheduling part of device management. So, it may or may not be DRA use the device abstraction. It probably would, but I just want to make it clear that the like there's a slight difference there. Two is these are very This is like a very early ideas of things we think the working group might be able

to contribute to. Um we do have some ideas. Um like one thing Patrick implemented taints and tolerations for devices. So, what that means is that if if you you have a controller that notices or your driver that notices that a device is performing poorly, it can add a taint to it, and then any pod that's using that device will be evicted, or or you can add a

no sched taint, and and no new pods will allocate that device. So, that's a way that you can kind of take a device out of service with a with a taint. but we have other ideas about um how might we propagate a signal from the device up through the to the resource claim of some pod that's using it to the application that's actually using it so that

the application can take some action, but it's all super early. This is like we don't even have caps written for this stuff yet. Yeah. Thank you. So, last question. I think we got a signal that we are about to end the session, but I think we can take one more. Uh so, for the extended resources, we kind of give people a way to continue using count-based resources

for GPU The question like what we struggle with is when you have that feature, why would somebody not keep using that versus using resource claims and like >> They're fine to keep using it if they don't have any specialized if they don't need the expressive power of resource claim then they they just keep using it. That's not a problem. But like what what mean like so I

guess so for stuff like DRA or sorry for GPUs like time slicing and MIG are like harder things to support. So can so the so the resource claim sorry I if you don't mind okay. Uh the resource claim has a few other things in it right? It has uh configuration you can attach it has um uh which you could you could also put it in a device

class and access through the extended resource. That's possible. Um it's got uh the prioritized list type of functionality and it's got constraints across multiple devices when you have when you're selecting multiple devices and it's got consumable capacity. Um so none of those things are accessible other than config you put in the device class. We made that accessible through that. Okay. >> And sharing between pods and containers

is a lot more flexible and more more explicit with resource claims. So all of these are things that you may need for your workload and then you have to switch to resource claims. All right, All right. Well, thank you all. Thanks for the interesting questions and