Slinky Expanded: Slurm, Kubernetes, and DRA - Praveen Krishna, Google & Marlow Warnicke, SchedMD LLC

This talk introduces SLURM, a high-performance computing workload manager that assigns resources in large clusters, and Slinky, a set of tools including slurm bridge and slurm operator that integrates SLURM with Kubernetes. The speaker explains how slurm bridge transforms the SLURM controller into a backend scheduling solution for Kubernetes, enabling the management of AI and machine learning workloads. Furthermore, the discussion covers the CPU DRA driver, which enhances dynamic resource allocation within Kubernetes, allowing for more granular control over CPU assignments by considering microtopology and specific hardware attributes. The presentation also highlights recent developments, like SLURM's external node support, which allows concurrent execution of Kubernetes and SLURM jobs on the same nodes. The speaker concludes by demonstrating a live workload application, showcasing the practical implementation of these integrated technologies.

Hi, I'm Marlo Boricki over at Nvidia. I'm the lead for Slinky. And >> hi, I'm Pravin. Uh, I work as a software engineer at Google and I've been like working on CPU DRA stuff. >> Um, how many people in this room know what SLURM is? How many people don't? Got. Okay, it's like 20%. So, we're going to run over this really quickly. Slurm is a leading high

performance computing workload manager. Um you can think of orchestrator scheduler. Um it prioritizes and decides which jobs run on which parts of the cluster and it has a resource manager which tracks node state and resources and launches jobs and assigns resources to those jobs. Um this manages majority of the top 500 computers and it also manages most AI ML training workloads. So training usually not all systems

use use slurm but most of them do. It also scales beyond 15,000 nodes in a cluster. Um I will make a quick note. It's a GPLv2. Um your Linux kernel is also GPLv2. What is Slinky? Well Slinky is a set of tools. Um the two main ones, well the two ones today, there's a group of supporting tools, but the two main ones people use are slur operator

and slurm bridge. We're going to cover slurm bridge here. Um what is slurm bridge? It's we're basically took the slur controller and turned it into the back end as a kubernetesuler. Um so it uses this set of functions. If you're not familiar with scheduling, that's okay. We only have so much time. um we translate the pod resources into a slur placehoer job once it's submitted and that

determines when and where the pods run and that holds the resources for the time. This is called an external job in slurm and we will also leverage the new workload API uh in 136 soon um scheduled by slurm launched by kublet. So if you're running inside Kubernetes, we schedule but we need the kublet to appropriately assign the resources. This is the Kubernetesuler framework. Um the thing on

top we use a subset of those items that we sync in. So we are using the real Kubernetesuler framework with you know the swarm controller in the back end. And then this second part is the flowchart um which basically we go through we find out where slurm would assign it we put in the annotations with post filter and then we go back through it and then if

we have all the annotations in place then we run the job. Um this is another cut. Um the interesting thing here is your bridge node has both kublet and slurmd running. The slurm node has slurmd and the kubernetes kubernetes node had kublet. So before we made these changes, we had to have slurmd and kublet and then it would be marked slurm or kublet and then you could

only run one type of job or the other on each node and then the underlying piece whether it was slurm d if it's a slurm job or kublet if it's a kublet job would be running would be choosing the resource assignment. >> Okay. So uh now let's see like how the resources allocated or assigned uh by slurm or slurm bridge gets translated to like what is allocated

on the node. Uh as a part of this like we have the CPU DRA driver uh which basically is like using DRA uh which is DRA stands for dynamic resource allocation which provides like a flexible Kubernetes API for requesting allocating and sharing hardware resources and this has been like G since 134. Uh so yeah like cube uh CPU DRA driver like is one plug-in that is using

DRA to manage CPUs. Uh why are we building the CPU DRA drivers? So basically currently we have like cublet that manages like guaranteed CPU allocation or uh cublet is able to pin CPUs to a workload uh depending on like its QS class and some node configuration which is called static CPU policy. Why we need DRA is like in the cases of like a IML or HPC workload

this is insufficient because there are two main gaps with what we have currently in cublet which is one it lacks awareness of like the micro topology on the node. So basically it treats like all CPUs to be the same. Uh and the second uh gap is like currently there is no way for a user to make very specific requests. So a slurm like job or slurm bridge

cannot request like specific CPUs at a numa node or a socket or a L3 cache specifically. uh which is why like we started working on the CPU DRRA driver with uh basically using DRA which provides like a more expressive way for a user to specify like what exactly they need and again this can be done using like cell expressions which DR supports to specify like whether a

workload needs like efficiency course performance course like course in a specific NUMA domain or socket and things like uh using DRA also makes it like extensible in the sense that like if we want to introduce like new uh like parameters based on which we want to schedule we can add new attributes in DRA like resource slice which I would get to uh and basically without touching like

any of the core uh cublet or scheduling logic it enables us to specify these in the workload and like schedule based on uh these attributes. So yeah, this is how it looks like if you're already using like DRA for DRA is currently mainly used for like GPUs and like highspeed nicks. If you are already using that like uh CPU DRA driver the the resource claim should look

like pretty much familiar. So uh basically in the pod you have resource claim. Uh within the resource claim you can specify uh the the all of the resources that we need including CPU in this case which I have in the example and along with just specifying like a count of CPUs uh which we normally do in the port spec DR lets us uh specify like more attributes.

In in this example, I have like uh like an attribute called core type and I'm requesting like five cores which are performance coursees. Uh so yeah, how do we get it onto the cluster? So this is currently done through a demon set which is pretty much the standard way uh of like getting any uh driver. Uh and once a demon set is installed, it scans the uh

hardware and it publishes like resource slice and uh workloads can use them to like request uh guaranteed or pinned CPUs through resource claims. One key requirement for getting like or making the CPU DRA driver work is like we would have to disable the cubelet part which also manages like exclusive or guaranteed CPUs which is the static CPU policy. So that needs to be disabled to make sure

like we don't have like two competing or two things competing to allocate like the same uh underlying resource. Um so yeah like within the CPU DRA driver we have once the driver is up we have like two currently we have like two ways in which like we can uh represent resource. So the first one is called like individual mode where each CPU on the node along with

like all of the attributes that it has is represented like independently on the resource slice. So in this case like you can see we we have like CPU device 0 through 48 and each of them have like uh core ID what's the L3 cache socket ID and so on and we can have more uh this mode is specifically useful for slurm use cases uh mostly because like

we can have slurm bridge or like slurm decide what exact CPUs that it wants to give to a workload and we can create a resource claim to inject this uh within the pod and basically the node the cublet would uh cublet and the DRA uh driver would uh enforce those allocations. Uh but yeah this is preferable only when we have like external scheduulers like slurm uh which

are which is managing like uh resource allocation but in most cases we would only need to specify like request based on properties. Uh in that case like like in those use cases we can use this mode called like group mode which uh pulls resources into like or pulls CPUs into logical blocks which can be based on like node topology like numa socket l3 cache or property uh

like efficiency course or performance coursees and this underly this uh like internally uses this DR feature called like consumable capacity which lets you specific specify a resource claim to draw down an arbitrary number of like CPUs out of the uh advertised like resource uh like pool. So in in the example like we have two devices one for each socket and like on each socket if you can

see like there is this capacity attribute that is publishing like 32 CPUs per socket. So yeah, now with like all of these uh information being made available to the scheduleuler, you can have like a pod with a resource claim which uh specifies like request for GPU, nick, CPU and all of them can be matched based on specific attributes. So in this case you we have like a

multisocket machine. We are requesting GPU and nick on the same PCI route and like also CPU and nick on the same numa node like which is currently not possible like with the uh cublet based guaranteed CPU allocation. Uh yeah like coming to the architecture at the node level basically we have like two core components once a demon set is installed like we have a a part that

talks to the cublet which is the dr cublet plugin uh and uh the nri plugin. So when a pod lands on a node if it has resource claim uh the like cublet make a makes a gRPC call to uh like the cublet plugin which prepares the resource claim. So basically what that does is like it it makes the allocation decisions but it does not enforce them yet.

Uh and all of those allocation decisions are like written to the CDI file or container device interface file. So when cublet is calling like the runtime API uh the the CDI file is like injected and that is what uh gets used within the NRI plug-in or the node resource interface plug-in uh using which like we can intercept like container life cycle events in this case like we

are uh intercepting create container calls and in that we we use the previously like uh decided added CPUs and uh yeah like uh basically through croups we enforce them onto the container and along with uh like giving a container guaranteed CPUs the other key thing the NRI plug-in does is like once we allocate some CPUs to a specific container we also need to go back and make

sure like for all the remaining containers we kind of remove those CPUs so that there is like no interference from other containers so basically yeah it does uh both allocation for resource claims and it also manages like shared CPUs for rest of the other containers. um so let's talk about slurm bridge with dr. So with all these changes we were able to get get it so the

kublet owns the nodes so we don't have to share anymore with kublet and slurmd. um external node support was added to slurm so the controller can schedule to nodes it doesn't manage um dra so that was a separate change that was made by slurmur so the last version of slurm has external node support um dra drivers have become available so like the cpr driver um we've been

working with uh Google and redot to get this thing out the door um and intel to allow us to propagate the chosen fine grain assignment to the controller to the node because slurm knows the slurm controller knows exactly what cores it wants to assign that assignment has to be available. Um so as a result we no longer need SLMD everywhere. Um so this is kind of a

slide from last year but this also shows you where the changes are. Um so each node can run both slurm and kubernetes workloads simultaneously. So this makes a big difference. So you no longer have to say is it slurm is it kubernetes I don't know. You don't have to choose. Um the Kublet will manage all node resource assignment for all workloads because of DRA. Um node controller

can now add and remove external nodes. That's part of the slurm changes I referenced earlier. And we can now handle multiple pods per node which is alpha. So if you go try it and things break, please tell me and we'll fix it. Um device plugins are supported and also DRA is now supported. So we have three drivers that we currently support. We will be adding support with

other ones as they become available in the community. Um, which is CPU DRA driver, the GPU example driver because we needed something that we could test with. So that's what this demo is running is those first two. And then the NVIDIA GPU driver, which again is alpha. We haven't done enough extensive testing for us to be comfortable. Um, so now we're going to try a trick called

a live demo. It's going to be fine. Let me turn that off. I thought that was It should have been off. Okay, whatever. Just going to show you what's running here now. Um we have these workers. So those are running on all the nodes. So that we have because we have operator running concurrently. Um this is out of control. That's better. Okay. Those are just the workers.

Those are just nodes. All right. I can tell what's going on because I have glasses. All right. Here's our DR CPUs. This is a nine node kind cluster. That's why it's a mess. Um, we have things like job set running. So, um, within Kubernetes, does anyone know anything about job sets, pods, groups, those things? There are certain paradigms that currently exist that let you run multiple jobs

or multiple pods at a time. So, there's job set, pod group, leader worker set. There's a new one called the workload API, which is actually pretty cool. You should look at it if they just released it, I think. so we we run these various controllers within the bridge so we can handle all of these multi-node uh assignments. Uh, just ignore all my Slack messages. So we also

have a dashboard that lets us see what is going on with Slurm. Why are you doing that? Try again. Can everyone see that? Okay. Is that very tiny? I can't tell. This looks all So, we're going to run a workload. Um, and we have these all in the repo. So if you want to play with it later. Nope. We want the DRA examples. Let's see what we're

running first. Yeah. So, we're asking for CPU. We're asking for a GPU. And you'll see more than one of these assigned because currently we're taking the whole node because we're doing exclusive. But if you have that particular feature of slurm off, it'll give you one. And then we have the sleep just so we have enough time to actually go look at what's going on internally in the

pod. So, we'll see that pop up over here. It starts running. So, it's pending. Let's give it a second. Come on, run. Do what you're supposed to do. Oh, there we go. It's getting assigned. Container should be created any second. There we go. So, we have a neat resource claim that was made. Hooray. Slurm did that. Um, here's our resource claim and move that over so I

can actually look this. So, you'll notice CPUs have been and GPUs have also been assigned. And this will run for five minutes. So, it's actually not going to finish before we're done. But this is a a technical job that's running. That's the demo. Uh, do we have any questions? Uh, somewhere there's a mic. Oh, do you want to Thank you for the interesting call. I um uh

presentation. So I have a question regarding the DRA for the CPU. So you know in C++ OpenMP uh is a bit tricky because sometimes it's not C groups aware um and we need to override with environment variables that we inject through the slurm. So how do you expect to deal with this without us to decide manually per job um where the specification of the number of CPU

cores uh is being set for the application that runs there. >> So I think like in the slum case slurm is what decides the CPUs and like in like creates a resource claim with exact CPUs and injects them into the potspec. the DRA driver just like respects whatever is there in the resource claim and through NRI actuates them on the at the croup level. So um yeah

>> yeah but the container so let's say that the container is spawn on a machine that has 200 cores >> and then the container just gets the injection of the croups inside of it but the C++ code actually can read the CPU >> and actually spawns more processes >> because it happens in the open >> I've seen that. So do we have any solution for that or

did you find >> there? There is a way to do that but I don't remember the precise way because we had to we had to figure out exactly what CPUs had been injected. So you're talking you're inside the container and you don't see what CPUs are available. >> Yes, >> there is a way to do that but I don't remember it precisely. Um if I get your

info after I can look it up and send it back to >> you. Because I know that in slarn you can actually define the MPIs >> and then the executor actually is able to define and constrain the process itself. But of course that is installed on the machine directly and now we have this layer of container and croups is the only way that the process is seeing

instead of having like the control directly from the OS level. >> Yeah. And you have no way to see that from inside the container. Yeah, I'm familiar with the problem. >> Thank you. says how many >> but which ones >> inside the container. >> Okay, we'll talk after. Yeah. >> Um I was here CubeCom well not here but London last last year when you were talking about

this and it seems you guys have come really far. It's very impressive. So, uh, thanks. Uh, I was wondering how you see Slurm as a as a citizen in the e Kubernetes ecosystem now. Is it now like first class citizen or is still a way to go? How do you guys see that? >> I think we're pretty close to there. Um, we need to get the networking

aligned I think other than that, I think the other pieces are in place. You just put them in the pod. >> Yeah, great work. Thanks very much. >> Thank you. Any more questions? >> Yeah, he has one. >> Question. How well does the work integrate with many systems out there which we benefit a lot from >> it's auler so if you're just using it like auler it

should just work um we have used kea to see it uh auto scale and we've played with it that way. So I think I think it should just be fine. >> Yeah. Just be loud. >> Uh it's still configured as usual. So if you have a topology, you can still put in your topology. We do make some assumptions with our containers, but if you want to configure

it custom, you can do that yourself. There's still a slim config. >> So, run their current config and they should be okay. Yeah, >> we could. >> So, uh currently we are using uh slum operator to spawn slur inside our APC cluster. Uh the drawback for slumbridge from the last time I checked is that it will allocate the whole note for slimbridge. >> So we just turned

that off. Um but we haven't extensively tested it. So I know what you're talking about. Um but you should be able. >> So now with dr we can schedule both kubernetes and slum job without put it slurm into maintenance mode. Correct. Correct. Yeah. Okay. Cool. Thank you. So yeah like this is the repo for both Slinky and CPU DRA driver. this at least the CPU part is

like an ongoing effort and there has been like some work within the community to basically uh expand on this like include things like memory and also there is work to build like tighter integration like within the uh Kubernetes ecosystem at the scheduleuler level and the node level. So yeah, like we are open to any feedback. Uh try this and like let us know how it works or

uh what we can do to make things uh work >> Has the has the CPU driver been released? >> Like there would be like a release probably we last PR is getting merged. So yeah, probably today >> you're going to have one more. So this slide the the QR code points to like the first release uh which is what like uh slurm is currently using.