Tutorial: DRA-matically Simple: On-Demand GPUs for MLOps - Doug Smith & Miguel Duarte Barroso

In this workshop, Doug Smith and Miguel present Dynamic Resource Allocation (DRRA) in Kubernetes 1.35. They discuss the evolution of resource management, highlighting DRRA's ability to improve resource efficiency and simplify administrative tasks. DRRA allows application developers to express hardware requirements more intuitively using a standardized YAML interface. The speakers explain how DRRA overcomes limitations of device plugins by enabling intent-based resource requests and enhancing observability of resource usage. The workshop includes a hands-on lab where participants can SSH into a lab environment to explore and implement DRRA concepts in a practical setting, further emphasizing the role of administrators and developers in utilizing DRRA effectively.

Yeah. Awesome. Hey, thanks everybody for coming to our workshop for Dramatically Simple. What we're going to do today is a workshop, but we'll have a little bit of time before that to kind of explain. I'm Doug Smith and I am joined here with Miguel. >> Yeah. Hi everyone. Great. So the kind of agenda that we're going to have is we're going to give you kind of a

introduction to DRRA dynamic resource allocation and we are going to talk about some of the like how we got there and what we think is compelling about DRRA which is just GA in Kubernetes 1.35 and kind of has been like in the works for a number of years and I think what came out of the work is pretty compelling and once we get sort of the basics

done um which we think will be in the 30-ish manage range then we're going to open up a um workshop style and we have got a lab that everybody can SSH into. So yeah, quick show of hands for a sense of it. Okay, number one is who here is this their first CubeCon? Yeah, nice. Awesome. All right, so we will try to give you uh the best

experience possible. Some of it is a a bit of an experiment like we haven't really scaled it um this big, but we think it's going to make it even more interesting to have more participants. Um, so to join our lab, we have a couple options, but show of hands, who has a GitHub account? Yeah. Awesome. All right, cool. Um, of those, do you know if you have

a public SSH key associated with your GitHub Okay. All right. Awesome. Um there's another way to do it too, but uh if you've got that, it should be like bone simple. Um so let's And just remember if you don't have any of this and you still want to follow through with the demo, you can just join up with somebody next to you that has a laptop and

required things. So you can at least follow that and make a friend. So perfect. >> Oh, motivation. That's me. So first thing we will try to do is to explain why should you care about this and hence motivation. So in Kubernetes the first thing that kind of led to the invention or to the evolution of uh dynamic resource allocation is semantics or actually the lack of proper

semantics. So what you have right now is something like you can request a resource but you have to explicitly tell what the resource is and how many of those you want. So if we for instance take the metaphor of a library this would mean that you get to go to the library and ask for a specific book and the librarian will give you that. You cannot go

to the library and request like I don't know a book on the destructive nature of obsession. You can just request for Moby Dick and here it is. The second thing that DRRA brings to the table that you did not have before is set of tooling or features that will increase your the resource efficiency within the cluster. with this. I mean, it's easier to get this with an

example. So, let's say you have a very expensive GPU costing I don't know how that how much it would cost, but like that GPU would have like a 100 gigabytes of VRAM and it's located on a node and somebody creates a workload, creates a pod that is using that resource, but will only use 5 GB of VRAM. So 95% of your super expensive resource is wasted until

that workload is gone. So we want to also um have something else that allows for a resource to be shared. And the third part of motivation is something to simplify your admin life. So I guess some of you uh work in administration administering the Kubernetes cluster and I guess that every now and then somebody tells you uh I have a pod that is stuck in scheduling for

the last 5 hours and I need this to run now what's happening so you go figure what's the pod you understand okay the pod has this resource which nodes have these resources these four log into each of the nodes who's using it's very very very cumbersome some. And what we would in a perfect world like to have is something like a digital receipt that says this resource

is being held by this thing since 9:00 a.m. this morning. And having these kinds of uh insight into the the actual state of the hardware resources is what we're after. uh and here I'm trying to kind of define the exact problem we want to have. So we want to in a Kubernetes native way have a common interface where the cluster can declare its capabilities. The admin can

write policy that indicates how the uh resources will be consumed and the application developer will get to express their exact hardware requirements for their workloads. That's literally what we want using a common standardized YAML interface. And with that, >> thank you. Okay, so speaking of kind of this Kubernetes native approach, what we had in the past and we still have and you can still use it's a

totally a legitimate uh solution is device plugins. And this comes from I 2017 2018 era and I think over time we've kind of evolved some of these things that earlier on we thought was the right way to do it. And we have a better when we have a better way to do this now. So device plugins idea is like take any arbitrary kind of device a consumable

physical resource and give a way that you can plug plug that in. However, as we've kind of grown with Kubernetes, we've realized that like this intentbased um kind of structure that we would like to have in Kubernetes where you you know you have a YAML spec and you say this is my intent to consume this many CPU cycles and I want to get scheduled according to that.

Like Kubernetes is awesome at that. It's a great scheduler. Well, with device plugins, you had to baby things a little bit more. So, you had to kind of like label Oops. Um, you kind of had to label all your nodes um, in a lot of cases. And it was way better for something that would be, I guess I would say, a like homogeneous or kind of congruent

environment where like everything is the same. Um, but the real world doesn't work that way. like like having nonuniform resources is really common. And if say you're using something like GPUs, this is like technology that's changing all the time, you might get another rack, different GPUs, your workloads, your applications that you're running on Kubernetes, those requirements might change, too. So that's where we start to get DRRA.

Um, and with DRRA, instead of having to say like I want one specific GPU from one specific node that's known, um, where kind of your admin was more like, um, managing that cluster infrastructure directly, now you can express it in more of an intentbased way to say like, I want a GPU that meets these requirements. Um, it's also worth noting that uh, you know, we're talking a

lot about GPUs, but a lot of the history for Miguel and I especially uh, comes from networking and similarly you have like purpose specific hardware that you want to use and consume. So you can take these kind of ideas for DRRA and apply it to essentially any type of consumable resource um including virtualized resources to some extent. Okay. So similarly um you've got this method where you're

like okay I'm going to wrangle all of these nodes by hand. I'm going to label say this box has A100 GPUs this other box has H100s all of that pain for admins. We want to say something where you can like express more of what you want. So, how much memory do I need? Um, what driver versions may I want to have? Maybe you have GPU workloads that

rely on a specific CUDA version or something like that. You want to be able to express that without having to go back to your admin and go, hey, I need CUDA 128 for this or whatever it may happen to be. Instead, put it in a YAML spec. Um, so the kind of the way that you can do that is with a cell expression. So instead of in

device plug-in land where we would have said I want uh A100 from GPU server04, now I can say I need greater than 24 gigs of memory for for this. So that's the way that I can express my intent without having to say like I need a specific thing for my infrastructure because this is something that we like to do. So you can have your way of managing

your infrastructure and you can expose a way to get your um scheduled onto the right places without babying it. remind yours. >> So okay, another aspect of this and this was something that kind of came later in the development of DRA as a feature for Kubernetes is this kind of um consumable capacity. Um, and this would be so that hey, you're taking this really expensive GPU. Maybe

it's a B200 and it's got 280 gigs of RAM and then you run a workload that uses 48 gigs of memory. It's kind of a waste, right? So, this also gives you a way to kind of slice and dice um that kind of stuff and you can integrate it potentially with like vendor specific type of um partitioning as well. Okay. And um thanks. No problem. >> Another

thing that is different in Dr. land is how it manages the life cycle. So before in the device plug-in world what you would have is the pod would uh reference a device request number of devices theuler would schedule that on a node it would bind and the moment the pod disappeared the resource would be deallocated. So in a way the life cycle of the device is tied

to the life cycle of the pod which could be exactly what you want. It could also be something you don't want to have. So the RA allows you to do something different if you want to. You will tie the life cycle of the device to that digital receipt. It's kind of uh when you left your cloak in the cloak room like that thing is being taken care

of and managed by someone else. Once you go there and you return like the paper they gave you, you will get the the receipt. When you once you hand out the receipt, they will give you your code back. So this means that you can for existence create that digital receipt and which can exist before the workload and can persist after the workload dies. So for instance, you

can create a workload or actually request the the device allocation that would be uh allocated to this receipt. You would then create the workload. It would bind to the device. You could then remove the workload meaning the pods and you would still have and hold that allocation and the scheduler would be prevented from assigning that to someone else. This could be a use case you want to

another thing by integrating directly with Kubernetes uh you get like resource quotas. You can enforce namespace quotas on your devices to uh ensure that uh proper uh usage or uh fair usage uh within your cluster. I guess that's more the words. And finally, it's not that the user themselves, the end user, the application developer that chooses to deploy their apps in Kubernetes is actually pointing at resources.

What they're actually pointing at is kind of a curated list of uh device of uh device configurations that is provisioned by the admin. So the admin looks at what is available, defines some properties and that's what the users will actually request. Another thing in the DR land which is incredibly better is observability. Before you had no way of knowing who's using what. With this you do have

and you do have because on that digital receipt that I keep mentioning uh you have it has a status sub resource and that status sub resource will say this thing is being taken by this pod right there. And uh another interesting thing is that the cluster introspection thing that is running in the background is actually able to uh monitor the device the health of the device and

will update the status of this uh receipt indicating like everything is fine or no the driver just took a tumble and this thing had to restart. So that will be seen in this uh And to put everything uh together, it's very important to understand that this feature actually require requires three different roles to coexist interact with each other. It's the driver which is a piece of software

but somebody has to develop. It's the Kubernetes admin and it's the application developer. we will see each of these roles now. >> Cool. So, we're kind of I think depending on the way that everybody in this room depending on your concerns and what your relationship is with Kubernetes. Some of these may be more interesting to you than others and some we'll dive into a little bit deeper

and some not as much. But we'll touch on kind of all of these things. But where it will usually start in order to start using DRRA is that you're going to have a DRRA driver and you might consume one that comes from your hardware vendor and you might just use that directly. um or you may have a particular opinion of what you think should be done with

your hardware resources in which case you could develop a DRA driver. So as part of the lab materials which we'll get into um there is actually a completely customized DRRA driver that I built and you could use as a reference. There's also the like um example DRRA driver from the special interest group that uh or working group that came up with DRRA um which is really really

excellent and easy to follow um once you have the basics down um but I won't go into it deeply deeply here but the main kind of thing to from this side is that as from the driver perspective and the driver developer perspective um we use kind of a like kitchen analogy later, but this is sort of your how are you going to stock the inventory for the

kitchen. And so this is going to be the part that looks around the pantry and figures out what you have and kind of makes that available. And uh it's also where you're going to have these kind of device attributes and capacity um type of factor that the driver is going to pick up on and it's going to expose that to Kubernetes so that you can later um

uh do this. So yeah, this slide the gist is essentially all right, you've got your server rack and your driver is going to look around in this stuff. It's going to pick out all the different parts that you might need and then all of this here totally customized. This could be whatever you happen to care about. So like I was saying like okay it's a GPU workload

it uses CUDA and you're like hey part of our stack is using CUDA 13 another part of our stack is using CUDA 128. So maybe that's something that you care about and you're like oh I care about what driver version is there. Um, I care about how much bandwidth is available for a particular network device. If you're using something other than >> how many num nodes I

have, uh, what's the throughput of the these network interfaces that I have? >> Totally. What color are the LED lights? Like what is it? What is it that you care about? It could it could literally be anything. Um, but yes. >> Okay. The admin now. So the admin is the next actor in our pipeline. So as Doug said, the driver will kind of stock the inventory for

the kitchen or the restaurant. And in this case, the admin is literally curating the menu. He's looking at the things that are available and he's saying okay with this I can make uh some sushi rolls uh Wagyu steak or whatever because I have the ingredients for it. So the admin will essentially create um a Kubernetes resource called a device class which is here. And this is important

because this is what your users the end user meaning the application developer will actually point to when they uh present the the receipt. It will point to a device class. Another thing the admin will be able to do now is to audit the existing inventory and look at what's available. They do that by inspecting the resource slices uh resources that are available on the cluster which indicate

what resources are available on a certain node. Finally, they will have to or have the ability to monitor and troubleshoot orders. So they will look at the resource claims and they will figure out in a very simple way which resource is being used by whom and uh finally once they figure out like a pod is stuck in scheduling for the last two days. The pod is requesting

24 gigabytes of VRAM they will look okay so uh user A and user B are hogging all the GPUs in this cluster. So you can they can just kick them out in a simpler Uh so in in a nutshell this device class it pretty much like is abstracting the hardware details. It's policy definition again uses uh sell and all that and the users point to that. So

in a way like the admin here is the chef of our uh little analogy. So yeah stepping up the stack kind of. So low level driver developer. Then we've got your Kubernetes admin who cares about the details of the cluster. Then you have your application developer. So this is somebody that really they care about like their purpose specific application and they just want it to run on

Kubernetes, right? So this is if you're a Kubernetes admin, this is kind of your internal customer if you will, right? like as a application developer like you don't want to have to say like hey I need a GPU on GPU server08 uh you just want to say hey I know what the requirements are for my application and I just want to put that in a YAML spec

and I want that pod to come up. So this is kind of like at the highest level. typically as your kind of app developer, you kind of just want to like launch the pod and maybe get feedback about is it running or not? And then as the admin, if they have a question, you want to be able to answer what the heck is going on. So that's

where you'd get kind of your uh digital receipt. So yeah, in this particular diagram, I'm trying to say like, hey, I'm a developer and I just essentially want my pod to come up. Um, so it's going to put in the order menu item that you want and get that associated with the pod and boom, comes So trying to wrap everything together here. So we've seen we have

three different roles. We have three different uh resources in Kubernetes and how do all of these things play along. So remember we have the resource slice which is created by the driver which introspects the cluster and kind of um is analogous to the inventory that you have on a restaurant. You then have the admin which is the chef that provisions device classes according to their needs. And

these are basically entries in the menu. Like I have sushi rolls and I have Wagyu beef. Finally, we have people that come to the restaurant and put up their orders via the resource claim. This is a digital receipt. I've been speaking all this time is called a resource claim. And this is how they actually point at the device class they want to consume and can give additional

configuration like I want the Wagyu beef but I wanted medium rare and I wanted to share with these five imaginary people right here. At the end of the day what you have is like theuler will make everything uh work together and it will also feed this back to the inventory. Like if you only have five Wagyu beef steaks and 200 people want that, 195 will need to

wait until there's more Wagyu available. D. Okay, so there's now we're entering the part where we explain two different features that are specific to the RA and um fulfill kind of uh two things we've spoke about in the motivation. The first thing is consumable capacity. This is the feature that allows one big very expensive GPU card. Let's say for the example it has 120 gigabytes to be

used by more than one workload. So the way it would work is let's say you have these three workloads each of them with a different requirement. it will slice this up and it would still have 70 gigabytes available for other workloads that would be scheduled afterwards. Uh so in a nutshell, what this feature allows you to do is the DRRA driver we've seen before will advertise the

capacity. It's just one more attribute in the uh resource slice, the capacity of that particular device. The scheduler will do the math and it will allow allow a device to serve multiple claims. That's literally it. That way your very expensive hardware can be more efficient by being utilized by more people because the requirements actually match. Uh it also has two extra mechanisms that I find extremely interesting.

First is request policy in which you can for instance define steps. It's like if you define like a five gigabytes step uh increment that means that you can request 20 gigabytes but you could not request 25. You could request 20 25 30 so forth. Another thing is like distinct attribute. This is very important for redundancy reasons. And again it's easier to explain with an example. Let's say

that for for redundancy you want your pod to have two networking interfaces. uh but for whatever reason these network interfaces will be spin off like will be MAC VLAN interfaces from the same physical interface you don't want that so you just put distinct attribute and that way each of these uh sub interfaces will be spawned off different physical interfaces that way you will actually implement redundancy the

other one is top topology awareness so you have for in this picture you have two different numa nodes and for the sake of the exercise let's say you want to have a GPU and one nick two things might happen here the first these are scheduled in the same numa node which is good because well they're inside the same numa node so they don't have to go via

this slow buzz over here but another thing might happen which is they will be scheduled on different numa nodes so they have to go via this very slow interconnect thing does wasting like 10 to 60% of the capacity of this again very very expensive hardware uh dr exposes topology awareness uh in the resource slices you can make that happen in a very simple way so you just

have here like two requests you see you have a request for GPUs you have a request for an accelerated neck and you can define constraints and the constraint we want to have is I want these two requests to match the attribute numa node magic. What would happen? Oh, this is slow. Okay. What would h happen is this won't happen but this will which is the good thing

which will lead to a huge performance boost and we finally reached the part of the workshop time we we've only lost two people so far. Okay, that all right that's pretty good. All right, so I promise the workshop is coming soon and we can roll up our hands and have some fun. Um, the way that I've sort of tried to organize this is like a sort of

like levels of abstraction. And in order to do that, um, it's kind of a like RPG that you get to play all of the different roles. So, I kind of wanted to start um the adventure with a little bit of kind of your like cyborg grandpa telling stories around the dumpster fire. So, that's me and the the gist is last year I was um uh working with

my team. We work on VLM and I'm going to talk about that a little bit, but we have people with like varying sort of like levels of I guess I would say like like you have guys like Miguel and I who've been using Kubernetes for like a long time like we totally get this stuff but we have other people whose expertise is in just like a totally

different area. So, I wanted to try to make something useful because the way that this story really starts is that we've got these developers and they need to use GPUs in order to do their work. And what they were doing was that they were using a spreadsheet to put in their reservation to say, I'm reserving this GPU and I'm going to use it for this long. And

my buddy Russell Bryant, great guy, good friend, he's like, "This is absolutely terrible. How can it be this day and age that we are reserving things with a spreadsheet? >> What if I don't look at the spreadsheet?" >> Yes. Exactly. And so that's what people did anyways, is that they would just go and use the resources. they might not use it or you'd be trying to be

polite and you'd be like tracking people down, Slack messaging them, all of this stuff. So Russell came up with this thing that he called CAN has GPU which was a CLI program that would allow you to on one box say can I have a GPU and it would reserve it for you and it put it in a like key value store and he also made it so

you could run it almost like a like you do like pseudo command. It would be like CAN has GPU command and then it would just reserve the GPU for as long as that command ran. You could also reserve it manually. And I'm like this is brilliant but it doesn't scale to more than one box. So I'm like oh this is like the perfect thing for DRRA in

order to do this. also like my relationship with Kubernetes and with this project VLM um is that there at the end of the day VLM is going to be shipped to a customer in my scenario as a container that they're going to run on a Kubernetes platform. So I'm also like there's a dog food factor here. Like these are developers that might not understand Kubernetes, but like

the consumers of this are going to use Kubernetes. So like there has to be a like dog food here. So yes, quick introduction to VLM. It's a inference server. Um put in another way, it's a way to run an LLM. So if you wanted to kind of like be your own chat GPT, this is the kind of thing. So if you've used something like O Lama or

whatever kind of like local llama stuff like Llama CPP, it's just the like um enterprisegrade kind of thing there. So totally open source. You can pick it up. You'll get to kind of see it like a little glimpse of it in the workshop. Um but yeah, just a little little background there. Okay, so just a couple more minutes on these different kind of characters. All right. So,

your first character here is your data scientist. And this is somebody probably is not like super into Kubernetes. They care about maybe they care about like writing like really fast CUDA code. They might care about like training models. They might care about like low-level access to the machines. maybe they can't do their day-to-day development with Kubernetes, but they need GPUs. Um, they may hog resources. They're definitely

the people ignoring the spreadsheet. Um, so we'll try we'll kind of like from the highest level we're going to try is this when we're going to use can has GPU. Next, we're going to be the VLM developer. So this person also needs some abstraction from something like DRRA. That's kind of like folks in this room. We want to understand DRRA. We want to be able to use

it. We want to be able to administer it. We want to maybe we want to develop our own drivers. Um but this person cares about doing their job for VLM. Like they're going to make modifications of it and they have to test it somewhere. Like they can't do this on their laptop. Same as the um uh data scientist. they want to pick up a pick up a

GPU and go. So, this is where we're going to um act in that role and we'll make a modification to the LLM. No actual uh like auto reggressive LLM knowledge required. It's like hello world style, but to show how you would like make that change and then launch it in Kubernetes and use the array. And then last but not least, I think that this character probably represents

um the majority of the people in the room and maybe some people sort of like scale in a range between these things, but like we care about how Kubernetes works. We care about how the gears are working. So we'll look at it from a like level, right? Like we're going to run the cube cuddle commands. Like the other roles might not care. Maybe we want to encourage

them to use it and so we give them some abstractions. But this is where we get into more of the details. But um kind of a like highlevel view of the driver side. thought, um, let's get let's get into it. Let's get into it. All right. So, here's the deal. Um, I'm going to have everybody that wants to participate, you can sign up for a shell account.

So, I have got a Kubernetes cluster running. And to sign up for a shell account, um, you can use this Google form. And there's basically two paths. So if you have a GitHub account and you have an associated public key, then you can just put in your GitHub username and then I'll provision a shell account for you so that you can just SSH into the lab. Um

if you don't have a um if you don't have a GitHub account um that is also okay. There's an option on the form to say like I don't have one and then you just pick a username and you put in your SSH public key. Is there anyone who wants to participate that was like that does not make sense to me? Because I'm more than happy to figure

it out. All right, this is a great crowd. Um, awesome. Um, so I'm going to give everybody a moment with that. and yeah this is just in case is basically the gist is yeah SSH keys private side public side public sides published. So the way that my script works is it's going to use the GitHub API and it's going to basically be like hey your name is

probably not Lionus Torvalds. Lionus if you're in the crowd let me know. But um it'll look up your name and your public keys. And so if you were unsure, you could just go to this URL with your own name, unless you're Lionus. Um so yeah, that's where I'm going to pull the key to provision your account. >> Cool. So now we wait. >> Now, yeah. So I

All right. At this point, give me a second. I'm going to try to reconfigure my display so that I can type at a terminal and also not look up at the screen. >> That's part of the fun. So, if you need any help with the form or something, just kind of wave and I will try to All right. I was not expecting that to be so fast.

Yay. All right, that's good. Um, I think there's one more link and QR code that everybody's gonna need. So, let me get that up. All right. Yeah. So, finally, um, yes, this is the link to our, um, GitHub repo. And there is a lot of stuff here. What you are looking for is the tutorial, the tutorial instructions. But if you are interested in this like like I

was saying like layers of abstraction like we you can go from just the tutorial which is like if you want it to be just copy and paste um or you can dig down deeper. So like everything about this lab is in this repo. So if you wanted to spin up this scenario, I've got uh it's primarily Anible playbooks to like provision a cluster and you can kind

of start from zero. It'll spin up a Kubernetes cluster and then it'll like load all of the um the demo resources and it requires zero actual GPUs. So if you just want to spin this up in whatever environment, um it's all emulated for the fun of it. So you can't do like the cool GPU stuff, but you can uh move all the gears to uh to see

all of that working too. does anybody need more time to sign up? Okay. Yeah, let me um Yeah, let me take your time and then I'll move on to the next step. Uh while we give everybody a minute here too, uh I'll just like point out a few things on the um GitHub repo. So yeah, tutorial steps. This is kind of the next if you're signed up

for the lab, this is the next thing you'll want to uh get up is the tutorial steps. then also um of interest may also be is this is the DRRA the custom DRRA driver. So what's custom about this is I wanted to make it specific to VLM developers. So I wanted to like kind of like figure out their workflow and then expose it so that they could

do this like with Kubernetes without actually having to like write YAML specs and all of that kind of stuff. But if you want to check out that um uh that DRRA driver, that's um that's where you would find it. um also is the um DRRA example driver is also really useful if you're interested in that perspective, but like I mentioned, probably not going super deep on that

one today. Uh let's see. Uh, and yeah, I guess last but not least is also this infra directory has everything for spinning up the um the lab yourself. Um, so yeah, it's a Kubernetes um 135 cluster. Also um I guess a pre-isclaimer in um for the tutorial steps is that um I developed the initial stuff with Kubernetes 134 and then DRRA went uh GA and 135. So

there's a bunch of deprecation warnings as you'll use stuff. You'll see it be like oh hey we're going to deprecate the v1 beta 1 flag. Um, but it's just a warning. Uh, and let's see. Yeah. Um, I am just going to check in with Miguel and the crowd and see how close we are. So, uh, just in case, um, if you've signed up for the form and

then you're trying to SSH in, won't be ready yet. Just going to wait till we've got everyone ready and then I'm going to provision all the accounts at once, which will be um, I'll run that part. But yeah, um for the tutorial, all you'll need is just to SSH in and then um we'll follow the steps from there. But let me just catch up with anyone. Anyone

else need time to create an account? All right, let's provision them. All right, so this is kind of a like behind the curtain sort of thing. You don't actually need to do any of this, but this is the results of the form. And then what I'm going to do is find the right tab where I was doing this, which is this one. And I'm going to move

unzip this oops. Um, great. All right. Now, this is where we start to load everybody up. All right. And then once this is complete, assuming that the playbook doesn't error, I'm just kidding. It should work. Um, but uh once this step is done, then everyone should be able to um SSH in. And let me see if I can make this a little bigger. Yeah, let me just

check from here. Does it look like hell? >> Okay, let me see about making some of these a little bit bigger, too. I should be able to sign in already, but wow. Does this have a timeout? >> Uh, it so when I ran it at lunch >> with much Yay. All right. It only took twice as long as at lunch, which is good. All right. So now

this is the part where people should be able to uh to sign in. So gist is SSH at draab.dougbtv.com and then you should get a shell account I'll give everybody a few seconds to catch up. Oh yes, no problem. >> Uh yeah, let's see about that. is that is that better in the light? I guess so. >> All right, good. >> Cool. Anybody need more time to

SSH in? Okay. Yeah, let's give everybody a And then I got to pull up my cheat sheet here too so I can follow along properly. we are here and then we will start to come down here and we'll try this out. So, this is this is my spot. This is Oh, look at this. All right. Cool. People are moving along. I'm happy. I'm happy to see this.

All right. So, all right. What this is sort of the like uh admin view. You can actually run these commands yourself, but um what we're looking at here is let's see if I can change the size of this a little bit. But we should be seeing here uh let's see. So what I'm looking at is this can has GPU status. I'm actually gonna exit the is so

if I'm looking at this all right so check it out we have eight GPUs virtual GPUs available on this machine and people have started to run this reserve command to uh to reserve the GPUs at some point will um will start to run out but this is our like data scientist kind of view so this was the person who was using the spreadsheet and then we came

up with CAN has GPU and then so you use CAN has GPU to say hey give me a GPU I'm going to run a command and then when it finishes release it so um this should be in the tutorial steps this is like your part right here so you can just do canize GPU run with your command if you're a data scientist you probably run something cooler

than sleep 100. But um hopefully >> require GPU for this. >> Hey guys, we're going to show you how to use a half million dollar GPU >> with a sleep with a sleep command, but >> at least increment a counter or something. >> Yeah, I know. I know, right? So yeah, that's uh yeah, how how typical is that? Your typical classroom environment. This is like uh for

junior engineers I've had join my team and they're like they're like trying to to figure stuff out and I'll have to be like, "Hey, welcome to the real world where everything is broken all of the all of the time." But yeah, in this ideal world, all you need to do is sleep 100. Um all right, so let's see where we're at. All right, people are people are

still doing it. Once once everyone has gotten a chance to run can house GPU. Oh, manual uh reservations. Okay, nice. So, remind me Doug, which one of these would kind of hog the resource? The manual, right? >> Yeah, that's right. Yeah. So, a manual reservation is like um uh is Yeah, you're saying um >> you have to release them. That's >> Yeah, you Yeah, you'll you'll have

to release it eventually. So, but yeah, so manual is saying like I want this specific GPU. Maybe you're um so I'll tell you when I use manual reservation with my team is I'll say I'm doing something like I'm building Docker images and then I'm going to run them on a particular box and then I'm going to test it out and what always happens is I'm like hey

I need two GPUs for this task and I'm probably going to like work for an hour or two on this but it'll be like I'll use the GPUs for like 10 seconds and I'll realize ah my build my build's no good. So I put in a manual reservation to uh to say like hey even though I might not be using these for five minutes while I build

a new image like I'm going to need them again in five minutes. So that's when I do that. Um, but for a lot of stuff that people are doing on my team, they do they'll just do the can has GPU dash run d- my command. Um, which would in a lot of cases probably be Python. Um, all right. Let's see. And yeah, you're going to see a

lot of this kind of like no usage detected. There's actually like a some pretty neat features of this where it will use your um device uh your hardware vendors underlying stuff to like see like hey what processes are processes are running how much RAM are they using and there's even stuff to like detect like what model you might have loaded on there so you can you can

see like oh cool somebody is using this particular model on this particular GPU. Or if you're like me, you can use too powerful of a GPU and then have somebody from your team come up to you and be like, "Do you really need to use that big of a GPU for this for this job?" So, that kind of stuff happens. Um, all right, let's see about moving

on to the next step. So, If you haven't gotten there yet, I recommend as the next step, it's not required because we can work around it and we'll we'll see how it goes because we're all using this on one machine. Um, but there's actually three available, right? So, next step, if you made a manual reservation, go ahead and do uh CHD release and we'll release that. And

then we're going to move on to using the GPUs as our VLM developer. Um, so in this step you are going to actually modify some VLM code. As I mentioned, you don't know need to know how to like invert a tensor or do like funky data scientist stuff. We're just going to say hello world to it. But we everyone should have in their home directory a VLM

directory which is a clone of VLM. So if I go here I can do get remote and you'll see that it is a um clone of my fork of VLM. Just one thing and please ensure that for this step you actually step into the VLM uh directory otherwise you'll get a very confusing error. Totally. So yeah and I guess this um goes into like this abstraction. So

for this sort of characters role your VLM d um developer I have made uh this be like specific to VLM. So, if you were to go and like take a look at my driver and see what it does, the the gist is it will take a diff of what you have in your working directory and it'll pack it all up and then it'll cache it on uh

in your Kubernetes cluster. It will also cache a convenient image to use so that you don't have to rebuild your whole image. So VLM bunch of code a lot of it's Python but there's also C sources that have to be compiled into binary. Um if you are changing the C sources and you need a new like compiled binary that you know gets loaded on the GPU and

all of that um that takes a long time. So you're kind of I would say out of luck as that developer. You're still going to have to wait for a build of that that cours. A lot of the time your like day-to-day kind of stuff you would be doing is you'd modify a little bit of Python and you just want to run that on the remote. Oh,

inside of my VLM directory, if I had just you know, it's like, you know, the kind of stuff you'd expect, especially uh claw.md, of course, you're going to see um that kind of thing. But the gist is this is your uh VLM dur. And if I do a get log, I'm going to see that there is one commit here. um that is the most important for this

demo which is my um kind of hello world. So you can take a look at this um this diff by doing uh uh so this is the diff and the last the last thing. So essentially what we're going to do to say hello world is that we're going to hook into a point like really really early in the VLM life cycle and we're going to output uh

a little bit of text cuz when VL VLM initializes it's trying to figure out stuff like hey what kind of hardware are you running on what are uh you doing for X Y or Z since this is emulated at some point it'll just wind up being like I give up. You don't actually have any hardware I can use unless you can do CPU inference and um there's

interest in that but uh at the end of the day like GPUs you needed kind of the GPUs to fuel the system. All right. So, one, uh, what I'm going to do here is let's actually go ahead edit this. So, you can put in whatever you want. This is this is optional, but more fun. And I always like uh my opportunity to put something in here. So,

I actually edited mine this morning to say, "What's up, CubeCon?" and um put that in there. So, if you just do a find for hello in here, you're going to wind up uh finding this block. And you could put whatever you want in there. You could if you want to try to invert a tensor or something like that, you could. But in in this case, um similarly

to sleep 100, um we're just going to say, "What's up, CubeCon?" And then we're going to exit. Um from there um next step in the tutorial should be after making your change is to actually do our Kate's has GPU VLM run. So this is our abstraction for your VLM developer role. So this is your person who wants to uh run this code somewhere that has a GPU.

they're doing this on their laptop. You don't have a powerful GPU on your laptop, chances are if you do have one, come talk to me. That sounds cool. Um, but generally we're going to say, "Hey, with this code in this directory, I want to run it." And so, you're a VLM developer, like you don't want to write YAML specs for Kubernetes. I mean, I like to, but

I'm a guy at CubeCon. So, um, but yes, this is their like level of abstraction. So, I'm going to copy paste that one let's let it go. You're going to see a bunch of output here. Um, which some of which might be more interesting when um you care about all of the aspects of VLM and what you're running, but I'm going to say, all right, VLM run.

Um, and I want to follow the output from this and this is going to be the name of my workload like a sort of shortcut and then VLM serve it. So taking us a few steps back like this Kubernetes has GPU thing. It has kind of the dimension of the cluster, right? It's like a bunch of nodes. It's aware of everything and you get to abstract how

you run your workload. just throw it at the cluster like >> yeah totally >> fully using Kubernetes and this is just like a CLI since I don't like or know Kubernetes I want give me a CLI that I can run. >> Yep. Um so I so this is taking a while to spin up. I have a feeling it's going to time out on me and tell me

that my um workload is getting cued or something broke which could totally happen. So, in a second here, I'm going to kill this and I'm going to look at it from a Kubernetes admin perspective cuz it's possible that everybody sort of beat me to the punch or something busted, which would be typical. >> Yeah, I would bet that somebody. >> All right, so well, a lot of

somebody's using everything. >> Next roll. This actually this starts to get more realistic. Like I was saying, it's the real world and everything's broken. Let's see what's busted. Okay. um, let's take a look. First thing I'm going to do is, so if I was VLM developer, first I'd say Kate's GPU status. Show me what's up. Oh, my box is down. This is not good. Let's find out.

>> whoops. >> Oh, that's no good. The load is 42. >> The load is 42. Oh, yeah. All right. All right. Um, cool. Well, that's um awesome. Um, the what? >> Yeah, check this out. So, yeah, the um we super overloaded overloaded the the boxes. Um, so 72 users and um a load average of 40. So >> this is I I guess I could have used the

bigger boat in this case. >> If you had a bigger boat, why didn't you bring it out? >> Oh, everybody's using this at the same time. It's the same folder. What is No. Well, and you know what? So, basically what at the end of the day, um I think I think we overload we overloaded the cluster is so it is these workloads probably will like fill up

a ton of RAM um when it starts. Yeah. Um so that's >> bummer. >> well, that's >> Do we have a plan B? yeah, let's see what's a good approach here. I there's like part of me that do like boot a bunch of these um >> Let's do that. >> Okay. Um let's see. Uh you don't have control plane access. Uh, you know what? I could stop

the cube blitz on all the boxes. >> Oh, it's working. All right. Okay. Magic. >> Thank you for the good news. Um, let Yeah. Let's see. Uh all right. So, okay. Yeah, we've got a bunch of these running. Um let's uh check out our case has GPU status and we'll see what's been used and not used. So yeah, this is our abstraction of the um DRRA driver

that's that's behind the scenes. And yeah, Kubernetes admin, I'm going to use cube cuddle to solve all my problems. Um or in this case, lock out and eventually we get some capacity back on these machines. But okay, great. So let's let me scroll up here a little bit. is yeah. So total GPUs 24, eight per machine, three nodes running. Um, everybody used a whole bunch as evidenced

by a totally overloaded machines with some cheesy cloud VMs. Um, and now um, we just have a few available. So if you're a VLM developer, like you don't necessarily care about like resource claims and like uh your like templates for for using this like you just want to like chuck in a command, have your workload go up there, get the results of your test, um see your

hello world output in our case. Um, so essentially this is like what you would look at as the VLM developer. Now let's hop back on with our Kubernetes admin hat on. So from here let's do a um cube cuddle get resource claims. This is sort of the um >> again um so let's see. No. >> Why don't I talk about it a little bit while we wait

for this? >> you can stop it. >> There we go. So, this is you know how Miguel kept talking about like you're going to get this like coat check ticket. This these are our code check tickets. So in this particular case um you're when as the your VLM developers went they launched their workloads and now we're going to see um the the claims um themselves. So this

is um what the driver went and figured out what's available goes and marks it claimed reserved. Um, something that you might see happening a little bit here too is um, sort of a little bit of the like behind the scenes here for this state. You might actually see some of these like kind of so this one here 12 days old. I left that one running specifically um,

to kind of show you that there that pod actually died. But the way that I um made this was that that allocation will stick around until you clean it up. I was kind of hoping that we might see I was seeing it this morning, but sometimes you'll see it like flip. There we go. It flipped allocated reserved. you're seeing a little view um behind the scenes of

the uh reconciliation process that's um that's happening there. And um I'm going to go ahead and clean up uh this one too and then we can see it go. But you have seen a like few others come around and then go. So if I um because if you remember the life cycle of the resource is not tied to the workload. It's tied to this particular >> Um

>> so once you clean the receipt you will get the device back not when the >> I think that's the one. watch. >> Okay, there we go. And now my 12-day old one is gone. Um, and how about just a like few other things to um to kind of take a look at as well to like look behind the scenes. If I do uh cube cuddle get

pods for all namespaces and I'm going to have here um this nameace where I've got the um the driver running. So kind of the like parts of this driver are a controller. That's really what does all of the like most fun kind of like uh your day-to-day um Kubernetes operator controller kind of life cycle stuff. You've got a cublet plugin and uh we also have this node

agent and this is what I'm actually using for the like caching side of this thing. So in the case of VLM like these docker images are big ones. So you're talking like um tens of gigs um kind of a thing. And then you also have your LLM models which are also monster big. So I uh made a way to kind of cache this stuff in advance. And

I mean depending on like your actual setup, you might like have like good shared storage and uh you're able to like have some source of truth what models you have loaded. And then I also cached um the like get changes as well. Um, so theoretically, if you kind of have like the system pre-loaded, it's got the right image you're going to use, it's got the models you're

going to use, and you're just changing a little bit of Python, then theoretically it should be a good user experience for your VLM developer because just changing a couple lines of Python. Everything's already loaded on the Kubernetes experience. I was trying to make it better than what they had before, which would have been stuff like SCP a bunch of stuff, kick off a huge build, and then

finally run it and then be like, "Oh, it's wrong and I have to change it." After that point in that workflow, like it probably would be small changes. Um, but I was trying to uh get a system where it'd be like a better life to try to encourage them and then sort of feed into that. Um, dog fooding. Doug, I think I have some news for us.

I think we're out of time. >> All right. >> Are we? >> Okay. All right. We're out of time. Thank you. >> Any time for questions? >> No. >> We're here. >> Thank you. I appreciate your time.