GPUs on Kubernetes: What Actually Happens When You Request Nvidia... Gulcan Topcu & Daniele Polencic

This talk explores the intricacies of using GPUs within Kubernetes, beginning with the fundamental differences between GPUs and CPUs. The speaker explains that GPUs handle operations differently, requiring a CUDA context to manage memory and kernel execution, which cannot be interrupted once started. The discussion highlights the challenges of allocating resources and managing visibility into GPU memory, particularly in a cloud environment like Kubernetes. The presenters detail the various components necessary for integrating GPUs into Kubernetes, including drivers, the Nvidia container toolkit, and device plugins, which facilitate communication with the Kubernetes control plane. Ultimately, the session underscores the complexity involved in optimizing GPU usage in Kubernetes and the performance implications of these differences.

Thank you very much for joining. Hopefully you starting starting the day with GPUs. That's a very very tough day, I guess. So welcome welcome everybody. So my name is Daniele. I'm one of the instructors at learn Q. Um and with me today I've got Gutcha. Um we're going to talk about GPUs. So we wrote two books. One of the one on on the basics of GPUs and

how they work and why they are so difficult in Kubernetes and another on optimizations for yield in GPUs. Again in in Kubernetes. So today we're going to take a sort of a look at both of them perhaps a little bit more about the the basics of of GPUs. We're going to see how those actually plug into Kubernetes, why they are so difficult to manage. Um and and

and you know the challenges you might face with with this kind of hardware. Okay. Without further ado >> [laughter] >> Thank you for the introduction then. Thank you everyone for um joining us today. I'm really excited um to be sharing the stage with Dan. Meeting my team for the first time and having you all in this room today. Okay. So to get to the good stuff actually

we need to start from the bottom like way before Kubernetes even enters the picture. This begs the question and this is how does Linux run things? So when you write an app it needs to do things like reading a file, sending a network packet but it has no idea what the hardware what hardware is actually there. Is it an SSD, a spinning disk? It doesn't know and

it doesn't care. Instead it just talks to the Linux kernel and the kernel figures out the rest. So we call this interface between your app and the kernel as system calls. Everything your app does goes through the system calls through the Linux to reach the actual hardware. If it wants to open a network connection still the same. There are no shortcuts and the kernel map is massive

like there are hundreds of system calls. So naturally people started asking if everything passes through here can we actually do something with that? Like can we group processes together or limit how much resources they use? Turns out yes and that's exactly what control groups are. You can think of them like rate limiting on an HTTP API except instead of limiting the calls the number of calls we

are limiting [snorts] how much resource like memory, CPU, network bandwidth we use. And we can create as many C groups as we want each with different processes and different limits. But C groups get us halfway there because containers need something else too like isolation. That's where name spaces comes in. The idea is give each process its own view of the world. There are different types of name

spaces and each one isolates a different part of the system. Take the mount name space for example. Mount name space is just a way for the Linux kernel to isolate access to the file system. So what does it actually look like? Our JVM app thinks it has its own entire Our Node.js app thinks the exact same thing too. But in reality they are just looking at different

sub folders on the same disk. But they just can't see each other. And there are other name spaces too like user, network, process name spaces and C groups and name spaces are powerful yes but they are pretty low level. Usually you don't want to deal with them manually all the time. So people thought what if we wrap all of this in a nicer interface and build an

abstraction on top of it so that we can interact with them more easily. That's basically what Docker is. You take C groups, name spaces, you mix them together and you get a container engine. There are other tools that do the same thing and they all help with this. How do I set limits with C groups and how do I create isolation with name spaces? So [snorts] that

was the container story. Now let's talk about how CPUs work. A CPU can only execute one instruction at a time. Yet most of the time you probably have a browser, terminal, Slack all running at the same time. So how does it even work? The kernel just slices time. A few microseconds for this process, a few microseconds for another one. So back and forth they appear to be

working at the same time. We don't even notice the difference anymore. And if you have two cores same thing happens across both of them. But I'm wondering if the CPU can execute only one instruction at a time how do we actually switch between the processes? The CPU has some state, some registers that it uses to execute instructions for a process. So when we need to switch we

save [snorts] the state of the current load the state of the next one continue execution and then we need to switch back. We do the whole thing in reverse. That's context switching. We [snorts] have to do it a lot. We see two multiple apps running simultaneously to see both of them running together. But in reality we are just saving state loading the next one, doing some execution

and moving on to the next one. what about memory? Same story. Your app never touches the RAM directly. Something else does this. Inside the CPU there is a component that takes the memory addresses your app uses and maps them to actual physical locations in your RAM. >> [snorts] >> Your app gets a clean unified memory space even if the actual physical memory scattered all over the place.

Again the hardware handles all of that for you. Now we really understand CPUs. uh is it the same with GPUs? >> No. GPUs are very very different from everything we just talked about. Now let's talk about kernel. I know that we already used that word for Linux kernel but here it means something completely different. In the GPU world a kernel is just a function, an operation that

runs on some data. In this snippet we import CuPy. CuPy is a library that talks your GPU. And we have a regular Python list. It lives in our RAM. There's nothing special yet. And when we call cp.array do you think GPU will access the RAM directly? No. Instead it takes our data from RAM and copies over to the GPU's own memory. The GPU has its own separate

memory and it can only work with data that is already there. Now we copied the data and we are good to go. Then what we actually that when we actually run a computation on the data like multiplying that computation is what we call a kernel in the GPU world. Okay. Now the kernel holds the result in the GPU memory. So how can we get the result back?

Since our program runs on the CPU we have to copy the result back to the RAM to actually use it. The data is always transferred back and forth between the RAM and the GPU. Now here's where it's really different from CPUs. Now if I have a second kernel waiting to run what will happen? Let's say I have a subtraction. A second kernel will trigger and execute and

this is where GPUs are completely different Remember how CPU slices time? There is a constant switching back and forth. GPUs do not do that at all. When a kernel starts running on the GPU it takes over the entire GPU until it's completely done. So it cannot be paused or interrupted. What if you have a long running kernel? Anything else that wants to use the GPU just has

to wait. So a GPU can only execute one kernel at a time and to completion. Let's take a step back and ask another question. How a program actually claims the GPU in the first place? And that is where the CUDA context comes When your program wants to use the GPU, it opens a CUDA context. That context is basically a session CUDA context keeps track of all the

memory you copied over to the GPU, the kernel you want to run, the GPU state, and all the other stuff like how you schedule and coordinate all the work that runs on the GPU. All of them are in one place inside one CUDA context. >> Okay. Now I know a program always starts from a CPU process. I also know that when I want to work with GPU,

this process will open a CUDA context on allocate some memory, and then kernel will start executing. If I have multiple programs that all want to use the GPU, each one of them has to do the exact same thing. And this design choice comes with some consequences that we will talk about in a second. But why are GPUs so different than CPUs anyway? On a CPU, your code

is full of decisions. The CPU never knows which way your code is going to go. So a huge part of the CPU's hardware is just dedicated to guessing. And on top of that, it has caches, all those extra logic to keep those math units fed as fast as possible. But GPUs are built completely different. When you work with them, there are no decisions. So instead of spending

hardware on prediction, you spend it on raw computation. That's why GPUs are so good at this kind of Now I know that GPUs are built for predictable, repetitive work, and CPUs handle messy, unpredictable stuff. Now let's talk about those consequences that shows up in memory. When you allocate memory on the GPU, that memory is visible to everyone using that GPU. If you have two programs running on

the same GPU, let's say one using 8 GB and another one using 8 GB, both of them can see all 16 GB. Not just their own chunk, which is a bit strange when you think about it. And it gets even more puzzling. Say you ask the GPU for 8 KB of memory, the GPU doesn't give you that. Instead, it gives you 2 MB pool from the If

you ran nvidia-smi right now, it would show 2 MB used. And what happens when you make a second allocation of 8 KB? The GPU doesn't reserve more memory, it just uses the pool it already grabbed. So instead of what you ask for, the GPU just hands you a chunk. The reason [snorts] is speed and simplicity. And as long as what you need fits inside that chunk, you

are fine. But if you need a really large amount of memory, the driver might reserve even bigger chunk for you. On top of that, that memory is being used, but it is not exposed to you. So you really don't know how much memory is actually being held by your processes. It is way interesting. So look at this diagram. Process A is running on the CPU. It opens

a CUDA context on the GPU. And inside that context, the orange block is the memory your kernel actually needs. But the driver grabs a much bigger chunk on top of that, as shown in yellow. But your process only knows about the orange part. I have no visibility into what the driver is actually holding. So let me quickly recap the key differences between CPUs and GPUs before handing

it over to Dan. On a CPU, a lot of the hardware is dedicated to predicting what the next instruction is going to be. The CPU can stop a process mid-execution, save where it was, run something else, and then come back, pick up exactly where it left off. So context switching is cheap. But a GPU works completely differently. It only has execution units. Once a kernel starts running,

it has to monopolize the entire GPU. And because there is no save and restore mechanism, context switching is very, very expensive on GPU. Memory allocation works differently, On a CPU, you ask for exactly what you need, and you get exactly that. But on a GPU, you just get a chunk. It's imprecise. You do not always know what is actually reserved, but it's fast and simple, and that's

the trade-off. On a CPU, everything goes through the Linux kernel. Cgroups, namespaces, all the things we just talked about at the very beginning. And the Linux kernel is open source. There are thousands of engineers contributed to it over decades. You can read the source code if you want, but on a GPU, none of them exist. Everything goes through the GPU driver. And the driver is a black

box. Nvidia wrote it, shipped it, we just install it. We call the API, something happens inside, and we get a result back, but what actually happens in between? We don't know because the source code isn't public. Okay. Thank you for listening to me. Now I'm handing it over to Dan. So we we sort of >> [applause] >> Thank you. >> We look at the hardware or how

GPU works, but how does this translate to what we do in Kubernetes, right? And then to look at this, we need to basically have a um go back and sort of understand what is the life cycle of uh of of creating a pod in Kubernetes. So generally, what you do, you would create a deployment or you you create a pod. It also will go through the control

plane, and then eventually, there will be something like a kubelet just picking up this specification and then running it inside um your node. Inside your node, there are like three components that we use to create these pods, right? There is There are three interfaces. The container runtime interface, container network interface, and the container storage interface. Now these three work together to provide the runtime, the storage, networking

for your container. And at the end of this process, then you will have the pod running. Now when it comes to GPUs in Kubernetes, there are a few things that we need to solve. And the And the things are And And these These two sort of challenges Where are these GPUs located, first of all, uh inside our infrastructure? Then we talked about how cgroups and namespaces and

how they isolate and control the resources for these containers. Well, it turns out that this um device that we are exposing in the Linux kernel is isolated for a namespace. So we need to poke to actually get to it. I don't know if you try running GPUs on Kubernetes, but you don't need to provide any driver inside your container. It just happens automatically. But who is doing

that work, right? And um how is the GPU allocated in the node? Now it turns out that to solve these four problems, we have four layers. We have the GPU, the actual driver at the bottom, then we run the driver itself, and then on top of that, we've got something called the Nvidia container toolkit, and on top of that, there is the Nvidia device plugin. So why

do we need all of these components? The reason being uh so we start with the driver. So the driver is basically the real um uh component that that they basically control the hardwares for these um for these GPU. And the way it works is basically it's just installed as as a driver inside the Linux kernel. And now it's it's exposing these these GPUs as a file descriptor.

So anything that goes through uh this time will have to go through through the driver driver itself. Now on top of this driver now, we need to find a way to actually run these GPU in Kubernetes. So we have the actual driver, we can actually send commands to the to to to this GPU, but how do we make these GPU available to Kubernetes? And how do we

actually say this this pod will run on the GPU itself? Uh it turns out we don't have any interface for that. We don't have a a a container GPU interface. So we will need to be a little bit creative here. So what we do instead is we look at the container runtime interface, and it turns out that this running a container for something like uh containerd, for

example, will go through several steps. And these several steps will basically be hooks in which you can plug you can plug in and execute things. And this is basically the magic that happens when you when you use an installed um Nvidia container toolkit. It will basically just plug in into this system and then ingest basically relax the constraints for namespaces. You you can you can actually access

that device. Um load basically the right driver, inject environment variables so your container has got a way to connect to this GPU to begin with and and and basically start the container with with the GPU inside. So all of this work happens magically. Your container doesn't know any anything about it. It's just going to be run and everything is ready there for you. Nothing nothing to do.

Um so this is basically how it works, this component do all of this magic? Well, it turns out that there is a demon So when you uh when you installed this inside your cluster, you get like this device plugin and it runs as a demon set. And what it will do, it will basically just communicate with with the kubelet and basically register this GPU inside the kubelet.

So the kubelet is aware that something is available on the node itself. And then from there, it will just basically make Kubernetes aware of this GPU. Um so it will say, you know, find a GPU and then it will basically just keep track of the uh status of of of this GPU running on the node. So how does this work end-to-end? Right? So what you see when

you install all of this with with and Nvidia, what you see is basically you create a pod, you create a deployment and you request for um a GPU, right? And then this request will go through and it will be passed inside the control plane. Control plane, we've got the um API server will get the request. You you will be authenticated, authorized, will go through all the steps,

stored um the resource will be stored in etcd and then what it will happen next, we will have a controller manager, will create the pods, we will have a scheduler, will allocate this pod to a a specific node. Uh and this is basically where spe- things start to get a little bit more interesting. So this scheduler will also be aware of the GPU as a as a

resource, as something. So what it will do, it will basically look at all the nodes you have and it will basically say, "Oh, first phase of scheduler is basically filtering out the nodes that are applicable for this pod. So it will just select the right ones and then we'll go through a scoring phase and we'll say, "Okay, I've got like these four nodes, but actually only one

is is probably the best node for this pod." At the end of this process, the pod is allocated to that particular node and that's when the device plugin um uh the toolkit and the driver work And what happens next, it will basically say, "Okay, I've got registered these these uh I've got I've registered this GPU for the node. I've been allocated the pod. It's now time for

me um to create um to create a pod." And it will go through this process of creating the pod through the container runtime. So containerd will go through we have pre- to actually create all of these environment variables, plugin the drivers as well, inject environment variables and then finally running running the container. So this is basically this section here. And then at the end of this process,

then finally I have my pod running with a full GPU inside with the correct drivers. Um that's that's basically the full flow for what um you would do in Kubernetes. So a couple of things to to remember from from all of this, um I really I suggest you read the book. So uh we put a lot of effort. So everything you see here is also in the

books and even a little bit more. Um but I think, you know, the takeaway from today is that GPUs are very very different from, you know, normal workloads. Why? Cuz most of the time what we do is um we are sort of assumed that things works in the same way that deploying a web application, right? But it turns out that the hardware is very different. We cannot

stop an execution, right? The allocation is made the the memory is allocated in chunks rather than being granular, right? So I think understanding the reason why this exist, which is basically performance, right? Um actually gives you insights on what's possible and what is not possible when it comes to GPUs. And the other things to keep in mind, which which I think, you know, tricked me so many

tricked me so many times, is this driver is basically the owner of the hardware. Whereas most of the time when you use CPU and RAM, you're going through the Linux kernel. When you're using a GPU, you are basically using a binary which we have no control over, which is closed source, we cannot inspect whatsoever and is basically doing this allocation of memory in chunks and is doing

this execution of kernels, which we cannot stop. So all of this is basically like a black box we have no control over. And and you have just to do to trust it's going to work in the right way. And then by the way, this is this is the reason why we also have a lot of escape, you know, sort of security issue with with with with GPUs

as well. So Kubernetes is primarily designed to run containers. So to run this hardware, which is so different from what we used to do, there are like several layers we need to patch inside Kubernetes itself and then just to make it work nicely. So I think all of these, you know, four different layers we talked about are necessary because Kubernetes perhaps were, you know, expect things to

work in the same way. Not to say, you know, it's not that Kubernetes is not embracing it, but there are some there is some friction at the moment that you can see on on on the number of things we need to install and patch to make it work it correctly. So that's that's basically um yeah, that the real challenge when it comes to to running um GPUs

on on Kubernetes. So thank you very much for for listening to to our talk. I hope you enjoyed and you learned something new. Um that this is the end of of the talk.