Peeking Into the GPU Black Box: Continuous Profiling on Kubernetes With eBPF - Zahari Dichev

This talk explores the intersection of GPU profiling and Kubernetes, focusing on how eBPF can enhance observability of GPU interactions. The speaker discusses the current limitations of traditional GPU monitoring tools, such as NVIDIA SMI, and highlights the need for per-pod visibility rather than just node-level aggregates. By leveraging eBPF, they propose a framework for monitoring GPU workloads with metrics that reflect both GPU demand and back pressure. The presentation includes examples of how these metrics can help diagnose performance issues in real-world scenarios. The talk emphasizes the growing importance of effective GPU monitoring as teams increasingly rely on these resources for complex workloads.

Good afternoon everyone and welcome to my talk titled peeking into the GPU black box. So quick intro, right? My name is Zahari and I work for buoyant as a software engineer where I work on the linker D service mesh. Now today we're not going to be talking about service meshes or networking. We're going to be talking about GPUs and how we can use eBPF in order to

profile the interactions between our applications and the GPU devices on Kubernetes clusters. So here is the plan. We are going to first look at why GPU introspection is important in Kubernetes and specifically why it's in this day and day and age. We're going to do a quick overview of the existing tools and approaches of how we can gain observability into GPUs on Kubernetes and we'll see what

they're good for and where they actually fall short. Then after that we're going to try and see how eBPF can help with all of that and to what degree. And then we're going to propose a simple monitoring framework that consists just of a handful of high-level metrics that can help us gain more insights into what workloads are doing with the GPU in our clusters. And then we'll

substitute that with a couple of concrete example of of incidents and see how we can how this framework can help diagnose them. And hopefully at the end we have some time for some easy questions. So first of all a couple of uncomfortable truths. We are actually living in a GPU gold rush at the moment. There are cases where cloud providers have wait lists for GPU hardware. Teams

across companies are hoarding their GPU budgets because it's really hard to get your hands on these expensive pieces of hardware. Why is this the case? Well, it's very simple. A single H100 costs nearly $4 an hour on a major cloud provider. You scale that to a node of eight of these devices and you're suddenly looking at more than $30 an hour. A cluster consisting of eight nodes

would then cost you more than $250 and that's just for an hour. You scale that to a day and all of a sudden your cluster costs more than $6,000. That's a figure. But for a month, you'd have to pay more than $126,000. you know, this is just compute alone. You don't factor in traffic and storage costs into that. So, this is all getting worse as teams grows

and projects fill in the pipeline, So, the question comes, when you're paying all of that money, what kind of observability do you get into what's happening with these incredibly expensive devices? Well, I'm afraid to say that we're for the most part flying blind, right? See, we've built over the last 10 or so years, we've built amazing observability tools for CPU-centric workloads. We have things like Grafana, we've

got distributed tracing, we have continuous profilers for Kubernetes. All of these great tools that can give you a lot of details about how your processes interact with underlying and to diagnose any problems. What do we get for GPUs? A bunch of hardware-centric metrics that are exported from NVIDIA SMI and DCGM that tell you information mostly around single single figure utilization across the device and memory consumption, temperature

of the device for whoever cares. Uh we don't have per pod breakdown. We don't have granular utilization percentages that we can rely on. So uh there's a lot to ask there. What do we actually need? Well, ideally we want per pod GPU visibility, not just node level aggregates, right? We want to understand why a GPU is idle, not that it just is, Um we also want to

be able to detect and explain changes in workload behavior over time and continuously. Which means that we want these tools to not just be um tools that we resort to when uh stuff breaks or after stuff breaks. We want them to be running continuously all the time on our production systems. So that's a pretty thick wish list, right? Let's see what we actually get uh out of

the box. So we get the NVIDIA SMI, which is uh hardware-centric tool that can give you a bunch of counters for your CUDA cards. It can give you It can give you things like temperature, power, uh utilization, hardware errors, etc. These are very hardware-level metrics. It's hard for you to actually um correlate them with the specific behavior that you see in your applications, right? And we get

tools like DCGM, which allow you to export a lot of these metrics into Prometheus um in your clusters and look at these dashboards displaying all of these Now, on the other hand, we also get tools that are a lot more powerful, such as Nsight, which is uh the proprietary profiler by Nvidia that's built on CUDA. It's very powerful, right? You can profile GPU codes, you can look

at kernel execution timelines, you can set breakpoints, you can do a lot of these fancy things, but it's not really it's not really um designed to run in production. It introduces 10 to 50% of overhead. It's manual and it's mostly targeted towards developers who are trying to optimize um the code that they are writing on their workstation, right? So, the fact is that the lightweight tools don't

really give you all that much information while that's that's useful, while the heavyweight tools uh that are very very useful in diagnosing problems are not really meant to run in production at all. So, let's see whether eBPF can help. So, eBPF, as some or most of you know, stands for extended Berkeley Packet Filter and is essentially a small runtime that runs in kernel space that allows you

to write programs that are driven by events. It's used extensively in networking and um observability. Now, Uprobes are particular kind of programs that can hook to device uh to user space functions, right? So, that can uh that allows you to essentially monitor for certain set of processes, what functions are they calling from a library, how often they are calling it, and compute statistics around like call frequency

histograms as well as uh arguments that are feeding into these functions, return codes, etc. So, that allows us to actually hook into the device driver API and observe uh what a process is doing with the GPU. Let me show you how that actually works. So, on a typical node, on a typical machine, the processes don't actually communicate with the GPU directly, right? They go through the device

driver library, in this case libcuda.so. And for example, if um if process one is a PyTorch training job that uh needs to launch a kernel, it will call into the C cuLaunchKernel function on the library, and then this library will do the usual hardware communication mechanisms in order to dispatch that to the device. So, having said that, we can use eBPF to hook ourselves to the entry

and exit points of these functions and trace calls for every process um that interacts with the GPU. So, this is not news, right? This is nothing groundbreaking. Folks are folks are doing that. So, let's see some of the products that actually use similar functionality to that and see what they give us. So, Polar Signals is a vendor that they're shipping a continuous profiling for Kubernetes, very interesting

technology. It relies on pretty much the same ideas. What they do is that they used um they used the CUPTI. They recently changed their Parker agent to use the CUPTI interface from NVIDIA, which is their instrumentation instrumentation API. So, they use combination of CUPTI to instrument the workloads running on the cluster and couple that with some eBPF functionality in order to collect um metrics uh around kernel

execution times, backtraces, kernel names, launch, memory transfers, Grafana Bela is another uh solution that uses a very similar to our approach. So, what it does is that it again, it hooks probes into the EP uh in uh via eBPF. And you get all sorts of low-level information about the interaction uh between your process and the device. So, you get um kernel launch histograms, you get rates of

memory allocation, etc. etc. So, all useful stuff, right? And then you've got interesting projects such as Eunomia BPF time, which is a research project that um I discovered recently. It does some pretty advanced stuff. So, it uses uh eBPF that compiles into PTX. And PTX is essentially what you would consider the CUDA assembly, right? Um so, it compiles this it intercepts the PTX that that's getting onto

the device and it introduces instrumentation points is the in this code. Um and that allows you to pretty much do some really advanced profiling. The problem with that is that obviously it's very intrusive cuz you're essentially changing the code that's getting uploaded onto the device. it's The other problem is that there is no clear path of how to run that on Kubernetes right now. It's still very

research level project. And but the biggest kind of showstopper here as I see it is that it requires the CUDA binaries to be packaged with PTX which I think for most of the um production level libraries such as PyTorch etc. is not the case at the moment. Now all of these tools are clear improvement that just looking at like Nvidia SMI metrics and wondering why the temperature

of the GPU is is going up, right? Uh however, they still give you very raw level data, right? They essentially concern themselves with kernels launched and memory allocations happened happening. Now an operator of a cluster looking looking at all of these all of these like thousands of traces across hundreds of pods would find it very hard to answer the question of well, why why is this particular

workload slow? Why are the owners of this inference service complaining, So if these things are too what what do we need? What do we actually need in order to make higher to kind of do higher level reasoning about these workloads? Well, we need to essentially move from raw CUDA traces um that just tell you what happened to metrics that tell you what what it means for the

workload, right? The diagnostic power is not in just individual signals. It's the relationship between them and how how these signals move over time. So we need metrics that are designed to disambiguate failure modes in in in how the GPU is actually used by the workloads. And in order to do that, we need to build a mental model of what the GPU actually is. And for for the

complex hardware it is, uh the mental model could be surprisingly simple, right? So, in its essence, the GPU is an asynchronous machine, right? So, you are launching a kernel or doing a memory allocation, and what's happening is that you're feeding work to the GPU. Um and this is your supply side of the of This is your demand side of the equation. You're enqueuing work onto the GPU,

and you need to feed the GPU some work in order to do something. But most of that enqueuing is asynchronous. So, on the other hand, the GPU is exerting back pressure. How is that happening? Through synchronous memory transfers from from device to host, through synchronization barriers, et cetera, through queries, et cetera. So, essentially, this is the back pressure side. This is the GPU telling you the the

host side, "I'm not ready yet." And that And these two sides of the story uh essentially represent something different. So, it's the relationship between them that needs to be observed, that matters over time when you're looking at your workloads. So, if we have to derive just simple metrics that can help us diagnose some incidents, what would they be? So, in its essence, we can use we can

look at GPU demand, and that could be as simple as kernel launches per second, right? which means how much productive work is being enqueued onto the GPU. You can make that even more complicated, right? You can you can factor in uh sizes of the kernels, etc., etc. But, in its essence, it's how much work are you queuing onto the GPU? And you can look at GPU back

pressure as a metric as well. And that could be as simple as the time that the the host waits for the device to be ready. So, the sync the the amount of time that you spend synchronizing with the device in a blocking call. And then you also need to use to look at like GPU contention, which is is anyone else actually on this device at the moment.

Right? So, having said that, uh how do we derive these metrics with eBPF? Well, we use eBPF Uprobes and your return probes, which are, as already explained, uh a probes that attach to function entry and exit. And we do that for we attach them for pets that uh are on a Kubernetes node. And between entry and exit, we stash that data into a kernel map. And then

we essentially compute all the statistics around what function call was executed, what were the arguments, what was the return code, etc., etc. Right? So, architecturally, this looks like this, right? We've got a profiling agent who's running as a daemon set This profiling agent its job is to monitor constantly the proc file system and look for new pets. So, when a new process starts up, it will check

in with the Kubernetes API, and it will see whether that process is part of a containerized workload. And if it is, it's going to get all the information about it. It's going to get um the name space, the deployment, etc., etc. Now, then it will start scanning the proc system to see whether that process loads up the NVIDIA device driver library. And once it does, it will

attach all of these probes to the functions that we are interested in. functions for launching kernels, functions for transferring memory, functions for synchronizing for introducing synchronization barriers. We like ingest all of that data. Once that's happening, the profiling agent starts receiving events. And these events are then um sent and forwarded to a metrics exporter process, which aggregates them and exports metrics in a Prometheus format that are

then vi- visualized in Grafana in a Grafana dashboard. So, that's how we essentially uh arrive at these demand and back pressure metrics that we look at. Now, the question is, well, having having these at your disposal, what you can do? Can you actually use them for something useful? So, let's actually look at three concrete incidents that can be um uh where we can use this these metrics.

So, imagine um a team deploys a new version of their PyTorch training job, right? And they only state that, well, you know, we've just added some logging. That's the only change. Uh so, however, a little bit after the deploy, they start complaining that this workload is slow. Ooh, it's, you know, uh our our our workloads got noticeably slower. So, they're turning to the operator, well, something's wrong

with the GPU, something's wrong with the cluster. And of course, the operator doesn't really know what some logging means, right? GPU utilization looks the same, temperature's fine, of all things. Um the pod is scheduled, the GPU is healthy, everything's good, right? Now, they actually pull up this dashboard and they look at these signals that we're talking about. Demand is flat. So, the amount of kernels, the amount

of work that's going into the GPU is actually the same. Back pressure, however, is rising right after the deploy point. most likely there is some additional synchronization that's been introduced in in in this job if everything else is equal, right? So, they drill down and look at another set of metrics that we are that we are actually collecting, which is the transfer rate of data between the

device and the host. And the host to device transfer rate stays flat. So, we're feeding the GPU the same amount of data. However, the device to host, how much data we are pulling from the GPU starts increasing to 4 MB uh per second right after the deploy. what happened? Well, they start chatting with uh with the team owning the service and it turns out that there's some

logging the some logging change is essentially pulling full-blown tensor copies from the GPU to the host and logging the bucket information about it. So, you see just like looking at these metrics, you can gain understanding that the uh that the problem points to an application-specific problem as opposed to environment problem, So, let's look at another incident. A team is running a real-time inference service. And all of

a sudden, they start complaining, "Oh, our P95 latency doubled over the last 10 minutes. Something's wrong with the GPU hardware. The pod is healthy, however. Um the GPU is assigned, everything's fine, right? A different team scheduled a batch processing job on the very same uh pod. So, the operator looks at the metrics and sees that both demand and back pressure are dropping, as well as there is

additional demand introduced in another workload. From another workload on the same device, right? And at the same time, contention is drop is raising to two. So, there's another piece of software running on the same GPU. So, both demand and back pressure have collapsed, which essentially points to the fact that uh well, there is just less work going through the GPU. Why is that the case? Well, it's

because we actually have a noisy neighbor. So, like correlating these metrics together, these high-level metrics together, and drilling into like lower-level metrics can give you an understanding of whether that's uh um that's an application problem or it's an environment problem, right? Um now, let's look at another another example. There is an optimization tool that's recently been installed on a cluster, and it does an optimization pass, and

it adjusts the CPU limits on a training job. Pod spec, right? And then the team owning the training job starts complaining that oh, well, you know, our throughput halved. Yeah? The optimization team responsible for the tool optimizing the limits is like, well, we only touched CPU stuff. Like, I there is no reason for your GPU to to to to exhibit any kind of behavior, your GPU-related code.

so, they turn to the operator, and they're like, well, you know what? Like, you the GPUs are faulty, something's happening, right? Um it turns out the GPU is healthy. It's exclusively assigned. There is no one else on on this node, right? Now, looking at the data, what does it show us? Demand is dropping. Back pressure drops as well, of course. Um, and then, more importantly, host to

device transfer rate also drops quite quite dramatically. Which means that the demand side of this is essentially collapsing now, right? So, what does that mean? That's less data actually being fed to the GPU. So, your GPU is not getting saturated. Why is that the case? Well, the CPU limits essentially introduce a bottleneck in the data loader of the training job that is causing the problem. So, with

looking at these metrics, you can quickly identify that something is happening in the application or the environment that the the the GPU to start being, um, essentially underutilized. So, there's not enough work, kind of, going to the this is, uh, these are all metrics that can point you in the right direction, but, uh, the fact of the matter is that these approaches come with their specific limitations,

right? So, let's talk about some of them. First of all, we need to understand that, CUDA devices, as already explained, are very deeply asynchronous, right? So, Uprobes API calls, um, they don't like see into the actual GPU execution. We can't reliably get kernel execution time. Uh, we can't get, um, synchronize individual synchronization, uh, points in the GPU or like detect bottlenecks in the bottlenecks in the actual

kernel code reliably just using We can't also reliably calculate, we can probably estimate, but we can't calculate GPU utilization per process easily with these tools. You know, there's no kernel level attribution, nothing like that. So, it was all it it so essentially kind of cuts at the border of the device driver inter- interface. And more importantly, eBPF was designed for um just network level observability and kernel

level observability. GPUs were not really part of the contract. So, as much as we push these approaches forward, um they still kind of fall uh in in in certain regard. So, that being said, uh what's the road ahead? Like, what do we need in order to actually be able to what's happening with these super expensive pieces of hardware that we pay for. Well, uh I would say

that I think hardware vendors need to work a bit more on providing first-class observability APIs for GPUs. I'm not pointing fingers at specific like har- hardware vendors. Uh but they need to work to prov- to provide uh APIs that are easy to integrate, that come with a set of SDKs for very popular languages. More importantly, these APIs need to be designed to be low overhead, to be

designed with multi-tenancy in mind, and to be designed for folks who want to use them in a cloud-native world, because let's face it, nowadays we are using GPUs quite a lot on Kubernetes. Like, that's a big topic. and more importantly, with that foundation, I I a lot of a lot of tools that are that are actually being built try to optimize GPU allocation, etc. and do various

tricks to do that. Um now that's a very hot and important because of cost reduction and the nature of of of the prices of these pieces of hardware. Having these these like readily available APIs that can give us information about what's happening with the device can designers of Kubernetes schedulers and vendors of like optimization software can use these signals to more intelligently optimize the way um the

the way workloads are placed on GPUs and nodes. So so that would be a great advancement. So in conclusion, I'm going to say that the gap between GPU and CPU observability tools is is evident uh at Uh and in my opinion, looking at higher level metrics such as demand, back pressure, contention, and and and then after that drilling into lower level metrics such as device transfer, individual

CUDA launches, etc. or like histograms of kernel launches uh is uh something that needs to happen before, you know, looking at these like lower level metrics. And then um obviously, we can use eBPF to profile some of these interactions. But um it's it's not ideal. There is a lot more that needs to happen in this space. So I really hope that in a year's uh we're going

to be at KubeCon again and there's going to be a lot more that has happened to advance the space further. So that's all I have and now I think there is time for some Q&A. Thank you for coming and to this talk.