Hacking GPU Observability: eBPF & Ephemeral Containers in Action on Kubernetes - Brandon Kang

This talk focuses on GPU monitoring and debugging in Kubernetes environments. The speaker explains the challenges faced when observing GPU workloads, as GPUs often fail silently and display unpredictable performance issues. He introduces eBPF and ephemeral containers as tools to enhance GPU observability while avoiding disruptions in production. Traditional monitoring tools like NVIDIA DCGM provide limited insights, primarily showing hardware metrics without explaining performance fluctuations. By leveraging eBPF, users can obtain detailed kernel-level metrics that reveal interaction between applications, GPU runtime, and drivers. The session also discusses implementing real-time observability, debugging practices with ephemeral containers, and ensuring secure monitoring of GPU workloads in Kubernetes.

Okay, thank you. Thank you for coming. My name is Brandon. I'm from Akamai. So, today I want to talk about a simple question regarding the GPU monitoring. how do we observe the GPU workload in Kubernetes? And I will talk about how do we debug them safely in production. So, GPUs don't fail like CPU. So, they fail very silently. And they often at the worst time. So, in

this talk, I will show you and how we use eBPF and ephemeral containers to understand and debug GPU workload without any breaking production. So, if you ever worked with the GPU workload, you've probably seen that some strange behavior that the GPU Sometimes you hit out of memory. And sometimes the performance suddenly drops. And sometimes the things just slow down without any evidences. So, if you see the

the left picture, CPU workloads can be predictable. But the but but the in the the right corner, the GPU workload I would say it's a kind of the alien landscape. So, CPU workloads are very different from CPUs. So, we have many issues like the thermal throttling from the GPU or the kernel stores. And they don't always show clearly. So, they just reduce the performance quietly. And that's

the real problem. So, we often don't notice anything is wrong. So, that's why I designed this presentation and this session for the GPU observability using the eBPF. So, some naturally we turn to tools like the DCGM, NVIDIA DCGM, or NVIDIA-SMI to see what what are happening right now from my GPU. They give us some metrics like the GPU utilization, temperature, and memory usage. And at the first

glance, they seems very useful. But then when you run into a situation like um you have the same GPU running the same model, but one this instance is running at 80% GPU utilization, but another is a stuck at 20% only. And the tools don't tell you why. So, I would say that the the traditional GPU monitoring tools, they just show hardware metrics, and then they let you

know that something happens, but you cannot find any root causes. So, this is what I call the visibility illusion. So, I think that we can see what's going on um if we using the eBPF and then all the best tools with some algorithms. So, let's talk about the DCGM first. Actually, it means the data center GPU monitor. But not only data center, but also your the GPU

instances on the cloud, your DGX on frame basic GPU machine that you can install this open source, and then you can utilize this for very basic level of the GPU So, it is actually a very powerful tool. It gives you a wider range of hardware level metrics including GPU utilization, which is very crucial for your GPU cost optimization. And then it shows you how much power they

are consuming right now, and GPU even error count that they check. And then you can know that. But all of these signals comes from the GPU hardware level. They describe only the state of the hardware. So, not the interaction between application and GPU, and the GPU runtime and the GPU driver. So, because the most performance problems we cannot see the reason of why this happens from using

this tool. So, we need another approach to check the GPUs more in detail. So, this is some examples. So, when you see the DCGM monitoring in Grafana, you can see some I made some scenarios for this presentation. So, this is what we typically see with DCGM monitoring tools. So, we can memory, and temperature. At first glance, this looks very useful, but here's the problem. So, if you

see the question mark in yellow color, we still don't know what's actually happening. Why did the utilization suddenly spike? Why did the temperature drop suddenly? Why is the memory flat, but performance is low? So, we see the signals, but we don't see any root causes. So, that's the key limitation of hardware level observability. So, it tells you what happened, but not actually what why it happened. We

cannot know that. So, another case, if we break down a typical element AI inference stacks, we can use we can see where the problem is hiding. At the top, there is application layer. Maybe there can they can be a PyTorch or the TensorFlow or llama VLLM, any inference engine you are using right now. At the bottom, you have the GPU So, it's like hardware and software that

between them, there is another layers. They are They CUDA runtime and the CUDA driver layers here. If your problem happens here, it's a very hard It's a very tricky to uh to find out the root causes what happened on my the GPU instances. So, where this information can be extracted from kernel level. So, where memory is transferred, where the kernels are launched, and where execution is orchestrated.

These are kernel level metrics. But traditional tools don't give you the visibility into these layers. And this is another example. So, let me give you one concrete example once again. So, this is transformer based model examples. You are not running one big computation, but you are running thousands of small CUDA kernels, for example. Each one is fast, but there can be a delay. And when this happens

thousands of times, it became a very significant issues on your GPU machines. So, the GPU looks underutilized. Uh but it's actually just waiting for some data from uh CPU. So, there can be a bottleneck or latency. So, another example on the the right side, this example describes the data movement from one GPU to another GPU. So, if you are frequently moving data between CPU to GPU or

GPU to CPU, or the between two different GPU model, when we order some the data movement between two different component, there may have some latencies as well as some um the slower performance as well. And thinking about the security once again, and now what's interesting is this same blindness but also created some security risk. So, because if you because you cannot see what's happening in the CUDA

runtime level, you also can't detect any malicious behavior inside your CUDA runtime. So, for example, this picture describes that the your container might start running unauthorized kernels. So, someone may use your GPU for crypto mining, for example. So, we collect a lot of the kernel. So, or you might see any normal the memory transfers, which could indicate a data exposure. So, unauthorized execution can be happen on

your GPU. And this kind of the security problems are not detected by GPU observability tools. So, if you don't have any observability on this layer, you don't have any security, you know, the strong security either. So, we need a more the stronger approach to detect the security issues in advance. And some may thinking about the legacy profile. So, for GPU, there are many legacy profilers. It's kind

of the software. So, why not just use a profiling tools to detect everything for performance and security? And yes, the tools like, you know, PyTorch profiler and NVIDIA Nsight can helpful, but they come with some trade-off. The most important the critical problem is that they require some your application code changes. They introduce some overhead on your overall performance and they don't hit fit well with the Kubernetes,

you know, principles. So, in particular, they break immutability uh on your Kubernetes the resources. Uh so, if you have your work node and then your deployment application on production environment, that's usually not acceptable if you use, you know, profiler and then you added some the runtimes and you your deployment uh at the same time, you know, there can be some issues when breaking your original applications. So,

legacy profilers are good tools as well, but it may break your stability of your application on Kubernetes and it can be a penalty for performance. And sometimes we may have some vendor lock-in if you use some spec- a specific the GPU models or the software layers. So, that's why we recall the eBPF. So, I believe that many of you have heard about eBPF. So, it's a, you

know, it's a kind of the Linux kernel technology that learn that allows the running sandboxed the custom programs within the kernel without changing any kernel source code or your application code. So, it enables high-performance observability and networking layer security acting as a the programmable. So, someone say that eBPF does to the Linux kernel what JavaScript does to HTML. This is a very well-known quote about So, in

other words, it turns the kernel into something programmable and dynamic and that's exactly what we need for advanced GPU observability. So, thinking about only HTML, it's a just kind of the static file. Uh you cannot make any dynamic content only with HTML. But, with the help of the JavaScript, you can make a dynamic contents, and very colorful UI and then funny UI that thanks to JavaScript features.

So, we can give some lives eBPF for your Linux kernel. And I believe that most of the Kubernetes clusters they are running on the Linux operating system. I know that we know that there there's some someone prefer the Windows OS and uh someone is still prefer to learn your the Kubernetes clusters on the Windows, but I don't have much knowledge about uh eBPF's the if eBPF is

also supported on the Linux OS, but I believe that 90% of the Kubernetes clusters are running on the Linux operating system and uh and the they are uh they can be communicated using eBPF together. So, uh if we see eBPF in detail, they are running inside the Linux kernel and it give us the visibility across the entire system. So, eBPF can observe everything in the box that

you can see right now. So, from application side was and system calls down to drivers and hardware levels. So, instead of some guessing, that we can see what's really happening inside the system, I mean a Kubernetes system in real time thanks to eBPF's the features. Okay, so eBPF they allows it allows to learn the small program inside the Linux So, this means that uh we can observe

a system behavior without modifying any application at all. So, you don't need to touch your Kubernetes application to use the eBPF. Only you what you have to do is that just um check the your uh Kubernetes clusters the CNI support eBPF. For example, rather than Calico, uh Cilium support eBPF, but if you are using the specific cloud vendors the managed Kubernetes, anyway, you need to check if

your uh Kubernetes the network layer support eBPF. Then, you need to download some open source source code or you can customize your own or you can develop your own the eBPF the monitoring the source code utilizing C language or C++ or the Google's the Go language. Then, you can connect that the source code to your Prometheus to visualize uh in the Grafana dashboard what's happening on your

by eBPF. They can be learned as a demon set pod on your every single the worker nodes. So, what's powerful here is that this works uh is it can be learning regardless of your any framework. It doesn't matter if you are using PyTorch, TensorFlow, or something custom. Everything passes through the same underlying system uh Linux kernel. So, uh we can get the many metrics from the the

via eBPF and they can be uh collected by eBPF. So, um after that, in the end, we can have more insight on what are really happening on your So, I would say that that DCGM we'll see the screenshot of the NVIDIA DCGM. It shows what happens on your Kubernetes cluster, but actually eBPF's they have explained why it happens. So, you know, root cause they are they can

be came from application layer on your user space behavior. They can be from system core layer or kernel layer. It's a very rare cases that your the Linux operating system or Kubernetes scheduler may have some issues and some problems can be from input output or data movement between two different CPUs or GPUs and it can be happen on your the GPU driver interaction layer and container Kubernetes

layer. So, thanks to eBPF the kernel level monitoring tool, uh we can find out that what are really happening and what kind of issues are happening on your Kubernetes cluster. Okay, this is the one example this is shows some more multiple metrics rather than DCGM. uh it doesn't take any it doesn't take you more than 10 minutes to make a this kind of the Grafana dashboard. All

you have to do is that you can find that there are many eBPF open source tools on GitHub. So, you may heard about BPF uh trace and then there are many fancy tools regarding the eBPF or you can make your own the customizable uh eBPF monitoring tools just using the C language or the Go language Google Go language, you can make your own. So, this is an

example how and then I made a some the stress on my GPU machine. you can see that not only hardware level metrics, you can see what application uh makes some errors. So, you can see the error count and what are really happening on the kernel level CUDA runtime level. We can see more the information here. So, this is the original the C language file. So, we need

to some expertise on the C language to make if you want to really make your custom eBPF system, we need some the coding skill or you can uh you may get some support from your Bible coding tool. So, this example shows that how I could get the eBPF the kernel level the metrics uh via eBPF the tool. So, you can see in the line number 1 2

3, there are they include some BPF BPF the headers to utilize the eBPF features. So, you may find this kind of the source code or tools on the GitHub. It's They are everywhere. But but um after that, you need to make your own Docker image utilizing this source code. And you can learn that Docker image on your Kubernetes cluster as a daemon set. So, there are many

ways to implement eBPF on your Kubernetes cluster. the simplest way, I think, is that to make a daemon set for your uh every single work node the Then uh there should be some interface uh source code uh to deliver this kind of the metrics uh to your Prometheus server. So, this can be another C file, another geo file. And then uh the Prometheus will send this metrics

to your Grafana dashboard. And they will start to drawing that this kind of the metrics. Okay, so I put some you know some explanation about my source code. And then I will share uh those source code on the GitHub. because I just completed uh this demo this this morning. So, I had to optimize the source code once again. Actually, until yesterday, I didn't supposed to make uh

this demo. Just uh you know, screenshots are everywhere. eBPF the monitoring tools the screenshots are everywhere in the internet. But the I found out that the this morning, I woke up earlier than usual, so 6:00 a.m. in the hotel. I found out that the more than 700 attendees they registered to this session. So, uh I decided to make uh the dynamic demo for eBPF the the Grafana

dashboard. But as I mentioned, it only takes uh it only took me uh 10 minutes. Yeah, you can trust me. So, I uh make some description uh for your understanding and the they will be on the uploaded on GitHub. It's a private right now, but I will update my slide to for you to uh access uh my GitHub to analyze the source code. Okay, so uh eBPF

in action. So, tracing CUDA runtime is also possible. So, execution tracking uh in the CUDA level, uh there is a function called the CUDA launch kernel. And then another data movement uh another cases uh when it comes to the GPU or CPU and GPU, uh in the CUDA function, there is a CUDA memory copy function. So, what you have to do is that this these are kind

of the event uh you you need to decide your requirements. So, do I have to have the CUDA launch kernel function as well as the CUDA memory copy function? And then uh they will collect uh required metrics on eBPF layer. So, you can see that uh after tracking this kind of the metrics, uh we may have some we may find some latency in the CUDA level. Actually,

they cannot be found in the DCGM like the the basic very basic the uh tools. They cannot be found. And then you can customize uh your you found your issue and then you can customize uh you can update the your source code or you can find issues and resolve them. After that, you can find the you know, the lower latency than before. For data movement, so host

to device or device to host, it can be happening everywhere. So, if you find uh it makes some uh significant latency or if you find out that it hurt your overall performance, you need to optimize the this layer as well. So, a sudden spike uh in device to host the transfer the flags on the immediate security problems anyway. Uh also, they can be make some uh security

problem. You can uh we can have some suspicious that if it's sudden spike happens on the data movement layer, uh and then your Kubernetes clusters are exposed on the internet even you have authentication. Uh there must be you know, if it's someone access to your and then start some crypto mining, for example, it's a very rare case, but anyway, then the uh the the CUDA memory copy

they take more time than usual. So, it means that uh something is happening on device level, host hardware Okay, so uh now you can start to see the differences uh with uh traditional GPU monitoring tool and eBPF based cloud native observability. So, visibility scope hardware only versus the deep runtime the kernel level visibility. And we don't need any uh heavy manual code changes, but uh zero code

automatic kernel attachment is what we have to do just for using the eBPF. And ecosystem lacking thinking about uh we don't have any vendor locking because eBPF is a kind of the Linux kernel technology kind of the open source. uh if you follow just traditional uh GPU monitoring tools, they only support uh you know, the two major the GPU vendors. And security context the process level and

network level context uh they are what you can obtain uh from eBPF level. sometimes we need uh debugging. Especially when something goes wrong in production, what do we usually do? We'll we check the logs in the container. Uh we attach some then uh we can run a command uh called the kubectl execute or we can access to the worker nodes directly using the SSH. But this approach

has some problems, especially the security problem, immutable problem. When it comes to production environment, it breaks. You know, it introduces some security So, it often uh violate the compliance requirement of your companies. So, uh what I suggested to use the ephemeral the containers. It means that instead of the modifying the running container, we inject a temporary debugging container into the same pod. So, in this diagram, the

ephemeral the container is the yellow one. So, it shares the same uh name space, so we can inspect the system in real time. But once we are done, it'll be disappeared. So, it'll it won't to touch your original application. So, observability the hack workflows, I suggest that we can have some the four steps. So, detect uh your what what are happening in on your Kubernetes cluster by

using DCGM and eBPF together. And deploy ephemeral container to debug uh to monitor the log in detail. And analyze the shared name space and limit it and fix the your problems we without any the zero down time. Uh this is a typical workflow uh how you can detect and how you can resolve. So, in this flow, you need some metrics and workflow uh tools called uh DCGM

Explorer, eBPF observability layer, runtime security uh layer. Uh for example, Cilium, Tetragon is a good for the security of the your container. And network observability utilizing Cilium and Hubble, uh they can be helpful for your full journey for GPU observability. So, my ideal uh structure architecture for full stack GPU observability is like this. So, there should be some visibility layer utilizing the Prometheus and Grafana. Under visibility

layer, uh we can have auto OpenTelemetry collector for option. Optionally, we can have it. And under that, Okay, time is up. So, it takes more time than expected. Sorry. So, uh we have the runtime tracing and security layer and then hardware layer in the bottom. Uh this is my final phasing. So, virtual 90 infrastructure is just cloud native infrastructure. So, I hope that my session will be

helpful for you. Uh now we know that the GPUs are not no longer the black boxes. And then we can put the data cost visibility utilizing open source and EVPF. Sure the secure debugging can be what we can achieve from this pattern. And then finally the at the end of day, the machine learning infrastructure, if you have the GPUs on your community's worker node, uh, it's it

can be a now it can be a safe safe cloud native for your enterprise companies workload AI workload. Okay, this is the end of the slide. Thanks for sticking around. So, you may contact myself uh, for this session or the the GPU on Kubernetes cluster. You can also download the session slide utilizing this QR code. Uh, due to time limit, uh, sorry for uh, we don't have

the Q&A time. Maybe I'll be there. I'll be the behind the room. And then we may have some communication or Q&A. Uh, in the end uh, the behind the room. So, I'll be there. If you have any question, please contact Okay, thank you for attendance. And then thanks a lot for sticking around and have a great day. Thank you.