DNS Tracing & Metrics Via eBPF in OpenTelemetry- Endre Sara, Causely & Nikola Grcevski, Grafana Labs

In this session, the speakers Nikola Grachevsky and Andras Szabo discuss the critical role of DNS visibility in modern application environments, particularly focusing on how eBPF technology can enhance observability. They emphasize the challenges faced by organizations in understanding DNS-related issues, which often lead to performance bottlenecks. By utilizing the OpenTelemetry BPF instrumentation project, they demonstrate how to capture DNS metrics effectively without requiring significant changes to existing setups. The speakers explain the architecture of the project, which interacts with the Linux kernel to collect DNS request data transparently. Furthermore, they describe how the collected telemetry can be enriched with context from Kubernetes, allowing developers to pinpoint clients causing excessive DNS requests and improve overall service performance.

Good, I know it's the end of the day. People probably tired, but yeah, here we are talking about DNS. Um Cool. So, my name is Nikola Grachevsky. I'm a maintainer at the OpenTelemetry BPF instrumentation project, also Grafana Labs employee. And I'm here with my co-presenter, Andras. Andras Szabo. I'm the co-founder at Causely. Has been working with Nikola on the OB project for a while. And I'm a

co-founder at Causely. At Causely, we are building a causal layer for your agentic pipeline. Feel free to find me later if you want to know more about this. Here we are talking about DNS. It's very exciting to be back. I'm glad you guys are here. You could all be having a beer, and you came here. Thank you. >> [laughter] >> So Nikola Yeah. a while back, while

we were working on OB, and Andras showed up at the SIG meeting and said, "You guys should do DNS." And it's really important, and eBPF is the perfect technology for doing that. show of hands here, who knows what eBPF is? Oh, everybody knows. All right, so I we don't have to talk about this. >> All right. Well, Andras, why don't you tell us about why you thought

something didn't click over here? All Why don't you tell me about what was the problem, why you wanted us to actually implement this? So, so it's it's very interesting cuz I'm talking to a lot of customers, and even in our own environment um people [clears throat] are looking at their services, and they they they don't know what's going on with DNS. They see their services are slow,

and they don't know why. And there isn't a really good observable layer for your your clients interacting with DNS services. Um so, um uh I thought that we should provide this visibility and make it as easy as possible. eBPF based auto instrumentation was a exciting opportunity to do this. Yeah. People always say this, like it's always DNS, right? Whenever you have a problem, I just never knew

why it was so important. So, why [clears throat] don't we look at the actual issues? So, so what what what's actually interesting is that it's never DNS. It's always DNS. But it turns out that it's not the DNS server, it's the DNS clients. People don't know what they're doing. Um and they make random DNS calls, they put in some random host name. Yeah. Um and then it

may resolve. If it doesn't resolve, it's obvious. But it does resolve, but it takes a long time. It's really hard to know why because if you just do, and I hope everybody here is doing auto instrumentation SDK, native instrumentation, whatever, that instrumentation only starts after you know what you're trying to connect to. So, how do you know what is going on with your DNS? Is it working,

how well it's working, and is your client doing the right thing? And if not, which client is doing the wrong thing? And and you know, it would be so if we were to ask how many of you had DNS problems in the past 6 months? Cuz everybody did. How how many of you had DNS problems in a way that you were trying to build your agentic pipeline

and trying to look up your various SMTP servers? Few people. Cool people. Yes. Those are the cool guys. So, so anyway, so this complexity is going to just get worse. So, it's important that you know, we'll uh We hope to have try to fix this gap. I Today I actually found out that apparently dot net has a built-in support that you can track some DNS metrics if

you know how to connect with a trace program to get it out, but hey. Dot net's cool. Um So, why does this DNS observability matter? >> Yeah. So, so as I said, um People don't know when things are not failing, things are just taking a long Um and it's it's it's often a good chunk of I mean, most of your services do look up, a chat completion

look up, any other database look up will start with a DNS look up. And quite often these silent slow services are caused by DNS. Um it's actually quite often the problem without people knowing why it is a problem, and is it how is it even happening? Um so, just just to, you know, pretty obvious, you're trying to make a request, you first look up what IP address

you're trying to connect to before you make a TCP connection and do all the other DNS handshakes actually make a request. From the TCP connect on, it's easy. Your instrumentation would tell you about this. But how do we uncover this blind spot? >> Blind spot. Yeah, and it seems like based on some statistics here, it kind of fails most of the time. how do we do this

in OB? So, I'm going to talk a little bit about now. So, first of all, for those that don't know what OB is, or OpenTelemetry BPF instrumentation, it's a relatively new project in OpenTelemetry. It's using eBPF to do both application and network observability. So there's couple of maintainers here, I know, in audience on the project. But it's a really vibrant new project, a lot of interest from

very many places. If you're interested in this project, please join the SIG call. Um but what we do is we capture this low-level data, and we make sense of it. So, in this case, we capture DNS queries for DNS versus. It's really low overhead. Based on eBPF, you can drop it in an existing environment. It doesn't do any You don't need to restart any services, you don't

need to add any instrumentation. And if it's too much overhead or whatever, you don't like it, you can take it out, and nothing will um happen. One of the premises of the project that we wanted to achieve and actually start working on is that um it should work if you have no instrumentation, but it should also work if you already have instrumented your services with OTel for

different reasons. Let's say you're using the Java agent to instrument and add traces to your Java applications. We should be able to capture the DNS requests on top of it. And finally, this project also itself can export traces and metrics if you have uninstrumented services. That's the whole premise. We do then take all these signals that we collected, we use Kubernetes API to do workload attribution, so

we'll be able to put labels such as what was the pod that did this DNS request, what service, and so on. what actual telemetry we capture for First, I'll talk about metrics. So, the DNS spec is DNS lookup duration metric. So, this metric will collect as a histogram um information such as the what was the name you were looking up, how long the DNS request took, and

annotated with various other attributes like the service name, service name space, the typical OpenTelemetry attributes, alongside with any additional stuff such as Kubernetes or cloud vendor providers metadata, and so on. At the same time, we will be able to capture trace spans. So, if you like to see this in your traces, it's not on by default, but you can enable it as an additional telemetry signal for

DNS. It will nest correctly in the places where you're making those DNS calls. So, a DNS request will become a child of, say, a parent HTTP call before the HTTP outgoing client call is made. In that case, we collect a little bit more information because traces don't suffer from cardinality explosion issues. So, in this case, you will actually see the IP addresses and which DNS call actually

managed to resolve your IP addresses, or how many DNS call you actually calls you had to make to get to the DNS response. >> [clears throat] >> I'm going to talk about a little bit of the OpenTelemetry BPF architecture here for those that don't know what this project is about. So, how does this magic happen under the covers? So, we plug into the Linux kernel, and we

tap into both TCP and UDP. Primarily, DNS is UDP, but there's also DNS over TCP. And we look for ports 53, and I think 5353, which is also sometimes used. And we, based on the DNS ID that's part of every request, we capture the request and the response, and then we correlate those together making the actual DNS event that or capturing the actual DNS event that happened

in the kernel. From there, we also extract information about which process this belong to. So, which process on the system, which application made this particular DNS Then this information is actually extracted from the kernel, shipped through a ring buffer, which is a low-level Linux concept, onto the user space side of OB, or OpenTelemetry eBPF The user space side now takes the time to enrich this information. For

example, talks to the Kubernetes API, cloud vendor API, or various process attributes to extract and enrich this event with context. Something that you can actually make sense of. And then finally, this telemetry is then exported in OpenTelemetry format as metrics or traces. Now, it's up to you later if you want to take this, export the telemetry, and maybe enrich it further with the OpenTelemetry collector, or you

can use it as is. And I mean, some cases people think that maybe the QName is proprietary or somewhat sensitive, and they may want to mask it out. That's part of the OpenTelemetry collector, write a pipeline, remove that field, or change it to stars, whatever you like to do. and as sort of broader kind of maybe more visual way, this is what actually happens. So, we have

the O B component that typically would attach to parts of uh various runtime libraries, or maybe the application itself. But in this case, for the purpose of UDP and TCP DNS tracking, this is only the kernel instrumentation part. Then, we get the workload attribution from the actual Kubernetes API in this case. And then, export the telemetry traces through the Open Telemetry Collector. That's the general kind of

concept of how this product works, what it does to achieve this DNS tracking. So, what's the value of all this in real-world scenarios? You have more experience in that than me. I just write eBPF programs all day long. It's It's really cool and amazing magic that and the whole team is doing in in O B in the O B project. But, what's really hard is Okay, so,

where is my problem coming from? Which client is responsible for this, you large amount of DNS requests that are being made? And And not only to see that on a per client basis, but to see this in the context of how is this part of this HTTP trace versus this other GRPC trace? It It's really important to see this in in context. In addition to getting some

of the metrics about your resolver, and I think that the biggest benefit really is that you could do this as Nicola pointed out without any additional change in your code, without restarting your code. Just simply running a a daemon set instrumenting all of the workloads automatically. As far as I know, this is the only eBPF-based implementation. I think Ground Cover has a another implementation of the same

thing, but this is the only open source implementation so far. And I think it's amazing that anybody can take advantage of this. Um So, so there are the obvious things is to know more about your server problems based on this instrumentation. As your, for example, your upstream resolution may fail, or you have maybe one of your replicas that's unhealthy and it's it's it's failing the the the

lookups, or you are resource or your DNS server is overloaded. But as I said earlier, these all look like DNS server problems. Just wanted to give a shout-out to the CoreDNS maintainers. I was at their session yesterday. They're doing an amazing work in putting improving the reliability, scalability, and the performance, you know, prefetching things, running local DNS on each node. The multi-threaded support or the multi connection

support in in CoreDNS is really making CoreDNS very stable. But the problem is actually not in the server, the problem is in the client. So, how do you know which client is causing the DNS overload so that you can actually fix the problem rather than trying to just infinitely scale your DNS infrastructure? Next slide. Um So, let me drill into a couple of examples where I've seen

this, you know, being a problem. Most of the people will not do fully qualified domain names, they do partial names. So, even something as simple as the Kafka bootstrap process will try to find your your your your your brokers by going through like, is this the name? No, this is not the name. You get back a next domain response. Is this the name? No, that's not the

name. That's So, so basically you're routing through this and you're getting making thousands of calls just to be able to bootstrap your your your Kafka cluster. Of course, this is not only Kafka. You you'll you'll see this if you turn this on with possibly a bunch of your services. And the sad thing is that at the end, it actually resolves. So, from a functionality perspective, you don't

know. Maybe I did take more than 1 second. Why did it take 5 seconds? I was just, you know, starting up this service, trying to connect to this other service. I've seen our customers doing like millions of these NX domain requests, you know, rolling through a whole bunch of them all the time. And once you see this, you could say like, okay, I can significantly reduce the

load on my DNS server and improve my client performance simply by using the properly qualified name. So, Simple fix, but you need to know >> but you need to know what's going on and who's doing it. >> Exactly. And And And one of the other common mistakes is is people doing a lot of search domains. And, you know, and dot five is the default. You're basically trying

to look this up by this domain, by the other domain. So, one of the other fixes to basically just reduce your your your your search domain. But, knowing where, knowing who is doing this, and actually just highlighting that this is actually happening, because the resolution at the end is going to be successful, is really Um So, you get all of this with the O B. >> Well,

well, just one. Yeah, instrumenting DNS. I'm going to talk about how this works together in the existing Open Telemetry ecosystem, because we didn't build this tool just so that you can run DNS and just added one more tool in your environment that instruments the same thing that you're already instrumenting. Um I like to point out that this is why I made the main point, is that O

B works with your existing instrumentation if you have it. If you don't have it, it will instrument for you. So, you can plug it in the background. No restart required. The unfortunate thing it works only on Linux. Um but maybe it's a fortunate thing. Um and then, you can choose to which namespaces or services you want to track. So, it's fully configurable. By default, when you drop

it in, it instruments everything. So, it will capture DNS metrics for everything that goes on into cluster. Um but you can limit those namespaces and resources. Um And I Like I said, if your applications are not instrumented, you can use O B to instrument them. So, if you want to capture, for example, HTTP metrics or traces for certain applications that you don't have added instrumentation, you can

use it to do that. One other thing that we kind of always talk about is that because it's eBPF, um we capture the event when it actually comes on the wire, if you will. Comes into the kernel as a payload. Somebody's asking for an HTTP request. We see it as the moment that kernel sees it. So, which is quite different than your application. Your application, so, the

event has to come through the kernel stack, eventually make it to your application. So, the timing data you get out of O B are the actual timing for the request that it took. All right, so let's say you have an application that's um maybe implementing a thread pool, and it's overloaded. You don't have enough resources that you can serve a request. The request will wait in that

thread pool until there's a worker thread available. So, your application when it reports the timing for the actual request, it's not including the time it waited, because it's usually the time that the thread pool is serving the request rather than waiting for the request to be served. So, instead of that, O B will give you the full transaction output. So, you'll get accurate metrics about how long

this took end-to-end, from the kernel first seeing it to the last byte that made it out of the way. Um So, with this data, you can actually, if you have existing instrumentation, correlate and get precise timing. And with the DNS metrics being present there, you can know exactly how much of that share of those metrics was DNS versus remaining part of your transaction. And it's very easy

to install. So, we have recipes on the project, where you can actually install it just like a Kubernetes daemon set with a cube kernel command, or we have a helm chart that we publish, where you can just use helm to install with a predefined settings, which we've kind of chosen to be the best sort of defaults to start with. And this is a picture of what I

just said, perhaps of maybe a little bit more visually. So, let's say in this environment I have a Kubernetes cluster, very simple reason, maybe one node. And on this node I have a Java service, and I have Node.js service, and I have a Go So, I've done the work and I've used O Tel instrumentation to capture, for example, traces on the Java service and the Node service

with maybe Node.js auto But I also have a Go service for which I actually haven't added the SDK. I haven't built anything in. So, in this case, you can actually use O B to instrument that Go service, produce metrics and traces, anything you like. But the DNS metrics are captured for all of these services because you've said I want to capture uh DNS for the whole cluster.

Now, this sort of detection of what O B does with the already instrumented application happens automatically. You don't actually have to specifically say, I don't want that service instrumented. Because O B monitors the whole traffic, it knows that this particular service is exporting its own telemetry. So, we know that the service is already doing the job, so we don't actually duplicate the So, it's sort of like

we're watching for OTLP traces export. Or OTLP metrics export, and whichever is happening, that service is deemed that it's already doing the thing. Don't Don't duplicate data. But DNS is a signal that doesn't exist in all these services, therefore, we're able to capture it on top. Once you actually correlate with that, you can ship it to your observability stack or enrich it further with All right. So,

let's see what this looks like and so much talking. I didn't want to put any dashboards and show histograms over here, but technically to give you a perspective, this is one trace captured. And you can see there's some payment service here, an example payment service calling some checkout service in below, but if you enable the trace correlation with DNS metrics, you can now all of a sudden

see the DNS request being made and captured. So, we can see in this case two DNS requests actually were made, one for IPv4, one for IPv6, so A and quad A happened for this particular call every time the service calls it. I know some programming languages internal libraries cache the IP address. In this case, this particular library doesn't, so every time you make a call to the

remote service, a DNS resolution happens. So, you have 100 requests per second, 100 DNS calls per second. Actually, in this case, double because it's asking for both records every single time. This is a kind of things you can see, observe, and maybe say, "Oh, maybe I can cache this in the next 5 minutes or if I get a failure of connect, I can re-ask the DNS." The

kind of optimization can do at the application level to help or remove the overload on the DNS server. So, let's recap, Andre. Yeah, so so it is I think it's really cool that you can easily have access to all of this information by simply enabling OB across your cluster or on your nodes. It's ideal because no change is required on the part of any of your developers,

no restarts are required, so your services are not disrupted. What Igor also mentioned is that not only that it measures the real timing, but it actually runs most of this work outside of your process space. Yeah. So, that your process is not burdened by capturing and exporting the traces by itself. It can just focus on doing the business that it does and OB does all of the

work for you outside, which you could constrain if you wish separately. you basically can run this in production. We are helping a lot of our customers to deploy this across all of their clusters, you know, hundreds of nodes, thousands of services getting benefit from not just DNS, but the whole OB instrumentation instantly without developers having to work for next couple of years. Yes, yes, yes, yes, yes,

yes, yes, yes, yes, yes, yes, yes, yes. Yeah, I often actually ask the question because eBPF is known in the world as, "Yeah, that's a technology that requires privileged access and it's something maybe it's a security concern." So, I've thought a lot and hard past years working on this project and how that impacts. I think it's opposite in my opinion, but maybe I'm biased. I just wanted

to share here with you. Essentially, yes, this is a privileged technology because it needs to read the network layer, maybe taps into things like libssl to extract the payload and all this information, but it's a standalone daemon and process that you can actually secure. Like the OB image itself, you cannot even log in. There's no shell in there. There's nothing you can do to that image. You

can clamp it down. Yes, it's privileged, but it doesn't export any ports. It runs outside of your application process space. It does any security perhaps flaw based on, I don't know, dependency we're pulling into the project or the programming language go itself that's discovered, it's only solely a security problem to that monitoring tool, nothing else. Well, if you're adding instrumentation through SDKs into your application, that's actually

part of your application process and you may be exporting this traffic to the internet. If you have a flaw in the instrumentation libraries, like a security CVE or something, that is all of a sudden a security CVE to your application because you added instrumentation. When you do it out of process with eBPF, that problem just doesn't exist. And any crashes or malfunctions of any way, instability into

an instrumentation project does not impact your runtime actually reliability of your service. So, if you have P9999, you have three 999s after you add the instrumentation. That equation never changes. But if all of a sudden I have an instrumentation that's misbehaving, maybe allocating a lot of objects, stressing out the Java garbage that particular problem may affect your P90 something, whatever you have as an SLA for your

application. Again, this is out of process, sits on the side, can be locked down, put in a separate namespace that only a handful of people have access to or maybe some automated tooling alone, and you can really secure and isolate your monitoring tool from the rest of your application stack. we are at the Q&A. We have a link to the project if you want to, reach out

to us. We're also on the CNCF Slack. All the maintainers are there. We have a channel OpenTelemetry eBPF instrumentation. We also run a sig call, which is on Wednesday. It's been an hour ago. Yeah, an hour ago. Yeah. >> Um So, yeah. >> [applause] >> Yeah, I didn't know what the setup is going to be like, so we said we're going to bring you a mic, but

I see two mics there. If you want to ask a question, please approach there. Hey, hi. Thank you for the talk. I just have one question. Does this require any special permissions when deploying? Yes, absolutely. Yeah, so like I said, this is a privileged tool to be able to read things like network data and so on, you need certain privileges. Um some of our examples do show

this container being deployed with privileged true, but that's not necessary. OB works with the detailed system level permissions. We have a recipe that shows which permissions you have you need to give the process to capture this data. Thank you. Thank you for the amazing talk. I just wonder in case if you have a multi-level like DNS system in our cluster, it would be nice to correlate, for

example, CoreDNS hotel spans with this trace. And I just wonder whether you consider it somehow to implement this. yeah. Yeah, I see. good question. I don't know off the top of my head. Sort of the trace support right now, unfortunately, OB is either or, so if you have your existing instrumentation, then OB will detect that you're exporting traces and it will not send these spans. We could

work on a way that we can just because we know the trace IDs for most of the application because we look at the traffic, we could potentially send them. They may not be correctly nested, but at least you see them. Um yeah, that's an option for us to kind of look into. Thanks for bringing it up. >> Thanks. Yeah. Hello. Thank you for the talk. I have

a quick question. Mhm. So, at the beginning of the talk, you said that you are doing the filtering for the 53 for the DNS. Mhm. Would us be able to expand that filtering to other stuff, other ports, or whatever? Yeah, it's possible. Right now, it's hardcoded to 53 and 5353 because it's something we've seen, but yeah, we can provide a config option that you can specify which

ports you want to monitor for DNS. Okay. It's not in the project, but it's a good suggestion. If you do have this suggestion, please make open an issue with the project. Some of someone will look at this. Yeah. Thank you. Thanks. Good idea. Hello. Thank you for the talk, also. I two questions. Hope they are not stupid. Um yeah, I've been trying out Grafana Beyla, and I

am I correct in understanding that OB is like the next step, sort of, of the implementation? Yeah, so this project before the mid last year was known as Grafana Beyla, then Grafana donated this project to OpenTelemetry. So, now Grafana Beyla is still a product, but it's just a shell on top of OB. OB is the main product. Okay, because I also quickly checked the repository and it

says it's still in development. Do you have like a ballpark when you try to do it, when when it will be like productionized? Yeah, we committed to release candidate of the upstream project this year, so yeah. All right. It's happening, release candidate, so Good luck. Yeah. I don't know if this helps. Obviously, not the official answer. I'm running this across >> [snorts] >> nodes in production across

many of our customers and it's working amazingly well. So, obviously, that's not a qualification on the official, you know, stability, but like practical experience, it works amazingly great. Yeah, so the Beyla, which is the downstream project, which is just a shell on top of this, that's built is officially supported by Grafana, but yeah, the upstream project release candidate is coming up. All And I also have a

question. When you have a service that is auto-instrumented and you have OVI that is capturing telemetry for the service that are instrumented, in the picture that you have, you are still getting DNS from the service that are even auto-instrumented? Yeah. How are you in the same trace? Because I imagine the service is instrumented, you have already a trace for that HTTP request. How are you injecting Yeah.

the span ID and parent span ID for that one? Are you making some kind of magic? Yeah, no, there's no magic there. So, that particular one doesn't quite work well. So, this trace if it was generated, we can't do this at the moment. If this trace is purely generated with OB, but if you have auto instrumentation and you want to add the DNS span to fit properly,

we can fit it within the transaction, but we cannot fit it exactly where in within your spans, especially if you added custom spans, that would not work. But, it will be we can find the trace ID and emit that span. However, you can get the metrics, and that was what this picture was showing. So, you can capture the DNS metrics, so you know that for that outgoing

call for that service, you're hitting DNS so many so many times. Uh but, we can add a feature, that's what the one of the previous uh pe- uh folks said actually here that we could provide it as a sort of maybe not a span, but maybe like a span event, and then you get an idea at least >> Yeah, but not only that, you mentioned that when

you are auto instrumenting, the time for the HTTP request is that one, and when you're using OBI, you are capturing from the starting really to the end, so maybe it's >> Yeah. side, right? That DNS is outside of So, time doesn't even fit. It fit. Yeah, so again, we have to look at if you look at the If you typically people instrument traces, I found so far

with OpenTelemetry is very popular. The that signal is telemetry signals particularly uh favored, uh but you can capture with OB the metrics, so you'll know the total time. And it's up to you to figure out then, okay, so I see my trace, this is the timing, and then with OB, I got the total time, so I know maybe there was something else going on. Uh and it's

probably some of the stuff that the application can't see, like a lockup on thread pool, or uh time took in the kernel for the data to come through. Um that kind of detail is really difficult to discern from. I mean, I think this this context propagation is is important. Yeah. But, I have a bunch of customers who are just simply saying that like, "Oh, I never realized

that I used to make 100 DNS requests, and now I'm making 1,000 DNS requests a second from this particular client." Yeah. Uh and that in itself is like, "There's something wrong here. We changed something." Uh so, not to discount the importance of the context, uh knowing which client is doing what Yeah. is in itself helpful as well. Yeah. Yeah, thank you. It's a good question. Oh, we're

out of time. We'll we'll be right here. Uh continue asking the questions. Yeah, thank you so much for coming. >> Thank you. Appreciate it.