Retroactive Sampling with OpenTelemetry: Cut 90% Distributed Traci... Roman Khavronenko & Zhu Jiekun

In this session, the speakers explore retroactive sampling in OpenTelemetry and its potential benefits for reducing costs associated with distributed tracing. They explain the challenges of tracking requests across microservices and the high resource consumption of traditional tracing methods. The talk contrasts head sampling, which is cost-efficient but limited, with tail sampling, which is more resource-intensive. The speakers introduce retroactive sampling as a solution that minimizes unnecessary data transmission while maintaining effective sampling decisions. They detail how this method can optimize memory and CPU usage compared to existing sampling techniques and share benchmark results from their experiments. The talk concludes with an invitation to contribute to the open-source project related to this concept.

Thanks again for everyone to who joins here. This is the last session of today. So, in the next 30 minutes, we are going to present retroactive sampling in OpenTelemetry and how this idea can help you to reduce your cost when using distributed tracing. Um a little bit about the speakers. My name is Shu-Jie Khun. I am the software engineer working at VictoriaMetrics. And my name is Roman

Khvoroenkov. I'm engineer manager at VictoriaMetrics. So, uh we are both from both of us are from VictoriaMetrics. So, we are the company behind open source metrics monitoring solutions also named VictoriaMetrics. And recently, we start to build our logs and traces solutions. And VictoriaMetrics is our trace solution and it's reason why this talk was here. Uh we start to trying to find some pain points from the user

side, especially when doing some sampling. And hence this talk. Today's agenda, we are going to start from the beginner's section. Uh what is distributed tracing? How does it cost? And what what's the typical way people use to reduce the cost? And we'll talk about tail sampling and what's the problem with tail sampling? Why is it expensive? And this helps introduce our idea the retroactive sampling. How does

it work? And what components does it need? And we will show you some numbers with the benchmark comparing with other sampling or no sampling solutions. And you know, every solutions comes with with the trade-off. So, we are not going to hide anything from you. Uh we'll talk about the trade-off variants and alternative. And lastly, the summary. Um before we could go into the slide, uh maybe like

uh there are a lot of experts here. We don't want to keep you from waiting. So, why not just show you the benchmark result? But with all those comparison objects name masked. So, you can maybe you can make a guess and come back later in the third and fourth session and see if you are uh making a correct guess. So, let's begin. Okay. Um hi. Let's go

into some basic understanding of what distributed tracing is and why we need it. So, let's say we're writing a service that should accept user requests, do something with that, and respond with something funny or useful. Um usually when running monolithic applications, we have all the context when processing this request. We can check them, our permissions, we can look into database, we can do whatever, but we keep

it everything within one process, within one unified memory. If something goes funny with this request like an error or delay, we have all the context in order to troubleshoot this request. We can log it, we can print a lot of debugging information. But nowadays, it's a bit different. Nowadays, we run microservice architectures where our service is represented as a bunch of separate processes. And those processes can

run on different machines and on different um physical nodes. So, now when service A receives this request, it can consult um service B about something. Service B will consult service C and so on and so on and so on. And this context um is not so easy to track anymore because these are different processes, the memory is not unified, and we need somehow to propagate this context

between this nodes, between the services to get understanding what happened to the request. How we do this with distributed tracing? Well, every time when we emit a new um procedure or an action or we hop from service A to service B, we generate a unit of information called span. Span is something that describes this operations that we made and put all the context that we will need

later into its um attributes. Later, those spans are collected into a trace like a parent entity that will unify all the spans within one user request journey. And all together, uh it becomes a distributed tracing, a tool that helps us to understand what happened within the distributed system to the user request. Where was delay? Where was error? Et cetera. Later, we can check this trace and we

can see immediately where every span was generated on the timeline or how much time it took. This is how it looks in the user interface as usually. It's a like tree-like structure when every action is every span is listed as an action. You can click on it, you can get all the information to what happened. You can see on the timeline when uh what the sequence of

the spans execution, et Uh one trace can have like 10 spans, 100 spans. It's okay to have thousands of But the thing is that spans are pretty expensive. On average, they taking about 1 kilobyte of memory. That's because they contain a lot of context, a lot of attributes within them which like represented as key-value pairs where values are like text information. This is why tracing becomes one

of the most expensive signals to process because like compare resources spent on processing 1 million samples metrics per second with 1 million spans per second. So, you'll have to pay a lot for network, for compute power to process this, and for disk for storing these traces. This is why people trying to find ways how to reduce costs on distributed tracing. And one of the techniques is on

the surface is in OpenTelemetry Collector, there are two types of something that you can that you can use. Head sampling and tail sampling. Head sampling is the most easiest one and also the most cost-efficient one. Basically, the idea of head sampling is that the first service that receive uh the request and initiates the first span, it can decide whether we sample this request or no. It can

make this decision based on the on the attributes of this request. Like for example, we do not generate trace for requests coming from the development environment or we do not generate traces uh specific um agent or whatever. Later, this decision is propagated to all other microservices and all of them knows that no need to generate span, no need to spend resources on that. In result, we send

to the tracing backend only that portion of the events that we know uh that we need to know. We can use probabilistic sample sampling here for example like to send 1% uh of the traces. However, there is a limitation with the head sampling. In the moment when we make this decision on service A, we don't know what will happen to this request in future. What if this

request will take more time on service C than our threshold? What if on service B something happened anomalous and it will be an error or something like that? On the moment when we take a decision on service A, we just don't know that. This is why there is a tail sampling. With tail sampling, we always unconditionally generate the spans and we send everything to OpenTelemetry Collector, which

will make decision uh later when it will receive all the And then it will decide whether it needs to be sampled or no and will send sample sample traces to the trace backend. There is a problem with tail sampling, though. And this problem is that it's pretty [snorts] expensive. And let's see why. So, we have a real real world example where we have a bunch of physical

nodes, um tens, thousands. Uh we run our microservices on those nodes. And we have OpenTelemetry Collector with tail sampling processor enabled. Those who configured tail sampling processor, you know that you need to specify the time window. The time window for how long we need to wait for all the spans of our of our traces to arrive. So, in order to make decision about was uh anomalous, do

we need to sample it or no, we need to wait to receive all the spans of this trace. And since they like happening at random at on on some timeline, they happen in different moments of time, we need a time window to wait for them. Because spans are pretty expensive like 1 kilobyte on average, the longer is the window, the bigger is the memory consumption. The more

unique traces you have, the bigger is the memory consumption. So, OpenTelemetry Collector becomes a bottleneck. At the moment when you can't scale it vertically anymore, you start scaling it horizontally. And another challenge becomes here because to make a wait decision on whether something whether we need to sample it or not, we need to receive all the spans within one trace in the same OpenTelemetry Collector. That's why

we have load balancers that need now to route requests charge requests based on the trace ID. Uh load balancers could be external services. We can also deploy them locally in the nodes in the form of daemon sets. But the thing is that we still need to do this extra routing uh additional to the memory usage that we consume. Another thing is network overhead. Because the decision is

made not on the applications that generate these traces, but on the OpenTelemetry Collector, we need to send all these spans, all this information to it. However, mostly with tail sampling, we want to keep only those requests that were anomalous, resulted into an error, or were just too long. >> [snorts] >> And I think in most of cases, requests that our services process are okay. They're within the

thresholds. We are usually interested in the small fractions of those requests that are unique bad. in reality, it's about less percent of the requests that we need to actually have traces for. So, we need to drop 99% of this information that we received on the floor and keep only 1% or even less than It starts to matter a lot when you have multi-zone deployment in different regions

because suddenly cross-region traffic is something that you need to pay for and spans are expensive. And when you send it from the different regions, you will have a bigger monthly bill in your cloud provider. Okay, and this is what takes us to the idea of retroactive something. Yeah, thanks Roman. I just woke up and when I heard someone want to drop 99% of the data. Yeah, so

what is retroactive sampling? So in this case, if you are really care about your network traffic, then retroactive sampling might help. What is it? So our idea is trying to eliminate network traffic or let's say let's call it unnecessary network traffic across those nodes or cross us. So in previous example, we are dropping 99% of data. What happened if you can find a way to like to

just send out less data, send 10% of the data while keeping the sampling result consistent with the 1% of data you need. It It brings you 10 times better ratio, right? But how to do it? So maybe we can start from taking a closer look at the what the spans look like. Here is an example. You can see that there are a lot of attributes on it,

resource attributes, span attributes, logs, events. There are a lot of things and most of them are not contributing to your sampling decision, right? If you are going to make the decision based on the duration of a trace, then why not just take the start and end time? If you want to sample error traces, why not just take the status? So our idea here is trying to drop

the other attributes. So here is an example to send less data to the tail sampling collector, but now it can be used to make the decision, but it's not something complete. In this case, the tail sampling collector need to return the sampling result back, basically a set of trace ID to your local agents. And local agent, it has to find a way to buffer the data, buffer

those data until it can receive those sampling results and it can Now it can combine the two and trying to send the sample trace span to the trace back end. This is the overall idea and now let's take a closer look. Let's begin from the local OTel collector's angle. So how our local collector works here, let's let's call it local agent, right? The local agent, every time

it receive new spans, it need to buffer them, like just write it into a FIFO queue. On this FIFO queue in batch. And in this during this process, progress, it has chance to scan over all the all the spans and it has the ability to extract some key information for those spans. Now it can send those key information to the sampling server and the sampling server, basically

it need to do what the tail sampling processor needs, right? It buffers just buffer less data, do the sampling decision and now send the decision to your local agent via pull-based or push-based manner. Now the local agent, it now has the raw data, it has the decision. Start a consumer trying to consume these FIFO queues and combine the two and make drop some traces in place and

send the rest to the trace back end. This is what happened in your local agents. And what about the sampling server? As I mentioned, the sampling server is basically a tiny version of tail sampling processor. It still cache data, buffer data in memory, grouped by grouped by the trace and each trace would have some key attributes out there. And every time it receive request from your local

agent, it trying to update the corresponding data. And after a while, it trying to rotate those There should be multiple hash set in the memory and it trying to rotate the current one with the previous one. And now the previous one waiting for maybe just a little bit longer, 30 second, it now can start decision making. And once the decision was made, trying to push back the

data. That's simple. So So far for now, everything is is on the paper. Why not Why not provide some numbers with the benchmark comparison with other solutions? So here is our scenario. We trying We have several nodes and running local agents on that. And we have load generator sending data to those local agents. And now the local agent need to find a way to send the data,

either sampled or unsampled, to the trace back The first one, no sampling as a baseline. So in this case, basically those local OTel agents are doing nothing. They just trying to forward the data to the trace back end. And as a comparison, the tail sampling, we need to set up two actual collector in between and enable the tail sampling processor on it. In this case, each local

agent now need to do the load balancing, need to route the data to corresponding OTel collector. And the tail sampling collector, now it need to buffer the data, wait for a while and make the decision and send the sample And the last one, retroactive sampling. Just as a side note, there's no no current implementation in the OTel collector, so we built all those prototype. We have our

sampling server to replace the tail sampling collector. And we also build a retroactive sampling processor and enable it on each local agent. In this case, each local agent FIFO queue, extract key information and send them to the sampling server, wait for the decision and now and lastly send out the sample trace. The data set we use is from the OTel demo. Why we use OTel demo? Because

it's is more close to the real world, more close to the production data. And why traffic recording on pay? Because it's it's hard to control or set up multiple OTel agents and generate those traffics. By traffic recording, it's more controllable and makes the result of between each runs more comparable. The workload here workload here basically means how many data we are we are sending from the load

generator to the OTel collect local local agents. So you can see that we are putting pressure starting from 50K and gradually increase it to 30 30K spans per second and you can see a very similar pattern in terms of network traffic. So here we go. The benchmark result we have shown you in the very in the very beginning. So if you you can see if you are

making a correct guess. So no sampling and tail sampling, basically they need to send out all the data. So the network traffic is pretty close to the input traffic, right? And retroactive sampling here as an as a comparison is sending 70% less traffic all of them up after the comparison after the compression. am I doing it right in terms of the sampling result? Let's Let's see what

happened in each trace back end. So you can see this is the injection rate in trace back end. Both tail sampling and retroactive sampling is injecting data capping at about 1,000 spans per second. This is about 3% of the input input rate. We trying to sample data longer than 5 second duration or contains an error. Otherwise, we sample only 1% of those sample of those healthy trace.

So now we can see that there are the sampling results are very close and very consistent. And how many resource does use to reduce the data? Let's see the CPU and memory comparison. Well, here you can see the tail sampling is using twice as much as CPU and memory than the other two, right? That means retroactive sampling saves 60 to 70% CPU and memory for you again.

And in terms of CPU usage, you can see that retroactive sampling is even slightly better than the no sampling. Why? The retroactive sampling actually needs to do more calculation. It need to make the decision, but it send out less data, so obviously it spend less time on the less CPU time on the data encoding and decoding. That's the reason. Yeah, just another side note. We Those CPU

and memory usage are captured from the local agent or plus the OTel collector or the sampling server. It means the application, the load generator and the trace back end are excluded. So all those resources are representing like how many resource you will need for your data collection and sampling pipeline. The last one, disk space. So this is actually not for comparison because you tail sampling and no

sampling, they are not using this at all. Here retroactive sampling is using 1.5 1.6 GB disk space at max. If you remember the previous line, the difference in terms of memory between tail sampling and retroactive sampling, that's 4 GB. So here it means we are trading 4 GB of memory with only 1.5 1.6 GB of disk space. I would say that's a pretty good deal. Okay, thank

you Jacob for showing these numbers. As I know from experience, everything comes with a trade-off. What are the trade-offs in the retroactive something? So you said that we saving a lot of resources on sending only those fields that we actually need to make a decision. What if someone needs more fields to make the decision? Well, that's a that's a good question because like a lot of you

may not sample traces only based on the duration or the error status, right? An error trace, it doesn't means it has to be something you are really interesting. easiest way is to Why not just put those attribute to your sampling sample sampling data data data structure. So, just send all those attributes to the sampling decision maker, right? Yeah, it it indeed we would help, but it only

work if you have only maybe one or two extra attributes you would want to send, but eventually it's not scalable. One day your user will ask, "We want to sample based on 100 attribute 100 attributes at a time." You cannot put all those things back. Otherwise, there is no need to use retroactive sampling, right? So, what we are trying to propose here is the local and remote

decision making. So, imagine that if I want to sample something based on let's let's say the debug flag, and each span will go through the local agent. Now, the local agent has chance to see all those attributes of this span. It already noticed that there is a debug flag, and I I can decide already there is no need to send this debug flag to the centralized server.

I can decide that this trace ID should be sampled, and what I need to do is um to try to notice notify other local agent that please send me the data of this trace. So, in this case for the the trace that local agent can decide to sample, just send all those trace ID to the sampling server, and let sampling server to find out those requests. Otherwise,

if you did haven't found something interesting, then just follow the classic retroactive sampling pattern to extract those key information, and let sampling server to do the job. It works here uh even if you have 100 attributes you want to evaluate um if the local agent can decide based on your based on based on your condition. So, in this case, we can just add one byte of this

decision flag into the sampling request data structure, and if the local um for this case, the local agent will send out two type of data. One type of the is the sampled trace. Just uh if we need the sampling server to find out the find out the results. Others is the pending trace to do the retroactive sampling sampling's way. And yeah, later the sampling server can push

back or provide an endpoint for those local agent to get them get the result. So, um I mean, there is no need to extend our data structure even if you have multiple attributes that want to evaluate. Just use just just trying to make the decision locally, and that's uh aligned with our principle trying to not letting any any any data escape from the node. One of the

biggest pain points of tail sampling is memory consumption, which becomes its uh bottleneck at some point. And I've seen um in the OpenTelemetry repo, there was a pull request of a person offloaded in this in-memory structure to the local disk to reduce the memory usage. Do you think it's a better way to go? Yeah, I think this question it exists uh ever since post tail sampling processor

was created. Um people always trying to replace the memory with this right. Uh so happy to see this proposal, and also it comes with some benchmark we can maybe check those numbers. So, indeed it helps with the memory usage reducing it from 2.6 gig to 400 megabytes. That's a great news. But, I also noticed that there is a some CPU increasement. Maybe it's because this is a

also a prototype. It's not mature yet. But, uh maybe we can like try to compare them from the data structure's perspective. So, in tail sampling, no matter it's a memory-based or disk-based solutions implementations, it it's trying to build a key-value database out there because people need to do trace ID lookup. You need to find all the span for this specific trace. And if you are offloading the

data structure to your local disk, it means you need to like uh ideally, when you want to accelerate this trace ID lookup, then you need to sort all those data by the trace ID. While those spans are coming keep coming continuously, it means you have to constantly address the data allocated on your local disk. It's more like something like a rotation in a B+ tree, right? So,

it takes actual it will will generate actual overhead. And in retroactive sampling, our we our principle is like if we want to use buffer something in memory, we try to buffer less. So, in our case, we buffer only necessary info in memory for decision making, and later we only buffer those sample trace ID in the memory. And the second principle, if we want to use some use

the on-disk data structure, we try to eliminate all the random IO. So, in this case, we are using a FIFO queue, and we see great outcomes from the benchmark. But, I think both implementation are that like both of them are the prototype and not mature yet. So, we are pretty interested doing the comparison again maybe in the Okay. So, let's summarize what we discussed. There are a

few takeaways from here. The way to reduce resource usage is to actually do less work. If we can trap that information that we don't need about traces locally and not sending it all the time, we can actually reduce a lot resource of usage. And the second thing is that as we experiment with tail sampling stage flushed to the disk, uh where the tail sampling processor should do

a quick lookups and seeks over the disk. So, you need fast disks, and you have spend a lot of informa- spend a lot of computes on unmarshalling and marshalling those data structures. With sequential IO and FIFO queue in this implementation, we just either write data in the sequential way or consume data in the sequential way. No random seeks. We save we saving a lot of resources. And

uh what's next? Time for open sourcing. I think this is uh might be the the questions might be the questions most of you are interested in. So, we will be open sourcing. So, we are definitely interested in contributing it back to the upstream to the hotel. Um the prob- we we are trying to find a proper way to do it because it's a the tricky things is

that there is something we need the sampling server to run outside of the hotel collector, and we don't know if it's a good way to put everything to the to the retroactive sampling processor. And anyway, we are definitely interested in contributing. I I think there are a lot of experts here. Maybe we can have some suggestion or feedback on you. And the last slide, we have some

material that we would like recommend to you. Um the first one is an academic paper published on in 2023 NSDI, The Benefit of Hindsight. This is actually the inspiration of the whole speech of the whole idea. They use a different different implementation. They use a memory-based implementation, but there are a lot of details, numbers, and comparison out there. So, I would definitely recommend anyone interested in this

idea to take a look. And the second one is some prototype we built. We built the retroactive sampling in hotel collector, and we built the sampling server as well. So, you can find out find all of them uh in the GitHub. Yeah, I and I think um that's that's everything we have today. So, uh you can scan the QR code to provide some feedback if you want,

and thanks again for everyone who uh who stay here. >> [applause] >> Yes, we have 2 minutes for the Q&A. So, if you anyone want to ask questions, so there are mic on both side. Hi. Uh one question. When you had the trim Uh I'm here. Uh Other side. Oh, hi. Yeah. So, one question. When you had the method mentioned you implemented a sampling server, did you

also have to change any uh doing things on the protocol of OTLP, or you could just implement on top of what's already there? Okay, so the the question is about whether this sampling server is receiving data in OTLP, right? Yeah. Yeah, so uh of course not with this. So, first of all, this is a prototype, and second the second one is that our our goal is trying

to eliminate all the unnecessary traffics, and we don't believe hotel is designed for efficiency, right? So, we trying to like uh implement only necessary fields out there in we compress them as well. So, the answer is no. okay. Uh so, you you could not implement on top of OTLP. Sorry, I didn't hear. Did we put what? So, you could not implement on top of OTLP. You had

to use your own custom protocol to do this retroactive sampling. That's what you're saying. Um I think the only customized protocol we use is only between the local agents and the sampling server because we only need some key information out there. But, afterward, when the sampling result was returned back to the local agent, the local agent will still export everything in OTLP. So, what's the the only

customized thing the only customized protocol is the communication between agents and server. Hello. Um author Yeah. author of the uh GitHub issue you just quoted here. Uh I'm Carson. Um I I had the same question as the uh gentleman just now. So, I'll ask another one. Uh what's the benefit of using the uh first-in-first-out queue uh compared to uh database, and how how is that queue actually

implemented? do you send back unsampled decision as well as sampled decision or just uh the sampled decisions? Maybe I can leave the FIFO queue's question to you. It's a Yeah. So, um why it is FIFO queue and not a database? Because we don't need to do lookups. On the moment when we consume data from the FIFO queue, we already consulted the decision server and we know what

needs to be sampled or not. So, we know what trace IDs we need to look for in the queue. And everything that is not matching the trace ID can be even dropped and marshaled. That's why we don't do lookups. We just consume everything. We're checking the headers. If it's not trace ID is not in the header, dispose it. In that case, this is a pull-based from the

agent to the sampling server, a pull-based uh mechanism. Is that true? Um it could be either a pull-based or push-push-based implementation. It's actually from our perspective, it's just not not that matters. Like currently, because we trying to secure everything by the time intervals. For example, we are chopping something here for 30 seconds and we want to make sure the decision will return back in 50 seconds. So,

you can communicate in the way you want. Currently, we implemented in a push-based manners. It means the server will need to find out the find out the result back. But, we also know that there are a lot of agent that cannot communi- Sorry, there are a lot of servers, centralized server cannot communicate with your local agents. In this case, I think it should be in a in

a push in a pull-based manner. Thank you. >> But, yeah, the the overall idea is try trying to eliminate those network traffic. I don't think the pull-based or push-based is matter here. But, it can be implemented based on your real requirement, I think. Thank you. Uh this was a really nice presentation and a really good idea. And here comes the question. Uh do you think this solution

and idea could be extended so that they can handle not only the parent-child relationship, these regular traces, but also the casually related traces that are connected via span links? Um sorry, can you repeat that special kind of traces again? Maybe I'm not getting >> related. That are connected via span links. Ah, I see. So, you mean those traces are not connected by maybe parent span ID and

span ID? So, in in some other ways? Yes. >> Um do they have the same trace ID or they don't have trace ID at all? Uh they could have different trace ID. Not necessarily. >> Okay, so it's more like something from uh maybe eBPF, something like that. Trying to come >> it's actually part of the data model for for spans. The span link. Okay, so at least

in our prototype, we trying to group everything by trace ID. So, I'm not sure if you uh can do it in like a specific for your for the for the trace you mentioned before. But, I mean the general idea, if you can find a way to group them in one place, maybe it's possible. And I think overall, this idea of like try to send out less data,

it should work. And if you already have some way to sample your data to the current current current way to sample the data, then it might be still beneficial if you can apply our idea, try to send send out less data. I'm just not sure about how your data could be collect, buffer, and sample. Just not sure about it. I'm not familiar with this. Okay, so I

think time is up. So, thanks everyone again.