Tutorial: Full-Stack Observability on a Budget: A Guide to Strategic Sampling and Da... Pavol Loffay

This talk focuses on achieving full stack observability on a budget in Kubernetes using open telemetry tools. The speaker discusses setting up a Kubernetes cluster and deploying an observability backend with Jaeger and Prometheus, along with the OpenTelemetry operator and collector. Key concepts such as sampling strategies—head-based and tail-based sampling—are explored in detail, highlighting their impact on resource consumption and telemetry data management. The speaker also emphasizes the importance of telemetry data profiling to identify unnecessary data that can be eliminated to optimize costs. Throughout the tutorial, the application architecture is explained, and practical examples are provided, including how to gather and filter telemetry data effectively. The session concludes with tips for managing log duplication and enhancing observability while keeping resource usage in check.

Today tutorial is actually sort of a continuation from the past tutorials. We did one tutorial for like overall observability on Kubernetes. Then we had one tutorial for uh using open telemetry for metrics on Kubernetes. Uh and the last previous one was about tracing how you can trace your workloads and infrastructure on Kubernetes. And today we will look at the full stack observability on budget. So it's going

to be roughly maybe like 1 hour, 1 hour 15 minutes. We will see how fast it goes. Hopefully we won't have any technical issues because last time we couldn't download some images. So let's see how it goes. Um everything will be hosted in the repository. You don't have to follow me on your laptop, but you can do that and we will have time for questions and maybe

will troubleshoot if something doesn't work. But you should be able to replicate everything at home. Uh if you will be interested in the topic, you can scan this QR code or go to my is it actually readable the URL or is it too small? Maybe I I don't know but you can scan the QR code and go to the repository. Uh and so what is on the

agenda? Uh there will be some introduction uh and setup. So we will need a Kubernetes cluster. There are instructions how you can spin up kind on your laptop. Then you will need to install the observability back end with Jagger and Prometheus and open telemetry operator with collector. It takes couple of minutes. So if you would like to follow me please do it right now. Then we will

do an overview of sampling. We will explain what the sampling is. What are the sampling? What are the two sampling strategies and then we will deploy our uh demo application. Uh I will explain what the app is doing. And then we'll be using the demo application or the telemetry data from the demo application to to improve the the amount of telemetry data we collect from that app.

Uh first of all we will look at the data profiling which is um really the fundamental step to understand from where the cost is actually coming. So we will understand which workload is emitting how much metrics traces and logs. uh and then we will look at the data and try to remove the the malicious telemetry data that we actually don't need. Then we will look at the

headbased sampling in more details. Uh we will configure different samplers. Uh and then we will continue with the tailbase sampling. Uh we'll configure it in the open telemetry collector. And then as the last topic is about logs dduplication. So let's get started uh with the setup. Uh as I mentioned we need a Kubernetes cluster. So you can install the kind cluster with this instructions. I have already

everything deployed on my laptop. So I will not be running these commands. Is any is anyone following me actually on the laptop? All right, quite a lot of people I would say. I didn't expect that but I think that's cool. Uh and if you will have any questions during tutorial tutorial please there is a microphone here in the middle. Uh you can go there and you can

like have the discussion. Uh so please raise your hand if you are done with the setup. I see. Okay, few people. So, let's wait a little bit. Um, so yeah, kind create uh then make sure it's up and running. Then we need to deploy the search manager because it's used by the open telemetry operator to get the certificates for the web hook. Uh you need to wait

until the web hooks are deployed. That's important because if you don't wait, your deployment of the operator will fail. uh then deploy the open telemetry and then deploy the observability back end. So this is the architecture what we will have today. Now we are just deploying uh the left part the the back end. So the back end is Jagger with inmemory store and Prometheus and open telemetry

collector. So we will be sending the data via OTLP to the collector and then the collector will send data to to Jagger and to Prometheus as well via OTLP. We will be configuring the collector in the tutorial application name space. So this one in the back end it's kind of static. We will not touch it. We'll just deploy it And then please pour forward the Jagger query

because we will be using the Jagger UI and as well Prometheus and then please clone the repository because we will be modifying some manifest files. So you need to have it locally to to be able to do that. And if you are done, please raise your hands. Or maybe better question who is still working on the setup. So let's wait maybe a couple minutes. Yeah the the

application will be again like microser application written in different languages. We will instrument the application with the open telemetry auto instrumentation libraries. Uh for that we will use the open telemetry operator with the instrumentation CR. Um like all the telemetry data will be reported via OTLP to the collector and to another collector and then again via OTLP to Jagger and Prometheus. Maybe this is better up. Then

the manifest files that we will be applying are in the in the app directory. There is as well source code for the applications if you would like to have a look. Is anyone still working? No. Okay. So, let's continue. Awesome. Um so let's start with a sampling overview. So sampling is a technique how we actually decide which data we want to store. It's not necessary to store

all the data uh because hopefully most of the telemetry data we never touch right. So we need to find a technique how we going to uh store only the the important data and the important data in telemetry is usually when there is an error or where there is a latency right something was too slow we want to have traces logs and metrics for it. Why we as

a summary why we do sampling is a cost management. It's very it's very costly to store all the data all the logs and all the traces. Um the observability vendors they they charge a lot for the uh for storing all the all the all the telemetry. Then what people don't think about a lot is resource reduction. If we generate a lot of the data it will cost

CPU memory and networking in our environment as well. And then if we have all the data in our back end and there is an incident it will be difficult to troubleshoot because we need to find the right data right. So all these three aspects uh we need to consider if you want to apply sampling or are the benefits actually for the sampling. So the first um kind

of methodology in sampling is so-called headbased sampling. In the headbased sampling, we make a decision in the start of the request. So when a request arrives to our first service or load balancer or ingress, the telemetry API or the tracer will make a decision. I'm going to keep this trace or I'm going to discard it. And this decision will not change uh during the kind of lifetime

of that request. It's going to be propagated to all downstream services and they will respect it. The this solution is very simple to implement because only first service makes a decision uh and makes decision randomly based on some probability. Uh so it as well doesn't consume resources. The disadvantage is we make a decision and we can't change it. Right? The request will fail and we decided not

to sample. We lose the data. Um so we might lose important aspects of the system behavior. How is the sampling config or headbased sampling configured in open telemetry is done in the SDKs. There are a couple of tracers or couple of samplers. There is always on sampler that will sample all the requests. Then you can sample nothing and then there are the probability sampler and trace ID

ratiobased sampler. So these two are very similar. The trace ID ratio is deprecated. We should not be using it. Uh it's replaced by the probability sampler which is the recommended approach right now. The way you configure it is you actually configure the parent base sampler uh with the probability sampler. I have an example here. That's because we need to respect the decision that has been made by

the upstream service, right? So we need to look at the the previous sampling decision and then if the decision is there, we respect it. If we have a root span, so we are the first service making the decision, we call the Now there are some explanation that uh explains the difference between these two. I don't think it's that important. uh but essentially the probabilistic sampler is its

implementation is consistent across the SDKs and it will propagate the threshold header to all downstream services. The threshold header is important because then you can uh in the back end in your telemetry provider you can calculate what is the total probability of a given of a given behavior. So for instance uh if we got 500 sample requests and we got five errors and we know the probability

was 10% we know that roughly 1% of the request in total failed. So that's is the headbased sampling. Now let's look at briefly tellbased sampling which is maybe more complicated. So in this case all services they send all data to the back end. Uh so there is no no decision made in the service. They send all the data to the collector. Collector will group this data by

trace ID. It will wait for some time. Um it depends on your environment how much it should wait we will talk about it later and then when it thinks it has the full trace when the timer is is done it will make a decision so it has all the spans for a given request so it can make very good decision right if there is an error it

will decide I want to keep this trace if there is a long latency it can as well decide okay I want to keep it is um in this case the collector deployment will be more complicated uh and it will consume more memory than the the hub base sampling. The pro as I as I mentioned the decision will be made on the traced outcome are usually the the

error the latency or your specific endpoints in your application. So you have very critical endpoint for which you need to have all the telemetry data. You can configure it as well or you have a customer for which you would want to keep all the data. Uh but there is as well a lot of cons. It will consume more memory. We will look at some calculations how much

roughly the memory is needed for the It's more complex to implement especially if you needs to have a scalable deployment. uh but we will see that in the in the next section as well. And what many people don't realize is it adds latency for when you can actually query the the trace from the back end. With headbased sampling and especially with the back end like Jerger, you

can immediately see the data in Jagger. With the tailbase sampling, you will be able to see the trace when the timer is done. So maybe after 30 seconds, maybe after 1 minute, maybe after two minute, it depends on your environment. All right, let's deploy the the demo Uh you have the instructions here. So please go to the instructions and start creating the manifests. Um but essentially what

we will have here are four services is the front end that is written in Node.js, JS the back end written in Python back end two in Java and back end three in Golang it's a simple dice game uh roll a dice so these players they roll a dice they and then the number is shown in the in the front end and the front end then decides like

which player won the game so I'm going to deploy it as Oh, and as you can see there is as well in it container which is the the instrumentation and we're going to deploy as well load balancer. to create some traffic in the So the apps they they are the services they are not instrumented by default. There's just the source code. uh the instrumentation is injected because

we configured the instrumentation CR and then each uh each pod has annotation for instance this one to inject Java auto instrumentation or NodeJS instrumentation or Python instrumentation. uh this annotation the open telemetry operator will see this annotation and will inject the open telemetry auto instrumentation to the pot that is starting. Um it it will inject their init container which copies the instrumentation libraries into shared volume that

is mounted as well to the application container and configures the runtime to use it. So in this case it will continue configure the node options to pick up the auto instrumentation and configure other open telemetry SDK environment variables. So for instance the exporter endpoints to export data to our collector in the Is anyone having issues? Right, looks good. So, let's go to the profiling. Um, by telemetry

profiling, I mean that we will look at the telemetry data. We will understand how much each workload is sending telemetry data. And as a next step, we will look at the data and understand if there is any malicious I call it I don't know profiling but you might probably find different terms on the internet. So by default the open telemetry collector exposes some set of metrics. Uh

you can configure it in the se service telemetry under the metrics and in this case we're going to send these metrics into the collector that runs in the in the backend name space. So we get them in the Prometheus essentially. I'm going to open the first link. It will show us number of total number of logs, metrics and traces that the collector is receiving. And yeah, we

see it here. Number of traces is much higher than the logs. And there are as well some But this doesn't tell us anything more, right? We know just that the collector is sending some data. We don't know which workload is sending and how much. Um the there is as well more fine grained metric. for instance in this case we are not doing any label replacement. So we

get a metric pair. We get as well like how many spans are accepted by the collector but we have two collectors and as well we have OTLP HTTP receiver and OTLP gRPC receiver. So we see a metric for for all of these um receivers but it's the same metric. It's just not aggregated. Uh but I want I wanted to show you here is as well refused spans.

Uh this is a metric that will indicate that your collector is not able to ingest all the data. Uh so is refusing the telemetry. In our case yeah it's it's zero. So we are good. our collector is keeping up with all the traffic. okay. So let's look at the profiling. For profiling we are going to use the count connector. You can apply the this manifest. It's going

to reconfigure the collector in the tutorial name space. So what does the count connector? It's able to count telemetry data. It's able to count metric like data points, logs and spans. Uh and then we can take this counter and export it again to our Prometheus. The connector is like one of the types that the open telemetry collector supports. you configure it as an exporter in one pipeline

and then as a receiver in the in the metric pipeline because it's it exports metric data. So what we have here is we're going to just count the metrics uh count these spans logs and data points. So each metric and we're going to split this by service name and Kuber's name space. So we'll be able to see which workloads is sending how much data. So there are

some graphs here what we see here is that the okay the front end is saying send sending way more than the the free back seconds. If I scroll down, the same thing for logs. No, the logs the back end free is sending way more logs than the other backends. And for metrics the back end free is sending more more metric data. Yeah, we need to wait some

time to get to get the some stability in the matrix. Yeah. So these are the metrics. Um and in this case backend one is sending more metrics but it's not by much right there is the the difference is not that radical as for spans for spans the front end is sending maybe a couple times more than the the free back ends. So this is something that you

know we should probably look at uh why is that happening? Okay. So what I'm going to do I'm going to go to Jagger. We're going to have a look at some traces. Uh let's for instance select this one. So this is the trace that we get. Um and we can we see that the front end or span there is way more spans from the front end than

the other back ends. Um so maybe this is because how the instrumentation is set up. Right. It's creating way too way too and then we see couple spans from each back end as well. Okay, it's it's a fact, right? That's how the instrumentation works. Um so what we're going to do next or what we should look for is like the noisy services like if one service is

emitting way more telemetry data than the others or we see telemetry coming from unexpected name spaces. So I didn't open this link but in this link we see uh how much spans are coming from each namespace. We have only single namespace. So in our case it's not it's not that important but if you have a large system we could see you know the name spaces that shouldn't

be sending any telemetry data So this is what we did with the the span um with a count connector and now we're going to look at the data and try to identify the data that we shouldn't be that the instrumentation shouldn't be sending. Uh for that we're going to again use the and for spend data we're going to identify spans that have more than 10 attributes. Um

so attributes are great because they hold the the meaningful information. Right? If I go here uh and I select the span, these are the attributes but we see there are so many of them uh and they actually tell us what the operation was about. So in this case it was HTTP request. We see the the status code, the URL, the endpoint, even the the IP address of

the client and the host. Uh and then for the process we again see we we are able to identify which pot from which name space uh the data is coming from but we see as well here stuff like you know deployment UID do we need this probably we don't right so we can we can identify it and remove it so let's take a look um so in

this case we are just looking at the number of total number of attributes. Uh then we are going to have a metric for dropped attributes. So this is something that is set by the open telemetry SDK. In the SDK if the tracer or someone wants to set lot of the attributes on the span, it will be dropped. Uh it's a limit that you can configure on the

SDK, but it will be dropped and it will be locked in the span and then you can identify it on the Why would you like to do that? You then you should look at the instrumentation and try to remove the number of attributes there because you are making so many allocations in your process that you you just waste. Then we have the health spans. Um so we

are deploying the services on the on Kubernetes. So we have the readiness pros with a health and end endpoint. Uh and they as well produce trace data which is not needed. We never use this to troubleshoot our applications. So we need to identify them and then we will get rid of them. I can show you the the the trace from the health endpoints. And this one is

pretty complicated because it's not a single span. It's actually multiple spans from the health endpoint. So there's a lot of traffic for for no reason. So what we have next is for logs we're going to look at the logs or log bodies that have more than thousand bytes. Again we will identify it by service name and name space. We're going to look at the debug logs. You

know, a developer um forgot to remove them or forgot to reconfigure the the final log level. For metrics, we're going to look at the attribute counts. We're going to do actually two two metrics. If a metric has more than five and more than 10 both for the attributes on the metric and attributes on the resource. Um I wanted to use this to identify high cardality metrics. Um

but this does this doesn't solve it fully because we can have low number of attributes but there might be a lot of attribute values for high cardality metrics. uh and there is no easy way to to do it or there is no way to do it with the count connector. So this will not prevent from the high Okay, I'll apply actually yeah we have already deployed that

and we're going to look at the metrics. There is a single link for all of them. And now we see the spans with more than 10 attributes. Again, we see the front end deployment more spend data, but that's normal because um it doesn't matter. It was sending the more spans before as well. There is no spans that have dropped attributes. There is log with large bodies. It

comes from the deployment free. Then we see metrics with more than five attributes from all services. I think yeah back and one to the front end resource attributes from all services. Again more with more than 10 resource and we see debug logs from the backend free service. So backend free service is really badly instrumented for logs because it has debug logs and large uh large log lines

and then we see the health spans again and way more from the front end. And that's because if I go to to Jagger, I see the front end is emitting is emitting five spans. But I think that doesn't matter because the the count connector is looking at the spans. So maybe the API is being called just more often. Okay. So in the next step, we're going to

optimize this and we're going to get rid of this data that we actually don't So yeah, I was talking about the high cardality metrics. The count connector can help to identify potentially high cardality metrics, but we will not identify the ones that have the high cardality attribute values. Uh now we're going to use the filter processor to filter the data. We're going to filter the debug logs.

Um, so the best way to actually optimize is to go to the application source code and figure out why the app is emitting any debug logs, right? We should that's that's the best way. But in our case, we're not going to do that. We're going to drop the data on the collector. Um, so we're going to configure the filter processor. It's going to drop log records with

the severity. We're going to as well drop logs with We use the OTTL statement for it. And then we're going to drop the health check spans. So if there is a span with the HTTP root attribute equals to health, it's going to be dropped. It's pretty simple. So, uh I'm going to apply this. Actually, we need to change this um in So, I'm going to enable the

the filtering for traces and filtering Okay, collector is up. And now we're going to look at those metrics again and see how it has changed. So we see the health spans they decrease. So it seems like it's it's done with the health spans. So if I search for the health endpoint from the front end, I don't see anything here. So it seems like we were successful. But

that's the trick. Uh there is a problem with the health spans. Um the way we configure it, let me go back to readme. This is going to drop span that has the health endpoint. But the health endpoints usually or not usually but they can produce more spans in that trace like we saw from the front end right there were five spans for each health endpoint invocation. So

what we did we removed the the root span but those other four spans they still stay there. Uh and right now it's even more problematic because we are not able to find them because if I search here for the health uh I don't see them anymore. I need to search for maybe this one. But this one is present in all other uh request invocations. But maybe this

one. You see it's it's marked as incomplete. It doesn't have other services. So let's take a look. Yeah. So this is the trace from from the health endpoint that doesn't have the root span. Uh and it's super tricky to find it. All right. Um it's incomplete trace. We can't identify it with the health endpoint. Um so be very careful when you filter the the health endpoints. Uh

I think there's another service here maybe I think the back end too it's Java and I think Java as well creates couple spans for the health endpoint yeah again I search for the health endpoint I see only something from 3 minutes ago right so it's old uh this is not this is before we didn't have any filtering so maybe I can search for all maybe C500. Yeah.

And search for incomplete traces in Jagger. I can search for incomplete traces which is very cool. I'm not sure if you can do it in other back ends but this is one of the techniques you can use in Jagger to identify um there to identify the missing parent or Yeah. So you see this is the the span from the RO controller uh and the method name is

health, right? So we need to get rid of this one as So the question is how do we do this properly? Um proper way to do this is to actually disable sampling for the health endpoints. But for our deployment that's not really that's not really supported uh because we are using headbased sampling with the instrumentation CR and there is only limited configuration. If we were using the

uh the declarative configuration who of you is does know the declarative configuration for SDKs? uh it's essentially like a config file for open telemetry SDK and it's way more um it gives lot of features and here you can actually configure a sampler for specific APIs in your app. So what I done here is I disabled the sampling for the health endpoint and in this case the the

tracer will create the span but it will mark it as not sampled so it won't be reported. So all these five spans that are childs will not get to the back end. That's really the only way how we can solve this problem. Um another way and we will look at it in the tailbase sampling we can uh in tailbase sampling we will get health spense as well

in the collector and there it's much easier to get rid of them. But with head headbased sampling can get you can get rid of them only with the the SDK config. So so what are the best practices? Uh we have done the configuration on the collector but it's much better to actually change the source code or configuration in your app uh and get rid of this data

before it's actually created or before it's actually exported to the collector. You will save the resources on the application. You And be careful with the health check endpoints because you can't really filter them on the collector. All right, let's take a look at more details at the headbased sampling. So how does it work? A request arrives uh at the entry point. So it can be a first

service in your environment, a load balancer or ingress. Uh there is a sampler API that makes a decision. Let's say it will keep only 10% of all the requests or 50%. Then this decision is encoded into header called trace parent and is sent All downstream services read the header and they respect the sampling decision and based on the decision all services either report data or they discard

it. So how it's configured? Uh you can configure it via environment variables. Uh this is what we are doing uh but yeah I will talk about it uh in a second in the SDK via environment variables or you can directly hardcode what sampler you want and with the declarative config. This is the the cool new feature in open telemetry SDKs. I think they just announced the stable

version right now at CubeCon this week. um you can configure sampling for specific endpoints in the application. So what we are doing we are creating instrumentation CR on Kubernetes uh where we configure the type and argument and then the open telemetry operator uses the environment variables to configure the SDK in the process. So essentially it's translating the instrumentation CR into environment So now let's change the sampling

rate to 50% in our deployment. So this is the instrumentation CR. I'm going to change the from sampling all the requests to 50%. And I'm going to apply it. So I did two things. I actually applied new instrumentation CR and then I restarted all the services. This is because the instrumentation is configured only when the pod is starting. So if I change the instrumentation CR and I

don't restart the pots, nothing happens because they will keep using the old configuration. I have to restart them to pick up the new uh the new config. So what we see here again the number of traces will drop number of spans as well will drop from It takes some time. It takes couple minutes to to see the result. This is the the span count from the from

the connector. But we see it's already falling. But it takes time. Maybe like five Okay, so we essentially with this technique we can drop number of traces by 50%. But we need to restart all the deployments. Another headbased sampler that is in open telemetry is called Jagger remote the sampler is uh implemented on the open telemetry collector. So there is um the component for it. Uh the

way it works the collector serves configuration for the SDKs. Uh so when the SD an SDK is kind of periodically checking with the collector for the config uh and based on the config it will reconfigure the the sampling in that SDK. Why this is a good approach is because you can control central you have a centraliz centralized configuration and you can configure sampling per I'll show you

the config. I think it's here. So on Kubernetes, you need to configure a separate config map. Um but this is how it looks like. You define your default strategy by default. Let's say you want to keep everything and then you can have per service configurations. Um so in this case we could have like 50% from the front for the front end and then for the health endpoint

essentially zero to disable sampling of I think it's really cool component uh but it doesn't work for all the languages. It's fully supported in Java and in in Go. But in our case, we are using as well Python and Node.js, so we can't really use it. But if you are using just Java, I think it's a great solution for headbased The headbased sampling uh you can there

is as well a component in the collector for headbased sampling. It's called probabilistic sampler processor and it works for traces and logs. Um so the way it works uh at the high level your SDKs will do some sampling or usually they will do the some sampling and then at the collector you can do additional sampling and drop more data. it's again a good strategy when you can't

really change any configuration in your services. you can't restart them and you need to quickly adapt the sampling because there is just too much traffic going to your back end or you think that you don't need the the data. So the the processor has um sampling percentage configuration and then couple of modes. There are three modes. there is the the hash seed mode proportional and equalizing. So

I think the equalizing I will start with the equalizing mode. Equalizing uh means that you don't want to send more than 10% of all the signals to the back end. So you essentially unify at maximum 10% uh of all the u traces and locks. Proportional means that you want to reduce sampling by proportional amount. So in this case by 25% you will reduce uh already sampled telemetry

data and h seed is similar to proportional. So again it will reduce it proportionally but you can configure it to use different fields to make the uh the sampling decision. Uh so if you have a random attribute uh some custom attribute you can use that as a sort of uh of probability. So yeah, there is some example you can take a look. Uh and yeah, the equalizing

mode it uses the uh the threshold header. So you need to be using the probabilistic sampler for for it to to work properly. In our example, we're going to decrease traces and logs uh by 50%. We want to maximally return 50%. So, we're going to use the equalizing mode. configure the collector. So, we just enable the processor in two pipelines for traces and logs. and apply the

manifest. Yeah. And we see drop from the previous from the previous setup. When we change the headbase sampling, I think the best way for us would be actually to roll back the instrumentation to 100% and then apply the the sampling on the It takes some time to to propagate but eventually we should see the drop in the number of of spans and locks. So as I mentioned,

we can use it when we can't really modify the application or the SDK config. It's good as a safety net to ensure that the maximum um ingestion rates to our telemetry back end doesn't increase. So let's see. Yeah, but we so we were sampling 100% then we changed to 50% in the headbase sampling and then we change it again to 100% but we apply down sampling to

50%. So it's at the end it will be very similar than 50% in the headbase sampling. So this one will not change much. Okay, let's continue then with the tail sampling. So as I mentioned the tailbase sampling it's more complicated. The all services they will sample everything. So they will send all the traces to the collector. kind of group spans by trace ID and will wait predefined

time which is by default 30 seconds and then it will evaluate the entire trace against some set of conditions and there can be condition to drop that trace or keep So this is the basic configuration. We configure the the wait time. In our case, it's going to be just 10 seconds because we don't have long running process. But uh if you have some messaging application, you should

probably uh understand how long does it take to kind of put the message into the queue and read it. uh and based on that adjust the the wait time because you want to sample only when you have all these spans for the given transaction. So if you have let's say your process that you know reads some really old data you probably can't really use tailbase sampling because

you don't want to like keep trace in memory for for a very long time and then you can as well configure how like the maximum number of traces you want to keep in the collector. Then there are the policies that will decide the on the sampling. Um there is you want to always sample. Then there is a probabilistic sampling. You can sample by the status code latency

based on the attributes in the data based on the number of spans based on the OTTL condition and you can as well construct more complex tracing strategies with a composite uh policy. So what we're going to do, we're going to have a policy that will keep all the errors. So if the status code is error, we're going to keep that trace. So if any span in that

trace will have error, the whole trace will be kept, not just the root span or the uh the leaf span like any span. And for latency, we're going to keep all the we're going to keep the trace if it has more than uh two seconds. And we going to drop the health the the spans from the health endpoint, not spans, the full trace. So in this case,

it's going to work uh as we want. And rest we're going to sample at 10%. So how it works? If policy decides to sample, it will be sampled. If it decides to drop, it will be dropped. Um what is the trade-off for this approach is memory because all these traces that will need to be buffered for the given time. Uh latency uh in a meaning that we

will not see the trace before it has been sent from the collector. So if our wait time is couple minutes, we will not see anything in the back end until the timer And yeah, let's apply it to the And we can copy this one. Yeah. And for this we need to really make sure that the instrumentation or yeah the sampling in the instrumentation is sampling everything uh

which is by default. And then we apply the tail sampling. Let's see. The tel sampling processor it's um it's more complex uh and therefore it emits some telemetry data itself. So we can understand its behavior. But we need to wait some time until these metrics are populated. But essentially we will see the number of traces that were sampled or dropped or not sampled. And we should as

well the same metric for every policy that we defined in the tail sampling. Yeah, it should be enabled. Okay, we get the metrics. So these are the global metrics the aggregated ones. So we see that the majority of traces are not sampled because we are sampling only at 10% rate. We see that yeah some are sampled and some are dropped. So these ones are the health and

point traces. We can look more closely per policy. So yeah, this one is the uh not sampled spans from the health endpoint policy. those these are the traces that don't don't come from the health endpoint. Um and they were not sampled by this particle policy. So they are marked as Yeah. And these ones are sampled. These um the rest policy that we defined um that are being

sampled. Health point dropped and the errors are at zero because there are no errors at the moment. So this way you can understand like how much how well each policy is performing, what it's doing, if it's dropping data, if it's sampling or if it's not You should as well look at the collector memory because we uh we said that the telbase sampling is more complicated and keeps

lot of data in memory. So let's take a look at that. And in our case, this top metric is actually the back the collector that runs in the backend name and the green one is the collector that in the tutorial name space. Right? So the backend name space uh consumes more memory uh and that's because in our case our load generator creates only 50 requests um per

second which doesn't generate uh lot of traces. But let's take a look at the calculation. uh we should understand how much we will need data for the tailbase sampling. So the way you can calculate it is to understand how much traces you will keep in memory. Uh which is essentially incoming traces per second times decision weight time. Uh and then the final memory that you will need

is this the traces in memory times average spend size per trace. Uh average number of spans per trace times average spans in bytes. The average spend size in bytes there is no metric on the collector for it. Uh use the file exporter to export you sample trace to to a file and you can estimate what is your your size and average spans per trace we were looking

at Jagger and in our case it's 15 or or 14 I think it's 15 there is still missing spense It's it's 17 even and incoming span rate you you can uh get it from the internal collector telemetry. This is the telemetry that every collector has. So you can take a look uh and get it from there. So let's take a look what it means for us. uh

in our case we have roughly up to 50 requests per second from the load generator. Then I was calculating with 50 spans per trace. It's it's 17 doesn't matter that much. Uh so we get roughly 550 spans a second. We wait 10 seconds which means that we need roughly 15 megabytes of memory. uh and the average span size is 2 kilobytes. Now if we increase the the

traffic to,000 spans a second, we will need roughly 600 megabytes. So it really depends on your scale, but you should be able to calculate this uh before and make a decision if you can use the tailbase sampling and how beefy the collector will need to be. The issue with the tailbase sampling is uh the scalability because we need to group all the spans from the same trace

uh in a single collector. So if we scale the collector, we need to have some logic that will send the same spans from the same trace into the same collector. The way it's done here, it's through the load balancing exporter. Uh essentially sends the spans from the same trace to the same collector. Uh and then each collector runs its own tail sampling configuration which should be really

the same configuration, right? Um good way to as well decrease your telemetry cost is to use the spanmetrics connector. Um so all spans they they have the the start time and duration. Um so we can extract metric data from Uh I was showing that in the Jagger pro I don't have it configured here. Uh but essentially you can get for each endpoint uh metric for the number

of requests errors and the the latency histogram. Um so you can drop more spans uh more traces and still kind of keep the same visibility with uh it's very good to use this connector with tailbase sampling because uh if you want to get accurate metrics you need to have all you need to apply 100% sampling in your SDKs because otherwise if you apply sampling in the SDKs

and then you calculate the metrics you will get skewed metrics right because you are calculating it from something that has been already sampled. Yeah. So this is the configuration. You can define your metric dimensions uh and the buckets. Okay. Uh in the last chapter I would like to look at the uh the locks. Uh there is as well log dduplication processor on the collector which groups logs

uh by their body content or attributes. uh and then if there are duplicates for a given time uh it will report only one and emit a metric that it was uh or attribute that it was uh duplicated which is again I think a good way to um to save some cost from the from the logs. There are as well bunch of metrics on the collector that help

you to understand if there were duplicates or not. Yeah, in our example we don't have anything but you can change the the app to create some duplicates. Apply the processor. I think it's predefined. Uh and look at the metrics. There is uh there are metrics here. This is just the rate. This is the uh the default metric that we get from the collector. Okay, this was everything

that I prepared for today. Um, wonder if you have any questions. We are as well working on one project called open telemetry agentic workflow. Uh we want to build agentic skills uh for different use cases and I think the cost optimization is a good candidate. uh and you would essentially use this skill. Uh the skill would reconfigure the collector for the most common use cases or help

you to understand which is like what malicious data you have on the collector and based on that uh apply the the right configuration to automate all this process because it takes a lot of time to to figure out what is actually malicious in your data and uh and apply the correct configuration. We can automate lot a lot of this. Okay. Thank you very much for your attention.