How Manual OTel Instrumentation Saves More Than Just Money - Juliano Costa & Yuri Oliveira

This talk focuses on the importance of observability in applications and the role of instrumentation, both auto and manual. The speakers, Juliano Costa and Yuri, discuss how auto instrumentation can illuminate a previously opaque application, akin to turning on lights in a dark forest. They delve into the trade-offs between auto instrumentation, which may generate excessive spans and data, and manual instrumentation, which allows for more targeted and meaningful insights. They present research on the CPU and memory impacts of different instrumentation strategies and share best practices for managing observability, including the use of OpenTelemetry to filter unnecessary data and ensure that the collected information is valuable to the business. By the end of the session, attendees are encouraged to define their observability goals and tailor their instrumentation strategies to optimize performance and reduce resource consumption.

Uh hello everyone. I I not even have clothes to be on this stage. Uh thanks for joining. Before we start, I'd like to propose an exercise to everyone. Um imagine we we are in a dark forest like really really dark. We can barely see anything. You can hear some O's in the at the back. uh some crackling from some leaves from the wind or I don't know

animals walking around you don't know and but there is no waterway you need to cross this forest like again really dark maybe a bit scary and you look at the side and there is a light switch you think like what the heck a light switch in the middle of the forest. Well, it's dark. It's scary. I'll turn it on. And then two flood lights, one on each

of the each side of the the forest. And suddenly you can see everything. You can cross the forest as it was daylight. But now all the animals are disturbed. And do you even need all that light? How much energy are you consuming on that? In your observability journey, your application is this dark forest. It's a black black box. You can cannot see anything on it. And you

hear about all the instrumentation and you give a try. Once you deploy on your application, you can see everything. And I would I would even say that maybe you can see too much. Do you need all the spans that you are getting? Do you need every single hop that your application is doing within this each service within the span? Do you need all the attributes that you

are cons that you are um ingesting runtime description really? Well, today we're going to learn how auto instrumentation can save more than just money and the pros and cons of auto and manual instrumentation. Hopefully by the end of the talk you'll also get some sanity back. I'm Juliano Costa. I'm a developer advocate at Data Dog and a CNCF ambassador and I'm here today with my friend Yuri.

Thank you Juliano. Yeah, I'm Yuri. I'm a co-founder at Olig Garden and I hope you enjoy. >> Cool. So again one more thing that we want to say before we start actually the the talk. Um the community the the open telemetry community which we are both part of uh work really hard on trying to impact the least as possible on your application. We are not here to

bash on anything of uh uh any work that the community is doing. We are just here to shed some light pun intended into the vantage and advantage of auto and manual instrumentation. Having said that, let's discuss a bit of what motivated us to get started on this talk. So you and I we were talking and uh one s one single request through a HTTP service that calls

a database and this example is uh express in JavaScript. we have eight spans and if we really look at it maybe we not six out of them. So if you think about you can control the incoming request or you control the the server side of your application and the database is the database query is where you would improve the query and the the performance of the query.

But all the the things that happen in between maybe there is just too much noise. When you do auto instrumentation when when you do auto instrumentation you get that out of the box. When you do manual instrumentation you could ah sorry. So this is one service. We know that applications usually run on multiple services and you have one service calling the database for the example. But in

reality we do 10 service, we do 100 services, 200 services. And when we see that we have too much data, maybe we think okay yeah let's go back and then we are backing 10 services which is not ideal. Ideally we want to actually have visibility across the whole application from end to end. So what we can do? Well, if we do a more intentional instrumentation, we could

focus on the things that we want to have from the traces or the expense that we actually have on our traces. And in manual instrumentation, we could say, hey, I would like to have just the incoming request and database query. We can argue and discuss more about what is important and what not, but uh bear with us for a second. Um let's take a look at a

bit of the the research that we have done. So we have a K6 lo generator in front of our application sending requests to this sums app that queries u a postgress to get songs details we have a spring java application and a express javascript application. We use uh uninstrumented as a baseline. Then we have a manual instrument manually instrumented service and an auto instrumented service. So here

just to like discuss what is obvious and hopefully everyone cares about that just when we think about the container image size um and on the the difference between the uninstrumented when we add manual instrumentation we we need to add dependencies. So we are adding increasing the size of our uh containers. And again when we do auto instrumentation we also add extra uh well you you do not

need to add manual instrumentation and auto instrumentation but just to make the calculation easier. we just uh show like gradually here how to what is what is increasing but let's say in express we are having uh 75 MB extra just to add the auto instrument we can argue again this is not much but maybe in a scale out scenario where you have a lot of services that

are auto instrumented maybe that may be something cool uh so let's investigate a bit on the cost of lightning the forest list. Uh here we have a dashboard or uh a chart where we have on the top the auto instrumented service and at the bottom the manually instrumented service and this is the average CPU usage on express in three different runs. Those are the numbers that we

that we got as average and that represents an increase of approximately 40%. this for CPU. When we talk about memory, the pattern kind of repeats again. The top is the auto instrumented, the bottom manually instrumented. And in three different runs, we have an increase of more or less 34%. What about Java? Well, in Java for the CPU, we have kind of the same again where the increase

is on 37%. And when we talk about memory, that was one thing that actually surprised us a bit. >> Yeah. >> Um they are pretty good on memory. The the the lines are pretty close. And when we take a look at the numbers, we have actually just 12% increase. So this is pretty good, pretty performant so to say. Uh but again there are if you have been

to cubecon or fos them or other conferences there are talks about uh benchmarks on auto instrumentation and uh the impact of auto instrumentation. So take a look at those talks as well. Do not take our uh word as a final thing. Do your tests. We are sharing the the the rep the repository at the end. and test your application, test the impact of it. But Yudi and

I, we were talking and say, "Hey, what else can we talk about?" So for that, we did a a couple of tests running for 15 minutes with uh 10,000 requests per second. And then we send the same number of requests for both uh auto and manually instrumented. And then when we look when we took a look at the at the storage we saw that the express auto

as it is more verbose and produce more spans per trace even though the number of traces is exactly the same. We have a 44 gig difference and this is for 15 minutes one service calling a database. If you take a look at your application, I'm pretty sure you don't have one service calling one database. Uh when we did the same experiment in again, same 15 minutes, we

got an increase of 42 GB. And if you are on the observability field already, you may just raise your hand and ask, "Hey Juliano, what about samply?" And to answer that Yi what about simply? >> Yeah thank you thank you Juliano for uh clarifying all these aspects uh to us. Yeah, when this question of what about simpling uh comes uh we are in front of a decision

on one hand uh we have 100 service let's say this is the average of services that uh most of company compan companies has and u producing 1% of sample rate versus on the other hand uh the same 100 service manually instrumented but with a 80% % of simple rate. So how to decide that because the storage consumption will be the same and then when you are in

front of this decision you're going to be switching like this me and to decide hey uh I have 1% on one hand 80% on one hand same storage but on one hand you have a lot of CPU and memory being consumed and in another hand reduced uh resource consumption. But if you ask okay if I decided to move forward on the manual instrumentation how uh can I

check what I will miss? So let's take a look. Let's go back on the first slide. On the first slide, we we had the opportunity to see a trace that calls a a song title artist uh endpoint with seven different spans. And if we move uh down, we are going to check that we have med spans. We have uh a couple of spans like the request handler

uh repeating the same information like songs, title and artist with the same information that we have on the trace information. Uh also pool connect and PG connect information uh DNS lookup GCP uh connect information. All of these uh does bring value to your business helps you u analyzing or finding the root cause for any issue. So let's compare uh with the You see a single span with

the SQL query statement and and this SQL query we truly believe that will help you out um having more precise data to see what actually does bring value to your business. Of course, you do not need to strip all the spins out of the trace. This is not a recipe. It's just a provocation because we know what you are doing. It's not about it's not about setting

the auto instrumentation and forget. We encourage you after uh you uh set your auto instrumentation in your NodeJS application, your Java application validating your data. So that's going to validate the data. So if we move forward here we have like process uh common arcs with the whole path of the node execution. If we check for example in the case of Java we have the whole runtime description

Eclipse adoption openk etc. All this information bring value to your business is a meaningful instrumentation. It's a meaningful data I don't believe so. So, but if you still needed those information uh in the auto instrumentation way that is an environment variable that you can set the resource detectors in this environment variable uh here we brought this host and I brought so uh different aspects about this uh

process uh command arts uh and information about how can you set host and name and in our case here we are calling a request for the uh song and the artist. But if this the same request contains your social number ID most likely if you have that and using out instrumentation this social number ID will be transmitted to your APM back end. So if we check it

out, we have in the URL this information about poly nirvana and um also we capture the request body which also can contains uh some sensitive information. You see we have like so uh songs poly nana in our request uh which is not uh anonymized with not sorry not anonymized but sanitized with the title and artist. So we encourage you join like the songs uh title and artist

to avoid uh a high cardality issue. >> And how can I do that? most likely uh have you heard about this open telemetry collector? If you have the open telemetry collector, you can take advantage of some uh processors. So here we have a processor uh to filter out the information that you don't need. So is is simply you have uh to pass a information that you don't

that that matches uh with the string that you want to uh filter out and the processor will remove uh the information that you don't need it. the same transform, right? To help you out fixing the possible PII leakage, you uh pass a regular expression uh to to match with the the social number ID. Let's say that and all of these will will be removed. The collector is

great is again powerful to remove or filter out information and you should definitely take advantage of that. But why do you need to first produce the data in your application and second process the data removing the data if there is one thing that I want that you can take home is if you don't need the data don't produce it because otherwise you're going to consume a lot

of CPU memory storage and so on and so forth. With that, I'd like to share a couple of recommendations. First, do you define your observability goals before instrumenting and as soon as you have uh defined the uh instrument your application with purpose. So I believe if you've instrumented your application as soon as you uh do that uh please validate your data again don't simply file and forget

go to your back end validate the data see if the data is there as expected with the proper information with the proper uh uh rate and consider by the end uh using simpling based on your traffic volume. All of this will have uh will bring you more precise which leads to a reduced MTX act X because it could represent a reduced time to detect reduced time to

recovery. And if you have a reduction on your MTGX, you're going to have more efficient telemetry pipelines. And if you have all of this precise data, reduced MTGX, more efficient telemetry pipelines, your team will be will thank you with enhanced mental health. And per consequence, you're going to have happy customers saving you more than just money. Thank you. Awesome. Um, thank you very much. Uh, please rate

the session. Um, we have two mics here. So, uh, if you have any question, feel free to to ask. >> Well, thank you very much for your presentation. was very well done and touches exactly on the topics we discussed tomorrow yesterday. So >> awesome. >> Awesome. >> Uh my question is as a platform team a lot of time you are trying to reduce the involvement of application

developers in this kind of instrumentation. So how do we achieve both best of both worlds give them the least involvement into the instrumentation but also provide this granularity that is good enough. Awesome. Great question. And um we do understand that for platform engineers and uh the majority of teams auto instrumentation is the easiest one. what you could do is suppress instrumentations. So uh on the for JavaScript

on the node SDK whenever configuring you can suppress for Java you can do that via environment variables and not have not even have the service instrumented and or deploy the agent but not have everything on by default or again following our our approach you set it on by default check the data see what you get and say ah okay so this this and this is not important

so you go back to the application, specify which which instrumentations you don't want to use, suppress them, and then your application will be automatically instrumented, but producing data that is more meaningful for you. Of course, all the business metrics that are related to your business, you won't get uh that would require manual instrumentation, but at least you won't get a lot of garbage uh in the back

end. Tricky oh sorry tricky question. Um manual instrumentation usually suffer from being inconsistent and not follow semantic conventions. How do we make them better? >> You want to take it? I I can answer. >> Yeah, I I can Yeah, you you can answer and then I can add my two cents on this. >> Uh thanks for the question. Yeah. Uh so the open telemetry has under the

project what we call semantic conventions and to ensure semantic conventions you could use we uh sub project called open telemetry weaver that you can add to your uh to your CI/CD for instance and do live checks on the telemetry that your application is producing even before it reaches production. So weaver would be my answer. Uh, of course you can do a lot of remapping and uh transforming

on the collector but I would go for weaver because then the telemetry that you produce is more uh aligned with the platform team. So the platform team can take advantage of weaver as well that will be super helpful. If you don't know the project take a look. Yeah, we for sure uh will be yeah also my my answer uh but also to contribute as about the slide

I just showed uh with the validation of the data also the semantic conventions are important but the data that you are transmitting over uh it's also important what value again uh bring to your >> Thank you. Great talk by the way. >> Thank you. Thank you. one one have you considered a mix of sampling and removing in terms of let's say your example is dropping TCP connect

on the database but what if it's starting to get too long for whatever reason maybe some kind of threshold after which it keeps being reported anyway >> well uh in that case uh what I would suggest ashes if this uh this kind of information is important to your business keep it and if you are facing any I would say if you're in the middle of an outage

and you are uh analyzing a root cause and you can enable uh some debug mode and so on to see how it reflects otherwise and then after that disable it again uh because uh yeah you're going to have a more precise information in your debug process, you know, but don't uh don't do that in a I'd say don't enable that uh for the full uh information because

you're going to have basically 80% or 90% of garbage in your observability >> I I can add to that. So basically the sky sky is the limit here. So you can pipe collectors and let's say or even in one single collector configuration you could have a tail sampling and this tail sampling would drop would do the decision of a sampling and then pass to the second pipeline.

So let's say traces for instance that was the focus of the talk. Um you could have a tail sampling based on the dur the latency. So like let's say TCP is taking too long and then if it's taking too long keep this trace then you pass to a second pipeline that will send everything to to your back end and if not then you drop uh then you

pass to a different pipeline that would drop those spans and then you have spans without the TCP and DNS connect. So you can do literally everything you want with the collector. Um yeah so >> thank you. Hi. So, um, with the manual and automatic part, uh, what I saw on the slides was, uh, if you do automatic, you get a bunch of spans. Not all of them

are important, but some of them, for example, there was a span for a DNS lookup. That sometimes can be useful, right? And when you do manual that's uh I don't you can't add that yourself because it's a third party library or something within it. So what would you suggest then like combine it somehow that you do manual for your business logic and let all the third party

libraries have automatic but then you also get a lot of garbage spans. What's your recommendation on it? Yeah, my my recommendation is uh out instrumentation is powerful and we do recommend out instrumentation if you are starting uh to yeah to do observability in your company or if you are a small team you know because again it's it's like I I mentioned if you produce the data it

consumes uh CPU memory consume resource and you are going to process the data in order to drop them out and consumes uh CPU and memory to do that. But if you uh evolved uh in your team to have a more precise uh observability, I encourage you having manual instrumentation in setting the attributes that is uh uh so that is important that is meaningful to your >> I

can add a little bit on on that. So there is another approach called instrumentation libraries that you can specify which libraries on your application you want to instrument. So you do the manual instrumentation and let's say TCP is meaningful to you. you can add the instrumentation uh TCP I don't know the name uh but yeah uh that that would take care of auto instrumenting the this TCP

connect and or DNS lookup if you go for both automatically for you on your application then you would have like both worlds together >> okay >> and also that would give you more power over the instrumentation because you specify what you want to instrument on >> But it's important to yeah to keep validating if this information is is still relevant uh to your business. You >> my

question was on that part of like combining because um you can also like filter the data as you showed. So if you do suppose automatic instrumentation for DNS lookups or something then you could have the collector filter out based on the value for example and then it only comes up when it's like above a threshold. So yeah and open telemetry uh collector has the concepts of pipelines

and then uh by default you can configure a pipeline that will uh filter them out and tomorrow if you need it is just uh removing the processor of the pipeline and then they will uh flow flow in and tomorrow or the day after you can put >> Okay. Thank you. >> Thank you very much. So the link to the to the repository with all the tests are

in this QR. Any feedback is welcome. Uh appreciate everyone. Thank you.