Day-2 Reality Check: Taming Wasteful Telemetry - Juraci Paixão Kröhling & Elena Kovalenko

In this session, Edina, a principal engineer from Delivery Hero, and Jasi Pashan Kurving, a software engineer, address the challenges associated with wasteful telemetry in software development. They discuss how the convenience of auto instrumentation at the initial stages can lead to an overwhelming amount of unnecessary data, creating noise rather than useful signals. The speakers explain various pitfalls such as poor data governance and the mixing of personal identifiable information (PII) with observability data. They also share real-world examples of telemetry mismanagement and suggest practical solutions like enhancing telemetry governance, utilizing tools such as Weaver, and creating wrappers around open telemetry SDKs to improve data quality. By focusing on responsible telemetry practices, the speakers aim to help developers produce valuable observability data without unnecessary clutter.

Hello everyone. That's long. Um hello everyone and welcome to the very last session of CubeCon. I'm very glad and very proud that you are here uh that you survived uh 2 and a half days or almost 3 days. It's been a long week. Um and I I promise that we will keep it uh interesting and relatively short I suppose. Uh but we have time. Yeah. Let's see.

Let's see how it goes. So welcome today to reality tech uh taming wasteful telemetry. Yeah. Hi everyone. Uh my name is Edina. I'm principal engineer from delivery hero and uh I have been taming wasteful telemetry for the past uh 5 years at least. >> Yeah. And my name is Jasi Pashan Kurving. I'm a software engineer and co-founder at creating that planetary for so many years now. >>

Shame. Shame. >> Yeah. Sorry. >> So uh let me guide you through the agenda that we have prepared for you today. First we are going to look at the social and environmental factors that make us create bad telemetry at day zero at day one. And also we're going to have a look at the exact examples of this bad telemetry. And then in the very last step we're

going to provide a set of simple solutions that everyone can implement and make your telemetry a little bit better. Let's go. >> Right. So let's get started with the roots of all evil. So when I develop my first applications or when I start developing applications even today I I try to make it work first right so I do the my software I get it working I deploy

that software and then I hope that everything works eventually things don't actually work so I need to understand what's going on uh and the way that we can do things nowadays at least I mean back then I would open the application add some logging to that and watch what's going on but nowadays what we would do is whenever we need to observe Oberve what our application is

doing. We would um perhaps get some vendor to help us there. Do a PC and perhaps start doing some instrumentation in our applications. Um so u imagine a situation that you are at a loud party or actually don't imagine anything. just remember how you were sitting at CubeCon this time and were listening to a keynote in this large auditorium and then suddenly um you start hearing the

thoughts of everybody else around you and in the beginning it is exciting it is interesting you're overwhelmed and you're thinking oh maybe I can listen to something interesting from a person I consider interesting but then there is so much noise and there are so many other thoughts and then an interesting person suddenly starts thinking about their cat or their shopping list and then it leads to you

being stressed being under pressure and processing all of this unnecessary information and this is not a pleasant place to be. So the same is happening when you're dropping an auto instrumentation in the docker file of your application and then suddenly a lot of information starts floating out of it and ends up in your telemetry back end. But then when you're starting uh to uh or when you

try to make sense of it, you frequently see a lot of noise and uh very little signal. So this is uh the problem of convenience, right? It's convenient to use auto instrumentation and this is something that many people defer to uh uh at day one or uh day zero but then afterwards it creates a problem on its own. Let's consider a different um problem. uh some organizations

like uh my organization are having a long history of uh producing observability but they're also having a special let's say um organizational structure um if you see so to u give a metaphor for this consider this map of Germany both of us are based in Germany so Germany has a federative um governance structure and some uh topics or Some decisions are being made centrally, some decisions are

being made locally uh on the uh Bundesl frequently in the organizations. So for instance uh organizations like delivery hero that emerged to acquisitions and previously many subsidiaries used to be their own businesses. The decisions about things like observability are frequently being made uh locally and this creates a problem of telemetry governance where uh every part of the company uh writes telemetry that does not um have a

common schema and this uh creates then problems uh for correlation of this telemetry or for analysis of this telemetry when it ends in the telemetry back end. Yeah, we end up in a situation where we end up collecting a lot of data and this is where I said at the beginning I is creating that telemetry. I'm I'm kind of responsible for that. We have this philosophy a

few years ago as well within open tracing even. So we would say let's instrument all the things. Let's collect all the things. Let's have telemetry about other things. Now that led us into the situation where we now have way more telemetry than we actually need. A lot of that telemetry is not very useful and uh but when we try to fix that problem or whenever we we

think about removing the that the user instrumentation that user telemetry that we have we end up thinking h what if I need this one metric in five years time for that specific edge case now if you've been thinking about that case when you look at your telemetry congratulations you became a telemetry hoarder Oh yes, imagine all of this telemetry that we haven't even looked at. Yeah, disaster.

So with all of these prerequisites of solving observability problem at day zero and day one, let's have a look at the concrete examples of the issues that might be introduced because of this. So um another u metaphor that might be very close to people who uh also um come from Germany uh is uh the separation of waste. So and the problem that is associated with this is

the um uh like if you if you categorize your data then PI data is not the same as application telemetry but sometimes we mix both because observability vendor frequently offer them in the same product. So for the users, these two are not necessarily two different things. They become the same. And this results sometimes in PI data or data that identifies a person uh landing in observability back

ends where it doesn't belong to like on this picture the nuclear waste landing in the organic uh waste bin. Completely unacceptable. Your German neighbors will kill you if you would do this. Uh so yeah and uh an example that I want to provide from my personal experience recently we had an accident uh or an incident where um so over 700 of the phone numbers of uh customers

of my company were leaked into the um telemetry back end and uh this happened uh because um uh the application that was producing this telemetry didn't have any uh PI masking on the application level. There was also nothing masking PI data in the telemetry ingestion pipeline. And uh what happened is that the application started ingesting duplicate records into a database that it was using. And then uh

the library that was interacting with database was logging the error um of the for the uniqueness constraint and then this error message uh contained the uh phone numbers. >> Of course, it >> of course why not, right? And then uh of course uh um afterwards this information has been removed from the obserability vendor back in Bastil. It created a security incident that could have been avoided. >>

But Ellen, I'm sure nobody had this problem before, right? >> Of course, nobody had it's unique. It's everything that you hear is unique. So another very unique problem for our organization are the traces of doom. And this is my personal favorite. So consider this trace uh that uh at the moment when it has been um inspected was running for 7 days and 12 hours. So um >>

can you repeat that how long is it >> 7 days and 12 hours? >> Yes. >> Can you run for such a long time? I can't uh I mean only ultra runners can do it but go going away from uh the running metaphor just imagine and I kind of uh anonymized uh information identifying the producer of this telemetry in my organization but it is something that is

called create blah blah blah app. So an app whatever that is is being created for more than one week. And then if you look at the spans uh that constitute this trace there is no useful information at all. So you don't understand what is going on there. What is happening? You can see that there is one get request that uh takes more than one day to uh

complete. Okay. But kind of yeah what is it? But what am I looking at? I don't know. I just know that it will be running forever and for some reason it's a it's captured in one trace. Uh so these type of problems originate frequently from the outer instrumentation and uh I've been seeing a lot of those when for instance auto instrumentation is being uh added to service

that interacts with some kind of uh popsup system. So then this kind of interactions with the cues are being captured as in form of spans and this thing can run forever. But yeah, this is also very interesting and uh clearly auto instrumentation generated. >> I don't know. I see some people smiling here and I think we are taking pictures and thinking we have the same there. So

I don't know if it's that. >> No, no, it's a unique problem as I mentioned. It's >> this one is very unique to you. I bet >> yeah in this manifestation probably yes but generally um I believe it isn't. So um just imagine that you come to a library and then you're asking for a book about Galactus. um for the fans of Kroan videos and uh uh

yeah the librarian gets overwhelmed because every bookshelf is being managed by a different person and sometimes Galactus is a service name, sometimes it's a service, sometimes it's an app, sometimes it's application and the poor librarian has to spend so many brain cycles just figuring out what is it and where to retrieve it from. So the same is happening when you have inconsistent attribute usage uh in your

company and asking simple questions uh about your telemetry becomes so hard just because you cannot attribute this data appropriately. And here since we were with the service name I really this is kind of a selection of my uh personal favorites uh um yeah from uh that I've extracted from our telemetry back end. Uh the first one is uh so what is not a service name? This is

kind of a slide to illustrate this. So a call to a database is not a service name. I I suppose the first one is is kind of auto generated thing that is uh about a database call but it's not a service name. Then IP address is probably also not a service name. Then net http is not a service name. AWS S3 or any other managed uh service

name is not a service name. Then probably if you have an app my app and it it interacts with the MongoDB then my app MongoDB is not a service name uh and the service name would still be my app. then something that contains uh higherity information about uh pods or any other Kubernetes resources is probably also not a service name and Java SDK is not a service

name and yeah this is also great production region is also not a service name the service names are so bad I can even show it on something else >> yeah I can you keep that slide I mean I I like one thing about this one is the IP uh did you realize that IP is a broadcast IP like it's not even the local IP like it doesn't

make any sense right um and those are service names that were actual like they found those service names there it's >> yeah they're real and they are with us at the moment of speaking >> um we also found a lot of useless attributes I'm not at the liver here I mean they don't have this problem at all but we found useless attributes also in telemetry that we've

been receiving and uh my favorite one is the pods back. So remember when I mentioned that I used to create a lot of bad telemetry um and one of the things is um reminiscent of those days. So we uh in in in those good old days we uh decided it would be a good idea to just get all of the resource all of the labels and annotations

from the Kubernetes pods as resource attributes for the resources that are running on there. Right? So why not store the whole podspec as a resource attribute because a specific tool is storing the podspec as a annotation. So let's get the annotations and store in the telemetry as well. So every single resource attribute from that from that application contains the full podspec and this is not the full

pod spec for the current version. It is the previous version. Right? So very useful. Um and also another example that I like well command args is of course it should be there. Right? telemetry hoarders. It has to be there and I might need that eventually sometime in five years time for some strange use case. But the my the other one is how many of those other attributes

here they're related to a specific vendor. And when I showed to them to uh to you know the owners of this telemetry they said oh yeah but we don't use this vendor for a few years now. So it's been there for a couple of years and they don't even use that vendor anymore. Now this is um another example of uh you know useless logs. In that case

we have logs that are not very consistent in the naming. So we have warns and warnings. Uh and we also see that quite a lot of the logs that are duplicated. So this is yes useless. Uh we also see a lot of useless traces right. So what are useless traces to me? um I don't need to store every single instance of um a health check trace, right?

Perhaps I need a couple of those from time to time just to see, you know, is it still working? Uh but not all of those, not every 5 seconds a new trace for that or even I don't need every single ping that I do to radius to become a trace or I don't need a spend for that or Kafka client chatter. And this results in what we're

calling the holy trinity of bat telemetry. Uh and the holy trinity of bat telemetry is composed on on three angles. So the first one is volume. So there's a lot of um a lot of bad telemetry in terms of volume there. So a lot of us use logs, traces and attributes and metrics. Um we have a lack of governance or a problem with governance. So we saw

the problems with the service names but it's not only that. how many uh other services or how many other attributes are being misnamed or how many how many times are we naming things uh using different attributes for the same things and how many attributes are we not using and that could be useful for debugging information. So there's a new semantic convention for instance for service criticality. So

perhaps we should be enforcing them um um from now on. And the third uh angle is nuclear waste. Uh it is the best monitoring in terms of sensitive data. So sensitive data does not belong to telemetry, right? So PII does not belong to telemetry. So that's the third angle of that telemetry. >> So how to get out of this mess, you would ask and we don't have

answers to all of this, but we have some answers that might be beneficial for many many um telemetry producers. Let's start with the um most obvious place where many central observability teams would be looking for um would be looking at to introduce some controls to the telemetry and it's um on the collector. So when talking about the PI data in uh telemetry then it's uh pretty trivial

to use the transform processor to uh manipulate the PR data and to mask it. Um and it is as easy to use as to create a um very good regular expression. >> So not easy at all. >> The second one is we can just delete um matchings. So I've shown you an example of a company that was shipping telemetry named after a back end that they didn't

use for a couple of years. What we can do if we own the telemetry pipeline. What we can do if we own the the collector that is being the proxy between the applications and the back end is we can delete the matching keys there by using the resource uh the transform processor. So the transform processor can look into those uh we can tell the transform processor delete

everything that contains that specific vendor in the name. I don't need that anymore. Uh so I don't need to go all of the engineering effort of going back to all of the sources and change that. Of course, it's better if I fix at the source, but I can also um fix the problem at the pipeline level. Uh the other one that we can do is I I've

shown the example of the logs, the duplicate logs. I I don't need a new log every five millconds telling me that you went to Reddis. I don't need that. So, what I can do is I can use the log to the processor in the collector that will aggregate all of those logs or group all of those logs over a specific interval uh period interval. So like every

10 seconds uh I group all of the logs that are similar based on this on the set of conditions that we have here and I store as a new log attribute a the count of how many events I've seen that match that condition. So instead of emitting 1,000 logs in 10 seconds I emit one log and then I I I add a counter I've seen 1,000 such

logs in 10 seconds. Uh the only way that we can we can tackle that the other way we can control the attributes or or gain control of the attributes that we are producing is by using weaver. Who knows what wever is? No maintainers. No maintainers of open telemetry please. Yeah. Okay. So a few users uh I've been uh talking about weaver for you know we had a

conversation about weaver the other day doing the uh uh project updates for open telemetry. There were a couple of other talks about weaver. So you should watch the recording or go back in time if you can. Uh but the way that we works is you define a schema for a manifest for your registry your registry. So you define what are my dependencies, what is my registry name

and so on. Uh and then you define the telemetry how your telemetry look like. So uh this is a metric and and the name of the metric is this one here and it is a counter instrument. Uh the unit is this one. It is table. Here is a description and here's a set of the attributes and this one here is required and for that attribute I require

this this and that um values or or subattributes for that one. So which members of my attribute should it be? So you define the schema for your telemetry and then you can use weaver tools to match the actual telemetry that you are generating uh and generate violations so that you can fix the problems at the source. So you can enforce the rules for the service names for

instance that we've seen earlier. Uh they should match a specific standard or they should not be broadcast IP addresses for instance. And now we come to my absolute favorite part because um frequently problem of wasteful telemetry is being pushed by organizations to somebody else because it's very hard to solve and uh more often than not is being pushed to observability vendors and uh customers are asking vendors

hey can we just send whatever we want to you and then you store it at a reasonable cost and then create some aentic incident management uh automation to make sense of it or sometimes we say hey can we push it out the ability team and make it introduce controls on the collector level but uh still it doesn't educate or it doesn't help the application owners and the

telemetry producers to realize what is it actually that they're producing and whether there is value in it or not. So that's why uh the most from my perspective the most um effective way to combat wasteful telemetry is to do it uh on the application side. Uh and um yeah the first solution that has already been in place at many companies for many years is to create um

wrappers around open telemetry SDKs. This has been done before. We had tools like we were to introduce the uh governance in your telemetry to enforce usage or automate usage of certain attributes and also to uh provide an opinionated way to use open telemetry SDKs and also control the SDK version that is being used in big companies and this has been working quite well. But it also has

a set of issues. For instance, somebody has to maintain these SDKs. You need to ensure that uh once you have a new SDK version also all of the users of your SDK have updated to this version and also um still it's possible to create a lot of wasteful telemetry with the SDKs right because uh people get creative when it comes to you know misusing tools and uh

that's why um it's worth to look into um agentic I hate this word I'm sorry it kind of uh Aentic solutions um to um control the shape of the telemetry and control its volume and they uh can be used to uh for instance uh make pull request reviews to access the cardality of the metrics that are being used and to give feedback. also it is um they

can be used to um uh instrument applications and do it in an opinated opinionated way that is uh provided in the context. So and uh I think you can build yourself uh these tools. You can also use some tools that are out there already. And yeah, this is the just an example of uh a pull request like how a pull request from such a tool can look

like like so in this case the uh tool has been evaluating the instrumentation of the application and then providing a feedback to the user with the suggest suggested changes. And this approach might be quite efficient and uh the benefit of it but it's kind of outer instrumentation in a way but at the same time it's actually so it's automation of the uh manual instrumentation and because it's

on the code level then there is a high chance that the owners of this code will be looking at it and uh will um educate themselves about how they actually can >> and it's not a human telling them to do >> Exactly. and it's a absolutely neutral party that is telling me them to do things so it's more credible. Um let's wrap up. Let's summarize what you

have just been talking um to you about for the past couple of minutes. So first we were exploring the reasons why we create wasteful telemetry and the examples of wasteful telemetry and we figured that uh one of the biggest reason reasons is the commodity of using auto instrumentation at day zero and day one where you just start or maybe when uh a team uh that is using

it doesn't know anything about telemetry and doesn't care. So this is uh a commodity but a commodity that also um needs to be used responsibly in order to produce um reasonable signals. Then also um absence of governance uh and telemetry hoarding are frequently social um factors that are responsible for our huge telemetry volumes and um yeah that observability vendor backend shouldn't be used as a back end

for PI data that there are better solutions for this PI is not um and let's not treat it this way and then when we were talking about solutions we have been talking about ways to um mask PI telemetry on the collector level. It's also possible on the application level. We have been talking about using out instrumentation responsibly and also using tools like weaver SDK rappers and a

gentic instrumentation to make your telemetry better at the source. Now before we um um close this talk, I want to remember very old or like not very old but like feels very old to me metaphor about uh that there is no um there is no cloud, there is just somebody else's computer. So in case of basically telemetry it's exactly the same. There is no vendor, there is

somebody else running a giant database for you. And many users because there are so many levels of abstractions, they are forgetting about it. And they are treating uh observability back end as a deaf now. And this is I think if we will find a way to solve this problem, we will also have a better and uh better quality >> Thank you very much for staying TILL THE

END. AND I THINK WE HAVE A FEW MINUTES FOR QUESTIONS. If you have any you have to start the question with uh I am a telemetry recorder. >> No questions. >> Okay. Uh no there are questions. Thank >> Um thank you very much for the presentation. Um I have uh one question here from the perspective of how to manage um the or better manage the retentions as

as you collect this with the collector. Um I didn't see mention much on the on the opamp perspective how to use that but I think I think it's implicit right or or or no uh where's your perspective on using opamp for the way to um indirectly or directly um um better manage retention policies across the board. Yeah, >> I think one way that I can think of

is propagating like whenever you see that telemetry, you generate a new rule and you propagate the new rule to the collector uh to the set of collectors that you have. But I think it is too theoretical for me right now. Uh I don't know any open source open servers that you could use that could do that for >> But in theory it would be that would be

the way. >> We we had a talk on Monday about this specifically from Google. >> Cool. So yeah, I should watch that. Yeah. need to go back in time. >> Yeah, exactly. Um the other question I had is on the agentic part. Um how do you see because of course the concern is really on the noneterministic part for that. Um how do you see what should be

the grounding for for those agents here because at least in my perspective should have kind of some analytical storage to provide the grounding so the agents can do the right thing and not just kind of >> shoot in every direction. But yeah. >> Do you want to take this one? >> Yeah. Maybe you start. >> Okay. No. So, um, LLMs, the B LMS, the offtheshelf LMS are

quite good at at at doing code in general, but you do need your layers of knowledge on top of that. So, we often see LLMs with knowledge that is outdated in terms of open telemetry. So, you need first to get a layer of knowledge that brings up information to the models so that the telemetry that you are going to generate out of that is then up to

date. you probably don't want to replace bad instrumentation with outdated uh semantic conventions and it's I don't know so you need uh some layer on top of >> because my concern is mostly as like all the all the resource layers all the things becomes too much for the context and then it depend on on the site on the amount of telemetry you have and then they can

start doing wrong decisions just because it gets us out of the context right so yeah >> yeah I think um Maria I think Marina mentioned uh earlier today to me that Hi Maria. Um that uh the new project explorer might make things easier for that. So uh agents go and look at that information there and uh it can become this hotel layer on top of the offtheshelf.

>> Awesome. Thanks a lot. >> Of course. >> And those slides folks they were all designed by Alena. So of course not mine. So >> a round of applause to her. So you know generation All right. Um, now I think we are at time. So, thank you very much again for coming and for staying here until the very very last session of CubeCon. Thank you very much.

>> Thank you.