Enriching Telemetry Signals Through Lookups in the OTel Collector - João Duarte, Elastic

In this talk, Jan discusses the concept of enriching telemetry signals using the OpenTelemetry Collector. The presentation starts by defining enrichment as the process of adding context to telemetry signals, making them more valuable for observability and analysis. Jan categorizes different types of enrichment, explaining how they enhance signals by providing metadata or correlating data from various sources. He highlights the capabilities of the OpenTelemetry Collector, detailing how it supports data transformations and proposes improvements for handling lookup processes. The discussion includes real-world implications of this enrichment for organizations, including better detection of threats based on enriched signals. Jan also shares a proposal for a new lookup processor designed to extend the collector's functionality, emphasizing the collaborative nature of developing such features within the community.

Hello everyone. This is not yet the last talk of the day. So I hope you still have some energy in you. Uh we're here to talk to talk today about enriching telemetry signals using the open telemetry collector. So my name is Jan. I live in Portugal, Lisbon. I work at Elastic at the uh inest observability team. Uh I say that my main hobby is to collect hobbies

and then not spend enough time on them and therefore I'm bad at them. So if you want to talk about any of these hobbies or about the contents of this presentation, uh yeah, hit me up after the session. So before we talk about enrichment, we need to understand what enrichment is. So of course I went to the dictionary and enrichment is the process of making someone wealthy

or the action of improving the enhancing the quality or value of something. And I got to say, I'm sorry, but we're not going to necessarily make anyone rich in this presentation as we're going to be focusing mostly on the second definition of the action of improving or enhancing here the something is or are the signals that are traveling through your hotel And why do we need to

talk about this? So signals by themselves are already valuable pieces of information. They capture a moment in time, an occurrence, a measurement. However, often enough they lack enough context or even structure for you to use them properly. So here enhancing the value will come from adding more context to the signals. This context can be useful for your S sur practitioners for example to understand where the signal

is coming from from within your uh infrastructure. It can be useful for your organization to understand uh to which cost center this signal is affected to or even to see which uh if a certain IP that is hitting your endpoint is a malicious IP using a threat feed. So the idea again is to take this raw signal and adding more to it so that you can then

use it on alerts. You can use it on dashboards. You can even use it to do better model inference for what you need. Another thing that we need when we talk about enrichment is to understand what kinds of enrichment there are. And I came up with a set of categories of enrichment. You can also put it differently, but essentially I came up with these six types. And

the first one is the most simple one. You already have a piece of data and you're just either taking that piece of data and just splitting into more information or just taking two pieces of information putting that together. But essentially, you're working with what you already have. Type two is about metadata. So again you need to understand what is the context of or the metadata of that

signal. Is it running in my infrastructure? Is it running in a cloud environment etc. Type three is about you have a dictionary you have a piece of translation of A to B and you need to uh for example translate an ID to a name. So it is information that usually doesn't change a lot but you still need to use it for your en enrichment purposes. Type four

is kind of the same as type three if you think about it but the source that you are looking up to is something that you don't either control it's something that is changing a lot for example it's something that is too big for you to just scrape that and just put it into a file that you can do a lookup on. Type five is you have multiple

pieces of information in in multiple locations and you just correlate them, aggregate them and have the one aggregated event be more useful than the separated ones. And finally, type six is very important for the day of today where model inference is very common but also even 15 years ago when you needed to do uh sentiment analysis for customer feedback. So if you need to do correlation, if

you need to do if you need to do classification, anomaly detection, scoring, then that's type six. So what does the collector have? The open telemetry collector, what does it support? Should it have all of these types or not? I say that not necessarily. I think this is something that we as a community of users and maintainers of the project need to come together and decide which uh

types belong in the collector and figure out a way to support it within the nature of what the collector is. So of course type one is something that is very well supported in the collector. So if you need to do data transformation within a certain signal then you use the open telemetry transformation language you use the transform processor and you accomplish that type two uh of course

the collector works very well within your organization within your infrastructure so you're able to understand where it lives within your Kubernetes cluster where the which cloud provider that signal is coming through However, as you are going to type three, for example, if you need to have a CMDB, there was an example of that in a previous talk yesterday. Uh if you need to put some information that

you need to translate on in your uh collector, then right now what we see users do is just put all of those values or those key value translations just in line into your configuration. And that works well if you have 50, but if you have 50,000 or more then becomes troublesome. And arguably the other types are not really supported by the collector today. And again that's okay.

That's us for the community to determine if there is a use case, a common use case and if it should be supported within the collector. And that is what we've been seeing for the past two three years within the collector country. For example, there have been a certain set of issues asking for the need for these two types. And the the discussions and the proposals that were

brought up and discussed were either from uh the need to just have one of these sources and therefore we should have one particular processor to support lookup against that source or a more extreme way to have just a generic gRPC processor that would just execute a request to an endpoint and the implementation of that lookup of that enrichment would be done would be left as an exercise

for the user. So what I did was to take a lot of these discussions, a lot of these learnings, a lot of these proposals and try to come up with something that would be useful for everyone and try to satisfy the needs that were described in these issues. And with this uh I created a proposal back in August of last year for a lookup processor that would

carry a set of built-in sources but also would be extendable if you need to add a lookup source that is um vendor specific and you then can put it in your own distribution of the collector. The proposal was reasonably well accepted. There was a lot of interest both in GitHub issues in the GitHub but also during the collector seek meetings where this was presented. And this is

something that we as Elastic that I work for didn't want to fully own. I think this is important enough and widely impactful enough that we would want to have more voices and more people from other companies to do this work with us. And for this I wanted to give a big shout out to Sam from Graphfana. He's not here today, but he accepted to be a volunteer

sponsor for this proposal. And also to thank for all the work that he's been doing so far uh to to get this processor up. So with this, the proposal was accepted in December of last year. And that means that now we can start working on adding these capabilities to the collector through this processor. And essentially, this processor is kind of what it says that it does, right?

It does a lookup against a source, uses a key, gets data back from that source, and puts those values into the attributes of your The idea is that you can do that in a consistent manner because you're going to still use the processor, the lookup processor multiple times, but then you choose which source you want to use, but the rest of the experience is the same. The

key can be an OTTL expression. We're going to see more about that in the slides, but it can be either just a path for an attribute or can be a more complex OTTL expression. And what you get back from the source can be either one value and just put that value into the attribute or you have multiple values and then you decide where to put each value

in each attribute of the signal. And finally the lookup can be done either at the record level or at the or at the resource level depending on your needs. So this is what it looks like in your configuration if you would use it. So you specify the lookup processor. You specify which type of source you want to use here using the YAML source. This source obviously needs

a path. So you specify the path to this uh to this YAML file. Then you determine which lookups do you want to do to that file. Here we have one lookup and this look lookup is to the source IP log attribute. Uh and again it can be a more complex LTL expression if you need to mangle the data first before use it as a key. But otherwise

what then you pass is to determine where to put the data that the source gives you back. So in this case the source gives you back a risk score I want to put that into threat. Score and of course if the key is not found you need to have a default value and you also can specify it there. So again the idea is that this processor carries

a set of sources that are useful for most people that are typically vendor agnostic but again if you want to build your own you can do that uh with uh this processor and to do that bit of code just two slides I promise the first one is that it just tells you which kind of interfaces you need to implement. So you would have to implement a source

factory that tells you what the type of that is, what default configuration and how to create the source. This would be called by the processor. And you also implement a source interface that has obviously the lookup function. It has a type which is the name and then two functions start and shutdown to do resource management for that source if the source needs it. And finally the lookup

function. So it's very simple. You take the context, you take the key, you perform whatever lookup you need, and then you get back either again a value or set of values, a map. You say if it's found or not, and if there was an error. So, of course, this is more interesting to see actual live. So, I have a very simple demo here. It's just one collector

in my mini cube locally and also there's a telemetry gen just sending the same signal um every 5 seconds so that we can see the evolution of a raw signal and to an enriched uh signal with all of these four types of enrichments. So to start I'm going to have just tailing the the the output of this collector and you already there was a talk already about

how to debug the data that goes through your collector and for that you can use the exporter the debug exporter when you enable the and then you enable the verbosity detail to to see that. So here we have one log that has a body which is a JSON payload. Uh it has one resource attribute which is service name checkout and then you have an attribute that is

app server. And this is our raw signal. And now let's start to add more value to it. So for that we have a new processor the first one which is again type one which is just working with the data that we have. And here we're just going to parse uh the body and put that into the lock attributes of the signal. And by deploying this configuration, we're

going to have now more attributes that we can work with. So before we only had the body and the app attribute and now we have all of these status, route, source IP, request ID, and now we can start building more things on top of it. Right? So type two we're going to add yet another one another configuration. So after transform we can now also add Kubernetes attribute

processor. So I want to know where this signal went through in within my Kubernetes infrastructure. So again same thing I will just push that to the collector. And what what I'll get now is a a bunch of resource attributes that tell me, oh, this is the pod name. This name, it is the service name that I already have, but now I also know that it's the hotel

demo, the hotel collector, mini cube, etc. So, the value adds up. And now we go to type three. So, we want to use the processor, the Luca processor to see if there's uh if this IP is in a list of risk categories. So, there's a list that I have locally. It has a bunch of IPs. It has a risk c risk score, the category, and even other

fields that I might not need. So, going back to the I can say here's my YL file, my YL uh source. Here's the path to it. I want to do the lookup on the source IP. And from that lookup, I want to take the risk score. I want to take the category. And again, let's apply that and see the outcome. And the outcome is will be that

I have two more attributes here. And apparently for this the threat risk score is high and threat category is that it it's a trap. Let's find out more why. So let's go to type four. Let's actually resolve this IP and get back a host name. So for that again we resort to the Luca processor with a different type of source but the experience is the same. So

you're going to use a DNS source with a record type pointer. So I want to do the resolution against this DNS server that I have locally. Caching obviously is a important part important part because you're going to do lookups against something remote. So you want to cache the values, but it's enabled by default. You don't need to specify it. And then again perform the same lookup on

the same attribute and get back the client address. Here you don't need to specify anything else. You just need to tell where you want to place it because the source is only going to give me one value which is the host. And if I apply this here, we're going to get back a host. And I wonder if anyone knows what this host is. I think that given

that we are in the Netherlands, I thought it was a good idea to give a shout out to this like nugget of internet history. So, if you know what this is, shout it out. If not, I'll show you what it is. I heard it. Yeah, exactly. I will show you. And you know that it's old because we're going to use Telnet. So if I just take this,

it's of course the rendition in ASI of Star Wars Episode 4, A New Hope. And this is a a very interesting thing to have enabled if you're waiting for your Agentic uh command line tool to finish its PR or its code. Just have it here and it's going to yeah show you 40% of the film. It was never completed. It's an actually interesting thing. this works, right?

But where are we today? So the base implementation of this processor was merged to main in the end of February. Uh it only supports the YAML source. It only supports log signal for the purpose of making it easier for reviews and getting it merged. The second PR that is linked there uh carries the DNS source. Again, it only supports pointer queries right now. And there is a

configurable DNS server that you saw. If you don't specify anything, it will use the local uh resolver. Uh and of course it required the introduction of a caching feature. And this cache was not done just for the DNS processor. The idea is that any source that will require it can use it and the experience should be similar. Of course, it needs to be fast otherwise it will

impact your performance in your uh pipelines. So what's next? Once this is reviewed and merged, what should we do next? Again, it's also up to us, but there are things that seem to be rather uh convenient. for example an HTTP source enhancing the existing sources again the implementation the current implementation of the sources is being done very simply so we want to add for the DNS source

for example more types of queries A's and AAA's add multiple server support so if you have an internal DNS server and an external DNS server you want to hit both during the resolutions for YAML of course doing reloads during refresh similar to the resource detection processor obviously support the other signals. So traces, metrics, profiles, you need to support those as well. Of course, this is a processor

that you need to keep a little bit of an eye on. So it it does it can impact performance. So you need to see if the cache rates, heat rates, and miss rates are within what you expect. So we do want to add enough telemetry to it so that you can keep it under eye and configure it accordingly. And of course benchmarking optimization although right now even

the two sources that are being added already carry a benchmark suite on top of also the core processor also has a benchmark suite so that you can understand what the base performance is the base overhead and then what the source will add on top of and of course this is only useful if it's used. So we do want to first move it to alpha and then eventually

carry it into the open telemetry uh contrib uh distribution so that more people can start using it and see its value and add more to it. So this is the time to give us feedback uh go to the repository uh go to the issues and if there's things that you want things that you disagree with or agree with uh any feedback is welcome and and that's pretty

much it. Right. So this was a very interesting process in uh going through what the the community has established as a way to add functionality to the collector contrip. So creating an uh a proposal going to the sig meetings to talk about it discussing it on GitHub. It is something that I think really validates the processes of the community and how well the community is working. So

it's something that I do recommend as well. So go out and go contribute and be part of the community. join the community and if anything else come say hi after the the presentation happy to help to talk about anything and yeah I think that that is it. Thank you very much. Uh have a great day. >> We have a lot of time for questions. So go for

it. Hi. >> Hey thank you so much. We are doing that in our site. So in in thousands we are actually checking in radius if um the field is there like we have test name we we get add test h sorry we have test ID we add test name so it's totally the same so thank you so much for that initiative is really needed cool >> the

question that they have is I saw that you are adding different source like the DNS the jl most likely you will add some database later on have you thought about having an extension that each one can build their own source for their own use case instead of moving to the on trip that extension for them to use. >> Yeah. So the the way that it's built right

now, you can still do that. The the way that it was the the interfaces are built is that you don't need necessarily to put everything uh into the lookup processor. You can still have in your in your own repository the sources and then when you instantiate the the processor, you can pass additional ones and that will be code that lives inside of your collector. I had like

a sample of that but we struggled a little bit that that there is some discussion in the issue about extensions versus not extensions. Um but we did find a way that it still allows you to have thing sources that are external to the collector but still expand on it and then you can put that into your own into your own distribution of the collector. So if that

does work that that is supported that's the intent as well of the of the processor. >> Awesome. Thank you so much. Um thanks. So my question is about uh availability and latency. So if you have an external source that's not available or that's slow, it could clog your entire telemetry data if I'm wrong. So how do you handle that? >> Yeah, I mean the the good and

the bad answer is that you're always limited to some timeouts. Like you can set a timeout to 0.0 0001 seconds to make sure that it never goes above that. And you're always going to hit at least once time that if your time even if your source is down and it's going to still hit that threshold once after it's gone once then it's still going to fail for

a long time. For example, for the DNS um for the DNS source, if the key is not found, it's still cached as a negative cache. So you don't lose that lookup again for a while. So, you're still going to have to pause or to to stop doing the lookups for a while and then you're going to start again. And then you need to specify how do you

want to mark that signal? Just not enrich it at all. And then you're going to have to decide what to do with that signal if it's not enriched. And it also depends on your logic. If the enrichment is absolutely crucial for you, then you just prefer to wait, for example, versus just letting it go without being No more questions. All right. Thank you very much.