OpenTelemetry Collector SIG: Project Updates - Jade G, Dmitrii A, Alex B, Evan B & Antoine T

This talk provides an update on the OpenTelemetry Collector project, now in its seventh year. The speakers, who are contributors and maintainers, delve into the organization of the Collector and its various components. They discuss the importance of stability within the project, outlining their ongoing efforts to reach version 1.0, which aims to standardize the OTLP exporter and receiver while stabilizing core libraries. The presentation highlights various components, including the Prometheus receiver and the Kubernetes attributes processor, while also addressing performance improvements and configuration enhancements. Furthermore, the talk covers new features, such as a profiling receiver and custom builds of the Collector, emphasizing the push towards streamlined distributions and refined telemetry.

All right, thanks for I guess thanks for staying after the project update. I think you really sold everybody on staying in the room. So, all right, welcome to the open telemetry collector projector up project updates, not projector updates. Um, this is I guess now year seven of the open telemetry collector, so it was probably about time that we put on a put on an update for you.

Um, so I'm Alex. Uh with me I have Antoine, Dimmitri, Jad, and Evan. And we're all uh open telemetry collector contributors, maintainers, approvers. Um and we're going to give you uh an update on all of the efforts we've been putting into the collector, various components, as well as the stability effort as was uh advertised earlier in the previous session. If you would like to have a time

machine and go back to the previous session, you can always do that. You're right. I have a button. I have a button and I can walk around. Great. Um, so just show of hands. Who here doesn't know what a collector is? I know you're lying. Okay. All right. Thank you. So, this you I won't spend too much time on the slide. The collector is a multi-tool. It's

a binary that the project makes available for people to use for receiving and sending data and processing it. So, I'm just going to move right along here. So um we're going to really focus on kind of the project the project status but also we're going to give because this is the first update that we've ever given. We're also going to talk a little bit about how the

collector is organized. So the collector sig is u maintaining all of these three different repositories. We uh we split them into three repos based on you know roughly the following responsibilities. So the main collector repository the open telemetry collector repo we also call it the core the core repo contains things like the core libraries that every component relies on for implementing uh the various aspects of what

they do. uh it includes also some key components like the OTLP receiver and exporter uh also the batch processor which eventually will I I don't want to spoil your your slide so um and uh it also includes things like the builder and m data genen so the builder is the tool that you can use to create your own uh open telemetry collector distro all of the distros

that we maintain and make available use the builder to build them uh we'll talk a little bit more about that and uh the next repo we maintain is a collector contrib repo. If you have you all run a collector in production, show of hands. All right. So if you've used any of the components there, it's very likely that it lives in collector contrib. Even uh core components

like the Prometheus receiver uh lives in that repository for historical reasons which we won't get into. Uh and the last repo that we maintain is the collector releases repo which includes uh this is where everything that gets packaged any binaries containers um live and that's where all the distributions are also defined. So if you're ever look wondering what's included in my DRO you can look at the

manifest file directly in that repository. All right one of the things that always comes up when we ask our end users what do you want to see on the open telemetry roadmap? What do you care about in the open telemetry collector? Uh is stability. So we've asked this question. I think this is over like three different years and every time stability is like first or second up

there. Some of the other things like config management which I don't know if we're going to talk about I don't think we're going to talk about today. That's all right. Um so we're going to talk about stability first. So, two years ago, I I had a I had a thought when you were talking about, you know, there uh someone I think maybe the injector was hoping to

go 1.0. Uh this year, I was like, "Wow, if I if you had said that about a collector in 2024, uh you know, I wouldn't think that it'd be standing here telling you that we're still chasing uh 1.0." But in 2024, we opened an issue where we wanted to make an open telemetry collector 1.0. The goal was to make a kind of an OTLP proxy. We wanted

to just stabilize the OTLP exporter and the OTLP receiver as well as as all of the collector core libraries. Um that was a lot of work. It turns out that stabilizing go APIs can be really difficult. We wanted to have stable APIs for our um collector core libraries because we wanted to ensure that any components that rely on these libraries don't end up um you know having

to work deal with backwards incompatible changes. So we put a lot of effort in trying to stabilize those modules. Uh we also wanted to ensure that any configuration that these components so things like the config http module uh exposes doesn't change from under you if you're a component owner or if you're an end user. So we spent probably the better part of a year and a half

trying to stabilize these modules. We're up to 25 modules stabilized. Um there are many many more that are not stabilized but um that was kind of the original 1.0 effort. And then sometimes last year there was some graduation recommendations uh along with stable by default which if you were at the last session you heard more about from the GC update and we gathered together all of the

u maintainers got together and we kind of started talking about okay well what can we do to enable stable by default meaning that um you know we we do away with any uh any surprises or any things that could be like affected the performance of the collector by default. or whatever. Um, and so we decided to come up with a list of components. This was uh based

on surveys that we got from our end users about what components they cared about. I have a list on the next slide. But we wanted to focus on stabilizing those components. And when we're talking about stabilization, we produced an RFC that identified what do we actually want as a graduation uh requirements. Uh, and this included things like having a stable configuration, having benchmarks, having clear ownership over

who owns the project. So this uh the the components. So this meant that we had to have at least two uh different owners and we also wanted to make sure that the component had documentation. This is the list of components we got from our uh user surveys and uh we are now going to talk about the Kubernetes component I think. >> Yeah. >> Yes. >> All right.

So Kubernetes and a couple other use cases. So um you're going to hear me repeat this a couple times. I'm going to sound like a broken record, but uh most of what we're doing upstream is focused around stabilization to some degree. Uh it might be taking a while, but uh we are working hard on it. So the Kubernetes components starting with the K8 attributes processor are working

on um basically consolidating on the hotel sem um just declared RFC. If you uh look at the presentation later, you can click on the link and check it out. Um, and also check out the processor readme. Uh, if you're updating and want to opt into these or are you're worried that uh we're going to get ahead of you, um, take a look and um, you can see

what's going to be changing. Um, additionally, uh, there's been some recent performance improvements. Uh, you can expect a about a 20% uh, memory reduction in the attributes processor uh, past version 0.147. Um so if you're using a collector uh an older one of a fine vintage uh and you're updating uh take a look for that. subsequently we're going to be seeing the other Kubernetes components follow. So

the Kublet stats receiver Kubernetes uh cluster receiver um those are also going to benefit from the semantic conventions. Um, some new developments. Uh, if you're monitoring your Kubernetes clusters and they have multiple nodes, which they probably should, um, the leader election extension is going to help with that. It's going to make sure that you're not, um, over scraping your uh, Kubernetes API. Um, and if you're a

user of the Kubernetes events receiver and you're worried about, uh, deprecation, uh, if that's something that's been keeping you up at night, um, it's not going to be deprecated anymore. Uh moving on to uh another popular project, Prometheus. Um likewise stabilization. So the Prometheus receiver should hopefully be stabilized sometime in the near future. Um and with that uh the collector's Prometheus endpoint. So if you were using

um the collector's SDK to uh get Prometheus metrics um that config is going to stabilize as well. Um some things to note. Um the way that Prometheus or the Prometheus receiver specifically determines when a metric series um began, uh you now need to use the metric start time processor. Um this opens up a host of new configuration options for uh exactly how you want to determine that

start time. Um but the Prometheus receiver will not set it. So make sure to include that processor. Uh, additionally, uh, in case you're, uh, a user of the Prometheus remoteway protocol, uh, there's now a receiver, uh, in addition to the already existing exporter. Um, and, uh, not something that we're able to really cover here, but, uh, maybe check out a Prometheus update for this. Um, the Prometheus

team is working on something that will allow you to wrap existing Prometheus exporters and make them into collector receivers. So if you're looking to consolidate your telemetry collection on the collector, uh this might be for you. Uh finally, um OTL, uh we're also working on stabilizing OTL. Uh we just promoted both the transform processor and OTTL to beta. Um hopefully stable is upcoming. Um and one big

effort that we've been um working on toward this is the inferred context uh feature of OTL. So this means that you can now uh basically just directly access the data without needing to worry about the concept of an OTTL context. Um it will just figure it out for you. Um so it's already in the transform processor and the filter processor. Uh routing connector is uh forthcoming. Um

speaking of the routing connector, there's a handful of features that are available for more advanced routing. Uh not going to cover them in depth here, but uh if you're interested, please take a look. Um, likewise with new OTL functions, if there's been something that uh you've been kind of itching for and uh you think you might be able to um use, uh take a look and see

if it's there. If not, uh open an issue and uh we'll see if it um makes sense. Um likewise, uh new pads. So if you're um there's some part of your data you haven't been able to access, uh again take a look. Um and then finally um various performance improvements. So if you upgrade um and you find that the performance is better, you're welcome. >> Batch processor

uh is the pretty important component that uh was introduced from the beginning of the collector and was recommended to be used for pretty much everyone. uh because it like as obvious from name it batches uh the requests coming to the collector and send them as one request. But there is a one uh particular problem with that and maybe not one but this is the most important is

that it has its own inmemory buffer specifically to for one batch and it cannot be configurable in any way and if you are getting back pressure or errors from your back end or for example your queue uh on the exporter side is uh running out of space so those errors will not be propagated and because of this like a synchronous behavior of the buffer within the batch

processor. So within uh effort of moving stabilizing the collector, we decided to uh deprecate that one and move batching capability to the exporter. So you don't need that processor anymore. You can just enable batching at your exporter and it's available for every exporter because it's part of the like core component API. And uh the batching here in the exporter is uh uh set after the queue we

and it utili utilizes that queue for like for getting all of those batched uh payloads and Q can be configured. You can even disabled in that case the batching will be synchronous. So all the requests will wait until batches batches uh complete and sent to to the back end or it can be also persistent. And here's how you configure it. Uh you don't need batch processor in

your pipeline anymore. You just specify additional section and the under sending queue uh called batch. Then it also has all the default um parameters similar to the batch processor. If you if you want you can specify like flash timeout, min max size. Uh but one another pretty important improvement uh is that sending Q and batcher now can be configured not only in terms of events uh in

terms of items like traces uh spans data points and logs but also in bytes. So you can specify uh Q size in in in bytes and also each batch can be um configured same way as well. coming next uh now I'm going to talk about uh improvements that are not specifically associated with the stabilization but also uh ones that we really like to mention. This one uh

shout out to another maintainer of open telemetry core collector Bogdan. Uh this one is specific about a performance optimization made to the internal uh data model of the collector called pipeline data or P data. uh we uh removed our dependency on go proto which uh added a lot of allocations and uh we use our own uh serialization uh which is generated from the same uh data with

the same with the same types that we use for p data. So we just add a serialization to them and uh uh by removing go proto we reduced number of allocations almost half. So you can see the graph of CPU usage dramatically decreased by that and also binary size is is decreased because like Google product not needed anymore and it takes a lot of size apparently. And

next >> um yeah so we've had improvements to uh the representation of the data flowing through the pipeline but the collector also emits some telemetry of its own to describe itself which we usually call internal telemetry. If you use the uh the Prometheus endpoint of the collector to um look at the metrics of how many exports failed uh that's part of that and there's been some improvements

on that front as well uh in the past year or so. uh one particular uh line of improvement was started by an RFC in uh November of 2024 which defined some uh general attributes to describe uh which component and component instance telemetry comes from. Uh and this is pretty useful because the telemetry from all the components of the collector is going to be get mixed if they're

not properly tagged. So um this was implemented in March 2025 and so now uh if you enable the right feature flag um the telemetry so the spans metrics and logs from all collector components will be tagged based on which component it comes from. Uh we also have some universal pipeline metrics. So that means instead of having each component be responsible for measuring how much data is coming

in and out we have some metrics uh collected at the level of the collector framework. uh which is a lot more general and uh also uh more specific. It incorporates the component attributes so that you can know exactly uh how much data is coming in and out of each component instance. We also have some uh basic error logs. So once again we used to uh delegate the

responsibility of logging errors in the pipeline to a component. So component would return an error and then the previous component would have to log it at some point at some point if it's not propagating the error but now we have some way built into the collector framework to log errors. It's not all by default again but uh it can be very useful for debugging. Uh some things

that are still under consideration uh with regards to the internal telemetry is whether to uh which pipeline metrics to stabilize. do we stabilize the new one and what do we do about the old ones because there are some some older metrics which are a little bit redundant with the new ones. Uh so there's the question of deprecating them but as I said it's still under discussion and

also spans. So we have some basic metrics and logs from the collector framework. There's the question of whether we should add spans. Uh if you're interested in all of this um some of it is already available. Um, if you've looked at the logs of the collector in the past few months, you may have seen the new component attributes. But for a lot of it, it's under this

feature gate here, telemetry. Pipeline telemetry. So, I recommend that if you're interested in uh improving your view of the collector's telemetry, you try it out and give us some feedback so we can try and stabilize it. Uh one thing to note is that the component attributes that we are injecting into the component telemetry are instrumentation scope attributes not resource attributes and not span/log/metric point attributes. So you

need your exporter or backend to support that and this was a bit of a technical choice we were forced into because of the design of the API. Um some other changes not related to the RFC. Uh we're working on making it easier to toggle the internal metrics produced by the collector. Uh this is still work in progress, but one step in that direction was allowing uh collector

users to specify a set of metric views. If you use the uh the Go SDK um before you can configure some metric views um that specify which metrics to emit and how and now this is available in the collector for its own internal telemetry although it's a little bit difficult to use but it's a first step. The next step will probably be to allow each component to

specify uh what verbosity each of its metrics is uh so that they can be easily toggled at the collector level when you change the the level configuration. Um some other improvements uh related to custom collector builds. So now the the implementation of the internal telemetry of the collector is a bit uh was modularized so that now uh custom collector builds can customize it. So you can use

a different hotel SDK and whatnot. Um and as well some other thing uh that that is under consideration is whether we should rename some of the older metrics from the collector's internal telemetry to fit OTA conventions because they use underscores like the scientifically superior metrics solution uh instead of dot. Um so yeah we're we're considering changing that but it's still in the future. And then uh entities.

Yep, it's me again. I going to speak about entities. in general, entities in open telemetry is a kind of new thing to uh added to solve particular problems that we currently have with the resource. As you know resources is flat set of attribute and uh typically that set of attribute correspond to several different entities and it's not clear uh how to like figure out which set of

attributes associated with particular entity whether it's like for example process container or kubernetes spot uh and also like what's the relationships between them and was the relationship specifically to the set of telemetry that is being sent. So that's a like set of problems that entities are intended to resolve and the collector is uh there are two separate sets of uh entity information that can be delivered. It's

all like currently experimental in the spec. If you if you're interested feel free to be involved. We have entities seek uh every Monday at uh 9 a.m. Pacific time. Uh so uh entity can be sent as a reference on the resource uh with every telemetry and it's it's attached to the same resource message in protobuff. But also there is another experimental side channel when you send entity

events which will be helpful for like to carry additional information to represent relationships between entities and also get uh like complex and verbose attributes to the entities which are not like meant to be sent with the all the telemetry on the resource and also some kind of life cycle updates like whether one port get deleted you you want to have events sent right away. both of the

both of those uh like information channels already available in experimental form in Kubernetes cluster receiver. That's pretty much the first uh uh open telemetry thing available that can you can experiment with and see how how those entities are uh like shaped and how to utilize them for your needs if you want Yeah, I just Thank you. So, um I've been tasked to talk about profiles now. Um

I have to say first um thank you for the profiling sig uh because uh with the open symmetry proto release 1.10 which just came out last week. We now have uh profiles in alpha stage as a signal. Um so that team has been working really really hard uh across all the collector for the pipeline data model to to work well and they've built a lot of helpers.

I wanted to just do a quick shout out to Demia Mafio in particular, but all the profiling six members to help kind of make this model easy to update and manipulate. And um in grcale films, we we have a small thing that we've done to try to also dock foot this and make it so that we can understand how this model works uh by creating it's called

the prof receiver, right? So um you can use it now. It's in alpha. It just was added to contrib. Uh it allows the collector to report its own profiles which is great to kind of see what's going on when you're under load for example. Uh it can also scrape a P prof point over HTTP or it can just read a P profile on the disk. Um so

quickly a profile would be maybe the state of your program at a given time for maybe a given location in your code with a set of samples. A profile itself is a complex object and uh can actually contain quite a few megabytes of data, right? So in itself the the profile data uh is worth a look if you have time to go look at that release. If

you want to learn more about profiles on Thursday this week at 3:15 the team is talking about that in depth and uh they'll talk about like all this alpha uh graduation. So um and of course where profiles are used the most right now is the EBPF profiler which is a special distribution of the collector. We're going to talk about that just in a few slides. We also

wanted to talk to you about components today. So as you know contrib many many components. So um we started to kind of say we need to stop um we have too many of them. Um so we're trying to kind of raise the bar a little bit. we want people to kind of uh work on their components um make it available first maybe get some usage. So uh

we're trying to make it easy and you've seen with some of the improvements we're trying to make it easier on people so they don't have their stuff broken with stability that we we're providing. That said, we added a few um and uh we're trying to do better, but you know uh so this uh as you know maybe from playing with the collector you have receivers receiving data

um exporters exporting data of course and then processors are important because they can change things. Uh and the extensions are the nifty things here that allow you to um extend what a a receiver can do. So uh a good shout out here to some of the things that have been done with encodings. Particular we have a lot of support now for some of the encoding extensions that

uh provide you a way to read structured logs from all those cloud providers like uh in particular the Google cloud um uh encoding has been kind of uh really really active. I think that's Oh, and well, we also said goodbye to a few, right? So, um we've had a few components have been either unmaintained or frankly just not not that needed anymore. I think the big one

that sticks out is the open sensors. It's been there since we started with a collector. Uh but it was time. Um it had been managed by all of us. There wasn't like a real coner, so we had to go. Um what happens is that most of the the ecosystem is starting to just use a TRP and we recommend that you use the TRP for all your communications.

You don't need to have your own protocol >> and we're going to talk about >> right so since we're talking about uh contrib and having uh hopefully uh kind of curbing the increase of the number of components in contrib let's talk about this a little bit more. Um so we have one more one new distribution uh that's officially supported uh and officially built by the collector sig

now which is the uh ebpf profiler distribution um it contains uh some basic uh processors and exporters basically just OTLP for exporting uh and the the the standout here is the ebpf profiler receiver this is a a kind of custom distribution if you want to get some uh using ebpf. And one thing to note is that the ebpf profile receiver does not live in contrib. It lives

it in in its own repository. And this is relevant uh uh we recommend building your own distribution. Not everything has to be in contrib anymore. And that's at least what we're trying to move away from. So that means we're going to have more components outside of official repositories and outside of the official distributions that we build. So we recommend trying out the open telemetry collector builder which

is this tool that allows you to build um your own custom distribution with whatever go components you want. Um and even if all of the components that you want are in contrib. So not like the ebpf profile receiver for example. Uh there are still some advantages to using it over just using contrib. Uh one of them is having smaller binary sizes. You have a binary with just

what you need and not 200 components or so. um a lower memory footprint. Uh even if you don't use some components, some of them will instantiate things uh at at startup. Uh fewer compromisable dependencies, slightly better security posture. And on the other hand, you can ruin your security posture by integrating components from anywhere. Uh so I count that I count that as a plus. in terms of

disadvantages, however, it is obviously more work than just pulling the existing collector contrib. Um, but it can be automated, you know, that's why we have a tool for it. Um, the ergonomics could be better, but that's a piece of feedback we've had. Uh, please give us specific feedback on how you would like us to improve the collector builder uh to make it as easy as possible for

people to make their own distribution instead of relying on the all-in-one contrib. And um there are some complications related to stability. So picking compatible versions of components and of core collector components uh is a bit complicated uh because things are not stable. So if they're built on slightly different versions, it might break. Uh but obviously this is something we're trying to fix with the the whole stabilization

effort. So I recommend uh if you're interested in this flashing the QR code the QR code. Uh this is from the the official docs for the collector builder. And uh yeah, we have one minute for questions, so you better speak quickly. Otherwise, >> two minutes. >> Two minutes. >> We don't have to rush. >> That's also what it says here. But yeah, or otherwise you can reach

out to us on Slack uh on the CNCF Slack. There's the OTL-COL channel. Uh and you can leave feedback on the talk on SCED. Uh thank you. >> I don't know what to do for questions. You have to >> you have to hand them the microphone. >> Oh yeah. >> Uh ah thank you for the session. Um I was googling about the uh builder thing because I

didn't know and it gave me the result of an archive rep >> Oh yeah that's that's where it used to live before it got moved into the core >> Yeah. And I couldn't find the cmd/builder until I actually had to look up the like lurking into the the documentation. So if you are recommending that in the slide, maybe adding the link would be nicer. >> So is

the link in the docs itself wrong? Is it old or? >> No, no, no. I looked it up because I couldn't find the direct link in the slide and uh I Googled the name of the project and it gave me the archive link. >> Um it's not really a question. And I just wanted to say that. >> Yeah, that's useful. >> Thank you. >> That's good feedback.

>> There was one more question over here. >> I guess we're the last talk of the day, so we can take a little bit more time. >> You can also come by and talk to us after. But >> yeah, thanks for the session. I wanted to ask you about the uh move away from the country uh thing. uh not not not that far away like couple of

hours ago during during the digital ocean presentation the guys on the stage there then then they said that uh they actually chose country because it has all the you know like batteries included right and they loved it so there is a discrepancy between the users what they're saying and what we are trying to do here >> yeah so contrib is very convenient for users that's absolutely the

case uh like there are some advantages that I try to talk about uh about about of building your own distribution over contrib. But for the vast majority of of cases, people don't necessarily care too much about the binary sizes. They care a lot more about the convenience of having a pre-built image. So this is partly us saying uh there's too many components for us to maintain. Uh

that's that's a big part of it essentially um of why we're trying to encourage moving away from contrib because there's too much and there's too many components that aren't necessarily well maintained by their own code owners either. So it's yeah it's it's partly partly for the these benefits for users but it's also partly for us essentially. If you don't want to, if you don't want to build

your own image, use the K8 Dro. Please stop using >> Thank you for the feedback. If you don't want to use if you don't want to build your own DRO, just use the Kates Dro. >> that's that was the comment. I'm just repeating it. I'm not supporting the statement, just to be clear. >> That was more like a heckle. >> If it has all the components you

need, >> that's right. So, it's me there. Um, yeah. Also, thank you very much for the update. Is pretty dense. Uh lots of info. Thank you very much. Um so try to uh use this session to ask something well not on the slides yet. So I'm I'm looking into the uh optimization options uh regarding the the traffic that you emit because it's lots of traffic goes cross

a or it's lots of traffic in general. So improvements towards like the gRPC load balancing make it like topology aware improve on the compression add dictionary support for compression. So because I mean you you're emitting in in general I'd say pabytes of data. So any any percentage of improvement on the component would be awesome. So >> not doing anything. Okay. >> No no you're not no not

you person. Well yeah but the component is is used for good reason so excessively that any improvement is is highly appreciated. So are you at all um looking into that uh currently? >> Yeah. Um I know this comes up all the time uh for load balancer. I think we had a good discussion about client load balancer sometimes on an issue on core recently but this is so

hard like we're trying to stabilize things not give ourselves three more years of work and we have to write a very fine line here but we would I think we welcome people who know what they're doing and help and >> I didn't want to interrupt you >> yeah no that's it I mean just we we I think uh yes this is true any percentage that you can

gain on on any RPC traffic would be great, right? So, by the way, there's like arrow which also is compressing traffic. You can look at Steph, which is a final project we have on the side to make it stateful so we don't send the same payload twice. Um, you can you can look at some of those things. There is a also like research that we can do

to to make this better. We should not we should not stop here like there's always more, right? >> That's exactly what I want to mention. Look at the STE or Arrow. those projects are specifically intended to reduce the amount of traffic being sent over the network and regarding the cross as traffic and everything that's not a collector concern I think you can do that with like some

other networking specific things and in kubernetes you can specify like services zonaware services and something like that so it's that's not what we want to solve but to reduce the amount of traffic like uh in a cost effective way there are some I think so. >> And that's true. I've been talking to people from Google regarding the optimizations um in gRPC for low balancing to make to

to leverage exactly what you mentioned about the capabilities of Kubernetes. You're kind of the victims of the incapability of gRPC to handle this. >> I think we're officially out of time, but if you want to come and talk to us after the session, I don't think we're going anywhere. Yeah. That way we can let the folks close off here. >> Thank you. Thank you. >> And you