OpenTelemetry Project Update and 'Ask the Experts' - Pablo B, Juraci P, Marylia G & Severin N

This talk focuses on the latest updates and advancements within the OpenTelemetry project. The speakers discuss the achievement of stability for declarative configuration, which allows for simplified YAML-based setup across multiple programming languages, including Java, JavaScript, Python, C++, Go, and PHP. They introduce Weaver, a tool for defining and managing telemetry conventions, and the OpenTelemetry Injector, which enables instrumenting applications without code modifications. The presenters also highlight performance improvements in the Go SDK and mention the deprecation of the span event API in favor of a log-based event system. The session wraps up with discussions on community involvement, ongoing projects, and the importance of stability in future releases.

Cool. All right. So, hello everyone. Uh, welcome to the open telemetry product updates. My name is Judas. >> I am Morelia. >> I'm Pablo. >> And I'm Severin. >> So, we have a clicker actually. >> Yes. >> Uh, as you all know, open telemetry is the open start standard for telemetry. Uh, we have recently reflected that on our website with a along with a new redesign that

you can check out right now. Um and today we are first going to do things a little differently from other sessions. So we're going to do a deep dive into three different topics uh that we think should be interesting for you. >> Well, so the first one I'm going to talk about it is about declarative configuration which just reached a huge milestone. It is now stable. So

yay, congratulations. So that that is a project that has been happening for a few years now. A lot of people working on it. So I know the people are here that also work on it. So thank you all for your work and this is going to be the the future how configuration is going to work on open telemetry. So we are able now to configure complex attributes.

You're not so dependent on environment variables that have a lot of limitation. Uh now is everything also the same. The configuration is the same language for all the places that you are using. In this case is a YAML file. So you have the same file that you can use for your Java application, your JavaScript, your Python, uh also for a collector. So doesn't matter which one. So

you can have the same and just share around. And we have the data model that you can see all the options that exist which is very helpful because like today you have like so many environment variables. They're like, "Where do they all live? Which ones do you have?" So, you have to like keep researching, but now you have everything in a single file. So, that's gonna make

your life easier. Uh here I want to show an example of a very simple file. Uh it's going to be two images. So, the first one just the file format to show that is the the version that is now stable. You have you can set up some resources. Here I have an example that if you are used to using like environment variables and you want to start

doing the migration, you can see here that you can put your environment variable on your config file if you still want to. We have propagators uh and then we also have like the providers or tracer uh meter and logger. So this is an example if you have this is a very basic one that you can already use to configure your SDK. For example, uh here the care

codes we have examples which have this one also have the one for migration when you want to start using this and you are used to environment variables. Uh and then you can also just ask on the slack channel for declarative like config file that we can also help you out there. Uh if you want to know where is this available today. So we have different stage levels

on different SDK uh languages. So we have for C++, Go, Java, JavaScript and PHP. They not all of them have all the features implemented. It's currently in progress. As I mentioned, it just became stable. Uh we are hoping to have all the languages uh soon. So if you want to check like here's the care code for the compliance with this one, you can see which option is

already working for your SDK and then eventually we'll have the majority of them. Wait a little. I see some phones getting >> All right. One topic that I find most interesting lately. Who here heard about Weaver before already? Uh maintainers down with your hands, please. Only users. Okay. Cool. That's cool. Um All right. So, Weaver is a tool within the open telemetry ecosystem that helps you define,

validate, and evolve your telemetry uh conventions. like your telemetry conventions not open telemetries. So you can define your conventions for your own company. You can define what is a good telemetry, what is bad telemetry and then you can you can find that um from the telemetry that is actually been flowing in your pipelines. Now the way that it works is uh you define a registry and in

that registry um it gets all of the files for river are resolved together. They they become a schema. They uh and with that schema you can then generate code. So you can validate the telemetry that is flowing through the pipeline. Now uh one concrete example is like this. So this is how you define a schema. So you have a file named registry manifest.l and in that you

define the metadata for your registry and uh you define then a few other YAML files. So you can split into many uh YAML files for organizational purposes. But then all of the other YAML files that you have that at your registry they become part of your schema. So you define what is the name of your tele of your metric in this case. Uh I define what is

the instrument for that metric. What is the unit? What is the kind of stability that I want for that specific one and a list of attributes and I can define what is uh what is what are the names of my attributes? What are the types? Whether they are required or not. Uh and I I define all of that in a set of YAML files and they become

then my schema. I can also define some reggo rules for my policies. So I can say I need all of my custom telemetry to have a specific prefix. I need all of them to follow a specific pattern. I need them all uh to have a specific set of values or uh they they should look uh on a specific way. So if you cannot express that in YAML,

you can you can do custom policies in those regular policies policy files. One cool thing that you can do once you have a registry and once you have the policies there and so on, you can uh start a live check and the live check listens or um takes your registry and you can send data to the live check. The live check acts as a collector or as

an OTLP server. You send data to that uh that specific port for the live check and whenever something wrong happens with your telemetry, the live check uh tells you. So it generates a a um a policy violation JSON that tells you what is wrong where uh and uh and you can get that information into a a another storage and you can alert on that. If you are

interested in in in Weaver, you can travel back in time uh because we had a talk about Weaver just today, earlier today and uh one yesterday. So if you cannot travel back in time, then you wait a couple of days and you can watch the talks on YouTube. Thank you. Uh now I'm injecting myself into the conversation. Thank you. >> Um I think we all know that

like there's two major ways how you can instrument your applications. There's codebased manual instrumentation and there's the auto instrumentation, zero code instrumentation. I'm wondering who have used using auto instrumentation or any any kind of that like java agents python auto instrumentation any of that and how's like who thinks that it's a good experience or yeah okay and who thinks it's could be more streamlined and better a

few people awesome but because I think it should be streamlined because like it's very different from language to language right you have to find out sometimes you even have to find out Okay, what kind of application am I even running there? Maybe it's very difficult to to change that. And we now have a really cool project that's called the open telemetry injector. And what you can use

that for is without changing any code and without even knowing like okay what what kind of application is running there. We can use LD preload to inject the right instrumentation into your application. Right? So how does this look like? Uh oh no this is supported for the following languages. So it's Java, NodeJS, net and Python. Python not automatically. So you have have to opt in for that.

But for any of those languages you can use the injector like the following. Um you add something into the LDSO preload file or into the LD preload environment variable. What this will do is like before any process starts it's loading this SO file and running through it. And this will just inject a set of environment variables that will then pick the right auto instrumentation and add this

into your application. So all you need to do from there is just run your application and you don't have to change anything. Right? This is now node example but think about it again. Could be Java, could be net, could be python and that's it. Right? That's all you need to do. And I think this is really cool. Of course there's one more thing you can do. You

can use your favorite editor. For me it would be Vim, but I did not want to get into any any wars here. So I I chose the editor very agnostic and you can just change the file and say like hey send it to this and that collector or here's some some other settings I want to do. If you want to learn more about the injector, we also

had a talk about this last year CubeCon on the observability day. Here's the QR code. Watch it. You can learn a lot about it. Okay, so uh we talked about a few cool things. Now it's even more cool things but in a quicker succession. Um so first I wanted to highlight uh some performance work that the Go SDK maintainers have been doing on the metric side. Um

their benchmarks show between four and 30 times of performance improvements. uh just by improving locking on the metric instruments and they're also considering doing hashing on attributes. Uh these are available today except for uh some metric instruments spec specifically exponential histograms uh which are under active discussion uh but they're coming soon and I mean in general I think it's it's great that uh now that things are

are more stable uh we are able to focus on on this performance work. Um, then another thing that I don't know if it's already on our blog, if not it will be out soon, is that we are deprecating the span event API. If you didn't know, we had two ways of sending events. One was the span event API. The other one was log based events. Uh, log

based events have been stable for a while and now we are making the move to make this the preferred way to um send events. Span events will continue to be supported. we will phase them out but um you can you can still use them and uh data and views that that work on them will still work. Okay. Uh then another cool thing that we've been working on

is stability and reliability. Uh we have a crosswide project uh that I nickname here stable by default like the OT uh that we have open for this which is a crosswide project to improve stability documentation and reliability across all of open telemetry and I wanted to highlight a few things where we work towards stability. Um first of those four things is profiling which is going alpha. Uh

we have a talk about this on Thursday. I'll put a cure code later so that you can see all the open related talks that you don't have to travel back in time to to watch. Um it will be alpha on OTLP version 1.11. Uh but there's not going to be any changes really between 1.10 and 1.11 for profiles. So you can already try it out uh for

example on the collector or the EVPF profiler. Um then uh on the EVPF instrumentation uh front, one of their goals is to reach 1.0 within this year. So they're working on uh release candidate in the next few months. Um mostly aligning with the clarity configuration now that it is stable, improving the installation on different um environments like Kubernetes and improving the documentation. Uh then the collector sik

has also been uh working pretty actively on stability. Uh now with a focus on the most used components. We have been working for a while on uh the general parts that all components use but now we are focused on on specific components. Um you can learn more about that on the collector 6 specific session that will happen on this room at 5. So right after after this

one and um sort of as a collaboration between the collector and the Kubernetes semantic convention SIG um there's now a release candidate for Kubernetes semantic conventions. This is also on our blog if you want to read more and this can also be tried out on the collector uh using some some feature gates. Um and last I want to talk about some community things. Um so earlier this

year we had uh hotel unplugged which happened uh right after Fostam we had a series of unconference sessions uh driven by contributors and end users. It was pretty fun and we got to make progress on on a number of things including internal maybe not things that uh are like um things that you can try out but also on the governance side and I thought it was it

was really um useful. I I hope we can we can make future editions of it. Uh then uh there is the QR code for the list of auto talks. Um I counted the order talks and the the sessions we have in the project pavilion and I think it's over 50. Uh so you have plenty to to choose from. On um we we'll still have a few sessions.

This would be on the tables that are uh nearby the project pavilion um with uh live meetings of different sigs. Um so yeah feel free to to join us whether you are already contributing to those six or just want to say hi. Um very much appreciated. >> Now we're going to talk a little about how you can also get involved. >> Hello. Okay. Uh so now I'm

going to talk a little how you can get involved because we have a lot of new sigs happening a lot of projects. So I want to share uh a few of them. So the first one uh blueprints we have a lot of people that come and wants to say oh I would like to have some guidance. This is the architecture that I have how I should be

instrumenting. So we have this project the blueprints that right now the the call for action here is if you are a user and you want to you have already something they're using we like to hear your feedback see how you're using so we can like collaborate any chance Dan is here no because I was going to say talk to him but find Dan and talk to him

so we can basically continue like just helping out the community like end users as well. Then we have a series of new uh sigs. So also we have the the collector MCP, we have the browser sik. Next one you all have to say together. And >> zigzag >> the zigzag >> zigzag. Just want to say it. Try it out. And then we also have uh PHP DRO,

the Cotlin. Uh we also have the ecosystem explorer. This one is gonna make your life easier very much. I'm gonna explain what it is. It's pretty much we have we are collecting a lot of meta metadata about like all SDKs. So for example uh we started with Java and we have all the information that what is the data that is sending which version of semantic convention is

using so now when you want to select something you can see exactly all the metrics that are sending which version is following so you can have exactly sense if you are stable if what is the compatibility would you have so we hope to have that for all languages. So yeah, please also join and help this initiative. Uh and then yeah, we'll tell for beginners. >> And well

before we end, uh big thank you to all our contributors. I guess raise your hand if you are an member of the open project. Uh we have quite a few of them in the crowd. Um, I don't think you all of you will be able to find your name here because some of them are pretty small. But, uh, thank you to to you all. And now we

go into Q&A. >> Wait, what? No, really. I mean, we've been doing that for 20 minutes now and no AI. Are we Do we don't we really have any AI topics to talk today? >> We can do better. If you've followed maybe hacker news or just are a contributor to any open source project, uh you can see that AI has brought some good things but also some

bad things to open source projects. Uh a lot of us are struggling with AI slope with low effort AI contributions. Um this is a topic that uh the governance committee has uh been discussing actively on the last few months and we are working on ways for contributors to um be able to keep having a a good experience while we have better moderation capabilities and we are able

to tell what contributions are AI focused and sort of change the way that uh we uh interact with you so that we can will have a good experience. And yeah, now we we do go to a Q&A. >> I'm glad that we got like 20 minutes outside without AI at all. Right. Thank you very much, folks. We are open for to your questions. >> If you have

a question, there are mics here, but we also can hand out some. So, just stand up and and go to the mic. We saved plenty of time for your questions. >> um graduation >> next question please >> just to I mean you saw what we sort of said about stable by default right I mean this is the biggest part of that like this is one of the

things where we said like hey this is a thing we have to work on towards graduation it's still in flight there's still let's say paperwork that needs to be done uh but yeah it's >> there's some adopter interviews that need to happen to get uh so that the the body within the CNCF that decides on this can get more feedback on what the experience is using open

telemetry and that should happen relatively soon. >> Yes, if you're an end user and you want to talk about your adoption of open telemetry perhaps uh we are always interested in hearing uh perhaps the timing is not perfect for the graduation but we are still very much open to hearing from you. >> Yeah. So from graduation perspective, if you are an end user, want to hear from

you. If you are a contributor thinking about things like becoming stable, uh making sure that the things you are creating are I know working. >> Yeah. Why are you laughing? >> Work no bugs, please. >> Yeah. Cool. >> We can bring I Hi. Um, the open telemetry configuration thing looks really cool. Uh, we got shown the open telemetry demo project yesterday. Is there any plans to update

that to kind of showcase how you can migrate a big micros service to using this? Now it's stable. >> Hotel demo maintainers here. There's already some of the >> sorry there's already some of the services that have been updated at least one in go uh I think there was mention of someone doing something in Java but yeah eventually >> yeah I think like part is that they

if they depend on something out like they're using SDKs on like hotel but those SDKs don't have yet the implementation so this why it is like a gradual process >> it will happen um I don't know if Juliano is in crowd the u person that gave the talk uh about the demo but I think like it is his intention to to make it happen. >> Yeah, there

was a blog post um just a perhaps a couple of weeks ago on the on making that config stable. So the spec for the config is stable and and Alex here can give more information about that. But I think now that things are stable at the hotel confs are there and so on. Now uh the real work be begins in like changing documentation and changing the demo

project and so on so forth. And you can help with that, right? >> PR is welcomed. >> More questions. >> Do you mind talking a little bit? >> Thank you. Do you mind talking a little bit about the road map? You mentioned stability stability, but something else that you want to highlight. >> I would say stable by defe like this is the one thing that we all

should have in mind right now. Um we have a few other things like we have cool work been done in with injector with uh profiling with quite a but really our focus should be uh graduation should be stability by default rounding up the things that we are working that that that we've worked on a couple of years ago and they are not they're not ready yet right

so those are the things that we should really really be thinking about >> for you as maintainers like this is the message that I think the GC wants to send as well >> yeah And of course like there are things that going to vary also by different SIS they have we have like those top priorities but there are things that happen for example like the JavaScript is

right now we are working stabilization the logs so are also like several things on different ones that can vary as well >> if you want to learn more about the robot for the collector specifically uh you have to wait until five >> stay in this room >> there's a more questions yeah >> there's a hand Yeah, I cannot bring the mic for me. >> There's like a

known um limitation with having isolated tracer providers um in that since they share the same context like you have often spans sometimes and this is like something we ran into as an end user. Is this something that you think you can address or is that not in the plans? Is >> that specific to a language? >> Python. Yeah. Python maintainers here in the room. No, >> I

I Yeah. >> Yeah. Um, >> cool. >> Yeah. So, I would talk to Josh. He think he's trying to solve this across languages. Um, for Python, I don't think there's anything super specific, but please come to the SE like, um, I think we all have ideas on it and we're trying to solve it internally at Google, too. So, yeah, please come reach out. Thank you very much

for answering the question. >> Josh saved us. Yeah, >> we still have time. We can tell jokes perhaps. I don't know. >> we have a few taps to review as well. So, please stay here. Yeah, >> we talked about OTPS. People start leaving. what is going on? Uh, one thing caught me the span events I'm not a fan of span events and I do agree it confuses

what events and span events but >> some uh places they got rid of logs instead of they are using span events which gives them just what they need without logs which uh when taking that decision was it on the table or >> so uh I guess the main message is don't panic bring a towel like uh it's just a deprecation that is going to take long long

time like if you use pan events that's okay you can still use pan events at the like at the backend side of things should be very similar still or even the same uh depending on the back end that you use uh so no need to migrate or to move from one to another right now for new code the idea is start using um proper events like log

emit I think it is the API for that um and then uh gradually you move things out >> but they won't be seen under traces by >> uh this is a visualization detail I think so the thing is uh log when you emit a log you can pass the context so the the trace context is still in the log is in that event in that log entry

so uh this is why it is mostly a visualization concern uh there is an option for the back ends to treat them the same way because they have the same information. >> Thanks. >> If you if you want to learn more about this, so the blog post on this topic is live already and at the end of the blog post there's a community issue. You can go

there and I think exactly that question was asked already and there is an ongoing discussion on that. So maybe maybe you asked this question. I I I just cannot remember who who asked this question but but that's the best place like if you if you have worries about that kind of deprecation please let us know right I mean it's the forum here but of course on the

on the GitHubs we had one more question over there yes I was thinking about the real user monitoring and client instrumentation it how what's the status right now >> browser sig Okay. >> Yeah. So, we've done quite a few work on on rebooting browser sie >> uh and there is some work going on there. I cannot tell you exactly dates, timelines or give promises but uh there

is a we know it's a weak spot that we have right now and we know a lot of users care about rum and care about uh browser events and client instrumentation in general. uh it is also one of the thing one of the reasons why we are having now a cotlin sig like this is also for mobile uh so this is also client instrumentation we know there

are few challenges there um like I can I could get in details on which which one of those challenges would be more challenging to solve but uh there are things that people are talking about and and eventually >> eventually we are going to have a client >> problem one like we have like the sig focusing >> on browser >> on browser specifically So yeah, right now they're

they already started like some feature and stuff like that. There's still some organization happening. So we can like publish thing like npm and stuff like that. Uh but yeah is is undergoing. >> I guess a good summary is that it's moving and it's moving faster than it was before because we now have the the new SIG organization and the new phased approach. Um I I think if

you go to a community repo you can see the the different like the project that there is right now for the browser for example and that can give you an idea of the the road map there. >> Okay we have we have two more minutes I think so we could still take one more >> Okay. uh I'm uh a noob when it comes to open telemetry relatively

started to adopt it recently or uh got a requirement for it and I've uh the recent thing uh we've done on our project is we've mounted all the uh instrumentations and uh it is really a clean job in a way that the way it works but it gets the codebase messy so I'm really exced excited about the injector. Uh, sounds like an awesome thing. Uh, the only

thing is that I'm too excited. Do you mind if I just run away and check it out? >> By all means. >> I mean, it works already, right? So, you you you should be able to use it. And a few of the maintainers of the injector also around. >> So, talk to them. >> So, so please talk to them. And I said like watch the recording from

last year. There's there's a lot of into it. And I think another thing like we're also thinking about package management for it, making it even more streamlined for you. That's still in progress, but yeah, making it more exciting for you. >> Are we done or one more question? >> Yeah. Um I think >> I think we're done. Yep. >> Okay. Thank you everybody. Thank you. Thank you

very much. Sorry.