The Fourth Pillar Arrives: OpenTelemetry Profiling Alpha in A... Felix Geisendörfer & Florian Lehner

This talk introduces the latest advancements in OpenTelemetry profiling, particularly focusing on the alpha release of profiling profiles. The speakers, Felix Gazender from DataDog and Florian Ler from Elastic, explain the essence of profiling and how the newly developed profiling format addresses complexities in performance and resource usage analysis. They discuss various use cases for profiling, which include incident detection, cost efficiency, and performance optimization. Visualization techniques like flame graphs and thread timelines are highlighted for analyzing profiling data. The session also details the integration of eBPF for continuous profiling and how it supports multiple programming languages without requiring extensive instrumentation changes, enhancing overall system visibility. Finally, they announce the official launch of the profiling signal into alpha and call for community feedback.

Hello everybody and uh welcome to our presentation uh the fourth pillar arrives open telemetry profiles alpha in action. With me today is Florian Ler from Elastic and my name is Felix Gazender and I work at data dog and we're both members of the profiling special interest groups which is why we're talking today uh about profiling to you. So to get started let's have a look at what

profiles actually are. What is the essence of a profile? Um to to illustrate this I brought you a simple CPU profile in a text format here uh where you can see this particular stack trace main calling into surf request calling into get balance call into DB select was collected 30 times by a CPU profiler which is a sampling profiler. So we saw 30 of these samples and

then for this stack trace we saw 20 samples five samples etc. And this is the simplest profiling format in existence. uh it's called the folded stack format by Brandon Craig who also invented the uh flame craft visualization and in a perfect world we would be done here this would be profiling and our job would be done we wouldn't have to do anything for open telemetry but the

world is not perfect uh the world is complicated it expects efficiency uh with efficiency comp complexity and so for example we realized that for example many of these function names here that you see in this profile would be repeated multiple times so you would see main multiple times and that's not ideal. In fact, in some cases, when you have not just an aggregation format, but you would

include the timestamps for each individual sample, which there's many use cases for, you might have to repeat the entire stack trace that you've seen multiple times for each time stamp you want to associate with it. So, you have to think a little bit about how to make a profiling format efficiently um and and support other use cases such as timestamps, labels, etc., which the open telemetry profiling

format does. Um, so let's talk about use cases for profiling. Why should you care about profiling? What can you use it for? I would say the three main use cases are incidents, costs, and performance. Uh for incidents, maybe one of the most common scenarios is that you're looking at a dashboard showing CPU usage, and you see a huge spike in CPU usage. Maybe that's what you have

an alert on. Maybe you have an alert on latency, but anyway, you find CPU usage has increased. And profiling can be really useful for telling you which code that was maybe cold before has suddenly become hot. So maybe your CPU profile before shows a little bit of JSON parsing and afterwards you see a lot of JSON parsing. That would maybe indicate that payload sizes have gotten bigger.

Or maybe it's actually the stack traces have gotten deeper. That could actually tell you you're seeing more nested JSON data coming in. There's many interesting things you can see in profiles that can give you an idea of what the issue is. Um also in some cases it's simply a regression and you might have many commits that went into release. You might just do roll back initially, but

you still have to figure out which code was the problem and profiling can help you figure that out and then you can make a fix to roll forward again. Um costs is another use case. Um you might find yourself uh spending at some point millions of dollars on the hyperscalers and so at that point it becomes very lucrative to actually ask yourself where's all my CPU time

and all my memory going to and how can I optimize this in order to spend less and be more efficient. And profiling is a natural tool for for doing that. And uh last but not least on the performance side, profiling can also be used to help things like improving tail latencies. So the profiling data format in open telemetry can actually associate uh trace and span ids which

every CPU sample for example that's being taken. And so you can ask questions like are my P99 requests using more CPU time than my P50 uh requests? Are they using CPU differently? Are they going down a different code path? And so you can do uh performance optimizations using profiling. Um now in terms of what you get out of profiling uh typically the main visualization that you end

up with is called uh the flame graph invented by Brendan Greg which is a code level breakdown of depending on the profile type CPU memory wall time you can do mutx contention a whole bunch of things. Um in this case let's say it's a CPU profile and uh each of these little boxes here is essentially one function. uh you read this one from the bottom to the

top. So these functions the main calls reader loop calls execute comment. Sorry it's probably a little small but um and the the size of the box is how much CPU time is spent in each of these functions and and all of their qualies. And that's this you can use to orient yourself. For example, if you want to pick pick a function higher up the stack like execute

built-in uh or function here might be an interesting target for optimization or then expand words might be an interesting one. And you can look at different layers of your function calls to to look for opportunities. Um, but flame crafts are not the only game in town. Once you add additional dimensions to your samples such as uh timestamps or thread ids, you can actually do visualizations such as

thread timelines or uh flame scope which give you a better sense of how resources are being used over time. And sometimes you see very interesting patterns of repetitive bursts that are maybe associated with a timer that are very difficult to to debug otherwise. Sometimes you might also uh identify bursts of inactivity that you don't want to happen because you want to actually maximize resource utilization. So the

open telemetry profiling format will let you do all of these visualizations based on the data that's being collected. Um, in terms of how we got here, uh, a lot of work has been going into the profiling signal and we, uh, shared it for the first time to a wider audience in, uh, 2025 at CubeCon in London. And this is sort of the milestones we hit along the

path. I'm not going to cover them in detail today. I'm going to go a little bit over the milestones that we've accomplished since then, leading up to today. Uh so in April 2025 we improved the resource semantics to be more aligned with what open telemetry understands as resources and moved our dictionary up to a higher level. As a result uh we clarified the correct usage of timestamps

to make those use cases work nicely and give implementers a chance to do it consistently uh across different implementations. Uh we simplified the stack trace representation again also to make it simpler for implementers u as we found we had something too complicated before that wasn't giving us a lot of benefits. Um the most difficult thing we've done recently is uh adding dictionary support to resource attributes in

open telemetry. Uh resources are like a highle concept that also applies to locks, metrics and traces and they can have attributes and we found that for profiles we would often see the same attributes repeated over and over again in our payloads. So we wanted to optimize that and we've seen up to 40% wire size reduction as a result. Uh Florian's going to talk about this in more

detail. Um and so yeah, this brings us to today where we're happy to announce that profiling is querating to alpha here. 30 OTLP pull requests were merged and many more commits to the collector EBPF profiler and other repositories were done to make this happen. And so we are here to launch the profiling signal into alpha officially. Uh we have Severron with us here uh who has the

commit access to the open telemetry block and he's hitting the merge button as we speak and within a couple minutes if CI uh is doing its job then we should have the a blog post online uh showing you how to use profiling and giving you a lot more background information should be available shortly after this talk. And if you have one more round of round of applause

in you, uh please also give a round of applause for all the contributors that made this happen. Um as as you can see uh getting the profiling signal to the stage was a collaboration involving many people and many companies. In terms of the companies, we also want to thank the employers for making it possible for people to spend their time. And some of these companies are also

vendors that might offer support for the profiling alpha. So you should check them out if you're interested in that. Um, now I will hand it over to Florian to talk about the profiling signal in more detail. Okay, thank you Felix for the introduction and uh now it's my honor to uh give you a short introduction into the profile signal. Um, as you probably see already, the profile

signal is a little bit more complex, more complex than than the open telemetry signals you probably know like logs, metrics and traces. uh but there are key concepts that are similar or that uh that all open telemetry signals do have. So we share the same uh resource and scope but then we become a little bit different. So um if you are familiar with profiling in general then

you already maybe know recognize the structure it come it's come it's orate originated from pro. So you see a lot of similar similarities here to give a more a better example. Um coming back to the first slide and stack we have the uh we have a stack trace here like we saw earlier main surf request get balance and DB select and how does it look like if

you put it into the into the uh profile signal. um I will just take the leaf frame and the function function name of the leaf frame will go into the last uh message. So here uh in the DB function uh you get the line information a little bit more about uh the location where this appeared. Um maybe if it was not symbolized you will get the mapping.

So um this is u if you use uh shared libraries this will pop up here. uh then we have stack and in the sample in message you get will see the information how often something did show up. So dduplication was a high priority for the profiling sick uh because you want to be as most efficient as possible and that's how we come came up with uh another

key component that we added and uh Felix already mentioned it is profile dictionary. So instead of writing everything every string for example every time in every message we have it only once in string table in the dictionary and then just use uh references to it. This uh brings us to the reduction of the protocol to up to 40% which is quite impress impressive and if you use

continuous profiling you have a constant flow of data and this sums up in the end. uh so this is a key value that we added and uh that was not there with PROV yet. Another thing that is important to us as a profiling sick was correlation uh correlation with other signals like metrics locks and traces and we introduced this new message on a high as a first

class citizen to the profile signals called link. Uh and link contains information about span I3 and span ID and trace ID. So you can directly correlate the information with other signals and have an holistic view on the whole system. Um but yeah um I guess most of you will probably not directly write something that uh um produces the profiling signal. So maybe the question comes up hey

um how can I get something that will just produce the profile signals and um the profiling did also work on an receiver in open telemetry. So you can just use an open open telemetry collector and use the profiling receiver to that uh is a continuous profiler and produces this um this signal. Uh how does it look like? How does it work? It's a little bit more uh

different to how uh other um signal providers work. So uh in the user space component you have an auto collector that is that is this uh deployed as a demon set. So you don't attach it to every pot and uh you just have it once on a node and you get all the information then for all the elements that are on the on the system. Um the

collector um installs ebpf programs in the in the kernel side. So it's really focused on the Linux environment. Um so we install uh programs, we install EVPF maps uh that do the ST trace unwinding and send the information back to user space and user space does additional uh a little bit uh little bit more work to symbolize your stack traces and then sends out to the hotel

compatible back end. Um one thing that I want to point out here with the approach of EVPF uh you don't need to instrument out auto instrument or change your application or restart your application. you just deploy it and it just works. So that's a key advantage and key feature. Um with the often the question comes up hey if you have evf how does it work stack unwinding

for uh for code that is not native so a not native code like um languages that are not C, C++, Rust and Go. uh we have uh custom stack unwindings for uh Java uh Java JavaScript V8, PHP, NodeJS, Pearl. So a lot of variety of interpreted languages and uh this will just just work in your environment. Um I think most of the languages that show up here

are uh running out there. These are the major uh versions but uh we are very interested to add more and more. Um with EVPF you get the frictionless deployment. So um you don't have any interruption of your applications. Um you don't need to do any uh anything special. Um an important part for us was that you have a whole system visibility is really uh get all the

information and all the power of open telemetry in one component with the receiver. Um attach additional processors to it and you get all the information you want. That's really uh really convenient and uh with EVPF already as I mentioned probably uh is the low overhead. So usually around 1% um uh of CPU usage that's uh quite nice for an observability solution I would say. Um but we

don't stop there. Um the road is just beginning. Um what we are working on is context sharing. So um that open telemetry SDKs are sharing more information with the EVPF profiler. So you get more detailed information from the SDKs that that can work together with the EVPF profiler. uh then we are working with uh on interoperability for especially with especially with OBI. So that we OBI solution

is uh open telemetry uh instrumentation uh EVPF instrumentation uh that we share on an EVPF level information between these components uh between uh in different EVPF maps and we are always interested in adding more support for interpreted languages. So just the number of logos you just probably saw um we will uh add more and more. So um this will become even more powerful in the future. And

this brings us to a demo. >> Yeah, Felix. >> Thank you Florian. So now we get to the demo part where I have to pray to the demo gods that this will go well. It did not go well and sometimes we tried it during practice but let's see how how lucky we get here. Um so for this demo what we're going to do is uh first we

need a tool to look at profiling data and I'm going to use a tool called defiler from elastic which is a debugging tool for profiling data used also by the developers of cbpf profiler. So it's a Mac OS application um cubecon defiler which I'm bringing up here um and so this will show us profiling data if we receive some I'll try to zoom in a little bit

so you can see it a little bit better. Um, and now we need some profiling data. Uh, the problem is I'm on a Mac OS machine which is not known for supporting ebpf. So I need a Linux machine uh which I have running remotely somewhere else. So hopefully I'll be able to get data from it. The first step to makes that happen is to establish an SSH

tunnel. So cubecon tunnel uh hopefully this will uh tunnel some stuff and then we need to start an ebpf profiler on the remote machine. So this shell session is now on the remote machine. Uh we have some files here and one of these files is a collector configuration file. Uh this looks exactly like a normal collector configuration file except it's targeting the EBPF profiler distribution of the

collector. You typically want to run this collector separately from your normal collectors because it has elevated privileges. So that's why why you want it separate. Um so we configure the profiling receiver here which is the ebpf profiler. Uh you can pass some options to it if you want to but you probably don't have to. Um, and then this uh endpoint here should be porting forwarding uh data

to my local machine if things go well. Um, so we've got that. Um, and then we have a uh docker compose file which I'm using to bring up the profiler itself. Um, you don't have to use docker compose for this. You probably want kubernetes demon set or something like that. But here we have the image. Uh, important here. The 0148 version is the first version of the

uh, collector that supports the profiling alpha. So that's the one you should be trying out if you're interested. Um the comments we're passing in the configuration which is mapped down here and we still have to enable this as a feature gate because we're in alpha. Once we're stable, this feature gate thing will go away. Um so with this out of the way, we can do docker compose

up and bring up the profiler. And if we get lucky, this is now forwarding data. There's some error messages here, but I think they're benign. So let's see. Um and yeah, we got profiles here. So first part is successful. Um but yeah this is just uh profiles from some random stuff that happens to run on this host. We actually have no way of knowing if these are

true or like useful. Um for all you know I have an LLM running here making convincing stack traces happening. So we want to actually check that this thing is doing what it's promising to do. And uh to do this um we are going to remember that earlier I showed you um uh here it is uh I showed you this um profile that was in the folded text

format. And so this is something we can use as ground truth. If we had a program that actually produced a CPU profile exactly like what's showing here, we could check that the uh EVPF profiler actually gives us the right profile. Um, so to make this demo a little exciting, we're going to use a slot machine to turn this profile into an actual program. I have prepared a

uh prompt for this uh which I'm going to show you. Um, so here's the prompt. So the prompt says, implement a main.py program that runs forever and produces a CPU profile with the same proportions as profile.folded. Use a burn duration millisecond helper that burns CPU time for a given duration. The program should be a main function with a for loop that calls the functions from the profile

which then call other functions or burn fun the burn function as needed. So if we get lucky uh we just tell the thing to just do it and we see what happens. Um so it gets off to work. It reads our profile. Um then it's thinking about it thinking pretty hard. I think it's going down a good path so far but we will see. Um, there's a

chance that this computer will stop mining bitcoins in a few minutes, so that would be fun. But, um, so far so good. We're getting a Python program. Uh, let's look at what this Python program looks like. So, we've got a main function here with a while loop. It's calling the surf request and report to IRS. I don't know if we want that, but we got that. Um,

then it's calling this burn function here. Um, let's see the burn function itself. Oops. I don't know what I just did. Um, just going to close this real quick. Um, let me go back up. Uh, the burn function here shows us, uh, that it's basically doing a for loop where it just checks the current time until enough time has passed. And so everything else kind of looks

good. So, uh, I think we can actually run this program. So, I'm going to do Python main.py. And if we get lucky, then this program is now running and producing the behavior that we want the EVPF profiler to pick up. Um, and so we we have to scroll down a little bit here to get to the point in time where this data should be coming in. And

we see the Python program coming in. Just going to scroll a little bit more here. And we can make this bigger. Um, so here we have the Python program that we just synthesized uh behaving the way that we told it to burning CPU times proportionally in all the functions. And we could now go back and check that it's exactly the same proportions as I told you. I'm

not going to do the facteing here but we have done done some factchecking as part of developing the profiling signal on the VPF profiler. So hopefully it's correct. Um what you can also see kind of interesting here is you can see how the time function works. So the time prof counter function in Python is calling into lip C which is then calling into VDSO. For those uh

not familiar with it, what VDSO is is a fancy way for a user space program to get the current time without actually having to make a system call into the kernel. So we actually have a very efficient way to get the time. Yay. Um but as we promised you earlier um the profiler is not just a Python profiler or particular language profiler. So this thing can profile

any language. So we could also ask uh this uh thing here to do a NodeJS program. So uh please make a NodeJS version of this in main.js. And if we get lucky we should quickly get a a node program doing the same thing. That looks pretty good. Um, so I'm going to put this in the background. Um, it's already running it for a second, but yeah, I'm

going to put this in the background and run run the node program myself. Oops. I want some main.js. And now if I zoom out here again, we should now see profiling data for a node program coming in. And it seems like we're getting pretty lucky here so far. Uh, so here we have a node program. And here we actually see a little bit more of the call

stacks of how NodeJS calls into JavaScript. So here we have our node binary uh calling node main instance load environment bunch of other stuff eventually we call into V8 which is the JavaScript engine NodeJS uses and then we actually get our user space uh JavaScript uh call stack with the main function surf request etc. And we can also see that node is actually when it gets the

time it is also using lip c through a library called lip uv and then calling in vdso as well. So node and python kind of behave similarly here. Um, and what's kind of cool about this is kind of the mixed stack traces, which historically profilers were sort of profiling a single language. And if you're uh like familiar with larger applications, they often are multi- language. So there's

often a C library involved that you're calling into. And for the first time, you can really see how much CPU time you're being using both in your runtime as well as in the uh native libraries that you might be using. So that's pretty cool. Um, but this is all kind of artificial and we have time for a little bit one more thing, something a little bit more

realistic. Um, so what I brought you here is a little e-commerce application that is currently chucking along. It's actually not, but I've pre-recorded the data here. Um, and what we see there is a v1 version deployed. It has two endpoints and there is sort of a 1% error rate that's happening. So there's constantly some errors, but for large applications, it's sometimes common to have like a background

noise of errors. Um, and so not great, not terrible. Uh, latency is pretty good here. We have like submillisecond latency because again this like a demo app. Um, and now we have to like wait for something interesting to happen where profiling could be useful to makes this happen. I'm going to skip forward in time a little bit in the data that was pre-recorded here. And I show

you the next 10 minutes. And what we can see here is seemingly something good has happened. All the errors have disappeared. So like a a second a second version has been deployed. error rate is 0% success. Uh so developer goes home then they get a call angry call uh because customer saying the application is slow the latency keeps going up and up and up. So that is

definitely not good. Um if we look at metrics we can see that there's some other stuff going on here. So CPU usage has also gone through the roof. So um that's not good. And maybe even most worrying here is memory usage is doing this. I don't know how much memory this machine has but I'm pretty sure this is not going to end well. And so this is

kind of an ideal case for profiling where a bunch of resource indicators tell you that something weird is going on. You don't really have a good theory. Uh in fact uh I tried debugging this with lox traces. I didn't really see much. Profiling is kind of perfect for this. Um so let's look at the profiles for the service. Let's go here. Um and what we see here

is the aggregated profile from before and after period. Here we see the CPU spike. And now the best way to analyze this is to uh compare the before and after period here. So you select this uh and do a comparison and this gives us a comparison flame graph and now we see a bunch of things here. Uh there's more GC but that's probably not too interesting. What

we focus on here is there's a function called process orders uh which has become hotter. And when we zoom into this uh let me see where focus on is. Um we see that this process orders function is calling a capture error context function which is weird. Why is the error code pass hot when we're seeing no errors? that doesn't make any sense. Um, and we see that

in the before profile, the process orders was not calling that uh call pass or at least we didn't capture it in our profile. Um, this is not enough yet to like be confident what the root cause is, but what it is enough for is giving us an idea where to look in our code, what we need to analyze to understand what happened. We know there's deployment, so

we need to check the diff. And now we have some function names to crap for and look at. And so we're going to go back here to uh to the code. And this is the code of this application. uh before the issue occurred. Um and what we can see here is there is a uh function called reserve inventory and when there's an error happening we capture error

context and then we continue with the next order if that item is not in stock. Um if you blink right now you'll miss a change. This was a change that was being made. It's pretty subtle change. Um if you did blink this is the actual change. Now you wonder who would be making such a change. Who's who's um thinking like that? Um, this was done through an

AI that was doing a refactoring. So, the AI copied some code from one file to another. And in the process, it's still kind of the next token predictor even though it's trying to get smarter. Uh, it saw this little comment here, continue with the next order. What are the chances of having a break statement on the next line after comments that says continue with the next order.

I mean, the break statement is logically oops uh correct. You jump from break out here back to these outer for loop. So, you continue with the next order. But if you refactor that to continue, then every time you have an error, for example, because an item is out of stock, you would actually jump over this line here, which is incrementing the index into the items list. And

so we basically are stuck in an infinite loop trying to potentially uh reserve an item from the inventory that's not in stock. And in fact, that's exactly what happened here. There was an item out of stock. That's the 1% error rate in the background where people try to order stuff that's out of stock. And now we've created an infinite loop around this. every time somebody tries to

order something out of stock, we have an infinite loop. And now you might argue this is clearly made up for like being a nice demo. But the answer to this, no, it's not. Uh this was actually a real production instance from a from a company using LLM to produce LLM software. This is where we are today. Um and they actually did use and this is the fun

part. They did use profiling to investigate this particular incident. And I can share many more stories of profiling being useful, but this was one of the most fun ones I've seen recently in the wild. And this is why I kind of tried to recreate this for the demo today. Um, and that kind of brings us to the end uh with concluding what's next for the profiling signal.

Uh, so we definitely want to get to beta and then stable as fast as possible. Uh, there's one known issue that we still need to address around adding units to the resource attributes which going to require some discussions but most of the profiling signal we think at this point is going to stay roughly the way it is. And then we also need uh to extend OTLP to

have a protocol for symbolication because if you're profiling native applications that are written in C, C++ or Rust, people actually people are wrong about this. If you ever throw away your debug symbols from a production application, you're wrong. Don't do that. Have your debug symbols on the host. But people sometimes don't listen to that and they strip things. They want small images. And so we need a

a way for people to send symbolication information uh to a back end so it can actually make sense out of program counters and show you function names. Um and yeah, this is basically the end of the presentation. Uh we uh would kindly ask you to please try out the profiling uh alpha. Uh the blog post hopefully is live now or will be live any second now. Uh

send us your feedback on GitHub or in the CNCF uh hotel profiles channels. And if you have um extra time or interest uh we're looking forward to people contributing to the SDKs because SDK profiling is something where you can get additional things like memory profiles and other things that are difficult to get an eBPF right now or if you just have ideas for features etc. Please let

us know and you can give us some feedback on this talk as well. Um so thank you so much. Um and if you have any questions let us know. I have one if they don't mind. Amazing uh talk. Thank you so much. Is there anything happen for the instrumentations uh instrumentation work? So we have like a not only eBPF solution but we have some really you know

something we can put into code and it will push um through the hotel SDKs like go and so on. Is there anything happening to get that kind of um you know done or is purely for ABPF and that's the only >> there there actually is um there are receivers uh you probably know know net http prof from the go side and there is a pro receiver in

hotel that can already translate pro from go uh into hotel profile so this is already existing uh there is a donation proposal for flight recorder go flight recorder that was introduced with go 125 five. So these are existing. Um there is also um work done in the async profiler. So if you're coming from the Java side, async profiler can also output uh auto profiles. So there there's

work happening. Um the ecosystem is quite large and it will still need work and uh we need feedback on this. Um but yeah um basic steps are already done. >> Yeah. And I think Jonathan is also working on the Java SDK. >> Right. So people are starting to work on this but obviously I think for the go SDK nobody has started on it yet. So if you'd

be interested or somebody else would be amazing. >> Okay. Thank you. >> Thank you. Thanks. >> Hey thank you much for the talk. I really like the demo and that you did things live. Um I'm curious like proof has a lot of what you already showed there. Um and there's obviously heavy inspiration from that. what are some of the selling points that that I'm getting when I

hook that into my o uh open telemetry signals that I already have? >> Yeah, it's a good question. So, we actually did work with the pro team at Google quite a bit. We were trying to not have something that's kind of similar to but different forel buttel basically said they need to own the protocol on their end for it to be part of hotel and puff was

like our use cases don't really mesh well. So, that's why we ended up with two things. We did make some improvements over our PROV uh based on the experience we have with PROV. I think uh one notable thing that we did is uh the the way we are supporting timestamps. Uh in pro if you try to support timestamps you typically end up repeating the same stack trace

over and over again. We have stack traces as a first class citizens. So we can refer to a stack trace with a single identifier which prof doesn't have. uh we made a few other uh smaller changes to to pros that are maybe worth noting but uh it is I think the main benefit is just the open telemetry ecosystem that it's just all going to work. Um we

do have support for actually if you want to keep some original puff stuff around uh to include the original payload but for in particular you shouldn't have any need to do that uh because we made it uh compatible in the sense that you can convert convert puff to OTLP and back without losing any data. So the original payload is more like for JFR or formats that are

even more complex. Um yeah, thank you. >> Yeah, thanks for the question. >> Thank you for the great talk. The link to traces and uh spans, does that already work? for example in the Java >> um for the Java SDK I don't I cannot speak uh otherwise uh yes everything that emits hotel profiles and where this field or message is populated it will just work uh we

are working with uh the guys from OBI uh where we can get this information from uh as profiles so uh yeah it's working nicely together um getting this information is sometimes a little bit tricky uh but correlation with other signals is an important part for us. Uh that's why we have it as a first class citizen. Yes. >> Thank you very much. >> All right. There's no

more questions. Uh thank you everybody for attending. I know it was last talk. Um and thank you very much. Thank you.