Jaeger V2: The Maintainers' Guide To OpenTelemetry-Native Tracing - Pavol Loffay, Red Hat

In this session, Pavle discusses the Jagger project, focusing on distributed tracing and its importance in observability. He explains how Jagger, a CNCF graduated project, leverages open telemetry for data semantics and tracing data models. The talk includes a live demo showcasing Jagger V2, highlighting its features such as span visualization, error tracking, and integration with Prometheus for monitoring. Pavle emphasizes the role of distributed tracing in root cause analysis in complex architectures and elaborates on how Jagger can streamline tracing and monitoring processes. Additionally, he shares insights on new functionalities added since the last CubeCon, including the Jagger MCP server and improvements in database support.

Hello everyone and welcome to Jagger project session. Uh my name is Pavle and now we're going to talk about observability. I'm curious who is using Jagger. Awesome. And who doesn't know anything about tracing? Okay. So yeah, in this 30 minutes we'll talk about Jagger. We'll do a lot of introductory stuff. Um there will be as well a live demo and we'll spend some time talking about Jagger

V2 which is the current uh currently supported version version of Jagger. Uh and yeah that's pretty much it. Uh my name is Powell. I'm a Jerger maintainer. Um I used to contribute a lot to the project in the past. Nowadays I do mostly open telemetry open telemetry operator and collector. Um and eager is a CNCF project uh graduated um it has been released I think 10 years

ago. Uh that's when I started VGER. So it's been long time with the project. Uh and so today we'll do an introduction to distributed tracing. I will talk about what use cases does it solve. Um and then we'll do a intro to Jerger. I will talk about the tracing data model and do a live demo that we actually publish for anyone to use. It's on the on

the website. Uh and then briefly talk about Jerger with Prometheus, how you can use tracing system to as well set up your monitoring and alerting. And then um finally something about Jagger V2. I will be talking as well about Jagger MCP which is one of the latest additions to the project. and at the end uh the new features since the last CubeCon and the road map for

2026 and beyond. So why distributed tracing? Uh and the answer is very simple. It's because our code is too complicated. We write very complex architectures systems uh and it's very difficult to understand these systems. it's very difficult to reason where the error happened and distributed tracing is one of the observability signals that can help us to answer these questions. Um it's as well great because with tracing

we see um um kind of we get the full lifespan of the request and we are able to kind of pinpoint the error and as we'll talk to the right people that maintain that service uh and tell them to to fix it because tracing system observes all the transactions. is able to kind of construct service um diagram which is very useful because we can plan for updates

in our complex systems. We understands which APIs are used, who uses this API and as well who uses our APIs. It's it's very important. Uh but in my opinion, why people use tracing is actually the root cause analysis. Since we see everything what is happening in the system, we can reason why it happened and why it failed. Uh and then last but not least, I mentioned the

tracing system uh can be used to set up monitor monitoring uh and so you can use you can as well set up your your SLAs's and I think this is very important in microser architectures because uh if you set up SLAs's in your um like up APIs uh and the latency increases with tracing you can actually reason what service introduced the the latency to the system. how

tracing tools work. Uh it's like three layer architecture. It's very similar to actually to like logging and metric system. So first of all there is instrumentation. For metrics we have the meter API. For logging we have the logger API. I think everyone uses that and then for tracing we have the tracer API. Um these APIs they create the data which is then sent to the collector. Collector

might perform some it collects the data. uh it might do some transformation, some filtering and finally the data arrives to storage and visualization. Jagger project at the moment handles the data collection with the storage and visualization. We used to have as well instrumentation libraries with SDKs but now we recommend users to use the open telemetry SDK instrumentation libraries. Um Jagger natively supports OTLP for the ingestion but

we will talk about that. Um so intro to Jerger um Jagger relies on open telemetry data semantics on its data model. So we use a trace is the main object. Um you can think about it as like end to end invocation in a system and trace consist then on of spans. Uh so every span represents some invocation in a system. So for instance a database call or

HTTP call. uh tracing system usually don't create spans for every method invocation in your system like the the private methods in your business logic but it will create spans only for like entry and exit points. Uh yeah that's the span and then we have text are you know data in a span so list of attributes um that kind of will tell you what the span was doing.

So for HTTP request you will find the URL the the method and the the status code for instance. Um I like to as well make this analogy to the logging system. Tracing is essentially like logging on steroids with three major differences. First one is in tracing we get a we get the duration because span has start and end. With logging we have only start we have only

the the timeline when the log was created. That's the first difference. The second is the structure. In logging system there is no structure. Uh and there is no semantic meaning of the values in the log. Uh so one service locks something for some event. Uh second service locks something different from the same for the same event which is very difficult to then analyze in large scale systems

because there is no consistency in data. Uh and the last difference is correlation. In log in tracing system we have the ability to cor correlate spans together uh which is not the case in logging and you can implement it with some custom ids but it's not enforced. So this is how a trace can look like. It can look like a tree or as a timeline view on

the right. Uh the timeline view is the view that most tracing systems support uh because it's great to analyze the um how long each operation actually took but we will see that in a second. So let's go to demo and actually anyone can go to jaggotracing.io and we have published uh a public demo with the demo app. Uh so I'll just this is the the hot rot

our demo app and this is the Jagger UI. I'll just refresh because it will shut down. And so the demo app is very simple. It's um sort of simulates Uber. If you click on one of these buttons, it will order a car ride for you. It's a micros service application. So I'll just create some requests to simulate some traffic. And what we get is that some driver

will arrive in two minutes. This is the license plate and this is the latency measured from the browser. And then we can jump into Jagger to find the trace. Um but the first thing what I want to do in Jagger is to actually show you the system architecture uh where we can reason how big this system actually is. So we see there are three there are four

services. There is front end customer driver and roots and then we get two databases. Jerger is as well able to understand like how many times these services are being called. In this case I see the customer driver they are being called the same the same amount of time but the root service is being called much more often. And then the same for MySQL and radius the radius

is u it's much more popular. is being called like 10 times more than the MySQL. Okay. But to actually perform the root cause analysis, we need to go to the uh main page and search for traces. I need to select the service name and then I get the trace instances that are recorded in Jerger. Um so what I see here is the total duration of that request

as it went through multiple services. These are the service names and I see the number of spans and number of errors. So when I click on it, uh I get this nice view. Um it may seem complicated but it's actually very simple to read. On the left side we see the application name with the uh with the operation name. And then on the right side we see

the lines and the longer the line is the longer the operation took. So the first span took 600 milliseconds and then the MySQL call took 255 milliseconds. So by looking at this I understand the timing of each operation. What I can do as well is to look at the the patterns. What I see here is this staircase pattern. So it means this operation was done in a

loop in a sequence. So maybe it was a mistake. Maybe it should be like this. Maybe we could optimize this and execute all these requests um in chunk. When I look further down, I see the same pattern again, staircase pattern. So maybe I could execute all these requests at done as at once and get everything done much quicker. The important bit here is the exclamation mark in

the radius tracing system. when the error happens it will log it. So we will get the the error tag and we will get as well the the log which should contain the exception message. So these logs that you see here they are like first class citizen in a tracing system. Uh but what you can do as well is to redirect the standard logging into your trace logging.

We will see that in a in a different span. Um so this is the span for the HTTP. We get the what was the status code, what was the URL, what is the client address. But here we actually have all the logs that the application was doing during this span was active. Okay. What I want to show you more is the MySQL. for MySQL we get the

the full SQL query. Okay. So this is the the kind of the base visualization for trace. We have as well the flame graph which is a new visualization that we added to the project. You can again select the operations. Uh we as well have the the trace graph which groups the same operations from the same service. This might be good if you have a large trace with

so many spans. It will help you to understand at a high level what was going on in the And then last we have the trace statistics where we calculate um the total and average time for each service. Now what I want to show you is the trace diff which is again a nice feature. you can compare two traces uh and the UI will show you like what's

the difference. So in this case the second trace contains one more span from the radius that's why it's a green. This is not a great example, but if you have large system with so many spans and you have roughly two similar traces, it's good to kind of find out what differed and then kind of reason why they differed and maybe those parts parts that differ um kind

of contain the the root cause of the problem or the service that is kind of one level above. Okay, I'm going to back to slides and we have as well added new dark mode which I like. It's this nice color scheme some screenshots and yeah let's go to the monitoring part with Prometheus. So as I mentioned a span has the the start and the end time which

implies duration. So we can extract metric data from each span uh and then store them in the in the metric system. This sort of brings Jagger more to the like towards traditional APM system where we have metrics as well. Uh I'm going to show you directly how it looks like in the UI. So we have this monitor tab. It's not enabled by by default. You have to

enable it and you have to configure your Prometheus instance. Uh but what we get here is the latency metric, error rate and the request rate and there is as well dimension for each u endpoint as well which is very useful and yeah you can you can select by the by the application or service. So yeah, I mentioned which metrics uh this is how it looks like. So

this is the Jagger. Um it receives spans uh it creates metrics and then it sends them to Prometheus. You can send them via remote write or Prometheus scraping or via OTLP. Prometheus uh supports OTLP HTTP as well. uh and then when you query them uh you can query them directly from Prometheus or you can configure Jaggery UI to query them uh from the Prometheus. Um that's the

only kind of back end that we support at the moment for the for the metrics. Yeah, that's the UI. All right, let's go to Jagger V2. Uh which is based on the open telemetry collector. It has been a big effort in the community. I think we worked on it couple years. Um so what does it mean is uh that we kind of implemented all Jagger functionality as

open telemetry collector components. So there is a Jagger storage extension, there is Jagger storage exporter, there is the Jagger UIs as well as an extension. in in internal we have as well refactored Jerger internals. Uh now we use the P data the pipeline data structure from open telemetry collector. So if you want to implement your own storage uh and you know how the you know the collector

API has become much easier right now. Uh but we as well support the the old uh API that we used to have in Jagger. Uh we natively support OTLP ingestion. Uh the query as well returns OTLP. Uh and yeah, the internals are as well aligned with open telemetry. Uh with the v1 we used to ship different binaries for different use cases. There was the collector agent query.

Now it's everything a single binary aligned to the collector. uh and as well the configuration is aligned. Now you can configure the eager with the open telemetry collector YAML file without differences. So you just enable the eager components it should work. So this is the architecture. Um the most important bit is the storage extension which holds connection to database. What I like about it is that you

can configure different databases. one for the main storage and one for the archive storage. Um, and then the query extension which is on the bottom and Jagger exporter they get the handle to the database. So we just configure it once in the extension. So this is the config uh you see the yeah Jagger storage extension where you configure your database. You give a name always in this

case it's a sum store and then you reference the the store in the query extension and the query exporter. It's very simple. Um and yeah you can in this case we are using inmemory for both but you can use let's say elastic search for one and Cassandra for the other. So what are the new features since the last cubecon? One of them is the Jagger MCP server.

Um, everyone has to do some some AI stuff, but no, I think this is really cool because the Jagger MCP supports progressive disclosure, which means that we don't or let's step back. The main functionality is that you can use the MCP to query the data, right, the traces. But the way it's done is that it doesn't return you the whole trace at once because that can feed

the fill in the context window so quickly. trace can have so many data. So uh we were thinking of approach that is more efficient for the model. So in this case how it works is you search for traces you get some trace summaries and then you ask for uh let's say the critical path of a given trace or for errors in that trace uh and then you

can ask for the span details for the spans that you would like to investigate further. So these are the the tools. Um you can get the services and then do the search after which gives you the trace summaries and then you can ask for critical path trace errors and finally for the spend details. So it should be very efficient way to to analyze large trace data sets.

Um so what are new features since the last cubecon? We have removed V1 completely from the codebase uh in January. There is as well has been done work on the click house. Um click house used to be supported before as well as a storage plug-in. Now it's directly baked in into Jagger. Um there are as well improvements to support latest um databases like Cassandra, Elastic Search and

Open Search. Uh there is the new MCP server. We have added Helm chart for Jerger V2. on Kubernetes you can as well use the open telemetry operator to to deploy Jagger uh and we have added the the live demo the live demo right now I don't think it's it deploys the latest Jagger version but it's something we will improve to uh quickly update the demo what we

when we release the new Jagger version so on the UI there is the dark mode uh we were looking as well at the trace timeline view uh you can as well configure Jagger UI to use the open telemetry naming conventions. So right now by default we still use the the tags and the old names but you can reconfigure it to be aligned with hotel and we have

added the incomplete trace detection which is a cool feature because many people get confused when they see a trace uh they think it's complete because um Jerger will show you a trace as soon as there is a just one span u but the processes they report spans independently and so it needs to refresh the Jagger UI And this kind of signalizes that you should refresh to get

the full trace. And what is on the road map? Um we would like to work more on the uh AI use cases. One of them is the AI assistant in the UI that will help you with the trace to analyze the traces. Uh and then we are thinking about the supporting observability for Gen AI workloads. Um, we could for instance show you number of tokens that you

used or the prompt. Uh, um, yeah, we we don't know how that's going to look like. There are just some thoughts. But if you would like to collaborate on this and you have some input, feel free to to go on the GitHub. There are issues. You can uh you can uh you can talk to us. Yeah, I think that's all and we have time for questions. some

data you get retention which I kind of don't really agree with but does >> I'll repeat or if you can please go to the mic >> I'm audible oh yes I am um yeah I was just wondering um about retention some of the Prometheus source um data sources they offer retention which as I said I don't really agree with but does does this uh framework offer anything

to deal with that like I take in mind that there's a retention of x period so therefore display it correctly or or you know decipher it correctly >> yeah absolutely I think for each storage there is a different retention mechanism how that allows you to configure how long you want to keep the data in the >> does that answer the question >> yeah can I offset it

as well like in the sense say if you to I don't know a 15-minute retention just ignore it for example. >> Um what do you mean by retention then? >> So like uh some of I mean this is more I'm I'm trying to work around issues with applications that erroneously report metrics like last error but then they retain that for 15 minutes. So there is no error

for the last 15 minutes, right? Is there any way I can offset that to say don't worry about retention actually worry about difference? >> Okay. I I was I thought that you mean by retention like how long you want to keep the actual data in the database. >> No, I mean the other way like how how much can I ignore it? >> Oh like the first I

still not sure I understand you completely. So um for example if you get like a last error and the retention time is 15 minutes >> then for the whole period of 15 minutes you have an error. >> Yeah. >> But that's not true obviously because the error happened whenever was the beginning of 15 minutes. Is there any way I can ignore that in in um you know

using ego? >> I don't think there's really a problem with tracing system because like once the error happened it's it's stored in the database and you need to search for it uh to to be visualized in the UI. >> Mhm. So but it still will be an error for the whole entire 15 minutes, right? Like visually it would >> Yes, of course. Until it's updated. >> Right.

Okay. That's okay. Thanks >> great s. So I imagine you know pers like dashboard as code and so on. Have you thought for Jagger to have some sort of way that each user can configure the dashboard how they want to visualize the the trace in Jagger? I don't know maybe importing a a dashboard from pers which is kind of the is going to be the def factor

standard. Have you explored those? >> Yeah that's a great question. I'm not sure if there is a kind of data source the Jagger data source in purses but pers in CNCF dashboarding uh project >> tempo for sure. >> Yeah. And uh the the the metrics that we store in Prometheus you can query them in purses of course it's it's a standard Prometheus metrics. Yeah. >> So the

plan will be to have like the same that we have for tempo like a data source and then you visualize the traces as you want in pers right. >> Yeah. I'm not sure if the Pers community is implementing the the dashboard for for Jagger. No, the database for Jagger. >> The data source. >> Data source. Sorry. Yeah, >> we will implement it. >> Awesome. >> Great job.

Okay. Any more questions? Uh I was I was wondering if you have um like any recommendations or if you um have any plans on on maybe maybe already works well. So I'm on tracing control loops like with for Kubernetes controllers operators you often have like don't have this one to one association where like one request results in one action but of sometimes you have like three changes

that result in a single reconciliation actually changing things which can be a bit tricky to put into Jerger as as I've understood is there like any yeah anything you and >> the issue with this use case is how do you model such a trace right um so in Jerger you can use the span links >> so you can have those three loops as a three different traces

and connect them with the span links I think if you open Jagger UI you should get you should see the span link if you open a span. It should be somewhere here. So, you can click on it and it will open a new trace. >> Yeah, I think I've seen that. So, basically you you need to collect whatever references you have and then attach them to to

this reconciliation. >> Yeah. And then Okay. >> Yeah. So, so all the controls that they have to be instrumented that they somehow have to export spend data. Now, the question is how do we model it, right? what is the best way to then model it and visualize it, right? Because it's sort of related, right? >> Many tracing systems then they will just show you the link and

you can jump to the next trace. You could maybe model it as a single trace, but it depends if there is a single entry point, right? Yeah, I think you often have like multiple entry points and then like so basically three traces that then result like in in a single >> a similar problem is with the messaging systems, right? You have like multiple consumers um or multiple

processes that will kind of send the message to the to the topic and then you have multiple consumers and like how do you model it like do you continue one trace or do you start a new trace for consuming the >> and and then you put batching in the mix gets interesting. Yeah. >> So, but yeah, there is the way with the span links. You should be

able to visualize connected traces. >> Mhm. Okay. Cool. Thanks. >> Hey. Uh and speaking about storage, the database retention. Uh I think last time I believe last time when I touched Jagger internal storage doesn't support retention, right? >> It does support. It depends on the storage. But for instance like in elastic search you can define like how often you want to roll over to the new index

and then you can just clear the the old indexes. you can as well configure which index is the Jagger UI should read and I think the same is for Cassandra when you set up the index or the table you can define like the TTL >> without dependent on open search elastic search uh Cassandra and Jagger have internal storage as well right >> Jagger has the in-memory storage

>> only memory okay there I think you can you can like define how much traces you >> okay then thank