I Can See Clearly Now: Observability of JVM & Spring Boot 2-3-4 apps @ Spring I/O 2026

This talk focuses on application observability, particularly within JVM applications using Spring Boot. The speakers, Tommy and Jonathan from the Spring team, introduce Micrometer, a library they maintain that aids in achieving observability. They define observability as the ability to understand a system's internals based on its outputs and discuss its importance in today's complex application environments, especially with the rise of microservices and AI-generated code. The presentation includes a demo of a simple application called 'tea house' and showcases how to diagnose errors using logs, metrics, and distributed tracing to find the root cause of issues. They highlight practices for migrating to newer Spring Boot versions and demonstrate how to leverage logging, metrics, tracing, and a new observation API for better insights into application performance.

[music] >> So, welcome everyone. Uh this is the observability talk. Uh I can see clearly now. Hopefully at the end of talk we will see clearer uh about application observability. Uh this is about like observing JVM applications and Spring Boot different versions of those as well. Uh I guess let's introduce ourselves. All right. So, I'm Tommy. This is Jonathan and we are on the Spring team focusing

on observability efforts across the portfolio and we maintain a library called Micrometer, which is what we're going to talk about today. to talk about observability, we should probably start with the basics and discuss what observability is. And I think that there are a lot of definitions of observability and a lot of people think about it in different ways and have different concrete examples of what observability is,

but at a basic level, observability is how well can we understand the internals of a system based on its outputs. So, the idea is you have this application running somewhere on some server. How how is it doing? How is how are your users perceiving the system? Is it working well? Is it working fast? Are they getting errors? Uh this is the kind of questions that you want

to be able to answer with So, meaningful insight into what's happening inside of your applications metadata about your applications. So, now we have an understanding of what observability is, but why do you need observability? Your application functionally works the same whether you have observability or not, right? So, nowadays uh applications are getting more and more complex. Uh microservices was heavily adopted and so you have more and

more of your applications, the interactions between your applications, dependencies are more and more complex. And nowadays we have a new uh factor contributing to this, which is that AI makes it easier than ever to generate a new app, generate new code, generate new features. And so, it's more important than ever that you have the ability to understand what's happening in your system. Who is running PHP and

MySQL on a one single web server? No one, right? That's what we are talking about. Good. You shouldn't should not do that. So, to talk more about why we need observability, uh environments can be chaotic. You think that if you ship a new version to uh production that that's when something might go wrong. But what if you don't change anything in your application? What if [clears throat]

the data in the database changes? You have more data, an index that was working suddenly is no longer working quickly, things get slow. And so, you can't always predict when things are going to happen, so you need to know be able to detect that and get to the root cause of it as quickly as possible. You have to deal with unknown unknowns. So, if you knew what

problems to expect up front, you could build that in, you could have a log message for when you a certain exception happens but in production, things that you didn't expect to happen happen. And then that's when you need observability to figure out what's going on. And things can be observed or perceived differently by different observers. So, this is the classic it works on my machine. You know,

a user tells you, "Hey, I tried to add this item to cart and it didn't happen." And then you go and try it and it works. Can you just say the problem's solved? No, of course not. Your user can't add an item to the cart and so you have to be able to figure out why that's happening. You have to even know that that's happening cuz you're

not standing right behind all of your users watching them do things, right? So, this is what observability can enable for you and why it's so important that you have this. So, let's uh go ahead and take a look at the demo. >> Yeah, demo time. So, I can put this We have a small demo application called tea house. Uh it's on GitHub, open source, you can download

it, it will be on the slides, you will get it. Uh and you can play with it if you want. But the whole application is like very very simple where actually like this is like three applications. Uh the user calls a web service called tea service. Tea service calls tea leaf service and water services over HTTP and they have their own databases and those calls are going

through JDBC. This is like quite simple, not a lot of applications, but just enough to demo what we want. uh let me show you. Go back here. This is our observability UI. Uh this is the UI of tea service. You might be able to tell that we are not UI engineers. Uh we are working on a but what it does uh what is your favorite tea, Tommy?

Do you like sencha? Yeah. Uh let's let's let's pick like a a big like a large sencha. Uh if you press the button, you will get like some sort of like a recipe how to make this tea. Uh let's say I like English breakfast. Uh let me get like a medium English breakfast and if I I press the button, I'm not sure I like the flavor of

this tea. HTTP 500 internal server. Did you break this? I you know, I I was making some changes last night [laughter] with with a few drinks and I I don't know what happened. So, maybe the drinks broke this. They don't did not like tea. Yes, I'm blaming it on the alcohol. >> Okay. Okay. So, HTTP 500 internal server error. What what what's wrong? Like it is not

not a lot. Uh let's try uh what is happening. But before we go there, the description promised you a cat trying to debug a HTTP 500 internal server error. So, yeah. You got this. Let's move on. So, first let's try to look into the logs because that's most of the people. Uh so, that is something that most of the people has. So, if I just like uh

try to query some logs not in the last 1 hour, but let's say like 5 minutes, then I see this error. Uh resource not found exception, resource not found tea leaf name English breakfast. Sounds familiar, but uh it is 404 and resource not found. But why it is not 500? I saw 500 on the UI. I still see it. It's there. What is what is going on?

Sorry? Uh I don't know. Like can you tell from this? I'm not sure. So, let's see. Uh let's drive uh a little bit uh like deeper. So, first of all, let's check if this is a one-off. Uh No, I'm pressing the button, nothing happens. Or is this broken for everyone? For this, let's look into the metrics. This is the application throughput. What you can see here, the

the lower graph, ironically the green is the server errors and the yellow one is success. So, you can see that there is a steady stream of errors. It's not a one-off, it's not just me, it's everyone. You can check like 4.5 requests per seconds, around Uh we have that many success uh like uh outcomes and we have server errors around 0.5 requests per seconds. That's kind of

like 10% of the requests are failing. That's not good. Uh we need to do something. In order to like uh dig deeper a little bit, I would like to demonstrate you how to jump between these big signals, looking metrics and distributed tracing. So, the very first thing that we can do is jumping between logs and traces, uh which is like quite easy if you have uh some

support for uh log correlation. Log correlation means that uh the trace ID and the span ID, which will basically identify uh these concepts in distributed tracing, will be in the logs. You can see them here. This big one is the trace ID, the smaller one is the span ID. Which you can just like get the trace ID, copy paste, and you can go to your distributed tracing

backend and just like look into what happened there. So, I can do that, but also Grafana lets me do this just by pressing a button. So, I will spare the copy paste. I'm not sure if it is faster, but maybe. Uh and this is the tracing backend looking into the exact same request processing chain uh that that error belonged to in the logs. So, let's see what

happened here. You can see that tea service gets an HTTP request to this {slash} tea {slash} name endpoint. And if I open this up, I can see some details about it. I can see the 404 in the error message. It says not found. It was a get to this endpoint, which is English breakfast, that's fine. But server error 500, something is fishy. Uh let's see what else

do we have. You can also see that this tea service, it made an HTTP get request, which was received by water service. And also then made another one, another tea leaf service. You can also tell This is a timeline on the left that these two requests, if I collapse them, they did not parallel. They were sequential. So, this timeline also helps you not just like debugging latency,

but also like debugging like timeliness for the lack of the better word. So, there is another interesting thing in this distributed tracing backend. Do you see this red button? That means something is bad. That means that there was an error with that span. So, you can see that as well here. T leaf service is giving an error. So, let's try to like dig down and look into

this. And what you can see here is that T leaf is giving us back 404. So, it seems that T service called T leaf service. T leaf service returns 404 and don't ask me who, but somebody remapped that 404 to 500. Terrible terrible code. Don't do this. But, we don't know why why this is 404. let's see what else happened here. So, after this like receiving this

HTTP request, you can see uh database connection here to the T leaf database. So, let's look into the DB connection. You can see that this is the JDBC driver of MySQL. Uh I am using HikariCP. And also, we are connected to the T leaf Okay, that's fine. What else can we learn from this? You also can see that there is a query here or a span and

that represents executing a query. And you can see the same details here. And also, you can see that there was a select query. I'm sorry, I did not write this uh JDBC query. This is Hibernate uh generated, but you can sort of tell that uh it we are selecting a bunch of fields uh from the T leaf table where the T leaf name is something and that

something is English breakfast. So, please please please don't ever do this in production except if you are very very adventurous. But, theoretically, you can grab this IP, SSH to this host, use this database, and run this query, and you can see what was the result set. instead of doing that, we have another span, an extra one here, which represents the result set. So, let's look into what

happened there and oh yeah. See this? JDBC row count zero. That's the problem. English breakfast apparently is not in the database. So, T service called T leaf service on HTTP. T leaf service called the database, got an empty result set, returned 404 as it should be, and then the problem arose, and uh T leaf uh sorry, T service got that 404 remapped it to 500, and mystery

solved. So, let me try to fix this error. And let's see if after putting English breakfast in this is now working. This is great. Uh let me show you two more things. How to jump between these signals because the only thing that we did for now to debug this is we were jumping from logs to traces. Nothing prevents you whatsoever to do the jump to the other

direction because here is the trace. You can grab the trace ID, you can go to your logs, and search for that trace ID. So, I teach Grafana how to do this. And there is this button logs for this span, which is kind of a lie because I am using the trace ID. So, it should be logs for this trace. But, if I press it, then what you

can see here is a fabricated query that basically querying this trace ID. So, if I go down here, I can see that there were logs from T service. There should be logs from other services as well. If I go Yes, water service, and there should be one for Yeah, T leaf service as well. So, this is all of the logs from all of the applications that had

any part in processing that HTTP request. So, that's really nice. Uh let me show you how to jump between metrics and traces because that could be a little bit more interesting. This was like very very easy. So, the easier part between like jumping is jumping from a trace or a span to metrics because if your metrics and spans they contain the same metadata, the same keys and

values, low cardinality keys and keys and values, then actually you can grab these keys and fabricate a query in your metrics backend. So, I teach Grafana how to do And if I click this button, then you can that every one of these uh like keys and its value, it was basically just copy-pasted from the span. So, I fabricated a query like that, which was basically showing like,

"Hey, show me all of the metric Also, show me an aggregated view for all of the spans, all of the events that happens that are similar to that." And you can see that this is basically the error rate, and it is going down to zero because we fixed it. So, that's awesome. Uh but there is also another interesting thing here because if we even if we can

jump from traces to metrics, how can you jump from metrics to traces? Traces to metrics is easy. You just try to find out what is similar to this one single event. But, with metrics during aggregation, you are losing data. So, how can you jump from a number 0.5 was the error rate in one in this one second, sorry, in this one minute. How can you jump from

that one single number back to one of the spans that caused this? That's hard. So, the answer to this is a feature called exemplars. Exemplars are metadata that you can attach next to the metrics value. And if you do that, you can actually like go to your metrics. I have this dashboard here, and you can see these data points, which are not connected. Those are the exemplars.

Those are sample data points that contributed to that time series. So, when that thing happened, a span was kind of like sampled and selected that, "Hey, this one span is like played some role for that aggregated view of that metric." So, if I just like go here, you should see that the trace ID is there. You can also do the copy-paste, but Grafana has like a button

for this query with Tempo, and I can go there, which is nice. So, that was basically what I wanted to uh throughout this application, and how can you basically use these signals and jump between them, and how can you use them to debug applications or get more insights from it. So, let's talk about how to do this. Before like first with Spring Boot 2.7. Who is using

Spring Boot 2? No one? Great. I tell you why. You all passed the test. Yes. So, the easiest way to do this in 2.7 is very very easy. Just migrate to 3.5, right? That's it. You're done. So, the reason behind this, and this is this will be like uh interesting if you are using Spring Boot 3 as well. Spring Boot 2.7 is out of open source support.

It is not getting uh like new updates except if you are paying for support. Uh so, that's a problem. And 3.5 will be in the same situation in June, end of June. So, this screenshot, this is from uh like the spring.io website. It's It's in the documentation or next to the documentation. So, you will see the timeline of every single version. is 3.5 is still in green,

but it is going to yellow very very soon. You have like 3 months uh and after that it will go out of open source support. So, I am very very happy that nobody is using Spring Boot 2 because this is the pixel-perfect representation how I look like when I hear that somebody is using Spring Boot 2. So, please don't do this to me and upgrade. Uh also,

in 3 months, I will tell you that I will look like when I hear that somebody is using Spring Boot 3. But, you are still good. If you are like Spring Boot 3.5, and you want to make it observable, it's very very easy. Just migrate to four. You did not see that coming, right? In that case, you will get like a little dragon because at this point,

3.5 is still supported. Uh but, yeah, it will go out of support soon. How many people are already on Spring Boot 4? Anyone? Oh, quite a few people. That's good. Spring Boot 3? Nice. Cool. So, we are going to tell you anyways like how to set these things up with all of these versions. Yeah, so thank you for that demo, Jonathan. Did Did that make sense to

everyone? Did everyone see how you start off in the application, there was some error in the application, and then we went to go check the logs, and everyone's familiar with logs, everyone has logging in their application, but then you want to verify in the metrics because kind of the opposite of it works on my machine is it's broken on my machine, right? Maybe it's not actually broken

for every other user. So, we verified in the metrics that it wasn't just our machine that it's broken on, it was broken for everyone. And then we went and looked at the trace tracing, and we got to find the root cause. And everything was linked together, and you can go between all of them. So is is everyone [snorts] using metrics in production? Who's using metrics? Okay. Who's

using tracing? Okay, pretty good. So maybe you all are familiar with this but let's go through logging works with Spring Boot and this talk we're kind of trying to talk about the differences between Boot 2, 3 and 4 so that you can see the the upgrade path But with logging it's actually pretty simple because it's the same for all of So Spring Boot provides logging starters and

the default Spring Boot starter logging includes SLF4J. SLF4J is a logging [snorts] facade and Logback is the implementation that implements SLF4J. So all you need to get logging in your application, add the Spring Boot starter. If you want Log4j instead, you can use the Log4j starter. And this is the same again in Boot 2, 3 and 4. But then if you want additional logging, you can add

payload logs with this library Logbook. Oh, yeah, Logbook, sorry. Logbook and it has its own Spring Boot starter and has Apache HTTP client integration as well. So you can log your HTTP client requests And for access log Spring Boot has these configuration properties that you can in the latest Spring Boot Undertow is not there, that's why that's crossed Steve. And for GC logging, you can add JVM

args to your application when you launch it. So that was logging. Let's talk about metrics in 2, 3 and 4. So for metrics we're going to talk about library called Micrometer. We mentioned that's library that we maintain. And so Micrometer started off as a metrics library and it's providing dimensional metrics. So before Micrometer, most metrics libraries that were available made hierarchical metrics but dimensional metrics make it

easier to slice down into different dimensions that you can add to your metrics. So say you want to have a dimension for the environment that a application is running in. You have your test environment versus your production environment or maybe you want to have a Jonathan showed earlier we had a status dimension so you can see status on what HTTP status code was returned for a metric.

So Micrometer is a metrics library, think SLF4J but for metrics instead. And it provides an API that you can use to instrument, add metrics and you can change the metrics back end that those metrics will go to by just swapping out dependencies and you don't have to change the actual code that you're instrumenting. And Micrometer has support for a lot of back ends. How many? We support

now? At least 20. And you can of course write your own implementation if you have a some proprietary metrics back end that you want to ship things to. And Micrometer's included in Spring Boot Actuator so if you are using Spring Boot Actuator, you actually already have Micrometer on your class path and can use it. And all of the Spring projects are instrumented using Micrometer so that's how

you get all of those metrics out of the box. So Jonathan did that demo and remember the code is available online but if you look at the code, you might be kind of shocked at how little code there is related to observability itself because a lot of the observability is provided out of the box because the instrumentation is in the projects themselves. What I showed you is

zero code. So Spring projects are instrumented with Micrometer. Even other projects that are not in the Spring portfolio also are These are the supported back ends and protocols. And I mentioned the actuator starter. So if you have actuator, you get metrics and you can add a specific back end. So in this case the Prometheus registry. And if you wanted to ship metrics to a different back end,

you would add just Micrometer registry and the name of the supported, we ship metrics there. And you can even add multiple meter registry implementations and have metrics for both or multiple back ends. And I guess it is not on the slide but this works like the previous slide. It works for Spring Boot Yeah, you can you can do that. No change. Yeah, so with Spring Boot 4,

if you're familiar, there was a big modularization effort and so some of the starters and options uh changed but the actuator starter is still there even in Spring Boot 4 but you have this new option, Spring Boot starter Micrometer metrics. So you can add Micrometer via starter without necessarily adding the actuator if you don't want the actuator in your application. the same as before, you add the

registry implementation that you want. All right, that was >> [snorts] >> with Spring Boot and I'll hand it off to Jonathan for tracing. So let's talk about what you can do if you want to add like distributed tracing to your application whether you are using Spring Boot 2, 3 or 4. 2.7 like I will be quick because no one is using it anymore but the distributed tracing

support was through a library called Spring Cloud Sleuth. And Spring Cloud Sleuth basically provided a abstraction layer over tracing libraries and two of them were supported. One of them is called Brave from the OpenZipkin community. The other one is called OpenTelemetry, that's a CNCF project. And OpenZipkin, sorry, Brave was kind of like the default because it is more like a mature mature project. OpenTelemetry is a little

bit like younger and more experimental. There are still parts of OpenTelemetry that are not stable yet but these are in heavily quotes because there is no default. You need to change, sorry, you need to choose one. You need to tell like which one do you want. So yeah, the a lot of Spring applications, like Spring projects, were like instrumented with Sleuth and also like Sleuth supports various

back ends because these two tracing libraries also supports various back ends. Sleuth needs these tracing libraries to operate. So what you need to do if you have like 3.5? Let me jump between these two slides. spot the difference? Not much. What happened in this Spring Boot 3.0 timeline when we released 3.0? We also released a project called Micrometer Tracing. So Micrometer Tracing was born by copying the

code base of Sleuth and putting it under the Micrometer umbrella. Without the Spring parts. Spring Cloud Sleuth it depend it had dependencies on Spring Cloud and Spring Boot. Micrometer Tracing does not. You can use Micrometer Tracing with any JVM application. You don't need to use Spring if you don't want to. Other than that and we renamed the package, it's the same. If you are familiar with the

Sleuth API, the Micrometer API looks, sorry, the Micrometer Tracing API looks exactly like that. If you want to use Brave, that's tracing library with Micrometer Tracing, that these three dependencies what you need. Actuator, you need to like choose the Bridge Brave which will give you like the support for the Brave tracing library. And then you need to tell Brave where to report the data. And this is

I'm just using a Zipkin reporter here and that's it. If you omit that last one, you will still get the trace ID and the span ID in your logs but the data will not be sent to anywhere. The spans will not be reported to a back If you want to use OpenTelemetry, let me jump between the slides again. You can spot the difference, it's not a lot.

See the second line? We just change the name from Brave to OTel and that's it. And then you need to tell OTel which like exporter you want to use and this is the OTLP exporter. So this was the Spring Boot 3.5 way. You needed these three dependencies. And a lot of people thought this is too So Spring Boot 4, we have a Zipkin starter and we have

the OpenTelemetry starter which is doing exactly like that what you saw in the previous like two slides. They are adding these dependencies and the auto configuration for that except since this is a OpenTelemetry starter and OpenTelemetry also supports metrics, the OpenTelemetry starter will bring in the metrics support of Micrometer. If you don't want it, you have two options. One, exclude that. Two, don't use the starter but

do those three dependencies So yeah, with Boot 4, you can just have one dependent, sorry, one one starter for these. For OpenTelemetry logging, I'm So for OpenTelemetry logging, we don't really have like a starter or like very very exhaustive auto configuration because like the logging support for JVM applications in OpenTelemetry is not stable yet. Once it will stabilize, we can add more and more bits. There are

also like new modules. So you can see like there is like Spring Boot Micrometer Tracing Brave and Micrometer Tracing OpenTelemetry. You can use this as well if you want. We highly highly recommend to use the starter instead but these are the modules that will be used under the hood. You still need like uh tracing library like a bridge and also like a reporter or exporter. So, you

will not like get away using those. If you want to learn a little bit more about OpenTelemetry and Spring Boot 4, then there is a very very nice blog post OpenTelemetry with Spring Boot. It's on the spring.io blog written by our colleague Moritz. Uh also, if you want just basically you don't want to memorize uh these dependencies because you are not Josh Long, then you can just

go to start.spring.io and start typing OpenTelemetry or Zipkin or whatever. And if you choose the Spring version like the Spring Boot version 3.5 or 4.0, it will give you the right dependencies for that version. So, if you are just typing OpenTelemetry, in case of four, you will get the OpenTelemetry starter. If you use 3.5, you will get those three dependencies that I showed you. I talked about

logging. So, logging support is there are some bits there, but it is not complete because those appenders support for JVM application for Logback and Log4j in OpenTelemetry they are not stable. read the docs, please. I'm not sure if you are aware, but all of the Spring Boot documentation, they are versioned. So, you can see like hey, what should I do with 3.5 or what should I do

with like 4.0? And also, uh there is a Boot 3 4.0 migration guide, which by the way it also exists like boot sort for boot 3.0, but since you are not using boot 2, that doesn't matter for you. Uh that guide is golden. Please, if you want to migrate to No, no. You want to migrate to Spring Boot 4. So, when that happens, please read that guide.

That contains very very vital information for for you and uh like migrating the project. So, let's talk about the observation API, uh which is a very very nice uh like thing in Micrometer if you ask me. Right. So, we talked about uh logging and metrics and tracing. What's this observation API thing? We kind of realized that typical flow that you go through when you want to instrument

your application, you have some operation that you want to time and so you add some logs, maybe a start and an end log or you log how long the thing took. Maybe you add some around how long that takes. You add some tracing so that you can get to the root cause, figure out the cause of latency. But, now you're having to do add logs and metrics

and tracing and so you're doing kind of not the same thing three times, but you're having to repeat this process of instrumenting your application. We thought wouldn't it be nice if there was just one API that you could use to instrument your application and get all of these things out of it? So, that's kind of how the observation API came about. We added this in Micrometer 1.10

boot 3 3.0 and this is in the Micrometer observation module. Micrometer core has a dependency on it. So, if you're using Micrometer core, you already have this on the class path. And so, as I explained, it's this higher level abstraction over metrics and tracing for timing an operation. So, think about if you want to time how long an HTTP request takes to process on a server. That's

something that you could have in metrics a timer for or in tracing you could have a span for that. But, now we have this higher level abstraction, the observation API. You can have an observation for that. So, the idea is to instrument it once with the observation API and then you can add handlers to the uh API configuration and that will add anything that you want. So,

we provide uh out-of-the-box handlers for metrics and tracing, but you could add it for logging and any other purpose that makes sense for you. So, the demo app that we showed, I said the instrumentation is provided by the project, so you don't you won't really see a lot of observation uh or observability code in there. All of that instrumentation is done using this So, what does it

look like? Uh here's some code for using the So, observation.start, you pass an observation name and you pass an observation registry, very similar to a meter registry if you're familiar with the metrics API. And then you after starting it, you do the thing that you're trying to time and then eventually you call observation.stop. And if an exception happens, you can tell the observation that this error So,

that looks pretty similar to what you would use if you were using a a timer or if you're using tracing API and that's intentional because we're trying to we're had in mind that we want to be able to get metrics and tracing out of But, if you just write this previous code and you don't register any handlers, it doesn't do anything. You're starting and stopping and nothing

really comes out of it. So, it's really these handlers that you register that cause anything to happen. And so, as I mentioned, we have metrics we have handlers that we provide for metrics and tracing and you can create other handlers that do other things that So, when the start method is called and the stop method is called or when the error is attached, you can have these

handlers react to those events happening and do something. The metrics handler is just starting a timer on start, ending a timer on stop. Same with tracing, starting a span on start, ending a span on stop. So, the observation handlers, Spring Boot has auto configuration for this and so if you have the right things on your class path, it will just auto configure the metrics handler, the tracing

handler. You don't have to do that. But, if you wanted to provide your own custom handler, you can just register that as a bean as you typically do in a Spring Boot application and Spring Boot will auto configure registering that handler to your observation uh registry. So, some other API that we didn't show observation has the concept of low cardinality key values and high cardinality key values.

So, metrics corresponds to tags metrics hopefully you know should not be using high cardinality tags. So, that's why we have this separation here. So, the observation handlers can decide whether they want to use the low cardinality key values or the high So, the default tracing handler will use both the low and the high cardinality key values. The metrics will only use the low cardinality key values. uh

I guess to explain that a little bit in case anyone's not familiar, low cardinality means that you have a bounded number of items that could be included. High cardinality is something that has unbounded number. So, something like a user ID or a unique ID or some something that you're accepting some value you're accepting from user. User input is inherently uh high cardinality because users could provide anything.

So, you want to make sure that you're not using that in the low cardinality because that'll get added to your metrics and cause a huge uh memory issue performance issue in in production for you. You don't want that. There's also this observe method on the observation. So, this is a a shortcut so you don't have to call those start and stop. You just pass it a method

reference and then we'll take care of calling start, stop and catching any exception and uh marking that as an error on the And then if you don't want to do things programmatically, uh similar to how we have a at timed annotation or uh at new span annotation, we have uh annotations for the observation API as well. So, we have an at observed and at observation key value,

so you can use aspect-oriented programming to achieve the same thing. since we have a little bit more time, do you want to show them the what's new? Yes, let's go there. So, you guys get a little little extra hidden slides here. You actually do a slideshow from hidden slides? Oh, it works. Cool. Right. So, uh Jonathan demoed exemplars. Who thought that was that was pretty cool? Is

anyone using exemplars in production? Exemplars, anyone? Thomas, of course we know. OTLP or Prometheus? Both? So, uh in the demo we showed Prometheus for the metrics back end. Uh we are adding support for exemplars to the OTLP meter registry and that's coming up in boot 4.1 Micrometer 1.17. So, that'll be something new. Yeah, basically next month. Yeah. So, I just showed the observation at observation key value

annotation. That is relatively new. Analogous to the at meter tag if you're using at timed annotation. Uh we added an OTLP metric sender abstraction. So, previously we provided an implementation of how OTLP metrics were sent from the OTLP meter registry to the back end and now we've just made an abstraction so that you could provide your own OTLP metric sender if you want to use a different

HTTP client or if you don't even want to use HTTP at all, you could use GRPC. Allows you that flexibility to implement it how it how you want. We added some additional metrics for virtual threads. Jakarta mail instrumentation. Uh J specify. Who's familiar with J specify? Okay. So, this was kind of a portfolio-wide effort in the Spring portfolio adding nullability annotation support. So, now Micrometer has J

specify uh nullability annotations throughout the This means that you know what is returning null and what accepts null or This is This is very important. If you have never heard about it or if you are not using, please look into J specify and NullAway, which is like a compiler plugin with error-prone so that it will the compiler will yell at you if you are doing something bad

around like nullability instead of users are yelling onto you because they are seeing null pointer exception in production. And then uh we added a meter convention. So, we previously had a a thing called observation convention. So, the idea is that if you're writing instrumentation and particularly if you're a library writing instrumentation, you want to allow your users the option to customize the semantic convention that's used. So,

you might come up with a name, but so, if it's say JVM uh memory metrics, it might be JVM. dot memory. dot max to get the maximum memory that's configured on on the JVM. But, maybe in your organization you have some predefined convention that says that that shouldn't be called JVM. dot memory. max, it should be called java. memory. max. And so, you want to be able

to change this, but you don't want to have to rewrite the whole instrumentation yourself. So, this uh convention concept here is a way to make that more flexible so that a instrumentation can be written and users can provide their own convention or we can have a default implementations of conventions that match something. So, OpenTelemetry defines its own semantic and this this meter convention and observation convention are

the way in Micrometer in which you can swap out the convention and so, you could make an implementation of meter convention or observation convention that matches the OpenTelemetry semantic convention. So, this is useful if you have dashboards that are built expecting a certain naming convention for what things are supposed to be called, then especially if you're working in a system that uses different languages and so, maybe

you have applications that are written in Python or in Node.js and those are obviously not going to be instrumented with Micrometer because it's a Java library, but they're going to have their own metrics and it would be nice if applications written in different languages all called the same uh metric the same thing or the same uh span the same thing. So, this is what the semantic convention

effort is for we've added meter convention and observation convention. And so, those are there. Uh we're working on making it more easy to auto-configure say you want all of the instrumentation to use the OpenTelemetry semantic conventions that are marked stable. It's currently possible to do that, but it takes a little bit of manual configuration. Uh the blog post that Jonathan shared earlier about uh OpenTelemetry in Spring

uh explains this. We're working on making that easier so that maybe it will be just one configuration option to say, "Give me change all of the semantic conventions for all of the different instrumentation to match OpenTelemetry." One more thing with the OpenTelemetry conventions, if you want to like use uh a lot of them are not stable yet. Which means that it will change possibly in the future.

So, that can break you. That can break your dashboards and that can break a bunch of things. We only use or provide uh like OpenTelemetry Sorry, these open observation uh conventions for OpenTelemetry conventions that are stable. Namely, the JVM conventions, they are stable. And also, we have conventions for HTTP clients. Uh other than that, as they stabilize, we are planning to add support for them gradually. Yep.

And then in uh the most recent we used to have an issue because the Prometheus Java client does not allow you to make metrics that have the same name, but a different set of tags. So, say you might have a HTTP server request metric and when there's no exception, you want to not include an exception tag at all. But, then when there is an exception, you want

to include the exception tag with the name of the exception that was thrown. That doesn't work in Prometheus because it expects, "Hey, I saw an exception tag. You should always have an exception tag on this on the same metric." So, that was a restriction that is now going to be lifted in the upcoming uh Micrometer 1.17 because the underlying Prometheus Java client removed that restriction. So, uh

just a little bit of a difference between Prometheus and other other backends. Talk about So, yeah. Uh as we mentioned earlier that there were like uh some modularization in Spring Boot. So, we have now many new modules. That's how like the OpenTelemetry starter receipt and starter is there. And look into the migration guide of Spring Boot 4 because that will contains all of these new modules. Not

just these are uh like uh included, but basically everything. Every single technology that Spring Boot supports now it has a new module and a starter as well. Uh we did like a few Spring AI enhancements uh export messages well. So, now Spring AI is more observable and we are planning to continue this. Also, there are improvements for SSL info and uh health check for SSL. I'm not

sure if you're familiar with that concept, but if you are uh using the info endpoint and if you are also using TLS, you need to enable this feature and it will tell you what are the uh like uh TLS certificates that your application is using, whether that's a server certificate or a client certificate. It will tell you the issuer, the subject, also the validity dates. So, you

will see when something will expire. Uh it also can warn you if something will expire soon. So, you can basically set a threshold that, "Hey, give me like a warning if uh my TLS cert will expire in a week." Because that's what the headroom I need to rotate the certificate. also, there were some changes in SSL bundle metrics, which is like metric support for this SSL info

thing, which can tell you that, "Hey, these many certificates will expire uh I don't know, within a week." And also, there were some enhancements for process info. Uh we added a few fields so that now your application can tell you what it thinks the current time whether that's correct or not. Uh and also like when the application was started uh and so on. You can You can

look into the process info contributor in the docs. It's uh it's it is very very useful in production. All right. And with that, does anyone have any questions? I think we have some for questions. Remember, the code is available online if you want to check it out, run it for yourself. Uh play around with that. Micrometer has a Slack if you want to ask questions later, but

since you're here now, if you have any questions, we'd love to take them. We also have stickers. Spring and Micrometer stickers in exchange for questions. Stickers and if you're doing the trading card game, come see us after talk and we can give you trading cards. Oh, yeah. This is This is something that you can put like next to your desk so that you can you can see

how I look like if you don't upgrade. Remind you to always be on the latest Spring Boot version. Questions, anyone? Mhm. I would say that >> Can you repeat the question first? Yes. So, the question was you've been using Brave for a long time and you were you looked previously into switching to OpenTelemetry for the tracer implementation, but it was missing some features that Brave had. What

does that look like today? Uh I guess my question would be specifically what features were were missing that you were looking for. But, the general answer is that OpenTelemetry is under active development, whereas Brave is more in a a maintenance mode. And so, you know, Brave is very stable and battle-tested in production, but OpenTelemetry is going to continue to add new features. And so, I I would

imagine in general it has caught up in feature parity and has some features that Brave probably doesn't even have. I I guess one feature that like Brave had you might use it is uh like the span continuation thing. Like uh when it it can continue a span without like creating a new one. Do you remember what was the name of it? Uh you mean when the client

and server span on HTTP where it's combining them? But, that as far as I know, OpenTelemetry cannot do this combination of of spans. Brave can. Uh other than that, they are like usually pair in general for like basic Yeah, but I we can talk after if you have a specific features that you're interested in. >> Yeah, I mean uh again, I for specific things, there will of

course be a concrete answer, but in general, I would just say that development. So, if something is missing there, it's more likely that it it will be there eventually if it's not already. What's with the observation API? It doesn't really matter. Like if you have support for something like a certain library that uses the observation API, then it doesn't really matter like what tracing library will be

used under the hood. You will get instrumentation out of the box for that for that component through it. Yeah, that is sort of the point of the facade of using Micrometer, both the observation API and Micrometer tracing is the idea is if you're instrumenting it with that, you it doesn't matter what the underlying tracer is so much because you can swap them out, and you should have

equivalent functionality. One last question, anyone? All right. Well, then thank you for coming to our talk. If you have any questions later, come see us. If you want stickers, cards, let us know. Thank >> [applause]