Building Observable Java Apps that Scale with Quarkus | Thiago Gonzaga (EN)

This talk explores the often-overlooked topic of observability within software development, emphasizing its importance for developers rather than relegating it to infrastructure teams. The speaker shares personal experiences and insights gained while migrating from Spring to Quarkus, highlighting how Quarkus facilitates better observability through OpenTelemetry. Key concepts like traces, spans, logs, and metrics are discussed, with a focus on how they help diagnose performance issues in production environments. The session includes a demo showcasing how to implement observability tools effectively and best practices for using Quarkus in production systems, ending with a detailed explanation of the stack used, including Grafana, Loki, Tempo, and Mimir.

Hey. Okay. Good morning, guys. >> Morning. Oh, are you awake? So, yeah. Um, glad to be here. Um, uh, I've been here like last year. uh but it was at a mentorship hub. So if you guys uh you didn't heard about it, go to the second floor. There are some really awesome folks there giving you career mentoring. So uh if you want to uh so uh actually

this is my first my first time in Europe as a speaker. Uh I've been in Brazil in in US. Uh so I'm really glad that uh that I could make it this year uh here at JCON. Um and today we're going to talk about something that uh people don't don't talk much uh because as a developer we normally treat observability as something for the infra guy and

we are developers so uh yeah so we want to code isn't it? Uh but yeah just just one question uh for those of you who use Quarkus can raise your hands. Oh that's nice. That's really nice. So, uh, actually this presentation came with a really nice product because actually, uh, let's be honest with you guys, this presentation was first first presented in Brazil like last year and

it was in spring. Yeah, I've been working with Spring for quite a long time and uh, and actually since last year. So, we started uh, a migration to Quarkus and then this come along quite well. uh and I'll uh I'll share with you how uh how it came along. so let's start what we're going to talk about. So, uh for those who are not familiar with, let's

talk about a little bit about observability and uh why does it matter for us? Uh let's talk about a bit uh open telemetry. Uh how does it integrate with Quarkus? And of course uh by the end I have a really nice demo for you and uh and I will share the code for you guys that want to start uh doing it yourself. Okay. And of course let's

finish with the best practice in production uh for the ones who are already running Quarkus. Uh glad to hear that. So uh I I'll teach you some some learnings that I had while uh maintaining uh Quarkus applications uh in production. Okay. So yeah uh uh normally uh as a developer so when we um we need to find an issue we normally rely on debugging it aren't aren't

we? So normally what we do have like to go to our ED and open a debug and trying to figure out what's happening but uh whenever we are handling some issues in production uh especially if we don't have the right tools so uh we might face some issues. So I I think that probably most of you have faced some similar thing like uh we see that our

tests are passing. So uh deploy is completed and fine. So nothing wrong with logs and CPU and memory looks fine although our API is still slow. So why is it responding so slow? um and if we don't have like the the right information so it might it might get a hard time for us to understand uh where is the this bottleneck come from. So that's why uh

that's why telemetry or observability is so important because it help us to solve these kinds of problems uh when we don't have uh we cannot reproduce this in our own computer for example. So uh what what does observability stands for? So uh it's all about understanding like what's happening with within our application uh just like by reading the outputs of it by reading the information the application

provide to us without taking to taking a look into the code itself. So and for it we have like the basic three things. So we have the logs. So most of us uh are are familiar with logs. Uh matrix uh for those of you have worked with Prometheus can you raise your hands? Yeah. So metrics are quite familiar for us as well especially when when we talk

about about signos. Yeah. CPU memory uh latency requests DB pool and things like that. and of course traces uh when we want to understand the path that our user is following inside our app and understand what's going on there. So uh these are the three three things that defines uh observability and uh how do we achieve that? So let's talk about about traces. One of now one

of the most okay not one of the most uh I would say underrated uh thing uh in development uh is collecting traces. Um so traces provide us a really nice a really nice information on uh things that is going on inside application. So it provides for example uh what we called spans. So it really it's the footprints uh of our user uh inside the application like how

many time how much time it uses like for example for for responding for getting an API for calling a service for quering uh or in seraring uh in a database uh or for example some IO if we're writing on a file or if we're reading a file or if you're exporting a PDF. So we have all those uh informations here in traces. So uh traces holds some

really important information for us uh when it comes to to telemetry because normally uh it carries the service that we are calling or the controller or uh whatever we are architecture we are using. So we want to call. For the ones who are familiar with uh service and controllers, I normally use service name and controller names. so uh the trace ID so the trace ID normally is

a trace that is unique for for every trace and then the span ID is unique for for each spun and then uh because of the trace ID we can relate spun remember each spun is a footprint uh and we can put some attributes there. So the attributes can be anything that we want. So we can add there for example uh user ID uh username we can put

like for example the HTP method that we want post get we can even get like some uh more fine grain information like the query we are using for for the DB for example the query that is returning for from JPA um we can get like infos for example we can put like uh infos Um let's say uh if we're talking about now we're talking about a much

about AI for example we can we can use like the tokens that we're spending or uh we can infer uh how much it's costing not only for tokens it could be like how much it's costing even for for in terms of selling a product in an e-commerce so we can put a price if if the user uh bought something. So uh the spans are really useful for

for carrying those sorts of things. Uh and the good things normally the the spans they can be correlated with logs. So that's a good the the good part of it. So normally we can carry like all the logs related to that trace. So everything that happened during that period that that user went through our system run for our for our controllers went to our services went to

our repositories. So we can get the logs from that specific time and everything that happens happen an error or a warning or anything like that. And of course we have the duration. That's the most important thing. So we know exactly the time frame on something when it started and it finish. So we know how much time it took. So we can find for example uh nice information

about Bronx inside of database. So uh we talk a little bit about this later. This is really interesting. uh and the metrics uh this is something that normally we don't we don't care much about but uh I'll explain a bit but we have like basically three types of metrics that we we can uh use we have the counter where we literally count things like request errors uh

events have the godge uh like connection Q size memory usage uh whenever things that like goes up and down and we have histogram where we are controlling like distribution like duration like payload size you know and and and things like and of course we have logging. Uh here I put like two examples of logging. So we have one structure in JSON and and a plain log. So

most of us are are more familiar with plain logs which is the logs that we see uh in the console. Um but when when talking about observability uh structure logs uh are really nice uh as well but why I'm saying this okay um let's come like sometime uh before like 10 years 15 years before so uh although we still have quite similar scenario yeah uh in the

past like uh having observability normally is not like something that uh developers care much because normally handled by an agent, isn't it? So if you have like data dog, if you have like New Relic um or or cloudatch, so normally it's an agent running on the on the host. Okay. And then this agent responsible for getting the metrics uh from that. The problem is like uh most

of those guys they have like different patterns. They have like different ways of of sending data. Each of them stores data in a different way. Some of them you're you're not even able to get the data especially if you are like uh data dog and new relic use user. So like migrate data from like different vendors if you're paying like it's kind of hard because the way

they store data is different. Sometimes they don't offer you a way to to migrate. Nowadays it's much better but in the past it was like really difficult to get uh like Datrace Zabics and other tools like to get information whenever we need to change vendors because of the price for example if you're paying like a lot of money for for your matrix. So changing it was like

uh and for tracing are the same. So, Zipkin, uh, Jagger, uh, and although the Prometheus guy that we, uh, we talked about, uh, that for for the open source community is like like well known. Uh, so yeah, most of them didn't have like a real good pattern to follow. So that's why in this case, open telemetry stands for because actually open telemetry is a cloud native foundation

um, graduated project. So it's an open source uh it's actually it's not an open source project itself. It's an open source standard. So it's a well-known standard defined on uh how you instrument like logs traces and metrics. Uh and the good thing is like they have SDKs for like every language available. So we have for Java, we have for lang for Golang, for JavaScript.NET net and a

lot of uh other uh tools. Uh we still have uh auto instrumentation. So uh we still have the ability to simply no code simply put an agent or simply uh add a library to our uh software and with no code it can start getting at least the basic signos the basic matrix and traces and things uh introduces a single collector. This is one of the most important

things. So normally we used to have like different collectors for different things. One for matrix, one for for logs, one for uh traces and now we can do with a single collector uh the one we call open telemetry collector. Uh and of course because because open telemetry is a standard. So normally the company the companies follow. So now with the open telemetry you can export your data

for any vendor that you want. So and just a matter of a configuration you can export your metrics to new relic to data dog to um data trace you can export to Prometheus you can export to Mimir to whatever you want. So you own your data and you send whenever you want and because it's it follows the same pattern, the same standard, so it's easy for you

to migrate between vendors. So we don't have like vendor lockins anymore in this case. Uh but some of you might think like why am I changing uh to open telemetry if I have Prometheus. Yeah, I think most of us especially handling with Java. So we have the the the slashmatrix which is exposed by the library. So uh where most most of the metrics are. Um but the

point is um when using Prometheus for for example it's quite hard for us to get some uh specific metrics that some custom metrics. So it has it's really good for for the metrics that are there. Uh and another point is normally Primeas does the script for you. So you have to install an agent on the same machine and and this is responsible for querying your your slashmetric

API every 10 seconds, 15 seconds, 30 seconds or or whatever to keep reading the metrics. Uh and another another point of when we talk about Prometheus is normally Prometheus have a limit especially in terms of scalability. So one of the main purpose of this uh is building like applications that scale and when we want to scale we want our metrics we want our observability stacks to scale

with us uh and because Prometers runs on a single node so it can it can reach like limits quite easily and it can concur of course in memory and CPU with your application inside the inside the machine. Uh and when we talk about open telemetry in in this case it's the opposite because actually the agent is not running on your machine. So the agent can can run

outside and what it does your application sends the data uh in batches uh to the collector. So it's not a collector who reads the data. So your application keeps sending the data the logs the metricing traces to the collector which resides outside and the collector can be distributed. So you can run literally a cluster of collectors. So uh you can easily scale uh your collectors to receive

as many data as uh although we we do have the same way for doing with primitives. Prime supports like what you call remote write. So if you we if you want to send it uh to a different place uh you can but uh there is actually not an easy way for parameters for example to handle like uh what we called uh short-term metrics especially when we're dealing

with microservices and cloud native. So what happen is uh if if your application lives true for short for like it runs like for a really a really small period of time and then dies. So what happened is whenever another application comes up so your your parameters we we start scraping a new application we have a new ID and have a lot of new information and what happen

is when you start seeing in your in your dashboard it doesn't look like a single application it will look like a lot of a huge amount of uh applications uh with like messed up data in your dashboard because we'll see like gaps because one application was up sends some metrics and then it here down then it spin up another another pod or another container and then it

starts another different point so it's not continuous and because uh because of the nature of the open telemetry because you can you can uh tie with your with your IP ID with your specific uh control so whenever your application goes up and down it it goes continuously in your in your dashboard for example because it doesn't have to treat differently uh the containers that is goes out

goes uh back and forth. So this is one of the good things when we are handling uh uh with open telemetry. although yeah something that that we might think is like uh both of them are are kind of complex because we need to handle of course the amount of data we ingest because of course as uh as much data we ingest we start getting uh more CPU

and more memory because we require more uh more processing. So we have the same the same thing with the collector uh because imagine that the collector acts like the same as per the difference is running on on on a different place. Uh but you have to plan well uh the metrics that you are sending there. So the the difference here is like the complexity change place. So

instead of like running locally alongside your application now it stands on a on outside. It could be running on the same cluster or even it could be done on on cloud if you want to. But uh the complexity is basically the same. Uh but some people some people tell me like what about the overhead because actually we are we are consuming like uh our uh our API

resources because we are sending data outside. The good thing is like uh when we talk we're we're talking about metrics we we're dealing with just a little amount of data like it's just like small chunks of protobuff um so it's not actually text and most of them are are done into into batches. Uh and the good thing like normally the the the size of the buffer is

fixed. So whenever for example if your collector goes down uh and if you start uh and you stop sending metrics so the old metrics will get dropped so it won't uh reach uh limits in terms of buffer because it will like clean up the old ones until your your your collector goes up again. So um so we don't need to expose anything. This is one of the

most interesting thing because we don't we don't need to expose like the lash metrics. I think for us who are handling um microservices uh we need to care we need to care about exposing metrics because um I don't know if you have ever faced this issue like when you are exposing an API uh if you don't control well what happens is like normally your slash health and

your slashmetrix goes goes public as well and and we don't want our our metrics to go public we want to expose only the API endpoints So if you don't control properly uh our endpoints so we might ended up that the slashmetrix goes uh public as well and in this case we're not exposing anything because it's our application which is sending data to outside it's not we are

not receiving any request to get the slashmetrix. So uh we don't we so we decrease the what would you say the the attack surface uh in this situation. So and for for this uh for this project what I did so uh I use this stack we we called uh LGTM it's not looks good to me okay uh in this case it stands for lock graphana tempo and

mimir so it's a stack uh by graphana labs so it's it's 100% open source so you can run self-hosted so you you you own uh your data so you don't need pay any licenses. So you don't it's production proof. So have like tons of big companies uh running uh their observability stack on on the on this. So this is something that uh this is very good. So

and what we do have here so um we normally know Graphana because of the dashboards aren't we? But it offers like a wide range of products. Uh so one of them is lock uh lock uh is responsible for uh storing logs in an efficient way. So the idea is like we store the logs somewhere with a cheap uh uh storage back end. In this case, it could

be like your disk or you could you could rely on on even cheaper things like S3 or um or other uh service which like it's cheaper than you know having EBS or high high throughput volumes which are really really expensive. Uh and the good thing is like different from from elastic search if you're handling for example logs with elastic search. So last search uh you have indexes

and indexes consumes uh your disk as well. So you have to plan plan well the index in this case because uh lock key is quite similar to uh to what Prometheus does. So it all stores using labels. So it doesn't need indexes for for that. So you can query uh using uh labels in a more efficient way and you can store uh so your data will use

like much less disk space and of course they use uh a language called loql. It's quite similar to promql which is the language for for promeus from for quering. So for the ones familiar with Prometheus, you will see that this stack will be quite familiar for us as well because uh all those standards uh all those those tools they are they are based on on on Prometheus

at least the standards of Prometo. So it will be quite uh easy for us to to see. This is just a query would look like. So querying for service name and trying to look for uh for errors. So uh we have matrix. So for matrix we use uh graphana mimir uh and graphana mimir is one of the most awesome tools that we have available today because actually

for the ones who know graph mimir is a primeus server. It implements actually the Prometheus but the good thing of mimir is it it has like all the good things about Prometheus uh but it can be highly distributed so it can store on a long term the problem remember that we discussed about Prometers the problem of permits like it's quite hard to store data because as much

it's getting consumed by by disk it's lower and slower and in this case so it can be like distributed uh you can use uh we can use a backend storage a backend storage again we can use S3 for example we're using our own disk um we have like two options for deployment we can deploy like standalone we can deploy like what we call a monolith or we

can deploy uh as a micros service so we it can create like a cluster with uh ingesters with uh uh uh query indexes and and everything to make things like faster uh and in more cloud native way uh and if you are replacing from prometers basically has no friction because it's it's a permitus and the query language of course is proql which is the query language for

for prometers as well. So if you are familiar with forters like this forgetting rate uh of user created and then we have traces. So uh the graphana temple uh basically uh follow the same pattern for lock and mimir for storing traces. So it can store trace traces in a in a very efficient way also using using the same pattern using labels. uh we can use uh also

S3 or min.io orever back end we can which is cheaper for for restoring long-term data and the good things like uh temple also supports other uh other ingesters. So we can ingest from Jagger zip king natively. So you don't need to ingest only for open telemetry. You can just for other um for other things as well. And of course uh we have like the error the rate

duration and we have traceql again quite similar to promql uh because it's like based on that as well which is quite easy to to to query from whenever you want and of course to connect everything we have graphana. So graphana literally can query uh all the data in all of those uh three services and the good things like uh because of the they work really nice all

together you can also correlate the information between them. So you can correlate your metrics, your logs and your traces which makes this easy for when you are seeing a log go straight to the the through the trace uh and correlate with uh what happened on that time in terms of metrics. Uh and of course you have like a wide range of uh of dashboards that you can

create. you have like the marketplace who offers uh dashboards ready uh for you if you don't want to write yourself but you you you can also create uh your of course 100% open source so uh no vendor locking it's still it's compatible like with like over 100 uh connectors so it not only connect with mirror lock and tempo you can connect with primeas you can connect with

cloudatch lask search so it has connectors for basically uh everything in the it. So, uh, Grafana is also, uh, an easy choice. Yeah. Uh, and of course, I would say the most important thing or the most important part when you talk about open telemetry is of course the Uh, and in this case uh for the collector also we use graphana alloy which implements the standards of uh,

open telemetry collector. We do have the open telemetry collector from the open telemetry. So you can download the open telemetry collector that and use uh or you can use graphon alloy which is also a prime scraper. So uh on a single collector so you can also receive data from um from all uh your endpoints open telemetry endpoints but you can also if you still rely on like

old Prometheus endpoints. So you can use alloy of course to uh scrape your data uh into your old matrix endpoint which like uh uh quite easy. So in this case alloy works well in this two cases and as I mentioned as alloy is a scraper as well uh it can only scrape for other stuffs that you want. It can even scrape files in your in your file

system. For example if your logs are not being exported as uh open telemetry logs. So you can directly read the logs file into a system and then send it uh somewhere else. And of course it has like a different tones of exporters. Uh again it can send data to uh to lock me and tempo but also it can send data to to data dog to new relic

and a tons of different uh vendors if you want to. So it's not locked in to any vendor again. So you can uh uh you can use also a it supports multiple protocols. So uh even though if you are not using open telemetry so you can use like permitus protocol you can use uh like the vendor protocols specifically for like data dog new relic uh zip king

jagger and other protocols uh which are not not standard so uh alloy supports it as well and the good thing this is one of the most the most uh interesting features of the collector that's why we had now the standards it because like uh we know exactly the the sort of data we are reading. So we can transform our data easily and send it in a format

that we want. We can use uh the collector to drop labels that we don't want. We can uh use the collector to transform the information that we're receiving into a different information that we want to to store somewhere else in a different uh format. Um we can also uh we can also set how how are we sending the data outside so we can control to avoid like

uh going um on memory issues or memory leaks. So we can control the bat sizes, the queue sizes and we can control everything to make sure that uh we we are not u overload our system. So uh again uh a really interesting choice. Uh yeah in this case we comes to this question. So why quirkus? Uh I think most of us uh I've been I've been working

with spring for over a decade. Actually it's I started working with spring like almost 15 years ago. and I think that most of us uh have the problem with uh dependency hell isn't it? So, Spring relies a lot on a huge amount of dependency and this is not this is not this is not Spring Boot's fault. This is uh this is like is a spring framework's fault.

This is why the spring frameworks were uh written for. So uh but that was like you know uh the tool that we have by that time to start writing and nowadays we still see uh companies is struggling uh using uh spring uh but now that we have we are in a cloud native environment so if you're running kubernetes for running docker so um if you're not running

on uh any environment that relies on uh a web So uh it's quite hard uh to to manage uh and we know that because we are sharing we are sharing resource we're running a Kubernetes or docker environment we are sharing resources with other tools as well with other applications so we are not uh using the the server for our own so um because quarkos is like uh

what we call like subatomic java because it supports like a microprofile IO it's like it's really uh fast to start up. So it it's a low consumer uh of memory. It requires like a just a little amount of of dependencies. So if you can compare the dependencies of course it is ground native. Uh so you can compile into binaries. Uh so it supports live reload. So this

is something that uh I don't know if you guys use but it's kind it's kind of useful. we saw in other language like Python node uh live reload and it's uh quark supports it and the most good things of it that we don't have unfortunately in spring what we called the the zero efort uh for uh for matrix uh although we we still can do it with

spring but with spring requires an external agent so you have to have like the open telemetry agent running alongside your jar if you want to have like the same thing that Quarku has with basically no dependency. So for Quarkus is just like a library. So that's why uh we can have we can literally have like uh half of our job done in in terms of seconds. So

uh like just just for this project what I use so I'm using Quarkus 3 uh uh I'm still using Java 21 just because of of compatibility. I see some some people still not migrated but uh the application do run on Java 25 also um hibernate with panic uh uh we use my SQL for production in in at at my company uh h H for for dev and

test and this one is just like a simple uh crude for like uh for users. So uh I have here the the code if you guys want to to download and play with it. So the the all the environments that you need it's in a docker compose so it's just a docker compose app uh so you can you have like everything mimir graphana at tempo everything that

you need to start the application uh and also the configuration that you need uh the start configuration for running an alloy I I also have the the dashboards set for you so uh you can have the basic dashboards for for the application as well so just go uh and download the the So uh so we starting this so if you are already using Quarkus so this is

actually the only dependency that you need. So just quarkus open telemetry. So uh of course I'm I'm using Maven. I'm still a node school guy. Probably most of you are are already in gradu or something. Uh but it's basically just just a matter of a single dependency. Uh and one one thing is like the matrix and logs are not enabled by default only traces. So uh in

this case you have to to add uh in your property files the enable logs and and metrics just to make sure that you are also having those uh and again this because because of we don't need any agent to run uh open telemetry using quarkus so this is fully uh growvm compatible so you can build your uh native image uh from it with no actually uh external

internal extend or jar jar dependencies. Uh this is uh what a configuration looks like. Uh so um one of the things that I normally suggest of course is like um setting at least the the some initial attributes like your app version, your environment. Uh, of course you can you can make it like uh dynamic if you are using um if you're using an automated pipeline. Uh but

uh it's a good practice to have to have those. Um as I mentioned the first the first one you don't need because actually uh it's already enabled. Uh trace is also enabled. So it doesn't actually mric and logs but I just put there uh so you guys can see uh what it look like look looks like and another thing that uh the telemetry for uh JDBC for

the database is not enabled by default. Uh so you have to uh enable telemetry for for JDBC as well. So you can get the information from the database and the queries and things like that. And you just need to point uh your open telemetry endpoint to your alloy. So it's where you are sending the data to. So and then alloy handles sending to uh all over the

So uh by default so which sort of information that we have so uh in terms of traces so we have traces for for basically every HP request. So we know the method the path the status the duration. uh if we enable the JDBC so we will have the JDBC queries the SQL that we are using the table name the duration of each call uh if we're using

uh gRPC also we have the service the method and uh for Kafka uh we have also uh publish and consume uh traces for panic we have the hibernate uh operations uh and formatics where we already have the heap garbage collector threads the class loading And um also we have like HP request rates, error rate, latency for metrics and for JDBC we have post, connection, weight, time, uh

Q uh and and things like that and logs. Um it's whatever formatable that we want. Uh in this case we can we can use like use text or we can use like the structure one that we discussed earlier which will be much easier to parse with. I'll explain you guys later. But basically uh this is what we get only by using that dependency. So we don't need

to do anything else. Basically no code. We don't need to do anything else. That's it. And we have like a pretty observable app already. Uh but of course uh this this talk actually is not about the ones that we get for free. Yeah. Uh so as a developer now it's becoming more important now that developers also own the metrics that are being stored and but I mean

metrics I mean the the metrics that are not provided by by these guys sometimes our company relies on other metrics which only our application knows and uh and an external agent won't be able to get that for us. Um oh this is the and that's why uh open telemetry has its own SDK which comes along with with the dependency where we can inject our own spans and

our own metrics there just like we do for logs. So we are so used already for injecting logs in our application. Yeah. Um but now we have to think more because only ma only logs are not enough. We need more information. We don't want to debug our application every time something goes wrong. We want to understand what's going from outside without looking at the application. So in

this case uh we can for example if you want to add some more information related to traces. So the uh we have the width span so where we can provide uh in this case I'm using the service uh the service method uh and we inject like we use like span set attribute the the same way we are using for example for logs you see logs info fetching

now we are just adding a new information like set attributes in this case we're adding what we are counting the number of users that are being registered in our system. This is an this is a new information we don't we don't have it by default anywhere else. So knowing the number of users we are adding or the number of products we are selling or uh the amount

the amount of products that we have in our cart and those are useful information. and of course we can also we can also set the attributes uh if you want to grab the attributes directly from uh from a variable. So you can use span attribute and map a variable or a parameter through that. it's just as simple as we used to to do with logs. Uh why

this is so important? Because uh imagine imagine like that you are selling and at some point at some point uh the user literally lost his entire cart. Uh and how do you know how do we know that? like simply by by looking at the metric simply by looking at the the the logs we don't know that our that the user lost our cart their cart it would

open a support ticket and it probably will come to you and they will try to reproduce that issue and how do you know that you don't know where where did that that happened so that's why now we can inject also the metrics there we can inject the the traces so we can know exactly for example the time that the user lost its cart losses products in in

the cart. We can actually add information to to see if the user completed or not the purchase. So uh we can now add more value to the metrics that we we are uh seeing and then we can have like more rich dashboards. So now thinking when we're designing your apps. So now we have to think also about the metrics you will you want to collect the metrics

that are important to your product to your company. Uh and this is actually just one one example. E-commerce is like wide used. So uh but imagine now for example if you're handling uh a tons of uh requests for LLM and of course we have to control the number of tokens we are consuming although some APIs already offer you Uh so you can also infer other informations there

as well. You can infer for example the uh the price you can estimate the costs of of something. Uh so these are are metrics that are not there by default and you you can output there if you want if we want to. Uh also we can we can for example uh uh uh and the good use case that that that we have. So, uh we want to

control the amount of uh the amount of memory and CPU uh our application is consuming but we want to correlate it with the the cost of the the cloud. Remember that currently uh we we only have the CPU and memory. We can have the CPU and memory of the container itself or we can have the CPU and memory the host itself. But from inside the app, we

can get more fine grain information and we can also infer like how much your app exactly costs instead of like relying on an approximation of what is outside. So imagine that you are running thousands of containers. You can see that okay so my containers so the containers that I own I know that their cost exactly so you are so if you are spending like this is a

really good this is a really good good thing if you are like uh handling with your with your upper management and they are saying that your application is is waste a lot of a lot of money say okay your cloud cost is this but my application just cost this I have all the metrics there so uh I can show you that how how how much each request

costs. This is something that you can get from it that you you can insert this data in your app uh that normally you don't have. So this something that now I do. So uh I estimate the cost of each request because now I know uh the how how long does it take for conclude like for example a DB request. So I know how much CPU it consumes

for that request. So I know how much uh it costs because we know the the price of the the time for for your instance for your Asia or the AWS is open. So you can uh there's even an API also you can also query the API and then uh do the math in your app uh pretty easily and then you can say okay so so this request

costs this amount of money. So if you are having like so we know so this is a like a new level of fine grain information that you you cannot find anywhere else in in terms of uh product and you're now with open telemetry you can do it in your app the way you want the the informations that you want you can put like any sort of information

that you want and of course uh and in this case we're just talking about about traces we are only talking about uh the the user path. Okay, because also uh in this case is just for injecting uh errors in traces is the same. So we have this error types. So uh whenever we run an exempt an exception so we can also add uh an attribute for uh

errors as well in our traces. But we can also rely on custom metrics. We can inject metrics also into our system. Uh so imagine uh so we can control for example the the requests we know this is one of the metrics that it comes out of the box so we know the request is going from a specific endpoint yeah so uh we don't need to so but

imagine that I want something more specific imagine that I I I want to see for example the amount of users I create or it's being created uh in this app. The amount of errors is being generated or the amount of user updates or the amount uh of the user changes. the we can even get some specific errors that we don't get for example uh in a regular

in a regular path or request metric because normally on a on the path we only get the the error code or the succeed code 200 4001. Now I can put more information on the metrics as well. I can count for example the number of duplications, the number of users who trying to add uh trying to to subscribe to my to my uh system using the same email

of someone Imagine that we can also even have even detected for example some sort of attacks just because we have diff we have more fine grain metrics. if someone is trying to uh to log with a specific user. So we can see that the ma matrix for a specific user if we want now. So instead of like having the whole uh the whole vision of the the

a the entire API we can have like per user the same with the same level of information. So uh we can even see like if if someone is trying uh to log into uh your username uh for brute force because you can count the number of uh uh of logging someone is trying with your username for example you can you can add this as a custom metric.

Uh and this one I just we just use like counter. You can also use like god and histograms if you want like you know uh more fine grained uh for example uh again we can uh the traces can give uh can give us information like for latency or search inside the database but if if you want to put it into in a nice dashboard for example if

you want to expose a nice dashboard we need to to expose of course the metrics for that as as well. So in this case for example I want to uh to have a metric specific for for users user search uh and then we can find the P99 latency for example. So we can uh add it like Uh and of course the last but not least uh important

is uh logging. By default, open telemetry has this uh you can add the format you want for the console and the format it will be using to to send to your uh collector. So in this case you can define the same log format that you would normally are normally uh familiar with which will be logged um and we can also control whenever we want to log to

our console or not. In case for example for we don't need actually to log for uh more in in production because we're already sending the logs somewhere else. So we don't need to have the the logs there. So if you want to we can simply disable uh we can define if you want the the the log format for JSON uh and things like that. So this is

what loy uh sees. So Loi can can can see the trace ID and the span ID. So it can correlate with traces So now uh see that now I have much more Now when we are trying to figure out why our API is low and we do not see any problems at all. So now we can go like to mimir see the latency to see if we

have any spikes uh in that endpoint. We can look for for temple to see if the database requests are taking longer than expected. Uh we can dig into a specific user if if see that specific user is locking for example the database in a and then we can finally with all the correlation we can finally go to the the exact point of where it happens. In this

case for example imagine that we we find that the users table doesn't have an index on email. we're discussing the latency for example. Uh so now we have like the whole picture of what's going on on our application and now we don't need to even need to debug. So with only the metrics we're able to see and find where the the issue is without having to trying

to reproduce in our own local machine and trying to figure out what's going on. Okay. Uh let me just let's just show you uh what we do have here. So So you see that that let me just my pawn XML is a little bit bigger because I I added a lot of stuff especially related to formatting and and and things like uh just because I've been working

with devops for quite a long time. So we we get used to make sure that our apps are formatted. We have some linting and things like that. But let's just just go into the code. I'll show you uh what it looks like. Oops. So you see as as just as I said I have like the display attributes and by by uh convention I normally use the same

method name but you can use actually whatever you want. I'm just using the the class name and the the method that we are we are using. Again we have like the uh the set but let's just let me just show you how does it work. I'm just go uh put all the the information up. If you check if you if you had had time to check the

the the source code, I put there on the snippet for you guys just uh showcasing uh Java 25. You'll see that there is a guy outside called low test java. Uh low test on Java is a singalone uh file uh on java 25 which runs without need to be compiled. So just like a script. So I just have like the uh the the bash instructions to run

and we can run this script without even compiling. I'll show you guys in a minute. Let me just see if if it's up Yes, please. uh in this case no uh the attributes that you expose is the attribute that that will be showing. So you you can control actually this at the at the permissions the permission level on locking mir and uh on traces and in graph

itself as well. So you can define the levels there uh >> uh yes in in this case you can but but because you can you can control the the label. >> Okay. Okay guys, it seems that I'm over time. Uh but okay. So it's uh just for you uh thank you for coming. Uh if you have have some time try it. It's just a docker compost app.

the graphana it's already in the it's already in a local host. So you have like a graphunnel like this. You have all the dashboards already uh there for you as well. You can use the load test file uh as as I just mentioned I can use like just called low test without even compiling. You see load test on Java and then uh it starts sending data and

then you can see there. But that's it guys. Unfortunately I don't have more time. Thank you.