Harry Kimpel: Tracing Kafka with OpenTelemetry

In this talk, Harry Kimple discusses the importance of tracing Kafka applications using OpenTelemetry. He explains how OpenTelemetry provides a standardized and vendor-agnostic approach to collecting telemetry data, including metrics, logs, and traces, which is essential for monitoring performance in distributed systems. The speaker highlights the ease of integrating OpenTelemetry with existing Kafka setups, noting that it helps to eliminate the black box nature of Kafka by providing detailed visibility into message flow and processing times. Kimple also compares manual versus auto-instrumentation approaches, explaining the trade-offs in depth and performance. He emphasizes the significant benefits of using OpenTelemetry for observability in cloud-native environments, predicting widespread adoption in the near future.

[Music] ladies and Gentlemen please welcome our next speaker Harry kimple presenting the topic tracing cough c with open Telemetry on the yes this side of the screen please do use slider to ask questions if you have any during the uh con the presentation after the presentation 45 minutes take some like 7 five minutes as much as you can for questions you can ask questions live afterwards or

again write them down you can also find the speaker outside of the uh presentation Hall afterwards and talk him and hit him up with questions or just chitchat so go ahead thank you hello everyone hello vs um I will talk about let you know as it was introduced on tracing Kafka with open Telemetry um I mention why this is important and you know what it what benefits

it it'll bring um you when you apply this um in your day-to-day job uh Safe Harbor slide I won't go into that um my name is Harry kimple I work for a company called neur Relic uh which is an observability tool you it's just a hype topic nowadays you know previously you know monitoring you know observing your systems um I may developer relations engineer at Nik have

been in a pre-sale um so capacity before that um I'm living in south of Germany and I'm you know excited to be here today so I represent um you know neur Relic and I work for neur Relic but all of the concepts that I'm sharing today are not related to neur Relic at all I will be using the Relic as a back end for the for the

Telemetry that I'm collecting um but basically you can apply the same concept in all of your environments independent of of whether you're using the Relic or not or using some open source component components so I hope this is um interesting and you know like I said it will apply to all the um the concepts um not specifically to neic so um I think I have the feeling

I don't need to talk about Kafka too much you know it isn't you know open source distributed events streaming solution it has a lot of benefits is an open source environment right you know high performance scalable you know persistent storage um so really interesting um topic there are some vendors out there that actually use and provide this as a managed Service as well like you know things

like confluent so this is around cka and the challenge is typically if you're running an application you have some kind of con um producer some um consumer on the other side that actually you know exchange messages over Kafka you know in general if you don't apply some Concepts that I'm sharing here today um you will basically you know see a broken connection right you will see like

the producer producing messages to Consumer consuming but if you want to trace all of the um requests through Kafka and you know some additional benefit that it brings it'll basically look as a blackbox so it's really important to apply um some some tracing in these applications so you can get a full endtoend perspective of your entire um application that is maybe composed of multiple microservices so that

brings me to um open Telemetry who in the audience has heard about open Telemetry yet okay so I would say good half of the um people it isn't really interesting um topic like I said I'm coming from a company called noric um we used to be in the APM application performance monitoring space for many many years um we had our own you know agents that you couldn't

attach to your applications but really um the benefits of open Telemetry is you know this is an open source um project it is part of the cncf it's an incubating project within community um it was actually formed through um two different projects that um existed prior to open Telemetry one is open tracing that was focusing on the traces the tracing aspect and then open senses which focused

on the the metrics so typically the typical Telemetry that you will um see is kind of like traces metrics and logs um that you want to collect um and open Telemetry covers all of these Telemetry data types in a in a single solution it's vendor agnostic it has some set of apis sdks that you can use and apply um you know it basically standardizes the way you

collect and you know observe your applications in a really efficient manner and then you can use that Telemetry that you collect from your applications your infrastructure to send it to another open source environment things like Jagger um you know or some other um environments or like a full observability platform for example like New Relic so really um you know why this um was introduced you know um

the industry wanted a a standardized way on having a vandor neutral um way of collecting the data um interop inter oper I can't pronounce interoperable um you know kind of like you know connecting many pieces together in a standardized way and form and highly configurable I will go into the details of why this is important where you can apply um these customizations but this is really the

benefit and you know the the statistics down there mention that you know gner predicts that by 2025 which isn't too far out um from from um today they expect that 70% of all you know Cloud native environments will be using open Telemetry in some way or form so like I mentioned there's a lot of benefits you know for various um people for various personas that you typically

deal with you know for management you know it's a standardized way you implement once and you know um don't need to um think about like any vendor login or anything um you know there's some really um interesting um benefits there for me I think the most important thing is really the render neutral aspect you know open Telemetry is here to stay um for for a long time

and you know if you using some kind of um observability of monitoring of your applications today you know it may be interesting to um you know think about switching to open Telemetry but if you haven't started um like monitoring your applications for performance um and exceptions that occur I highly encourage you to look into um open Telemetry because this is really the the future of um you

know observing your applications infrastructure um the concept of collecting your um data from your applications and services is kind of like that you will use some kind of pipeline that you can configure and what you see here on the screen is you know really just um a you know typical snapshot of a open Telemetry implementation you can see on the left hand side you know you can

use some um applications some languages if your applications are written in you know java.net whatever languages you use there are some ways on how to you know So-Cal instrument your applications like observing your metrics traces and loog applications you can then send this to into this pipeline um and I'll explain what the benefits of these pipelines are um another aspect of you know sending data into that

pipeline can come from your Cloud environments from your infrastructure from other services like the hyperscalers like AWS Azure Microsoft is also heavily invested in open Telemetry so if you're using some kind of services it could be even you know serverless functions that you use typically open Telemetry is already baked in and you can also use that to um you know add additional information to environment so like

I said the first thing is to collect the data and this is where the so-called receiver um comes into play it'll receive um your data your Telemetry this whole pipeline that you see here that can that can run locally in your own environment like if you have you know some kubernetes environment some local infrastructure that whole pipeline can run in your own premises this does not need

to run in some kind of um Cloud environment the middle piece the processing part is interesting because you know it helps you to for example batch your um traces your metrics and logs so that you don't you know send the data you know immediately to the to a back end you can batch it you can process it you can decorate the information with additional information you can

opos skate data if you identify anything things that you don't want to capture in some kind of observability back end you can also upate that data or also remove data if someone for example um you know decided to send in um some debug logs you can identify those and you know um remove them already in this pipeline um of the open Telemetry um environment and then lastly

the exporter is then you know a component that helps you to send the data to some back end that you're using in order to analyze the data the traces that you typically capture you know are nice you can look at them in maybe a console output but really in order to help you analyze the data what is going on see all of the traces in context and

maybe you know identify any anomalies that are happening you want to export the data and you can use various kinds of you know exporting backends like I said you can use Jagger for example for traces you can use some you know lock management system to send the locks and it the important thing is also that it can it it doesn't need to be like one environment that

you send the data to you can send the logs over here you can send the traces somewhere else so you have full control of what is happening in this in this Pipeline and that is really in my opinion the biggest um you know benefit of having such a pipeline because it helps you to Define um and process all of the data that you collect and of course

you know we as neic we also have like an end point in so-called OTP endpoint open Telemetry line protocol that receives native open telemetry data and then you know stores it and you know does some um interesting aspects on how to visualize and an help data all right um but you know you know let's just all go into open Telemetry right or is there anything you know

some design considerations or some things that you may want to um look at and this is what I'll um quickly talk to in the next couple um um sections so you can imagine you know open Dy also um supports basically two different ways on how you can instrument your applications on the left hand side you see kind of like your ingredients like if you want to bake

a cake you can use the different powders and stuff like that right you have full control over what is happening and what you want to observe within your applications which is nice because I can decide you know maybe which pieces in my application need some you know deeper instrumentation maybe I need to get some details into you know some more important aspects of my application for example

if you think about like a retail um environment the the card and the checkout is probably one of the biggest uh and most important um aspects but you can decide what what it is you want to um instrument and observe on the other hand open Tel also allows you to have some kind of um you know like an um Auto instrumentation um which is it called and

you can use that to um you know basically just attach some form of agent to your applications and and then it'll automatically instrument your application and try to identify things that of that are of interest but in an automated fashion the car the you know the pros and cons are you know on the left hand side you have full control but you also have to um you

know add something into your code so you will bake in some dependencies you will you know change your code to maybe reflect some you know custom spans that you create some custom data you know on the other side you can just attach an agent but it only gives you that much level of detail right so it's kind of like a balance and you can pick and choose

as well but um these are the you know typically the designed considerations and what I'll show you in the next couple minutes is kind of like you know how this actually looks like and how this you know could um appear in applications um what I always um encourage you to look at as well is if you go to open Telemetry doio um there's like a status and

releases um page in the documentation this highlights the maturity of the different environments like if you're using java.net these are you know probably the most mature um environments depending upon what is what it is you're using you know based on the Telemetry like traces metrics and logs um you will have you know full support already and you can use it also recommended in in production already but

in some other languages um you know I'm thinking like rust for example this is basically uh in beta Alpha stage whatever but I highly encourage you to look at that page on a regular basis because this changes frequently this is a snapshot I took last year in in the summer I just went in today um to gather like an updated snapshot and you know you can see

a lot more support a lot more is happening you know a lot of um things that are in stable um configuration now PHP went into stable for all the different Telemetry types which is kind of kind of interesting so always highly encourage you to look at that before you start um implementing all right um you know design considerations you know I won't go into details of that

specific slide you can um um read through but basically do you already use traces in your applications if not I see a lot of um customers um and environments that are heavily relying on logs you know maybe historical reason you know things that to to consider when when starting with Telemetry you know what is it that you want to observe what type of telemetry do you want

to collect and these kind of things are highly um you know of interest when you start your journey in Telemetry but it is like I said earlier it is here to stay and you know worth looking into all right so let's get into the details how this could look like in your application what I um am showing in a sample application in that um specific scenario I'm

using a Java spring boot um environment ment the concepts really apply in all the different languages in a similar fashion the syntax may be a little bit different but basically the concept is really the same in all the different programming languages so what I'll first go through is kind of like you know remember that that cake um I'll start on with the auto instrumentation so basically just

using um an agent which you can see here um this is the like a you know a run script that will you know start up my application and what I'm doing here is kind of like exposing some um environment variables in this case the dash Java agent um you know adds um to your jvm um um options and what you see here in the in the back

of it is kind of like the a jar file that I'm just attaching to the jvm at startup time this is basically it what I need to do in order to you know get my application up and running using open Telemetry I don't need to change my source code I need to change anything in my my application itself I'll add this um Java agent and then of

of course you know I need to send the data somewhere you know remember that pipeline that I showed earlier in this case I'm using this OT LP open Telemetry line protocol endpoint like I said this applies in in all the different observability concepts in this case I'm using our neic um endpoint for OTP for open Telemetry um Telemetry and then you know since this is an endpoint

I need to attach some kind of um Header information and API key in order to send the data and you know for the back end to accept that request coming in this is basically it um I can start up my application I can start up my um sample like a producer a consumer and some other services and you know without anything you know um that I need

to configure in the application I will get then insights into you know how that application um looks this is kind of like a a screenshot um from our environment but basically this is the the high level architecture that I got um generated automatically you can see on the left hand side the producer you know the consumer I have added some additional um things you know you know

to the consumer just to show some some additional more interesting information but basically Kafka sits um in between right so the producer sends in some data into Kafka the consumer you know subscribes to it and reads the data from that Kafka topic if I then look into like the tracing and again this could be some you know open source component like Jagger in this case this is

the um standard view for distributed racing in The Relic I will then see you know the the breakdown kind of like that waterfall view of you know what is a single request look like this is what we're looking at is a single request you can see the trace ID at the top as well but basically I can see you know the producer sending in some data into

Kafka I see a huge block of you know you know empty space this is kind of like the lag this could be the lag of the consumer reading from the Kafka topic and at some point you know the consumer kicks in and reads the data and then my Trace um follows through this is great insights already right I mean I didn't have to change my application but

I can see a full Trace all the way from the beginning like the producer to even like some some backend service that I added as well but I see also you know not Kafka as a blackbox but I can see at least you know hey there's something happening kafa sits in between and basically process it that day and so I can you know easily see you know

how that breaks down I can see some um huge chunks of code that are um you know being executed and where the time is actually spent in the whole journey request um Can anyone spot like a challenge with that approach like you know I'm using Auto instrumentation so I didn't change my code does anyone you know notice anything on that screen that may be of interest exactly

that's exactly it you see at the bottom there's some you know um a span you know a trace is typically you know composed of multiple different spans like you know parts of a uh whole trace and what's happening is that you know there are some uninstrumented time with the auto instrumentation you know the the visibility into my code ends at some point this is the the caveat

I mean it's easy to configure Auto instrumentation but at the same time on the other hand hand it's like it only goes so deep into my code which is on purpose right and this is exactly what I um referring to I can see you know this part of the span you know takes 3 seconds but what is happening in that 3 seconds I have no idea and

this then brings me to um you know the other option of you know manually instrumenting my code with open dmetry in this case what I'm um need to do is in this case I configuring some you know properties of my application again the OTP endpoint where do I want to send that Telemetry data to from the um open Telemetry pipeline I have some license key again for

our API key and maybe important I can also of course configure much more details of that environment I can specify do I want to capture the Kafka um Telemetry right do I want to you know add any additional information um from my Tomcat in this case as my web server so I have full control over what is happening and I can highly configure um this environment the

details can be found in the um docs of open Telemetry but this is just an an example so if I then look into the the actual code um like I said in the um case of the auto instrumentation I don't need to do anything you know it's just adding the agent and and it works on in contrast in this example I do need to add some code

and some you know configuration for my Telemetry configuration in this case I'm highlighting one section here this is the the tracing aspect and it basically you know configures the a tracer provider um that I can then use within my code in various different um parts of the application so this is kind of like a one-time thing typically in a static mode within my application but this is

um what I need to do once for each of the applications that I need to um configure you can see you know I'm also you know basically oops I'm sending um you know or adding in the LLP endpoint here the API header and so on but this is now in my as part of my code um that executing when it comes to the Kafka instrumentation um there's

also something that I need to configure basically for the producer side I um need to um add in a um tracing Interceptor this is a standard way for Kafka to you know intercept something in in the case of you know some producer adding some messages into um Kafka topic and the same is then happening also for the um consumer side so on the consumer side I also

have to add some kind of interception that actually um you know then listens to some information within the a message and the topic what actually happens behind the scenes is you know what open does that it actually adds some um Header information to each of the messages so that the producer will add some tracing um context to that message and on the consumer side it'll read that

context and basically process it it from there so basically what we've seen with the auto instrumentation also works in here but I have to um do some um and add some additional um things to code for the manual instrumentation a trace looks pretty much the same you you will notice that maybe the the span names and and the information there is a little bit different because I

can you know customize this to my specific needs maybe I have my own naming conventions across my entire organization I can um highly customize this and name it you know in a in a meaningful way and also you know the uninstrumented time that we have seen earlier um now I know where this is coming from this is a long running task you know I could go into

more detail in this and break it down even further but just as an example I U basically just created a custom span for this specific method method that I'm calling basically what happens is I just you know in order to do something I added a um a sleep in here um just to add some you know um you know span duration for this specific T task but

importantly what you can see here the um the Tracer provider that we used earlier I'm creating a custom span here you I Nam the span and what you also see is kind of like I can add some additional meta data in this case I'm just adding the the time it is about to sleep in that thread but basically you can ask uh uh add you know any

kind of custom data from within your application think of again that example of a retail application you know a customer puts something into the card I can add the product ID or the the product name the product category ID as custom attributes spans why is this important because this will then help me to in in you know in analysis cases if I have some root cause analysis

that is happening maybe it is only a certain product that is having an issue in the cart or maybe it's a a category or maybe the region where the users are coming from so I can highly um customize this and add as many um you know custom attributes and information that I think I may need in the future as um information to that span on top of

that um what also happens out of the box you know we're not collecting only traces from um the environment there's also a lot of metrics that are being captured in this case I decided to capture metrics you know it is in my control but if I do so I can also look at you know some producer specific metrics like you know the the buffer available the average

IO ratio and so on so I can add a lot of more detailed information that will maybe give you give me information about the producer side of things and the same is also true for the consumer so also for the consumer I can send in an add some additional consumer metrics that maybe help me you know Gathering some information about why is it slow is it the

application was it something happening in the application or was it some infrastructure aspect and also um finally you know if you want to capture additional details about you know the Kafka environment itself this is then going into like you know Kafka specific you know infrastructure metrics what is important here is if you you know remember that that span that Trace that we have seen earlier and that

you know WID space in the tracing I could then you know look at data like consumer lag for my kfka environment so how how much time does a typical message spent in kfka before a consumer reads from that um from that topic so a lot of detailed information about um the the infrastructure aspect as well from my Kafka environment in general um open Telemetry you know has

great support for many um different libraries um typically again I would highly suggest to go into the and look at you know what are the libraries that I'm using in my application does open Telemetry already have support for these um libraries you know it is the um the you know the the the challenge for the developers of these libraries to add actually open tmy into the applications

but once they done that you know like I mentioned open tmy is an open standard you can then use that you know and use it for any other libraries that you may be using in this case Kafka is really just as an as an example example I have the links here to the um documentation um I can share this also later with you so in summary um

you know be mindful of you know what it is you want to achieve you know is you know maybe Auto instrumentation is a good starting point for you to you know see what is happening you know it's not much effort but as we have seen before as well it only goes to a certain level of depth within your application if if it is enough then fine you

know you didn't put much effort in and you got you know fast you know insights into your application but if you want to have full control over your applications um you know there is a way of how you manually can um you know instrument your applications The Collector the pipeline that we have looked at earlier is also you know very useful you can use it you know

to you know add additional information maybe delete data opk data a lot of um configuration but it is also something that you know if you're running it inside of your infrastructure as part of your application you also need to make sure that this also scales with your application right because at some point you know you don't want this to be like a bottleneck in your infrastructure um

and you know the support in open Telemetry is really um is really great you know like I said a lot of you know big companies are investing heavily in um open Telemetry AWS you know Azure Google Microsoft they're all investing we are also um a big contri distributor in open Telemetry you know our agents that we used for many years they have a lot of knowledge and

you know information on how to gather data from certain different environments so we add um also to these open source projects um very heavily and with that um you know there's some additional reading material you know I can share the slides um you can look into more details um in this but with that thank you very much for your attendance um any questions thank you Harry again

so just a reminder you can also ask questions on slider otherwise I'm ready to round uh to run around and give you the mic to ask the questions do we have any oh come on it's pretty clear right okay perfect oh yeah there is one um so like you mentioned it already like Azure application insights is you know a Microsoft service right you can also ingest and

other you know Telemetry data into app insights open Telemetry is really like an open standard right it is vendor agnostic you know independent of what environment you use maybe you want to you know use you know um Azure app insights or some other back end to today but maybe tomorrow you want to switch it is just a matter of changing that configuration the OTP endpoint the Header

information whatever is needed to send in the data and then you can you know switch back and forth but you implement and instrument once and the idea is that this will be you know there for the future for you to stay but then you can decide on you know what what back end do I want to use to actually analyze that data because that is really the

the aspect but um I guess you know to your point I guess the the the aspect is it is vandor neutral it is an open source project you know a lot of contributors to that project um so I hope that answers your question um I guess if you um are at a starting point like with new services that you're instrumenting it's probably worth looking into you know

what could op Telemetry offer me in that um case and they basically go into that direction um for existing services is you know it is probably worth um you know switching um but then again you have to have some environment you know some other environment in place that receives that data um and so you know more questions no okay oh there is one one second you'll get

a mark uh thanks for your presentation uh do you have any recommendations for rate limiting of um tracing uh for example if we want to um Trace only one of n uh requests uh can we do it yes so in yeah in general so the question is you know you know can we you know add in some sampling maybe want to only add or Trace 10% of

the request is that the the question right so yes you have you know full control in in in the standard configuration it will sample the data because you know there can be a lot of traces you know thinking of you know if you have an environment with like hundreds of requests from customers it will um do some sampling by default and then there's also some configuration that

you can apply on some you know it's so-called head-based sampling so where you um make the um configuration of do I want to capture that Trace at the beginning of the trace but there's also some you know additional configuration on some tail based sampling where the decision to sample that Trace yes or no is made at the end of the trace why is that important because at

the beginning you may not know if in the whole Trace there is some error right or maybe it was slow at some point and if you make that decision to sample at the beginning um you may you know not capture these interesting use cases because that is actually where the you know where the things is that you want to optimize and and maybe do some root cause

analysis if you do some um tail based sampling you know you know already in that entire Trace did we see any error was it slow at some point and then you can decide I want to capture all the requests with errors and I want to maybe capture all the requests that are slow for some reason or above a certain threshold but yes um it is possible to

sample and that I have a question yeah so technically your talk was about tracing Kafka right yes and Kafka is as you also mentioned is more of a blackbox part of the system but very reliable one yes so where is the balance between collecting all the possible metrics and drowning them versus actually just not caring and then restarting the any service that goes down yeah so I

guess in the case of like Kafka like you said it's very reliable in general you know there's not too much you need to worry about especially if you consume it as like a manag service but the the tracing aspect of what is happening before and after a request this is really where it's where it's interesting and you know like I said the the the Header information or

the trace context that is added to a message doesn't consume that much you know additional overhead it will have some overhead but having the visibility into a full Trace end to end is really where the you know big um benefit lies and this is where I highly encourage you you know also to you know to add in some um some of that tracing um um aspect if

whether you or not you want to Capt the Kafka metrics um that I also showed that is you know something that you can decide but I think for the tracing aspect it is especially interesting yeah gotcha thank one oh there is one oh there are several ones cool so right let's start with the top one uh any recommendations to run inside kubernetes yes um so there's also

um and I think I have a link in the appendix there's also some um kuber the specific you know operators that you can deploy which will add some Auto instrumentation to your services if if you choose to do so for maybe different name sets and so on so yes there is a specific aspect you can of course you know depending upon whether you use Auto instrumentation or

manual you could add for some services and bake it into your containers and your applications maybe for other services you can add in some um Auto instrumentation and this is where the operator for kubernetes can add in the auto instrumentation aspect for open to lry thank you uh I will bump up personally one of the questions yeah uh we've heard that New Relic can be OTP backend

do you know whether data dog accepts uh this Trace Trace in end metrics as well um I'm pretty sure it does um I'm haven't used it of course uh but I'm pretty sure it does um have capabilities but I don't have any insights into um what it is you need to configure there um okay so I guess we have to just figure out and are there any

statistics about added latency using open telary so adding added latency in the case of how much longer does it take to send the data into Kafka maybe someone can speak up who added the question or what part of latency are you are we talking about so I guess also if you have to like my guess not my question um you have to introduce specific code changes right

you instrument your code and it's a call right it's a resources do you think it can add a significant overhead in your code as well so for your application itself right if you add open Telemetry as part of your instrumentation it will have some overhead you know nothing comes for free right in general I would say you know the the overhead within the um instrumentation of your

application is probably in the lower percentage rate you know maybe 1 to 3% a dedicated number is not possible POS depending upon how deep you want to instrument um um also the aspect of you know maybe how chat is that service with the database and how much is happening inside of that application so it's I would say in the lower um percentage rates the the overhead but

basically this is um you know the standard you know um aspect you you gain some invisibility but you have to pay a little bit of um overhead um if that is the question around the latency um I hope I answered it if not you know I hope I understood the question also correctly okay and the first one was do you know the current status of the 70%

prediction I have no updated information on that um the data that I showed is I think from 2022 um and that was the prediction for 2025 um I don't have an updated in uh statistic for that I'm sorry still thank you so if we don't have we do thanks um one more question we had a PC with one of your competitors using Auto instrumentation on um kubernetes

and we saw for um our applications like a 10% memory increase for most Parts MH um do you have any recommendations in which cases you actually uh want to use Auto instrumentation uh compared to manual instrumentation because I guess like um just writing custom spans will um not result in such a high memory consumption um it is a trade-off right I mean um having 10% memory increase

is seems like a lot um and maybe you know that is an indication of maybe Auto instrumentation does too much within my application for various reasons I could you know look into configuration of the auto instrumentation as well like you know maybe if I'm seeing you know or if I'm capturing also database requests right if my application that service talks to a database maybe that is the

the bottleneck this could be um switched on or off and maybe that is um um a way to gather information on why where this heavy um overhead comes from and if that doesn't solve it I would look into manual instrumentation and maybe start like really high level and only go deeper in areas where I think um the bottleneck lies and then you know going from there um

I'm having like another talk tomorrow where I'm instrumenting like a game like an 80s um jump and run game um and I played around with open Telemetry as well right and some point I did way too much um instrumentation of my um code and you couldn't even use the game anymore because it was too much happening right so it is a balance and a trade-off um that

you need to apply but basically looking into maybe the configuration of the auto instrumentation and if that doesn't help you know going into maybe the manual instrumentation route perfect thanks one more question from slido yeah what do you think about using open Telemetry for monitoring troubleshooting kubernetes API long requests um that is an interesting topic and I would consider this to be more like an infrastructure aspect

um open Telemetry um started with like the applications and services um but um it will go into the infrastructure aspects as well so you can you know gather information you know metrics of your you know infrastructure like API request to the kubernetes API um we actually launched um also um a open Telemetry um support for infrastructure aspects that you can use in your kubernetes environment or your

other um parts of um the infrastructure so yeah definitely if that is um seems to be like a a pain for you then definitely helpful to also you know look into the the um you know Telemetry for your infrastructure for the communities environment thank you I I think we have time for one more question if there is yes hello thank you for for presentation uh what do

you think about uh is it possible to configure any alerts for loaning requests yeah so the short answer is yes of course that is then part of the you know your back end whatever you're using um I mean we have ways and how to configure like you know static thresholds like if the if a request takes longer than two seconds then I want to send out an

alert or send a slack message or whatever but basically what I would recommend in that case is more like looking at maybe some baseline house your application behaving in a typical week from a historical perspective if I see like a spike on Mondays every Monday There's a spike in um duration and it goes down and on Saturdays it there's you know it is very fast all the

time I would follow the the Baseline and not configure like static alerts because they typically don't tell too much but yes um of course possible yeah sure we can use percen or something else like to uh look about uh common behavior and then if Behavior changes uh we should uh alert but uh will it be configured on back end or open symetry gives us such possibility uh

typically on the back end um because there there this is the case where you have that that back end and that could be you know Jagger or some um um other form alert manager maybe um but this is typically something that is outside of the open Telemetry pipeline um in general okay much and this concludes this presentation thank you thank you again fork for attending thank