Eran Kinsbruner: Troubleshooting Distributed Workloads ...

In this session, Aran Kinser discusses the challenges of troubleshooting distributed workloads and complex architectures, emphasizing the importance of shifting left observability. He explains the disconnect between development and operations teams, which often leads to reactive rather than proactive incident response. By empowering developers with observability tools integrated into their IDEs, the collaboration between teams can improve, leading to faster resolutions of production incidents. Kinser presents various use cases where organizations struggle with debugging, including Kubernetes clusters, serverless functions, and CI/CD pipelines. The talk also highlights how optimizing logging practices can reduce cloud costs and discusses the role of feedback loops in enhancing software development processes.

[Music] hi everyone hello uh my name is Aran Kinser and uh I welcome you to my session at devops Pro Europe um I am a devops evangelist and Global head of product marketing at Lon which is a dynamic developer observability platform we'll talk about it uh if time permits uh last in the session uh but in my session today uh I don't have too much time but

I'll try to explain why troubleshooting distributed workloads and complex architectures is so complex and so hard but it can be uh made more possible and easier when you shift left observability when you employ uh a new method on top of your APM and logging uh infrastructure and tools so developers uh as we say are much more connected to the application that they actually developed that are running

whether it's in production in CI QA staging doesn't matter but most most importantly if they are running in production and you have an issue this is where all of what I'm going to talk about becomes very critical very complicated and time in this case is the most critical uh parameter or uh value that organizations should care about so uh let me start by uh highlighting a bit

of the pain that we see today most of the organizations today if you read surveys from the cncf or devops uh Dora or other uh surveys within the marketplace uh we know that the majority of organizations are running containerized applications distributed workloads microservices serverless all of these model type of applications with that uh aside there are still organizations that are doing a mix of uh deployments we

see organizations mostly large Enterprises that are still know have much more newer and modern applications based on microservices but they have Legacy applications that are not retired yet and they need to support them I'll will talk a bit about this Nuance between the Legacy and the new and modern architectures as we dive deeper into this session but a bit about the pain so if you look today

at a we are in a devops uh conference so right you have a devops organizations it can be a squad of different uh practitioners with QA with developers with uh other devops engineering product owners it doesn't matter at the end of the day what does matter is that the development team the engineering team is using a whole set of different tools than what the operational guys that

are you know in the shift right uh equation uh uses so just giving you an example the operational guys are using tools for logging and APM alerts right splank data dog din trace and the likes developers live within their idees intellig pycharm and uh the likes okay gitlab GitHub okay so there is already by definition a this connection and this is a natural one I'm not this

is not a bad thing right this is the reality it's based on skill set it's based on objectives and job that needs to be done the developers by definition are using different tools than what the Ops guys are using for their daily job but this reality which is not bad but this is what it is causes some reactive rather than proactive approach to when things go down

when things are not performing as they should in production or even in pre-production environments and this causes when things like that happen uh for the developers as you see on the uh most bottom uh bullet here the developers are disconnected from their live applications that's the bottom line right because if something happens in production the first guys that are getting alerted are the Ops guys they're getting

the alerts from the dino trace and the APM tools and this is a bit late in the game because when you need to troot something you then okay what do I do the OBS guys might might not have enough Telemetry data and insights into what causes the issue this is where they turn back to the developers the developers find that they don't have enough logs or uh

insights into what is happening so they need to add more logs right and this uh drives a new cicd Pipeline and this is very inefficient very slow and unfortunately also very expensive so this is you know the reality and what uh we're going to talk about uh in this session is how by shifting left observability by empowering developers to be more connected to what's happening in the

distri uted workloads in the remote environment from their idees from the intelligence and vs scode of the world they can react much faster and just making sure that it's clear the developers are not taking you know the dinner lunch breakfast and job of the OBS guys the OBS guys have a very critical and important job but this disconnection makes both teams miserable and less productive so with

a new approach that I'm going to present to you in this session you will understand how the collaboration and the empowerment of both teams by the way can help the organization to react much faster to production incidents or pre-production incidents and by that you know if anyone is following me on LinkedIn in this session I've just ran uh two two and a half weeks ago a live

webinar with Google Dora and price line one of our customers and what Google Dora is all about right is the productivity metrics the mttr the uh failure rates the percentage of failure rates and these kind of things so what I'm going to describe here is also something that contributes to Improvement in developer productivity mttr and uh the likes so what I am seeing and what lighton sees

from a troubleshooting perspective as key use cases which are very challenging are the following uh seven there are eight here I just touch on seven and I leave one which is not a troubleshooting use case it's more kind of an optim optimization use case okay the first one is troubleshooting kubernetes clusters we know how hard it is to travel sh kubernetes environment remote and distributed especially when

the scale is massive not to measure when you are shifting TOS serverless like Lambda functions from AWS and Google functions from Google uh and the likes right so seress is even more complicated and I will touch a bit about why each of these use cases is so complicated I mentioned earlier that the organizations have also done an event recently with MNT Bank MNT Bank told us told

the market that they are like a museum right they have uh applications from different ages from different eras some of them are very Legacy and monolithic and some of them are very modern and microservices oriented which means they need to tackle both types of uh environments the next thing or the use case that you need to troot is cves and how do you prioritize them cves for

those of you online that are not familiar are security vulnerabilities which are attributed to open source libraries and we are all using open source libraries even if you're not using you have heard about log for Shell log for J and all that uh very famous uh you know severe and critical vulnerabilities being able to understand which cve impacts which environment and which client of yours is a

challenge that can take you a lot of time so that's another use case a fifth use case that is how to troubleshoot and again I'm going to explain why is code flow investigation using feature Flags even within L run when we are building our SAS environment our SAS platform for troubleshooting right we have different features behind different feature Flags so uh think about an uh an incident

which uh is very specific to only a feature which is now enabl but the other environments are not relevant how do you troubleshoot only this isolated environment in production Without Really replicating and doing every uh setup from scratch from development standpoint it's very very hard okay so that's also a challenge that it's not just by the way everything that you see on the screen it's not something

that Aran is inventing these are live real life use cases and challenges that we are hearing from existing customers of flight but also from Market uh practitioners I'll jump to number seven because I told you the optimization is a bit different and I talk about cicd debugging right if something fails within your CI pipeline whether you're running GitHub C or other CI pipeline what causes that you

need to troubleshoot and basically when you're using CI you don't put any logs or break points within the flow so being able to understand whether it's a fail test a bug in the test a bug in the software it's very very hard to pinpoint and that's also something that slows down the troubleshooting and is a pain point developers last but not least is the performance issues we

know how important is to have a production environment that really functions and work as as it should when you have a performance degradation being able to understand what is the anomaly that causes that what is the root cause is very very hard and very very complicated troubleshooting that is a nightmare uh so the E use case which is number six is the log optimization and this is

where I actually told you in the previous slide that when something happens you know developers are either adding Pro uh proactively uh logs during the development stage just in case that's the word just in case right and this is really causing a huge amount of cost to the cloud uh uh bills that the engineering team is paying being able to minimize the static logs that you have

within your code you know the ones that you put in data dog and the likes logs Zio is very very critical especially when we are in a very economic stretch uh for the past few years right so so being able to optimize the logs that you are using only to the uh logs that are very very effective the ones that you're really going to use is a

new practice that organizations needs to employ so the a reduces the cost of travel shooting and B are being much more uh agile if you like to add Telemetry data only when and where they need when breaks and don't just take my word for it this is a survey that was uh done quite recently by uh Microsoft and what Microsoft is telling us maybe it's not a

huge surprise but this is scientific data okay that's real data coming from Microsoft is that the majority of the RCA when something breaks you know like I've shown you in this previous slide when something breaks in a kuber NES cluster environment when something breaks in a Severus um uh function or other uh out of these eight uh use cases seven and a half right the RCA statistically

shows that the majority is related to code related bugs and people would say d right we write software we have bugs no but if you look at what you see here in different colors there can be so many different reasons to why things breaks why incidents happen it can be a dependency can be infrastructure or network it can be deployment and can be configuration and stuff like

that so RCA category shows that code related bugs software quality if you like lack of testing I don't know human heror uh these are the main root cause for issues but the mitigation you know is very very important either because the majority of organizations when something happens is to either do a roll back or to do infrastructure infrastructure change both are very very timely and very very

costly okay think about to do a roll back and do a full cicd Pipeline with all the testing regression and and the likes and especially when it's in production can be a huge huge hustle for the entire team not just for developers right so culated bugs that leads to roll backs and infrastructure changes can be very very complicated and if you look at what impacts you know

uh you know these Cod bugs and what is happening from a uh total time uh or uh time to deliver and time to mitigate you know you can see here still from the same research by Microsoft that the code related bugs like I mentioned in the previous slide are the ones that are taking the longest from a time to mitigate you see TTM here time to mitigate

uh uh issues that are related to either codb and dependencies mostly codb is the highest and the longest that it takes okay with that in mind let's understand why you know cordinated bugs takes so long to fix in this uh different use cases that I've talked about and starting with kubernetes clusters right we know that kubernetes clusters are at high scale and you need as a developer

usually you don't have right but to troubleshoot issues that are happening in a service that is deployed in uh I don't know how many replicas and how many pods in production you need first and foremost you need access to this environment as a developer sitting at your home or your office usually it's going to be very very restricted and very hard to get access to this environment

so logging and obtaining Telemetry from this environments is tedious costly getting remote access is usually not approved by sees by the security gu and okay you found something updating the code in production is obviously not recommended and when you need the access sometimes you also need to employ port forwarding and special permissions that is also a pain uh for it and Engineering uh by definition so you

can understand that something when it's breaking in production you have seen what Microsoft is saying but even like even if you do have the time and you do have I don't know uh the understanding of what happened being able to troubleshoot it uh in such a remote environment is very very complicated if that is that is not enough then you look at what is happening with serverless

right you have sever functions that are uh available from both uh all from actually all Cloud providers starting from Google AWS and Microsoft Azure Cloud they all provide you the technology and the infrastructure to build sess architecture sess applications which are uh by definition very very lean in uh you know you just build a code and the cloud providers actually uh provide you with the INF structure

that you need and the scale that you need to run this but the again uh it's in the the definition of serverless it's it's uh the code is running locally but everything else is being uh populated by the cloud provider so it's very difficult to debug and analyze what's happening locally in your IDE right as a developer understanding the resources that are impacting the issues is also

not under your control as a developer again talking about locally versus running in the cloud and we know I won't read you the full slide but we know that by definition serverless functions live for a very very short amount of time sometimes it's just one or two minutes sometimes it's even less so even if you know what is happening you don't have that much time because the

runtime of such a function is very short okay so um I think I scared you enough right kubernetes cluster debugging sess architecture debugging it's very very hard and you know these are the major architectures that organizations are using today to deploy and build modern applications as I mentioned earlier many organizations do still have a mix of legacy and monolithic I don't want to read you the full

slide because we don't have time but I just want to give you one or two points about why it's so complicated today uh when you you are dealing with incidents in a monolithic application just building such a huge monster you think think about a bank right I mentioned earlier MNT in my one of my webinars right just building a new version of such a monolithic architectur is

very very long not to mention adding new logs running a full cicd or a full cycle of testing before you deploy it right so many dependencies and long restart these are things that are by definition comes with the definition of Monolithic architectures this is by the way why many organizations neglected or moved away from monolithic to sess and uh Cloud native uh architecture but when you do

have these applications and even though there are not that many uh uh depending on the organization maturity it's also quite hard and challenging sometimes it's even harder than kubernetes clusters because with kubernetes clusters at least you you are dealing with more modern and very uh pinpointed uh pods and environments here you have the full blown application with all of its dependencies that you need to build and

uh analyze when you have an incident I spoke a bit about the the cves I just say well you know everyone today including my engineering team is using SCA and SAS tools to scan sometimes it's it's a part of the criteria when you're building a new ver new version when you're building a new iteration of software right you need to show that you don't have any P1

or critical cves security vulnerabilities that you have within your open source libraries that you are relying on but as I mentioned in the in the uh intro during the session you know you as a developer think about a dashboard that consists of I don't know 20 60 80 I don't know depending on the size of your application uh you have dozens of cves how where do you

get started how do you know which of those are impacting your environment in production and which users are impacted usually you have no idea because you for that you need to understand what is running in which environment okay so trouble shooting cves in production to determine which are impacting which users in which environment with which feature Flags it's a very very difficult uh pain and talking about

feature Flags again I mentioned it earlier being able to uh understand a specific environment and most importantly to replicate it to understand which environment with which characteristics is being impacted by the incident is very very hard and also very very complicated because you're dealing with very uh specific data sometimes it's customer data sometimes it's production data that you can simply not create or recreate in your uh

preo environment so debugging feature Flags in production is another challenge another pain that we keep hearing in the marketplace and I have the full faith and I truly support this is the best Mo modern Eder way to to build software using feature Flags Progressive delivery this is the way to go right but keep in mind that also it's the way to go with Sever and kubernetes clusters

but it comes with a toll it comes with some challenges because when something breaks you need to understand how you're going to approach it you need to understand what uh or when the uh Ops guys start and when the development team begins uh because or takes takes it away from them so it's all in sync and kind of Dev compliments uh Ops in a very uh efficient

way I will not spend much time on the uh cost optimization but I mentioned earlier you know and this is taken from the phobs organization empowering engineering I just focus on this uh response empowering engineering to take action on cloud cost this is one of the biggest challenge that fops believes in uh and it makes sense right engineering usually have the biggest budget within the organization and

with that they not always look at how much really uh a new log line a new uh deployment I don't know a new amount of cloud computing actually costs them because they're not really seeing the uh uh report or the uh purchase order they just care about building the software the best they can and this is their job but as economics become uh a critical pain with

like phop shifting also left towards the engineering it's important to allow engineering to have visibility and accountability into how much they spend to build a software because that there always is there is always a better way to build a software when you take also cost into consideration and specifically with regards to the log everything and analyze data that I mentioned earlier right being able to uh debug

in different ways and only put logs when and where you need during the development phase knowing that you can always get back to this C in production is a new mindset a new shift uh in optimization and productivity that I'm going to talk about in the coming minutes last but not least is the challenge around cicd debugging I mentioned earlier something breaks in your cicd how do

you understand what's going on is it a test issue it's a feature that is breaking it's a feature flag it's a dependency Cod buug you have seen what Microsoft is saying but being able to troubleshoot within the CI is also a very Big Challenge again because you don't have all the Telemetry and the developers usually are flooded with so many things going on which makes it very

hard performance issues I'm going to show you very soon a demo you know because everything that I've just said will become even more clear When You See It live you know you understand all the nuances of trouble shooting uh um reactively you know as a response to something that happens versus doing it on demand when you need only and where you need okay so dealing with anomalies

and performance issues is also a huge pain and it's also a pain for the business because when something is not performing as it should or you have discrepancies in data in production like user login failures uh for specific users it's very hard for you to manage the business and the business actually uh is being impacted in the bottom line when I talk about shifting left observability and

I'm going to show you soon what I mean by that it's also important to understand that there is a feedback loop okay we always uh like in the past decade we were always familiar with this uh devops uh endless infinite Loop right building uh debugging deploying uh all this shift left and shift right everyone was talking about right but no one was talking about the real feedback

Loop which makes everything much more efficient okay and I won't touch on platform engineering and the tool stack I'm just going to talk about the feedback here okay when you're dealing with a production issue that uh happens in a specific environment again doesn't matter if it's KU severs or Legacy monolithic or whatever something happens right you did all the work troubleshooted it how much time it took

you I'll put it aside right now but you fixed it are you able to uh share this feedback loop of the root cause we talked about earlier the root cause can be dependency can be code related bugs can be infrastructure whatever are you sure that you sent also the feedback back to your testing team so they build modern and new upto-date regression tests right we know we

know the um test pyramid actually wrote books about continuous testing but this is much more than continuous testing it's about closing the loop we talked about Ops and Dev you know being more connected by shifting left observability but all this feedback also needs to be shifted left towards the testers so this is not being kind of um a recurring procedure a Rec recurring event uh in the

future so once you solve the issue regardless of the method please make sure that testing also receive the feedback the root cause analysis and where in the in the code the issue isue failed just so they know which kind of test cases they need to build modify add so this is not repeating in future iterations after I gave you all these scary uh use cases and scenarios

let me introduce to you what I mean by shifting left observability or observability driven development when I talk about shifting left observability I'm not disconnecting Dev Ops I'm actually really connecting them one to another but in a much more uh I would say natural way in the right tools that everyone uh is used to use if you are a developer as I showed you in the earlier

slide they use the IDE they use the GitHub areas if you an OP you're using the APM and the logging uh tools if you're shifting left uh observability when something happens the alerting happens in the APM Ops are good okay this is the alert now the developers TT charge they don't need to rely on the op guys to give them Telemetry which usually they don't have the

developers have a Dev native environment to troubles shoot in real time uh in a proactive way with Dynamic logging as you see here logging exactly what they need when they need and they really now own you know their application their code whether it's running in production uh in serverless kubernetes Legacy doesn't matter but the developers are owning their own success which is kind of a mind shift

uh from where the most of the organizations are today if you go go and ask today Enterprises that are using and paying so much money to APM tools what happens and what is the cycle when something breaks they will say yes it's a lengthy cycle because when something breaks the APM gives an alert to the OBS guys looking at the dashboard the OBS guys don't have uh

the the knowledge of what to do with the code so the troubleshooting process begins which is very long up until it gets to the developers sometimes I'm hearing weeks you know it takes them to understand and reproduce the issues and add more Telemetry and build the full uh software from scratch it's very very inefficient when you are able to shift left observability you are really cutting I

I would say even more than half the time it takes I have experience minutes to few hours of uh reproducing and solving P1 and P2 production issues by clients that have implemented this shift left observability developer observability as we call it and the way uh they benefit from such a new method is everything that I mentioned earlier I mentioned Google door I mentioned uh you know uh

observability cost mttr uh cost optimization time to Market and quality so the way things are working and let me just move to an environment that I can actually show you in real time uh what is happening what you can see on the left screen is uh and you can see here the the Java Java application a very simple one that goes every half a second to the

Wikipedia website and plls for an article you can see here every half a second you are getting uh a new article title from Wikipedia this application is running uh it's still not running but it's deployed in two remote environment and it's running locally on my intellig ID in Java before I launch the application I just want to show you that nothing is running you see here in

my light run platform you know you see here I have no agent running okay and I have no actions that you see here this is an expired action from March but you see I have no agents what I'm going to do right now I'm going to run uh an access uh to remote environment one is uh an orle environment and the other one is uh an AWS

and what I'm going to do right now I'm going to run this exact replica of the Java application uh with my light run Dynamic instrumentation agent uh attached to it okay I'm not touching the code I'm just what is what is called I'm just instrumenting okay and the application is right right now running in two agents one in Oracle and one in uh AWS and you see

right now every half a second you see the skipping I'm getting a new article on both both deployments and you see here on my uh plug-in in intellig I see two agents orle and AWS and now you also going to see them I'm going to refresh my browser you're going to see two uh environment AWS and Oracle okay and what I've just shown you is that as

a developer I'm Living in a Dream everything is good the application is running in production and it's fine but what is happening when something doesn't work I'm getting an alert from Ops from the APM guys that something is broken uh on both environment and I need to Traverso it I have two ways of doing it with developer observability you know the logs metrics and uh um traces

right in lighter we have logs metrics and snapshots the virtual breakpoint I will first put a virtual breakpoint okay what we call a snapshot and here I'm going to select the source I go to AWS okay and I will ask for three break points on this line so I can get uh uh the ability to investigate three articles once the breakpoint was placed the application is still

running it's not stopping no matter what which action I'm requesting from a Telemetry data perspective the application is not going uh to stop I'm going to get the the breakpoint you see it here I got a SN Snapshot from production and you see here I got another one and get another one and you see here the title of the article and you see the title of the

article in the second snapshot and the third one I didn't need to stop the application I need need to do any uh full uh rerun of cicd I was able to debug my application at runtime uh remotely directly from the IDE and you see it here it's not I'm not uh it's exactly May 20th at 902 a.m. I live in the East Coast so that's my current

local time I can also do something more exciting and I can do what we call a conditional log and here I'm going to see Hello Dev Ops Pro Europe okay and I'm going to put uh a variable from the code rwa I'm going to do it and pipe the the log both through the STD out so you can see it uh in uh the console and in

the IDE I'm using AWS okay and you will soon see once I go to the ID light on Console okay you see on the left hello devop SP Europe okay oh I I did a I have a mistake here see I didn't put the brackets here now it's okay it just show you that it's live and I'm not stopping the application you now see Hello devops for

Europe with the actual title of the live application running in production I didn't need to stop it you see it here it's running on AWS but as a developer I don't really care where where it's running I do have here the ability to analyze and troubleshoot from my environment which is awesome and last but not least as I mentioned earlier the performance um performance debugging is also

something that uh is a pain what I'm going to do right now so we won't neglect uh oracel so I'm going to do something like that I'll call it Dev Ops Pro matric and here I'm going to analyze and I can say to Prometheus and other uh environments but I want to an analyze how long it takes to run this specific block of code from line 33

to 37 again at runtime in production without really stopping the application and here you see you see on the Left Right devop SP metrix I'm already collecting timings okay metrix from the environment and here soon when I go to the Tik Tok because that's how we call it you already going to see that I'm building a graph of performance so I can analyze the peak environments I

can analyze them here in the IDE or from the Prometheus and other metrics related uh environments that I'm using but the most important thing is that I'm collecting data at runtime from production environments without really stopping the application and you can see here in the dashboard when I click clicking on actions you can see here exactly that I've added metrics Dynamic logs and virtual breakpoints during the

application at runtime without really stopping it so it's all managed and uh fully uh governed within uh within the uh platform and if I need again to connect my uh metrics and logs and everything to other provider connecting Ops and devs as I mentioned in my session I can do uh I can easily integrate to Prometheus inana din flank so it's all a better together if you

like approach so everyone uh can continuously work between development and operations but in a more streamlined and efficient that I've done the demo and I want to leave just a couple of minutes uh for anyone uh online if you have any questions uh I'll just stop my screen so I can see the chat uh let's see if there are any questions okay uh okay I see that

there are a few questions um I will read for those of you are not able to see because it's a private chat I will just read uh the the few questions that are coming in the first one uh today at another session a statement was made uh that the lack of local testing is a big problem for the organizations where you mentioned it is a problems with

some aspects what do you think the the impossibility of testing something locally is due to the organization or the tools used it's a great question uh and uh as an ex uh director of QA and an author of testing books um I usually don't like to blame you know tools or organizations or testing uh quality is the is the responsibility of everyone but I would actually say

that it's the nature of the applications you know if in the past you have le you had much more time for deployment and for testing nowadays you I I will tell you from my experience at Lon we released a new software version every two weeks okay in these two weeks both the development team and the testing team and the documentation guys and the product everyone needs to

work in a very coordinated way and we are building Cloud uh product right you have seen the product demo right it's fully suas so being able to locally also build new tests for new features like actions and logs and everything that I've shown you and also build the feature itself is very very challenging so I think that there has been a change over the past few years

with the modernization of software where time is very short for both developers I'm not blaming developers or testers the time is short on both the developers and the testers which means that they need to think a bit outside of the box and uh kind of readapt the processes to what is really happening because you already know today that in two weeks you have a release you already

know that this release is going to be deployed with each amount of feature Flags across 10 different clients or 100 different clients so you need to have uh kind of an automated setup environment uh with a baseline that is matching the previous version on top of which you add your new test case that that you added plus regressions because things happen right so it's all a new

automated mindset and techniques that developers and testers needs to come up with and uh things that I just demonstrated Lun what we call the dog fooding you know uh my my developers are using their product sometimes to debug issues that are coming from our production customers when something happens no one is perfect right but when you have a tool that allows you to shorten the time that

you need to recreate every complex uh scenario you are in a very bet better or not a very you're in a better position to address uh issues that are happening in production so uh I don't think that it's impossible to test I think that it's just harder and organizations needs to optimize the way that they are doing that the next question was how much is lightron affecting

performance for high load system so uh that's a great question uh and we have actually metrics that we can share I won't do here I'm not going to promote light here I can only tell you based on technology right that as you have seen up until the light on agent is running you know nothing happens and when it's running it's lazy it's not doing anything it only

waits for the server to ask for an action so if everything is working in production you don't need anything you know there is zero impact not near zero it's actually zero impact when there is a need for a snapshot or for dynamic log it takes you like a very very micro micro micro second because we're very very fast very optimized so we only ask you to debug

and request Telemetry when things are not working so when everything is fine you don't need to do anything but you are prepared for when things are actually broken uh broken and then you can add these Telemetry request or actions uh and uh you you're going to uh use them but the uh payload and the impact is very very uh uh negligable the next question is what should

be the first step to move to OD uh observability driven development so the first step is to identify and acknowledge the pains where you have them and uh as I mentioned there are different use cases some of them are very generic you know just troubleshooting my kubernetes cluster or just troubleshooting sess there can be specific cve uh troubleshooting there can be things around uh log optimization there

can be things around cicd which is unstable so the first step is and by the way I'm already uh seeing many organizations starting to move toward PL toward platform engineering so if you're already looking at a new mindset because something isn't working well for you you know uh platform engineering is is is a good sign of uh a reshuffling of your tool tool set this is a

good step or a good sign for you to look at okay how is my observability uh uh strategy looking today what is my metrics what are my metrics for mttr for time to U deploy okay for uh failure rates right if you have these metrics and you have the goals or objectives to what you want to improve you already know uh what will be the step the

first step is of course you know moving to obser to observability driven development means that you are trying to not break the silos to actually connect the silos between Dev and Ops you're trying to fix a broken Channel between your operations and uh the first step is yes using uh a developer observability platform right that this is a growing technology today which was built by Developers for

developers because of the pains that everyone is talking about and I've demonstrated them during the session so yes mixing and matching Technologies uh many organizations today have 12 10 to 12 uh tools within their cicd pipeline not all of them are very optimized and not everyone every tool is working great for that you need to think okay how do I merge and manage multiple tools in the

observability tool chain that actually moves the needle and improves my mttr my developer productivity my developer experience no developers want to spend days and weeks on just troubleshooting issues if you can do it much faster and last question because we are very short to the end and I appreciate everyone that was uh asking these questions thank you so much uh what if there are multiple instances of

the same application you don't know from which instance exactly the breakpoint is going to be hit it's a great question for that we have obviously for each deployment you are going to have an agent like an agent and you're going to use tagging okay like you use feature flag you can use tags and with the tagging you have going to put a tag on a snapshot or

or a log and the ones that jumps to you you know that it's coming from this environment you can add conditional snapshots Dynamic and conditional logs and metrics so you are easily getting all the Telemetry from the exact point of the failure directly as I've shown to your IDE to the intellig vs code pyam and the likes so you can trouble shoot uh in run time uh

as fast as you can with that I am exactly on time thank you everyone for being in this session and asking the questions if you have any uh questions to either me or about Lon uh feel free to reach out to me on LinkedIn or uh slack or WhatsApp and if you want to just try it yourself just go to playground. lion.com it's free you can use

W implement that it's deployed on Lon website and troubleshoot it and see how it actually works so you get my word and it's not just I invent things you can play with it first and with that thank you so much and I hope to see you in my next session thank you so much for