Costa Tsaousis: Agentic Observability — The Path to AI Co-SRE

This talk focuses on the concept of observability in the context of machine learning and artificial intelligence to enhance infrastructure monitoring. The speaker discusses the challenges of querying vast observability data for anomalies across numerous monitored metrics. They describe how Netdata employs machine learning to analyze metrics in real time, thereby detecting anomalies without bias from future data. The talk emphasizes the need for discipline in AI model operations to ensure thorough investigations and reduce false positives. The speaker illustrates how Netdata's hierarchical approach can reveal correlations and dependencies among metrics, further enhancing observability and enabling better decision-making. The ultimate goal is to shift from a reactive to a proactive monitoring strategy, helping operations teams manage alerts more efficiently.

What I I I will claim is that we are here to help you at 3:00 a.m. This is why I love 3:00 a.m. No one loves 3:00 a.m. of course, but we make it easy. That's the the whole point. So, uh let me start. A genetic observability, machine learning, all of them combined. What is the hardest question for an AI assistant? The guys that you have tried

it, you know, is this. So, let's assume you have a thousand nodes, and you ask, "Hey AI, hey Claude or Codex or whatever GPT, how is my infrastructure today?" No way. Huh? It's not going to happen. Why it's not going to happen? Mainly because observability data are vast. Observability data are the biggest database that you have in your infrastructure. And not only that, it's a lot of

uh signals out there. Probably uh for example, in in the case of the data, we collect about 5,000 uh unique metrics time series per server, per node. This can go up to 20 or 30,000 depending on what the node is. Uh and of course, it is important that all these metrics and logs have time. So, it's not just the latest value. It's how the latest value came

to be there, huh? Or what happened a few a few moments ago. So, if you ask this some AI agents, what they will do is they they will check alerts. Okay, let's check alerts if if you have alerts. And if something is raised, if something is raised, then they will go to the part of the infrastructure that raises alert and try to figure out what is wrong.

Uh if you don't don't have alerts, they will look at the infrastructure, probably pick a few nodes randomly, check CPU, memory, probably they understand from the matrix some application of interest, some database. Probably they will do two or three or four iterations and they will give you a verdict. Eh, your infrastructure is great today. Eh, it may not be. eh this is why this is this is

how a mail machine learning comes into the play. Machine learning is part of artificial intelligence, eh, so it's AI with AI, but anyway, Netdata has uh machine learning across the board for all metrics. This means that we train machine learning models per metric. We train these machine learning models based on your data at the edge. So, you install Netdata agents to your servers. For every metric we

collect, we learn the patterns, the behavior of this metric based on the collected data, and then we detect anomalies on the fly. when we collect a sample, we can immediately say uh if it is anomalous or not. Of course, there is an anomaly bit that is stored in the data in the database. So, when we decide if this is anomalous or not, this bit, true or false,

is it anomalous or not, becomes part of the sample. It's stored in the database. You can query it, eh, for the past. It's not biased by the future. Uh we see a lot of a mail all over the place where it is biased by the So, it knows what happened later, so it's biased. Uh this is decided at the moment we collect something, so it's unbiased totally.

And it's pure real time. Now, let's go to the next. Using this machine learning, so when a data what we have achieved in a data is that we have built a tree of information. So, let's assume that you have 1,000 nodes. This information, imagine that you have 5,000 metrics per node, so 5 million metrics in total. So, what happens in a data is that we have built

a hierarchy where the 5 million metrics become I don't know, 4 500 contexts we call them. It's the It's like the the metric of Prometheus, eh? Not a time series of Prometheus, the metric, the first part. it goes to metrics, it goes to nodes, so you have then the ability to say which nodes are anomalous. Which time series Which metrics are anomalous, eh? And then drill down

to individual to individual time series. Now, with this process, an AI can say, "Hey, okay, query from the last hour, the last 2 hours." It will immediately reveal which nodes are anomalous, which metrics are anomalous, and eventually it can drill down to the individual metric, to the individual time series. this is the beauty of machine learning. Machine learning reveals correlations that you have not configured. You are

not aware. So, let's assume that you have these 1,000 servers, and we observe continuously that, "Oh, this application on server one gets anomalous concurrently with this application on server 3522 to 52. [laughter] Machine learning is noisy. There are random noises over the place and I I will show you how we managed to remove all the false positives. But even with this, if you have two applications that

you believe are irrelevant, and somehow they get anomalous together, it means they are synchronized. You're not aware of it, but they are. Somehow they share some workload or they share a common dependency or they share something that makes them go anomalous together. This is very important because at the end of the day machine learning reveals and dependencies that you probably don't know. This is also very important

and this is a very good example. What it says what So, let's assume that we have a metric is the speedometer of your car. What do we learn? Do we learn the speed? No. It shows the speed, but we don't learn the speed. What we do is that we build vectors of the deltas. So, we collect every second and then we try to understand how you drive.

How you stop at traffic lights. How you accelerate to pass another car. How you predict what happens on the road. Are you a gentle driver, an aggressive driver? What you are. Now, machine learning learns this on the system itself, on its own metrics, and then it can can tell that oh, wait a moment, that vector never happened before. That's the first time I see it. So, this

sudden change in the speed of the car didn't happen. I I never experienced it before. So, that's an outlier. So, it's very important to understand that we don't necessarily learn the we learn machine learning learns the behavior, eh? It's the same thing like a passenger that he sits in your car, sleeps or whatever, relaxes, eh? And can tell that wait a moment, the last 2 minutes or

this event that just happened is not normal for you. Eh? You don't drive like that. >> [snorts] >> Now, how we remove false positives? Every 3 hours, we we train a machine learning model covering 6 hours of data on the past. This means they are overlapping, eh? Every 3 hours they have 3 hours overlaps. Every two of them have 3 hours overlap. And we learn 18 of

them. Why we learn 18? We learn we learn 18 because it it changes how much on the past we remember in the past we remember, eh? So, 3 hours * 18 is uh 2 days. Okay? So, we learn the machine learning knows has models to learn eight the last 2 days of your infrastructure metric by But then what what we do to avoid false positives because every

every model uh has 1% false positive. But only 18 need to agree. Only 18 need to agree. Let Let me go to the next. This. 1% false positive per metric, 18 need to agree, that's the probability of false positive. And when we go to host level anomaly. So, what's the probability of 30 metrics being anomalous concurrently? Huh? It's that number. The last number. So, this is how

we remove all the false Huh? Or most. In practice, there are a few because these are shared resources on the same servers, etc., etc. So, they're on the same workload. So, the false positive rate is not that that big, that tiny. But, the theoretical model is this. what kind of anomalies it can detect? It can detect the the most common, eh? So, I have a a sudden

a sudden spike or dive. I have a sudden change. So, it change it stair steps, eh? these things learn the the model is learn the periodicity of your metrics. So, a period is missing. Or a new period has been injected. So, all this is what it can do. It it usually detects all the all the anomalies, let's say, on the normal behavior of your metrics. And it

doesn't matter if your metrics are, you know, usually uh stable and gradually evolving or not. What it cannot do? Short-lived workloads. So, it needs to train a model. To train 18 models. If you have a workload that starts for 5 seconds and vanishes, what model? >> We don't have models. The second is crash services, but this is tricky. So, you have a service that crashed. So, no

data, no anomalies. But, all the services around it, every dependency will go nuts. So, although we cannot detect in that case a crashed service that this crash this service is the the surrounding gives you a blast radius for you to infer that come on, that's That's the common denominator, the common dependency among all of them. of course, it cannot detect something that has that grow grows extremely

gradually over weeks. Uh huh? So, the idea is that my machine this kind of machine learning is not, you know, one size fits all. It cannot detect everything. You cannot use it for But for operational needs, for the things that happen commonly in our infrastructures, this is magic. And the idea is that it is unsupervised, so you do zero for it. It just happens. It's stolen data.

That's it. Always on. now how we how we we we combine this agents? What we do there? The first thing is that as I said before is that if you have if you ask just one agent to do the work, give it all the tools, even machine learning, give it give will it be good? Can it find the root cause? The key problem, no. Most likely it

will not. Uh but why? Or it will not be a reliable. Sometimes it finds it, sometimes doesn't find it. So, what does it change? What is why this happens? Uh why Claude, Opus, or Codex 35 5.5, or whatever great model exists out there cannot do it. So, the problem is discipline it it needs to be consistent and have some discipline in order to to perform the kind

of investigations that need to be performed. Usually what happens with models, AI models, is that all they they have some good clues. Okay, I see something. Hmm, it's good enough. Let's give a response. This is what They don't have the discipline to go through all the signals that must be investigated in order for them to decide uh what is the root cause. That's the That's the default.

That's what happens if you open cloud code and What needs to be done? How do you enforce uh discipline? How do you make AI assistance consistent? we all know skills, huh? And what skills and sub-agents and all this kind of stuff. So, we try to do it in a way that uh enforces proper investigation, no matter what. And this is the design. So, we have a lead

investigator, an AI model. This is Opus or GPT. And this model, the only thing that can do is ask other models to perform specific work. So, it's just one. Now, the tricky part here is that you need a model that is extremely capable in investigations, in reasoning. And you need very fast and cheap models to perform all the work. So, for each of these work, there are

22 Uh for each of these work, the idea is simple. Establish a baseline. So, they the lead investigator will say, "I am interested about this. I'm interested about that." What a problem that it faces is an alert or whatever. Each investigator Each Sorry, worker needs to establish a baseline. How the infrastructure was before the incident. What's the baseline? What happened during the incident? And what was the

the new baseline after the incident? But then, they need to our playbooks, for example, will say enforce some kind of things that must happen by the workers and the investigator. For example, they need all to do to identify actors. Which are the interesting actors for this event? They need to perform hypotheses. Huh? What if this is the problem? That is the problem. The other is the problem.

They need to have a discipline to check before and after the fact. Uh so, the idea is that the way this has configured, let's say, in prompts and enforced with schemas in the output of the workers, is that we guarantee that a number of things do happen I said about this. Workers, so the idea here is one of the thing that is interesting because sometimes we tried

in the beginning with the workers knowing and having an opinion about the root cause analysis. This didn't work. So, suddenly workers said, "Oh, I found it. It's this. That's the problem. Take the evidence. That's the problem." And the lead investigator said, "Oh, okay. That's what the worker says. That's okay." That's your report. Wrong. It was not Huh? It was a hint for the problem. So, what happens

there is that we we configured all the workers to uh deliver facts, not opinions. All right? Don't don't say anything how to interpret the facts. Just deliver uh a list of facts. Now, this is for one of them, the process, for example, huh? The uh the data has the ability, you know, it collects operating system level metrics, but then it is able to drill down operating system

information, resource usage, etc. per process, per service, per container, per whatever you want, huh? So, this is one of the workers, the processes worker, that actually says that, okay, you are going to be given a work to be done. No matter what that request says, you check CPU usage per process, the top processes that consume CPU for the time frame of interest. You need to check this

IO. You need to check uh threads. You need to check out of memory kills. You need to check this. You need to check that, huh? So, there is a discipline enforced by the schema. If If the agent does not output the right schema, retry. You're not doing it right. If the agents themselves, because some In some cases, you know, we have seen this a lot in a

lot of cases, people do it. People introduce a problem. So, there is one one worker, for example, that is trying to figure out user activity. What users did? Okay, this happened. The server was rebooted, Was the kernel panic or a user that did it? Huh? And how these are related. The same with logs. Logs can be vast. There are workers for Windows logs, for Linux logs, journal

logs, but there are also workers for custom your application, your open telemetry So, the idea is that at the end I forgot to say that, this also an important thing. So, the way we structure the workers, the workers must know what they need to do, why they need to do it, and how they need to do it. This is very important. If you omit any of these,

workers are not good. So, a worker is just a a it needs a reasoning because it gets data and it needs to investigate, so it needs to it needs it needs to iterate through the data, but the whole point in order to do it right is for it to know what needs to be done, eh? So, it needs to do this and this and this and this

and this and this and this. for each one of them. And then, how. How is usually the nuances of the metrics themselves or interpret memory or processes like that, interpret whatever that is that that has some nuances in the metrics themselves. so, the idea is at the end, you have a reliable agent. These are by the way they are these are validated automatically. So, after each worker,

there is another validator that is running in order to ensure that this report of the worker was accurate and independent different model. Now, what what this ensures is that when the leading investigator says, "Hey, I want to know which processes I see a CPU spike on the system, I want to know consumed that or which containers." Or I want to know which name spaces on the containers

or whatever uh uh uh uh investigation is needed. What we have ensured is that the workers have a discipline. They will do all the work that is needed and transfer back the data that the the lead investigator needs in order to take the proper decision. Nothing will be lost. I can give you some examples in this. We have a Do I have time? What time? It's okay.

It's okay. So So sorry. Ah, okay. Great. So I will tell you an A customer a prospect customer comes and it was a funny thing. Uh these guys have a problem on a service or that the service crashes for the last year randomly in various And they cannot figure out why. The DevOps teams that they have uh they go through everything and again and again and again

and again. But randomly their servers the physical servers, not VMs, reboot. And when they came to us during the POC they said, "Okay, uh guys, look. It's very simple." the CTO. "It's very simple. Uh we have this problem. We're going to install in a data. We expect this to happen in the next 15 If you find it huh? If you find the we'll buy whatever. It doesn't

matter. This is irrelevant. So we were waiting for this to happen. >> [clears throat and snorts] >> the the the Swarm found it. So what was happening is the following. After a threshold, this was about sending back patching messages about something I don't know, SMS or something. It doesn't matter. But the idea is that there was a threshold under which So if the load was above that

threshold then auditd the audit demon because this was a regulated service. The auditd demon was incorrectly configured to panic the system. Right? If it can if it reaches the queue reaches 8,000 messages. Right? That's It's simple. There was a misconfiguration on an on a on a demon on a security audit demon that they never thought will reboot panic the server. So how what the swarm did, of

course most interesting part is that they didn't have a log server, so everything was there and because the system was panicking, the logs during the incident, the last few logs during the incident were lost. No no logs, nothing. Empty. And no copy other elsewhere. So what did what they did mainly because we we asked them we asked these agents to establish a baseline before evaluate what happened

during the incident and then understand the baseline after the event. What's the new status after the event? What happened is that the one of the agents, the logs agent, went through after the after the reboot so the reboot. So okay, it panicked, it rebooted. So it went after the thing and found out a log entry that says, "Hey, I am auditd. I am running with failure mode

equals two and queue length 8K." What? That's a panic. This assumes, and this is the interesting part, models have vast knowledge about uh how to interpret stuff, eh? So, they if you ask a model to give you an essay on what ODD is, it knows very well. The same happens with any technology. If an assistant an AI agent to give you uh the mental model of how

you should be managing Kubernetes, top of mind, it will do a great job. They will do errors here and there, small hallucinations, but the the core principle will be there. How How I manage this, how I manage that, what do I do in this case or the other. This is why the investigator, the least in the leading investigator, needs to be one of the best models you

can get, So, to have the least uh the least hallucinations possible. Now, what does all the this mean, eh? So, it's 3:00 a.m. the thing that we are trying to solve. 17 alerts are firing. What does it change? This is what change. So, you wake up, you click a few Today it's all reactive. So, we are working in to make it proactive, but today you can either

chat with this or uh click on alerts and ask investigate. It's a one button. What we're trying to do is to make this proactive. So, instead of you reacting to this, the the the system by itself to go through all the alerts, all the issues, blah blah blah, blah, come up with reports. We are thinking of this like This is still brainstorming, but the idea is that

we believe this should have like a status page. This is how you should view it at the end. Uh much much like all the uh SAS companies have a status page. Uh this should be a status page that is generated automatically by uh AI assistance. Um So, the idea is that this will give you So, if you if you check one of these uh I think there

is one. Ah. This is a very extreme summary. The reports are much bigger. So, the full report, for example, has all the information about, you know, which actors, uh what hypothesis we evaluated and why we dropped them. Um why we believe this is the case, what evidence we have, what is missing, what limitations we have, and we cannot actually say 100% what happens. What recommendations are there

for you as a uh as an operations team to deal with the situation. So, the whole point is that the reports I didn't forget to tell you of course this is about 3:00 a.m., but reports are of many kinds. You can have capacity planning reports, AI. uh investigation of anything. You can imagine uh on whatever. It doesn't need to be a fault or an alarm. A performance.

I have a post grace and I want to evaluate the performance. I didn't uh I didn't tell you that uh Netdata, apart from metrics logs, we are adding now tracing traces, has another feature that allows Netdata to get live information from uh its plugins. So, the idea there uh of course you collect, for example, for post grace the uh the metrics [clears throat] that most post grace

has, the logs that Postgres has. But, we are also able to select to to query Postgres for deadlocks, for slow queries, for top queries, for whatever else Postgres allows that is not metrics, logs, or traces. It's live information, probably. you can have as a list on your on your screen. So, much of the console, let's say, of the database, on Redis, on MySQL, or whatever, 14 databases,

actually, you can have them on in in a data. The data also has the ability to give you topologies with dependency trees. So, you can run a Kubernetes cluster, it will visualize by itself all that it takes all the sockets, finds who connects to who, and gives you topology of the entire The same, of course, for SNMP networks, what is real today? Today, all these that I

I show you exist. machine learning is there, troubleshooting assistants are there. They are reactive. Uh you can even chat with this. They are reactive. Of course, if you want to do it yourself with your own AI assistants, the cloud code, etc., Netdata has MCP server. So, the data are yours, of course, you can do whatever you want with them. What is coming is proactive monitoring. And we

believe that with SLO tracking and proactive monitoring, we're going to solve the alert the alert fatigue. the last thing is, of course, this is future, uh next steps. these are the things because we see a lot of vendors out there claiming this and that and whatever. what I will tell you is the following. The first is, if you don't have high resolution data, let me say the

other way around. When I started in a data, I believed observability industry has some cultural faults. This is why I started this. What are the cultural faults? Guys are saying, "Hey, monitor only what you need." Huh? Monitor only what you need. So, this means that I can predict everything I need. Huh? They say, "Dashboard only what you Uh again, so I can predict what I need. But

if you check all the reports, most of the world creates dashboards during crisis. Huh? 90% of of outages lead to new metrics being collected and new dashboards being built. So, this this is failure. For me, this is failure. This is no go. No go. So, what I said back then is, "Wait a moment, guys. The observability thing must collect everything. Everything. No questions asked. No configuration. I

I and select every single signal that exists and I collect them all in real time, every second, as a standard. We lower granularity only when the application that exposes the metrics has a problem with its rate. So, if that thing is low, yes, let's go gentle. But if the if we can collect the the and the information, the insights easily, in a lightweight fashion, then uh it's

uh we we go that way. per second. The next is that dashboards need to be automated. Netdata has algorithmic dashboards. So, you collect everything and suddenly, boom, dashboards happen. The dashboards allows you to slice and dice the data. So, you don't need to learn PromQL or another query language. No language at all, nothing. We have developed a little ribbon upon the chart. You can group by, filter,

slides that we you see fit, by label, by this, by that. So, with two clicks, three clicks, bam, you're done. The result is that every Netdata chart is the equivalent of about 25 Grafana charts. Huh? So, you have 25 Grafana charts, one Netdata chart. I am I'm telling you this because I believe that high-resolution data and having having the AI access high-resolution data, this is where the

the magic happens. At the per second level, you can see how the events the events cascade. Oh, this happened first, this happened second, this happened third. And that's it, guys. Try Netdata, free open source, 80,000 stars, 1.5 million downloads per Open for questions. >> Okay, thank you so much for your presentation. >> [applause] >> Uh we have quite a few questions. So, the first one is, do

you have any recommendations for how to get started? For example, any open source tools? >> Hm. Okay. To tell you the truth, in for [snorts] our agentic software that we use, we use it to also internally in Netdata, we have a CRM agent that is connected everywhere, to billing, to CRM, to ticketing, everywhere, everywhere. So, we have developed our own harness. The problem with most of the

harnesses is that you you need what we wanted what we were looking for is a harness that will allow us to focus on the prompts and the contracts and the constraints, not on everything else. I we don't need to to code. We need to to to to do anything else in there. We just prompts, constraints, period. Let it run. And the idea there with this is that

you need to to segment it in many many many many little parts, many little parts. Of course, each agent has its tools, has its prompts, has its output contracts, etc. So, we have developed our own. one of the things that I want to do is to open source that thing. We call it AI you can experience it if you go to learn the data cloud, you're chatting

with it. uh it's free for our documentation. the whole point is that people believe that this will derail the data of from its original observability natures and nature and this is why we have not open sourced it. Uh but I am sure that today, because this has been developed over over more than a year now, uh today uh there must be such tools. I don't know them,

but I believe there must be >> Okay, next question. If an AR co- sa- >> SRE >> Handles the on-called toil that juniors learn from, what does the path to senior SRE actually look like for someone starting just now? >> Oh, guys, wait a moment. AI agents need the extreme babysitting, extreme. It's not that simple. So, you don't If the AI agents are not fire and forget,

it's not like I wrote this piece of software. I know it takes it takes this inputs, gives that outputs. No. No. You need to think. And the biggest problem is you have an engineering problem, huh? You have an an assistant there and a model that can reason, but it needs to reason for the right things. So, it needs to have some Also, it has amazing prior knowledge,

but not everything is activated. So, you it's You need a way to enforce some work in this. And this is only trial and failure. This is only trial and failure. So, you cannot do it. Even if you are good, you cannot do it. Forget that oh, I built an agent is there serves customers. No. So, it's a lot of iterations all the time paying attention to the

little details. What it thought, why did that, why did do the other thing, uh So, to my understanding, AI engineering is a agentic engineering is an engineering task like software engineering or whatever that that has some different process, let's say. It's not just oh, waterfall. I developed, I tested, I delivered. Bye-bye, the application works. I know. No, this is not the true for for agentic engineering. So,

you need to be on top of it. The your experience makes it better. Your Um a slide mentioned the Minimax um 2.7 model. What types of models, LLMs or specially trained, are typically best for investigating the different signals? >> Um Yeah, so the whole point with these guys is that if you use Opus everywhere, it's going to be, I don't know, $10 per investigation. Eh? Every alert,

uh $10, $15, $20 per investigation. So, you don't want that. Obviously, it's wrong. What you need to do is find something that is extremely fast and reliable for workers. So, keep the lead investigator in the best model, in the most most expensive but use workers that are trivial, that can be called and called again and investigate more and get a lot of data and do this and

do that and they will not cost. Eh? This is the slide that we mentioned Min Max. What we found out is that actually we use we try continuously to identify this as snapshot. We found that Min Max is one of the of the best value models. It's reliable enough. It has low hallucination rate, the specific model, not the previous This 2.7, eh? 2.5 and 2.1 are a

different story. Uh but 2.7 has uh low hallucination rate, is extremely fast and cheap. Cheap. We have not tested now after Deep Six 4 V4, we are testing now Deep Six V4. So, we will because this is even cheaper Deep Six. Uh so, the idea is that you need Imagine this, if you if an agent runs or a swarm runs for every alert, we have a prospect

customer who had an interview here. We started the the POC last week. The guys had in the last the last 8 months 36,000 alert transitions. 33 36,000. You cannot deal with it. Eh? If you put a a swarm behind that thing, of course it will be so we we developed in order for to deal with its customers, we go and develop an AI assistant that actually says

which alerts they don't So they are configured wrong. You're not doing it right. This thing flip flip flops all the time and it's your fault. Fix it. Because otherwise the cost skyrockets, of course. >> Okay, for banking only traditional observability costs may reach 30% plus of cloud spend. How do you build a cost-efficient AI co-SRE without losing auditability and RCA in in a fixed budget? >> So

the first thing is the data is designed to lower the observability cost. It's one of the reasons that we developed the data data like this. So you can expect less than half of the cost of any other observability solution if you try the data. With the data the data remain remain as as close to the edge as possible. It's fully distributed. Why we did that? Because the

biggest constraint that all providers have is ingesting vast amounts of data and then trying to bill you. But if you keep the data as close to the edge as as possible then this storage is cheap. This storage is nothing. You can even keep keep if you don't need a high availability of your observability data, you can keep them in The data a single the data agent on

on any node with 3 GB of storage and if I remember correctly 15 KB per second writes can give you 14 days per second, 3 months per minute and Uh, almost 2 years per hour data for Uh, 3 GB. This is cheap. On every on you will you will not even notice it on the VMs. And actually it uses only 5% CPU of one of a single

core. And 200 MB of RAM. So it's cheap. It's very very lightweight with ML all of Now, with NetData the first thing is that it lowers the cost significantly of the observability. So 30% uh, it's not there. I saw a 30% of the cloud cost. No, guys. It's less than 10, huh? With NetData. The the this cost. Then it depends on the AI, it depends on how

much you want to use it. Huh? So if you want to delegate your entire team to this, probably it will cost you a few salaries. But what can we do? This is not our cost either. This is the the cost of AI agents, huh? >> And um, cuz we don't have any more time left, let's do you one more quick question. Um, how do we differentiate between

a signal being a symptom versus a root cause? >> Oh, that's easy. uh, it depends on the case, of course. So in some cases like the ODT problem that I told you before, the signal itself says that uh, so the the agent saw that ODT was uh, severely stressed during that time during the event. So it was obvious that it was a kernel In other cases, it's

not it's not that obvious. So especially if you don't have logs or if you don't have something for the assistant to to conclude that this is it, the root cause will give you a direction. What we know is this. We eliminated all these other causes. This remains. It can be either this or this or this, but we don't have the data to say for sure. You get

it? So, it depends. This is why actually you need more data. This is why 90 data collects all the possible data.