From Noise to Signal: Using MCP Servers for AI-Driven Alerting and Monitoring - Devyani Kota

This talk addresses the challenges of gaining clarity amid the chaos in site reliability engineering. The speaker emphasizes the dual roles of architects and firefighters in maintaining application stability, highlighting the importance of error budgets and root cause analysis. They discuss the concept of alert fatigue caused by overwhelming noise in alerting systems and how traditional AIOps solutions often fall short. The presentation introduces a novel AI-driven agent, the template agent, which integrates with existing observability tools like Datadog and Splunk to enhance decision-making. By automating the triage process, this agent can prioritize alerts and provide context-rich responses, ultimately reducing manual toil and improving operational efficiency.

Today, we'll be talking about this talk which is noise to signal. That is, we are going to seek clarity out of the chaos that exists. So, in this world, we have architects and we also have firefighters. So, they both have their own role in this world to maintain a balance in the society. So, as important it is to you know, take care of all the design patterns

and come up with a blueprint and build an application, it is similarly important to maintain that application and to protect it from say possible bugs that come by or incidents that happen in production and a couple of other things. So, they both are equally important. So, uh I would like you all to focus on the third point, which is the most important point. That is, the pain

of toil. The false alarm fatigue that site reliability engineer has because of the chaos amidst which they have to focus on certain bug and the amount of time and duration that is spent on a single one. That is something that we'll be speaking about in length in the upcoming slides. And error budget, of course, the credits that we get assigned to and it's the responsibility of the

site reliability engineers to you know, take care of that credit so that we do not exhaust it within a single month, but rather over the period of time. Last but not least, I would also say that analyzing the uh ash and the incidents, that is, the root cause analysis that we do, those are one of the most important things so that a fire that is caused, it

doesn't repeat itself again. So, what is the problem that we are trying to solve here? It's too much noise, but not enough insight. So, if we have like we see in an org, we have an alerting channel, we'll have 100 alerts, then that is equivalent to having no alerts at all. Because which one should we focus on? So, that is which causes alert fatigue. And having a

having a couple of metrics will you know, looking at dashboards is not does not imply that we have an understanding of our applications and get to the root cause of an incident. So, this is what traditional AI ops, they do not fail per se, but they fall short in uh by a small margin. So, let's see what Agentic AI brings in and the true intelligence behind it.

So, let's consider a scenario. On one fine Monday morning, there's an on-call engineer and an incident comes by. They go through the docs, ADRs, runbooks, SOPs, whatever the source of truth that they have and they go through all of that and the observability tool that they are part of, the application logs and come up with a solution. And they present it and they work on it and

then they present it. But while the on-call engineer was actually trying to dig uh dig into the logs and figure out a solution and come up with something to fix it uh immediately, there was a backlog of requests that has been pending in queue and which stay unattended and that causes customer escalations and you know, uh uh a lot of um stacking up of those tickets which

get unattended and gets moved over to the next person who is coming on call. So, what does this end up in? There's a lot of manual overhead. A lot of toil on the on-call engineers. Yep. So, what are we going to cover today? We are going to cover the architecture and the patterns that exist, and I'm really excited to share this with you all. Uh the code

walk-through of the template agent that we have come up with, and a small demo where we'll be presenting on how the agent actually works uh in our day-to-day lives. Uh we'll also discuss risks and mitigations at the very end, which is also very important because um there there's always supposed to be a human in the loop to actually uh give permissions before executing that operation, right? Okay,

what is the high-level architecture that we have over here? Uh we have observability tools. We have like every org has various observability tools. Some use Datadog, some use SignalFx, some use Splunk. So, whichever observability tool exists, we ingest the logs from there, and the AI agent that is a template agent that exists, we engineer it to look into those logs and come up with a decision layer,

and then execute it based on that. So, the key patterns are done by the template agent, the RC analysis, and the automated postmortems of those incidents are done by the agent itself. let's uh so, this is the overall architecture of that um AI model that we are speaking about, where the MCP server, it takes in the logs, the metrics, the previous incident reports, the run books that

exist uh uh in the MCP server as the source of information, and they they have this a they have all this wealth of information that exists, which a human cannot come up with. Like, they cannot go through all of that. Like, they cannot go back and see, "Okay, 2023, we had an incident, and something of something failed, and such deployment version did not work out." A human

would not be able to do that. But, an MCP server, the template agent, it can very well go back and let us know that this has already happened once before and they can come up with those insights. So, how does the template agent How is the template agent organized? We have the agentic template is similar and we integrate it with our observability layer, which is say SignalFx,

Datadog, or whichever of your choice. And then, on top of that, we also have a validation layer. Because once the response is generated, it needs to be validated whether it has, you know, PII information or something that can not shouldn't be shared with the customer. So, all of those information. And lastly, we have a human in the loop. That is the final approval before executing those operations.

So, this is one of the module of say the Splunk dashboard that we have. So, you see not just this, a lot of dashboards underneath operate in the same way. They give a lot of information and they have these metrics, but what is the one the the answer the what, but they do not answer the why. That is why is the about cause and how are we

supposed to get past that? So, those are a few things. So, let's get to the demo. Before that, I would like to walk through the code once. So, this is the template MCP server that we have and template agent. I'll share the links at the end of our talk. So, how does template MCP server work? We deployment uh repositories, docs, examples, and the most important one that

we need to notice is the tools folder, where we add our SignalFx modules to actually communicate and add the logic and the intelligence behind it to work through the metrics and what to achieve based on those. The we have we can have a Yes, it it it work as an SRE assistant to retrieve those metrics and actually come up with insights for us to take it forward.

So, this is the overall um architecture. Mostly the code walkthrough. The most important module is the tools module where you can integrate your own observability metrics, the logic behind it, like DataDog, SignalFx, or Splunk, whichever. You can add it over here. And let's see the uh agent in action. So, I'll just run it our local. So, I'm just bringing bringing my server up and uh So, this

is how the agent looks today. So, we can uh ask a couple of questions to it. Say like uh how is our Snowflake environments performing today? Yeah? So, it uses these tools to come back with an overall information, that uh the number of credits that it's using, the performance, and the whether the uh status is uh fine or if there are any degradations that are possible. And

we can also uh compare the system from 24 hours ago. How are we doing? all of these uh all of these questions and queries that we have, what it's doing is in the back end, it's communicating with our SignalFx server and is retrieving the metrics. It's retrieving the Snowflake environment metrics and to verify whether the systems are healthy. So, we see that the trend it's it it

actually degrading from the past 24 hours and it comes up with a uh with a why that it it's recommended to investigate and uh couple of other things. So, I actually have already asked a few questions beforehand and I have kept it uh uh ready for demo purposes. So, we can also query like what tools are available, then we can ask like uh this using the search

metrics. This is one of the tools that the template agent that we have added using the search metrics, get the Snowflake query failures in the past 2 weeks. So, there might be one SQL query that is taking up a lot of resources of your uh um of your warehouse and a couple of other uh resources that it is exhausting and is not available for others to use.

So, you can pick up that query and you can being the human in loop, you can actually stop that query to restrict the resources from being uh exhausted. Again, um so, I'll go back to the demo for a uh for a bit because So, uh one other tool is I'll I'll explain a few of the tools that this uh template agent provides that we have added is

the morning briefing. So, suppose you come up in a fine morning and uh you need uh to uh know that uh an on-call engineer was uh available in the previous time zone, but now that you're starting your day, you want to know that what has happened while you were sleeping. So, you try to ask what is give me a morning briefing of my entire platform and if

there were whatever incidents happened, whatever fires were already put out to get to know uh to to uh catch up with what has happened overnight and to retrieve that So, this is one of the tools. The other tool we have is uh this name again the Snowflake dashboard. We uh retrieve information on how many active warehouses are there, whether what are their statuses and a couple of

Um using the SRE assistant, get the memory usage detectors, we can have set up SLIs and SNAs, and we can actually retrieve that pieces of information on how your system is doing and monitor its health. Uh this is the most popular tool um amongst like on on my team. Uh what's on fire? So, on one fine morning, if you want to come up and you even want

to know that what are the active alerts that you need to look at on priority, then this would be something that you can communicate with the SRE agent, and it would come back with information based on the rank that you need to take care of. Again, these are a couple of other examples, which are to retrieve any failures that exist. You can also ask it if there

are any alerts that are going on, which one should you fix first. So, um all of these are fun and games. You know, the template, we are communicating, and an SRE engineer is asking queries and questions on uh their systems and retrieving information. But how do we make it agentic? How do we automate this? How do we integrate it with our current pipelines and let the agent

do the heavy lifting on our behalf? So, what is the outcome over here? So, the trigger point is the alerts. The alerts come in on our Slack channel, and you know, there are so many alerts that we need to verify which one is on priority, which one needs to be fixed, and which one is something that can can be ignored. So, these AI-driven triage, so how did

our team come up with this? We integrated our MCP client along with our Slack API, and so every time an alert would be triggered, the agent would also get into the picture and come up with a context. They would add context to the alert, that what the alert alert has instead of the on-call engineer going behind in the back end and going through the graphs, going through

all that observability tools and coming up with an info, we given the agent to come up with the solutions. What are we doing over here? Observe and assist, not execute, just observe and assist. So, it's going to come up with the context and it would share all the graphs that exist and whatever issue is there and come up with a resolution. And it's on the on-call the

human that is there in the loop to actually make a call whether to execute that or not or to revert that operation. Uh So, let's see what happened. So, earlier we used to get these alerts like so, which you can see these are PagerDuty alerts, I suppose, and we would get all of these alerts and the alert channel would be bombarded with notifications, and we will not

have way to prioritize which alert to take forward and which one can be ignored. But now, with the onset of including the template agent and adding it to our workflow and pipelines, we have context-rich errors, root cause is also mentioned that a recent deployment is the one that has caused this issue, and you can actually go back and roll roll roll it back, and your system will

get back uh healthy. So, this is one of the examples that I picked up. And yeah, so this is one of the important things, the risks and mitigate mitigations because no matter how much uh uh how much governance and policies we add, we need to add these guardrails so that we do not allow the agent to take decisions on our behalf, which cannot be rolled back. So,

to add this, we have added governance policies on what all can be performed by the agent and what all cannot, what rollbacks and logging. So, how do we uh get the agent across? We uh, do it in a phased rollout, which is observe, assist, and only then act. So, the first two things, even when you guys uh, would use a template agent and connect it with your

observability tool, you need to add so that the uh, it observes for the initial few days, and then the runbooks are generated, and the solutions are added based on the problem statement, and use that to train the models, and then only then give it the authority to act on those solutions. So, uh, mitigations are the same, the human in the loop, we need human confirmation thresholds that

exist, continually retraining the models, and um, coming up with anomaly explanation. So, the key takeaways, if you have to take away anything back from this talk, it would be to understand the MCP, the template agent, and build your own AI alert agent assistant for your team, integrate it with the observability tool of your choice, whichever that might be, and uh, leverage the metrics and logs, the past

information, and the runbooks that exist to come up, leverage the past to actually come up with solutions in the present. So, what uh, what it will uh, what what will happen is that uh, the alerts and notifications that come up will actually hold our trust and will not be something that will be causing noise, but the ones that actually need our attention will be uh, will be

in front of us. Like in a fire, how you know the smoke actually hides the actual source of the fire, so consider this that uh, the noise would be avoided to remove that smoke and to focus on the actual source of fire to put it out. And uh, these are the few references. Uh, these are open source and readily available for uh, for all to you know,

use it, the template agent and the MCP server. And uh, yep, that's all from me. Thank you. >> [music]