AIOps: Leveraging AI for Incident Root Cause Analysis - Sathish Kumar

In this talk, Satish, a senior principal engineer at Atlassian, discusses how AI operations (AI ops) are implemented in Jira Service Management to enhance automated alert and incident management. He begins by defining software incidents and root cause analysis (RCA), outlining how incidents are detected, investigated, and resolved. The talk emphasizes the importance of using AI in RCA, particularly in complex microservice environments, to analyze dependencies and derive insights quickly. Satish explains building blocks for an AI Site Reliability Engineering (SRE) agent, including context engineering, an agentic interface, and LLM orchestration, aimed at improving metrics like mean time to resolve (MTTR) through effective incident investigation. He also shares challenges faced while integrating AI into the incident resolution process, including issues related to data scale and interlinking various domains within incident management.

I'm Satish and I'm a senior principal engineer in Jira service management at Atlassian. Uh we work on a lot of products around support services, alert management, incident management, and AI ops. So, today I'll be talking about how we do AI ops in in one of our sub products. And and how we leverage AI effectively for uh automated alert management, automated incident management, right? The agenda of the

talk will roughly cover the following topics. Like uh Like most of us will have some idea of what is a software instance, what is a root cause analysis, but I'll still start with a brief intro before going into the details. So, we'll talk about what are software incidents, what is RCA or root cause analysis, and the role that AI can play in in a root cause analysis

both both in terms of like product in terms of in terms of operations, right? Like I'll be coming from the perspective of how we are building this into our products, but uh it's it's applicable even if you're like running production systems and you want to use it outside of the product, right? And uh I'll talk about the building blocks of uh how we've approached an AI SRE

agent and the challenges that uh we have encountered along the way. So, what are software incidents? Software incidents are usually like unplanned interruptions and uh degradations or any kind of abnormal behavior in your software services and applications. It usually manifests in the form of some reliability or performance or functional issues to your customers. And a software incident usually goes through these following phases. So, you have a

problem that is detected. Most of the time the problem is detected by your monitoring systems, but if your alerting or monitoring is weak, it can also be detected by let's say customer issues or support desk or even like if your product is like super popular, it can even be discovered on Twitter by your users and complain, it results in some kind of customer impact and the the

most important thing in any software incident is to reduce the customer impact or address the bleeding. Most of these incidents usually translate into something negative for the business. Like it can be revenue loss, it can be reputation loss, or some kind of like in worst cases even like data loss, right? And any software company or tech company usually goes through the following phases in order to effectively

manage an incident. Like it it varies from organization to organization. The the rigor varies, the process varies, the tools vary, right? But still some of the common things are there's an investigation step. There is a effort to identify the potential root causes. Then we try to mitigate the issue. Like short-term mitigation is usually done with a goal of arresting the customer impact without like even if you

don't find the actual root cause, right? And um once that is done, we want to update our stakeholders and we can spend the next few hours or few days finding out the actual root cause. I'll I'll go into details of what is root cause analysis and why it's important in this context. So, I'll be using the word root cause analysis to mean two things. Yeah. analysis in

two different contexts. One is from the perspective of during an incident, how do you find out the problematic code, the problematic infrastructure, or the error to address the issue. The second is what we do as 5Y RCA at the end of the incident. Like we spend couple of days or couple of weeks to come up with a rigorous 5Y RCA, right? I'll be interchangeably using it in

both, but it means both, yeah. So, what is root cause analysis? An RCA is a structured process to identify the reasons of why a problem has occurred. Um the the intent of an RCA is to understand the what, why, and how of a of an outage or an incident. And the most important purpose of doing a root cause analysis is to go beyond the surface level symptoms

into the most fundamental or the underlying root cause. And the objective that we're trying to achieve by doing this doing this is that you can prevent similar problems from occurring in the future. And 5Y is a very popular technique in terms of doing an RCA, right? So So the basic idea behind 5Y is that you repeatedly ask the question, "Why did X happen?" And until you find

the root cause. So there are a lot of uh conventions and good practices in terms of how you write a 5Y. It can't just be like some random 5Ys about five different aspects of the incident. Um so each Y has to ask a question which is trying to answer the previous Y. So that it's it's sort of like peeling the onion when you're trying to uncover the

root cause of the incident. And most companies try to follow a principle of blameless postmortem because unless you follow blameless postmortem, you're not going to identify the fundamental root cause and prevent the issue from happening again. And the the better your RCAs are, the better you get a holistic understanding of your system. Like even people who have joined newly, like leadership who have joined newly, get a

much more better understanding of the overall systems and the failure points in the overall system. Once you start doing sitting in more and more 5Y RCAs, right? So So eventually it helps you in from coming up with your architecture plans system resiliency plans based on what is what is a hotspot or what is frequently failing in your systems. And the the uber goal of any kind of

RCA system, be it a manual RCA system or a automated RCA system, right? Is is to attack this metric called MTTR, which is mean time to resolve. And the less time you take to resolve these incidents, the your you your customers are impacted. Like um what we're seeing with GitHub and Cloud these days, right? So, there are like different breakdowns in terms of uh MTTR. It is

based on the phases that an incident goes through, right? The As soon as the incident starts, the first thing that we measure is what is called MTTD or mean time to detect. Like did your monitoring systems detect the incident? Did a human user or a a user report the incident, right? And MTTD is usually good only if your alerting is good. The The next step is uh

Did the on-call or did the incident responder acknowledge the alert? Like most of the time the the time spent in incidents is about getting the right people in the room, and the right people, the right owner services, and right stake right uh subject matter experts in order to solve an incident, right? This is what we call as MTTE or mean time to engage. So, if you optimize

your mean time to engage, it's like the work is half done. Like you know which service to engage, which uh expert to engage, and which metric to debug, right? The next part is mitigation. Like mitigation is most important from a business perspective, because that is where that is the duration of your outage. You could You could do whatever hack it takes to fix the system, and resolution

is about architecture resiliency, right? Like what are the long-term action items that we're taking after discovering the underlying root cause of the incident. And And all of these are important. MTTR is the uh uber metric, which which kind of represents all of these subparts. So, now that we know what is software incident, what is root cause analysis, let's look at what role that AI can play in

in root cause analysis and software incidents. The Like when you're doing a root cause analysis of a complex microservice environment, like uh typically large enterprises deal with uh hundreds or thousands of microservices and their dependencies like databases, cache, and message queues, right? You you're dealing with a complex environment where you have to understand the dependency of multiple changes. So, there is a challenge in sifting through change

logs in the form of deployments, pull requests, commits. And And what is shown here is an approach used at Facebook for divide and conquer of changes. Like every time there is a faulty broken build or faulty release that happens, they do a process like divide and conquer to to batch through the changes and find out the fundamental root cause. And how can AI help us irrespective of

whether we are a small small enterprise or medium enterprise or large enterprise, right? LLMs have very good code understanding and they can look at a PR or a commit or code code snippet like a diff and try to conceptually understand what it means. So, using this, they can semantically relate how this commit or code change could have led to a certain incident. I'll go into details of

this later. This is just to motivate on the different signals that that are important from a AI root cause analysis. The The second most complex part of root triangulation. Like how do you triangulate an incident with a code change with an observability signal that indicates that something is wrong. So, the the challenge here is that we're dealing with lots and lots of observability data. Depending on the

scale of the company, this is this can literally be like terabytes of logs and uh terabytes of metrics, right? So, the different kinds of signals here are collectively called as melt, which stands for metrics, error events, logs, and traces. Like alerts, sentry errors are are a kind of error events. And the role of AI in helping us manage this voluminous data is that it can help us

in pattern matching, anomaly detection. And most of these systems do not have very straightforward queries like your analytics database. You cannot just run a SQL query on a Prometheus database, right? So, it helps you in understanding the problem, trying to formulate like what is the right query that you need to run, >> [snorts] >> and and run those queries. And there is also correlation causation. Just because

there are like 10 5xx errors at a point in time doesn't mean they're related. You have to know which error led to which error. Like the kind of output that you you you would have at the end of a human RCA of 5Y, that's the sort of thing that we're trying to do in minutes. Like by by relating those 10 5xx errors. And the other big challenge

is that there are complex microservice architectures. Like most enterprises have hundreds or thousands of microservices, and and they are connected in complex ways. They have sync flows, async flows using SQS and Kafka. Each service like like we might in our head we might know that a service depends on let's say five services. But once you start looking at the network graphs, looking at the traces, you start

realizing that you depend on like 20 or 30 services which you did not even know in the first place. And the the goal of RCA is to come up with what is called as a fault propagation graph, right? Uh you start with an outage, like a business outage, checkout is not working for bank XYZ. Like checkout is failing for bank XYZ can be a business error. How

does a fault propagate all the way from a customer-facing cart cart service or checkout service all the way to the underlying service? The The purpose of this fault propagation graph is to show you in your system how this is actually happening. The The role of AI here is that it it kind of grounds your investigation. This is sort of like uh the context graph for AI to

investigate in the right places instead of getting confused. And there is also unstructured understanding in the form of what does a service actually represent. Like a promise engine service can be different from what is a procurement service, right? What do the What are the concepts that they represent when you're facing a checkout error? So, those kind of relationships are unstructured relationships also have to come from the

complex dependencies that you're seeing in production. And that's not all like just sorting these does not end the story, right? There can be like various other changes like you can have static feature flags, you can have infrastructure as code changes in deployment.yml, Terraform, or like something being merged does not mean it gets deployed, right? There are complex deployment topologies like canary deployments and progressive deployments which make

the whole debugging story much more difficult. Like some of the examples that we have seen is there could be a deployment that happened like 1 month ago and the feature flag got rolled out 3 days ago and that caused the outage. So, it it doesn't mean that once you deployed a feature it's it starts immediately impacting and timeline correlation is is what it takes, right? So, let's

let's look at what are the building blocks of like if you have to solve all these challenges, what are the building blocks of a AI on-call or a SRE agent that that needs to solve this? Like I'm I'm using the word SRE here a bit loosely. An SRE does a lot more things. An SRE writes code. An SRE improves systems as they go. I'm just taking a

very narrow use case of during an incident what does a on-call person do in order to resolve the incident or what does a SRE do to resolve the incident, right? It doesn't talk about any other aspects of an SRE. the principles are there is first principles thinking. Like humans have a runbook. Like you you can have a specialized runbook in your confluence or documentation which says this

is how we solve issues of type X in the system. Like when there is a rate limit related error, this is how we resolve it and it can be codified into a runbook. But when you're giving it to an AI, it it can come across novel incidents. It can come across recurring incidents. Your runbooks are going to be completely useless when you when you come across a

novel incident, right? So, that is where some generic runbooks will help. So, there'll be a typical runbook which helps you how to deal with code change related incidents. Like incidents where code change is the root cause, incidents where feature flags are the root cause, and incidents which can be explained by metrics, logs, and traces will have a typical runbook, right? So, uh the insights that we've gotten

is that you need some kind of runbook. A runbook is like a like a cloud code to-do list here, but except that it's it's trying to debug in And uh you also need to correlate with these runbooks, right? Like, based on the incident, you need to have a dynamic plan. And you're trying to correlate between a 5XX error in your Splunk with a connection pool error in

your metric with some code change that happened like 3 days ago in in GitHub PRs. So, this kind of correlation and triangulation is happening all the time. And and the biggest challenge with doing this in AI is that a human expert will have tribal knowledge or subject matter expertise, whereas the AI is always starting from scratch, and it has to somehow find the shortest path from the

symptom to the underlying symptom. Like, the way the what you're effectively doing in a 5-way RCA is causal chaining of symptoms, and you're expecting AI to do that from scratch every time. So, do triangulation becomes a challenge with AI. I'll I'll show a typical architecture. This is like a representative architecture. I've not used any real systems here, but just very high-level systems, right? The three building blocks

of a AI SRE agent is One is the agentic interface. The second is the context engineering part, and the third is the LLM orchestration piece. I'll I'll zoom in on this. Most probably, this is not visible here. So, the first part is the interface, right? Like, um an on-call engineer or an SRE engineer receives alerts and incident notifications from an alert management system. In Atlassian, we have

Jira Service Management for alerts and Jira Service Management for incidents. You can plug and play that with any any tool. And these alerts are actually coming because a threshold got breached in an observability system, or a a says something is broken, or even employees find out that something is broken and raise an incident. Once a notification comes to the on-call engineer or SRE engineer, they have like

multiple interfaces to start working on the problem. Like we have RCA as a product, like root cause analysis as a product, and there is a product UI that on-call engineers can go to. We also have an agentic chat called Rover chat, which is a which is similar to like ChatGPT for enterprises, this can this also has custom agent custom tools where the incident investigation can start. The

third one is we have an equivalent of cloud code called Rover dev, which is like a CLI-based interface. So, you can just substitute this with anything, right? Like cloud code or cursor. And this is sort of like the agentic interface for someone like a on-call or SRE to start debugging an incident. The second part is context engineering. one of the new ones in AI is that in

traditional ML, the more data you give, the better the system performs. In normal in LLM-based AI, the more data you give, the worse it performs, right? So, the the challenge is all about finding the right context and feeding it into the AI. It can easily get distracted if you give it the wrong context. So, we we have three ways of giving context to the AI. First is

what is called as service catalog. So, think of service catalog as a what a one-stop service dashboard for your company where information about any service, metadata, services, owners, on-call can be found. Like a canonical open source example of this is backstage, and we have a product here called assets which does this. The second one is teamwork graph. Like teamwork graph, you can you can think of it

as a context graph that that we built. And teamwork graph gives you context of what is the work that has happened in Jira, what is what are the RFCs, tech documents that got written in Confluence, what are the PRs that got raised in Bitbucket or GitHub, and what are the alerts that this service is currently receiving? So that is a one first-party context that we have, but

we don't just stop there, right? Like Teamwork Graph is trying to build a like a context graph of of entire work, like So it it's not restricted to Atlassian. So we go beyond that into Google Drive, Salesforce um Sorry, Salesforce is a bad example here. Like GitHub pull requests or or even your Dropbox documents and whatnot, right? So the integration service pulls all of this and and

builds a hundreds of billions of object graph, which which is like multi-tenanted per customer. And this graph is accessible to you whenever you want to get context. Like let's say I'm dealing with a checkout error. I can find out what are the recent payment gateway-related changes that happened in Confluence, discussed in Slack, and where have a have a PR in GitHub, right? So all of this information

is something that I can retrieve in in a couple of seconds using Teamwork Graph. Like uh I can also mean that the agent can retrieve it on demand using the Teamwork The third part of context is the orchestrator. You're not going to be able to get all data you want upfront. Like there is always going to be some data that you want to pull on demand. So

that is where tools like MCP CLIs and API calls will come in. And this is especially useful for us when we're dealing with large-scale data like metrics, logs, and and The the third important part of the architecture is is the LLM orchestration itself, which is once the RCA back-end service has received a received a question, the question can be about how do you uh like what is

the root cause of this incident, which is a end-to-end question, or it can be something very basic like which service has a high number of alerts right now, which is a query-based question, right? So these queries are answered by the RCA agent. The RCA agent is made up of multiple sub agents and multiple skills. An example of a sub agent is something like a hypothesis or reasoning

agent or a planner agent. An example of a skill can be something like change analysis, metrics analysis, log analysis. We We use a lot of custom ML models here so that it's it's not just like MCP integrations and tool calling. the agent is connected to some kind of gateway in order to in order to decide the next tool to invoke or the orchestration to do. good data

often leads to good machine learning and AI. And Teamwork Graph is at the center of how we do good data, right? So, you can think of Teamwork Graph here as the connected data ecosystem for AI. Like the way rag is used in AI, we use a graph rag for this. And at a very high level, it is it is made up of nouns and relationships. Any first-party

entity or third-party entity is converted into nouns. And the relationship between those entities are converted into what is called as graph relationships. And And we have an interconnected system of nouns for work. Like we call this as our system of work. Like all the Atlassian products are somehow connected to this whole Teamwork Graph and system And in the context of RCA, what this gives us is a

subgraph of deployment entities, pull requests, commits, Jira issues, repository, service. Like the catalog contains a repository and the service. And it helps us in grounding our investigations in facts. The The other part that that we don't actually own, but it's available in most of the observability systems and network monitoring systems is the service graph. So, a service catalog like Backstage or a service graph like a New

Relic service graph is the is the system of record for And it gives you information about what are the upstream dependencies of a service, what are the downstream dependencies. Let's say you had an outage where checkout is not working for bank X and the underlying cause turns out to be something in a dependency service like promise service, right? So, these systems will trace the dependency from your

user-facing checkout service all the way to the problematic service. And we get this information from SOR, but we also try to store this in our teamwork graph, so that at the time of incident, we have a holistic view of things. the end goal of this data is that we're trying to construct a sub graph which the AI can use. This is actually like a pretty big sub

graph. it it connects the incident with the service. It connects a service with all the changes like which repository is the service hosted in, what are the PRs, commits, and deployments that happen on this repository. It connects a service with a service. And and all of this is queryable in a uh like you can think of teamwork graph as a Neo4j like graph which the AI can

use anytime it wants. And and the last important part of building block of an AI SRE system is the agent orchestration itself, right? I'll I'll start with a logical view. So, in the logical view of an RC agent, it had it is connected to like three different sources of data. So, there is change data, there is system knowledge, and then there is observability knowledge. And majority of

this context comes from our teamwork graph context. And whatever is not available in teamwork graph, that is where we start using MCPs and CLIs to to to augment it with the additional So, an example of change data can be GitHub or Bitbucket deployments, PRs, and commits. And static feature flags and Jira issues. So, the teamwork graph connects all of this and and gives a how does change

influence the current The second part is a system knowledge, which is there is structured knowledge and unstructured knowledge of a system. An example of a structured knowledge is your backstage system backstage software services and software service But anything that is config driven is not going to going to reflect the real world. So, this is also connected to active systems like neural network service graphs or uh distributed

tracing based service graphs, right? So, it it reflects the real world and it's not some stale JSON that that is present. An example of unstructured data is architecture documents and RFCs and PRDs and confluence which tell like what each subsystem is all about. the teamwork graph is obviously per tenant, so it is grounded in the unstructured and structured knowledge of that specific tenant without using world knowledge.

in addition to world knowledge, right? The third part is whatever data is not available to us, we we depend on MCPs and other systems to integrate that data. Like things like uh Prometheus metrics or Splunk logs are something that come through on demand on demand queries, right? The end result of the RCA agent is that it tries to come up with multiple hypothesis, like a ranked list

of hypothesis. It's It's not just integrations and data here. There is also like obviously some amount of intelligence supplied at each place to know how to traverse all these dependencies, right? So, the structure of this is that it's made up of multiple subagents. It's made up of multiple skills. And for data that is not available through the context graph, it depends on multiple MCP servers. And there

is also some uh internal intelligence like in the form of ML models and uh ML models for changes, ML models for anomaly detection, ML models for entity understanding, and so on. This is like a typical stack that we use subagents and skills. It's It's just a layered architecture, which means that there is a agent harness or runtime. You can think of the agent as uh as a

chef here. The analogy is a chef. And the agent harness depends upon multiple skills and multiple tools. You can think of the skills as some kind of recipe in prompts and some kind of recipe even in code, right? Like prompts cannot express 100% of the scenarios that a complex system is trying to solve. An example of skills here is again how do you do feature flag analysis?

How do you do uh pull request analysis? How do you do metric analysis? Log analysis? It can't just be expressed all in English. So, it also needs a connect collection of MCP servers which translate that into uh API calls. And you also need to connect to your backend services which are acting as the intelligence layer, right? So, the interfaces like MCP CLI or function tools and even

the backend services act as like the like the analogy to a chef here would be that these are the ingredients and appliances which are actually used to get the job And this can contain the intelligence that you're using in order to do an effective RCA, right? And obviously there is evals and stuff like that which I've not shown here. So, every problem will require some kind of

evals data set that that is being used to solve solve And that concludes the building blocks. So, to summarize the building blocks of a AI SRE agent are one is the agent orchestration, the second is the data that you're using it up for grounding and context, and the architecture elements like context engineering and uh LVM orchestration and the agentic interface. And the other part is how do

you actually represent runbooks? How do do an investigation? I'll briefly talk about some of the challenges we've encountered before going into questions. we've encountered a lot of challenges, right? Like uh some of the significant challenges in change-based analysis would be like these there are causal relationships. Like these are uh unstructured relationships. Like your problem is in a different domain language, your features are in a different domain

language, right? So, you're trying to interlink a problem domain with a feature domain. The second kind of problem is that a service graph is not enough. Like a service graph does not tell you which API calls which API, right? It it just tells you that service A calls service B. And the third is that uh like because it's a product like the the third part will vary

between a product solution and a platform like an internal solution. Because it's a product we we deal with different levels of code understanding. Like we don't get access to the entire GitHub code of a customer. We get access to maybe the PR description and the commit description. So what you can do with code understanding and without code understanding are widely different. Like some customers give us code

understanding code access, some customers don't. So deep code understanding makes your RCA way better. But the challenge is also that you have to do it in a repository. Like what we are trying to do is a RCA without like it's not like we are using cloud code to login into repository and do an RCA, right? We have thousands of services and maybe like thousands of repos and

we are trying to do an RCA across these thousands of repos. Observability has its own set of challenges. Like first is the scale the sheer scale of observability data. And second is that most of the MCP tools for observability are very low level in nature. Like it's it's mostly about uh how do you get metadata? How do you get raw data? How do you construct a query?

It's it's that that is not enough for you to translate from a problem to a query. And the third part is you need to come up with way lots and lots of dynamic plans depending on the data sources, Yeah, these are the list of challenges that we've encountered in this. Yeah. Yeah, thank you. >> [music]