Lightning Talk: Building AI That Ops Teams Actually Trust - Robert King

In this talk, Rob King discusses building trustworthy AI solutions for operations teams, which often struggle with trust due to past experiences with unreliable systems. He highlights the importance of validation when developing AI agents that can accurately identify root causes of operational issues. The speaker advocates for a modular approach, where smaller, focused sub-agents handle specific tasks, such as context gathering and telemetry analysis. This method allows for independent testing of each component to ensure reliability and traceability, overcoming challenges related to non-deterministic outputs. King explains the need for comprehensive testing that includes operational, end-to-end, and output layers, emphasizing the necessity of grounding tests in real-world scenarios to foster confidence in the AI solutions. He concludes by reflecting on the importance of transparency and gradual rollout strategies to build trust among engineers.

My name is Rob King and I'm a engineer in the Chronosphere division of Palo Alto Networks and I've worked in the observability space for well over a a decade now. So, the topic of today's talk is building AI that ops teams actually trust. So, if you're building AI for operational tasks, you're most likely and that is that ops teams inherently don't trust AI and honestly they shouldn't.

They've likely been burned by noisy alerts, black box tools, and also systems that just call themselves intelligent but aren't. So, when you show up with a an AI agent that claims to be able to pinpoint the root cause the default response is most likely going to be prove it. So, the AI might work but it needs to do that consistently and getting engineers to actually rely on

it requires solving a validation challenge. So, how do you prove to a skeptical ops engineer that gets woken up at 2:00 in the morning while. So, you know, the alarm goes off, their eyes are heavy, and they quietly creep to their laptop. And all the while they're considering, is this going to be a 5-minute quick fix or is this going to be the end of my entire

night? here's here's the entire problem. are non-deterministic in nature. So, if you give the same agent the same input twice, you're likely or possibly going to get two completely different answers. Both could be valid but one could be subtly wrong and in a way that only surfaces in context. So, traditional software testing assumes determinism. So, expected output equals actual output and that breaks down completely here. So,

how do you test something that has to be correct or can be correct in multiple different ways? I've played competitive chess, I've won a national competition, and I'm currently rated over 2,000. So, naturally, I spend my time debugging microservices. And a strong chess game isn't one brilliant move. It's a facet of multiple different things. So, opening preparation, um ensuring that your tactical calculation is accurate, strategic positional

evaluation, and then just in you have really solid end game technique. All distinct skills that work together. And if a player loses a game, you don't just say they lost the game, you're actually able to pinpoint exactly in the game where things went wrong. Was it a tactical miss? Did they strategically make mistakes on the positional side of things? Or did they just technique? So, it's important

to take the same approach building AI for incident response. Instead of one long monolithic agent, um that takes a problem and produces an answer, design a system of small, focused sub agents. One gathers the context, one forms hypotheses, one queries the telemetry, and then you've got the evidence evaluation that then leads ultimately to the root cause. So, each one of those is going to become independently testable.

So, let's make this concrete. This is a real validation scenario. service starts to exceed its expected So, the AI agent then kicks in and then performs a complete trace analysis, identifying all of the upstream and downstream services and and technical touch point dependencies. And it then identifies a downstream service in regards to where which problem originates. So, it navigates to that service, and it observes crash loop

errors. It then investigates the application uh logs, and it notes an increase in high severity log events. So, all of that is then actually, uh, associated to a feature flag flip that was made in production just was detected. So, each step there is a different kind of challenge for the AI. You've got pattern recognition, telemetry analysis, service navigation, and correlation. Now, most teams validate agents by checking

the final answer. Did it get was correctly? And that's outcome evaluation. It's useful, and it's, uh, a decent first step, but it is coarse. And while the answer is, um, you know, when when the answer is wrong, ultimately, you're not going to have an idea of in regards to which, uh, part of that trajectory was incorrect, which makes able to debug it. So, what you should be

thinking about, as mentioned, is trajectory evaluation. So, validating the full path that the agent took, not just where it ended up. So, that would include every tool call, every reasoning step, and every decision point. Did the agent query the right Uh, did it follow valid reasoning chain, and did it take unnecessary details during that process? And this is where that modular architecture really plays pays off. If

your agent is one monolithic chain of thought, you can't evaluate the trajectory in any meaningful way. But if each step is an component, you can score them independently. When we validate that when we validated that five-step, uh, scenario, each step was assessed on its own merit. And if the agent finds the root cause, but skips a step in that reasoning chain, the test fails. Because in production,

skipping a step means can't feel confident in the reasoning. And ultimately, the engine is then not going to trust the AI and its insights down the line. So, conceptually, it's beneficial to think about your evaluation in three layers. So, you've got the operational layer. Did each individual operation do We've got the end-to-end layer. Did the full execution path make sense in its entirety? And then the output,

ultimately, did the agent was it able to pinpoint the root cause and solve the problem? And you need a signal on all three. So, as an example, an end-to-end pass but with an operations failure, that ultimately key. An output for pass but with an end-to-end failure means that the agent got to the correct decision, was able to identify the root cause correctly, but ultimately, it took a

nonsensical path to be able to get there. So, if that agent was then to come across similar incidents in the future, evaluate them incorrectly. Oh. So, in terms of what you test against, synthetic test data is is again a good first start and it works for those kind of fine and happy paths, but it's really going to capture the complexity of a in production. So, a synthetic

test might give you a clean latency spike with an obvious database bottleneck, but in reality, that spike coincides with a deployment, a config change, or an unrelated set of services upstream that are throwing a whole bunch of errors. And the agent would have to work through all of that noise. We ground our test suite in real production scenarios. Our suite covers an ever-growing library of root causes

such as feature flag flips, bad deploys, resource saturation, and infrastructure issue. I could go on like lots and lots of different root causes. And each scenario includes multiple different challenges. So, it scale Our agent runs thousands of investigations per week, all in production, and it performs a complete multi-step reasoning for for each. Now, on the testing side, we've got well in excess of 600 tests, including hundreds

of golden trajectories, which are the uh the annotated reference paths. And so, when something breaks, the team within minutes. So, your engineers care about whether AI is ultimately going to waste their time or not. So, you have to dis- design for skepticism. This means showing your working, every recommendation, and all the data that it considered. So, you're not the the ideal is is not score, but actually

that full reasoning chain. So, if you've built using that modular approach that we just spoke to, that reasoning is real. Um and it's because it's ultimately how the system actually works, and it's not something that's reconstructed after the fact. Then you roll out gradually. You want to start in shadow mode. So, the agent runs along this. Um and they can then carry out their uh typical uh

problem resolution workflow, such as dashboards and single panes of glass. And you'll start to notice the AI is correct. They'll they'll start checking it earlier, and they'll that trust will build from observation, rather than the AI being mandated to them. So, we've ac- who was our the biggest skeptic. He wouldn't even look at the agent's output. And then one night, it pinpointed the root cause in under

a minute. That would have taken him over an hour typically. So, the next morning, he Slack'd the entire team, "All right, the robot's not bad." And now he checks it before anything else. So, to uh up, whether you're building LLMs or agents, um machine learning, AI agent validation and trust are are the real problems. Take a modular approach, test to operation, um ground your tests in real-world

scenarios, and show your reasoning in the full chain. So, if you get that right and you've built something um teams will actually rely on. Thank you for your time. My LinkedIn is on the screen. I'll be around all day. I never decline a conversation or a game. Thank you.