Project Lightning Talk: Safe By Design: AI-Powered Auto-Healing For SREs - Arik Alon, Maintainer

This talk introduces Holmes GPT, an open-source AI Site Reliability Engineering (SRE) agent designed to enhance incident response by integrating with various observability data sources. The speaker, Ari, demonstrates how Holmes GPT connects to cloud providers and monitoring tools like Prometheus, Loki, Datadog, and New Relic. The system proactively analyzes incidents and provides root cause analysis in real-time, significantly reducing troubleshooting time. The speaker shares a practical example where Holmes detected a critical issue after a deployment, identified the root cause, and even generated a pull request for remediation. By defining health checks in plain language, Holmes automates monitoring and response tasks, showcasing its capability to streamline the operations workflow effectively.

Hi everyone. So, a bit about me. I'm Ari and the CEO of Robusta and one of the maintainers of Holmes GPT. So, Holmes GPT is basically an open-source AI SRE. And what it actually means is it's an AI agent which is connected to all your observability data sources. So, anything basically that can be used when troubleshooting. So, it connects to your cloud provider, to your observability data,

to Prometheus, to Loki, to Datadog, New Relic, whatever you're using in-house. And then whenever there's an incident or someone is trying to investigate something, instead of an engineer going and digging through the data, you just ask the agent why do I have this pod crashing or this latency issue? And then the agent is going and querying the data and then within a few minutes you get a

real root cause analysis of why the issue is happening and how to fix it. And Holmes can also do remediation, which is what this talk is about. Holmes is being used by a lot of companies. We submitted it to the CNCF together with the Microsoft which is using it for the Azure CLI. Uh So, everything basically or a lot of issues start with someone want to uh

merge a PR. So, I've got this PR recently one of the teams wanted to a big change. Um probably most of it was done by Cloud Coder or some other coding agent. I reviewed the code, looked decent, I went ahead and merged it. And then something went wrong. There is a big drop in one of the in the application metrics like the bidding rate dropped 70%. Now,

in the old world, it could have take very long until someone noticed it. Um but luckily for us, we have Holmes which monitors our environment. And then within a few minutes I got a notification on Slack saying an incident detected by Holmes. And Holmes is telling me there was a deployment of deployment of version 241 and there is some critical issue. There is a drop in the

metrics of 70%. There is an a drop in the cache hit ratio and then in the latency there is a big increase. Holmes is showing me some visualization of the data. And it even went and dig deeper into the code base. It basically tells me someone basically set the TTL cache to zero which basically disabled the cache. And Holmes even done more than that. It went on

and opened the PR with actually a fix. And now all I have to do is go and review the PR and merge it and I'm basically done. Um so, this is great. So, how how this actually happened? So, basically we defined a Holmes health check which is basically telling Holmes what to do after each time I'm changing the bidder deployment. I'm defining it in human language. I'm

check the bidder deployment health, check the logs for error, check the latency metrics, check if more than 10% degradation in any of it. And then if you detect an issue, try to go on GitHub and fix it. And if you're not, just suggest a PR to to roll it back. And what happened in the background, we have the Holmes operator. Holmes operator is basically picking up those

health checks and then following the the instructions in it. So, in this case, it picked up the health check, it done some queries to Prometheus, it detected the metrics degradation, it checked Kubernetes for the pod logs and then went to GitHub, detected what the issue was in the code and even opened a PR to fix it. Now, in this example we've seen a PR remediation which is

basically a human in the loop. Um someone need to review the PR and actually approve it. Um but Holmes can be can integrate it with any remediation handler. So, it can go and trigger some GitHub action or it can go and run some Q3 cube CTL commands to restart the pod or to scale it and so on. So, it basically can run any remediation. Um yeah, and

that's practically it. If you want to ask a question or talk to us, you can meet us on the project pavilion or on the Robusta booth. Thank you.