From Alert Fatigue To Self-Healing: Building AI-Enabled Control Planes... Nuno Guedes & Yury Tsarev

This talk discusses the integration of AI into control planes in banking, focusing on enhancing operational resiliency and compliance. The speakers, Nunu and Yuri, explain how they utilize Crossplane v2 and AI-driven operations to address issues like alert fatigue within a multi-tenant environment. They demonstrate an AI-enabled control plane that autonomously diagnoses problems, implements remediation, and identifies cost optimization opportunities in cloud resources. By leveraging AI for low-risk, repetitive tasks, they showcase how banks can maintain regulatory compliance while improving efficiency. The session highlights the careful methodology employed in deploying AI, ensuring that operations remain bounded and auditable.

Welcome everyone. This is from alert fatigue to selfhealing building AI enabled control planes in banking. And the reason why it says banking is Hi, I'm Nunu. I'm head of public cloud at Millennium PCP which for those of you who do not know is a bank in Portugal. And with me Yuri. Hey everyone, I'm Yuri from Abound and we are creators of course plane project. Super excited to

be here. Please kick us off. >> Thank you very much. So, first of all, some warnings. Sorry, this session may use AI. With that out of the way, it does use AI. The promise of us is you kind of know these boxes signals. Let's keep the reason signals decides reconcile. If you know crossplane or other universal control planes, you're kind of familiar with this pattern. Something happens,

there's an event, there's a decision being made. For instance, drift reconciliation and something reconciles. Today we are putting another box in the middle which sometimes reasons and why following the story that hey we're a bank. We kind of have a lot of regulatory compliance things to take care of. It's people's money on one side of the room and on the other side it's AI. Uh we do

not want to hallucinate and we have things like specific regulations things like the digital operational resiliency act in EU. We regulators keeping up with what we do. have operational resiliency honestly a board topic. It's there there's there's a lot of people involved there. We cannot harm customers. We cannot have regulatory exposure. This means auditability, controlled change, predictable remediation. So it's perfect fit for AI, right? The reason

why we pursue the AI story still is well, you know the story, right? challenge doing it manually as you grow in the size of your platform, in the complexity, in the in the service count, uh in the amount of signals you have to process, it just doesn't really scale and and this is something we all know, right? It's not know something new. It's always a story about

toil. And I'd rather have people focusing on important stuff instead of here's incident number three and number four and number five and no cleaning up cues instead of going ahead of the curve with the additional topic of what happened, what changed, who approved, what was the outcome. All of this being tracked, being auditable, being shown to everyone, including the regulators. Some years ago, we could say we

had an observability problem. We're not there now. Perhaps we even have, sorry, too much observability, too many events. We have a big hose of of everything that's happening everywhere. And the side effect is is it done yet? Is it done yet? Have you fix it? That's a P1. That's a P2. You need to go after it. You need to do it. a pressure on people that should

be focused on kind of a different pattern to do stuff in instead of just being a choke point in this process. And the way to address this that we chose is the typical control plane approach. And sure, we could do a crossplane story on crossplane adoption and all of that, but the fact that there is an observed state triggering and showing intent, receiving and showing intent, which

is something that Judy is going to drill a bit into later on, is the way we chose to reduce this pressure on SRRES and operators. And the cool thing here is with crossplane v2 we are enabling day2 operations. some of the reactive work, some of the optimization work, some of the observe and react patterns are now going into the control plane, focusing people on other stuff, eventually

some more important stuff. And also with crossplane v2 we are now implementing these operation patterns because everything we do is very multi-tenant. We are also implementing those patterns in a best fit for workloads approach. So workload equals namespace, namespace service level equals operation patterns. Now we're putting that down. This is a critical system. It's running in that name space. These guys get this sort of automated Yudi,

you're going to show some cool stuff around this, aren't you? I probably should. Is it time? All right. So, uh, first thing first, I'm going to check if the terminal is visible. Uh, I ordered it hardly this morning. So, um, uh, the setup, right? So we have a control plane and we already running some Azure caches that we created together with Nuna for the bank. Uh we

can assume that it is already standard high quality crossplane obstruction. So it's not the main part of the story. Uh what we are going to demonstrate is this AI injection part. What we're running next to the uh to these uh XRS names by XRS is so-called watch operations. And these watch operations obviously in working some standalone operations. So what is this all about? So if you check

the code so for example the first one is stuck diagnosis operation right. So we have a watch operation spec uh we have a standard uh way to control the things and we have a pipeline. So what we're running in this operation is a function open AI and we deploy it in Azure AI as a model. In this case, it's a G GPT4 mini to not to pay

much for the decision- making. It's really cheap, right? Like when I was running it for active experiments, it was around five bucks a day. So, uh, and it will be cheaper in a standard operation, right? So and we run uh function open AI with a system prompt and we tell to operation like UR crossplane cache diagnostician tool and we creating the prompt how to diagnose the stuff

the interesting part how we already can benefit from the corresponding within within a control plane. So in this case we're running watch operation and we washing these caches our custom and BCP cloud caches and we are not washing all the caches we only matching labels where we enabling explicitly AI operations. So we we are fully under controlling what is happening within the control plane and we invoking

AI only intentionally. Uh this prompt like does diagnose stuff like a kind of debugging. You also can generate this prompt and that's what I did with AI and test it right so that you are under full control. So let's test that stuff out. Uh again we have a couple of caches right one of them is stuck for some reason. So I have a demo script that will

help me uh uh to run the consist consistent demo and like in the before stage uh we are we running crossplane in standard operation we observing that one of the cache is stuck and now uh well we don't know why we will have to debug it so we on our own and that's a classic we'll have to drill in XR composition logic and all that stuff figure

out the event stream uh we will have to uh debug manually instead of that uh with AI enabled operation watch operations that I just demonstrated we can uh investigate uh the annotations so uh this annotation is set by watch operation by actual operations that invoke by watch operation on the on the cache right so it's there is no magic if you look at the uh at the

spec uh so it will be cache demo stack and I will pop it to yq two uh standard spec uh composed objects right and this is di diagnosis uh annotation so our AI enabled operation looked at the spec and figure out that there is some bug and it actually knows what to do what to patch there is a there is a bug in a spec so uh

it's already good knowledge right so we as an operator imagine we are we are devops engineers right we are looking at the cach and we want to fix it uh with this kind of readon information we already know what to do. Uh we definitely can either fix it manually or uh we can activate remediation and remediation uh will look like the following. So before I uh activate

it uh I will just uh shortly show that like it's the same watch operation with slightly different prompt and the cage here that we are going to watch for the same uh caches but now it's like a double filter of the labels. Uh it's AI enabled and auto allow auto remediation true. That's how it got into the filter and it will invoke auto remediation logic. So let's

uh run the remedy. The script will just set the label on this uh uh on this cache and let's see how it goes. Cool. So now uh we got a uh auto remediated annotation uh in uh our annotation list and we can double check that uh the job was actually done uh in our demo stack spec and yeah it was uh parameter id uh was patched to

the proper one. Nuno can tell what means I don't know. So but so it's a custom enterprise logic. Uh so AI helped me uh I was able to fix the cache even without uh internal knowledge of the um implementation of this of the cache and uh uh not even necessarily knowing the semantics of the spec. Cool. Now uh as we have uh this stage like intelligence assistant

uh gave us diagnosis annotation with a recommendation how to fix stuff remediation acted on it so like mitigated the incident uh we can build on top of it some interesting things like a cost optimization for example. So we have this uh second cache uh running uh that is something costly uh again situation before uh before AI right so just manual observation we look at the spec which

is parameter qxl we might guess if it's it is kind of looks slush but we don't know how much does it cost uh should we downscale it or not so we need to investigate again manually instead of that we can use our uh watch operation uh that does a cost analysis for us. It's the same uh same principle watch operation you can look at the its effect

cost after. So it sets it sets this recommendation right downscale from XL to L and save some bucks. Uh again implementation is uh pretty straightforward. If you look at the cost analysis operation watch operation uh AI enabled and or cost analysis enabled so additional label right so here we kind of hardcoded the cost data right you can imagine that we can we can retrieve it from some

database or external source. So for the demo it's part of the prompt to make it pretty much constrained but the principle works in any kind of complexity scenario. So now we have the cache uh that is uh uh analyzed uh and we have a cost recommendation right so let's check its spec for uh to keep it true right so again the standard spec and cost analysis cost

recommendation to downscale and uh we can follow recommendation manually but why we have a another watch operation that can actually downscale it for us uh and Uh if you look at the operations spec it is cost optimizer and it's going to react when we have we put the third label allow cost optimization true and then react to the cost analysis cost recommendation. So let's run the cost

uh optimize it will effectively put the label on the demo costly. So demo costly is labeled and we wait for downscaling event. Okay, we can trust annotation that it's cost optimized but it's better to double check it and uh uh get uh the full picture if it was true or not. Okay, spec parameters and SQL. Uh so it basically followed the sound recommendation and downgraded the cache

for us everything automated uh with u uh uh with the prompts and operations that under our full control right so uh that's roughly the demo uh to recap uh we have intelligent assistance like a readon recommendations and intelligent control where we can uh ask our AI enabled operations to take over and mitigate the issue or act on some recommendation like cost optimization the core um promise of

the crossplane right so in a core crossplane you abstract your complexity away you create your own obstruction your own databases your own anything caches your own infrastructure this idea is uh uh scaled to the AI scenario where you inject just enough AI in form of uh crossplane v2 operations and you also like under full control you uh uh you inject only uh minimal AI that uh is

required for uh control plane to be intelligent. You test it, then you promote it, you run it in read only mode, then uh when you trust it, you promote it to act on your spec in a readr mode and mitigate the issues automatically. So that's the overall idea. Thank you so much and back to the back back to the slides. Yes, stage is yours now. in summary

what we saw you get AI reasoning with a very bounded proposal. There's no change outside of what's being typed for that specific operation and the rest is typical crossplane running. And again you saw simple operations. You can imagine that a larger implementation would ensure things like logging evidences, making sure there's an evidence trail, no who, what, when, all of that stuff. And so the way we are

keeping it safe is schema bound actions. No, everything is pretty much well defined. The boundaries are clear. the constraints on policies uh who can do what when or for instance you know it's a good example here in the slide change windows no maintenance windows when can that operation take place we never start by coding an operation the operations you see come from real life experience it's by

observing what people do by converting practice into that we pick that experience. We know how it should really behave because we're seeing people doing it and offloading it not the other way around. And again audit of course here in the demo you saw GPD40 in our case it also makes sense to look at running some local models for this not only cost but there's always stories around

uh data sovereignty and that sort of thing. So some of this should in some cases run locally. And just highlighting something you saw in the demo. The efficiency of the reaction. This isn't pulling stuff. This isn't I'll go check every 30 seconds if the state changed, if there's a drift, if there's an event or whatever. We are reacting. Now we are subscribing to stuff and reacting to

stuff. So it's pretty efficient when we are scaling. results what changed? What didn't change? Automate lowrisk repetitive tasks. Yeah, that's it's simple to put it in the slides, right? The the challenge in real life is saying what exactly low risk means and what's repetitive enough. But then again, observe there's no rush. No, observe. No, if we take care of the 20% that's really repetitive and has low

risk. Sorry for being repetitive when I say repetitive. Um, it's a pretty good help and it's a good way to start. And if you're if you're not completely comfortable with saying, you know, identify the issue and fix it, just the fact that you get an annotation with, hey, there's an issue here. You need to fix this. It kind of speeds up the process, don't you think? Because

at 2 a.m. when when your system is down and you need to know bring it back up if you're if you build a habit of hey I'll look at the annotation. Oh yeah yeah yeah you're right. Let me fix that. I think that's pretty good as well. Now even if you don't go the full story with the auto remediation thing I think there's value there. And we

definitely are not target targeting everything with this. There are stuff that no it's up to people to decide on. But there's a pretty big improvement in efficiency by using this. So start with the deterministic workflows then add AI with the reasoning to help use schema bound AI there's no drift and hallucination if that container is small enough at least that's what we think prefer this watch pattern

over chron on and polling and all of those things that we see in multiple tools. Keep some sovereignty options on the table like local models and design for evidence. The proof changes posted for people in regulated uh industries is truly important. No one's going to forget to get the right data into the incident tracking system if it's in the operation. No, no one's going to close an

incident saying fixed if there's an LLM writing the right content to push into that tracking So with this where do we want to go? many many many uh infra where we want to do this. We saw an example about cash. No cache is cool but if cache isn't available typically the app should survive no for some time. Let's do it with databases. Let's do it with messaging.

Let's do it with everything. We have few dozen uh resource types that we support with crossplane. All of those should be enabled at least in the analysis stage. is to help people get some context into it. And for instance, now with Crosswind V2 where you can do claims at the application level, going into an app claim and seeing, oh, here's an annotation, your app is down because

you need to fix that. I think that's pretty cool. We are improving all of the data we keep for reporting. And when I say reporting things like uh we we need to uh do uh recurrent reporting to the European Central Bank on incidents and fixes and all that. The data that we get from this approach is a lot better than someone at 2 a.m. typing stuff into

a a text box. And because we're doing this name space by namespace, workload by workload, we only stretch the elastic as much as you want. You know, critical apps are one thing, uh line of business apps are a different thing. We set the pace to adopt this with a managed amount of friction. So to leave you with a takeaway, our point of view is is this AI

becomes safe in production for instance in places like ours where we get a governed control plane doing it. The boundaries are clear, the expectations are clear as with everything control plane wise. So we know what we get when we get and it and it's a cool story, you know. So with that demo code and session feedback and questions, please. >> Thank you so much. There's a mic

up ahead if you guys want. No questions. AI explained everything. Yeah, we have a question. >> Hello. Uh, great talk. Thank you. Um, on a multi-tenant environment where people can modify labels and such, how do you handle cooperations between the the operations? Because the labels are not safe. Everyone can modify them. >> Sure. Well, watch operation is kind of a mini resource, right? >> Yeah. >> You

can you can have two stories uh for someone changing an annotation. >> One can be someone improving the annotation and saying for instance we saw uh cost cost optimization saying uh you should move from extra large to large Imagine an S sur doing no move from extra large to medium. Where's the issue in that? Right? We are going to do an even better change because someone added

knowledge to that. If someone removes an annotation so that the operation doesn't match and you stop looking at the resource, I certainly hope someone had a reason for that. Now, maybe that is in a condition that someone wants to look at and keep AI out of it. It's good. No, someone should should have a reason for that, you know. >> Okay. Thank you. Thank you.