Fusing FinOps, Forecasting, and Kubernetes at Scale - Ankur Singh & Satyam Bhardwaj

This talk addresses challenges associated with Kubernetes' PHOPS, focusing on implementing a solution that leverages AI and machine learning for better forecasting and resource management. The speakers, Ankur Singh from Red Hat and Satyam from Mirantis, detail their backgrounds in site reliability engineering and open-source contributions to multi-cloud management systems. They propose a modular framework designed to integrate existing Kubernetes tools like Prometheus and ARMA while enhancing them with AI-driven forecasting capabilities. The solution aims to optimize Kubernetes infrastructure by using time series data to provide actionable recommendations, reducing the typical 6 to 8-week cycle for cost optimizations. The project, called Auger, emphasizes open-source principles and incorporates a human-in-the-loop approach to validate AI recommendations.

I hope everybody is having a good day. Uh today we're going to talk about a painoint that we have in the Kubernetes part which is PHOPS and we are going to present a solution onto PHOPS where all the things that we can do in a Kubernetes native way and keep it open source as well. So before we start with what we're going to present today, let me

first introduce ourselves, first is me. My name is Ankur Singh. I am a senior site reliability engineer at Red Hat and I work for managing and building solutions for one of the cloud managed uh Open Shift solution called Azure Red Hat Open Shift. My day-to-day things include building all the features around ARO, making sure that the observability is in place and also troubleshooting any customer incidents that

come to us with respect to ARO as well. Hello everyone, good morning. It's a big crowd. Thanks for being here. I am Satyam. I'm a software engineer with the open source program office at Mirantis. We contribute to a lot of open source projects. I have been involved actively in building open-source multi cloud management systems. So I have been contributing towards um there's a project called cluster API

from the kubernetes eggs and I've also been contributing towards cod and kzeros. These are all multicluster management solutions and kz is like a kubernetes distribution. So today, yeah, today we're going to build a solution whereby we're going to utilize some a IML models to um you know really time series models to do forecasting for the observed data from your clusters. And here's the you know agenda for

today. So we're going to leverage so yeah we're going to build the system we are going to first talk about what's wrong with the phops today and then there's an architecture there's a pattern that we saw in this in building this project and we have built a project around it so really a p around it and we'll showcase the pattern that is observing the state of the

cluster using those matrix to uh feed it to a model and use that forecasted data for recommendations to optimize your Kubernetes infrastructure. We'll also see how you can extend this model to build your custom logic for your business solutions and we're going to see how it scales al together and finally we have a demo around it. So uh talking about phops we we know that this has

been one of the things that we always want to consider when it comes to the cost for any organization and it has been there since we had virtual machines. But right now with Kubernetes taking the pace we it is more important than ever that we look at how we can do it in a more Kubernetes native way rather than relying on uh vendor locked in uh products

or solutions. Talking about uh the state today we know that whenever there is a cycle for a phops it takes around 6 weeks to 8 weeks of time where we identify uh what is you know taking up the cost or wasting the cost in the right sense and out of that what can we fix onto onto those things then we go ahead and deploy the fix okay

and then we wait for the next cycle altogether for the fix to show the results. It might be successful, it might not be successful and the loop goes on again. But again, this loop takes 6 to 8 weeks which is not something very suitable in the in the ecosystem where deployments or releases happen every week or every day uh or every hour on the on the right

side. So we are talking about making it much more visible or making it much more quick and making it sure that we do that when when it's the right time rather than waiting it for 6 weeks or 8 weeks right there. We have autoscalers which do a decent job of scaling your clusters and saving some of your costs but those work on the current data available. We

do not have a solution that works on any of the forecasted data and makes the right decisions there. Now talking about creating a solution, we don't want to create a solution from scratch. Why? Because we already have the tools which we can uh leverage to actually build a glue with all these things integrated together and present a solution that is more uh robust and modular. Here for

example whenever you want to have uh observe the cost or observe the resource utilization we have a lot of tools which we can use for metrics like Prometheus Thanos or any of the open source tools available. Now for the forecasting we have really good models like a oneshot model for Toto or we can use ARMA or times FM as well and coming out to taking actions on

to how your cluster infrastructures can be made more efficient. We have tools like carpenter and kada and all those features. Now the problem that is isn't that we don't have the tools. The problem is we don't have any integration of these tools. Now to build an integration you can have it customized for your organization or we can have an open-source system or open source tool that will

be extensible and also utilized by anybody who has to use these things in an ecosystem for their own. So the solution that we are going to present or they're going to talk about later talks about integrating all of these into one single platform and the end result for all these things would be a clean recommendation about how you can save your cost based on the current as

well as forecasted values based uh with the system. >> Okay. So what would it look like to build such a system? Um so we are talking about agents and it's not agentic. What I mean here is um automated loops which going to solve this problem for us. So I deal with a lot of multiclusters you know on my on a daily basis and we have really standardized

the way we collect matrix from those you know for the observability of the whole cluster. you have open telemetry collectors deployed on clusters whereby you can collect data related to um CPU utilization, memory utilization etc. And when you have all of that, maybe you want something um you know cost related matrix from your um cluster. And for that we have a mature ecosystem with a lot of

tools. You could use something like open cost to get those data. And once you have this observability in place you we want to use this data to feed to a time series model and let it generate some forecast So these these forecasts are they have like um let's say certaintity bounds. So you have a mean forecasted data plus the confidence levels at which it's going to work.

And we can use these metric points to to generate recommendations. Now generating recommendations, it has to be deterministic because we can't rely always on nondeterministic AI models. And once we have this recommendations for um let's say a particular resource in your cluster um we want to have a human in the loop to approve those because we know that AI models today there are model drifts there are

you know hallucinations and Kubernetes makes perfect sense to loop this all together because we have a reconciliation loop where we look at the state of the what we have defined and what we want to achieve. So yeah so around this pattern we have built this project a score auger we have uh to achieve all of this goals we have defined three custom resources it goes through a

few stages and you have one you know cubital apply to do all of this sorry so yeah we call it auger um it's from a Latin which means somebody who reads science and for what's going to the first stage you know the first custom resource that a user want to interact with is called a forecast pipeline. This is where you define what would be the cadence at

which you want to get the you know forecast let's say because you know as as soon as you have newer data you want to have more forecasts possible fresh forecasts and when you have that sorry yeah so when you have that you you want to define the configuration for your how you're going to collect the matrix maybe you're using hotel maybe you're using Prometheus maybe you're using

something like promzy if you don't know it's sort of aggregator for matrix altogether. And once you have that um you before feeding to a model models all the time series models at least they need different sorts of input format they expect and so you need a transformer to do all this together. Then finally once you have a data in a certain format and you feed it to

a model um you should be able to communicate to the model and for that we have a custom resource called forecast model. The idea behind is really to abstract how you're going to deploy these models because Kubernetes ecosystem has tools like K-serve which are really good at let's say model serving and inferencing and we want to utilize that and not you know couple the model deployment with

the project and if you know about the um v2 open inference protocol from case we are using that so that we have a large number of models that we can support. So every time a model which you know really um defines a set of APIs defined by the contract we will be able to contract with them. So so there are actually a lot of ML frameworks which

already does this and so we are good with it. And finally um the produced data would be fed to an optimizer. Um optimizer is a custom logic again and it produces some outcomes. So it the outcome that's produced it's produced as a um another CR we call it for recommendation. It targets a particular resource and as a human in the loop you want to approve that to

you know apply to a particular cluster or a set of clusters. Okay. So this was a bit of things that we have built around or we have created for the project to have an end toend integration. But even with these things what we also want to um demonstrate is how we are doing is doing these things under the hood in the project. What are the main points

that uh or we can say what are the main differentiators that the project will have so that these things have uh been you know working in in a in a right flow. So we have come up with four major pointers of our project. The first one we are talking about is a stand standard data formats. Now we know that the sources or the inputs of data that

are coming in can be of a different you know different ways different formats and different stuffs. So we are defining a particular metric format in which the source data should always be there. So we can plug in any of the source tools but they have to convert the data into a specific metric frame. This will make sure that we have more extensibility with the tools that we're

using on the source data side and we also maintain the integrity with all other um uh stuffs we want to have in in the cluster itself. The next thing we are talking about is a multimodel inference. Now this makes sure that we have we are able to integrate multiple models. we are not sticking to something proprietary or something that we you know don't want to use. The

customers or the users can actually make sure that any of the things that they uh model that they want to use can be integrable with the software itself. Not only that but we can even check about having multiple models and each model can be used for the same source of data. With this we'll be able to compare on the results of each and every model that is

coming out of it and we can choose the right model for the right uh uh data of the so the right workloads there. So this will make sure that we do not have to rely on any of the models that do not fit with the workload or with the data that we have uh right now. The third thing we are talking about is uh uh about accuracy.

Now we know that when a model is being utilized in any form it might be you know a state or you can say it might be not so accurate after a period of time. We we will have mechanisms that will be in place to detect these drifts and not only detect but even raise Kubernetes events onto to check eventually. And how does it do that? The first

thing that it does does is with the predicted uh predicted values that he had or the forecasted values that it had when the time comes it compares the forecasted with the current value of at that time and sees that if the accuracies are up to the mark or not if not it raises raises an event and with that event we'll be able to make changes or we'll

be able to uh take actions in time so that the recommendations always remain fresh and always remain you up to the mark with respect to accuracy that we want and as a as a precaution we also have a human in the loop mechanism. So it might be possible that with all these things that we are doing there would be a possibility that the recommendations might not be

what the user requires and I'll also have to quote that as per state of phops 2026 report we still have more than 50% of the people setting uh limits and requests manually. So nobody relies on AI to that particular extent. So we have a human in loop mechanism wherein with all the recommendations that came in there will be an audit by a u by a user where

it will decide on if these recommendations are to be taken into consideration or not with a proper approval. These recommendations will be then converted into actions and uh will be taken up for um optimizing the cost or saving the cost eventually. Okay. Now talking about what's uh with the architectural stuffs and what do the contributors have to do if they want to extend this particular uh software

in their own use case. So the first thing if you if you see uh you can have your own data source. The only thing you have to have is an collector which can have the metrics in a metric frame format. Now this is the standard format we want to maintain for the software so that any source can be can be easily integraable to the particular software. This

can be done by only working around one Python class itself. So this will be an easy to go uh approach. The second thing we are talking about is the model. As I mentioned the forecast envelope is the output format they are looking for. So any model that you integrate should have this particular format as an output for us to make it uh work with the end to

end approach that we are using. So for this thing if you want to have your own model you'll have to have to deal with only two Python functions nothing on the go side as of now. The next thing is you can have your own policies implemented. Now again the recommendation that will come up and for this thing you'll only have to deal with one Python class. Why

I'm saying about dealing with only short uh stuffs here? Because the software that we have created has an mixture of Golang and Python where Golang uh takes care of the Kubernetes plumbing on the Kubernetes side so that it is you know much more robust and has the mechanism working on and the inferencing and all the AI side of things that we are doing is taken care by

a Python sidecar. So anything that you want to integrate with it only have to deal with the Python side of things wherein the reconciliation loop and the operator side of things which is done by the Golang part of stuff will be as it is for any of the use case. We are also talking about having a multicluster approach here so that uh this product can be deployed

on something like a managed cluster wherein it can scrape all the metrics. You can get all the metrics either using you know Prometheus federation or something like that and um be utilized for all the clusters at once or you can even try it on one cluster itself and uh this this will actually make it uh more extensible more modular and give us a the um flexibility of

introducing more plug-in based uh approaches to actually utilize that uh for any software. >> All right. So, let's see a demo. By the way, this is a completely open source project, so you should be able to um play around with it. Just give me a second. Sorry about the background. I thought it's going to be black. So I I actually have a um a kind cluster running

locally. So So that's cubectl get pods um for uh K for cubectl. So these are the nodes. I have two worker nodes and a control plane and I have a couple of deployments in this name space and so we have a augur controller manager which is the main you know reconciliation loop and then we have ARMA serving and to serving so these are the two models I'm

going to show the demo on and what what serving means really that they are just running on in a pod as a separate deployment. Let's see the um forecast models that we have in this name space. So we have two of them. Um ARA really or to really references those deployments the to communicate to the spots to be able to do inferencing. Um let's see one of

them in a little detail. So this is total model. Um if you've heard about this this has like this is a pre-trained model uh close to trillion data points. So I do not have to train that on my matrix from my cluster. It just gives me I have to just provide it some contextual data um let's say the past 7 days past month and it's going to

produce some forecast for me uh sorry yeah forecast for me in the for the upcoming future. So if you see um it's just referencing this uh particular URL that's uh running somewhere and we have built a a thin wrapper around uh you know implementing the v2 inference protocol for toto because toto doesn't natively supports it um and you see a couple of conditions which is like very

uh kubernetes uh you know native things for example a transformer to in to you know proc process the inputs and outputs and health checks are ready so we are good to go with this the ARMA model that we're going to test. It's a it's a univariate model. So it's it's sort of a classical ML model and it natively supports the uh V2 inference protocol. So on this

cluster on this sky kind cluster I have actually some um open cost deployed and I have I have um you know given a custom pricing for this kind cluster um you know something random um and we're going to see this um in the forecasts. So one of the pipelines that we are running it's uh I have already ran it so that we don't have to spend time

here. This is a remma model. So basically in this forecast pipeline which is the main you know this is what you mainly interact with. Um you define a cadence define the horizon for which you want the forecast uh configuration for the collection of matrix. So for me it's Prometheus. Prometheus is just you know um giving me all the data aggregated from my kind cluster. It could be

anything else though. This could be just a open co um hotel collector or it could be a let's say um promzy or just open cost and I just say I'm going to refer this model ARMA um which is like a forecast uh model CR and this optimizer is a custom optimizer that we have defined and I have given some silly numbers so that we have some forecast

and this is very similar for the Um let's see the no forecast my plans. So we have two of them. Um and if you see the printer columns it already has a lot of things. For example our um you know the last run of the the pipeline and the next run when it's going to be. So let's see one of them in a bit So I have

defined a spec and you know um Kubernetes does it thing of adding conditions. So we have u couple of things like if the model is ready, if the collection is ready, um if the focus are ready and finally if you see this line here u focus URL this is where we are actually storing the focus um so that while this is right now it's in a cluster

I mean I'm in my controller cache it could be any back end where we want to store it and maybe use it for um drift detection later on and I have built actually a graphana panel Sorry. So this is the graphana panel for you know visualizing the forecast that we got from the pipeline. Um we have currently selected the toto and I could select one of the

work uh nodes from the drop down. So if you see the data it's uh the model what you see in the forecast is as good as the models are and this cluster is relatively new like a day ago two days ago I created this so we have less contextual data but this is what we get the middle one is the you know the mean forecast upper one

of the confidence levels for the forecasts similarly we have different pipeline arma and let's see the contrast between the forecast from you know them it's pretty much flat because ARMA needs a lot of data points to train on and then you can see some data. Um we have also defined a couple of things like map. Um so uh so this this is the error rate in the

model how accurate it is and the way we have built this algorithm is we just give it the you know the whole of the data and we say uh from the first 70% of data try to produce the next 30%. And how it performs but we could be in a production usage it would be different. It would be something like you have a lot of data and

then you compare it against the you know forecast and the past data that you have that you know is true and that's where you can get some real data. If you see there's one recommendation as well. I'm going to quickly show that. So there's one of them and it targets a particular note. So spec has a lot of things um because this forecast recommendations here was created

by the you know uh the optimization algorithm. So you have a couple of things for example where the forecast is um some metadata about it and finally some the impact that the recommendation is going to have for example this is the value you're going to save and the the peak CPU and memory limits um and finally the recommendation it's in the it's in a very readable language

right now and if you remember I give some silly number for optimization and it says um this particular worker node is okay for removal given it's going to be below 80% CPU and 90% uh memory or which is bad but yeah um and yeah so how does this going to work when you want to approve it so you just got to you know um patch the spec

with approval status you could also interface it with a Kubernetes MCP server and do that but you know that because you're doing that um it's going to be okay for your cluster you don't want to just AI models to run on on its Okay, so that's all about the demo. That's that's I don't think that's the one. >> Okay. Fine. Uh so with the demo done uh

we have a specific uh version of this particular software already working which we showed in the demo itself. It has some certain features already implemented but we are targeting for uh separate versions after that uh which will include um the hintcast or we can say the drift detection. It will also include the multi multicluster support that we want to look at so that we have the things

in place. So I'll just have a shameless plug-in here for a call for action. If uh you know any platform engineers who find this thing uh to be utilizable, you can have it and contribute to the Golang side of things. Uh for people who are on the ML and AI side of uh respects, we can work with the uh plugins for different models inferencing for different models

and uh with these things I think uh the remaining aspect of uh building the project would be something uh we'll be targeting next. And uh with this I think uh we are done with the talk. Thank you. Thank you for being a loving audience.