Cutting Metrics Traffic, Cutting Costs: The AZ-Aware Observ... Iris Dyrmishi & Rodrigo Fior Kuntzer

In this session, the speakers from Miro discuss the challenges of handling observability costs in cloud environments, particularly concerning data transfer across availability zones (AZ). They reveal that a significant portion of their cloud expenditure stems from network costs associated with observability components. The talk emphasizes the importance of making architectural decisions that consider both performance and cost, highlighting how default observability setups often lead to inefficient data transfer. The speakers propose strategies to optimize metrics collection by making monitoring systems AZ-aware, which helps reduce unnecessary data transfer and costs. They present their journey of implementing these optimizations, including the use of tools like Prometheus and VictoriaMetrics, while maintaining system reliability and performance.

Hello beautiful people. Good morning and thanks for stopping by. Really appreciate you staying with us until this very last day of KubeCon. Uh and I hope you are enjoying the show. Uh my name is Rodrigo. I'm a staff SRE engineer at Miro. Uh we are here today to talk you about uh cutting metric traffic and cutting uh costs. And I'm delighted to share the stage with Iris.

Hello everyone and welcome. I'm Iris Durmishi. I'm a senior observability engineer at Miro and also CNCF ambassador. Happy to be here. Uh yeah, so uh we have provided a link for some Q&A in the end. So, if you have any questions during our talk, please feel free to scan it. And if for some reason you cannot send your questions, don't worry. We have a mic. So, uh

yeah, we're going to answer everything in the >> And you also find us elsewhere later. Absolutely, yeah. We're open for discussion. we have come here and we want to ask you a question. Who pays for your observability? Do you know? Well, we have a spoiler alert for you. Well, you do. And probably even more than you think. Sometimes there are some hidden costs that we bet you

didn't take into consideration when you were evaluating the value of your observability stack. Uh over the past uh 2 years, we have uh been uh have been being much more cautious uh about our uh costs uh our cloud costs. Uh at Miro, we store a lot of data on behalf of our our customers, especially in terms of storage and and uh databases. That's it's a lot of

our cloud spends. And then, there is not that much left for for computer network. And that's where we have to be really uh cautious about. So, we have done many uh many good improvements uh in terms of computer optimization like right-sizing workloads, improving nodal location radius, and so on. But, there was one AWS bill that stayed untouched in our billing reports, and it was actually kept growing

in a terrifying rates. That item is the data transfer intra AZ one. And then, when we start digging into it, we found that 60, sometimes even 65% of that cost was coming from nodes running our observability components. And the actual network cost of those components was often often equal or even higher than the compute costs needed for the compute for the for for the compute power of

those components. for us, something was clearly wrong, and you have to do something about it. So, when we start to research about the problem, myself from a technical angle and Erez through her contacts contacts in the observability community, it was interesting to find that even companies with mature FinOps practices and robust observability were also not able to handle this properly. And there was also a blame a

kind of blame blaming game going on here. Observability engineers were saying that it was a network problem. Network engineers were saying that it was just the nature of public clouds, and FinOps was just lost in the middle, you know. And we are here today to show you that tell you that every architectural decision has a cost dimension, and you can do something about it. But, before we

move move on, you might be asking why multi-AZ at all? If you are in a public clouds, you pretty much have no choice here. Uh the obvious obvious reason is fault tolerance, as everyone talks about. Uh if one AZ is suffering, the other uh can absorb the loads. Uh Kubernetes itself makes this is smoother with pod topology spread and self-healing capabilities, if you configure it properly. Uh

but even if you say uh that you don't really need high availability uh across AZs, uh you might still be forced into a multi-AZ setup, uh mostly by capacity constraints uh constraints on your provider side. if you try to run everything in a single AZ, you might not be able to run, for example, on your AWS side that shiny new Graviton instance or a GPU-accelerated instance uh

in the AZ that you decided to stay on. and in terms of costs, what are actually the the cost and and the price that you're talking about here? Uh each cloud each cloud provider charges it differently. So, if you're on AWS, like we are, uh AWS charges it 1 cent per gig on each direction, uh make it effectively 2 cents per gig uh of data transferred. Uh

GCP charges it at 1 gig per uh 1 cent per gig only in the egress direction. And Azure has made it completely free since 2024. and to be clear here, we are just talking about uh cross-AZ data transfer within the same region. Uh we are not going to be talking about cross-region uh transfer, because that's another uh different level of of of expensive. so, circ- circling back

to observability, why observability is the heavyweight champion of traffic costs? By default, most observability deployments are not AZ aware. Agents see targets across the entire cluster environment with no awareness of AZ boundaries, crossing them on every scrape. This is from data collection perspective. Now, during ingestion, still in the right path, multiple pipeline components are talking between each other and maybe running different AZs as well. So, more

boundary crossing. And the same, of course, later on in the read uh path uh can happen as well. from our side, and just to give a quick number here, that was becoming that was quickly becoming a million-dollar-a-year problem for us. And finding the actual workloads generating high cross AZ is not easy. Most of the time, it is just some item in your billing reports. To understand it

properly, not only for observability components, but for your environment in general, you might you need a right tool for that, either in the network observability space or in the FinOps one. Uh in our case, we already had all observability components running on a dedicated node pool with proper cost allocation tags. That made it straight- straightforward to open the AWS cost explorer, filter by that tag, and confirm

where the costs were coming from. Uh that's our personal advice from where to start from here. If you can col- collocate your observability components, or most of them, on the same nodes, have them properly tagged, you can isolate and filter it filter that view immediately and just see if the if the costs are coming from there as they were for us. Uh and for a deeper per-workload

visibility, because as I said, just uh looking to nodes, you might have different workloads running there. If you want per-workload visibility, we are adopting Grafana Beyla, a lightweight eBPF-based solution that gives you network observability with zero instrumentation. That's a an enhancement on the setup of bringing uh cost visibility for our on this talk, we want to give you uh blueprints, but you still have to decide if

it fits your context. We don't really want you to just chasing a problem that you don't have. And here is just a few bullet bullet points that you can evaluate whenever the added complexity is worth it or not. And just to summarize them, if high high cardinality and volume is already a problem for you, if you are already in a multi-AZ uh Kubernetes cluster environments, if you

have hundreds plus uh scraping targets, and if cross-AZ costs are already visible on your bill. So, now let's uh learn a little bit more about how we actually implement it, but first, let's get something straight. Uh It is important to be able to distinguish between the two primary metrics collection methods. The pull method and the push method, for example, the pull method, most of you are probably

aware, uh tools like Prometheus, like Victoria Metrics, or if you're using Prometheus receiver in uh in Open Telemetry, and of course, the the push method, which is the standard right now for Open Telemetry. So, general in a push metrics architecture, industry standards involve running an OTLP collector as a daemon set right next to our the So, when it comes to the metrics collection itself, there is not

much um optimization that can be done. But, when it comes to the ingestion in total, of course, that is more something that can be solved with native Kubernetes features. Uh for example, the network solutions like the traffic distribution, prefer close, or prefer same AZ. So, in in this case in this talk, we're going to focus on the pull metrics because this is our use case in Miro

as well, and we have studied it and implemented it in our in our uh stack. So, what does uh our architecture look like? We have named this slide anatomy of a scandal because it is a very expensive scandal, let's say. Uh so, in this slide, we're sharing a snippet of our metrics pipeline. So, it's a very small uh part of what we're doing. So, as shown, the

metrics ingestion is going into hopping into several AZs. For example, we have a scenario where the target, the VM agent, and the VM insert are all in different zones, so you can imagine the costs accumulating here. So, and then let's not talk about the querying layer, which is again accumulating more and more uh multi-AZ traffic. So, in uh architecture of a big scale like ours, and I'm

sure uh of many of you, uh the cost of uh this traffic will add up fast. But, don't get us wrong, and let us be clear here. This is not a problem that we are only facing for example. I mean, not really a problem, but it's an architecture design like for tools like Victoria Metrics. This is something that we're also seeing in Prometheus, for example, and other

pull-based tools. So, it's a side effect of how the systems are discovering the targets in a dynamic environment like Kubernetes. So, for example, we have Victoria Metrics as we mentioned. We're doing the collection through the VM agent, and that we have already optimized. We're going to talk about it. But, then we have the rest of the ingestion, which is the VM insert until it goes to the

VM storage ready for querying. The same thing for Prometheus, Thanos, Grafana Mimir, Cortex. So, all these tools in in their center they have the same issues when it comes to collecting the data as long as it is pulled base, of course. let's talk about Mimir's use case. Before we went to a project of this scale, we first wanted to add some um big prerequisites and some big

conditions because uh messing with your metrics pipeline is not a good idea unless you're sure of what you're doing. So, we wanted to optimize the traffic as much as we wanted, but we have some non-negotiables when it comes to optimization that we can make. And the first one is to maintain the current computer resources. We do not want to save on networking cost and then add a

crazy amount of computing. That means that we're not really doing any optimizations in this case. Uh no additional spending on the architecture, which is uh coupled with the first part. And also preserve availability, reliability, and the quality of the observability metrics. So, for us cost is important as any other organization, but when it comes to reliability and availability, that always comes first. So, we want to make

sure that that was not something we were messing with when we were making all this all these changes. And of course, scalable architect architecture. We're going to show some numbers to you later, so you're going to understand why we really need the scalable architecture. And yeah, of course, the big prerequisite is the centralized alerting and querying. We also predicted some impact and we actually saw them happen.

Of course, higher architectural complexity. We introduced a lot of changes, so the architecture became more complex and there was a lot more um work for the observability team to maintain it. But, okay, we we predicted it and it did happen. The cost saving actually justified the investment because it is a time investment to have some engineers working doing all the all the troubleshooting and actually implementing the

solution. Uh the best part that we had was that there was zero end user um interference. So, basically, they did not have any impact. Uh the the change was flawless. So, for them, it was as using the the platform in any other day. And yeah, we always entered this with a mindset that we would never have a zero traffic bill. There would always be something that we

were going to pay, but our goal was to optimize as much as we could. So, just to give you some numbers, we will not mention all of them, but for example, we have around 140 million of time series per day, over 13,000 targets. And here we're talking about one of our largest environments, but we have several of them. Uh when it comes to the amount of data

in gigabytes, around 12 to 19 gigabytes of uh data per scrape, and this is uncompressed. And in general, per day, around 25 terabytes of uh metrics uh per day. So, yeah, if in the beginning we were thinking 2 cents per gigabyte, it's nothing. When it adds up, and this is just one environment. I I'm reiterating it just to to uh to to show the significance of the

scale, the bill really, really adds up. Yeah. Um So, yeah. Um But, for our use case, we decided to do the optimization in two phases, and that is because of those strong prerequisites that we mentioned before. The first one was because we knew that there were going to be a lot of architectural changes, and they they to be rolled out very slowly, of course, and we wanted

to protect our reliability. So, in the very first phase, we have optimized the metrics collection, and that's what we were going to focus heavily in the rest of our talk. So, what we did, we made our VM agents AZ aware, in your case it could be the metric collector, for example, like Prometheus. We are scraping only targets in the same availability zone, and it was a quick

win, low risk, but ingestion still crosses between phases, and that's what our phase two of the project is, which we're still not implemented, but it's in the plans. So, what does it mean? That our whole VictoriaMetrics stack is fully AZ aware. We're ingesting and querying fully AZ local, federated multi-level query, and of course, much bigger architectural changes. So, if we were to visualize how this project has

felt for us, here's the observability team with a very complex architecture that is going to get even more complex, working hard, but there's our end users that are absolutely loving it and they didn't feel any change. So, yeah, that's exactly what we were saying in a picture. And from a project execution perspective, that is it was it was also great because we could do that by our

own without much friction or interaction with other teams. So, before we go into more technical details on how we actually solved our problem, we had a very strong prerequisite, and we of course we're giving you all blueprints so you can make your own decisions, and of course we wanted to present this prerequisite, and that is to have a centralized storage for alerting and querying. Otherwise, the solution

will go outside the industry standards, and you will alerting that is not working well, and query that is not working well. For example, in VictoriaMetrics, we have that case. Uh so, there is all the metrics that is ingested is usually stored in a central storage, and from there we are querying and auto scaling and alerting. So, that is very important to keep in mind if you are

deciding to make this optimizations that we're proposing here. So, what would the architecture look like uh for these changes? So, the first one, there would be a zone isolated deployment for each AZ that you have. Um so, that means uh for example, in each of these AZs, you're deploying one uh VM agent deployment or one Prometheus deployment. Or if you're using OpenTelemetry with Prometheus receiver, the same,

you would be deploying it there. Um there would be intelligent scraping, so you would be using target relabeling to only scrape the targets that are from the same AZ. And uh of course, horizontal scalability. So, all your deployments would have n + 1 shards depending on the load uh that they would need uh the amount of targets they would need to scrape. So, this is also something

non-negotiable for big-scale architectures. You need to be able to scale properly, and uh yeah, it's very important to maintain the scalability. Uh in this picture, we have mentioned uh Prometheus, but as I mentioned, you could use Victoria Metrics like we are or OpenTelemetry Collector, and this is kind of how it would look look like. So, uh your metrics collector is only collecting uh from a service, the

pod that are in the same AZ. And uh of course, n + 1 shards for scalability. Yep. So, enough talk, uh show me some code here. Uh as you are said, we are now running uh zone isolated agent deployments. Uh but that alone is not enough. Uh just placing agents in a given AZ uh does not make it it understand which targets are co-located at there. Uh

the prerequisite for a Kubernetes runtime in this case is having topology metadata available in the Prometheus Kubernetes service discover. Uh to achieve that, we have pretty much three options here. Uh how many of you are already on Kubernetes 135? Raise your hand. 35? Oh. You're lucky. So, if you are there on 135, we this problem is solved, so you can just go home and deploy it. Uh

thanks to kept 4742, uh which graduated to beta and now is enabled by default, all pods now automatically get topology labels propagated from your nodes to the pods itself, and you can see uh a guardian S pod there uh from a Kubernetes running 135. So, the topology labels that before that were just on your node side are also now on the pod side by default. the second

option, uh which if you are in a older Kubernetes version as you are for now, uh you can use a mutating policy for that. In our case, uh we use a Kyverno mutating policy to copy the labels during pod binding. So, you have pretty much copied from the nodes uh to the to the pods. Uh the only catch here is that binding time can only mutate annotations

and not labels. Uh but this is fine. Uh Prometheus relabeling also works with pod annotations. It's just a uh uh uh uh a small uh detail when writing the relabeling config. Uh and the third option is a Prometheus native one. Uh you can configure your scrape job to attach meta uh node metadata directly in the SD config. So, there is a snippet there, uh but we decided

to do to not go deeper on this one because we are a little bit afraid about consequences like increasing API server pressure or high memory usage on the agent side. Now that you Now that we have topology metadata available, we can make a scraping AZ aware. Uh and I have two snippets here again. By adding the relabeling condition at the end of your relabeling configs, this is

the technical secret sauce here. You can keep only targets that match the current AZ where the agent is running. So, it's pretty much that the service discovery will find all the all the targets in your environments, but this rule there will dropped uh targets that are not in the same AZ, so data collection will never happen for and consequently uh no uh cross AZ boundaries will be

crossed. Uh for targets outside Kubernetes, like for example EC2 instance in our case here, you can apply the same logic via relabeling. There there will be a a meta uh label for the EC2 availability zone, or you can filter it directly on the EC2 SD config as we are showing here. Uh Moving forward, now you might be thinking I I don't really manage agent configuration directly. I

use like a Prometheus operator CRD like the service monitor or pod monitor CRDs, and I have thousands of them. So, I don't want to go and change each one of them, right? Uh so, the good news here is that most platforms often offer a way of injecting global relabeling. I have two examples here, one from the Prometheus agent CRD. So, it lets you define a default scrape

class uh where any relabeling definition condition added there uh is automatically happen to every job. So, we just create your engine in that way, adds that re-labeling condition there to this scraping class, make it enabled by default, and you get the same that we were that you could be doing by in a job in a pair job basis. The VM agents here the works similarly with re-labeling

template per monitor type. So, you have to define one for your service monitor, another one for your pod monitor, and so on. And one particular useful feature here is that the VM agents, you can reference environment variables inside the re-labeling condition. As you can see, I have two environment variables here, the zone name and the dedicated AZ AZ placement. And those are used there in the reg

exp field of the template. So, it helps you keep the configuration clean and portable. In some ways, and then because also because it's an environment variable, you can use the Kubernetes downwards API to get the value for the agent the AZ where the agent is running is straight from that label that I just said. So, comparing both here, you see that in the VM agent side, we

don't really have any hard coded value for the AZ. And in the other side, you have to to to to do something like that. You might be You might be wondering what is that dedicated AZ placement environment variable for? We have one more and tiny problem to solve here. As you guys mentioned, we are not really chasing a zero dollar bill. Some Some targets simply do not

expose or easily expose topology metadata. You just get like an opaque URL out of them, and you can really do much about that. One of good example here is for AWS managed services like the Amazon MSK. Uh each Kafka broker in a Amazon MSK exposes a metric endpoints, but URL is randomized and no and doesn't carry any AZ information attached it. So, our solution was to designate

uh the agents in one AZ, like for example in the 1A, uh to collect to be the collector for those topology blind targets. Accepting that there is small amount of cross AZ traffic for a limited set of endpoint endpoints would happen. Uh so, in practice here for the for the agent in this 1A AZ, we define that dedicated AZ placement to a value that creates a logical

or between the real value for the for the AZ uh uh for in this case matching an empty string for the targets that that has no AZ metadata at all. So, this covers the blind spots without uh complicating every other agent agent configuration. Uh the final picture here is to fast forward to the final solution once you get the phase two uh covert. Uh so, phase one

was just about metric scraping and phase phase two once it's roll out, uh it will be a fully per AZ Victoria metrics cluster deployed. The hard part, just to touch a little bit on that, is the querying parts because we still want clients to read and aggregate transparently across all AZs without any specific logic on their side. Uh to solve that, we are introducing a federated or

a second-tier or second-level VM select layer for querying on top of it. Uh it's sitting on top and funnelling out the queries to the various uh VM selects deployed on each one of them. Uh but the interesting part here is that this won't really provide any cost savings because there is no aggregation whatsoever happening locally on the regional VM select. So, it's just a to send the

raw data back to uh to to the global uh VM select and then the the the aggregations are are done there. Uh this is an acceptable trade-off for now uh because at least from our from the right path, uh we are sure that no data is leaving an AZ and there is a lot of uh savings uh for what we have proposed so far in terms of

architectural change. time constraints prevent us for showing you some demos or or going deeper into into phase two. Uh but you we don't want to leave you empty-handed, so we have prepared a GitHub repository here uh covering uh a few different topologies and and uh observability platform from at metrics perspective flavors. Uh you can just uh deploy that in a kind cluster setup that that is available

uh and that setup is simulating a three AZ uh environments uh and you can explore by your own there. And to leave you with a last piece of information, in case you were wondering how much did we actually save uh because we were going on and on and what we did and what we can do. Well, no, wrong. Well, we all we can say is a lot.

Unfortunately, uh we cannot divulge uh a numbers uh because of legal reasons, but we can say that our savings were in the six figures. So we were very happy with the optimizations that we did only for the metrics collection part. So this is me and Rodrigo swimming in money that we saved. Um Yeah, I would be happy to be. Uh thank you so much. Uh if you

have a moment, please make sure that you give us some feedback. We would love to hear back from you. And I don't know if we have some time for questions. Let me check Slido, but if we do not, we promise that we're going to answer your questions on a post on LinkedIn uh or you can meet us. We're going to be around. Oh, we have one question.

Do we have time for one question? Okay. Did you measure the difference between the ingestion and query path traffic? Uh who ans- asked this question? Yeah, that's uh that's interesting uh question. We I guess uh uh we haven't proper uh measure uh both sides of the coin here. Uh we have we were just working with a global view of our costs and working as fast or as

as much as we could to reduce to reduce the overall bill there. But as I I mentioned at the end, uh from the query part from the read path, unfortunately, there is not much that you can do or at least that we have found so far, right? Especially if you want to have this global view at querying time where you have to fan out the queries to

your local AZs and get the data all back to where the the the the the client is is doing. Of course, there is techniques that you can power there in terms of down sampling, aggregations, and so on, uh but that you increase much more the the the complexity and won't be transparent for for your users if you want to focus also on the querying part. Uh yeah.

I hope you I have answered that. Thank Thank you so much everyone for being