Smoothed and Anchored Rate Calculation in PromQL - Björn Rabenstein, Grafana Labs

This talk focuses on advancements in rate calculation using PromQL, particularly smooth and anchored rate methodologies. The speaker, Bern from Kofana Labs, highlights the limitations of the traditional rate function in Prometheus, identifying issues such as dependency on sample count within a range and its inability to manage edge cases effectively. He introduces two new methods: anchored rate calculation, which uses the last sample before the range to provide integer outputs, and smooth rate calculation, which employs linear interpolation for better coverage and resilience against data loss. Bern explains the statistical underpinnings of these calculations and the implications for monitoring systems, sharing insights on how recent developments have led to the introduction of these experimental features in Prometheus 3.7. He concludes by addressing potential future directions and user feedback on these new approaches.

Hello, thanks for being here. My name is Bern. I work for Kofana Labs. I also do things on Prometheus. Um, I guess that's why I'm here. I want to talk about rate calculation promql, which sounds like a really old chestnut, but we are doing something new now. We are doing smooth and anchored rate calculation. So, let's see what this is. I mean, first of all, let's look

at the old chestnut, right? You probably have all written an expression like this one, right? Rate request total over 1 minute. It's a range selector over a range of 1 minute and then we take the rate very like 101 promql and I claim rate is the most used promql function. Um at least if we include its friends um increase and delta right so in for the purposes

of this talks these function are all doing the same thing I mean they are not right. So delta is just the delta over this range for gauge and then increase is the same thing for counters with counter resets and rate is syntactic sugar to divide the whole thing by the length of the range in seconds. But what we are talking about here that doesn't all matter. We

are just talking about how we do this fundamentally this calculation of an increase or rate. Um this is very fundamental to Prometheus. has been around forever essentially but also the discussion around it has been forever and people have suggested many different ways of uh calculating a rate they have even named it like x rate y rate h rate a rate m rate and these are just the

proposals that got a name right um so something is wrong with rate you might say what's wrong tlddr nothing um it's a lot of all render into the classical current real official endorsed rate calculation but there's always a trade-off right it would take like 3 hours to explain this trade-off and all the aspects and edge cases and this talk is just like 20some minutes so I give

you a very short version and I also focus on the downsides so we we we have actually a motivation to change something right so properties of the current existing rate implementation it strictly looks inside the range inside those one minute or whatever you picked and that's actually a very good property. It it makes a lot of sense. It's easy to reason with but it has implication. The

one thing is um hard to get full coverage for graphing and um if you have done this and you run into this you know what I'm talking about. If not we'll look at it a bit later in a bit more detail. Um the other thing you have certainly run into is since we are only looking inside the range and you need two two samples to calculate an

increase you need those two samples in the range and if there's only one sample in the range you get nothing back right that's annoys a lot of people. Um there are more properties and there are kind of consequences of the first one. Uh we have to extrapolate to the range boundaries because we're only looking inside the range and uh this would give incorrect results. So we have

to extrapolate to where the range actually is. This is a statistical thing and it results in non- integer values and many people are really annoyed by that. I actually think it's a good property and it's not surprising but maybe I'm too nerdy and too much of a stats guy. Uh this last one this is it's not the last one actually the next one. This annoys me actually

quite a lot. Um sometimes you should not extropolite to the end to the very end to the range boundaries. uh this would give wrong results if the series starts and ends within the range starts or end. So you we need some uristics because Prometheus classicalally has no notion when the series starts and ends and this can go wrong and it's very confusing. The uristics is complicated and

if it goes wrong it's hard to see and gives you weird results. Right? So this annoys me the most of all of these. And then there is something which is not a property. It's a not property. It's not composible. So what does that mean? I I put this little expression here. If you take a delta over a minute and then you tell and take another delta over

a minute, which is the previous minute, and you add it up, you should get the delta over 2 minutes, right? Um approximately that's the case, but not precisely with the current rate implementation. And that wasn't a big deal. We didn't really have this on the radar when we designed this, but now people create like distributed promul engines and stuff. Composibility would actually be a very nice property.

So that's something new on the radar. Okay, so Prometheus team is maybe infamously conservative on many things. Pushes back sometimes. I hope for good reasons. But last year things got moving and we started to embrace some change for the rate calculation. So first of all I did one of my infamous brain dumps. Um I wrote everything down I know about this what I have read about the

proposals. Uh those brain dumps are very indigestible very hard to read and um that's why I mean you could read it maybe it's not recommended. it goes into depth but uh Julian Pivoto he did the actual work he distilled this into a readable and actionable design doc and he even implemented it so he deserves all the praise he should actually be here and give this talk but

he couldn't make it to CubeCon so it's back to me to give this talk this whole thought process resulted in something that is um a bit annoying because we came to the conclusion that we cannot just fulfill all the users requests with one better new rate calculation. We decided we have to go two different we have to go down two different path. So we created an anchored

rate calculation and a smooth rate calculation. Um we also decided to not just have like new names uh like smooth rate and anchored rate and smooth increase and anchor increased. We just int introduced this modifier keyword. So we don't have this multiplicity of newly named functions. And uh by the way you could even do it without the rate the range selector just with the anchored and or

with the smoothed it even makes a bit of sense but it's very niche. So usually use this for rate calculations. since we only have 20 something minutes here I am glossing over many details but you can read the design. It's linked here with a QR code and that design dock has all the other links to my brain dump document and to all the other proposals and you

can do all the archaeology and go as deep into the rabbit hole as you want. You can also just try it out yourself. Uh this was introduced in Prometheus 3.7. Uh it is an experiment. Uh it might also change behavior and all those things. uh promql extended range selectors is the feature flag you have to set. Um secret pro tip also say create a time stamp zero

ingestion which is a other experimental feature which plays well with this one and we will look a bit more into this later. Um yeah this is I mean use use the most recent version to be honest because there are bugs that are getting fixed all the time. Uh the biggest downer is that it doesn't work with native histograms yet. And as you might know, native histograms are

my favorite topic. There is a tracking issue uh to add support for histograms and Julian is planning to work on it in April. So we will see. I mean it continues to do all the work. Um okay, now let's look what those things are actually doing. Let's start with the anchored because that's easier to understand. This is essentially there to satisfy the demand for integers in integers

out. So what we do here we take the last sample in the range and then we take the last sample before the range and the difference direct difference without any extrapolation and nothing is the relevant increase. Right? So this means integers in integers out. This works with single point in the range because we only need one in the range. The other one comes from before the range.

And it also gives sorry it gives perfect coverage in graphing. Uh again we will look at this in more detail what this even means. If you have run into this you you know it's it's actually it's not always giving you that but it's pretty easy to get it. You could say it's like request total minus request total a minute ago divided by a minute. If you want

to rate that's more or less what anchor is doing just that this this expression here this expression here down here this is um not doing countries detection and is confused by stainless markers so we couldn't just give you this as a workaround we needed this anchored thing so smooth is the other way um essentially I heard that you doesn't like um extrapolation so let's do interpolation Right?

So we do interpolation here. We take the last symbol in a range and then we take the first symbol after the range. We draw a straight line. Linear interpolation and wherever the range ends and intersects with this line. This is the one relevant value. Then we do the same thing at the beginning. Last sample before the range. First symbol in the range. Draw a straight line. Take

the intersection. And these are the relevant values between which we have an increase. It's an interpolation. Statistics again gives you non- integer values. But it's a really nice estimation. It has really nice properties. Um, it works with zero points in the range because you could just have a range that goes from here to here and the algorithm would still work. Magic, right? Gives you perfect coverage and

graphing always. There's just one big downer needs data from the future. So what is that? Let's say the present is here. And then I would love to know this point, right? You could say it's AI, right? weak age of AI, we can just let the AI predict the future. But we intentionally decided not to use any kind of crazy heruristics to do this. So we just um

ignore that there is something in the future. So you get an underestimate of the range if you if you do it for now. And this was a big big no no for our original rate design 12 years ago or something. We really wanted everything to be real time and the normal query is for now and it should work. So we never even considered any method that would

need data from the future. This has changed because nowadays many of you are not using a single node Prometheia server. You're using some distributed promql engine like ingestion via auto collector into some cloud product. So you are already used to some delay right in ing inestion and there's actually a query offset parameter in the rules config that has been there for quite some time for exactly that

use case like your alerting expression should not like be underestimated because you query for now and you are still lagging behind with some data. So this is what people got used to. So that made us a bit softer about this and allowed this nice smooth thing. we can check how this all behaves. Um, I wanted to look at real production data, but real production data has all

kind of stuff going on, right? So to see certain effects, I created artificial data. I created a little program that simulates two to 20 20 tasks that are running maybe on Kubernetes, doesn't matter, right? They serve 123.45 QPS in aggregate. And then we simulate a rolling restart. Each of these 20 tasks gets stopped and replaced by a new task. Batch zero gets replaced by batch one. Then

you can collect the metrics from all of these and uh sum by batch. And then you see nicely how um yeah the first batch serves 123.45 QPS slowly fades out the batch one kicks in and in the end it serves all the requests. This is normal rate calculation as we know it. It looks as we expected all good, right? So now let's do smoothed. And guess what?

It looks smoother. That's maybe where the where the word is coming from. Um the this is actually an artifact we should not have. This is there are many edge cases that I can't explain in this and this is something we realize is kind of implemented in this slightly wrong way. So this shouldn't be there but it doesn't matter. It's just zero. So yeah. Huh. normal one, smooth

one. We can overlay the the um uh anchored one as well. And then you see they if you squint there is no difference, right? And that's kind of expected. It's benign data. It should be fine, right? Uh they shouldn't be radically different. Radically different. Sorry. Uh but let's zoom in. So now we zoom into the middle where they are all crossing over and the blue and the

orange line are the smoothed one. And you see like they're perfectly straight lines which is cool right because this I know because I wrote the program with the artificial data they should be the truth is perfectly straight right um so we could argue smooth is representing reality whatever that is the best way and then we see the others they are kind of lagging behind they're a bit

of like rough and I mean easy to understand is anchor that's a nice property of the anchor calculation because it's so easy to understand. Uh it looks back, right? Because you take a sample from before the range. Um and this sample could be a whole scrape interval away or it could be just before the range. So you get a variable delay and this annoys me actually quite

a bit that you it depends on the phase angle of your scraping Prometheus uh how long the delay is. And it's different for every of the 20 tasks because that's what Prometheus does, right? it it it it it's it spreads the scrapes. Um and and that's that's a feature of Prometheus, right? So you see those little steps and the delay goes down to here almost zero and

then it goes up to 15 seconds, which is the scrape interval. So it's it's pretty interesting to watch. The classical rate calculation is even more interesting. There are at least three different things that influence it back and forth. And again, I cannot explain them all in the shortness of this talk. So this would be a perfect topic if you want to come to our project booth 12A

project pavilion I will be there for yeah quite often I guess then we can have a chat what this is it's kind of in between right it's a bit delayed but not so much it's a bit more steppy but also not really it's anyway what's arguably more interesting is to look at the complete sum of everything because this should be just a straight line because in aggregate

these these 20 task and the old 20 and the new 20 in aggregate they are serving 123.45 four five QPS and it's not a straight line it's even quite a dip right um thinking about it it's not surprising because that's how Prometheus works it scrapes more or less regular intervals and it's all by design if a task dies it's not scraped in this moment of its death

it uh will have served a few queries that Prometheus has never accounted for and that's fine This is fine, right? Um, what's interesting is that all these three methods see this and they are also more or less the same, right? You could say smooth is a bit less wobbly. The others are more wobbly. Maybe the classic one is the wobbliest one. What's annoying here, this is what

you can't see. The classic right calculation has a huristic to guess the lost samples. and we put a lot of brain power into it and it's kind of frustrating that you can't hardly see this. So, ideally the the green line should be straight, right? But it's arguably a bit higher up than the others, but this whole effect that we worked so hard for is actually not not

working out nicely. So, you could also argue let's not just do it. Let's let's do it like in the new smoothed and anchored where we don't have this uristics at all is fine, right? It's also like the bottom line here is the very lowest one is 117.5. Like it's not a lot. Like if I if I were using like zero as the baseline, you might not even

see this. So it's okay, right? So now let's try something else. Um I always like to say you need your monitoring system most dearly when everything is on fire. Um and the problem is that your monitoring system has a shared fate with the system and monitors. um even if you use a cloud provider because they all run in the same AWS ability zone or something right whatever

if something happens you might uh lose data and here I have removed 10% of the scrapes randomly which is something realistic I guess and then you see everything becomes a bit noisier um but interestingly the orange and blue line the smooth one they're actually still smooth so the smooth rate calculation is really resilient against statistical noise essentially, which is cool, right? Because the reality hasn't changed. We

just removed some samples from how we see it. If we now sum everything up, it becames even more impressive because both classic and anchored is so noisy that I mean, you could say the positive thing is we don't see the dip anymore because it's it drowns in the noise, but the the yellow line, the smooth one, is almost no difference to before. And note this is now

like uh 104 or something, right? And the 117 line is here. So this is also a much wider spread than before. So I'm pretty happy with smoothed here. But let's go like let's summarize the all the properties. And now I finally want to explain this perfect coverage and graphing if you haven't done it. I mean traditionally in Prometheus you would always pay attention that you have like

four or five samples in the range for proper like averaging out. But some people want this like dynamic zooming in and then you have this infamous dollar interval uh variable in graphana which gives you exactly the spacing between two samples like the one in the range query and then you you line up all those ranges next to each other and if you do this with the classic

rate calculation you lose the difference between the last symbol in the range and the first symbol in the next range. People got very annoyed by this. There are workarounds but both with anchored and with smoothed it's very easy to get this perfect coverage. Anchored is not perfect because if the range is so um short that you have zero symbols in the range you get weird effects with

smooth even that is perfectly handled. Right? So this is what people wanted and not entirely by coincidence. The step function is something new in promql which um is essentially what graphana dollar interval is doing. So now you can do it even independently from graphana. It's just the annoying thing about the step function is it doesn't make sense in an instant query and then it just returns zero

and um yeah nothing works. But anyway so this is what you can do now. Users really wanted this. I hope some of you are here and are now grateful. Um, both new methods are almost composable. Again, I'll have to gloss over the details. There are edge cases where they're not composable. We can chat about it at the booth, but you can detect those. And I guess it's

pretty usable this composibility. Neither relies on uristics about beginning and end of series. This is super cool because it's it removes this source of noise and confusion. Anchor needs only one symbol in the range. Smooth needs zero symbol in the range. magic. So, pretty cool. Um, smooth underestimates rate for now. Not cool. Needs a query offset. Um, this is the one thing I don't like about it,

but otherwise I'm in love with smoothed. Uh, anchor. It gives you this variable. Sorry. It gives you the integer results you wanted. Um, you wanted I didn't want it. For better or worse, I still claim that many of you don't know what you wish for, but at least now you have the poison to easily apply and you can try it out. Um, anchored. This is the one

thing, right? It has this variable delay into the past and I'm really ah, that rubs me the wrong way, but yeah. Uh, smoothed handles multiple workers and loss scrapes most gracefully and that's really nice. I'm very happy about the outcome here. All right. So, why did it take so long? Why are we doing this now and and not like 5 years ago or 10 years ago or

something, right? I mean, it's it's a complicated thing, but then all those among you who who came up with suggestions, you could rightfully claim it wasn't even a breaking change. We could have added X-ray rate and Y rate and M rate and A rate all the time. It's just a new function. Uh it has a new name, so it's not a breaking change. Why haven't we just

done this? And you could have all played with it. I mean you could have played with it because it got implemented in forks, right? So it wasn't even the work that was missing. You could you could do the fork or use something where this is implemented. What we said the mantra of open source no is temporary. Yes is forever. We implement this I don't know seven different

proposals. We have a lot of new functions. People are very confused and for half of them we will later notice that we should not have implemented them and then we cannot take them away because there are some people in a widely adopted open source project that are now using it and depending on it right well no it's temporary we can always reconsider which we have done now

essentially right but this person in the audience or wherever might use a counterargument you can mark it as experimental then you can remove it again right maybe because the problem is users just use experimental features and uh they are not may maybe they are not even aware it's experimental and they put it into their critical path and then we cannot really take it away. most infamous example

Prometheus remote right literally billiond dollar industry depending on Prometheus remote right when it was completely experimental so there was no chance we could change it or take it away or something right but there's another argument we could hide it behind a feature flag and feature flags are really nice Prometheus discovered it relatively late in its life cycle it's very good to make users opt into an actually

breaking change because we cannot roll it out to everyone but you can opt in because you say, "I accept the breaking change." But it's also cool to make you explicitly opt into an experimental feature. You cannot claim you've accidentally used it. You didn't know it's experimental if you activated the feature flag, right? You you consciously have to do this. And that's pretty cool. But even then, I

mean, we did this for many features, right? Also, for things where we weren't really sure if it's a good idea, but let's put it behind the feature flag and try it out. The rate calculation is really really fundamental to Prometheus. So we really didn't want to confuse all of you by adding like five different implementations to it and see what sticks. And also like there was none

of it that really wowed anyone in the core Prometheus team. So we really needed this whole thought process and uh soul searching and everything. I like to compare to go generics because that was a similar thing. Go came out everyone I mean not everyone but vocal ones said we want generics. what language is this without generics? And the Go team could have said, "Okay, implement something and

they could have thrown five different generics implementations at the Go community, but they didn't, right? They took their time. They took a long time of consideration and then they carefully launched one implementation as an experimental feature into Go generics uh into Go and then people could use generics and it was iterated on and now we have stable generics in Go. That's kind of what we trying to

do here, right? We we the the bad thing is just that we need two different implementations, but we thought about it as an informed decision. Okay. Still having said all of this, last year had some changes that also triggered the Prometheus team to reconsider things and I want to quickly mention them. The one is the star time stamp which is what you activate with this feature flag,

right? Create a time stamp zero ingestion. This gives Prometheus a notion when a serious starts and that's very cool and it helps to avoid some of the concerns we had. The second one is what I already said like now PromQL is used in many places that have ingestion delay. So we should totally forget about our original idea that always if you run a query right now it

should always give you the readymade result. So we accepted that you need some query delay for something like smooth and we got a nice new way of rate calculation for it. And the third one is really hilarious and this is this is such a nice story to learn from. Um u the range selectors in Prometheus were traditionally um inclusive on both ends. It's a closed interval mathematically

speaking. So they included a sample that might have landed exactly on the range boundary left and right. And that is a really bad property. And we realized this pretty soon in the process but we couldn't change it. And in version three, we said, "Okay, now's the time to change it." It's anyway very theoretical. Nobody will notice. And u I mean, it was marked as a change. So

range selectors and look back delta are now left open in version three. We just forgot sub queries. It's kind of was implied that it also um relates to subqueries, but we forgot about it and we thought this is a bug fix. So in 3.0.1, we fixed it for subqueries. And what we weren't aware of I mean I was aware of that there is a workar around to

essentially implement the anchored rate calculation with sub queries. I was just not aware that many people are using this. It's very brittle. everything has to align right and it's it's also like you should not try this at home right but so many users tried it and they all I mean they just upgraded a bug fix release and their nice workaround stopped working and we couldn't just tell

them ah why are you using brutal workarounds right that was really a strong incentive to say okay now let's let's let's get our stuff together and and give you this anchor thing even though it has many caveats right okay with that lesson Lesson learned. We finished this here. We have two and a half minutes for questions. Thank you very much. If you have a question, there are

microphones. You can walk to the microphone and then I don't have to repeat the question. Somebody's moving. Ah, Ben. Okay, you are first at the mic. Perfect. So yeah, given that now we have this, should we ever use the classic raid? >> Good question. Very good question. I was wondering myself, should we deprecate the classic rate one day? Um, I think it has still many good properties

although they are not as good as we we aspired originally. I mean, I would probably say I if you ask me, I would say go for the smooth one. But many of the people in the room, I'm sure they will love the anchored one because they used this brittle workaround and now they have a proper implementation there. We will just see how it plays out. Right? This

is why it's an experimental feature for now. We can still iterate on it and maybe one day we deprecate the classic one. But let's see how how what what you all think about it. Right. Thank you. Okay. Now Ben, >> can we have a feature flag to make the default smooth? >> Ah, also good question. Yeah, let's let's say if if it if it turns out like

this that everyone loves smooth as much as I do and we really think this is the way to go, then maybe we should consider that. All right, we have 20 52 more seconds. One more. Um so when will this drop in Mimia? >> Um pretty soon I guess because I mean this workaround works particularly well if you use Grafana as your dashboard front end because Grafana always

aligns the query to like full minutes and then the workaround works better. So Grafana users were the the the ones that screamed most about the anchor thing. So there's a strong incentive to see this in in Mimir because that's what our cloud our I mean Grafana had cloud offering back. So I guess if it's not already there it should be pretty soon. I don't know somebody in

the room who who is in control of mim features I don't know not yet. Um, ask at the Grafana booth making like booths are great, right? This is what we are here for, not for talks. Okay, I think we're done. The time has run out and um, yeah, thank you very much. See you at the booth.