Prometheus V3 One Year In: OpenMetrics 2.0 and More! - Jan Fajerski & Bartłomiej Płotka

This talk focuses on the recent updates and future developments in Prometheus, particularly since the release of version 3.0. The speakers, Yan and Bartek, discuss enhancements such as stable native histograms, improvements to the OpenTelemetry Protocol (OTLP) receive endpoint, and the integration of resource attributes into metric labels. They elaborate on experimental features in PromQL, including easier outer joins, advanced arithmetic on timestamps, and the addition of new keywords. Furthermore, they touch on the importance of community contributions and their governance, as well as ongoing initiatives such as the transition to Parquet storage and the introduction of new storage formats. The session emphasizes collaboration and solicits feedback from the Prometheus community.

Great. Welcome everybody. Uh thanks for coming. We uh have a pretty jam-packed presentation. So, uh yeah, let's not spend too much time on intros and everything. We'll talk about what happened in Prometheus since V3, so about last year and a half, and where it's going. And um my name is Yan. Um I work with Prometheus and around Prometheus and some other stuff uh and recently I moved

to a forest with great internet connection, so I get to look at these things when I don't look into the magic box. And with me is Bartek. Yeah, my name is Bartek. I work at Google. I'm responsible for Google managed service for Prometheus and generally observability at Google. And um I love, you know, open source. I maintain a bunch of stuff. I love mentoring. Um so, let's

do some mentoring together if you want. And I also wrote a book called Efficient Go about optimizations, and I I love that stuff. And in my free time I tend to do motorcycling, but recently I have two little humans. So, it's a little bit more chaos, but it's definitely enjoyable anyway. And but I miss my time with motorcycle as well. Let's go. All right. So, here's what

we have planned for today. Um we'll we'll dive right into a little recap, uh then sort of the last year and a half, and then what's coming, couple of community updates, and then we have some requests to you guys, too. Okay. Um let's start. Who here doesn't know what Prometheus is? Oh, okay. More than I thought. Um who here runs Oh, sorry. Who here runs Prometheus? All

right, keep your hands up. And uh lower your hands if you run 2.something. Okay. Okay. So, keep your hands up if you run three. All right. Pretty good. Not a whole lot, but maybe, I don't know, 6 7%. Um awesome. So, since we have uh more people than I thought who don't know what Prometheus is, here's what Prometheus is. Um it's uh TSDB at its center, um

and it gives you a service discovery to discover scraping targets, where it gets all the metrics that it ingests into the TSDB. And then on the other side, it gives you a query interface with PromQL, um and you can define rules, which can be also alerts, but also recording rules. And ultimately, you get out dashboards uh of your metrics and alerts. So, that's sort of the speed

run. Uh since we don't have much time. Um Prometheus V3 uh was a pretty huge deal about 16 months ago. Uh the first major version in uh 7 years, I think. So, at the time, Prometheus 2 was like 2/3 of the lifetime of of Prometheus. Uh was time, and um it created a lot of buzz in the community, so there's a tons of features, uh new UIs,

you know, changing of character supports, and for us, very important, we could introduce some breaking changes. Um if you want to know more about this, there's plenty of talks already. Um just scan this QR code. There's a YouTube search um with all the talks. And um if you want to migrate, if you're still looking to get to 3.0, there's a migration guide that you should check out.

Okay, this is uh sort of the adieu of getting started. So, now we want to look at um what you maybe don't know about yet. native histograms. Native histograms are officially stable. Um if the histograms have been a part of Prometheus um for a long time, if not the beginning, um but native histograms basically improve all aspects of it, and you really want to use them. So,

um do go check that out. Um you no longer, since 3.8, need to define the feature flag. Um but with uh since 3.9, you now need to take action and uh configure this on a scrape level that you want to ingest native And they're great. We We'll keep working on them. There's still lots to do uh on the uh PromQL side. I think there's a couple of

gaps still, but also, of course, on the ecosystem level. The exporters all need uh need to expose Kubernetes is uh one offender that we really want to push to give us uh native histograms, cuz um it can, you know, have a serious impact on your head series if you uh monitor Kubernetes. Um OTLP. Um we added a whole bunch of stuff to the uh OTLP receive endpoint.

So, um there is the first signs of delta ingestion. Um if you don't know what what that means is you In Prometheus, you typically give Prometheus always the the the counter, the global state of the world. Um with delta, uh that's a different approach of sending metrics, uh where you always just send the differences. Um has different trade-offs. Um we'd still prefer accumulative, but um delta is

sort of taking off, even though very primitive so far, but there's more What else? Oh, yeah. Uh so, um OTLP now goes straight into the TSDB, or straighter into the TSDB, which has quite large performance uh uh improvements behind it. So, you know, 25% faster is a number that gets thrown around. last but not least is the um resource attributes. Uh you can now promote those to

actual metric labels, so uh normally they would be land in an in a so-called info metric um in Prometheus that you can then use to sort of um extend the your queries. Uh but if you have uh attributes that you find very important, then you can uh um raise them up to the level of a label, which will have impact, of course, on your cardinality, but uh

you have all your labels that you want. Uh speaking of info, um this is how you would sort of use uh these info metrics. So, you sort of do a join on an info metric on an info metric to get all this uh additional information. Um we nowadays also have Oh, wait, this is already the next topic, experimental PromQL extensions. Um so, nowadays PromQL has the info

function, which is some uh syntactic sugar, which um makes this look a whole lot neater. All right, more experimental PromQL. Um I promised this was going to be fast, eh? outer joins uh are now much more comfortable. Um if you know Prometheus, you know, the semantics are usually more uh of often like an inner join if you um do operations on on metrics. Uh there was always

a way to get outer joins, but they were kind of awkward. So, um nowadays we have these PromQL extensions uh that fill sort of the the um metrics that would be missing from an outer join um with default values that you supply. Um you have access to the timestamps of interesting samples in in a certain timeframe now. So, you know, the timestamp of the the peak in

the last 2 hours, for example, you can extract that nowadays. Um there is the arithmetics on durations. This is um sort of part of a work. The The original issue actually asked to be able to do um arithmetics on uh arithmetics on um timestamps that follow the at modifier, which uh to me sounds a lot more useful than this. Um but um that's the, you know, we're

on our way there. And uh last Oh, there's two last but not least, but on this slide, um the type and unit labels. you can you can enable these, and then you get these extra labels, uh but really this is just sort of the user-visible change to a to a much bigger change that's sort of in the storage layer, where um the the metric type and the

metric unit are supposed to become sort of part of the metric identity, right? So, that um eventually you might be able to overload metric names, as long as they have different types. Sort of that's the idea. But you can also use it to um look at all the histogram uh metrics that you have. Okay, now really last but not least for is two new keywords that follow

vector selectors, um smoothed and uh anchored. Uh I think smoothed will be very interesting for a lot of people. Um go check it out. There was a talk on Tuesday. Um so, I'm just telling you about a talk someone else gave. Um Björn gave the talk, uh and he presented the work that uh Julian did. So, you know, that's how open source communities work, I guess. now

I'll take a breath and hand it over to Bartek. Let's go. Yes. So, generally, we are there are plenty of things we still um we added recently that are part of next Prometheus version, let's say, but it's already merged. Generally, like bunch of, you know, fixes and optimizations. I tried to kind of capture them in some, you know, graph of where where were the most kind of

attention there. But um the point is like, as a community, as contributors, we are constantly thinking of what else we can optimize for for Prometheus users. And especially when we want to fit a new feature, which is useful, but has some overhead, we try to make room for that. So, um just a reminder, and please help us as well to you know, notice things that could be

faster or, you know, less expensive. We have that as a as a top of our mind. So, we constantly improve Prometheus. Um Two things we I want to mention that already kind of you can try out. We were We are progressing on the start timestamp um story, and essentially, you know, adding a native storage implementation for this. Quick reminder what start time start timestamp is. So, you

imagine those orange boxes are samples and before start time you only have a pair, you only value and a time stamp, right? And when the value represents a gauge, right? Gauge semantics, it's kind of enough data. It's like gauge represent the current observation for a given time, so you don't need to any you know you don't need you don't need any additional information. For cumulatives, this is

different, right? Every sample represent an absolute absolute count that started counting at some moment and uh usually the same moment across the scrape target, usually process start. So, in this this moment this kind of moment is called start time stamp. And Prometheus historically didn't would was never even saving this information um because we have some characteristic that guessed this. And having this has lots of many benefits,

you no longer have to guess this reset detection uh thingy what PromQL rate is doing. So, essentially waiting until the value of the counter is essentially essentially lower than the previous value, we know reset happened, we don't know when exactly. So, there are many edge cases where that was not fitting, especially short-living jobs, very long scrape intervals. So, start time totally helps with that and compatibility with

OTEL and any any kind of like systems which are delta-based. Um and really enabling delta in future. So, that's already um there and how we implemented that, we literally in every place where we have a sample, now you can store a triplet. So, another information uh called start time and this is kind of coming into 3.11 and what implemented is essentially a storage layer, so you can

save this information, you can programmatically for now retrieve that, and then you can send that through remote write to your external system. And we are working on PromQL support that make, you know, use of it um you know, like for for rate accuracy. And and we need to implement histograms, so there's still kind of things to do, but like the float should work and we will be,

you know, improving it over time. Right now, you need to enable this special flag SD storage. And really connected to this concept uh to really be able to persist this start time, we uh we had to change XOR format. So, our beautiful compression format where we encode every sample one by one together for each series and it we we did always, you know, a huge work to

make sure it's, you know, fast enough for general public. And now, you know, there were two streams of work, start time stamp that has to kind of increase add another information to this, but also there was another stream of work from Julian, from other maintainers who who who we wanted to apply lots of learnings from this very old format, probably from the beginning of Prometheus. We want

to improve it and there are already we already improve it and and, you know, improve like a tiny nitty uh you know, optimizations around encoding to support staleness markers better, to support new jitter between scrapes. So, everything is just, you know, compressed better and reads are faster and writes are faster. So, we kind of combine those efforts and now you can really use XOR 2 already um

in your Prometheuses. Don't use it in production yet because we reserve the right to maybe change it for some time just to optimize it further, but you can test it out and tell us, for example, maybe in your data it's slower or actually a lot of better, so we should, you know, maybe maybe stabilize this faster. But this is something for you to test and we should

I mean, our focus is to stabilize it as soon as possible, so you can test this and run this on production with compatibility guarantee, right? So, this is something that's coming already in All right. Now, even more experimental stuff. I will be fast here, but I I want you to kind of understand some kind of big initiatives um so you can help, so you can join, you

can give feedback if we want to uh if you want this or maybe you are worried about this. And you can learn more on on some links we are leaving on the way. So, first one is really related to the start time stamp, so delta metric, right? So, we used to have a primitive way of doing them and receiving them from OTELP and save them as a

gauge in certain form. Now, we are really trying to make sure there's a native support for delta metrics and start time stamp storage enables that. So, you need to have enable this flag and suddenly start time already kind of have natural um natural kind of support for delta because you can think about uh delta sample as a very short cumulative counter, like one sample cumulative cumulative counter.

So, rate supporting start time in PromQL already supports delta in some way. So, um this is happening and, you know, we still want you to use cumulatives, they are much simpler, much kind of more reliable um in general, but, you know, there are cases where you have to push data like, you know, um serverless or short short-living um job kind of environments we want to support better.

So, this will be available in Prometheus very soon. So, follow those two proposals um give feedback, mention, "Hey, I want this fast, so I want to deprecate my statsd pipeline", stuff like that. So, make sure to follow on those things. That's my Can we select our our thingy? I'm sure it'll fix it. I don't think it's you. I mean, I will continue, okay? Maybe one of those

Maybe one of those makes sense. The top top level. All right, I will restart for a second just to quickly see if there's anything on my side. There we go. Now, it's better. Thank you. All right. So, keep in mind those discussions and we're almost there and we should have a delta in Prometheus. Next topic, Parquet storage. I won't going to go deeper into this, there are

good talks from PromCon you can visit about um this potential future. So, there is a perspective where Prometheus next uh storage for metrics should be kind of based on Parquet model, Parquet storage. And uh there are prototypes for this already as and some production company some companies use it on production for Cortex and Thanos. And and what's amazing, this kind of idea was really started from, you

know, top-down. So, started from those Cortex, Mimir and Thanos um projects collaborating together into this new storage that is super important for, of course, those high, you know, um high-scalable setups first for uh high cardinality, but, you know, it can be mapped into Prometheus and Prometheus itself can have some benefits. So, this is still happening, working right now mostly on Thanos side, but you can totally join

this uh this effort, make sure you motivate us to get us to get this higher cardinality support into Prometheus and we already have a lot of good uh really results on chunk itself, compressions on on on a disk and object storage, but read times, especially for like detail matchers, um you can kind of select only certain columns with this columnar storage. So, super super beneficial for high

scale. And yeah, help us on. Similar related to Parquet, we are working on this metadata storage um part that really utilizes um the resource attributes data and kind of integrates and scales that part, but we also can use it for like a proper type and unit help support and even like details like, "Hey, scrape interval." So, you know, when you are querying what scrape interval was used

for those metrics, which just improves accuracy of dashboards and alerts and everything. All right. And few last things um composite sample storage. So, this is essentially um you know, something that we see is converging in Prometheus. So, you know, usually we had like classic histograms and we, as you know, we introduced native histograms now. So, essentially instead of like having tons of series, we have one sample

representing one histogram as it should be. And recently kind of we introduced native histograms with custom buckets and HCBS, which are generally like native histogram with explicit bucketing, also sparse uh that really can map classic histogram one-to-one. So, the golden idea is to make sure uh we we all can use it, right? We all can transparently swap this classic histogram into into the native one and and

have all the kind of beautiful parts for uh for doing so. So, it's just compressed better, you avoid indexing cost and yeah, it's transactional because you no longer have separate metrics for different parts of the same histogram, like count, you you have everything within one sample. So, on when you are having like a complex pipeline, you have assurance it will go together, it will be sent together,

right? And you can already use it and convert classic histograms into the new histograms and like not native, but not native histograms with custom bucketing. So, really data is the same. And the only kind of blocker right now, or not blocker, but some challenge is that this is changing your queries, right? Because you no longer have those bucket suffixes metrics and count suffix and sum suffix, you

no longer have LE label, but you have essentially um you know, a simplified one sample with a different name slightly, right? Uh and this LE less equal bucketing boundary is built in within sample, so uh you don't need to like reference this. And that's was becoming kind of challenging and some companies totally, you know, rewrote their dashboards and alerts and it's already kind of working for them,

but we still see some you know, lack of adoption because of like it changes queries and users would like to choose the time when they rewrite the query, right? So, we are prototyping with this feature PromQL HCBS classic, which essentially translates this composite samples like much better stored information and represent them as a classic one. And this is controversial because you are querying, receiving data which is

slightly different than you have in storage, but represents really the same information. So, you please give us feedback if you like something like that. Um I personally think it helps uh with compatibility with ensuring we can migrate to the better storage sooner, but uh feedback welcome if it's too magic, too complex, and too many edge cases. Um so, we we try it out and you can comment

and give feedback on this pull request and we hopefully maybe have a feature flag for that for you to try. And there are tons of things we have to uh still do even within the composite storage and NHCBs, there is like a new prom cool function we can add. There is federation story we have to kind of sorted. Remote write two has to be improved around NHCB

metadata. And we have opportunity is to have summaries as well optimized like that and state set which is also composite sample. Wow, and then last thing related to this, open metrics to zero. Um so, this is kind of a new, you know, version of the of the protocol open metrics that was created like 7 years ago and we are kind of going one one key change is

that we um we kind of try to make this composite sample uh possible. So, essentially, you there will be no classic histogram like that. Every classic histogram will be represented compacted form. It will be one line. And the native histogram is are now supported as well uh also with one line. So, that's going to be kind of probably the first thing you will see when you have

a the metric page uh with this new protocol. And um you know, we are working heavily on this and we're just about to release the first a release candidate so you can you try to use it and read and and you know, give feedback. But, there are a bunch of stuff we changed to essentially up uh apply learnings from using open metrics on on on scale um

and uh you know, around reliability, efficiency, and compatibility with new Prometheus features and auto features. Um so, yeah, keep that in mind and test it soon. All right. And then kind of like eventually um you know, we don't have time for much else, but I want to list three bonus points. First one, we have we are working on optionally schematizing your metrics. So, you will be able

to use auto schema and and and use some kind of nice benefits of it. You can version your metrics better. You can generate instrumentation documentation. All of those things. We can build some translation layers so you don't break queries when the new version schema version happens. Um so, really really kind of a big initiative, but again, help us, give feedback if you want this or you you

you don't want Um it's super import important that you share. We are playing with some fun experiments around alternative prom QL maybe syntax that has variables and variables and pipes. So, maybe it there are more readable. Very controversial. Not everyone likes it. But, again, please us uh please add any feedback in what direction you would see this happening. And the last thing is like we are playing

with some stochastic kind of you know, models to um yeah, uh kill your se- reduce maybe drop some samples, drop some scrapes, um drop maybe a target, but avoid like the whole Prometheus going down. So, those kind of like very controversial algorithms that maybe are scary, but kind of makes the whole Prometheus experience much more reliable. So, that's from technical aspect. Hopefully, that's not too much. And

if I if you're unsure about any of this, just reach us after. We can we can, you know, explain more. All right. Then uh quick look into the community. Um we'll start this with a big big thank you to Björn. He's here in the room. So, maybe a round of applause for 12 years of dedication and hard work. >> [applause] >> Yeah, thanks a lot, Björn. I

think um without you uh Prometheus would certainly not be what it is today. Thank we have a new governance. Um actually, um well, my my notes here are wrong. Um so, we have the document uh sort of agreed upon. This took us only a year. Um we now actually voted on it. So, it's officially accepted as our new governance. And now we start the process what is

laid out in the governance. Um How is that interesting for you guys? Well, the the goal of this rule is to make onboarding to become a Prometheus contributor um a lot simpler and a lot less scary. So, um we really hope that we get uh attract more contributions that way and uh specifically sort of long-term contributions, right? So, um keep your eyes peeled and hopefully this will

take off as we hope it it does. We had uh PromCon uh last year in Munich. Very much sort of the polar opposite to um this event here. Um single track, very nice and uh cozy, you know, uh super technically detailed talks uh not really companies. Uh good stuff. Uh there's a cool little teaser video if you want to check that out. Um and keep your eyes

uh open on promcon.io for uh future plans. We definitely want to repeat this um but uh nothing nothing definitive out Then contributions. Um this is sort of the um CNCF uh um stats on contributions to the uh Prometheus org. Um you can clearly see uh Prom- Prometheus Prometheus V3 has caused a lot of buzz and attracted a lot of new contributions before and also after. And of

course, right in the release months uh we we had a flurry of activities um that has sort of uh um regressed to the to the mean a little bit. Um that might look not so good, but it is actually still very much a healthy ecosystem. I just pick out one of the projects here. So, this is when you look at a project level. Alert manager has classically

struggled a bit um with with finding uh maintainers, but then we had PromCon and we raised this topic again and actually four people raised their hand and now we have a completely brand new maintainer team for alert manager. And I think that's also worth a round of applause. >> I believe one of them should be in the room, too. So, carry that on. we can't have a

presentation these days without mentioning this, right? Um From our perspective, uh it's not all unicorn unicorns and rainbows, right? We we feel the sort of pressure and the impact of it like many other projects do. Um so, maybe it's worth taking the time here on the stage to sort of um show you the the, you know, hairy underbelly of this. Um and to give you actually a

concrete example, I picked out uh um one individual who um uses AI to contribute, at least I suspect so. Um what do you think? I think you see this number up there, I don't have a laser pointer. Yeah, uh almost 200 contributions in a week. That's not bad. Um but what this actually looks like is this. So, all those red dots are closed PRs that somebody went

looked at, saw that it was bogus, and closed it. Um and that's this is just one person. So, this is the pressure we feel. This is sort of the overhead this causes. this is not cool. So, what to do? Um we don't want to discourage anyone from using AI. Um this has already produced uh valuable contributions. Um we use it, too. Um we would just like to

put the word out to use it responsibly, right? And um if you want to really uh become a Prometheus contributor, here's sort of the the high-level things you can do, right? Um it's all about trust. It always was, but it is more even now. Um one thing you can do is just tell us that you used AI for this. You know, there's there's no shame in it

or anything, but it uh gives us a lot of important context. And at the same time, um we still want you to know what you're doing, right? You're signing the DCO. So, you you should um have checked what it does anyway. And uh even better if you can can tell us like, you know, um this might have impact on another feature of Prometheus. We don't expect you

to have all the answers, but um if you demonstrate insight, um that that creates a lot of trust and that um and unlocks the interaction with the maintainers and reviewers. And then, as I said earlier already, um we really want people to stick around for a while. Uh not not just a drive-by contribution, right? Stick around, help us triage issues, help us review code. We've got a

lot of code to review as you've seen. Um this is really where we need most uh most help with, actually. It's not the writing the code was always easy, right? Um this stuff is where it's hard. And then one more thing. We have a user survey up. If you visited our booth, you might know already. If you use Prometheus, um there's a couple of questions. Really not

not many. It takes 5 minutes out of your day. You would help us a lot by giving us some Um there's a link also on the slides so if you don't catch this one here, but it'll be on the next slide, too. Um then um yeah, do it. So, this this red QR code is still the survey. Um there's some feedback for us if you want to

give it to us. And uh I think we have a minute if there's a question or two. Anyone of you want to ask questions, please go to the microphone. It's in the in the middle. >> Yes. Audible? Yeah, cool. Uh I was just wondering if is there any Do you guys see um uh an alternative to Thanos for like uh managing fleets of um Prometheus or is

Thanos still going to be the way? I mean, I'm biased. I create I kind of co-created Thanos, but um so, no. No, but like there are there are of course. And like uh the space is like kind of really nicely collaborating between Cortex, Thanos, and Mimir. So, really there there are differences, but generally they have similar architectures, so they would have similar trade-off, but you know, on

details one might be more, you know, for you than others for sure. And and that's from open source CNCF side and there are of course vendors that can help you, but this is kind of the spectrum. I don't I wouldn't say even Cortex or is worse or than Thanos or better. I just very equal and but just different on details. Yep, thanks. Hi. Uh I would like

to ask last year there was a renaming of metrics session and I didn't see anywhere in back or what's the situation there? Yeah, so uh last year I presented about automatic conversions based on your schema. So, um and already we have uh you know, this this kind of discussion. This first kind of like point is that I can talk that happened on KubeCon and you can rewatch

with details, but this is kind of one progress where we work on tooling that actually works with Prometheus to generate this stuff to even define your metric in a schematized way. And that's kind of, you know, the first block that has to happen before we even do any prototypes, we merge any prototypes uh to Prometheus. So, right now it's in the prototype stage. This this translation is

still happening, but we don't have people really defining their metrics in Prometheus format yet. And we have to and there is also like some mapping um work to do together with OTel, but it's really lined up and we have this telemetry policies is very relevant topic that um happens on OTel, but we collaborate with Prometheus, so you can translate the telemetry based on schema. We need to

aggregate those so many you know, initiatives into one thing eventually, but it takes time, but it's happening with the point is, yeah. Thank Hello. Hello. >> Nice to meet you. Um I see on the slides prom {dash} something in almost all the slides. Uh could you tell us a little bit about what those prom stuff is? Oh, yeah. Certainly. So, um Prometheus has a somewhat formalized proposal

process nowadays. So, if you go to uh the GitHub repo Prometheus {slash} proposals, uh you can see There you go. You can uh read sort of what gets proposed and leave comments um and And we just create we just create like a short uh you know, uh shortcut shortcut number, so every pull request that is against this proposal process repository has its number and that's number is

the ID. So, it's very easy to once you work with it, you can just exchange the number and it's much easier to um I identify the proposal. So, if you see prom something, you know, it's a proposal you can really visit and that's the number of the pull requests, which is 48 for example. Is this aimed more for maintainers or do >> No, for everyone. We reference

proposals for Prometheus. >> We reference. We even have like some nice automation where where you mention prom 60 uh or prom something at the Prometheus repo, it will actually automatically, you know, expand to to the link. So, yeah, thanks for the good question. All right, I think we're out of time now. Yeah, but we'll be here. Thanks everybody. Thank you.