Discover Cortex: High Scalability Metrics in 2026 - Friedrich Gonzalez & Charlie Le, Apple

In this talk, Charlie and Frederick, both software engineers at Apple and maintainers of Cortex, present the latest features available in Cortex version 1.20 and discuss upcoming updates in the 1.21 release candidate. They explain that Cortex is an open-source project designed to collect and store metrics from Prometheus and OpenTelemetry, offering multi-tenancy capabilities that allow different teams to query their own metrics while preventing conflicts. The session covers enhancements in ingestion, including support for remote write 2.0 and optimizations for native histograms, as well as improvements in query handling and performance through features like dynamic query splitting and Parquet format integration. They also highlight Cortex's path to graduation from incubation status and promote community engagement in the project.

All right. Shall we start? Thank you uh everyone for joining today. Um I know it's the last day and every everyone's a little tired now, but uh hopefully we can keep you awake during this talk. Um so, hello everyone. My name is Charlie. Um I'm uh a software engineer at Apple. Um I'm joined by Frederick Gonzalez. He's also a software engineer at Apple and uh we're both

maintainers for Cortex and uh today we're going to be talking to you about uh Cortex, some new things like um that have come out in the 1.20 release. I'm going to be going through some of those new features that have come out and then Frederick will go over some of the new things that are coming out in the 1.21 release candidate. Um so, um it's pretty exciting.

And then uh we'll round out the talk with uh the road to graduation for the project as well as um sort of what's in the road map for things to come. And uh if at the end, hopefully there will be some time for Q&A. So, keep in mind uh some questions that you have and uh we'll try to answer them at the end. So, before we go

into those uh features, I just wanted to quickly introduce some of the people that um maybe haven't used Cortex. Can I get like a show of hands, people who have used Cortex before? Okay, people that are new to Cortex. Okay, wow. A lot of people, great. Thank you, you're in the right place. Um so, let me just quickly uh introduce you to what Cortex is. Um I've

been at the booth for the last uh 3 days, so I've been getting this question a lot. So, hopefully I've figured it out by now. But, uh so what is Cortex? Uh what it does is allow you to send metrics from Prometheus or OpenTelemetry collectors uh to a back end that can then store it for you and allow you to query it for a long duration of

time. Um essentially, you can run it as a service for your whole company, allow other teams to send you metrics, and then separate them out by tenants. Um so that way each team can only query and send their own metrics. There's a bunch of other knobs that you can add to each tenant so that uh no single tenant can cause an issue for another. So this solves

the noisy neighbor problem as well. this is a completely open-source project, right? Subatomic 2 and uh part of the CNCF. So you don't have to worry about um you know, the licensing changing or anything like that. and uh if you're using Prometheus, it should be completely compatible with that. So you shouldn't have any issues using Cortex. As if if you're using Open Telemetry as well, you can

send metrics via OT TLP or the remote write uh protocol. So hopefully that solves a lot of the kind of uh questions about, you know, what is Um this is essentially the high-level architecture. Um there is um you know, your your Prometheus uh scraper that's scraping your metrics. Um it's going to be sending metrics to Cortex via the remote write And uh the distributor will be taking

in all of those metrics. And um each of these components here, like the distributor, the ingester, uh these are all horizontally scalable. So if you have a lot of traffic coming in, you should be able to scale out horizontally and not have to worry about any single like components failing because they can't scale up, right? So all of these components were designed to be horizontally scalable. So,

this also means that let's say you have some periodic periodic traffic in terms of you know, certain times of the day you're getting a lot of ingestion, some days you're not, you can scale up and down, you know, automatically to save on costs. So, once your metrics goes through the distributor, this is sort of like the gateway. It distributes it out uh to the ingesters, which is

where all of the time series actually gets stored in memory. Um and uh you can think of it like as a horizontally scalable Prometheus, where um each ingestor basically has a, you know, block that gets created after 2 hours. And um those blocks, once you have enough of them, they uh get uploaded to some block storage like S3, Google Cloud, Azure Blob Storage. So, any S3 sort

of object storage, you can even use Rook. Um and then that's that's where you get that long-term storage, right? It's not bound to a a persistent volume uh to a node. And so, you essentially have infinite retention if you wanted to, which is really, really cool. Um finally, you have this compactor here, which touches those blocks and makes them more efficient to query over time, so that

way you're not you don't have like a thousand 2-hour blocks, you just have one over a 24-hour range. And um that makes it easier to query in the future. So, all of that red line, that's the right path. I mean, sending metrics. For the blue path, you have something like Grafana or Perseus that can query Cortex to get those metrics that you were we're in. Um When

you're querying, you specify this header, and that header allows you to define which tenants am I actually querying metrics from. This allows you to have that multi-tenancy feature, so um, each each query that comes in needs to specify that header. Um, and so all of these components I said, like I said, are horizontally scalable as well. So, if you have a bunch of queries coming in, you

can horizontally scale that out and be able to, um, withstand all of the traffic that's coming in. that goes into the queryers, which can parallelize your query over a long period of time and make it, uh, super efficient to run. Um, I should mention that there's this protocol called gossip ring, and this is how all of the components kind of talk to each other, so that way,

um, if a node goes down, they gossip with each other and tell you, "Okay, don't don't talk to that node anymore. Talk to this one instead." Um, this is like a really cool way of being able to handle, um, you know, failures in the in the network or failures in hardware. And, um, this is something called gossip ring. Um, you can read more about it. And, of

course, there's a lot of caching involved, so you don't need to fetch it from S3 or whatever block storage you have. If you query it recently, it'll store it in the cache and then make it available for future queries. There's also the ruler as well, which handles your recording rules, so if you have those in your Prometheus instance, then you can just set them up on the

ruler, and there's one per tenant. So, each tenant can have its own recording rules that are just querying its own metrics, and then you create new, uh, recording rules based off of that. and then there's also integrations with PagerDuty, uh, and, uh, you know, Slack through the alert manager, which is just, you know, the regular alert manager, if you're familiar with that. Um, yeah. So, that's kind

of the the overview or the the high-level overview for the folks. There are quite a few folks that were uh, new to Cortex here, so welcome everyone. Um, hopefully that was explanatory enough. Um, okay. So, for the folks that have been using Cortex, which is the the lesser half. Um, uh, so I wanted to go over some of the new things that have come out in 1.20.

the first one is support experimental support for remote write 2.0. Um, and what is that? It's basically a better version of remote write 1.0 that uh, uh, that improves like the reliability of being able to send. So, if there's a failure, it'll retry and make it handle those requests better. So, that's experimental support uh, for remote write 2.0. uh, the second thing is this change in how

uh, ingestion works. So, instead of a unary call, there's a stream-based push between that right path, the distributors and the ingesters. So, these connections will, you know, live longer and you don't have to reestablish each time and it's it's uh, improving the efficiency for the right path. The third thing that I wanted to call out is um, better support for native histograms. So, uh, you may not

know but in 1.19, we already have native histograms, but in 1.20, there's improved support, so you can have out-of-order ingestions. Um, and uh, there's better just overall support for knowing for each tenant how much uh, of those tenants are they sending um, native histograms. So, you have better, basically, observability around native histograms. And And plus, there's more ingestion rate limits. Like I mentioned earlier, you there are

knobs that you can set so that others. So, there's there's a a limit now for being able to just restrict um uh the amount of samples that are coming in for native histograms. Okay. So, the on the multi-tenancy, we're still on Uh on multi-tenancy, um there is the ability to query multiple tenants at the same time. And the way that you do that is using these uh

pipe delimited character. And so, in the first example, I have uh three tenants here, user one, user two, user three. And I want to be able to query all three of them at the same time. You can imagine like you have multiple clusters. You And you want to query all of them for an aggregated global view. Um if you have a bunch of these tenants, it can

kind of get messy, right? With all these pipe delimited characters. But you can use regex now to query all of them. And your your use case can be more complicated than this, obviously. But this is a really cool feature that I uh really like. the next is dynamic query splitting. what this means is like when you're querying metrics that are really old or just far back in

time, uh the way that the uh queries get split up depends on how you configure it. And before 1.20, it was basically static. So, if you had a large amount of query or a long amount of um uh the time range that you're querying these metrics for, the sharding would be super like inefficient. Um and we found that out by basically testing and and realizing that the

number of shards was like hammering the back end pretty hard. Um this is how you would enable it. It's like a configuration in Cortex. But um let me show you what the static splitting looks like. So, on the top right, you can see this is like an unhealthy amount of sharding. There's like a bunch of these small little squares. Each square represents a shard of a query

that's happening in parallel. But what you really want is these uh larger kind of shards that are happening so that you can query more efficiently. Um and the way that you accomplish that is through this dynamic splitting. So, the farther back you go and the more vertical um splitting that you have, these shards will be calculated automatically and isn't hard-coded to any specific value. So, this is

a really help if you're seeing slow queries. We have a blog post um about this. We have a QR code as well if you're interested in reading more about how that's uh query parallelism works and the query sharding and the dynamic part of it. Um it goes into detail there. and then um on with the reliability section, um there's a new feature for being able to reject

queries if your ingesters, which hold them the metrics, if they are above a certain CPU or memory usage. So, this is useful like you don't want your ingesters to run out of memory. So, you can start rejecting queries if you start to have um too much memory utilization on Um so, that's a really cool feature. So, let me show you what that looks like. On the left

side, you see that there are no limits being applied. And then right in the middle is when the limit is being applied. in this case, we've set the limit to be 60% of CPU. So, any queries that were coming in are getting rejected. You can do this for memory as well. Uh we have a um a guide guide on how do you set this up. And I've

even cleared the QR code as as Um on the compatibility point, um we have now support for UTF-8. Um so on the top, this is your, you know, your normal um your normal metric with an underscore in that. That's pretty familiar. But now there's uh support for using a dot in the metric name. But if you wanted to do that, you'd actually look at the third line

here for how you do that. And the way you do do that is by including it in that curly bracket and then quoting it. So, you can have emojis in the metric name and the metric label values. Um so that's a pretty cool. Yeah. Is that funny? Okay. Uh the really, really cool feature that uh was added to 1.20 is this Parquet format. You may have heard

of it. It's an Apache uh format. Um but instead of querying metrics by row, you're now querying it by columns. And this is really efficient because if you've looked at um how queries look like in Prometheus, they look like they select just certain columns from all these metrics. Why would you query all of the labels when you just need only a few of them? So, that's what

Parquet is aiming to resolve, especially for long They have to query a bunch of data. So, Parquet aims to solve that. And the way we've introduced this is by um adding a converter, which converts your time series blocks into the Parquet And then um once they're converted, they're stored right next to the time series blocks. Um but then eventually the old one gets cleaned up, the uh

queryers then know how to query them. And uh uh Frederick will talk more about this uh in his in his uh update on 121. uh so, we have a guide as well on how do you enable this. There's a QR code here. Um it goes into detail about why you would want to use this and some of the benefits. So, please check it out. Uh there's also

this talk from Philip from Shopify who uh who did the experiment of um running this in um his setup. And uh is it was really helpful for him. It basically solved all of the issues with um like cardinality. And uh I would recommend highly recommend checking this video out if you are interested in Parquet. He did a great job. Um and I'm going to hand it off

to Frederick now to talk about uh the Parquet mode. Thank Thank you, Charlie. Um so, yeah. So, Philip did this work uh for uh uh Thanos, right? So, he's a is a almost Thanos engineer. We collaborate a lot. A lot of these projects to collaborate together. So, uh it uses the same feature. We use the same feature that they use, but we do have uh some specific

differences, and I do want to call those those out. Um for uh in 1.20, uh the target for us were was actually to get rid of the stateful set in the store gateways. So, we wanted to get rid of the storage gateways and we were able to do that. So, you can use today 120. This was a release we did in November last year. It's already a

few months old. So, you can use that and not have a storage gateway at all. The reason for that is because it's so efficient to send the queries over to the object storage that you can actually do that now. this diagram that you're seeing here is kind of like the updated architecture for that. So, now you have a parquet talks to the object storage. Just run that

and that makes it possible to have the all the all the new parquet blocks. In in 121, which I'm going to start talking about, um we actually went further and re-added the storage gateway. It's kind of funny cuz we we decide we discover in a in the test in that how using the storage gateways to still use the caching in the storage gateway was actually also beneficial

for the caching. So, in 21, which is not yet released, I mean it's released. We have the release candidate that just came out last week. run it if you like bleeding edge technology. You can and test it out and you can have better It's because of the way the storage gateways are named that we're able to specify specific blocks per storage gateway, which is different from the

queries. So, the queries can still do the work, but the storage gateways are better at And this is updated architecture. We're back at the all that you know. It's just that right now the storage gateways are also doing the work getting the parquet files for all the couriers. I want to speed up. I I don't unfortunately don't have the time to go all over the features we

have. We have selected a few. I don't think we did a good job because there are too many. I do want to call out in the last 3 years we've been increasing the number of changes that we're making. That's why we're doing a release right now. Um And and when we all we had a release in November but it made it made sense for us to have

a release. As number of changes um one that I want to call out um as we are there was a call from the community to knowing which features were experimental, which were production ready. And we went through all the features and move a lot of features out of experimental because we're using them in production so they're ready for production. They were just not called out as experimental

as production ready. So all these features you can use in production. They're ready been used for a number of years. Ruler API alert manager API tenant federation cache instance limit. So we have all these features already to use for a This is another feature that's really interesting. Uh if you folks uh uh no we have specific support in Cortex support for sending a pair of Prometheus. So

you can configure a Prometheus pair, send metrics over to Cortex and Cortex deduplicates the metrics from the from from the push. That means you're only ingesting the metrics from one Prometheus not from the other. So we have this support for a long time. Um and the way the feature work normally it required you to have an LCD cluster so that you would send you would synchronize with

that LCD cluster which one is the right pair. With the 120 release we actually uh added support for the chain into member list. So, what that means uh you don't longer need the STD anymore. So, you can just run it and and this is a cool feature uh that was contributed by Sanjay in uh in in Kakao Enterprise. Uh so, yes. Uh be sure to check it

out. Um Uh this is my dog. Um uh one of the features that we have right now um is uh there's a typical question we all have is are my query I are my metrics queried at all? Like I'm storing all these data. Is it being used at all? And uh we add a uh actually it was banned who contributed this feature, which allows metrics from the

courier the the query the query metrics that are exposed. It's a very complex uh thing that we did there. I I I won't go all over the details of it, but basically allows you to have metrics about your usage. Like are you actually querying the metrics that you're storing? This um we're hoping it's going to help people figure out if the um what query patterns are they

getting. And it what kind of metrics are you querying? So, you're going to be able to say, "I'm act Am I actually uh querying the metrics that I'm storing, right? Why am I using all this data for?" And uh Um also important for this 121 release, we are adding more more support for OTLP. This is a feature um came from the community as well. Uh success is

metric success is a feature for OTLP. We support it already. This is a uh a feature that's coming from upstream. Some features we come uh come from upstream, we just get them, and some features uh come from us. This one is from upstream mostly. Um this feature is from us. Uh override API is the ability to modify as a user your own limits for your own tenants.

So, you don't no longer have to modify a config map. A user you get the ticket, you modify the config map for the user, you just let them know. Now the users are in they can also modify their own their own limits. To a certain extent, you can actually set up specific hard limits that cannot be modified. So, you can you can juggle around and let them,

you know, modify their own metrics. Okay, so we have 5 minutes. I do want to call out a bit on the graduation. Cortex is a project that has been a long time running, 10 years. We almost This year we're going to make 10 years. So, still in incubation phase. We're trying to get out of it. Uh last year we asked a lot of you to uh put

your company name in the in the in the adopters. A lot of you did that. So, if if you were an adopter of Cortex, you can put your name there. Put your company. It helps a lot to get this uh graduated. Also, in the process of graduation, we're right now doing the Cortex security review, third-party review. So, this is currently running. It's going to be running for

the next 3 months. So, we're going to see this is going to be the first public review that we had. We had a number internal, but this one is public. Uh this is how can you get involved with us? meeting every 4 weeks and we all have a Slack channel. You can go there, ask questions. We always active there. Uh if you want to file issues, if

you have any requests, please welcome to do so. I want to move to questions. I think we have 4 minutes for questions. Thank you. >> This is Mike. Hi. Thanks. For me, Cortex sounds very much like Thanos. Can you give me a pro and cons why I should use Cortex or when I should I use Thanos? Yeah. So, I I think you can also add to So,

so Cortex and Thanos collaborate. They're both CNCF projects. We have maintainers for one maintainer of Cortex is also a maintainer for So, we collaborate a lot internal. There's some features they focus on, some features we focus on. They focus on having the sidecar. We also we focus on the multi-tenancy. It's not really clear right now if really makes sense to have two. Let me put it that

way. We we collaborate a lot on upstream Prometheus projects. In PromQL engine, you can also use PromQL engine from Thanos. We also there's also compatibility. You can put Thanos blocks into Cortex. They work. You can take Cortex and put them in Thanos. They'll both work. Uh So, we are not competitors. We are more like collaborators. Okay. If if you want to know like if you want to

use Cortex, Cortex works more like a SaaS project. Like you just have an API that you point at it. The users are able to control their access in any way possible. The case of Thanos, you still have like some configuration files that you have to go through. It's just a different approach. Okay. Thanks. Thank you. Sure. First of all, thank you for the talk. You just mentioned

that you're closely related with Thanos and they also had a talk here where they pitched Parquet. Is this a shared thing or is it implemented in both projects? >> it's uh it's a shared thing, yes. It's a shared thing. There are some parts are shared, there are parts that are not shared. Um but yes, the main core of the feature is actually a shared one, yeah. Okay,

thanks.