Accelerating Thanos at Scale: Faster and Cheaper Queries With Parquet - Giedrius Statkevičius

This talk covers the advancements in using Parquet format for enhancing Thanos's query performance and storage efficiency. The speaker discusses the challenges currently faced with Thanos, including slow query responses and excessive memory usage, and how transitioning to Parquet can address these issues. With the ability to reduce data storage by over 300 terabytes without losing precision, the adoption of Parquet offers significant benefits for managing metrics data. The session provides insights into the execution of queries using Parquet, highlighting how its column-oriented feature allows for more efficient data fetching. Additionally, the speaker presents a migration strategy for existing Thanos users to adapt to this new format, emphasizing the ongoing developments within the Thanos project and the opportunities for community contributions.

Hello everyone. Welcome. Thanks for coming to listen to me. I will be talking about accelerating Thanos at scale, faster and cheaper queries with with Parquet. So, the current format is already quite fast, but with Parquet we can achieve even faster queries and we can use much less space to store the same exact amount of data. Invented, we are quite huge users of Thanos. I think in the

biggest region we have over 700 terabytes of metrics data over 1 year. So, we are very excited about this project. We contribute a lot to it and I hope that you will be hooked onto it as well after this presentation. this is kind of like a I would say maybe even a trilogy of series. So, there's this one presentation 2 years ago, I think, by Philip Kominek

of who inspired our upstream work. Philip goes into detail about what are the current problems with this TSDB format and how Parquet helps with it. He goes much more in depth. And there is another presentation from last year where people from Grafana, AWS, and Michael from the Thanos team presented this work also. But from the upstream Prometheus point of view. But today we will be only talking

about Thanos specifics. So, Thanos is another project in the space. And I believe yeah, that after this you will be able to take advantage of the Parquet format as well. I will add links to these videos in the documentation, which and and the link to the documentation you will find it in at the end of the slides or you can just go to Thanos.io and to the

store page. So prehistory, uh Thanos also I would say Thanos spearheaded the current uh way Mimir and Cortex work in in the sense in the sense that they upload objects to object storage and read off from them. So Thanos also suffers from a lot of the same problems that are mentioned in other talks. So I'll just quickly go over some of the issues that we have on

our tracker. So the first one is Thanos storage too slow for query and don't know where the time went. So depending on your luck factor, you might your queries might be quite fast or quite slow. It all depends on the form of your queries and how the data is laid out. I will explain in a bit more detail shortly what that means. Another issue, very old one,

offset table size exceeds 4 bytes and block table size exceeds 4 bytes. Uh So it is possible to lift this requirement in the old format, but we still haven't done it. Thanos compactor employs some uh tricks or you can call them hacks to work around this problem. So if the if in the compaction group the sum it has some heuristics. So if the sum of index files

would become too big, it drops some of the blocks from the compaction and some blocks never get compacted because of this. Uh document drove guidelines on sizing store gateway 4,802. So, another very old issue. Uh because of these issues regarding unpredictability, it's really hard to size the store gateway properly. So, what happens in practice is that a lot of people over provision resources for the store gateway.

And the final one might be familiar to a lot of users that uh Thanos store consumes a lot of memory at startup and that leads to out of memory situations. this is caused by the building of what's called the binary index header. So, and we need to build it because uh we need to amortize some of the costs associated with fetching data from the index file. Without

this, we would uh get even worse performance. Uh so, what is Parquet? This is the definition from the official Parquet website. Um in my own words, I would just uh describe it as a project from the Hadoop where uh it's a a data format which has a schema and it's a column oriented and it uh lays out data very nicely in the sense that uh there are

these uh things called data pages and uh uh data pages are the minimal uh amount in a Parquet file and you don't need to fetch lots of small things from the file. You always fetch pages, essentially. So, why is uh Parquet a good fit? Uh majority of queries use some amount of aggregations, so we don't actually with a column oriented format. We don't need to fetch all

of the columns. We can fetch only a few for so in this example we would only need to fetch ABC. Because when you use some by over columns or labels just disappear. You just need to some by those three dimensions. So it's really not needed to fetch data of other Good compression and current decoding possibilities. So can fetch multiple pages and the code them concurrently. And yeah,

through data pages we can fetch data efficiently instead of fetching small amounts of data from the same object. In a in a random manner. So just a very short overview of how uh query execution works like on the storage layer and Prometheus and parquet. It's a very simplified model. Doesn't have lots of details. But the point of this slide is that you need to fetch for each

matcher postings. So lists of series IDs for each matcher of the current format intersect all of that data. All of those numbers then fetch the definitions of series which contain references to the symbols so the labels and then you fetch chunks. But with parquet you only need to fetch matching pages. Scan them and fetch chunks. Parquet has this feature called statistics. So it Each file contains some

metadata about each page. What's inside of it. So for example if you are fetching uh I don't know. There's a matcher handler equals to API V1 whatever. Statistics allows us to tell whether that data that we need this is whether it's inside of some page or is it not in that page. So, we can filter out lots of pages and don't even need to fetch them. Yeah,

and this is a table from the TSDB documentation from the Prometheus repository. So, as you can see, postings are laid out one after the other. Series information is laid out one after the So, if we need to fetch let's say series from one to 10, then from 1,000 to 1,100, and so on, you need to do a lots of random reads. And if you are unlucky, there's

quite a huge gap between the data that you need to fetch. And then you need to do multiple requests to the object storage. a lot of people might not know that in I probably all object storage implementations, you cannot read multiple byte ranges in one request. So, for each range that you want to read, you need to send another request. that's as far as I know. So,

we need to send multiple requests, and more requests you do, the worse the performance is. Uh the schema. So, as I've said, the Parquet formats have a schema. It looks quite similar to what you have in other projects like Cortex. So, first we have the uh bitmap of populated label columns per row. So, it allows us to know whether some column is populated or not. We have

the hash. Um it's used for project projections. So, if projections are enabled, like you have some by some labels, you only need to compare the hashes uh to know whether that series is unique and then we have the chunk data. So, uh I don't remember exactly why, but probably to have a more granular uh fetching capability. So, for example, if you have uh if you your query

uses last 12 10 12 hours of data, you don't need to fetch all of those columns. All you only need the first two. But, with the old format, you might uh uh fetch more data than needed. So, it's a trade-off between uh how much data you need to fetch versus uh how much time uh you will spend decoding all of the time series data. And finally, each

label becomes a column. and why does the each column have that underscore underscore CF prefix? It's because co-maintainer of Thanos, uh one of the maintainers of Thanos, uh made this POC initially, inspired by Philip's work, and he added this prefix to all columns. But, there's an initiative to unify all of these column names in the Prometheus Parquet work group. Uh yeah. So, some data from our production,

uh quite uh different cases, I would say. So, I I converted a few tenants. So, first tenant is a very small one, no churn at all. You can see that the reduction is 85%. The second case is uh stores recording rules results for based on node exporter. So, node exporter, I would say rules have quite low churn, but for example, some things change like network interface names,

uh if they are Kubernetes workers, and so on. So, in this case, I got a 42% reduction. the last the last case is just server with recording rules, alerting rules, and I got a 60% reduction. So, in vintage case, let's say we would always hit the worst case scenario, so we would go from 700 terabytes of data to, I think, 400 terabytes of data. So, that's 300

terabytes less to store the same thing without any loss in precision. The migration path. So, how can you take advantage of this? Currently, the Parquet project is in an another repo in the Thanos organization. Uh in the You can find here the Docker images and also link to the documentation. So, how it works like is that of course the converter converts, and the Surf gateway is the

equivalent of Thanos store. It reads the Parquet files and responds to queries. Um there's currently a 2-days grace period before converting the data. That is so that the compactor would be able to vertically compact if needed. And so so that there would be consistency consistency in the input and through the parameter on the bottom left corner, you can uh gradually migrate one tenant by another. So, for

example, in our case, we have a matcher in this parameter, tenant ID equals to those three. And it's the converter will only take those three streams and convert them. Store. So, how does Thanos store on the other end know when to unload those that have been converted? We introduced a new parameter in Thanos store and through it, Thanos store is able to read metadata files from the

Parquet bucket and inside of those metadata files, Thanos store is able to find the source IDs. And it just drops those source IDs. in this way, you can gradually migrate to the Parquet format. How does the big picture look like? Store is connected to both buckets and serve is connected just to the Parquet bucket. And as time goes on, the converter does its work. And the data

served by Thanos store the original one shrinks and the one by the serve gateway increases. for those who don't know, each Thanos component exposes the minimum and maximum timestamp about the data that it knows. So, if the Parquet gateway the Parquet serve gateway starts serving more data, it knows Thanos query not to send queries to the Thanos store. If it doesn't match the minimum maximum timestamps. So,

in this way, you can gradually yeah, migrate bit bit by bit. How does the data look like on the object storage? We currently chose this structure, but of course it's subject to change. We are still it's very new project. So, on the top level, we have the external labels hash to separate individual streams of uh of blocks. After that we have the date encoded and and the

and the in the path. This is allows us to have a It's very convenient because you can just take a look at the year, month, date and you know what data is inside of there. After that we have a labels file. So this one contains just the labels as it says. Uh chunks. As I described earlier, each uh there are three columns for 8 hours worth of

chunks. Metadata file uh contains various metadata about the day. And the stream file contains the actual external labels that should be attached to the results. What's interesting thing is that uh there can be multiple shards and we use this sharding uh thingy because we want files or objects on the object storage not to be too huge and not to be too small. So there are parameters that

you can control the maximum shard size. And through that uh you can control how big those files are. Because for each parquet file we need to also parse the footer. So the footer is content contains metadata about the parquet file itself, where to find various columns and rows and so on. Uh so for example, if the shard files would be very small, we would need to do

lots of parsing and vice versa. Yep. Uh just wanted to add this slide that to emphasize this because it came up in a few conversations. And yes, the compactor is still needed at this point in time. There are various conversation conversations in the community to in the future have some component that produces parquet files natively. So, after that we won't wouldn't need such a thing, but right

now it's still needed. Yeah, some benchmarks. So, this is quite again, I made this in our production environment. And this is the simplest thing simplest query you can think of. I think 99.9% don't do this kind of queries unless you are exploring what's available. Because it's just the equivalent of select asterisk from some table and that's all. So, you don't do that in reality, but even then

Parquet is faster because you need to fewer less data. Yeah, and I made a small program and I calculated I would say that our Grafana deployment quite reflects reality. We have I think half a thousand engineers that use Grafana. I don't know, like a few hundred. People love creating various dashboards and I just did the calculation. And in our case the median and average PromQL query has

three matches. So, what what happens if you add those three matches? Remember that uh with each match you need to fetch possibly more data from random places in So, this is where Parquet shines. In this case it's becomes five times faster. So, it's quite a huge difference. I of course changed like the few matches to not uh leak internal label values, but yeah, the uh Uh, made

this benchmarks in production using Grafana K6 uh, running on a physical server and we host Thanos in Kubernetes. And I did many iterations, like 50 I think, and it's the result is consistently more or less the same. And of course I had to defend uh, disable lots of caches. For example, Thanos store by default enables index cache, so I also had to disable it to make it

uh, as equal as possible. Another thing is that in our Grafana deployment I found out that uh, the majority of queries can take advantage of projections. So as I've said, people in reality don't explore everything what's available. They want to slice and dice the data by certain dimensions or labels. Uh, so I hacked together a version again, with no caches just to understand how it helps. So

for example, in this case I ran this query many times and the difference is 25% compared to no Of course, you know, I could come up with some benchmarks that show like, you know, 10, 20 times difference, but I would I like, honestly, I think it wouldn't reflect reality. I tried to uh, here present uh results that I think reflect the reality of what your users will

feel in the majority of cases. And yeah, conclusion uh, the Parquet format solves lots of issues for us in the Thanos project. But at the same time lots of things are still missing. You uh So it's also a call to action. If you ever wanted to contribute to Thanos, I think it's a good uh uh good time to start because uh lots of I would that that

we could say low-hanging fruit are available. So, if you are interested, uh you can start using it, I think, right now and then contribute issues or code or whatever else. And we have a dedicated channel on the CNCF Slack, so you can drop jump on there and discuss with everyone. So, thank you. >> [applause] >> Have time for questions. Yeah. I have two questions. First, uh what

is the roadmap from POC to uh becoming uh the uh replacing TSDB uh in the normal uh uh Thanos repo? So, so in in the major version. And uh secondly, do you think that the parquet file is going to change in the meantime? Uh meaning that if I now choose to uh run this uh uh project in uh pro pro pro production, I have to convert again.

Okay, so uh about the long-term feasibility, I know there are like a few other formats that are you can call them hot right now. I don't remember the name. But I want to say that those improvements that other file formats have, they are coming to the Parquet format as well. And because we use the Parquet Go library under the hood, it will be enough to just implement

them because lots of projects use it, so they will come sooner or later and we will be able to transparently, for example, take advantage of new compression schemas. Uh for example, there's this uh thing called the fast static string tables, I think, FFST, that are available in other formats, but not in And with that compression scheme, we will be able to compress the labels even better. I

think it's like two, three times better than ZSTD. But yeah, like all of these things will come through the library and I think we'll be able to change that transparently with no with no changes needed from the user's point of view, I think. So, I think it's I would personally invest into Parquet. >> [laughter] >> And regarding the first question, for the longest time we wanted to

make certain improvements to the how compactor works, for example, in in my opinion. Like to make compactor separate, for example, the planner from the workers. Just like one example. Uh and there are quite a few other ones, so I think once we migrate to the common library that is in the Prometheus community, make these kind of improvements. Uh and after that, I don't know, half a year,

year, we will migrate the code to the main repo. And regarding components that would produce Parquet files natively, it's still a I would say a long time ahead until that's reality. We we we as Vinted and other companies I think from Reddit want to do the same thing, so maybe it will be become a reality faster, but I don't Because yeah, to have the converter right now

it's yes, extra complexity because you need to take care of the and it would be much nicer from the architectural point of view to only not have these converters, compacters and just produce parquet files from the get-go and to operate only on It's really the thing that we want to implement. But I don't know when that will happen. So we have there's a ton of booth. We

will have other maintainers there. So if you want, come around, chat with us. No more questions. I guess that's it. Thank you for coming.