Rasmus Selsmark: Processing Millions of Events per Second with Apache Kafka on Kubernetes

This talk focuses on using Apache Kafka at scale on Kubernetes, specifically through the Streamsy operator from Red Hat. The speaker, representing Unity Technologies, discusses the architecture of their event streaming pipeline which processes large volumes of data generated from video game advertisements. He explains how data flows from devices to ads endpoints, where it is processed using Kafka for normalization and storage. The session highlights key features of Kafka, including the recent KIP-932 improvement that allows multiple consumers per partition, and the use of tiered storage to optimize performance and reduce costs. The speaker shares insights on operational challenges, monitoring tools, and software architecture best practices involved in managing Kafka clusters efficiently.

[applause] Thanks, Eric. Yes. So, uh, as Eric introduced, thanks a lot. Uh, we're going to talk about Kafka. And, uh, so this is the sort of official title, the alternative title, which the question we get now and then is, can you run Kafka at scale on Kubernetes? And basically, uh, yes. And this is how we're doing it using the Streamsy operator. And when going through these slides,

I say a lot of this is about Strimzy um the open source project from Red Hat. So um Unity technologies, Eric mentioned uh I don't know how many of you are familiar with it. The the primary product of Unity is the Unity game engine. Uh the editor looks like this. You build games. Um it's sort of a it's for software industries who focuses on games and the

sort of a lot of the challenges are the same. Uh, and you might think, uh, games, what does that have to do with Kafka? Um, but we're back to the fact that games are everywhere. Uh, so this is a computer from the 60s at the computer history museum at the in in Mountain View in San Francisco. One of the apps on running on that computer is a

sort of asteroid game. Um, and I say myself, you said that introduced me when I started programming. I started programming games um and then I went into uh finance systems other things and then I bit by luck got back to uh into Unity. So sort of feel bit back to the roots. Um and then we have um the first graphics card that came out that was invented

in the or came out in ' 90s because people wanted to play faster games. Fast forward a bit until today where we have the one of the latest Nvidia cards which is used for machine learning training machine learning models. So there's a lot of places where innovation driven by people wanting to play games actually is is used for other purposes. So um so I think games are

fun but they also actually uh they're also serious. Um and we talk about numbers. Um in our case what makes back to the topic of uh Kafka and uh event streaming. So uh every month uh 3 billion games built on the Unity platform are downloaded um from app stores. Um and important to say Unity is not building those games. Unity is the the company providing the technology

platform for game studios to build those games. And we uh serve ads. Um and that is the data we're going to talk about today. um uh 68 billion uh ads per month um are being uh being shown to people playing games. Um and this is a whether we like ads or not. This is a way for if you're a game studio want to build a business uh

one of the models for that is uh to show ads in your game. yes, let's quickly go through what actually our pipeline looks. Um, we start with a uh a device playing a that sends data to one of our to our ads endpoint. Um, we use a protoraph definitions. That's sort of our schema registry. Uh, we we don't use Kafka schema registry. We have our sort of

own implementation. But then goes into Kafka. And um then the next next part is using a passive link for doing the normalization/enrichments. So enrichments in this case could be taking a IP address turning that into a country code. Um making sure that all times are in UTC so we can compare them things like that. Um that's what happens there. And um then from there it goes into

either Pyman um or Kafka. I say the um this is the one this is bit experimental. We have uh some internal uh customer teams at Unity requiring they say they require uh seconds uh latency from actually and we would only be able to achieve that using Kafka. Whereas when you go up here, there's some um checkpointing in these databases, which means that typically uh you there'll be

five minutes or something before you actually have the data available to be able to query. This is not used yet. We have it uh but is actually not been used because it turns out that usually minutes latency is is good enough for most teams um for actually all teams. Um and then our concept is that we talk about the data over here as being L0. Uh in

other terminology it can be raw data. Uh and when it has been enriched we talk about it as being L1. In some other cases it could be bronze, silver. Um and after this uh then the data basically goes into um into reporting into training our machine learning models which then goes back to being used for showing ads. But also this is where um our our customers the

the game studios the publishers basically um get getting their money based on this. So if this pipeline is down we both for unity is a problem but also for our customers who won't get the revenue from uh from ads in that case. Um and yes so the uh areas we're going to talk about is uh this and I said Kafka is also being used here but uh

we have the Kafka cluster standing mostly idle um to be honest for for this one. Um yes if you want to play along uh I have sort of my own uh hobby project uh where I also use for trying out ideas but this is a um a GitHub repo uh with a small uh Kafka setup running on scrimsy with yeah three Kafka brokers one topic a producer

that writes um data um and then the other things red panda control the UI cruise control, Prometheus, Red Panda, Kinion, things we'll get back into this in this presentation. Um I have a I have it running in the background. So if there are any questions sort of might be that we can use that. Yes. Um then important to say that there is a sort of a world

uh in Kafka. There's a world Kafka world before and after Kip 932 which was released released here with the Kafka 4.2 to um the spring. Before KIP 932, you could only have you would have one consumer per partition, which means in terms of sca scaling out. So if if you need to scale out the con the number of consumers, you'd also need to scale out the number

of uh of partitions at the same time. with Kafka with this keep 932 you can have suddenly have multiple consumers per partition but um the sort of the is that you won't have this strict ordering so if if strict ordering is a requirement for you um you you still need to use uh single um consumer per partition I say we have not implemented this yet uh says

is fairly new and we also need to actually figure out what it exactly means in in our case. Um so in in this presentation assume that there will always be one consumer per partition and I I'll get back to that. Um bit about the Kafka architecture. So in our case we run in GCP on uh three different uh three availability zones US central A, B and C

which could be repeated A, B and C. And in this example uh having six brokers spread across those zones and uh two topics and a topic is such is is is the stream of data um and that is distributed across all the brokers back to the talk about partitions. So a topic is split into partitions. So in this case simple example we have um two partitions for

this uh for this topic and uh with three replicas. So that means that we have a the one marked with bold is the is the lead for that partition. The two others are followers. And the same for this one. Um has one lead position. And another topic, the same concept can be spread across different brokers. And then when when you produce data into Kafka, we will uh

you have a producer that produces to the to the leader to the leader replica of that of that partition. And then Kafka afterwards will then synchronize the the data into the other partition or the other replicas for that partition which is where we get the reliability. So so even if if one of these uh brokers would go down um we still have two other replicas available. Um

and uh consumers can then consume from uh any of these replicas. It doesn't have to be the the leader. So that is uh in the highlight how Kafka works internally with terms of topics and partitions and replicas. bit about our history. I'll get back to the speed sign actually right now. So we started out uh 2018 running Kafka on VMs in uh in GCP and um this

cluster is actually still running. Uh you probably all know that when you uh start spin up a system, it's uh it's very it's easier to introduce a new system than taking it down again. And uh we have have a lot of internal consumers on this cluster. Um so we currently actually running 30 72 brokers still on that VM based cluster and keeping it uh actually feel like

it's it's it's running quite well. um we still want to move. Um yeah, but that started out by running on the SP spinning discs which has the um capacity limit of 400 megabytes per second. We later uh one and a half years ago um migrated into running on Google hyperdisks which uh then allows uh has 2.4 4 GB um or gigabytes sorry per second capacity which means

that we could basically actually reduce the we could we could move to larger brokers um and fewer brokers because of this. So that's also a um sometimes people think that Kafka is elastic but uh at the back end there is still data being stored on disks and you need to take capacity planning into actually what what are the what's the capacity of those discs? Um yes then

in 22 we introduced uh our first uh Kafka generation sort of running on on Kubernetes using scrimsy um we are trying to uh actually decommission this um uh in this in the autumn last autumn it was 12 clusters now we're down to eight uh we are slowly trying to actually move users out of this um but basic Based on the uh learnings from um from the from

these clusters uh we have now last year um started this the what we call the our third generation of running Kafka also kubernetes with streamsy moved into the from zookeeper to kft for instance still running hyperdisks the big difference here is the tiered storage I'll also get back to that which means that we can offload data into uh GCS buckets in this case into Google cloud uh

reducing the need for for local storage and also thereby reducing the the cost. And what we see is when we migrate from our second generation into this third generation, we see a 65% cost uh reduction um by being able to have fewer brokers uh because the these can actually be smaller because of the tiered storage. Uh so one example was that we move we migrated a 48

broker cluster from second generation and only needed 32 um in our in this and this is the one that we're Yes. Uh bit about uh how much traffic we see. So uh across all three generations um we get uh 20 million events per second. Um and that is uh 22 clusters in total which means uh almost 17 GB per second. That is the scale that is possible

to handle uh with Kafka. And you might think all this just comes from ads. Uh yes it comes from people playing games. There's a lot of people playing games on mobile. Um and also you can see uh so what what we see here these variations is basically at what time. So this is a one week of data. Um and this means what what time are most people

on this planet awake and playing games which is about as recall around 400 p.m. uh UTC that is when uh China is still awake, US is waking up. So that is the time of the day when most people are playing games. Uh and you also see in the weekends uh there's a bit more traffic. Um we uh we don't scale up or down. We just keep a

fixed size of our clusters. Again, Kafka is not that that elastic. Um, but we have to also do sort of our capacity planning against what is the what is the peak traffic that we see um and the uh throughput on on one of Uh yes, I mentioned tiered storage earlier. Um and the point about tiered storage uh is that you can uh we can we can offload

data from uh from disks on the brokers into GCS. uh in our case we're using uh GCS yes as mentioned could also be AWS S3 it could be any other storage it could also be if you have a data center it could also be just some cheaper storage that you have in your data center um of course comes with the uh you have to be aware that

typically getting data from that um will will have a higher latency um also get back to that but uh in short how you enable when you have enabled the other parts but how you enable on a on a topic is to set the remote storage enable to true and specify a local retention. Um and then that defines how much of this data is stored locally versus um

remotely. uh that one and just to try to illustrate we have Kafka topic in this case uh two hours are stored locally the rest of it is stored in uh seven days is that what we store in GCS uh 7 days might be a bit too much but it's for uh we sometimes see that we need to run back fills um and it's given that it's relatively

cheap to store data in GCS uh we just said okay we have seven days uh so if if there would be a need for teams that come to us say we actually need to back fill this data for some reason. Uh they can still do it um up to seven days after and again um we assume that we have processed all the all the all the data

within two hours. uh normally would actually we normally we talk within a few minutes the two hours is to allow us if there's some incident that at least gives us some response time uh not too much stress um and still we can um I said we we can still backfill when it's in GCS we just see it's it takes longer because of course it needs to fetch

data from GCS bug in that case yes bit about scale so that means that we are we're storing um roughly 84 terabytes of data um locally and 5 pabytes in um in on GCS remote storage. Um yeah mentioned uh between two and four hours of retention locally, three replicas and running on hyperdisks and seven days in remote and then it's just sort of that's numbers uh just

for sort of comparison was looking what can you compare this with. So internet when you look around internet estimates that on Netflix there are 35,000 hours of content. If we uh that means four years you can watch four years of content on Netflix without breaks. uh if we assume that each uh hour of movie on the Netflix is 20 gigabytes that means that Netflix in total all

data on Netflix is uh less than one uh pabyte and from serving ads for a week we get uh five pabyte of data. So it's it still surprises me now and then actually how much data um we we have for this. yes um just to show you uh it's always of course nice with the UI. Uh we use Red Panda console. Red Panda is actually for you

those you know those of you who know it is sort of a competitor to Kafka. They've built their own Kafka compatible uh streaming platform. Um but they took over this uh this open source project was normally called the Apes cow. Um oh not Apaches but the Kafka Kafka Owl. Um and they took that over and made it into Red Panda console for historical reasons. We used cow

back in the days. Uh so we could just continue it uh using Red Panda console. Um it's nice it gives you overview of topics. Um and you can so it's always nice to actually sometimes be able to visualize and go in and see okay u is the configuration applied correctly. You can find other open source controls out there. Um yes um bit back to architecture. So uh

we have uh a cluster consisting of Um we are running as mentioned 22 uh clusters and the reason for that as such uh we could run I believe in we could run one big cluster but what we have seen by experience and you also saw that our first uh original cluster is now 72 72 brokers. What we have seen is the noisy neighbor problem is that suddenly

some service some team they scale up and they forget to give us a heads up beforehand and not only get are they getting affected by that it also affects other uh teams using the same cluster. So um we have learned to uh of course there's overhead of spinning up a new cluster. Um but we are we're now at 22 clusters for this reason of uh avoiding noisy

neighbor problems where actually suddenly one team can cause problems for for other teams. Um and yes largest cluster cluster we're running in Kubernetes uh currently 32 brokers. Uh so again we're running more clusters a bit smaller than our original VM based cluster. Um I'll also get back to the uh replica settings. So we're as mentioned three replicas um and we're running the um what Kafka is called

minimum insync replicas um of of two. I also get back Yes I mentioned scrimy operator earlier that is a redhead project. Um if you want to run Kafka on Kubernetes, you can do it without scrimsy. I've seen that. Um but I would certainly recommend you to do use a streamy operator that does a lot of uh nice things for us. Um I just copied uh some u

bullets from the streamy website. Um yes has uh handles um doesn't handle but it it helps you uh define your authentication for a cluster. um define record awareness is what we're using for spreading across multiple zones. Um and uh as operators uh which allows you which basically simplifies the main maintenance of our cluster. Um and be behind this also uh with what streams uses is this streamy

pot set. So instead of setting up a back in the days would stream would actually use a stateful set but that has some disadvantages because there's still some Kafka logic in terms of if you if you want to roll a Kafka cluster you should roll the so there's a controller in a cluster and you want to roll that last because if you when when when you when

you take down the controller Kafka needs to find a new controller which can take a few seconds for to dedicate new controller roller. So if you are rolling a cluster, you want to roll the the controller last in order to avoid this from happening. Um so it comes with a lot of nice built-in features. So if you want to run or are running Kafka and Kubernetes, I

would recommend using StreamYC. Um was there a uh yes and but what you get uh or how you define a Kafka cluster in scrimsy is basically a Kubernetes manifest using um their CD in this case Kafka version uh defining the listeners um and the advents replication factor So it's a very uh what do you call it familiar feeling working with with Grimy is similar to what everything

else on in Kubernetes. Um on top of this I guess uh many of you have been through the journey of starting with plain uh YAML files uh which works for a while. uh plain ylifiers has the advantage that it's usually very easy to go back and pinpoint where's the uh if something is uh where where something comes from but also at some point it becomes unmaintainable especially

when you have in this case multiple clusters and you want to make a change across all those clusters so we uh using what called the KCL Kubernetes configuration language uh another open source project but we're using that as templating um for Um uh sorry for the yeah for helping us defining these clusters in this case uh yes KL is based on Python so we can define methods

so in this case there's a topic method which takes in some parameters the name of the topic how many partitions local retention things like that um and then used this is an example of using it so we're calling that uh method And sorry I had my There we go. Um yes and basically that turns into a cluster into a topic definition. Nothing magic there. Um but just

makes it also a bit easier for for teams themselves to define topics. um they just have to define these three lines which then uh behind the scenes turned into uh this topic definition which for instance is that we uh we set default for the local retention bytes for instance um and that also remote uh so for inance when we move to remote storage then it's a matter

of introducing this as a as a default and then we get that um across all topics um I hope you can see this an example of uh just storing the configuration as code. In this case, scaling up a a cluster from 24 to 32 brokers. a matter of making a pull request for that and uh hope you can see this uh applied using Argo which uh and

we can always go back to Argo and see what is actually what is the actual uh communities manifest that has been thank you um that has been applied for this Fun fact about those 32 brokers. It was done uh can actually see when it's done was done some sometimes last year. Uh if you realize 32 is not divisible by three and given that so normally we would

actually go for uh for broker count that is divisible by three because we have three uh zones. I don't know why we decided 32 in this case. Um but um but it works. Um yes, misone mod zone works by using the pot topology uh spread constraint in uh again uh supported by streamsy. So um on a in kubernetes we'll provide this label saying which zone or uh

which zone is this pot running in and using that as topology key for scrimsy. Then Kafka with Kafka has the aspect or the concept of rack awareness back to when you actually had racks in a data center. Uh you want to spread uh across those and that's the same we use the same concept that we use for spreading across So you see in this case uh brokers

running on uh on the three different Uh some of you might uh have attended the uh CI/CD for data pipelines yesterday um where he talked about actually he showed a um a diagram of a GitHub githubs based flow for uh for for data pipelines. One of that I'm not sure this qualifies as a contract uh but this is also part of of how do we actually in

practice manage all these uh events that we receive. So there need to be some structure. Um we have a for our latest uh our latest data pipelines uh we introduced this naming schema basic domain uh which is maps back to the team or the service what event it is and what layer so the layer is this L L0 for instance or L1 um and you see in

this case so we have a uh unitiz network uh let's say the operative event so the operative event is basically all the information that we get from devices saying uh this this game was started uh things like that and you see there's a versioning number most of these are on version one of them is on version three because we don't allow any breaking changes um another thing

on on this scale uh you'll learn that at some point it's not especially when you and that depends on the number of consumers you have you will likely always miss a consumer when you make a breaking change to a uh to schema. So we we've said that we don't allow breaking changes. If you make breaking changes, you need to create a new version of the topic. So

that could be u renaming a field uh removing a field and um in that data pipeline there is a GitHub actions check to verify that there was no um breaking changes when you when you change the protobuff definition for that Um yes, partitions and throughput. Again, back to actually doing some sort of capacity planning on when you create a new uh topic in know in terms of

how many partitions do we need for this and again back to this is a world where we have one consumer per partition. Um and given that uh so the basically the throughput of those consumers they define what is the um how many partitions do you need? If you have only so if you only had one partition it would be limited to how much how much is that

consumer accessing that topic able to process. Um and u if you are interested in this uh there's old blog post from confluent on um on what the considerations when it setting the number of partitions. Um so as I say more partitions again this is back to the world of one consumer per partition but more partitions would lead to usually lead to higher throughput because you can then

scale out the number of consumers. Um but there's also an overhead of course um a partition uh means files on a disk. So more partitions also means for more files on the disks um more file handles um and uh the main increase on availability is because if the controller goes down there's some work related to each partition to actually find a new um or if the leader

go down find a new leader. Um and the most important uh once you have set a uh partition count for a topic you cannot is reduce that. So for that reason we actually have one topic with uh 960 partitions because at one point we needed to back fill and it had to go fast. Um but there's this blog post um most of it is still relevant even

in a world with multiple consumers. Um this I said uh the how much each consumer is able to process uh also defines number of so we have this formula uh basically saying let me continue um basically saying uh we look at how much is the consumer able to process in uh in data wise or how many messages is a consumer able to process. In our case, our

downstream of our data pipelines can process roughly 25 megabytes per second uh or 10,000 messages per second which is basically based on our experience um which for just took one of our topics uh which gives so we have a topic receiving 2.3 GB per second uh on on 1 million messages per second dividing that by 25 megabytes or 10,000 messages gives us a max of 10 and

100 which we then rounded that's the cluster with 32 brokers. So 96 is a nice round number for that because that actually means that you have you you spread the number of topics evenly across um the brokers. So there's still those considerations even in a world with multiple consumers. I believe there will still be those considerations of um how many topics do I actually need in order

for for for Kafka um to handle this traffic um and consumers to be able to also uh handle. more side note uh on our old clusters uh also the second generation we had a lot of man manual work related to uh certificates authentication um handing out uh certificates uh for for the for teams with our latest uh we as you said we run in Google cloud uh

Google GKE has built-in support for this workload identity which is based on spy spy bas It means that the it's servicetoervice authentication which has reduced a lot of the it makes the onboarding experience way easier. When we ask teams to make sure that you have workload identity enabled we then put a trust uh your name space or the Kafka and then um they can authenticate using that

with the service account. Again, mostly a developer experience improvement and makes our lives easier for maintaining Kafka. uh observability of course again Kafka StreamY makes that uh simple by uh having this metrics config setting where define um what kind of exporters you want um and that's uh more or less it and then you can uh have Prometheus Kafka Prometheus and Grafana from there um bit about what

alerts we see some of our so we the primary alert search we have is related to um underreplicated partitions. So there as I said we have uh three replicas underreplicated means that out of the three replicas one of them is unavailable. Uh so we have two left. is not critical yet. uh everything runs as such but is a early sign for us saying that hey um it

could be a uh signal that either uh a broker is too heavy loaded it cannot keep up with the traffic or a broker is down or network issue things like that I said there's no uh data loss at that point at this point but it's a it's a warning for us that will typically we will look at because it could turn into this one which is the

under mean insync replicas. So that means out of our three replicas there's only one replica left at this point. Uh we don't want the last broker to go go down because that would mean a data loss. Um another thing that we've seen that in uh so when when a producer sends data to uh it can send using uh this a setting basically acknowledge setting. Um if you

use acknowledge all that means that you uh you you ask Kafka to acknowledge when all the insync replicas um have received this data. Uh at this point if we are below uh min insect replica that means that producers using egg all they will actually they will wait because they they they won't get the egg from the last um from they will only get act from one one

replica. Um and uh in some cases that is what you want if you have a if you really need to make sure that data are written. In our case when we're processing this m much data we can we can manage without we can manage with some lost events um but actually this we've seen that the producers using e all they will basically stop in this case even

though as such kafka is still running so for that reason uh we are requiring our producers to use uh leader um yes and then of course the user suspects uh monitor your disk uh IO low uh CPU load on your brokers. Um, also consumer lag uh is a metric that we're looking at. It could uh indicate that uh Kafka for some reason has a problem. Couldn't indicate

a network issue. It could also be purely on the downstream consumer side. Um but at least it's it's interesting interesting for us to know that um we have a uh a topic where the consumers are are lagging behind and typically we measure um in this case one hour that's usually our threshold we can we can manage with one hour um I said usually we process data within

minutes um and often teams will also start notice um before This uh this is more like my uh one of my own uh panels in the graphana. It's it's interesting to see uh what is the uh the growth of of topics. Sometimes uh a team will forget to tell us that they are scaling up. Um and so this basically tell shows what what was what is the

difference on the topic size between compared to the day before. So gives us some idea about um if uh if some topic suddenly received a lot more data um and yes uh consumer lag um and yes another nice feature of Stressy is that you set this uh Kafka exporter. So Kafka exporter is a open source project. If you uh manage your own Kafka setup you need to

install that yourself. uh stream takes care of that by you just set the Kafka exporter field in the uh in the Kafka cluster and then it's there for you. Um but uh so uh that will give you this Kafka consumer lag um metric which is basically is a calculated metric that will tell how much are we behind on consuming in this case. Um so this is basically

normal uh in this case this we're talking uh a few seconds of lag um on a topic but this would be the a normal um graph for consumer lag. Um then we also have uh another red panda open source project K minion. K minion is basically uh measuring the latency of the Kafka thank you uh the Kafka topic the Kafka cluster itself by sending data receiving data

purely on the topic no data processing but just telling what is the actually the uh the average P 50 and P 95 percentile um of of the cluster that gives us when we some team reaches out to us and ask if Kafka has problems we can use K minion at least ourselves to see uh if C minion has a problem then we know is certainly Kafka having

a problem um rebalancing what will happen over time is that again we're back to disks data stored on disks when you add a new broker uh or just the hardware variations will cause that over time uh some brokers they they end up storing more more data um this is from a streamy uh YouTube talk you can And um so in this case we have one broker uh

having more data add one broker balance it out and it looks like this afterwards. Uh another thing when you add a new broker to a uh to a cluster only uh it would be empty as such. Uh so you also need to do something to move u partitions into uh into that new broker. number of goals, different goals that you can set on um on cruise on

cruise control and cruise control is a LinkedIn uh project also. An example of where what we have used the cruise control for. So this is a new cluster starting to receive traffic for some reason one of the brokers um get above our threshold for request handler which is basically the CPU processing on that broker. Uh so in this case we notice this we uh run a what

do you call it a targeted uh cruise control balance plan uh with these goals and see after this it actually starts switching. So what happens is that we'll move partitions um between brokers. So she actually uh this broker uh will traffic from that moves into this broker instead which then uh so as such rebalancing a cluster moving um and apparently is slido time quickly uh a lifetime

of a cafka cluster. This is actually our old cluster. You see this over time has evolved into two groups of brokers. In this one group of brokers storing less data, run a cruise control rebalance uh gets more even. Uh we get some uh new topics introduced. Um and we remove some brokers. Uh so cruise control is a way to keep your Kafka cluster healthy. Um yes, enabling

cruise control. This is from the um Stressy documentation. You uh specify on the uh again on the Kafka topic, you specify the cruise control, you specify broker capacity and with what goals you want to run as default and uh quickly yes everything in scrimy is about um annotating. So in how you start a cruise control plan is that you annotate your uh your your cluster with this

uh rebalance label and then it will start um rebalancing and at some point it will be ready. So it it's very uh what do you call it? Uh we do this from command line. Uh there is also a uh old uh LinkedIn cruise control UI project uh but that has been uh unmaintained so we just use from command line uh tiered storage have talked about that uh

again uh related to uh cruise control and rebalancing the less data you have on disk the the faster rebalance also is. So actually for us uh moving data to GCS means that also we can it's easier for us to rebalance the cluster because there's less data stored on the brokers themselves. Um and uh enabling shared storage uh again look up in the uh scrimy Um and that

was it. And again, if you want to play around with this, I have a