What Happens in Kubernetes SIG Scalability: Intro + DeepDive - Wojciech Tyczyński, Google

This talk focuses on the Scalability Interest Group (SIG) for Kubernetes led by Voytech Tachinski. The speaker discusses the group's core themes: defining scalability goals, monitoring performance, preventing regression, making improvements, and working with the community to ensure best practices. The speaker explains that Kubernetes scalability is a multi-dimensional problem, encompassing aspects such as the number of nodes, pods, and features. Key concepts discussed include the scalability envelope, SLOs (Service Level Objectives), and scalability testing using the Cluster Loader framework. The talk highlights various improvements made to Kubernetes, such as the watch list feature, serialized list requests, and caching techniques to enhance the control plane's performance and memory predictability.

Okay. Hello everyone. Uh my name is Voytech Tachinski. I'm one of the um TLS of six scalability and in this talk I'm going to to talk a little bit what is happening um in the sik and what we are primarily focused on these days. Um so let's start with like a short introduction um of what the SIG actually is is is doing in general. Um so there

are like five main themes that uh we are we are involved in or we are working on like first one is like defining what scalability of Kubernetes really is um and like what are the goals what should be the goals from the scalability perspective for for for our community. Um so how far should we go what should be the the the current goals and so on. Um

the second one is um oops sorry um monitoring and measuring the performance of the system and the scalability to ensure that um things that we work on have the impact that we expect them to be. Um and what is tightly uh coupled with that is um um preserving the the and monitoring if we are not actually regressing the system in the meantime. either we or like someone

else by by um introducing new features that accidentally or not accidentally accidentally actually um affect the scalab negatively affect the scalability of the system. Um the fourth thing is probably the most interesting one which making the actual improvements and that is something that I'm going to talk more um in the second part of the presentation. what exactly are the top like improvements that we are or we've

been working on and are either delivered or like um in progress. Um and the fifth thing is like working with the community and ensuring that uh they understand the best practices they understand how to ensure that features that they work um will be scalable and so on and so on. Um so okay so let's talk a little bit about like what scalability really means for Kubernetes. Um

in general like the the the primary thing that we want or the primary principle that we that we want um that we follow through that we follow with is u is to ensure that we are not optimizing things for the sake of optimization. like it should always be driven in like real user requirements because every um every optimization brings or almost every optimization brings additional complexity to

the system. So we don't want to like introduce additional complexity if no one really cares about that. Um so when we when you ask users about like um if they want really scalability the answer is like of course we we do but if we ask them like what what exactly they want what are the exact like dimensions that they care about it in many cases the answer

is like we don't really know and it's even more it's like it's often like they they just they not just don't know they just don't want to know because like it's in the end it's like the infrastructure for them. So um so they want the infrastructure to just work and not think about like how it what is happening underneath and like how it's um how it's working.

They want to focus on their application and their business. So um historically what we were doing is like we were many years ago we were thinking about scalability as like size of the cluster in terms of number of the nodes when it's it's it's not really true. Um like in fact that Kubernetes scalability is really like multi-dimensional problem with like many dimensions really dozens of dimensions. Um

nodes are are just one of the dimensions but like there are many others like churn and the cluster number of pots number of like gateways number of secrets and so on and so on. So like uh we you should like when thinking about scalability we should actually think about like all of those um together to to to realize like how what it means for the performance of

the cluster. Um so what we did is we defined this concept of like scalability envelope which is like a zone within which like your cluster is supposed to be happy. Um what does it mean the cluster is actually happy? Um it's it's basically that all this all the SLOs's all scalability SLOs's are are really satisfied. Um so I think you probably all know what SLOs's are but

just just in case like um there are like two primary concepts here like SLI which you like service level indicator which you can primarily think about as think about as as a metric some metric um an SLO that is like conceptually um a metric with a threshold. So um we have like two primary SLI and SLOs's that we really focus on and scalability like being API call

latency and put startup time. Um it's not that this is like good coverage we just are missing people to to to extend that to to multiple other things. there are there are more things that we have defined as SLIS and we actually measure them in in in our tests but they are not uh we didn't spend enough time to like actually smoon though enough to the to

the level that we can call them or mark them as mature enough like SLOs's and start like blocking releases on on on those. So um one example here is that um or one thing that needs to happen for those is to to think through about like the like how how precise they are like historically we were defining the SLI and SLOs's like very very coarse gra in

a very coarse grain. So um you can see the like the the first definition from like the beginning of the project from 2015 like how it was defined and the problem with that definition is that like um the exact same system or the exact same system depending on how exactly you interpret that um you could say that it either satisfies the this SLO or it doesn't satisfy

and it's like the the worst thing that we can say because we can we as maintainers can say like it works But for the user it may actually not work because that's that's not how they interpret it. So um we need to ensure that like all those definitions are very precise and uh we did that for the first two but we still need to think through it

for like all the other things and like and increase the coverage of the system in general and it's something that where if you are interested in that work like we would definitely appreciate help here. Um okay so um going back to like scalability limits um the precise definition of scalability envelope is like pretty much impossible um with like that many dimen that many dimensions that we have

in Kubernetes we are often like using approximating it but but by assuming that those dimensions are somewhat independent it's not really true so while it often works works like if you are relatively close to the boundaries of that like u every single use case or scenario you should really think about like validating that before um trying to use that in production. Um okay so moving on to

scalability testing a little bit. Um and I will try to go quickly through it like there are um scalability testing is like in general super important part of the work that we are doing although it's not like super fancy it's like very critical and the primary tool that that we are um we are using here is like cluster loader um this is like our framework um you

you can think about it a little bit as a as bring your own yam it's like a semi-declarative framework where you can define what is the desired state of your system and also to some extent how to get to that desired system and the system will actually or the framework will actually drive the cl the the cluster towards that state um to and in the meantime it

will be validating like um how it behaves with respect to the SLOs's we have like valid or measuring a bunch of other stuff dumping um dumping other like observability and debugging information if you if you need to dig deeper into that. Um so we have also like deeper integration with graphana and promes we are gathering a lot of the data for um for like debugging purposes profiling

and so on. Um we have this um this tool called perf dash. It's pretty simple tool, but it's effectively allowing us to track the history across different runs for different metrics to to observe like how it like how it what happens over time and whether like the improvements what we are doing actually bring the the impact that we expect to do, whether we are not like regressing

or slowly regressing and or to to show when we regressed and so on and so on. yeah I think that that I already mentioned like that the the the SLI and SLOs's that we are um that I already mentioned are kind of natively supported in the framework. So they are measured like even if you start creating even if you if you create your own test case or

the test as a as a whole like you can effectively get those for for free and thanks to this framework. Um so regarding the testing itself like there are um there are three things that I wanted to mention here a little bit. Um first is like we have a set of like release blocking tests to ensure that um we are effect we effectively are not regressing um

all across releases. Um it's super important that we are running them on the real clusters like we historically were were doing a lot of that validation using the simulated clusters like we were using cube mark or quark or others and those are great those are great for like faster iteration and coarse grain like debugging and understanding and so on but we there are many cases where they

don't cover all the issu all the potential issues and we had multiple cases in the past where those tests were passing or or working fine, but the actual clusters what when you when you tried to create the real clusters, they didn't really work. So, um all the release blocking tests are are running as real tests on the on the real infrastructure underneath. Um and it's important to

mention that they are running not only on a single provider. Um we are the primary tests are effectively alternating across AWS and GCP. Um we have other tests also running on Azure. So uh like pretty much most of like the the the biggest cloud providers are involved in those testing. Um and this is important for multiple reasons like not only for cost and so on and for

involvement of different companies but um also this is an example from like February this year where like this is those are the gra or the um the diagrams from for like 5,000 note cluster tests um and you can see that like the AWS S test were completely green, the GCP tests were red effectively and that was like a valid situation. It's not that the tests were incorrect

or something. It was the the regression went into the or the there was actual regression here that went into cube proxy. Um that is something that AWS stack is not using. They are replacing it with a different component. And so um having the tests being run on on two different environments allows us to um first see first validate that like it's not validate more environments effectively if

we would only have like AWS test for example we would we could release the uh we could have a release that that was effectively faulty and it worked on AWS but it didn't it didn't work on on other like environments and on the on the second hand like it allows us to faster like diagnose what is happening because if one is consistently failing and the other is

consistently green we pretty much know that it it it is related to the setup itself. So like it it we can exclude um components or like parts of the system that we know are using the same the exact same setup across both um environments. Um and one more thing to mention here is that um we are encouraging very much feature owners to to either like adjust existing

scalability tests or even create their own tests. Um it's super important because like we as a SIG cannot really hand cannot really catch up with like everything that is happening in the community. So like it should be a responsibility for feature um to ensure that the feature they are working on um actually scales to to the levels that they they would like it to scale. So in

part in in principle like one of the one of the things um we need is we need to ensure that even if the feature is not needed it's not used like it doesn't break it it doesn't actually affect the scalability of of the rest of the system. So um okay um so I think that's mostly the introduction part and now probably the the something that you are

or many of you might be most interested in it was just like what are the actual improvements that we are working on or we've been working on and I I I will focus on the latter. So what are the things that we already um have in the system so I think there are effectively two main categories um of things. one is explicitly pushing the the the limit

on in a one in terms of like one dimension. So like allowing to to run more nodes in the cluster or allowing to now run more pods in the cluster. And the second like category of things is like increasing reli reliability at scale. So it's not really pushing um the individual dimension of the system but it's it usually translates to um increasing the scalability envelope with like

in the other dimensions when when a given dimension is um is um sorry is uh is set. So like as an example like imagine that we we want to support say 5,000 nodes and we don't really ch change that but within 5,000 nodes we want to support like higher pot churn or whatever. So um we we are kind of thinking about that as increasing the scalability envelope

um without really pushing the any individual limit above what we currently scale but trying to make both of them be exercised to significant like both of them be exercised at the same time. Um so few things few improvements that I wanted to mention here. The first one is like um watch list feature that we started working in I guess 127 or 128 like it's definitely more than

three years at this point and what it allows us is um is effectively to replace the the regular list requests which are pretty expensive like the the amount of data that we need to send in the single list request can even can easily exceed like a gigabyte of data in a single request. So um that is not something that like or that is something that is super

expensive from the system perspective. So um we came up with this idea where you act we actually don't want a single request to be that large but you can re we we can use the watch protocol to send the exact same data data just like in in chunks kind of and uh this is what this feature is really about. So like thanks to the like additional parameter

to the list request uh to the watch request sorry um um you can effectively request the initial content. So like the the the usual pattern that that that we use in majority of or pretty much all controllers is that like you first list the objects of a given type and then you start watching from that point. And now you can with with this with this feature you

can effectively boil that down to a single core where like the watch within the watch you also request the the the actual state and and and then the watch starts sending addition the events from that point. So um what that gives us is like effectively it it significantly helps like with memory usage of the of the system. um we the the predict like the the memory gets

much more predictable the memory I mean here the memory of API server really um because the the footprint of individual the individual calls is now like static like static it's like up to roughly 2 megabytes per per such request whereas for list requests it it could be like arbitrarily long because we we were um or we still are for list requests um actually serializing the whole memor

the whole response in memory and then there is a moment in time that where we actually keep it um as a whole. So um and memory is or the spikiness of memory is is one of the primary problems that we are actually facing in terms of like scalability of of control plane. Um, so this is this is now enabled for my pretty much all like components. We

all we also like switched in 135. We switched the or yes 135 we switched the default in client go libraries to to use that instead of like the regular wrist requests. But um so the API server itself is much less um exposed to to regular risk but still we don't control everything everyone and in order to use that you need to explicitly change your usage patterns or

like optin for that for um in in other libraries and so on. Um so we also did another thing which is um changing the or changing the serialization patterns for for list requests for some most some of mostly used like um protocols like JSON and protobuff primarily and just those two um where we no longer we no longer serialize the whole whole response in a in memory

before like even starting to send this to to the client, but we actually serialize individual items of the list kind of independently and already send send them to C send them to clients. So, um the the impact here is pretty much the same. So, you may you may um wonder why we actually need both. And there are we don't have like enough time to go deeply enough

there. But like first the the former the watch list is a little bit more um it allows us to a little bit more flexibility. Um and it allows us to do more tricks like try to reuse the same objects and if when we are relisting and so on. Um and on the and and additionally it it allows us to also um not do like once that feature

the previous one the watch list lands it it will it doesn't really require any additional changes per per se per per formal per requested formal like JSON protobuff and yamlin and so on. This feature requires additional work for for every single one of those and like any adjustments or any extensions to those like will need much more maintenance here. So we are focused on only on like

the most frequently used here because we don't control everyone and like people can still be requesting those those like legacy or like regular list requests. Um okay so moving on the other thing that is like worth mentioning um is snapshot tableable API server. Um so couple releases ago I think it's like 132 or something um we enabled the feature or we implemented the feature that we called

like consistent lists from cache. So um historically every single um every single list request was um reaching CD and that the response was served first or returned first from CD to the API server and then API server was additionally processing it and then like sending the response back to the to the client that is like super expensive. Um so at some point we started we allowed for

opting in for ser for returning the response from from the cache and API server. We have this like this like logical comp component called watch cache in the API server that contains or like um that yes that has the the whole state of of the cluster in memory. So we allowed that but but that that is by definition eventually consistent only. So it it doesn't work for

everyone. Um so at like in 132 we for serving like consistent lists there too. So the mechanism was effectively we send the request then the watch cache is sending the request to to HCD to to get the current resource version and as soon as the like watch cache is up to date to that that version we are we know that we can now return the response to

the to with this feature. We actually um expanded the the number of use cases or to that that we can serve from from watchkash to pretty much every single list request including the the pagenated ones because we um we changed the format in which the um in which we store the or which we have the which we store those object in memory into B3 and we can

now like easily clone using the copy and uh mechanism the the snapshots so that we can at the same time serve the the the request from from few seconds or even like tens of seconds ago but at the same time keep keep updating the the the cache itself and and potentially serve other requests with um that are like more requesting more upto-date um data. Um so again

the the gains are relatively um were focused on the same goals here like we wanted more predictable memory usage and we wanted additionally to that we wanted to reduce the load on the Ccb itself because that that in many cases is the bottleneck of the system. So as much as long as much as we can reuse the load on that, we actually should do that. And um

at this point um it pretty much every single list request no matter if it's like pagionated, non-pagenated, consistent, non-consistent, whatever, it's already self served from um from API server. And the only list requests that like SCD handles are those that are feeding the data to the API server cache. okay the the the another thing that I wanted to mention here is um the is the stale controller

handling. So we we observe or we had m multiple different incidents incidents where um for whatever reason like the control plane was lagging and and in terms of like not being not having up-to-date speed uh up to-date state and by control plane here I mean more than than API server I mean also controllers and so on and in such case when controllers were um operating on on

the stale state they were actually making either incorrect decisions or they were repeating their steps that that they already did because they weren't yet reflected in that state and that was causing like additional load on the control plane itself. So we introduced that like relatively simple at least conceptually mechanism where um can register that like during reconciliation of a given object they the next time they will

decide to reconcile it. need to ensure that whatever um whatever right requests that that were result of that that reconciliation are are already visible in in their state so that they operate on for some definition of of fresh enough on they operate on on fresh enough state. So they they don't reconcile again the same object using like the the whole state and um and again here like

the the primary reason for why we were doing this is like making memory um of of control plane or on controllers in particular much more predictable. okay and there is like the the second um which I have I guess two examples here um one is like the opportunistic batching that is more like the scheduling improvement where we imp we change the scheduleuler um or we introduce this

concept that is likeulular internal concept uh but the the concept of like pot signature where scheduler can say that like those two pots or those 10 pots or whatever um are identical from scheduling perspective. So like they have the same requirements, they have the same shape, they have the same um constraints in terms of like what nodes they can be running and so on. Um and thanks

to and if if two pots have this the same scheduling the same signature we can actually reuse the a bunch of like computation thatuler um for the first pot we can reuse it for for the subsequent pots. Um and thanks to that like the um scheduling for subsequent subsequent homogeneous pots um is is much faster and that helps a lot for um latency scheduling latency in general.

So the focus here was like ML or AI workloads were like primarily one pot per node. There are multiple things that we or multiple corners that we cut here. So not every single um not every single signature is really supported for many signatures. We are still falling back to regular like full potby um computations of of physibility of those bots. um we we were focused on like

pot per node use case which is like the most common one in for AI workloads. We know how to how to like generalize it. It's just it didn't yet happen. But um here the the goal was not the memory as for pretty much all cases where we were discussing before. But here the the um the goal was to improve the churn or push the boundary on how

how large churn and we can achieve in the cluster. Um and the last one that I wanted to mention is the the resource size in general. So histo like we had we observed multiple different cases where when trying to simulate a workload using in our test like every everything worked fine and then when real user was trying to use that using pretty much exactly the same pattern

and same like scenarios and so on this didn't work and it's it appeared that like the only real difference between the synthetic tests and and And the actual use case uh the the actual um workload was that like they were using larger objects, not more of them, not faster, not whatever, but they were just larger. Um and so like we um we started looking into what is

happening there. There are multiple things that are in flight now. some of them like some some initial like uh things will be landing in 136. So it will be a little bit better. Um but it's it's something that we pretty much started looking at in the last two months or so. So there is a lot of work to actually do to reduce the impact of large objects

on the system. Um okay and that's pretty much what I had. So if you are interested in any of those topics, I'm around here both today and uh in the next two days like feel free to catch me. Um uh if you are interested in this topic like please join our meetings. We need we need a lot of help like there's a lot of things we could

do if we have like capacity. So like if if you are interested interesting like um please join us and please help us. And I think that's that's all I had. So thank you very much. Um I'm not sure if we have any time for questions or no we don't have. So like I I'm I will be around here. So if you have a questions I'm happy to

answer them like in the corridor or whatever. Thank you.