SIG Scheduling Update: Transition From Pod To Workload Scheduling - Kensei Nakada & Maciej Skoczeń

This talk covers updates and future plans for the scheduling components in Kubernetes, specifically focusing on the transition to a refined scheduling cycle. The speakers highlight improvements made in the latest release including gang scheduling and workload APIs that facilitate atomic decision-making for scheduling groups. They discuss the features of the scheduler such as pod affinity, resource allocation, and the means to avoid deadlock situations with gang scheduling. Future developments are outlined, including enhancements to workload preemption algorithms and topology-aware scheduling, which aims to optimize resource utilization across distributed systems. The talk also introduces a new scheduling strategy that allows for improved handling of workloads in heterogeneous environments.

All right, let's start it. Uh, hi everyone. This is six scheduling maintainer session. Uh, transition from part to rock wall scheduling like fancy fancier title than I than usual I guess. So we are going through all the updates that we made in the last release. I mean actually this release as well. And then like we see how the future looks like. So I'm can say Nakada >>

and I'm Mati Scott. >> So we are presenting today. All right. So and yeah we have a lot things to present. So most likely we don't have enough time for Q&A but we're going to you know be around this room after the session. So you can like please feel free to come to us and ask question after that. All right. So s scheduling uh we are maintaining

components that makes uh power placement decision uh basically scheduling. So the schedule recommended schedule is the biggest part of us. Uh it's the viewing component making a power placement decision based on some you know n condition or whatever and then we have sub project we have actually a lot more but uh like three like I would say popular sub projects right now as desk schedule rock. uh

Q is uh fancy job queuing system like other mission mechanism with C and sharing and other feature as well and uh the scheduleuler it's the component to reschedu the parts quark it's the fake cluster with some quark is actually kubernetes without cub right yeah I think so uh it's like you can create whatever amount of nodes was uh on your laptop and simulate what happens in the

in that scale. So yeah and the scheduleuler is the component. Yeah we explained it and we implement like for example resource request port affinity node affinity some other scheduling preferences and then those features are implemented as plugins internally. So basically like for resource request we have resource feed program it's like you know each program has a dedicated responsibility on it and we also have another concept called

extension points. So this is where program is one on so basically one on any extension points that they want. So first line is filter. Uh I mean these two are like major extential points. Uh filter is like if for example if nodes don't have enough resources you shouldn't run part on it right. So like basically filter is the extension point to exclude such nodes that shouldn't run

this part and then like after future we get the list of nodes that that can run run this part. So after that we get this score extension point that uh like each plugin says how good each node each node is. So like after that we can see which which node is the best node basically and actually there are a lot more these two uh the thing that

we so and then this green box uh this is called scheduling cycle. So it basically calculates the scheduling decision and this yellow box is called binding cycle. So it so green cycle just decide where this part goes to and then this yellow cycle uh yellow box uh this actually applies I mean makes API calls to uh bind this part onto that node and actually this euro part

like binding cycle this is asynchronous uh we running so that is the reason we're like kind of separate writing them and this purple one is called scheduling queue. It holds all the pending parts. Uh there are some uh like a lot of I would say like compre like complex mechanism around wuing and sorting mechanism and but we're going to we are not going to touch today. So

what happens is basically so as I said this queue has uh all the pending parts and uh basically schedules pop one by one. So it goes through filter score as we like as I explained and then finally uh arrives at binding point. uh this is the extension point that actually like bind B part B onto N and the scheduling decision is applied. So this is the very

basic for it and it's like this. So this is the this is the only uh scheduling flow that we had but here we go we have record area scheduling. Oh okay. So what are we scheduling? Uh so this air for started around last week uh which is 1.35 actually we had uh this like kind of design uh from before but uh we actually like implemented some yeah

those uh gang and API at the at 1.35 and then we introduced Yeah, we also introduced some like opportunistic batching feature for uh this is more for like optimization stuff yes so what are we scheduling uh this uh 1.36 uh cycle uh we improved it a lot more uh which Mach we're going to go through. So let's see how it look how it looked like at last

release. Uh so we had this work object sorry work resource and you can say like it's very simple like you can say gang mean count four. It means uh this workload is a gang with four members basically like four members have to be scheduled at the same time. It's like all or nothing and at the okay at the pot side you can like specify which work row

this pod is belonging to. Note that this is old API that we previously had. And how gang scheduling is working is like as we saw we each part goes through like scheduling cycle and then this is uh like permit extension point like basically you can stop part to pro from proceeding to the binding cycle. So what we do is like this. So basically uh until we see

all members to be scheduable we basically stop all parts here and then once we see okay we have three parts scheduable then we release them like admits them to uh binding cycle and binding cycle we were going to make API calls but there's some challenges that new. It's like this implementation is very rough and let's say for example one pot is unscheduable and it never reach reaches

this permit extension point. What happens is like scheduleuler keeps there those those two places and there should be other gangs as well. Why? So like you know it happens that like multiple gangs partially reserves their prices and then like they kind of deadlock each other. >> Yeah. >> Yeah. So now we can move to 136. So first of all start with the APIs. So as Kenz showed

you in 135 we had only the workload API which was a grouping object for the whole like workload entity and it had the problems for example with replicated like jobs or anything because each of those was like separate scheduling object so the port instance but it was consisted in one API object what was the workload API and we had for example problems how to introduce the workload

status and other things. So we decided to split these two APIs. So we have now workload API which is the policy. So static specification of scheduling policies for the for the workload and it consists of templates that the to the workload controller can use those templates to create the port groups which are the okay which are the dynamic instances on runtime. So each instance is like for

example separated replica or separate scheduling unit. So scheduleuler only cares about these port groups not about the workloads. And now it can now pots can point to that specific port group instance. And it can contain a status to show the scheduling state of this port group and other features like the array can use that to clean up clean up some states. So do some life cycle management

and yeah and we now have this workload API. We have the port group which is created from the port group template of the workload and pots can now reference specific port group using the the scheduling group port group name field in the pot and yeah on theuler side as once again I showed you that we had a problem with gang scheduling because the port byput nature of

the scheduleuler is not enough to handle port groups Well, so now instead of using this port by pod scheduling cycle, we propose something different. So we have port group scheduling which currently looks like this. So we have podbased scheduling queue still pod based. We have this new port group scheduling cycle that checks if the pod is physible or not and then if it's physible pots go to

the steel potbased finding cycle. So in particular it the the cycle itself takes the port group it snapshots the cluster state and runs some algorithm that checks the physibility of the entire port group and then yeah go to binding. So this allows us for atomic decision for the entire port group under the same cluster state eliminating potential deadlocks that you saw. So in fact we still reuse

the potby pot scheduling but we do that inside the single cycle. So we run it for each pot and then we bind each pot as well. So when we see how how it works now we have this port group the whole port group goes to the algorithm as the whole we determine which of the port is scheduulable and based on the physibility check we can say okay

the photo is physible so we bind it or we can say it's not feasible because we don't have like enough pot schedule to form a gang so port group goes back to the scheduling queue and binding is is not executed. Or we can even schedule something partially. For example, we have more ports than are needed for the gang. So more than mean count or we use basic

policy that is best effort and then some ports can go to the binding and some points come back to the scheduling queue. But still it's the atomic decision of the scheduleuler to do that. And obviously this algorithm now has some limitations and we have some future plans for that. So now as I mentioned we rely on the podbas based scheduling queue and we need to implement the

pod group based one soon because yeah we have the problems like we still pop the single port from the queue then try to get the rest to process them together but we could just store the port group objects inside the queue and all of the member ports and process them together from the beginning to the end of the scheduling process. The preeemption activation is not decoupled from

the perime calculation. So now we just use the post filter which does the preeemption for each pot. But if the port group is not feasible, we just remove some post from the cluster unnecessarily. Right? So to reduce the disruptions, we can move the actuation to the end of the scheduling queue or to use the workload preemption that I will show you soon. Yeah. And the algorithm has

some also limitations. So if the port groups are heterogeneous as we process them in each port in the specific order fixed order it there can exist different order in which the port group can be scheduled but we don't process that that options. So if you schedule homogeneous port groups it's all okay. We can find the placement but for heterogeneous we may not find the placement. if it

exists even and interp dependencies are partially incompatible. So they should work but maybe not as well as as they work for pots but yeah but we can try to model these dependencies in the future on the port level. So if that is like use case we can try to address it on that layer. And coming to the workload preeemption now we have the new preeemption algorithm that

is called after the port group scheduling algorithm. So when it when it considers the port group as not visible we try the preeemption like single preeemption phase for the whole port group. So don't run this post filters per pot but we have only one and yeah it now like reuses the concepts of the pot preeemption but pree but pot group is the preemptor not the individual pot

and the entire cluster is the domain where we try to evict the victims and not calculate the preeemption per node and we have new fields in the port API needed for the preeemption. So you can now specify the priority class name that is translated to priority in the port group and it's used for the preeemption. So to determine which port group is is like more important than

the other and we have disruption modes that where you can select how the port group can be disrupted. So either you just say pod each pot can be evicted and port group is still running or you set port group and then you like preempt the whole port group even if preempting only one member is enough but for example your application can't run if you don't have all

of the pots right so that's why you do gang scheduling you want all of them and we have this this our uh graph And we have the port group it's not scheduulable. So now we do the workload preeemption for these pots. Right? And then based on the the result we evict the the victim pots port group comes back to the scheduling queue and later it will schedule

once again when yeah the cluster uh doesn't have those and yeah and here is an example of the disruption mode policy. So if P wants to preempt some victim when it's disruption mode pot it can preempt only two of the the victims spot group but if it's spot group it has to preempt the whole port group even if it doesn't have to and this algorithm obviously has

the limitations and we have some ideas for the future. So the default portbase preemption so that executed in post filter for individual ports still ignores the existence of port groups. So the disruption mode port group doesn't work when the port group can be preempted by the individual pot. But we obviously have to address that before beta. Other post filter plugins for example the array that do some

cleanup of resource claims also require adjustment to integrate better with the scheduling cycle. it could use the new phase to to just do it once per per the whole B group, right? And we have the new extension point proposal that will be done in 137. So now we we have this extension point, but it's not exposed to to plugins. We only do that for the preemption, but

we'll expose that. So other implementations of post filter could use also the vlo preemption to do their own uh preeemption right and we can move now to topology of scheduling which is yet another feature that we added in 136. So we added new fields to the API. So now we can define the topology constraints with the topology key. So it's a key where all pots from the

pot group have have to share the same have to be placed on the nodes with the same topology val the same label value set by topology key and we have the new placement based scheduling algorithm with the new extension points. So even if the feature says it introduces the like new framework for other use cases as well. Yes. So we have this placementbased algorithm that can be

used by another plugins in the future. Currently TAS implements that but for example DA can support that as well with some shared resource claims and the algorithm now looks like that. So the difference is that we have this which looks like this. So first we generate placements then then we have this our funny uh scheduling algorithm. So now for each generated placement we have to run the

the port group scheduling algorithm which for each port from the port group runs the port by port scheduling right then we have the scoring if there is anible placement and we bind to the selected one. So we can generate for example if you have the topology for racks we can generate three racks in the generate placement plugin for task. We then evaluate those racks. So we check

and two of them are feasible one is not. So the racks go to scoring. We score them have the the the best rack selected. So for example it has the the the most parts from the port group. we can place on the rack or we try to bin pack to to to make this rack as packed as possible. Right? So we select the rack and we can

bind to that rack and as well we have some limitations and future ideas. So the array plugin doesn't yet implement uh the the points that I told you but yeah they they will do that in the future. Now we we can only have one plugin which is SAS but it will be uh like simplified in the 137. We have also only initial version of task. So we

have single level topology constraints but we can add more levels algorithm algorithm is cano optimized. So for basic policy which is only best effort placement we can start the scheduling event with with one pot if we expect 10 and we can take the like nonoptimal decision for for that for that port group. So we maybe need some counter to know how many pots we expect and then

schedule the the port group using task right and it lacks integration with any preemption algorithm. So both both face default preeemption and workload preemption doesn't work with task but yeah we we have to figure out how how to integrate what workload preeemption with task in the next and we have also other caps related to fast integration in 136. So it's the array resource climb support for port

groups which adds port group level resource clims to port group API. So now claim can be bound to the life life cycle of the port group and we have the MVP job console integration. So job is now integrated with with the pass but obviously this integration with will follow with inside the job but also with with with other to the workload controllers. So our future plans for

us. So these plans are you know our current plans they can change and we even have more plans that that we want to address in the future. These are our closest goals that we want to achieve. So the first one is to promote to beta in 137. So we want to address the limitations and improve our current features and we want to extend the port group API

multilayer structure. So allow for for multiple layers of of port groups so they could form some kind of the tree for example. So we can introduce pot subgroups and pot set to have multilayer support for the pot API. We also have the topic of integration with other controllers as I told you integrate with orchestrators and with autoscalers. So as you can see the vast effort is not

only the scheduling but is also like the whole bubernetes ecosystem. So controllers orchestrators autoscalers scheduling everything to be honest. Uh uh yes so there are other updates still a lot of DOAs uh DOA is still big for us and we've done a lot of works which we cannot go through this time so today we have another session from John and Patrick so if you are more interested

in like DAS Sorry about that but please move to the this session and yeah I'm going to go through sub projects from now very quickly. So the first one is Q. Uh so what is Q? Q is AI work level scheduleuler. It's like it supports a lot of thing that we are doing at commanded schedule actually like over or nothing. This is basically gang scheduling and some

like uh topology scheduling as well. Uh fair sharing uh coder management based like yeah coder management and some other mission mechanism and those are all already implemented in Q. So recent highlights yes we released a lot of features. The first one is API graduation and yeah DOA support and topology D scheduling implements as well. So it now supports uh cub flow train job and it now supports

balanced placement algorithm which is like basically like instead of be parking uh it can now try to like spread parts across like over multiple domains and this free capacity algorithm for non-constraint model works uh yeah it makes has unconstrained more much closer to required mode which is already existing and other like module improvements as well. Uh we uh it now supports enabling both task and pro provisioning

request. Uh it's like until now you can you could only uh enable either of them but now you can enable both of them and authentication to work cluster using profile API uh which is the project from sigma cluster I think uh is now you know securely authenticate the cross cluster communication and good news we also have another uh tomorrow uh from uh Yuki and Kevin. Uh this

is a walking world batch update and it covers a lot more stuff around Q and others. So please join if you interested in also they have a booth of Q at the project pavarian. So if you have some some specific like topic to discuss you can also visit uh that booth. Yes. Lastly, we have deskuler. Uh the scheduleuler as I explained at first it's like reschedule parts

based on cluster state. It's like you know the uh scheduleuler scheduleuler only schedules parts. It means it doesn't care what happens to those parts after scheduling happened. So even if you want to you want your you want your parts to for example spread across the like zones for example nicely. Uh scheduleuler only cares it um at at scheduling timing and it doesn't care after that. So maybe

if you your your deployment for example scales down uh your like spread might be you know there could be some unbalanced uh spread. So this scheduleuler solves this problem. Uh and what's new uh it now implements a default avictor plugin and also it now get uh supports a port lifetime plugin. As far as I know they also have a similar mechanism like you know similar plugin and

framework concept. So there are some extension points I I I believe I'm not sure if we have a desk schedule maintainer in this room but you can ask if there are all right so yes uh we need more reviewers and long-term contributors so if you're interested in all those works please join us and we have six scheduling Slack channel obviously and we have community meetings uh two

like six scheduling meetings yeah two scheduling meetings one for Asia Europe region and the other one for like between America and Europe and also we have another meeting uh which is uh like mainly for road API integration stuff and yep you can join based on what you're interested in. That's all. Thank you so much. And if you have a question, please come to us.