Instrumenting Kueue Scheduling for ML Training - Amy Chen, CoreWeave & Gabriel Saba, Google

This talk focuses on Q, a batch job dispatcher designed to effectively manage computing resources in machine learning training environments. The speakers, Amy and Gabriel, introduce the challenges of scheduling jobs across multiple GPUs and highlight the complexities that arise from resource contention and utilization. They discuss how Q facilitates quota management and queuing to ensure efficient workload scheduling. The speakers explore common issues such as guarantee violations, fragmentation, and slow admission rates, providing a step-by-step approach to diagnosing these problems. Throughout the session, they emphasize the importance of monitoring and clear visibility in the scheduling process to address the needs of researchers and operators alike.

All right, let's get started. Hi everyone. Thank you so much for coming to our talk um titled instrumenting Q for machine learning training. My name is Amy and I'm a software engineer at Cororeweave and an upstream Q contributor. >> Hi, I'm Gabriel Saba. I'm a Google software engineer and a maintainer of Q. >> So for today our talk will start with briefly introducing the Q scheduling problem

space and then we'll understand where Q fits as a batch scheduler and then after that we'll provide a highle overview of Q. So the reason why we're going through all this is because it's important to understand some concrete Q scheduling issues later in the talk and finally we'll offer a recipe to answer researchers when they ask why isn't my workload scheduling. So let's start off by chatting

about the motivation for Q. Q is a batch job dispatcher. The reason why we need something like Q at all for machine learning training clusters is because compute is expensive and researchers need to split up GPU compute time in a reasonable way. So if this hardware were cheap, each researcher could just sit on a bunch of capacity without worrying about whether that hardware is being used or

not. Given these constraints, compute organizations really care about the percentage of utilization overall for the cluster. So you basically want as close to 100% utilization 24/7 as possible because again this stuff is expensive. Um in a large multi-tenant training environment, people really care about the GPUs they are vended. So large organizations for instance have cost centers, budgets, a lot of different financial models for how they're vending

GPUs to underlying teams. However, using a batch dispatcher requires complex queuing, prioritization, and quota rules, which can be difficult to reason about, especially when you're trying to understand if there's scheduling bugs along the submission path. So this opaque scheduling queuing process directly leads into a confusing um question where researchers often ask why practitioners of slurm for instance a scheduler that has been here way longer than Q

um and that is over 20 years old also have problems answering this complex question. This is actually one of the most important questions for training cluster operators and one of the hardest questions to answer because of the many togglers all schedulers have. So, as platform operators, we need to be able to answer when a workload is expected to remain pending or fix it when there's a failure

in scheduling. >> We'll start with an introduction to scheduling and illustrate how difficult of a problem it is. Scheduling is the temporary assignment of jobs to tasks. If resources were unlimited and free, this problem would be simple to solve. As we well know, this is hardly the case, especially when we're speaking about accelerators. Scheduling is hard. Even problems which turn seem to be very simple at a

first glance turn out to be computationally inractable when we try to find the optimal solution. Consider the following problem. We have three identical processors and nmp compute tasks which may have different running times. Our goal is to schedule these tasks across the processors to minimize the total time known as makes span. The decision problem associated with this optimization problem turns out to be npmplete. There are often

polomial time algorithms which are nearly as good as the optimal solution. In the three processor problem above, there are greedy algorithms to solve the online variant of the problem where tasks come in one by one with a completion time that is at most 67% worse and the offline variant where we know the whole problem ahead of time and a completion time that is at most 22% worse.

As we can see, even a very simple problem may end up being computationally intractable. And the problem which Q is solving is much much more complex in comparison. With that introduction, why is Q's problem so hard? Firstly, we have many more machines, potentially thousands or tens of thousands. These machines are likely not the same. Additionally, the set of machines may change over time. Next, the problem is

online. We need to make scheduling decisions without knowing all the workloads we need to schedule ahead of time. Finally, the objective is multifaceted and configurable. We must enforce quotas. We try to schedule high priority workloads first and we try to maximize resource utilization. Additionally, we try to avoid unnecessary preemptions to share resources in a fair manner and to have a high scheduling throughput. To handle this difficult

scheduling problem in a relatively efficient manner, Q uses heristics with some examples being priority cues, sorting, scoring, and fair sharing. So what is it exactly that Q does? First and foremost, Q is a quota management system. Consider the scenario on the right where we have a cluster capacity to run three pods total and two jobs that consist of three pods each. Q enforces that resources used by

admitted jobs are not greater than total cluster resources. This allows the overs subscription of a cluster without jobs deadlocking. Many batch jobs require that all of their pods are running to make progress. If we are not careful, we may end up in a state where several workloads schedule some but not all of their pods. And that's exactly what happened here. When we try to schedule these jobs

without Q, some pods from each of the two jobs scheduled, but neither will make progress, a deadlock. With Q, in contrast, we can guarantee a schedule like this where a job gets all of its pods scheduled and other jobs Q until enough capacity exists to run all of its pods. Q's other primary function is that of a queuing system. It decides which workload should run next when

there are limited cluster resources. Suppose that the first workload finishes and a new workload green cues up. Assuming these workloads are of equal priority, Q will take into account that the orange job has been waiting for longer and schedule it first. When deciding which workload to admit next, Q considers priority, creation time, and when enabled, fair sharing. If you're interested in the details of Q's fair sharing,

check out the lightning talk by me and Yuki at the CubeCon in Atlanta. Next, we'll talk about some of Q's basic data structures. The resource flavor is Q's abstraction for different types of hardware. Taking the example on the right, we have two flavors representing two types of Nvidia GPUs, H100s and A100s. Administrators can map a flavor to a subset of nodes via node labels. Upon job admission,

Q adds the appropriate node selectors to the job depending on which flavor was assigned. Next, we have cluster cues. Cluster QES allow the definitions of quotas and handle the queuing of workloads. This Q defines quota for the H100 flavor with a guaranteed or nominal quota of 8 GPUs. Finally, we have cohorts. Cohorts allow administrators to model their organizations, grouping cluster cues and other cohorts into arbitrary tree

structures. This feature is also known as hierarchical cohorts. Note that this cohort has defined quotas for both the H100 and the A100 flavors and has a parent cohort root cohort defined. Now let's quickly look at Q's scheduling internals. We will go deeper into select scheduling topics later on in the presentation. During a single scheduling cycle, Q considers workloads from each cluster Q which has a pending workload.

Usually this is the highest priority workload of the cluster Q. Though there are some cases where this doesn't hold, such as when the highest priority workload is in the inadmissible queue or when we're using admission fair sharing. Next, we simulate resource assignments and preemptions. This is the step where we match workload requests to resource flavors, which we talked about a couple of slides ago. Recall that resource

flavors are cues representations of different types of hardware. When making the selection, there's often the trade-off between preempting in one flavor and borrowing in another. Administrators can configure how to handle these situations using flavor fungeibility policies. After making these flavor assignments, we order the workloads and try to admit them one by one. This ordering step is important as several workloads may be competing for the same resources

or trying to preempt the same workloads. We will discuss the latter issue later in this talk. Process processing the workloads in order. We admit workloads where there's enough free capacity, initiate preeemption for others, and skip the remainder that either don't fit or have some interference with earlier processed workloads. Thanks, Gabe, for covering some fundamental aspects of Q scheduling. This will be useful for understanding which aspects of

Q we should be instrumenting. So if you've worked on any machine learning training platform, no matter what batch scheduler, so like Slurm, Q, Volcano, Unicorn, researchers will ask, why is my workload pending? What you're looking at or what they're looking at is a guarantee violation. So what's a guarantee violation? What it is is it you have a large multi-tenant cluster where each tenant bubbles up to a

different cost center. So what I mean by like tenency is often like different projects, different teams, um, and it's all split up on the on the same cluster. So in this type of cluster, every tenant really cares that they have access to the capacity that they were vended. A guarantee violation means a tenant is unable to access their vended quota. In the example above, the user has

submitted eight workloads each with one GPU. In this example, four are admitted and four are cued. And again, each of these have one GPU. Um in total the user has four remaining GPUs of nominal quota or guarantees with four uh four GPUs of worth of workloads pending. This is a guarantee violation because theoretically these four remaining workloads should be admitted. Users will emphatically ask why can't I

schedule this workload despite having guarantees available. It's actually deceivingly difficult to reason about. Um and users really want this uh understanding and visibility. So guarantee violations can be due to like reasonableish reasons like fragmentation or due to unacceptable reasons like scheduling bugs at various scheduling layers. So in summary, what is a guarantee violation? It's when cued workloads can fit within unallocated guarantees, but they're still not scheduling

for whatever reason. This is a very important concept because it is the first signal of when something is going wrong. Um, and that's why I wanted to offer a conceptual understanding of what this is. So when you initially get alerted about a guarantee violation, there's three tiers that the failure could occur in. First and and an optional layer is a submission layer. So this is often needed

if you're using the pod workload type or if you need another like DAG layer on top. This layer is optional because sometimes if you are using a smaller platform which wouldn't have this layer, um users will just directly submit their workloads directly to Q and just like kind of use Helm charts a little bit. Um, some platforms uh have a submission layer if they need to manage

a DAG. They may also need a submission layer if they're using the pod workload type. And because pods don't have an operator to bring up a minimum number of replicas, another layer is required to handle this uh logic. Platform operators may use the pod integration if they need to define some custom scheduling semantics that don't already exist within existing operators like job or job set. Second, you

have the Q layer. So this is what is most relevant to us, right? Um Q exists as a controller that gates and ungates Q workloads. It does not work as a Kubernetes plugin. And finally, you have the Kubernetes layer. So after Q removes its scheduling gate or unsuspends the workload, the workload is released to Kubernetes for scheduling. And each tier has different characteristics. Okay, so now we

know how to signal when something is wrong from the admission perspective. We start off with seeing a guarantee violation. We specifically see that there's enough unallocated guaranteed quota, also known as nominal quota, to fit the pending workloads. So then what? Let's go ahead and follow this recipe. This is just a highle description of what we're looking for and we'll copy and paste this template for multiple practical

issues that we've seen submitted upstream. Um, so I will provide more specific metrics when we run through some concrete bugs. After this, as a next step, let's rule out fragmentation. Um, and I need visualizations for this, so I'll hold off describing this um until the next slide. If we've ruled out fragmentation, we should look at slow admission and preeemption rates. This gives us further clues that there's

a blockage within Q specifically somewhere. Next, you can also check if Q has released the workload to Kubernetes and if Kubernetes is the one actually having issues with scheduling the workload. So, Q has two flows to release workloads to Kubernetes. the first flow and they're uh not interchangeable so it's either one or the other. So the first flow is they remove scheduling gates. This applies to resources

like pods, stateful set and deployment. For the other flow um it's unsuspending the workload. So this applies to resources like job or job set among other ones. Why could Kubernetesuler have issues? The scheduler could be overwhelmed. There could be hardware failures um rendering the hardware unscheduable or for instance like PVC binding errors among other issues that I'm sure are missing. Finally, once we've seen all these clues,

you can start looking at the logs to really narrow down where the problem lies. I would actually recommend configuring your uh log to zap log level five for the Q controller. Um because it actually gives you a lot of uh interesting logs that wouldn't otherwise be shown at log level two. So as promised, let's go ahead and look at fragmentation again because I needed visualizations. So it's

important to understand how you conceptually measure fragmentation. What even is fragmentation? Fragmentation is when the total amount of resources is sufficient to run the workload. However, the workload is unable to run because the physical distribution makes it impossible. Using the example in the image, we have one workload that is eight GPUs large. across eight nodes we have eight unallocated slots for only one GPU each. So this

workload is physically unable to run. Instead of eight separate uh different nodes with one GPU each, it actually needs one 8GPU node to be able to run successfully. With this understanding, how do we even measure fragmentation? Um and we'll see this in later visualizations as well. What you do is you sum up the quote unquote shapes of quota available per node. So for instance, how many 8GPU

slots do we have? If there's at least one 8GPU shaped hole available, you can rule out fragmentation for this specific example. This issue manifests itself differently when you have topological aware scheduling, otherwise shortened as TAZ, turned on versus off. If you have TAS turned off, the failure occurs at the Kubernetes level. Q will admit the workload and send it to Kubernetes. But because the workload cannot physically

schedule on the Kubernetes layer, it will hit the pod ready to pod ready timeout if this feature is enabled and fail. With TAZ turned on, Q will prevent admission of the workload, allowing for other workloads that do not topologically fit to be considered for scheduling uh for admission. However, that workload is still um not able to be admitted by Q, right? So, top topology is still an

issue. So let's go ahead and put together our knowledge from Gabe scheduling overview and my recipe for un for instrumenting Q to walk through some real issues practically. I should make a disclaimer that all the following graphs are presented from simulations and not from actual production workloads because customer information is very uh proprietary and delicate but um hopefully it gives you a good example of what you

can do on your own production uh environments. So now that we've gotten through a high level understanding of Q instrumentation, let's go ahead and dig into concrete bugs reported to upstream Q. For the upcoming bug, it's reported when users uh run through two different scenarios. One, they're migrating to a new cluster and they migrate one cluster queue at a time. And then for the second scenario is

is when they have a sudden spike of high priority inference workloads that come into a cluster queue. So going back to our recipe uh for this bug, first we notice that they are guarantee violations. Short guarantee violation spikes are considered acceptable. Q needs time to admit workloads. So we expect to see temporary guarantee violations. Longer flat that persist over 30 minutes to hours are when researchers start

to notice and escalate to platform engineers, which is us, and that's abnormal. There's three metrics important to finding a guarantee violation. One, how much quota is being allocated to the cluster Q. Two, the total nominal quota or guarantees that the cluster Q has. Three, the quota of the cluster Q's pending workloads. I should note that this final metric isn't doesn't actually exist in Q yet. I'm working

on this for issue number 9741. In the meantime, the workaround is is you can actually create a custom metric in cubestate metrics. Um to construct this at a high level, you admit a metric when you see that the workload exists to request capacity, but the but that the condition quota reserved in admitted conditions are false. Or you could just wait for me to make a PR. Um

the step two is ruling out fragmentation. Often times platforms standardize workload sizes. So for instance, they could be standardized to 1 2 4 8 and then greater than eight is multiples of eight. So what we're looking for are the number of slots that can theoretically admit the pending workload. In this example, we're submitting 12 8GPU workloads. If you have enough AGPU slots free, you can rule out

fragmentation. If there are no HGPU slot size slots free, the cluster is currently fragmented. Fragmentation is loosely a quote unquote acceptable reason that prevents workloads from being scheduled. Um I should note that topology decay is out of scope for this talk but as at a high level topology decay is something where you can imagine a state where on your cluster each node is occupied by one GPU

workloads but it prevents a large workload which is usually much more important like say 240 GPUs over 30 nodes from being scheduled. The community still needs a solution for topology decay but it's a very difficult problem face for this bug. We can see that fragmentation is not an issue and go ahead and rule it out. For the third step, let's go ahead and check admission and preeemption

rates right after submitting a bunch of workloads. Remember, we are current we currently have a guarantee violation. We're expecting the admission for the cluster Q demo CQ to be close to instantaneous. Instead, what we see is admission is very slow. So, one workload is being admitted to the demo cluster Q every 15 minutes. We're also seeing that the preeemption is happening very slowly where a set of

workloads is being preempted by the demo cluster queue every 15 minutes. Again, this is a simulation where in real life the gaps wouldn't be this perfectly timed. So this pattern is weird. Why is this pattern weird? When there's a guarantee violation, we expect admission and preemption of workloads reclaiming their nominal quota or guarantees to be extremely fast. Instead, I would describe this as more like a trickle.

So step four, we would uh I would recommend checking to see if Q has released the workload to Kubernetes or not. As covered previously, depending on the workload type, you'll either see that the pod has its scheduling gate removed or for types like job or job set, you'll see that this uh suspend field being marked as false. So it means it's it's still suspended. So I'm using

the pod workload type. So I'm checking for scheduling gates. Here you can see that Q is not released has not released the workload to Kubernetes. If we search for the corresponding workloads admission condition, it would be listed as not admitted. This means that scheduling is actually stuck at the Q layer. So if we remember the cake, um this is where you can start looking for issues. In

comparison, if you saw a buildup of pending workloads, it would mean that Q has released the workload to Kubernetes, but the scheduling failure is happening on the Kubernetes layer, and you could direct your attention to that layer for debug debugging instead. um while not as relevant for this particular bug, but it'll be more relevant for for the other bugs. Step five is that what I would say

is just broadly like searching for weird logs. And then step six, checking for overall system health issues. If you look at control plane metrics and things like that or if you uh notice any configuration, major configuration changes. So for this specific bug, I didn't find any weird logs. And what I mean by that is I usually just search for the work workloads name and logs and look

for anything that stands out. Also, I didn't notice any overall uh system health issues and also there were no major configuration changes at the So, let's go ahead and summarize our checklist that we just ran through. Again, to recap, we noticed this issue when uh when users in one situation were migrating cluster cues to a new cluster in the second situation when a wave of entered a

cluster queue. And when this happened uh what we saw was one guarantee violations that's always the first step. Two fragmentation was not an issue. Three we saw uh trickling admission and preeemption. Four Q did not release the workload to Kubernetes meaning that the issue is probably scoped to Q and that's where you can focus your attention for debugging. Five there were no weird logs. And six we

checked overall system health to see if anything stood out and no configuration changes and nothing nothing really stood out. So what's actually going on? Let's call back Gabe for a quick sidebar on >> Another detail from scheduling. When we're waiting for preeemptions to complete, we ensure that the workload that is at the head of the queue remains there. This is to make sure that another workload which

is deeper in the queue doesn't steal resources or preemption targets while the head workload is waiting for the preemptions to complete. And we can see that illustrated in this diagram here. >> Okay. Um, so we're using the pod workload type, uh, which waits for pod preemption targets to finish preempting before releasing the admission candidate. So the root cause was head of Q blocking when using the pod

workload type. At the time for the pod workload type, it would wait for preemption targets to complete preeemption before being admitted. However, the user had configured the pre-top life cycle hook and grace period termination for 15 minutes. So at worst case it could take 15 minutes between workloads reclaiming guarantees to be In the migration case the entire cluster was empty. So the first cluster key you migrate

it can admit everything and then borrow a bunch of capacity within the cluster. Right. and uh so right uh including admit everything pass guarantees into the burst capacity. So incoming cluster cues being migrated needed to preempt workloads bursting past their capacity to reclaim their guarantees. As a sloppy mitigation, you could manually preempt a bunch of workloads bursting past their guarantees to finish those uh preemptive targets to

finish preempting. In the inference case where it was a bunch of high priority workloads coming into the cluster queue, the high priority workloads needed to wait for low priority workloads to finish preempting. So again, as a sloppy mitigation, you can manually preempt a bunch of low priority workloads in the cluster queue. Luckily, you don't need to do these temporary m uh mitigations anymore because the GitHub handle

tscillion, thank you. Uh in upstream Q implemented a solution for the pod workload type where once something is marked as a preeemption target, its quota will be immediately released. So this is similar to actually how the job workload type works. Let's go ahead and move into the next bug that we saw. I won't go through as many visualizations for this since the charts are pretty similar. So

what we can do is verbally run through what had happened. So using the same formula as previously laid out, the first step as always is that we're alerted on Um for the second step, what we can do is rule out fragmentation. For the third step, we noticed a slowdown in successful admission rate across all cluster cues. And then we checked that there was no buildup of pending

or unsuspended workloads. Then we checked for a buildup of potentially quote unquote weird logs. What we saw was that the workload was the log workload has overlapping preemption targets with another workload. The reason why this I consider this like medium weird and why I labeled it with like a a yellow circle um is because in normal operation you actually do see these logs pretty often. What stood

out was seeing this log for the same head of cluster Q workloads across a long period of time. And then finally, we actually did see something weird in the system health um that uh we didn't notice until until purposely checking on it. So what we had missed was alerting that we had failed garbage collection for a pod living past its grace period termination time. So what's going

on? Let's go ahead and check at check out the Q scheduling algorithm and move back to Gabe. >> During preeemption, we use the opposite criteria that we use for scheduling. We preempt from cluster Q's and cohorts that have the highest dominant resource share. This is the measure that's used in fair sharing. That is those are which are borrowing the most in the graph. Note that cluster Q2,

this is the cluster Q in the middle that's painted red, has the highest DRS value. In cases similar to this, many cluster Q's will target workloads in this cluster Q for preeemption first. The issue is that resources can only be reclaimed once and therefore we'll have to invalidate all subsequent workloads that choose the same cluster queue the same workloads for preeemption. So what happened for this sample

issue is that there was a large zombie pod that multiple heads across cluster cues were targeting. Recall that previously for an admission target to be admitted it waits until the preemption target completes. On top of that, if downstream cluster Q heads share the same preemption target, Q will skip this cluster Q for the scheduling cycle. As a result, we saw that admissions slowed down across across cluster

Q's because a lot of the cluster Q heads targeted the same stuck workload. So, the root cause was a stuck zombie pod. The temporary mitigation was to forcibly delete the zombie pod. Zombie pods occur if the cublet fails or another known bug is if you terminate the pod while pulling an image. As a follow-up change to Outstream Q, Q now releases the quota for preempting pods if

they exceed the grace period termination time. So, as a platform, you should still actually configure notifications when you detect a zombie pod to ensure that preemption completes out of band of Q. It's more like at the Q layer, we just now won't stop scheduling as a result of it. Let's move on to the next bug and again using the same checklist. First, we saw that there are

guarantee violations. Second, and I'm again repeating myself, uh fragmentation was not an issue. Third, we didn't actually see an overall slowdown in admission preemption rates across cluster cues, which is like different from the other bugs that we saw. There was no buildup of pending or suspended workloads, and we also didn't see any weird logs, well, at least at first. Um and then in terms of like guarantee

uh instead of um configuration changes uh that occurred during this time um we noticed that guarantee violations coincided with system changes. So specifically we had enabled preemption fair sharing at the time that we saw guarantee violations. Um at the time we had the log level at two. So then after we turned on the zap log level to five, we sort of noticed something odd with the log

dominant resource share values used during tournament. So, what's going on here? Um, I'll let Gabe go ahead and dig more into Q scheduling. Again, >> with fair sharing enabled, we simulate dominant resource share, this DRS value of cluster cues and cohorts after the admission of workloads. Using this simulation, we run a tournament to order workloads for admission starting with those cluster cues and cohorts with the lowest

DRS values. Zero is the lowest possible DRS value and it means that the cohort or cluster Q is not borrowing any resources. Um, in any words, it's in other words, it's within guarantees. This matches our intuition. Cluster cues and cohorts that are within guarantees should be the highest priority for So, right, it turned out there was a rounding error where in some cases we assigned a DRS

value of zero to cluster Q's which were actually borrowing. So, the root cause was incorrect dominant resource share values. When you see the DRS value equals zero, it means cluster Q's are reclaiming its guarantees. When you see a DRS value greater than zero, it means that the cluster Q is bursting past its guarantees. So in this situation, some cluster Q's bursting past its guarantees were actually incorrectly

being marked with a zero DRS value, meaning that Q incorrectly thought that the cluster Q was reclaiming its guarantees. This also means that for the cluster Q's correctly reclaiming its guarantees, there are no more remaining preeemption targets for this admission candidate to select. We should also note that you'll no no longer face this issue in production because we fixed it upstream. Uh shout out to Dominic. Thank

you. Um yeah, so that concludes our deep dive into more specific issues that were actually seen uh reported to upstream Q. So to recap, here are some starting recommendations we have when building out a dashboard to monitor Q scheduling. Most importantly, tracking guarantee violations can initially alert you to an issue before your researcher's notice. We're adding some additional metrics which will make it easier to build out

such a dashboard and to visualize cohort guarantee violations. These metrics include pending resources at the cluster Q level, quota and usage at the cohort, cohort level, and finally hierarchy information so that you can plot all cluster cues or cohorts in the same cohort tree easily. So you should go ahead and stay tuned for these new metrics. Um so for the total pending resources per cluster Q, you

can track Um the reason why this is important is instead of using the hack that I mentioned before to determine pending workloads with the cube state metrics or maybe oftentimes organizations have their own ways of tracking this with this metric introduced, you can see live what pending resources Q has. This would help you measure guarantee violations easier. And then for the second thing that I mentioned uh

for cohort tree uh resources monitoring you can track issue number 7539. Um although we haven't covered hierarchal cohorts it's useful to monitor resource usage at each level of hierarchy in the tree. So with these new metrics you can do something like answering these questions. How much GPU quota is available for my organization? Which organizations are within guarantees and which are bursting past guarantees? Next, let's remember that

measuring admission and preeemption rate is a must. These metrics can alert you to a problem at the Q scheduling layer. Finally, plotting available nodes by number of GPUs to track fragmentation is invaluable. And then for the the pending or schedule gate thing that I mentioned, you can do spot checks for those. So, you can decide whether or not you need dashboards for that. I just want to

say thank you everyone for your time. Um, if you're interested in learning more about Q, I've listed the website where you can read the documentation. Also, the Kubernetes Slack channel working group batch is actually pretty active. I can say hi to you there. So can Gabe. Um, and the batch working group meets every other week on Thursdays. If you have more questions, we'll both be at the

Q booth right after this talk. Thank you.