Making Topology-Aware Scheduling Practical for AI Workloads: From Discovery to Simula... Weizhou Lan

This talk, led by Weijie Lan, focuses on making topology-aware scheduling practical for AI workloads. It highlights the significant networking communication volumes in model training scenarios, particularly in dense models and MoE models, which require efficient data routing to avoid network congestion. The speaker discusses the challenges related to multi-tenant workloads and the necessity for intelligent topology scheduling, outlining key strategies such as isolation and affinity. Various scheduling enhancements, including Volcano and CAR from NVIDIA, are introduced as replacements for the default Kube scheduler. The importance of a topology detector for accurate network mapping and health monitoring is emphasized, as well as solutions for validation across different network environments using tools like NVIDIA Air.

Thank you for joining my session and today I would like to give a talk, give a session about making topology aware scheduling practical for AI workloads from discovery to simulation at scale. Uh, I'm Weijie Lan and senior tech lead at discuss the goes beyond the the scheduler. It includes uh, network topology discovery and cost effective approach for function validations. So, let's get started. As we know in

model training scenarios, uh, networking communication volume is enormous uh, for conventional dense models such as Llama or reduce communication uh, uh, dominate in this pattern. Uh, data exchange uh, typically occurs among the GPUs within each single network area. As a result, most uh, traffic can be handled locally by leaf switches. In contrast, MoE models like Deep Seek routes data to different uh, experts and generates intensive all-to-all

communication. This traffic frequently traverses the upper tier switches to reach destination across different areas. Uh, consequently, uh, in a large-scale cluster, uh, workloads must be placed in an orderly manner according to the topology to avoid network congestion. In inference scenarios, uh, complex RDMA traffic uh, patterns are also unavoidable. Uh typical example is PD disaggregation, a prefilled and the decoder workloads needed to synchronize KV cache data. Uh

so, they must uh be placed uh on topologically close nodes. Uh this shortens the switch uh forwarding path and it helps avoid uh traffic conflicts uh with other jobs. In addition, MOE model uh inference also involves all-to-all communication, often requiring traffic to traverse multiple layers of network topology. So, related tasks should be uh close together uh within the same network domain, and unrelated workloads uh spread out

to uh minimize path level traffic contention. Network traffic is further challenged by multi-tenant workloads. In a multi-tenant environment, uh nothing is static. Uh we are constantly driven by requirements like canary releases and SLA. To maximize GPU utilization, we cannot afford to isolate resources for each tenant. Instead, um the cluster is treated as a single fluid pool. Uh workloads from different tenants uh must be free to dynamically

uh scale across all nodes. Uh therefore, the flexibility comes with a challenge. If workloads scale without topology awareness, uh network contention becomes inevitable. Uh or it can lead to unfair allocation of network resources uh, within certain parts of the topology among tenants. To adjust this uh, challenges we've discussed the in in the past the slides, we need the intelligence topologies scheduling. It's no longer enough to uh,

just to find the free slots. We must find the right slots. We rely on three core strategies. I think first pass isolation. We ensure that uh, unrelated AI workloads do not uh, contend with each other. If a job A and a job B are independent, their traffic should never collide on the same network path as much as possible. Second, affinity. For affinity sensitive workloads uh, like the

prefilled and the decode pods, uh, distance is the enemy. We make sure communication paths are as short as possible. Third, health health uh, awareness. Uh, network can fail and links can degrade. Our system should constantly monitor network health and triggers uh, disaster recovery rescheduling when necessary. Um, um, ultimately we should like to uh, achieve several objectives um, to guarantee job SLAs by minimizing communication latency and to

minimize uh, ROI ensuring that our uh, expensive network infrastructure is fully utilized and to ensure network resource uh, fairness among the tenants and the applications. Uh, but it's the only native Cuba scheduler. Uh in some scenario simple scenarios, someone might consider using node label combined with pulse node affinity to control scheduling. Uh I think unfortunately, um this approach is fundamentally uh is unsuitable for AI workloads. Uh

first the reason, manual orchestration overhead. Uh node affinity requires uh us to manually map a physical topology uh to logical labels. This depends heavily on human planning. As the cluster grows, static partitions can cause conflicts between applications and operational costs quickly um um escalates. So, second reason, topology drift. Labels are static metadata. They have no awareness of dynamic physical infrastructure. If a network cable is removed or

a switch is under maintenance, but the label isn't updated in real time, uh the scheduler is operating on a updated map. Third reason, lack of hierarchical global view. Node affinity is essentially set-based matching. It only knows whether a pod belongs to or does not belong to a node. This flat view cannot represent a global network topology or quantify complex architecture. As a result, users often uh define

node affinity rules that truly reflect the physical network. Final reason, gang scheduling. It is a uh all or nothing game. Without the scheduling the entire group simultaneously, we may end up with the partial allocation. Some resources are locked while others are missing. It leads to resource deadlocks across the cluster. On this slide, we outline the core requirements for topology-aware First, hierarchical topology view. The physical network is

modeled as hierarchical key tree covering both scale-up and the scale-out architecture from smallest to largest. The typical hierarchy is node and a super pod and a rack and a spine. Second, hard and soft topology constraints. First, hard constraints are strict. All pods of a job must stay must stay within a specific domain. If there's sufficient nodes, the scheduling stays pending. Soft constraints are flexible. The scheduler tries

to pack pods tightly in a domain like rack and only spills over to adjacent domains if necessary. Third, optimize the switch path for communication. For workloads like ML inference or PD aggregated tasks, we should place pods close close together within the same In the open source community, Volcano, Kai scheduler, and the queue already support the topology-aware On this slide, we have queue and enhancement over the native

Kube Queue acts as a a gatekeeper for job scheduling. We could use node labels to define a topologies that are deeply replicated lines to the a hierarchical network topology. Each cluster each cluster node is labeled accordingly. So Q can understand the logical topology. We can specify hard hard or or soft topology constraints using annotation labels in the workload YAML. When a job is submitted, Q first calculates

the remaining capacity within the relevant topology domains. If sufficient resources are available, Q injects node selectors or pod affinity to pod templates. This guides the native kube scheduler to make placement with decisions that respect the topology Now, we are uh let's look at the Volcano. Volcano serves as a replacement for the default kube scheduler. It builds a logical topology tree using hypernodes CRD. We can manually assign

labels to nodes for each switch tier of the topology and uh Volcano will automatically sync these updates to the hypernode instances at periodical intervals. When uh submitting a Volcano job, uh we use the network topology field to specify a topology constraints. I think a key advantage of Volcano is the integration of InfiniBand InfiniBand topology discovery. Uh Volcano can interface directly with the InfiniBand UFM to automatically pull

the network topology. This eliminate This elim- elim- um eliminates the manual effort of maintaining node labels. On this slide, we introduce CAR developed by NVIDIA. Uh it's a a replacement for the default Kube scheduler. CAR scheduler use a topology CRD to understand how a network is uh structured. Applications can define topology constraints either in pod group CRD or directly through pod annotations. These constraints can be specified

as required replacement or preferred replacement. A key feature of CAR scheduler is subgroups. Within a pod group, subgroup allow multiple sets of pods to have their own the independent topology while also enabling a group-wide constraints across all pods. This ensures that affinity-sensitive workloads are placed uh uh close together. Uh it is especially suitable for a pre-fuel and the decoder workloads in PD um disaggregation. So, let's look

uh let's look at the list uh typical topology example that combines with scale-up and scale-out networks. As shown in the diagram, within each GB200, every 72 GPUs are interconnected via NV switch to form a super pod. Nvidia refers to this as multi-node NVLink domain or NVL domain. These super pods are also interconnected through rack and a spine switch, forming a scale-out fabric. So, the topology hierarchy from

smallest to largest is node, super pod, rack, and spine. Now, suppose we are deploying a group of vRAM inference instances. As shown in the YAML on the right of the slide, we can apply a hard topology constraints to ensure that the entire workload stays within a single rack domain regardless of how it scales. And we adopt a bin pack strategy and apply soft topology constraints to keep

instances tightly grouped within a few adjacent super pods. This allows communication to fully leverage the performance advantages of of NV switch within the As a result, all pods are deployed in a structured manner. So far, we would discuss the advanced schedulers. They are powerful, but they are also one critical dependency is node label for each topology layer. These node labels represent the logical mapping of underlying network

topology hierarchy. If the labels are wrong, the scheduling decision will be wrong. Humans cannot reliably integrate complex the network topologies. It is not a reliable to manually label node at scale. That's why we need a topology detector or builder. Think of it as the bridge between your physical network infrastructure and the Kubernetes. It runs inside the cluster and solves two key problems. First, network auto discovery. It

connects directly to external systems such as InfiniBand fabric manager or NVLink manager tools automatically discovers how the network is wired uh from scale-out switches down to scale-up GPU internet connects. It builds an accurate topology view and it tells us uh the position of each node uh different network hierarchy levels. uh network health awareness. It is not enough to know a link exists. We need to know it

works. If a cable is uh degraded or a switch port fails, the detector should identify it immediately and update node status accordingly so the scheduler can make the right decision. Now, let's look at the uh topology detector solution from NVIDIA network environments. topograph automates the entire workflow uh workflow. It supports the full NVIDIA networking stack. When you're running uh RoCE on Spectrum switches or InfiniBand on Quantum

or leveraging NVLink switch for scale-up connectivity, it has visibility across all fabrics. continuously talks to network infrastructure. It builds an accurate topology map and the light it automatically applies multiple topology labels to each node. Each label represents the node's position at each level of the network uh hierarchy like super pod, uh rack, and the spine. Uh because this process is automated and uh synchronized with uh the

with the fabric, the labels stay accurate. So, scheduler like a cast scheduler can immediately uh make right decision of topology aware scheduling. So, what if your network hardware comes comes with vendors other than NVIDIA? In a non-NVIDIA network environments, uh we may need to develop a custom a custom topology builder. For the scale-out network, many problems many modern switches uh run Sonic OS. Uh topology builder can

query the switches via GNMI protocol to collect all the network neighbor information. For the scale-up fabric, each GPU vendor repre- uh provides CLI tools or API to expose domain IDs. By collecting these uh IDs, we can uh determine which GPU are physically interconnected by scale-up network. Therefore, they can build an accurate topology map and apply multiple topology labels to each node for the scheduler. Now, we are

facing validation challenge across multiple scenarios. First, scenario for scheduling decision. For instance, after deploying and configuring the scheduler, we need clearly we need a clear visibility into its scheduling behavior. second, uh like scenarios for network topology construction. For instance, if we want to develop a custom topology builder, we need an environment for developing and testing. So, we need a a network environment environment with rich and realistic

topology layers covering both scale-up and scale-out architectures. However, it may require a dozens of switches or GPU servers. The cost is uh prohibitive prohibitive. So, when it comes to validating the schedulers like Kai scheduler, we can how we can access a GPU cluster with a multi multi-layer network My answer is It is It mean means Kubernetes without the Kubelet. Uh functionally, simulates the behavior of the Kubelet.

It allows developers to generate thousands of virtual nodes inside a kind cluster within seconds. These virtual nodes can be configured arbitrary resource specifications such as GPUs or memory regardless of the actual actual host hardware. When a pod is scheduled to a node immediately updates the pod status to running. But no actual containers are started. The mechanism operates entirely at the control plane level. Uh with cap at

least with this cap capability, we can label the virtual nodes with topology tire attributes such as spine or rack. So, we we learn deploy kind scheduler in the kind cluster and submit simulated inference workloads to these virtual nodes. Well, at least allows us to accurately verify validate the scheduling results. When we are developing and testing a custom topology builder for non-Nvidia network environments, the the builder needs

to actually communicate with switches at different network layer and construct a This means we must have real switch available. So, how do we obtain such a Kubernetes cluster? My My practical solution is Nvidia Air. Nvidia Air is a free SaaS platform to simulating network infrastructure of a data center. It allows It allows us to deploy Linux VM and the virtual Ethernet switches with Sonic OS or Cumulus

OS. So, we can build a multi-layer network topology exactly as we desire and deploy Kubernetes directly on top of it. We can deploy our custom topology builder on a cluster. It can communicate with the switch to collect topology information and the generate a hierarchical node labels. In addition, NVIDIA provides APIs to create infrastructure based on user-defined topology configurations. This makes it easy to to build a CI

pipelines for automatically testing. Okay. Uh Uh that's all of my presentation and thank you. Any questions?