Volcano: Orchestrating the Full AI Lifecycle – From Training To Inferen... Chen Zicong & Hajnal Máté

This talk focuses on Volcano, a unified scheduling platform that orchestrates the entire AI life cycle, from training to inference. The speakers, Matty Hel and Jesse, share insights into Volcano's evolution from a simple batch scheduler to a robust system that supports diverse workloads, including TensorFlow, MPI, and Spark jobs. They discuss key features such as multi-cluster scheduling, workload collocation, and innovative scheduling policies designed for latency-sensitive applications. The presentation highlights new developments like the 'agentuler' for managing bursty workloads, as well as integrations with other tools, emphasizing the importance of adaptability in the rapidly changing AI landscape. Jesse introduces complementary projects within the Volcano community, such as Kina for LLM serving and Agent Cube for AI agent workloads. Together, these advancements position Volcano as a critical technology in managing AI workloads efficiently across various environments.

Hello. Hello everyone. So um let's uh let's get started. Thanks for joining us today on the on the maintainer tech uh maintainers track. Uh we will speak about volcano um how it's orchestrating the full uh AI life cycle. Uh our focus here won't be just one product in isolation. It is a the full AI life cycle from training to inference and uh and highly bursty agents in

the end. But uh before we start uh a quick uh speaker introduction uh let us introduce ourselves. My colleague here is uh Chong. >> Yeah. >> You can call me Jesse. Yeah. >> Yeah. I will call him Jesse too because that's what that's his GitHub handle. Um he's an R&D engineer and volcano maintainer from Huavi Technologies. Um and I'm myself uh is Matty Hel a senior machine

learnings engineer at Movio and a core working a contributor recently uh we have worked worked together on a lot of features um let's move uh to the agenda what we will have today um I will cover uh the big picture uh the volcano evolution uh we will start with that that how we evolved uh evolved a unified batch scheduling system uh scheduling platform from just being a

simple simple uh batch scheduler. After that we will jump uh into the scalable and efficient foundation we tried to lay down in volcano in general in uh into the past few years and uh uh we will extend that knowledge uh with the the topology of our uh section. Uh so I will cover these parts and uh the next uh part will be about kina an enterprisegrade uh

llm serving platform um and the agent cube a cloudnative agent and code inter interpreter platform and volcano global the component that's capable of multicluster federation and these uh these these will be covered by Jesse so uh volcano as I said earlier evolved from uh The cube batch project uh it was developed throughout the years uh from being just a standard batch system to be a greater unified

scheduling platform. Um just a little history uh it became part of the CNCF volcan the CNCF volcano became part of uh the CNCF ecosystem in uh 2020 and uh two years after it's reached a maturity level of being an incubating project uh in the CNCF foundation and in 2026 this year we really want to achieve a graduated status. Um so uh some listeners may not be familiar

with the volcano in general but uh uh let's just uh go through a little bit that what it's capable of doing. So uh it's a unified scheduling platform. It's it's it it supports just a wide variety of workloads. uh we can we are integrated with TensorFlow uh MPI jobs uh Spark jobs uh and a lot of uh community contributions were were uh were implemented in working already.

It has a queue management um system which uh which is able to handle multi-level cues uh in a in a in a Kubernetes cluster and uh also uh it has a really unique feature which is called workloads collocation. Um we I will actually dive a little bit deeper into that. We are able to uh separate online and offline workloads on the cluster uh and schedule them pretty

efficiently. Uh also multicluster scheduling with the help of volcano global uh we support a wide variety of heterogeneous devices. Uh GPUs, MPUs at etc. So uh it's not just uh concentrated on GPUs itself. we we can support MPUs and TPUs and things like that. Um and it has insanely a lot of uh scheduling policies. Beam packet network topology aware scheduling gang aware scheduling uh numa aware scheduling

device share scheduling topology aware scheduling task topology scheduling. So uh we we really have a lot of um task there and uh and capabilities there. So why now and uh and what's what's now? AI is drifting really fast, diverging the workloads really fast in this ecosystem. A few years ago um we had uh batch training um uh workloads mainly dominated on the Kubernetes ecosystem um which we

were actually uh handling and uh recently in 2023 uh we started seeing these massive LLM distributed training workloads on the clusters and recently we evolved one one step away with the with these disagregated LLM uh patterns where you need to run the decode and the prefill for part of of these LLM workloads inference workloads in the cluster with really different runtime patterns and now also with that

the ca the agentic workloads uh came into the picture which are short-lived latency sensitive and extremely bursty um yeah so these these uh these different characteristics uh bring new challenges and and uh we can't really uh optimize both for uh latency and throughput. It's it's a really really hard challenge and and to tackle these challenges we we need to evolve fast. So um that's just a brief

introduction about the technical road map of volcano. Um as you can see from the top level uh we don't we have right now we will have not just volcano jobs in general we will have hyper jobs which are capable of multi defining multicluster uh jobs with the help of corm and volcano global and also on this application layer we have kina which is really really new project

Jesse will talk about that agent cube um for code interpreter and and agent runtime time workloads uh and uh and these are uh these are all in the application layers uh coexisting in this in this ecosystem. So uh beneath that we have the the traditional volcano um core scheduling features and below that we have the heterogeneous devices uh which uh which pulko handles via unified heterogeneous device

allocation pool and uh yeah it supports as I said a lot of uh scheduling features like the network topology of our scheduling to improve AI workload performance in general. Okay, let's uh let's bring us to the uh scalable and efficient foundation. Uh we realized that we can't really um tackle these problems in the in the same. So that's why uh the recent volcano release the v.4 14

has introduced the the agentuler and uh the volcano existing batch scheduler has has a really outstanding performance in in gang scheduling but uh but it has a session based mechanism. It uh it schedules the pods every second by default and we cannot really meet uh the bursty uh latency sensitive workloads uh that are new in this ecosystem the agentic workloads. Um and we decided okay that uh

we need to have a dedicated fast scheduling path for these uh for these type of workloads. And this is uh uh which was um implemented in a man manner to serve these serverless like traffic in a multiworker parallel scheduling style. As you can see on the right side of the p picture, you can see that uh the schedule worker, the agentuler has more workers with with snapshots

about the cluster. So it's it's able to um uh to to schedule and uh and tackle latency sensitive applications like the like the agentic applications. But we and and it in the design we had a lot of uh thought in place optimistic concurrency conflict aware binding and uh an optimized scheduling queue and the the best part is that it's seamlessly coordinating with the existing volcano B scheduling

system. It's not like uh there is no coordination and we will see that how that happens. Uh and but we don't really wanted to throw out the existing patterns which the batch scheduling system uh provided for us. So we basically just reuse the the old code we had had in the volcano batchuler and we we still provide uh the predicate plugin the node order plugin the the

beam pack plugin uh which were working really well in the in the batch scheduling uh part of volcano. Okay, but uh how do we shard the cluster uh from these two types of scheduulers and here it is the uh we needed to introduce a new sharding mechanism uh which we which will be coordinated by this component called uh sharding manager and uh in general the sharding controller.

What it will uh do in the end is that it will uh provide uh uh dynamically uh controllable manner to calculate note pools on the cluster uh with uh with extendable policies. So um rather than an inefficient static partitioning for example when you have when you can when you split the cluster between uh with with node labels or something like that. This is a dynamic mechanism and

we are not just uh partitioning the cluster uh horizontally but vertically. We are partitioning them uh horizontally too. And uh this will give us all the leverage to uh do uh resource from stupensation and scalability and the workload isolation in general. And uh here is the uh the actual CRD that helps us in it the node note note chart CRD and this will be the the contract

between the the schedulers um uh about the coordinations. uh it will dynamically candidate node pools for each each scheduleuler and uh the the best part is that it's pluggable uh it has pluggable sharding strategies. So you can say that okay for this particular scheduleuler uh you want the you want to configure it that okay for CPU utilization we want to aim for an 80% CPU utilization for

the batch scheduler but for the agenticuler it can be it can be used by the agentic scheduler above that threshold. So these two types of workload can coexist on a on the nodes themselves with with two different schedulers and there is no uh congestion and no uh and and and no conflict uh between the two twoulers when the when they are using their node charts. Okay. And

uh the next uh good new feature is the walk in a collocation uh which lets us to to define uh online and and offline workloads on the cluster. Uh our online workloads uh have three uh separate uh really comprehensive quality of service model. We have latency critical high latency sensitive and latency sensitive workloads that are running on the cluster u basically 247 if you want. But for

offline typed workloads, the best effort workloads, you can uh you can run them in off peak hours. And uh the other thing is that uh you can actually configure with with cloud native collocation uh dynamic resource over subscription on the on the nodes themselves. So you can say that about 30% uh of the nodes can be still used uh by the by the offline offline workloads if

the online workloads are not using them. And this is happening with the with the volcano agent. is SLO agent um which can be seen at the bottom of the of the picture um that it calculates that how much the the CPU or the memory is utilized by the uh by the pods on that node and the SLO agent will decide that okay even though that pod requested

CPU resources uh but it not doesn't use use them maybe we can uh schedule these offline workloads to that that node and they can utilize these these fragmented resources on the cluster and uh this basically just improves the the cluster utilization by a lot. Um and also uh CPU burst is is a thing like that um you can add an added annotation to your pod and it

will uh it will let your pod to go just for a specific amount of time above if it's its CPU uh limit. And this is it's really uh important because j because just because your uh workload actually uh doesn't consume more resources uh than its limits on average that doesn't mean that it's not throttled by by the Linux kernels fast fairuler it it will still maybe for

a specific amount of times would like to spike a little bit above that port limit and with CPU burst you can actually uh control that spike. a little bit so your workload can get a little bit more resources even though it is it's for a really tiny percentage of the time it's above its limit. Yes. And uh yeah we did a lot of work uh in the

cola volcanic collocation the collocation configuration became a specific CRD and uh we adapted the full croup v2 integration uh into volcano and and for now it's not uh not specific to uh uh to to to the op operating system too. So this is really cool. Uh use this if you if you can and I will just jump into the network topology aware scheduling and what's new. So

Volcano in the in the way.12 release uh had a concept called hyper node based topology aware scheduling um and it it abstracts away the your network topology into these hyper node levels. You can see at the below we can have leaf hyper nodes this s0, s1, s2, s3 and these leaf hyper nodes uh are containing a set of set of nodes and uh also you can have

above them these non-leaf hyper nodes uh in this tree structure and uh this represents your network topology on the cluster. We implemented label based hyper node auto discovery and recently we added richer subgroup level constraints and uh and gang coordination and one thing which is really uh unique to volcano is that uh we have hyper node level beam packing uh now so you can actually beam pack

based on network uh this is an example in a in an LLM u aggregated service you can see that your prefield part will be scheduled on a on a leaf leaf topology and your decode part uh of your LLM serving workload will be scheduled on more nodes on on tier 2 on a on on a higher network topology. Um yes that's it. We have uh hypernode pluggable

auto discovery which means that uh both for heterogeneous devices like Nvidia and Dustin uh which has uh MVL link fabric manager API uh infinib bander um uh shared manager in place or rocky lldp the link uh layer discovery protocol. We can actually translate uh these kind of resources with u with the UFM, the unified fabric manager discoverer, the node labels discoverer and the rocky discoverer to actual

hyper nodes on the cluster automatically. You don't need to manage and uh yeah so we we still have some work to do. What's next? We we want to collaborate with the DRA uh much more as uh as as the uh volcano hyper node is actually extending the support for node members but for multi rail architectures uh where GPUs are directly connected to envelinking and envisage we don't

really have um a saying in that uh the dynamic resource allocation of kubernetes comes really handy in this microscopic PCIe MV link alignment uh the inter topology handling but intra inter node topology handling and hyper node provides the microscopic topology handings and the network boundary the the intra node um yeah we need to work more um and integrate other other type of u network protocols like the

rocky rocky v2 uh and the open source providers like the nvidia topograph and dcim um and also uh let's not forget our are uh most good clouds we need to integrate with them much more. Uh so that's why volcano uh calls for contribution. So we want to promote hyperno to be a kus for class citizen. Uh we really think that the the hypernode is the right abstraction

layer uh for the for the top network topology of our handling. uh every other uh scheduler like the KO scheduler, the Quler, coordinator scheduler. Um these are handling network topologies based on node labels and uh it's really cumbersome for administrators to to handle all all these lab labels and it's just not not the right abstraction. So we are we want to co-create the hyper node in initiative

with the CNCF committee um uh to jointly define the standards uh what hyper node should be. Let me give you give it to test. Yeah. Nice to meet you guys. Uh I'm from the Huawei technology. You can call me also Jesse. uh I would like to introduce your uh the three sub project of our volcano community uh and what's the what's the latest updates in in this

sub project. So the first one is called the casino uh casina is our uh kubernetes native a serving platform. Uh it has uh uh supports lots of serving platform. Uh it supports V like V and SG long uh and it can do auto set scaling based on the uh token swer and has a intelligent uh data plan uh can do the uh KV cache routting uh per

cache PD group aware routting and and other we can uh cooperate in uh natively with the volcano because the uh volcano has support the uh GAN scheduling and network topology aware scheduling. So the volcano can natively uh integrate with the casino. Uh and the casino has a a native perfume and decode disagregation. Uh uh it's a the the schedule union code serving group can do uh xp

yd deployment. Uh the the p per because the perfu and decode has a different resource requirement. uh at the prefuel can do uh needs more computer resource but the decode needs a more uh memory bandwidth. So the uh we we can uh build it as a uh schedule union to uh automatically scale the decode and per uh per instance separately. So uh the CA casina natively uh

support the per and decode disegregation and the casino routter is intelligent. It's a a native uh routting router. It it has a lot lots of uh load balancing strategies not just a simple roundroing uh load balancer. uh it can do uh per cache uh low KV cache aware lower aware and uh it's a tok uh token level traffic management like the uh rate red limiting uh and

it's native perfield uh disagreing routting uh router uh and it's other uh sub project we uh released last year it's called the agent coup uh it's a kubernetes native platform for agent AI agent workloads. Uh from the left left left picture uh it's a status core of the uh Kubernetes system. uh but uh for the AI AI agent workflows the AI agent workflows uh needs to reuse

session uh it's a bursted uh it's you needs to re review some status so it's not made suit for enough for the uh AI agent workloads so uh the kubernetes also built the six uh sandbox uh in the last year so we also uh build top of the AI AI agent sandbox box uh in the last year to build a agent cube project. Uh agent pro agent

cube pro project can uh reuse the warm pore uh can do the uh auto can provide auto out of box SDK COI uh and and it's uh can do can integrate more uh like the define frameworks more AI agent uh frameworks. So the AI agent uh cube builds on the uh 6M box today but we will introduce more uh native uh preempties and border integrations with other

uh frameworks. Uh it's a session aware uh routting and life cycle management. Uh it can reuse warm pool to uh reduce uh the fast shot starting sandbox. Uh it it's uh can suitable for the AI agent workloads. uh and the agent cube also provide out ofbox uh Python SDK uh for example here's a uh picture show that uh you can use the uh Python SDK uh to

create a uh a session uh and then and to reuse in the in in the next run uh to uh send the prompt to the uh same sandbox to reuse it. So you can uh just get uh get check check of off of of our uh starting docs to to use it. Uh so what's new in the volcano global? Volcano global is our uh multi multicluster scheduling

uh platform. It's built on uh kamada and can do queue management based on the on top of the uh multicluster. So uh we also updates the uh two key features in the last year. One one of these uh hyper job it can do uh multicluster uh job split split splitting for the uh large scale AI training and and the other is the data dependency aware scheduling can

do the uh data aware placement for uh compute storage distression. So why do we need hyper job? Because uh are getting uh large larger uh and hyper job can automatically split your uh messy training job across the multiple uh hogenous clusters while giving you a single uh unified uh entire unified view of the entire process. Uh and here's the some uh two example of hubber job use

case. You can do uh large scale uh training job to uh across the multicluster and it support the hedrogenia cluster. The the picture like kind of small but uh what I want to show you that it can do also the hogrogenia cluster uh training. So another is the data dependency uh aware scheduling. what what it want to do is that uh the uh the the compute and

the storage disagregation uh com computing is is become more uh uh famous. So uh the bottle is that the bottleneck is that the cross region data access uh in the disagreeing architecture cause mouse uh massive IO latency and the uh bandwidth cost. Uh so we have uh developed this uh data uh dependency aware scheduling uh to allow you uh to uh query the metadata of data links

uh and to schedule uh schedule affinity through the data dependency. So uh thank you guys for uh listening and uh here we're call for a contribution for the volcano community. And do we have any uh question want to ask? >> Yes. Go ahead. >> So, so what we uh figured out that um you can always partition statically if you set the node boundaries thisuler will work on

this node and this scheduleuler will work on this this node. That's that's the the um vertical uh uh splitting. But uh but we can do uh horizontal splitting which means that for CPU utilization uh above for example 80% one scheduler will uh uh schedule the pots uh and it's actual utilization and below 80% the other will will do the the utilization and it's pluggable it's configurable. If

you uh check check this slide you can see that uh for eachulers there's a config. One of them is the agentuler and the other one is the volcanouler. And this is working really really well together because uh bash training jobs has these re these fragmented resources on the on the cluster which which aren't really utilized by other ports but actually agentic agentic workloads are really good to

be run with them uh by default. So above just just this above this threshold uh the agentuler will schedule the uh the agentic workloads uh on on the clust on on the node itself and below that the actual volcano bestuler will schedule and uh there is this contract the node shard CRD which uh which actually shares these po these nodes between the two schedulers. anything I said.

>> Did I answer your question? Did Did anyone also have also Okay. Subar. Okay. Okay. Yes. Sorry, what there is what is the boundary of volcano? He's um he's asking is that what's the connection between Q and and volcano? Um I will let you >> Yeah. Yeah. The queue uh also did did some uh the same uh features in uh with volcano but uh volcano uh born

uh very early because it it originated from the uh cubic batch uh and and and has a different uh framework with uh cubisuler can do native uh GAN scheduling and we we have uh uh do the network topology we scheduling uh earlier uh and we had uh Q man Q management uh define it earlier. So we have already uh a lots of users uh users uh integrate

in the volcano ecosystem and we wish to uh beyond the volcano not just for the uh batch scheduler. We want to do a unified scheduling platform uh to not just for the uh batch batch scheduler uh but also want to do the AI inference AI agent workloads just just we uh sold in the in the last of the slides we we have built some sub project uh

for the volcano community to to build as a unified scheduling platform. Yeah, we we also we want to uh we can see the from the road map of the volcano uh we we we will not just support the uh training inference we will also support like the re reinforcement learning uh and we can do the hogenous device pour like integrate some uh cooperators with like the hammy

project to do like the GPU slice shutting uh the and and other uh hedrogenia devices devices. So, uh yeah, yeah, we do have some overlaps, but but the volcano uh has some uh great in performances and we'll be beyond more. Yeah. Anything else? Super. Okay. Thank you for joining us today. Thank you so much.