Cloud Native Theater | KubeVirt Summit: KubeVirt on GB200: Vi... Fan Zhang, Kevin Klues & Alay Patel

In this talk, the speakers from Nvidia, Fan Fan Jang and Kevin Clues, discuss their work on enabling virtualization for GPU data centers and AI workloads using the GB200 architecture. They explain the challenges associated with creating a virtual machine (VM) that behaves like a single computer tree, focusing on key aspects such as memory backing, translation, and topology. The importance of maintaining coherence in system memory and GPU communication while ensuring that the virtualized environment accurately reflects the hardware's characteristics is emphasized. They introduce concepts like the extended GPU memory (EGM) and describe how they adapted existing cloud-native frameworks, particularly Kubernetes, to support multi-node NVLink communication fabric, enhancing workload scalability. They also address challenges in managing IMX demons within VMs compared to containers, proposing potential solutions for optimizing the user experience in Kubernetes environments.

Hello. Hi. Um, good morning. Thanks everyone coming here. Um, my name is Fan Fan Jang. I'm from Nvidia. I work for the cloud native virtualization for GPU data center and AI factory. >> Yeah. And I'm I'm Kevin Clues. Kevin Clues. I'm also from Nvidia. Um, and we also have Ali Patel with us today whom we're going to be swapping back a bit a bit back and forth

on stage here. But, um, yeah, happy to be here. >> Okay. So, I'm going to I'm going to start the first part of this talk. Uh, I'm going to share how we achieved the covert on the GB200 with a proper VM model. Um, so our goal was not just to boot a VM on GB200. Our goal was to make one covered VM behave like one computer tree.

To look at the system from the left to right, we started from the GP20072 rack, then zoom in to a single computer tree and then turn that tree into a tenant VM. A typical GDX uh GB200 computer tree is a two socket system. It has two grace arm CPUs. Each CPU is conducted with two black well GPUs by the MVLink C2C. MVLink C2C is a has a

900 GB biodirectional bandwidth at the same as MVLink on GB200. And what we wanted was very simple to say but not simple to build. One VM equals one computer tree except for passing through all the hosted devices. It has to make sure the guest see the CM NUMA and the devices locality reflecting the host. Why is that important? Because in the guest for the AI workload using

CUDA niko locality is not only about performance. It affects communication behaviors, routing decisions and overall stability for token outputs. VM boundary is not just about isolation but also about prediction. So the promise we were chasing was coover provide a deterministic VM boundary on GB200 for AI factory. The twist in this story is that grace virtualization is not just passing through a larger GPU devices. GP200 is a

coherent system and that changes the virtualization contract. There are three key key parts. The first one is translation on GP200 nested translation accelerated by the SMMAU is mandatory. The guest owns stage one, the host enforces stage two and M SMMEUv3 sits in the middle and the command Q helps reduce invalidation overhead on the acceler accelerated path. The system memory used by the command Q must be contiguous.

The device assignment path also relies on the OMU FD framework instead of the legacy group centric MU model. The second one is memory backing. GP200 use a cache coherent CPU toGPU interconnect. So the memory behavior matters more than on our CP PCIe only system. And this is where extended GPU memory EGM becomes important. EGM is a hardware feature on GP200. It allows the GPU directly to access

system memory over the inve C2C. Once EGM is enabled on the host, a dedicated EGM character devices will be created for each CPU socket representing a region of the host memory of the socket. This EGM region memory is reserved and hidden from the host and linearly mapped into the guest. In our virtualization flow, virtualizing the EGM will allow the VM memory to be backed by all the

EGM devices of the host along with the GPU pass through. So the whole guest memory is contagious and naturally supported the virtual command line Q. That said for the VM VGM matches a gra platform much better than huge pages. One important constraint need to call out here. VGM is only supported at the computer tree boundary which means you have to you have need to pass through all

GPUs of the sing of the same host into the single VM partially GPU assignment is not supported to using the VGP vGm. This system constraint effectively pushes us toward a computer tree side VM The third one is topology. On the legacy system, topology is often just a optimization. But on GB200, topology becomes a part of the correctness. On on typical PCIe only system, mirroring the host PCIe

topology to the guest is usually sufficient. But on coherent system like GB200, the numa constraints are nuanced. For example, a gas need to expose six uh auary v numa node to for each GPU that ensures uh all and and also need to ensure all the numa distance and the devices to numa affinity reflect the That means the guest ACPI and the numar relationship must be carefully constructed

not only the PCIe tree. So the takeaway on the grace typically GB200 virtualization correctness depends on whether translation memory backing and the topology all line up with the platform correctly to support the virtual grace virtualization features. Nvidia delivered customized liver cumo patches and all of the source code can be found in the Nvidia GitHub repos. Now we know why grace virtualization is different. Why expressing that expressing

that inside the cub is hard? Look at the graph on the left side. This is a bare metal virtualization typically type one. So that is a world where QVM runs in the kernel. Liber Q runs as a host managed services. The host owns almost everything like hypervisor memory lock cg group uh driver binding etc. In grace platform if the grace platform has a requirement you can usually

wire it directly into the hypervisor configuration. For example, the GP200 features patches on the librivver cumo can be directly pasted into the uh host kernel. On the right, this is a cloud native virtualization with cubert. You can see the the cumu and the liver are now running inside the pod to launch VM. That means there are more layers and correctness gates between the hardware and the VM.

All the constraints of the layers becomes part of the success path. Also, we have to handle the Nvidia customized liber package patches in a different way. Packaging them into the uh into the RPM files and wire them into kvert as a dependency for compiling and building images. The the lesson is kvert did not in invent these platform constraints. It exposed them across more layers. So the lay

the lower layer issues do not always show up as a hyper hypervisor errors. We can ser they can survey that the kubernetes failures p failures or covert failures. And that that is why Kver um enablement was not just about teaching the hypervisor new features of grid spec about the grid specific. It also about teaching the whole cloud native stack to express the right GB200 platform So, so

our enablement work can be understood as four contrasts we had to satisfy one after another and when I say contract I simply mean it's a a set of the conditions one layer expected to be true uh before we move to the next layer the first one is a boot before we can talk about performance or topology in the gas the VM must be able to boot reliably

with four GPUs and highspeed IO's attached on TP200 converter must treat 64bit MMIO aperture as the first class platform parameter with every large PCIe bar mapping default assumption that work on legacy platform is in sufficient so the VM address space need to be modeled correctly ly and configured explicitly to support a large resizable bar of the black whale GPUs. The second one is container. The cubert in

the cubert cumul is no longer a host process. It is a part of the container. So that means uh memory log behavior and process cap capabilities matter in the way they often do not in the bare metal with VOM UFD the memory lock accounting applies to both VM RAM and decides devices bar mappings. So missing the cap IPC log can cause the bar mapping failure. So here

the container runtime policy become part of the virtualization correctness in kvert. The third is the kubernetes we are not we are not in the cumul configuration alone we are dealing with cloud native stack including scheduling cublet admission topology hints and device allocation etc. The Kubernetes cap described the fitter model we hit on GB200. The system can expose 34 NUMA nodes. Only two of them are CPU numa.

Others are HBM related NUMA from the GPU. That's because GP200 firm enumerates the eight additional numas nodes per the That sounds very uh very same to the guest numa constraint, right? That that's that's another reason why the guest must keep 101 mapping to the uh computer tree host. Topology manager can get stuck when calculating the topology hints by exploring combination across a large bit mask. The time

complexity is 2 to the 34. So in that state the cublet suffered the CPU burning and hunging never reaching the allocated core. So the port start stays pending and the VM never starts. This cap tracks our improvement efforts and the PRs to address its limitation at this layer. Um we um we have also needed to handle the we have also have supported GPU uh grace vfio driver

binding by the GPU operator and the GPU allocation EGM device mount by the curver GPU device plugin at this time. Um we were going to uh implement that in the DRA driver in the future um in the cloud native way for the cubert. The last contract is a grace platform parity. Even after the boot the container runtime community everything works well the guest started the guest still

need to reflect the real or grace platform semantics in coververt user user has no control to construct a combination of the correct parameter to the cumu so we designed a numar planner in the curver in tree codebase to align the complicated constraint for grace virtualiz ization. That means hubert can support a right memory backing with VGM, the right invalidation path with V command Q and the right

NUMA and the devices topology including the NUMA distance and the PIE bus bandwidth were uh where the network devices attach to is overrided to the maximum capacity for the RDMA. So when the CUDA driver is loaded in the guest it see the same semantics of the TB200 computer tree mode. So um we have achieve so right now we have achieved the single single um single node uh

pattern for implement the quer VM on the single GB200 computer Let's see how the guest CUDA driver maps out the data path of the GPU and the CPU for the VM. So inside the VM GPU to CPU access is consistent by the MV link C2C whatever the CPU is on the same socket or not. The between the VMs of the different computer tree of the same rack

the GPU to GPU traffic will go over MV link and the GPU toGPU is over MLink plus C uh C2C. So all GPU and the CPU data traffic are running on the highway because MLink and the C2C have much more bandwidth than the PCIe and the network fabric. So at this point we have solved the single node story but GB200 is a rack scale system and that

brings us to the next challenge. Once communication and semantics extended the single node the single computer tree how do we all treat them correctly in the right scale. I will be very pleased to hand off to Kevin about this point. So H will in introduce Nvidia amazing solutions to address this challenge. >> Thanks Fan. Um yeah, so as Fan mentioned, you know, everything he talked about up

until now has been how to enable a VM on a single compute tray in one of these larger GB200 systems. Um but as we know, you know, the scale of modern AI has outgrown the single node. And the bottleneck is no longer the number of GPUs you get inside that pod or that VM. it's this node boundary and how those GPUs are able to communicate across that

node boundary for for much faster communication. Um, and the solution that Nvidia has for this at the hardware level uh is this technology called multiode and vlink. Um, and uh the team I'm work uh that I work on built uh kubernetes abstractions in in order to allow us to to use this multiode and vlink um communication fabric. Um, and what I'm going to talk about today, uh,

to start off with is what we did in standard Kubernetes for being able to use multi vlink inside standalone pods. And then Ali's going to going to wrap it up with, okay, well, given what we did for normal pods, what is actually what do you have how do you have to change that mechanism to work inside of VMs? Because what we've done for normal pods doesn't quite

work exactly the way we would in that environment. Um so before we get into that uh the first thing we need to kind of understand is how do you partition these multi-node and vlink systems and this is the terminology for uh how you describe the way that this partitioning can happen. Um at the top level you have this thing called an invinc domain. That's the set of

physically connected GPUs over in Vlink. So in this picture we've got uh on the on the two outsides we've got what we call the compute trays. There's 18 compute trays with four GPUs in them each and in the middle we've got nine inv switch trays which allow us to have this full uh full uh GPU to GPU uh bandwidth across all the GPUs in the entire mesh

and this invinc domain kind of encapsulates all of that and says these are the physical physically connected set of GPUs that could potentially talk over in Vlink and in the case of GB200 that's one rack one rack is one in Vlink domain um but within inside an inv domain you can partition that into what's called an inv partition partition and this is something you can set up

on at the NV switch level. So you have to have access to the NV switches. You log into them. You say, "Okay, I want to partition off this set of nodes with these GPUs to only be able to talk over the NVLink fabric to the nodes in this other subset. Um, and then within one of those NVLink partitions, you can run some software level partitioning in the

form of what's called IMX domains and IMX channels. Um, and without going into the details of what that is, the only things that are really controllable from with inside Kubernetes are setting up these software construct of IMAX domains and then making sure that whatever application you're running has access to this construct called an IMX channel to allow them to communicate uh with with GPUs on other nodes.

Um, so the Kubernetes abstraction that we built to kind of express all of this and and take advantage of these multi connections is this thing called a compute domain. and we implemented it as a as a DRRA driver in in in Kubernetes. And the goal of it really is to hide the complexity. So from a user's perspective, you declare one of these compute domain objects as a

Kubernetes CRD. You point your workload pods at it and all of the workload pods then come up with multi wheeling capability to all of their peers. So from an end users perspective, it's very straightforward, but there's a lot of magic obviously going on behind the scenes to to allow that to happen. Um, one thing we wanted to make sure of is that we didn't want to impose

any kind of artificial scheduling constraints inside the compute domain construct itself. So workload placement is still handled entirely by standard Kubernetes primitives. You set a pod affinity, you set node selectors, you do what you normally would to schedule your workloads on a node, and then all the things that we need to do to set up these IMAX domains and get these multi-node and VLink connections kind of

brokered amongst them uh follows that to the nodes where your workloads land. So you don't have to change the way you normally would schedule these things. You just know that whatever constraints you always would have put on them, you do the same thing and the multin connections will just kind of magically pop up for you because you've pointed them at this compute domain object. Uh yeah, and

the DRA driver does all the heavy lifting for that. So it watches workload pods land. It sets up these constructs and it makes sure that your workloads have access to a common IMX channel so that they can uh communicate. Um so again, yeah, from a user perspective, this is what it looks like. You create one of these compute domain objects. You have this field in there called

the resource claim template name. You add a section in, you know, in this case I'm using a deployment of of 18 replicas. The podspec template inside there points at this resource claim template name and that basically advertises that I want all of these pods to have membership inside this compute domain so that they can get have those multiode and vlink connections uh set up for them. Um,

and yeah, like I mentioned, you can use standard Kubernetes mechanisms to to get to influence the scheduling of these. So, you know, an example here, you could have plopped this into that uh into that deployment to say that I want all of the pods associated with this deployment to land on a node that have the same topology key. Um, the click name at the end might be

a little confusing, but that's just another name for what we call the NVLink partition. So you might have set up an inv partition. You want to make sure that all of them land on that same one so that they actually will have the ability to talk over multilink once they once they land there. Um and so yeah, this is the actual process that you that you go

through step by step. So when a workload is linked to a compute domain, the first thing that'll happen is the DRA driver will deploy an IMX demon onto each of the nodes where those workload pods land. Uh and then it blocks each worker fra from actually starting until its local IMX demon comes online. So if you look at the graph that's there, you know, the two nodes

on the right have had their worker pods started. These IMEX demons have been deployed there. They're marked as ready. And so the workloads were able to also move into the running state. Uh but the one on the left, the worker pods still blocked from demon reaches the ready state. Um all of these demons are configured to form an ensemble with each other so that they know who

their peers are across the the different worker nodes. And once any one of these individual IMX demons has been started, marked as ready, the DNA driver will inject an IMX demon into them uh and unblock them from running. And eventually all workers come online. They start doing their thing. And this whole setup stays alive with these IMX demons discovering each other, adjusting to workload scaleups, node failures,

whatever you whatever you you know normally might have inside a cluster until the workload completes. At which point the worker pods then you know exit normally. IMX demon pods are cleaned up and now these nodes are ready to support a new compute domain in the future if you land some workload pods there later. Um, and to kind of segue into to Allay's portion of the talk, the

one thing about all of this that doesn't really work well in in a Cubert environment is how you configure these IMX demons to form an ensemble with their other worker nodes. We do some tricks that allow us to do this fairly easily in a uh in a pod environment, but for VMs, you have to to do something different. Um, and at the end of the day, it's

all about having these this file called a nodes config file set up properly um to uh have all of the different IMX teams that come on know about each other and communicate with them. And we run some tricks where we put static DNS names in these files and then we overwrite the local Etsy hosts file in each pod to make sure that it can discover all of

them uh without having to actually know the IP address uh a priori. And so again, I'm not going to go into the details of this, but it's this nodes config file and this kind of interception of the Etsy host file and having us to do this dynamic update of it that doesn't really work well uh inside VMs. So, >> so in the next set of slides I'll

be talking about what exactly are the challenges with making this ensemble work for cubeword VMs and what are some of the ways in which uh we can solve it with uh cube native constructs and then forward-looking ideas on uh making it easier for for the users. So the biggest difference with uh cubert VMs is that assuming that one node u has one VM which has all the

GPUs passed through um to the VM. It means that GPU devices are actually running inside the guest VM which is that white box because the guest VM is isolated from the host. the IMAX demon needs to run on that guest VM. So it is that orange box. This is different from how it happens with containers and pods. In containers, you can sh you can have the IMX

demon run on the node and it can be shared between the pod and the uh uh IMXDmon demon set part. Right? So, so in this case because the IMXD demon is isolated inside the guest VM, it creates a set of new challenges. The challenge is that the guest VM does not have access to the API server. Because it does not have access to API server, it cannot

communicate its identity outside this isolated environment. because it cannot communicate the identity outside the isolated environment. It is hard to coordinate the different IMX and their IP address or DNS names uh to form the ensemble. So something needs something needs to be uh uh plumbed into the IMX demon in such a way that the ensemble is formed without access or coordination. because the coordination is cut off.

It also means that the magic that compute domain does to uh dynamically map the D static DNS name to the pod IP is not possible with VMs. This is because the C host file also needs to be isolated inside the guest VM. and writing the SC host file from a controller that is outside on the node. Uh it is challenging. The challenge is that sometimes the guest

VMs have their own networking configuration. They can overwrite uh HC host files or in some cases you don't even have access to write them uh write the host or update the host uh file directly. So these are the uh three sets of In order to solve the use case for IMX demons, before going on to the solution, let's see what the IMX demon needs. The first thing

it needs is this static config file. This config file has all the DNS names that each IMXDmon can use to talk to the other IMX demon. So in the diagram you can see we have the static config file but currently these demons don't know how to talk to each other. In order for us to in order for the demon to find the other demons in the uh

ansemble there needs to be a mechanism to dynamically change the DNS names that are present in the node config file and map it to the uh IP address of the VM that comes up. So, so there is two distinct need. One is to have the nodes config and then second is to have dynamic DNS uh mappings that can map to the IP address. So the ensemble can

be formed. Let's go step by step on how it can be done today in a a manual manner. So the VM can be configured with this nodes config file. This node config file has all the DNS names and you can see they are in order VM1, VM2, VM3 up to 18. This covers all the 18 nodes in in a rack. Now we create a a Kubernetes headless

service. This service has one job which is when the VM comes up the the name or the DNS name that is present in the node config file is dynamically mapped to the pod IP or the VM IP. In this case, we assume that the pod IP is equal to the VM IP because we are using a pod networking on that is plumbed into the Here you can

see the red box. It gives an index based approach. So the nodes config file is statically rendered with VMs. And then on the on the right when the VM comes up uh the green box is the IP address that gets populated by the headless uh service because now because of the uh headless service we have this dynamic mapping when the VMs come up with the nodes config

file each demon is able to communicate with the IMX demon on the on the other uh VMs in the ensemble and have the networking configuration needed for the IMX demon. You can see in the diagram that the VM needs to be configured with certain cloud init configuration so that the IMX demon automatically starts when the uh when the VM boots up and once it does that because

of the headless service all the VMs form a a mesh where they can talk to one another talk. So there are two interesting aspects in this configuration. One is scale up and scale down. Because of the headless service, as soon as a new VM comes up and tries to join join this uh ensemble, the headless service again uh adds the dynamic uh name to service name to

IP address mapping. And this uh VM was already preconfigured in the nodes config file. So, IMAX demons are all happy and they can accept this new VM and and scale up is achieved. The second uh po the second uh part of this is scale down. So, in the diagram you can see that VM 5 no longer wants to be part of this uh configuration. Maybe something happened

and the VM crashed, right? So, in that case the VM will just uh go away. the headless service again will remove that uh IP address from from the uh configuration and because the IMX demons are configured in a way that they can work even if the DNS resolution is gone the new uh ensemble will only have four four VMs now. So it can be it is uh

evident that the current uh mechanism for VMs is much more uh manual compared to what we have with the pods and the compute domain construct. Hence we are working to make uh make user experience better. We it would be amazing to have some kind of uh abstraction like compute domain or maybe an integration with a compute domain so that all of this manual manual uh work that

admin needs to do now can be hidden away behind the abstraction. The challenges that we face is that in building that exception part of the configuration has to be inside the VM and the other part has to be outside in the Kubernetes environment. So this is something we are actively working on right now and we have some plans on integration of of making this user workflow uh

better. Yeah. Uh so that's it. Uh thank you so much for attending and we'll be around if anyone has questions.