Cloud Native Theater | KubeVirt Summit: Breaking the Performance Barrier... Jian Li & Yves Weisser

This session delves into the challenging aspects of virtualizing high-performance AI storage using KubeVirt. The speakers, Gel from SK Telecom and Eve from NetApp, discuss their experiences and findings from their work with AI data centers where GPUs play a central role. The presentation explores how to effectively virtualize AI storage and fabric, emphasizing the importance of high IOPS and bandwidth for various steps in the AI data pipeline, such as data preprocessing and training. They highlight the use of Kubernetes' CNAP project, specifically the Petasense AI Cloud solution, and the integration of RDMA technology to enhance performance. The talk also includes real-world performance evaluation results demonstrating the efficacy of their approaches, particularly in the context of GPU Direct Storage (GDS) within virtual environments, ultimately showcasing how they manage to achieve near-native performance even on virtual machines.

Okay, good morning. Maybe it's a noon. I'm So, I'm Gel from SK Telecom and today I will have a presentation with my um coworker Eve from NetApp. And today I we would like to present a talk which is about the breaking the performance barrier, high performance AI storage virtualization with the KubeVirt. And very brief introduction to myself and I'm currently working as a principal engineer at SK

Telecom and I've been working on the AI data center like the infrastructure virtualization with the KubeVirt for the last 2 or 3 years. Yeah, and here I'm good morning everyone and I'm a solution architect at NetApp and yeah, well, I spend my days working and playing around with Kubernetes and KubeVirt. So, yeah, that's a fun. Okay. so this is today's agenda. First of all, we'll briefly introduce

what we have done so far on virtualizing the AI AI storage specifically. And then deep dive into how how we virtualize AI AI storage and the fabric specifically using the KubeVirt and how we integrate that the whole solution using the KubeVirt. And also we we're going to show the real world performance evaluation results on the virtualized AI storage and fabric. And finally just showing to show some

key key takeaways and the conclusions. So, when you start to look into the AI infrastructure recently, unlike the traditional data center, it's a full of a GPU. So, the in traditional data center, the the key player is a CPU, but the way we move to the AI data center the key player become changed to the GPU. So, we have to deal with all the GPU fabric and

the GPU virtualization and all as well as a GPU interconnect as well as also we need the AI storage virtualization, the AI fabric virtualization as well. So, uh those are a bit full of uh like complexity, but today we will be more focusing on talking about the AI fabric and the AI AI storage fabric and the AI storage virtualization specifically. If you feel feel interested about some

other part like NVLink virtualization GPU fabric virtualization, you can attend the AI ML track talk at 2:30 today. They will be talk more about that side as well. So, actually, so in order to virtualize the whole stack, we fully leverages the Kubernetes CNAP project including the Kubernetes and the KubeVirt. And that works as a core engine as our solution. Our engine our solution called the Petasense AI

Cloud, which is a SK Telecom in-house built cloud solution specifically target to virtualizing the AI infrastructure. And we got the several award from the from the global like GSMA. Um and uh since since we our solution based on Kubernetes and the KubeVirt, we can run both the virtual machine and the container and the even the managed Kubernetes on top of the the two frameworks. And on top

of that, run directly run a set of services like AI ML IML Ops and the some other cloud native DevOps stack and the and so on. So, those are just very brief like architecture what we are using to infrastructure. So, when we talk to the AI storage virtualization overview and actually, there are the AI data pipeline is a bit kind of long and in each stage it

require different like a characteristic. So, for example, for data inject, it require very high capacity storage. Actually, for this stage, the performance is not the issue. It's more or less it require very large volume size. But when we when you move to the next step, which is data pre-processing, it typically require data normalization and the data deduplication and so on. And the from this stage, you you're

going to require a lot of like IO perform IO intensive like workload going to run on top of that. After that, the preprocessed data will be become a input of the training and then the train data will become a error model like a weight model. Then that going to be the input of the inference. And then eventually you're going to archive that kind of data and and

so on. So, the three steps in the middle require high performance. And the for that area, you somehow need AI storage specifically, which require high bandwidth as well high IOPS and so on. So, in order to realize that, there are a bunch of like a software stack to support it and support that. And today I'm going to mostly focusing on talking about the network stack mostly. So,

it's heavily rely on RDMA software stack. So, RDMA is a remote data the remote direct memory access technology, which you can by using that technology you can bypass the entire TCP/IP kernel stack and you can interact with the outside of the the storage using it. And the so far the InfiniBand and the Rocky v2 are primarily used in order to realize this. So, there are several way

to interact to realize AI storage virtualization. And I just list up three of them. The left-hand side one is a CSI based NVMe over TCP. It's a typically just block type of storage, but if you integrate that with the QEMU, you have to go through the whole like a software stack. So, your guest like a kernel uh send some kind of you know, messages to the outside

want to send a message to the outside of the stack, then it has to go through the QEMU like hypervisor stack. And then inside that there is a virtual block device and you know QEMU block layer and a lot of intermediate layers we have to go through. And also inside the host kernel you have to also go through the Linux network stack and the device driver and

so on. So it's a whole bunch of stack that definitely degrade a lot of the IO performance. So in the middle is a we try to bypass the QEMU like a hypervisor layer. In that case the intermediate layer like a virtual device and those kind of thing can be immediately can be bypassed. This can give you a lot of like a higher performance already. But the things

like it still is not enough for running AI storage like a workload. So um the another option is just running the RDMA kind of uh Yeah. All right. So RDMA kind of you know stack. So with all with RDMA communication you have to install the like RDMA compatible like a client like a NFS and or maybe Lustre client in your VM. Then you have to specifically load

this kind of RDMA like a driver into the VM. Then with that you can directly bypass the TCP IP stack inside the guest kernel. That can also give you a lot of like a performance gain. Then definitely you can bypass QEMU and inside the host kernel there is a virtual PCI driver and then you just directly call that driver and send out your all the you know

IO intensive workload to the outside. So we adopted the last one to run our whole test. And the the cons of the last one is in order to get those kind of a performance gain you have to uh need to manage your additional software stack in VM definitely. You have to install the drivers and kind of thing. And also you have to do some additional configuration to

realize the HA because the storage IO somehow need to be HA. It's not It's nothing like a GPU fabric. For storage, if the connection drop, it's immediately just black out all your services. So, HA is very important in this case, and we are using the VF virtual function link aggregation technology to hedge these kind of issues. And the last one is that you also need to do

a lot of optimization and configuration for storage RDMA. So, by using our solution, you can handle the three all parts like costs. Those are being automatically handled by the Pedasus cloud. So, this slide will show you that um how to realize the GDS. GDS is a technology called a GPU direct storage. So, once you have some direct like a call from the from the remote storage, typically

without GDS, it has to go through the CPU and also system memory. And that in that case, CPU and the system memory become the bottleneck. Then, how to eliminate those kind of uh you know, overhead? So, the things is like it's it's sort of like a very similar to a GPU direct RDMA GDR, but the the invocation like like a source is remote storage. So, with the

GDS enabled, uh the all the storage like IO like a workload will be directly goes into the GPU memory directly through the HCA or NIC without going through the CPU or system memory. That can also intensely can improve a lot of performances. So, with KubeVirt and our solution modified like a like a function what we have implemented, we realized the GDS even inside the VM. So, I

think this might be the first time that I didn't see any community members are trying this with the KubeVirt, but I someone is doing this. And then we co-work with the NetApp to you know, deliver the this kind of real world use cases to the community. And we want to and we also run a set of like a benchmark test in order to show that how this

outperform compared to the regular one. So, I will So, yeah, please yeah, explain the next part. Thank you, Dr. Lee. So, this slide may look very familiar to some of you, the ones who are familiar with KubeVirt actually. And it displays how KubeVirt managed virtual machines can use native Kubernetes mechanisms to to to scale. So, on the left side you have the control plane and as you

know, virtual machine is managed through custom resources. The device plugin will allocate diverse hardware such as the the network cards and the the virtual functions. And the CDI is in charge of managing partially the the disks that you are attaching to the virtual machine. The virt controller will reconcile the custom resources and on the right side of the screen you will see what happens on the worker

node. The virtual machine or the libvirtd and the virt launcher will spawn the VM and attach the disks that are configured as well as mount the disk into the inside of VM. Bottom line, you can manage your virtual machines the same way you will manage any traditional workload in Kubernetes. As seen before, the benchmarks were done the newest NetApp system called AFF A400. So, for those of

you who are already familiar with NetApp, this is our newest platform and still uses our our operating system called ONTAP. So, ONTAP has been around for more than 30 years now. But the the I guess this new architecture brings you something that is pretty useful for AI workloads, which is disaggregated aggregates. Meaning, if you need more compute, just add more nodes. If you need more storage, just

add more disks in a nutshell. However, it is still on tap. So, with that, you get seamless scaling horizontally and vertically without over-provisioning what you need. You get also a whole lot of data services very useful for AI workloads for snapshotting and cloning and data protection. And since it is still our operating system on tap, you integrate your existing systems into platform and migrate data pretty easily.

Last, of course, you need security. And in the AI workload, we're also bringing you secure multi-tenancy to create multiple tenants on the same platform, as well as automatic ransomware Of course, when it comes to Kubernetes and storage, sooner or later you're going to talk about CSI. And when it comes to NetApp, we have a CSI driver called Trident. So, Trident has been around for more than 9

And in a nutshell, Trident can be used with the any ONTAP platform in any form factor. So, hardware, software-defined, or even in the major three hyperscalers, Amazon, Azure, and Google using first-party services. Trident is free and open source and maintained by NetApp. It supports various storage protocols, as well as all the existing access modes, which is pretty important. Of course, Trident does more than just provide provisioning

storage. It comes with a whole lot of data services and to to control and use your storage. But when it comes to VM, there are two that are quite important. The first one is the possibility to create efficient snapshots. And something that I would call probably smart clones because it's not a copy. It's just a bunch of pointers that we keep in in memory. Which means that

when you want to create a clone of a disk, you don't have to wait for minutes or hours. It is done instantaneously. Even though even if your volume is 1 gig or 1 terabyte, your clone is done in just 1 1 second. So, here we will see two different KubeVirt integration architectures you both using an NFS over RDMA storage providing by NetApp. So, on the left side

of the screen, you see the traditional CSI integration with with Uh and the all the the VM will use two disks, the the the boot disk mounted as a persistent volume, as well as the data disk. On the right side, a different approach which is probably more static. So, the the boot disk of the VM will also be mounted as a persistent volumes. However, the second disk

will be directly attached to the VM using NFS over the RDMA and the volume function that is provided by the network slot On the bottom, of course, you see KubeVirt and Kubernetes to sustain that In terms of network topology, no need to spend too much time on this one. The the the back end storage from NetApp is sustained by a 100 gig network and the NVIDIA platform

was connected to NetApp through a 200 gig network and with a bond over two ports. So, we did this benchmark with a tool called GGSIO. So, GGSIO stands for GPU direct storage IO, and it's here to measure various things. It will measure basically how fast your data can flow from your storage to the GPU memory. And it will give you It will give you different results such

as the throughput you can get, the latency you get in these measures, and it will also compare the two modes with or without GDS enabled. So, let's go through these BIOS results. On the first on the top left side of the screen, you see bandwidth comparison. So, here what's interesting to see is that if you look at the blue parts and you compare it to the green

parts, so blue is bare metal and green is virtual machine, the results are pretty similar, which means that just by implementing GDS, you really don't get loss of performance. On the right side of the screen, so top right, you see the CPU comparison. And here what is very interesting to notice is that between on both bare metal and virtual machine configurations, when you enable GDS, you decrease

the consumption of your CPU by roughly 40%. So, it I guess it it is it makes it a much much more efficient. On the bottom line bottom left side of the screen, you see the usage comparison. And you see that the GPU here is zero simply because the GPU is not spending its time trying to deal with the IO coming from storage. It It can actually focus

on what it sees it for, which means well, working on your AI inference. And on the bottom right side of the screen, you see the bandwidth that was measured on various pieces of bare metal and VM. And what is interesting here is that if you look at the two first columns, we have reached the same throughput on both bare metal and virtual machines. So, numbers, numbers, numbers.

So, what what what do you get out of this this slide? Simply, just by putting together the right configuration on your PCIE uh cards, especially the IOMMU and the ATS parameters, uh well, you get the same throughput in a virtual machine compared to a bare metal platform. Which in a nutshell removes the barrier of actually using your running your AI applications on bare metal. Well, historically, I

would say that um running AI inference on virtual machine was a barrier because you have a lot of layers between the storage and your GPU. And yeah, if you set if you configure your environment correctly, now nowadays, you can really reach the same amount of throughput. Yes. On this slide, we we have various results comparing the different data sets that we had during our benchmarks. And you

can see that on the top side of the screen, there isn't really any difference of of behavior depending on the the data set. And on the bottom left side of the screen, you see again that the by using GDS, well, there there isn't really little consumption on your on your GPU for AI workloads. That's for you. Okay, yeah. It's just really back to me. So, actually, GDS

I have definitely we can get a lot of benefit from enabling the GDS IO GDS in VM. And as you can see even you can run GDS inside a VM, we didn't get much of the performance like penetrate like degradations. Um so, what is a real real world use case of enabling GDS and so, one of the killer use case might we can envision is just KV

casual loading to the storage. This might be one of the killer use case we can envision. And actually, this left hand side is a software stack what we in order to run those kind of offloading, we constructed this kind of the the software standard. The top side is Nvidia inference servers like Dynamo and the VR and kind of thing has been installed there. And in the middle

side is our own solution which virtualized the entire Nvidia GPU and their servers. The underneath is a NetApp AFF storage. So, we try to offload the KV cache direct directly to the NetApp storage using the GDS path. So, that can further accelerate over all like performances a lot. So, without the KV cache offloading, the prefill So, if there is some kind of a user prompt going to

the the inference engine, the prefill will always be recalculated and that is a computing intensive kind of job. So, um it takes a lot of time to re-recalculate the whole thing. So, why not just store the KV cache to somewhere, maybe in memory or maybe offload it to the storage and if there kind of a need, we can we can reload the KV cache from the into

the HBM memory and in that case, we can just bypass all the prefill stage. So, that can which can further inference performance a lot. the So, those are the four like use case four like a setup in order to run this kind of a KV cache offloading scenario. depending on the memory hierarchical memory tier, we can differentiate as a four part. One is a G1 and the

other one is a G4. So, G1 is definitely it's not offloading anything. It's not offloading KV cache to the storage or to anywhere. It just use a GPU VRAM to um store all the KV caches. If If the KV cache exceeded the the RAM, then there will be some some eviction will be happen. So, in that case, the preview will also need to be recalculated as well.

And the second case is uh just try to offload all the KV cache to the system memory. And the third is uh using the GDS to offload the KV cache to uh the remote storage. The last one is uh using the CPU POSIX IO to offload the uh the KV cache, but uh the last one didn't uh leverage the GDS technology at all. It just remote access

the the storage. And so, the last one has to also need to go through all um the system memory. So, you will see that the performance is uh very poor in the last case. So, So, those are some setup that what we have. So, in a client, it runs some multi-turn or multi-round uh RM like um uh context injection. And in the middle side is is a

VRM engine. Yeah, we running two we we are running this engine uh against two GPU uh RTX GPUs. The last one is uh try we try to offload the KV cache using the RM cache to the remote storage or the system uh map the memory as well. So, since it's a multi-turn, at the initially, the uh cache uh hit rate is zero. But, uh as you adding

more and more like uh user context in the second round or third round, uh the overall like a cache uh hit ratio will be increased. So, depending on the cache ratio increase, you get a bit more like uh benefit like a by by hitting the caches, which means that we do not need to re-calculate redo the preview as much as possible in this case. So, this is

a war like performance what we measured with the experiment. So, with very small size. So, we actually use the Q138B. It's very small model. With this model, we uh differentiated the user context length from 2K to 16K. As you can see, when we increase the the context lines to 16K, we can immediately get a lot of benefit by using the GDS in these cases. So, but just

without using the GDS, as you can see the interface RDM shows that the very poor performance compared to the other offloading mechanism. But with GDS, you can get almost like a double performance compared to that. And also, the ITL ITL is inter token like like latency. And the lower is better actually. So, we also can get a lot of benefit from that as well. And the last

one is is a cache. So, oh So, they the hatched graph shows like is cache hit ratio actually in these So, once the cache hit ratio increased, we can we do not need to re-compute the the preview. Then So, overall, the performance will be goes up in these cases as well. So, the green shows the the best performance. And also, we increase the area model size to

the 62B. The overall the the result shows very similar to the small size, but uh overall, the GDS outperforms in all cases in our case. let me just conclude. So, uh In AS virtualization, we try to need to bypass the hypervisor layer. It's really essential for like achieving the near native NFS over RDMA and the GDS performance. And therefore, in the we also showed how we integrate

this with the Kubert and then we leverage the SR-V CNI combined with the VFs like offloading to the like a ConnectX and the BlueField adapter uh to realize this. And also we enable the ATS and IOMMU to realize this. And also with the uh storage IO performance-wise, as you we can show the near-native IO performance for NFS over RDMA and the even inside the VM and compared

to the bare metal. Uh last but but not the least, we also run this against uh the inference like a purpose benchmark test using the VM uh using a VRM and the Iron Cache and G4 definitely delivered the higher like a throughput than G1 and G2 uh and which leverages GDS for like KV cache offloading in very large context uh scenarios. So, this is end of my

talk and if you're interested about the NVLink of translation the GDR a bit more than definitely you should not miss the talk in the in the afternoon in the 2:30. Uh and yeah, that's it.