Virtualizing Large Scale GPU Cluster for Sovereign AI: Petasus AI Cloud Journey with Kube... Jian Li

This talk discusses the virtualizing of large scale GPU clusters specifically in the context of SK Telecom's in-house solution called Pedestria Cloud, which utilizes Kubernetes for orchestration. The speaker describes the challenges faced during operation, such as GPU resource fragmentation, provisioning delays, security risks, and operational overhead in AI infrastructures. They highlight the importance of effective GPU resource management, including optimization techniques like virtualizing Nvidia's NVLink and implementing GPU Direct RDMA for inter-node communications. The session further touches on the significance of maintaining performance while ensuring security in a multi-tenancy environment, and the results from benchmark tests indicate that their virtualization solution achieves performance comparable to bare metal systems. Overall, the presentation illustrates the advancements in infrastructure optimization necessary for supporting AI workloads effectively.

Nice to see you and uh yeah today I would like to give a talk about the virtualizing the large scale GPU uh cluster for so I uh this basically target to our in-house build solution called the pto cloud journey with the uh uh kubernetes so I'm from uh SK telecom from South Korea and I'm Jen and currently working as a principal engineer at SKT so this is

a just brief overview that u about our slide Right. So first of all I just want to uh provide some background of today's talk. First of all, just from the last year uh around July uh we started our GPU as a service at the South Korea uh with the over 1,000 Blackwell GPUs uh which is fully managed by the Pasi cloud and uh we call this cluster

as a H cluster uh which has been uh fully uh constructed and designed and by the SKT uh with some partners like Penguin Solutions and the Super Micro. Um we are using the HGX type uh like a server scale uh GPU servers uh in our production and uh in this uh environment we fully virtualize the entire AI infrastructure uh using our solution and uh but uh now

there are two end customers are using our solution to train their uh RLM model and actually the HANE cluster the name HANE is inspired by the Hannes hub which is one of the very famous temple in South Korea and we position ourself as a core element of the Korea's sovereign AI infrastructure. So uh before I go dive deep into our my my talk I want to just

want to address uh the several uh issues what we faced uh when we start to operate the large scale like AI uh data center especially the infrastructure specifically. So the first uh challenge what we had is a GPU resource fragmentation and waste. So as you know the HSGX or uh DJX type observer uh by default equipped with the eight GPUs uh in each node and statically um

allocate those GPUs to a specific team and uh it leaves the low overall utilization and uh which in turn can increase the TCO. So for example, some team or maybe some user just need two GPUs just run their uh ARM like inference workload for example. In that case we cannot just provide such a infrastructure without virtualizing the entire infrastructure. Somehow we need to split the uh ATGX

node into several parts especially the GPU perspective. And the second uh address uh the issue what we address want to address here is uh uh AI uh environment provisioning delays. Actually uh once we provide for example like you know uh provide the biome type of services to end user somehow the user need to be dealing with all the drivers and the library installation and optimizations. But typically

uh our end users are not quite aware of the details of how to manage the infrastructure thing. So uh this can slow down all the deployment like schedules a lot. Um and sometimes um there will be and more like like customers they sometimes just require maybe one day or maybe two days uh uh like like infrastructure but sometimes yeah but we but without virtualizing the entire like

the the infrastructure we somehow need to like provide only the reserved services for those customers. The third is very important which is the security risks. So actually we are supporting multi-tenency in our environments and uh without virtualizing all the infrastructure will be visible by the any others. So this can pose another uh potential risks. So definitely yeah multi-tenency support here the multiency means not soft multiency it

it be it's so it is a hard multi-tenency. uh so somehow we also need to support this in order to fully uh increase all the you know C G CPU utilizations to the end and the last not the least is the scalability and operational overhead. So uh although we just running over 1,000 but uh so without the proper platform there will be a lot of manual like

a physical operations and uh you know that can also error prone and somehow we need to provide the very systematic way to detect GPU failures and uh if there are some issues in underneath uh somehow we need to find a way to immediately detect the issues and uh uh do a proper like a treatment. So this is the uh what our uh vision for virtualizing the entire

AI infrastructure. So underneath is um GPU node. So here I just uh depicts like a GTX type of GPU server and unlike the traditional data center AI data center uh comprise of a large like amount of GPUs and all the GPUs some somehow need to be interconnect internally also externally. So internally Nvidia uh used the MB link to provide a very high speeded interconnect uh of the

GPUs for the external uh connection. Uh typically there are two ways. One is using the uh infinity band. Another way is recently a lot of people just talking about the using the uh Rocky V2 type of RDM uh the RDMA protocol to interconnect GPUs and uh uh we so we need to somehow when we start virtualizing the thing we somehow need to virtualize the ambulink first also

we need to virtualize the GPU fabric and for the regular purpose like a north south traffic is dealing with the BPC so it's been virtualized already and the last one is uh the storage fabric And the typically the training job require very high speed like a storage fabric as long as the IOPS for the storage storage appliance perspective. So somehow when we start to virtualize the storage

fabric as well the storage we some we also need to care a lot about the performance. So with this virtualization layer we somehow need to be need to minimize the overhead as much as possible. So there are many ways to virtualize uh infrastructure and previously actually a lot of people are using the open stack to spawn VM and use this way to virtualize infrastructure but uh it's

secure because uh it provide a guest konel which play just like a sandbox. So even with one VM has been compromised for some reason the neighbors VM still be safe because the host economy will be still be protected. uh but the open stack way is uh sort of like it the performance is quite poor because u they really don't really target to this aid virtualization scenario specifically

and second is unlike kubernetes it's not really extensible in kubernetes as far as as you know we can use a cd to extend the the resources and the that kubernetes to uh to orchestrate these newly added resources as we want so uh so the container based uh location getting a lot of popularity recently and it's it's a yeah for sure it's a flexible it's extensible it's a

high performant but the one like a drawback is it's not insec it's not quite secure compared to the VM and it's a little bit uh complex to manage so we ended up with the the last one which is containerized VM uh so uh the good thing is uh uh kubernet CNCF already imaging kind of use cases. So they there's a cube open source which exactly does this

does this thing. So by uh side of by side running the cubvert with the kubernetes we can orchestrate the VM just like uh orchestrate uh like containers on top of Kubernetes. So uh this uh way we can purely can got get a lot of advantages uh from Kubernetes platform but one downside is still it's it it's poor performance. The reason is cubert so far didn't really specific

target to this ac use cases. So we actually made a lot of efforts to optimize the cubvert stack to to make it be possible to run in our environments. So I would like to go through what kind of optimizations need to be enforced in order to run Kubernetes and cure and use that to virtualize your infrastructure and and so on. So uh as I mentioned that we

uh it's a we just put all things together and we added a lot of optimizations and then we we packaged that entire solutions and we named that as a pedestria cloud. This is a SK Telecom in-houseuilt solution. uh we have built this solution for many years and recently since AI getting a lot of traction. So we added more support for uh this AI workload. So underneath we

support uh AI accelerator from future genius vendors not only Nvidia but Intel and AMD and on top of that we can run hoggenous workload not only limited to the container and the VM but we also can run other uh manage to kubernetes on top of this and on top of that when we can run multiple like services as uh the user uh to meet the user's demand.

So yeah I will dive in deep for uh the each item. So in link and the GDRization what what is this for? So actually if you buy SXM type GPU server it typically equipped with the eight GPUs by default and all the GPU are interconnected with the through the MV switch through the MV link actually. So this is a uh I think this is a hopper but

the bandwidth is shown like blackwell. So for blackwell um all the eight uh so each uh GPU can communicate with each other birectional uh the link speed is up to 1 uh 8 terabyte per second which is very high speed. So we somehow need to utilize this bandwidth even if we virtualize the you know the VM uh like a GPUs in the case. The second is GDR

this is very important because uh so MB link is fine it just happen inside the inside the GPU node but what if you want to uh enable like a inter node GPU communications with high speed. So typically uh in that case we somehow need to fully utilize the GDR. GDGR is a stands for uh GPU direct RDMA. Um so uh in order to use that somehow you

need to be aware of your uh server topology. So for example in this diagram it shows like it's a HGX B200 uh baseboard which equipped with the eight GPUs and it has a uh six PCI uh switch internally and each PCI switch uh interconnects the nick here you can consider this as a nick or HCA and the GPU no directly. So each uh so in order to

fully utilize this GDR you have to carefully manage you we you have to carefully manage the nick and the GPU. So to make sure that your paired nick and the GPU should be uh under the same PCI switch otherwise uh it has to go through the QPI if if you uh incorrectly map the GPU underneath. So those topology kind of thing should be aware and then how

to virtualize MV link actually MV link is a is kind of link fast link and it has been provided uh uh by the MV switch. So MB switch is a special it's sort of like a just switch but u uh it provide very high speed and uh by default they has 18 links in each uh MB switch and the MV switch can be managed by uh some

external software called the fabric manager. So the fabric manager was uh pre- shipped by the Nvidia. So if you want to uh certain GPU toGPU uh communication allowed it or maybe blocked you somehow need to carefully manage the uh their uh like a MV link like permission like it's not like a regular Ethernet switch uh Nvidia like introduce something like a partition concept. So each partition mapped

with a certain uh set of GPUs. So for example if we enforce partition one to blackwell MV switch then it will allow all the communication between GPUs. So by default you can just use that concept to manage the thing and if you want to specifically block certain link you have to infor you have to enforce a certain partition number into the fabric Yeah. And the second thing

what you need to deal with is what I mentioned GDR is very important. So the concept of GDR is if there's some traffic comes from the nick and the target I mean destined to the GPU memory without GDR you have to go through uh all the all the passes like you have to go to the system memory first then it will direct you to the GPU and

the GPU memory and in this scenario the CPU and the system memory will become a bottleneck definitely. So uh by utilizing this GDR uh technology have I mean properly you can get a lot of performance gain. So in real AI work if you want to run the AI workload without any uh performance degradation you somehow need to care about this thing a lot but in virtualization like

a word how to realize this so in bare metal it's quite simple because you just need to follow the Nvidia's guide to just realize this but if you start virtualizing it you have to care a lot of about the the direction so first of all if you virtualize something the MMU should be enabled in your GPU server. Um, so if this IO MMU has been enabled, all

the traffic comes from the nick, it has to go through the uh PCI switch and the root port and the root complex and also it's interact with the CPU and the memory as well. Then we'll degrade a lot of performance a lot. So uh one way of uh mitigate this issue is uh you can enable the ATS which is address trans translations translation service which has been

introduced by the Nvidia. So with this option enabled all the traffic comes into the nick will be directly go to the GPU memory. uh so actually IMU will be in charging of mapping the physical quant virtual memory uh like addresses but uh those addresses can be cached into the HCA using the with the ATS service enabled. So you have to enable this option so that uh you

still can use GDR in your uh virtual machine. This is very important. Then um how to then run the Niko? So the so this is not sufficient to run the Nikico test because um as as I mentioned if you spawn your VM just using the vanilla cube uh all the feature topology will be mapped in a flat manner which means GPU and knee topology will be just

mapped in the right hand side way like it's just flat but what I mentioned previously this is not uh directly uh reflect the uh physical PI topology right so you have to rearrange or maybe yeah we have to rearrange the PI topology by uh by considering your like a physical like bare metal node. So uh after you rearrange on the entire PC topology somehow that should be

exactly identical to the uh bale metal p topology as well and there already some web has been proposed by the cubvert community some patches has been upstreamed but some are not so I think uh in cubvert 1.9 most of patches will be will be upstream so you can utilize fully utilize this uh uh like a feature in your production but for us we use a different way

because we didn't open source that part yet and uh we are using another way we call this is a uh PCI PCI topology hook side car and this is a kind of a sidecar container which uh typically watch the physical uh PCI topology uh in the bmetal and then it will rearrange the domain XML in the breeput time then in a runtime rearrange domain XML will be

inputed into the lirad then VM will be spawned by using this kind of mechanism so this is all like a workar around from SKT but uh yeah already yeah the community has been yeah been aware of this issue so start working on this so I think the feature will be available soon so once you've done all the things what I mentioned so far then you will be

eventually have a very nice topology view inside your VM. So once uh you put up put up your VM uh somehow your uh GPU and nick topology should be looks like this. If the topology does not looks like this, you might be have some trouble to run Niko Niko test. So, so GPU to GPU connectivity should be all shown like MV18 which means it's connected with the

18 MV link and then GPU to nick at least one of the GPU and nick uh topology should be shown like a PIX. So pix means u here means it's connection traversing at at most one single pci bridge which means the two paired nick and the GPU should be placed under one identical one PCI switch otherwise we cannot get the full performance and the others should be

shown like a sis and node that is fine so once you done this part then you can start to run your nikl so I just want to briefly show that uh even with the uh the performation is not that severe. So I just want to uh show the benchmark result what we had. So we ran a lot sort of like a ben the benchmark test in order

to find the virtual how much is the virtualization tax. Can we is it okay just to run uh this in our production. So for H100 we run the uh in link bandwidth like test the benchmark as you can see it almost show like 700 uh 70 gigabyte per second it's a birectional and theoretically Nvidia claim it should be uh like reach up to 900 but in reality

it only reach up to 700 like a 70 some number but this number is almost same as a bell metal but right hand side is for blackwell It's a it's a double. So we compare the you know the uh MB virtualized MB link performance to the biometer in biome case and as you can see the performance wide the degradation is quite negligible. It's almost like one less

than 1% degradation. So which is quite fine and without properly uh virtualizing the MV link which means you only relying on PCIe for example communication then uh compared with that MV link you can outperform almost like a 1,700% performance improvement you can you can observe we also ran the uh hyperformance uh hyperform impact score as you can see also the overall uh the score looks quite fine

uh with the uh the virtual Yeah, definitely we also run ran the nickel. So nickel is a the Nvidia communication link library. It's very important library because if you want to run the ARM training workload, this is must have. So you it's kind of mandatory that to run this. So we run the Niko test in single node. As you can see uh you cannot see the blue

color. The reason is the VM the optimized the VM the bomero like a score is exactly over overlapped which means there's no performance degradation at all. So which is quite fine with us and also uh in order to fully utilize the topology wide optimizations you also can consider you you also can combine the PCIe and the in linker topology to further reduce the the traffic flow and

optimize the network traffic. So uh I think those are typically automatically done by the nickel but uh just just uh for your uh for your reference I just want to briefly mention that. So in the best way we should not uh go through the out like the external like a link like the infinity band because uh in link provide a hu the way much higher bandwidth compared

to the um the infinity band. So for example, if you want to make uh make the GPU node three at GPU 3 at the node one want to communicate with the uh GPU five and node two uh there are two ways just going through the external link but it is a bandwidth capped but if you want to go so if you can use the fully to use

in link then it can further improve the the performance a lot. So we call this is a pxn. So then isolation. So so far what I talked about is just virtualizing your infrastructure. But uh in order to provide the multi-tenency hard multi-tenency uh somehow you also need to find a way to isolate your GPU fabric. So there are two like types of GPU fabric. One is uh

uh using just infinite band. Another one is using the rock key. But the rock key is is Ethernet based. So you can utilize whatever like a tenant isolation mechanism like using the vlan or vxn it's all fine but in this slide I specifically want to tell want to talk about how to isolate infinity bandan so the concept is very similar to the ethernet but they introduce another

concept called the partition key. So this is uh very similar to the v vlan ID. So once this partition key has been generated uh the HCA which has been configured with this exactly same partition key can be communicated with each other otherwise all the communication will be blocked. Then how we can automatically implement the whole things. So we are uh in this cases we are we implement

our own kubernetes operator by strictly follow the operator pattern. So once the user or the mean spawn a new VM and that the VM be included a certain GPU cluster then Kubernetes will be automatically uh uh be aware of this because they will there will be an API call uh from our portal to the uh GP like a Kubernetes API server and then the Kubernetes operator will

watch the VM event. If the VM has labeled a certain num like a number then the operator will be in charging of u um query the GPU cluster policy for example like a P is setting up a certain number then it try to discover the VM's HCA from the UFM and then it try to contact with the uh Nvidia UFM to enforce the PT to the uh

HCA uh which has been assigned to that VM. in that case. So all the things are done in in automatic way. So the end user will should not be aware of what is happening underneath. So uh we also uh evaluated the uh performance uh with the isolated GPU fabric. Um as you can see um here. So I think uh I think the right hand side will just

show that result. So with 16 uh like GPUs with without partition and with partition doesn't really show any differences in terms of Nicole all reduce test like benchmark. So overall even with partition or without partition it doesn't really affect the performance at all uh based on our uh like benchmark test and also virtualizing uh the the like infrastructure it also doesn't really introduce any uh like overhead

too much. So left hand side you show the result. So so bare metal throughputs almost overlap with the uh throughput of the VM. So that is the case. Then we also run the scalability test over 1,000 GPU uh blackwell GPUs and the topology looks a little bit complex right. This this not really in like production like topology but it's similar to production. Yeah. So it comprises of

six spine switches with the 24 lip switches and then we uh implemented the eight rail optimized topology with the 64 port NDR switch. So in this topology we also ran the niko test and try to see uh what is the trend of uh the overall like uh and the throughput and things and we did this inside this VM no battle and as you can see uh as

the number of GPU increase the performance degradation is not that severe it's still okay and uh even you run this in the bomero you might observe the same trend so we can claim that even with large scale uh like uh GPU cluster uh it's still doable just virtualizing your like infrastructure doesn't really hurt a lot of your like performance so our end user do not really claim

about the performance at all because it's almost similar to the bare metal for AI storage virtualization I already presented at the cuber summit like a couple of hours ago so if you want to check how this has been done then maybe you can check the uh uh just capture the link and uh I already shared some slides you and yeah check by yourself. So due to time

limitation I just want to jump to the observability. So we are we heavily utilized a lot of uh uh CNCF project to implement our uh the observability stack. So as you know um we run the v we run the VM using the containerized hypervisor uh and inside the VM we by default install two uh agents. one is a node exporter and another one is a DCGM exporter

and then we expose those matrices to the um promeicus to collect all the you know GPU and the VM related matrixes and if there's some like anomaly detected then it'll be automatically uh send it to our manager and it can eventually can notify the uh some components like slack and email whatever you prefer and the rule is very straightforward because some Nvidia already provide a way to

identify the GPU feder type uh using some sort of like XID concept. So yeah, you can the operator can compose its own like our rule to enforce it to the parameters and to look in into details of what happening inside the GPUs. So this is just written Korean but yeah we already have a very nice GU to represent the GPU status. So each of the rectangle denotes

one GPU node and inside one GPU node there are eight GPUs has been running and then we we just showing the GPU sizations and the details through the GU. So yeah key takeaways. So um so uh a infrastructure optation and isolation. So actually it's uh categoriz one is a compute. So we are using the converged VM and the container orchestration via the cubert and the kubernetes to

imp to realize this compute uh virtualization uh for the for infrastructure for networking wide. We added a lot of optimization like a stack like in link virtualization, RDMA virtualization, PCI topology alignment and optimization also the automated partition and the management via the Nvidia UFM for storage for storage uh we realized the the the uh we implemented HA enabled the storage fization with the GGS supported. So inside

the VM you also can take advantage of using the GDS. Um and the second point is I want to mention is a performance assessment on the virtualization isolation and uh as I shown like uh we achieved the near bare metal like performance with less than 1% of uh overhead in in following the benchmark like MVL link test and the HPL and the Nikico and finally our solution

is a fully yeah fully uh it's a full stack like cloud native architecture for Yeah thank you for your attention and this is the end of my talk. Thank you.