Project Lightning Talk: Hami: Dynamic, Smart, Stable GPU-Sharing Middleware In Kubern... Mengxuan Li

This talk introduces Hami, a smart and stable GPU sharing middleware designed for Kubernetes, presented by Limousine from Dynamia AI. The speaker highlights the limitations of traditional GPU allocation methods, which often lead to underutilization of GPU resources. Hami addresses this issue by facilitating GPU partitioning, allowing multiple tasks to share GPUs efficiently. The speaker explains the user-friendly allocation process using resource limits and the importance of hard resource isolation provided by their custom CUDA hacking library. The integration of Hami with Volcano for resource scheduling is also discussed, showcasing its growing adoption in production environments, especially in China where GPU efficiency is crucial.

I want to make a proper introduction. My name is Limousine and I'm from the Dynamia AI and I'm the maintainer of this project called Hami and the topic is smart and stable GPU sharing middleware in Kubernetes. So, Mhm? So, if you are using the traditional way of allocating GPUs like by using a video device plugin to allocate it exclusively, you may get a very poor performance GPU

cluster. And these two pictures are taken in a real production environment. You can see that the overall environment of certain GPU is about below 20% and that is it that is a very natural scenario. So, to solve this problem, we we we produce the project called Hami uh which is a GPU partitioning system and you and with it with its help, you can uh sharing the GPUs

uh between multiple uh task. For example, in this mhm so in this scenario, both user A and user B can share can share their task on the on these two GPUs and leave the rest for others to use. Also, we can do that by either using traditional Hami or by using the more advanced Hami AI. Yeah. So, I guess many of you may wonder so since we

have already have a DI solution which can uh sharing GPU dynamically, so why uh Hami is about? So, mhm there are two advantage of Hami. The first one is you can allocate your GPU uh uh GPU slice by just allocating uh resource limits here. You don't need to write your resource claim which is more user-friendly, I guess. So, in this so so in this example, you wish

to allocate two GPUs and each with the device memory of 10GB here. Uh and the other advantage of using Hami is we guarantee the hard resource isolation inside the container. The core component of Hami is a self-implemented uh CUDA hacking library which is injected into the container. So, you can see the output of SMI here as the upper limit of the device memory has been limited to

10GB. Yes, which which is done by the by by our library. So, the method of install Hami is very easy. You just fetch your fetch the Hami repo and install it by using helm install. And then you may want to label your GPU nodes which you wish to be managed by Hami by labeling the nodes with the label GPU equals on. And the requirements of Hami is

very easy to satisfy as long as your Nvidia driver version is larger than 440 and the Kubernetes version is larger than 118. Yeah, but if you wish to use Hami DI, then your Kubernetes version need to be larger than 134. Yeah. Hami has been integrated by Volcano as well. So, if you enable the device share plugin in Volcano, you can you can just use Hami like the

Hami project and the scheduling part is done by the Volcano. We have integrated the the scheduling part into the Volcano main branch and we we use the Hami core I just introduced to manage the in-container resource. And you can just specify your GPU partitioning spec here like that you want to use two GPUs each with the memory of 10GB and 30% uh 30% of the total computing

cost. You can just specify the limits here, resource limits here. Yeah, it is very easy to use. And this is the comparison table. I won't go through the details here because the time is limited. If you are interested, you can meet me at the pavilion tomorrow and and Thursday. And it has been widely adopted by many end users. Uh and there are many of them are coming

from China as you all you already know that uh Um in China, we have to make every GPU counts. Yeah. Uh so, if you wish to learn more about our project, you are free to search Hami h a m i on GitHub and you can join our you can you can search our website projecthami.io. Yeah, you can also join our Slack and we have a pavilion tomorrow

and Thursday. Yes, see see you there. Yeah. Okay. Thank you. >> Awesome. Thank you, buddy. >> [applause]