Lessons Learned Orchestrating Multi-Tenant GPUs on OpenShift AI with NVIDIA KAI (G/H2... Luca Berton

This talk covers the implementation of multi-tenant GPUs on bare metal using OpenShift AI, focusing on creating a sovereign full-stack AI platform. The speaker, Luca Berton, discusses the challenges of safely and efficiently sharing GPU resources among different teams, emphasizing the need for safety, fairness, and efficiency in the architecture. The session highlights the use of Dell AI infrastructure, Nvidia GPU, and tools such as GitOps, Argo CD, and Kustomize to achieve operational excellence. Berton shares practical lessons learned from deploying this system in a regulated environment, such as the importance of visibility, monitoring, and compliance. He introduces different user personas and their interactions with the platform, illustrating how a structured approach leads to better resource utilization and management.

Welcome everybody. Today I want to share a practical blueprint of running multi-tenant GPUs on bare metal OpenShift AI delivering a sovereign full-stack AI platform that drives immediate result. The key point is this. Getting GPUs to run is not the hard part. The hard part is sharing them safely, fairly, and efficiently across teams. My name is Luca Berton. I work on AI-ready infrastructure and enterprise cloud environments at

Dell Consulting. And these tools come from hands-on practical platform work delivering to an avionics customer in France in production today. What I will show you is the result of operational lessons from real systems. You're here for the architectural overview. And this is based on the Dell AI factory. This was a bare metal environment layered on Red Hat Kubernetes OpenShift. OpenShift AI will then capsule all the best

project that drive AI innovation. Oh, I forgot to tell you. All the slide are available throughout this QR code, but I will show you also at the end of the presentation. Where was I? Okay. OpenShift AI is the component that encapsulate all the open source project related to AI. But other two components are very important. Nvidia GPU and network operator, and as well having good GPUs. In

this for this customer we use a Grasshopper H200 GPU nodes with more than 137 gigabyte each. And we're leveraging the latest Dell AI server that can manage up to eight GPU card per node. Uh Another challenge was RDMA for storage. And another small thing, this is air-gapped setup with local mirrors. In the cloud some complexity is abstracted away. Here every layer is our to own, configure, and

recover. In total we installed more than 2 TB of data downloading all the intricacy of dependency between all the various component. What we learned at very quickly that a working GPU cluster is not the same as a shareable GPU platform. The real risk were noisy neighbor, queue starvation, mismatched MIG behavior. Enough familiar with MIG is a way of slicing GPU in predefined way, like a half, quarter,

and so on and so forth. You can divide by the tensor core or the GPU memory. And there are some already provided profile. Driver drift is also another problem, especially because this doesn't allow a core here visibility across the cluster. Um Networking failures around SR-IOV and RDMA, I would say that were the most interesting thing part of the project require a little bit of more love. The

challenge become how do we prevent chaos before tenants ever feel it? We use a framework. Every platform decision had to pass through three lenses. First, safe, meaning one team cannot break another. Second, fair, meaning contention is deterministic, not political, and nobody can override the decision on another team. Third, efficient, meaning we optimize for useful outcomes per GPU hours, not just utilization graphs. Remember these three lenses. These

will be a guideline for all the presentation. Safety, fairness, and efficiency. We design for three personas for a full service delivery. The end user want a fast access and stable latency. The LLM ops teams want repeatable deployments, safe upgrades, and observability. The tenant admin wants boundaries and visibility into cost. If any of those persona loses, platform adoption collapses. Our answers was GitOps. Argo CD and Kustomize become

the single source of truth for operations, infrastructure, and tenant configuration and applications. The reason this matters is not just automation. It is auditability, repeatability, and rollback. In a highly regulated environment, this is more critical than ever. In a multi-tenant GPU platform, manual fixes become future outages, and they can introduce single snowflake. We work with Kubernetes for years, so this is the best advice that I can give

you. We use We use a particular setup of a tree. We use the app over app of a Argo CD. But more or less this is the directory structure that we use all One thing, I told you this is a air-gapped environment, which means that in the beginning it was just a bare metal, and we need to install the node using the agent ISO created for OpenShift

for a specific node, which means specify the MAC address, specify the path of the dev device for each node, because this node has a lot of drive, PCI Express, and it is it become complicated very quickly. Once we boot the system, we use Ansible for the initial handshake, especially in the air-gapped environment, and then handle control to GitOps as early as possible. The goal was simple, bootstrap

once, then let Argo CD own day two operations. That reduced drift and makes the platform behavior predictable over time. Every time we need to introduce a new application, a new integration with a a different type of component, for example, the level of backup, we just create a new application and redeploy to We use a laboratory a scale-down environment and then deploy to the customer side. For us,

safety means hard is isolation boundaries. We implemented this at every level. Namespace isolation, scoped service account, least privilege with a rollback, and integrated with the local active directory of the customer, and implemented with the deny by default network policies. Both security guidance, the latest security integrated in OpenShift, as you see, quotas, limits, and admission checks. The key idea is that safety is not something operation remember to

do. It is something the platform enforces by default. And here you can see all the different layers. Admission webhooks were fundamental for every for every Each tenant came from a standard Kustomize build deployed throughout Argo CD. That bundles include namespace configuration, RBAC, network policy, quotas, supporting networking pieces. The practical outcome is important. No ticket, no manual steps, no tribal knowledge. A Git pull request become a tenant

provisioning workflow. And this is what the customer really asked to improve because he's a highly regulated customer, so he need a to prove compliance that what is running in production is exactly what we were expecting. One subtle lesson, this was a This took couple of weeks of our delivery because Uh, learned that the bare metal force you to care about the actual hardware mix. Specifically, different NICs

require different operational model. Uh, Nvidia um use this type of uh, network card, the Nvidia ConnectX. Uh, this have as a version six and version seven. And uh it's complicated because it serve as a like a six different network card and uh with different capabilities connected in a different way with a switch and uh everything was uh uh, need to be fast. One thing that AI need

is a faster access to data. Faster storage faster and we need we need a really millisecond access. The Nvidia ConnectX paths were managed by the Nvidia network operator for RDMA and GPU direct compatible flows. While other NICs paths were handled differently. We implemented also different network configuration, different VLAN and so there there is like a main machine network, there is a a different network for NFS, a

different network for backup and each of them is uh, separately physically separate. The main takeaway is do not assume that one network recipes fits every GPU node. You might have a slightly different Uh, what we discovered is that there is a an incompatibility between the uh Nvidia Mellanox ConnectX card and the Intel integrated in the server. There was a uh, an RDMA collision at kernel level and

we were not able to uh enable us SR-IOV. We discovered hardware and we need to blacklist the Intel uh the Intel driver that was taking the uh, connection with the kernel and we were not able to move forward with the project. Uh, another things that we did, we reduced upgrade for fragility by moving away from the more tightly coupled legacy approach toward open kernel module and DMA

buffer. These are the new driver designed by Nvidia and they are fantastic to be honest. The important message here is that it's not the terminology. It is that safer upgrade require reducing coupling between drivers, kernel dependency and data paths. Previously, the previous driver was directly connected with uh uh, the Docker driver. It it create like a a strange dependency graph between the network and the GPU. And

one was dependent by another creating like a compatibility matrix that we need to follow when we deploy a new cluster. in shared GPU environment, fairness does not happen naturally. Without a clear rule, the loudest teams wins. So, we use hard tenants GPU caps implemented through the Nvidia Kai GPU orchestrator. Uh, priority classes, explicit uh preemption posture scheduling constraint and GPU aware scheduling visibility. The goal was simple,

make resource contention predictable, explainable and intentional. Here you can see the different priority level and the interactive is always uh the one that have the highest priority because a user need a fast interaction. Whereas the training can be scheduled in a batch overnight. We looked at efficiency as outcome per per hour. That means using between time slicing, MIG and full GPU based on workload type rather than

ideology. Interactive notebook training and inference each behave differently. So, the platform has to optimize for real workload outcomes, not just maximum theoretical utilization. Upgrade path is taking into consideration because in AI world, everything move super fast and especially there is a one software delivery every month. One of the most important lesson was that the upgrade path is not an appendix. This is part of the platform design.

So, we manage and known good compatibility matrix in Git and we use like a canary pool. A big period usually is like 48 hour. We evaluate how how the platform behave and then we were able to to move forward on the next stage, validation gate and everything is a Git based rollback based on Argo CD. So, if there is a one component that doesn't behave like we

want we can always revert to a configuration that we know is uh is stable and working. We also use this when we enable new feature for the GPU operator. We started with a minimum viable configuration and then we start enabling one feature after another. Make the invisible visible. We invested in a per tenant monitoring. Chargeback views, scheduler report, GPU per hour visibility and I really believe that

observability is one of the things that really stand out when you create a new platform. Because this hardware is expensive and we need a way that we can uh, verify what is what is doing and make sure that we can report to our stakeholders. Uh, the big change was behavioral. Once teams can see their own usage and impact, they become much better platform citizens. A lot of

developer want to get assigned a GPU only for themselves, but what happen when they go in holiday or when they become uh when for example they have they have some day that they are not working. This become part of the pool and we can reassign on demand. Let me explain you with a one demo. There are three different persona here. LLM Ops engineer. His goal is to

deploy LLM models on premise so developers can securely integrate the feature into application. William start by identifying model on Hugging Face. Here we have a Microsoft fee mini for example. He logs into the Red Hat OpenShift and navigate to the OpenShift AI dashboard. After verifying the model files are stored in an object storage bucket it could be it's just a S3 compliant. He configure the model deployment

and he set up a model server using the hardware accelerator. So, here we can see that these are all the files downloaded directly from Um it's just a it's just a simple deployment and here I created already the model under OpenShift AI. One interesting part is that here we are using vLLM and we can allocate the Nvidia GPU for acceleration. We can decide how big is the

deployment model. We can decide how many GPUs we wanted and once it's deployed we can retrieve API endpoint and review performance dashboard monitoring inference benchmark. Here for example inter token latency and throughput and OpenShift system metrics like CPU and memory usage. And also we can correlate this over So, this is the view from William that our LLM Ops engineer persona. Another important persona Michael, the OpenShift administrator.

His goal is to guarantee the performance, availability and optimal utilization of the platform's GPU resources. Michael checks the OpenShift AI dashboard to see the active model deployments including the one just set up by William. And so you can see that there is a model a fee four, but there is also GPT-2 that is allocated by another user. So, he has visibility all over the cluster so he

can see where the resources are. He then switch to Grafana to view an Nvidia DCGM dashboard. he monitor detailed real time hardware metrics to ensure efficient GPU scaling and utilization. Tracking data points such as GPU power usage, temperature, SM clock speed memory clock and specific memory usage per MIG profile. We can see that this cluster is pretty busy. Mark is a Python developer. His goal is to

access an internal AI-powered coding assistant to boost productivity while remaining compliant with internal security policies. Mark works inside Visual Studio. Like many Install the open-source Continue AI code assistant access extension and you just need to edit the extension config.yaml file pointing it directly to the custom internal API endpoints that William deployed earlier. Rather than a public cloud service. You just use endpoint and API key. Finally, you

open a Python file. We're all familiar with this. This is just a simple sorting algorithm and uses the Continue chart that is alongside on the left, not on the right. And it can ask for like make it more readable or successfully generate a refined code. As you can see, these are the three user persona that in every organization can the deploy and use of this type of

Now you can extend to multiple domain, to multiple tenant. And understand the impact of a solution like that. Mhm. If you take only a few things away, start with GitOps first and a standard tenant template. Those two alone remove a large amount of operational pain. And then add quotas, clear networking ownership, a few stacks, a few stacks, useful observability and an explicit upgrade playbook. The bottom message

is mature shared GPU platform are built from gather rays, not heroics. As promised, the QR code is being into the full slide deck. Um multi-tenant GPUs on bare metal work then safely is provable. Fairness is explicit and efficiency is measured. In our case, GitOps was the mechanism that made those principle enforceable by Thank you. >> [applause]