Keynote: Making Kubernetes for AI Optimized and Reproducible - Nathan Taber & Mark Chmarny (ASL)

This talk explores NVIDIA's efforts to improve Kubernetes for AI workloads through the introduction of a project called AI Cluster Runtime (ACRE). The speakers, Nathan Tabor and Mark Schmarni, discuss the challenges faced when configuring GPU-accelerated clusters for AI training and inference, highlighting the complexity of managing over 250 configuration values across various components. ACRE aims to alleviate this pain by providing machine-readable recipes that encapsulate optimized configurations for different use cases. Mark elaborates on ACRE's functionality, including its CLI and REST API usage, validation of deployments, and the ability to generate bundles for various deployment methods. The talk emphasizes the potential for community collaboration on ACRE, as well as contributions already received from tech companies like Microsoft Azure and Oracle.

Good morning. I I think this is my favorite event of the year. I'm Nathan Tabor and I'm joined with Mark Schmarni and we are both building Kubernetes at Nvidia. And so we we're really interested in how can we make Kubernetes easier to run for AI? When we look out across the community, we've been doing a number of investments across Nvidia. We've been working at projects like the

DRA driver, the GPU operator, Kai scheduler. A few moments ago Aaron was talking about all the work that Nvidia has been investing in and we know when we look at GPU accelerated clusters that these things are just the start. When you go and you stand up an AI cluster for Kubernetes, you have to install all of these components in your cluster and more. And we know that

installation is just the beginning. So when you go ahead and you want to do, let's say uh uh a workload on EKS with an H100 accelerator um and you add in your operating system, let's say Ubuntu, and then you say you're going to do training. There's over 250 different configuration values that you have to make across 15 plus components. And then let's say you want to change,

you want to go from training to inference. Well, that's swapping out five components and additional 40 plus configuration values. And so there's a lot of work that you have to do to manage and make an optimally tuned cluster to run AI. And so what we've been doing at Nvidia, we we feel this pain every single day. One of the responsibilities of the team that Mark and I

work on is to not just help make Kubernetes better, but also to run the accelerated Kubernetes infrastructure for all of the different workloads that Nvidia does that span from world model and simulation building all the way to robotics. And we've realized that these tuning and this optimal configuration is locked behind the walls at Nvidia and we want to open that up and we want to bring the

community in. And so we're proud to introduce this project AI cluster runtime which brings our knowledge about how to configure AI clusters for GPU training and inference into machine readable recipes. And so I'm going to turn it over to Mike, he's going to talk to you Mark, he's going to talk a little bit about how ACRE works. >> [laughter] >> I'm Mark. Hi. So like Nathan was

talking about these recipes, they represent a very validated, optimized, and reproducible set of knowledge that we've acquired over many years and wanted to share with the with the rest of the community. These recipes represent a set of components that are version locked that are combined with set of dependencies for each one of those components, my any manifest optimized values files if you're in a helm world that

are specifically tuned for set of use cases. Each one of those recipes recipes also include a set of constraints that represent the minimum set of requirements that have to be present on a cluster in order for the recipe to be deployed on a cluster and it comes with a self-contained set of validations that demonstrate or validate the actual cluster once the recipe is deployed on a cluster

spanning from everything from the actual components and their readiness all the way to a CNCF AI conformance readiness. want to acquire one of those recipes, what ACRE does is provide both the CLI and a self-hosted REST API that you can deploy in your own data center. And when you use ACRE to create a recipe, you you have to provide any number of variables. The more values the

more inputs you provide into the recipe, the more optimized the recipe is going to be for a specific use case. So you can specify the actual service, the accelerator type or GPU type, you can specify the intent training versus inference, the OS types, and these are some representative values in here, but the output of that is going to be a recipe that it's either written to a

file system or to OCI image. And because ACRE encodes the entire graph of that knowledge inside of the binary, you can actually query that graph in real time in your CI/CD pipeline without connection to the outside world. Once you have that recipe which represents the desired state inside of the cluster, you can actually generate a bundle. The bundle is just set of artifacts that are going to

be used to materialize that recipe inside of the cluster. So ACRE comes with number of deployers supported already, currently Argo CD and Helm. Other ones are coming very soon. And what you when you deploy when you create a bundle, you provide the information with the recipe and the desired output. And because bundle is before because ACRE is also itself attested during the build time, we can attest

each one of those bundles during the actual generation time. What that means that later down the pipeline once you have persisted the the bundle in in your GitLab repository or GitHub repository, you can later months later actually validate that bundle and its accuracy that it wasn't compromised all the way with a provenance down to the specific version of ACRE from which it was generated. Once you deployed

the the bundle that we've created using the recipe into the cluster, regardless which way you've done that, ACRE provides number of validators that can be used to validate that the recipe on the cluster. We talked about the readiness of components. What that can also do is validate the conformance of that cluster against a CNCF AI conformance program and it will provide you actually an option to the

artifacts that are required to submit a PR to CNCF to to get a conformance certification. Cool. So we seeded ACRE with recipes for H100 and GB200 accelerators across EKS and GKE. But the work Nvidia is doing here is just the start and we have a a long pipeline of different capabilities and recipes that we plan to add, but we've also made this open to everybody and we're

excited to work with hyperscalers, AI clouds, developers, and users to add new recipes and new capabilities into ACRE across all types of different environments. And so I've been really excited to see in the last week since we've opened ACRE up, we've seen contributions from both Microsoft Azure and Oracle and we're really excited to work with you to add many more recipes and capabilities into ACRE over time.

So go ahead and check out the project that's at github/nvidia/acre and have a great KubeCon. >> [applause]