Keynote: Making Kubernetes for AI Optimized and Reproducible - Nathan Taber & Mark Chmarny

In this talk, Nathan Taber and Mark Smalley from Nvidia discuss their efforts to make Kubernetes easier to run for AI workloads. They highlight the challenges involved in setting up AI clusters, particularly the numerous configuration values needed for optimal performance when using GPU accelerators. To alleviate this complexity, they introduce the AI Cluster Runtime (ACRE), which encapsulates their expertise in configuring AI clusters into machine-readable recipes. These recipes are designed to be validated and optimized for various use cases, with each containing version-locked components and dependencies. The speakers emphasize that ACRE simplifies the deployment process through a command-line interface and a self-hosted API, allowing users to generate and manage cluster configurations effectively. They invite community participation in expanding ACRE's capabilities with new recipes and integrations.

I I think this is my favorite event of the year. I'm Nathan Taber and I'm joined with Mark Smalley and we are both building Kubernetes at Nvidia. And so we we're really interested in how can we make Kubernetes easier to run for AI? When we look out across the community, we've been doing a number of investments across Nvidia. We've been working at projects like the DRA driver,

the GPU operator, Kai scheduler. A few moments ago Aaron was talking about all the work that Nvidia has been investing in and we know when we look at GPU accelerated that these things are just the start. When you go and you stand up an AI cluster for Kubernetes, you have to install all of these components in your cluster and more. And we know that installation is just

the beginning. So when you go ahead and you want to do let's say a workload on EKS with an H100 accelerator and you add in your operating system, let's say Ubuntu and then you say you're going to do training. There's over 250 different configuration values that you have to make across 15 plus components. And then let's say you want to change, you want to go from training

to inference. Well, that's swapping out five components and additional 40 plus configuration values. And so there's a lot of work that you have to do to manage and make an optimally tuned cluster to run AI. And so what we've been doing at Nvidia, we we feel this pain every single day. One of the responsibilities of the team that Mark and I work on is to not just

help make Kubernetes better, but also to run the accelerated Kubernetes infrastructure for all of the different workloads that Nvidia does that span from world model and simulation building all the way to robotics. And we've realized that these tuning and this optimal configuration is locked behind the walls at Nvidia and we want to open that up and we want to bring the community in. And so we're proud

to introduce this project AI cluster runtime which brings our knowledge about how to configure AI clusters for GPU training and inference into machine readable recipes. And so I'm going to turn it over to Mike, he's going to talk to you Mark. He's going to talk to you a little bit about how ACRE works. >> [laughter] >> I'm Mark. Hi. So like Nathan was talking about these recipes,

they represent a very validated, optimized, and reproducible set of knowledge that we've acquired over many years and wanted to share with the with the rest of the community. These recipes represent a set of components that are version locked that are combined with set of dependencies for each one of those components, any manifest test and optimized values files if you're in the helm world that are specifically tuned

for set of use cases. Each one of those recipes also include a set of constraints that represent the minimum set of requirements that have to be present on a cluster in order for the recipe to be deployed on a cluster and it comes with a self-contained set of validations that demonstrate or validate the actual cluster once the recipe is deployed on a cluster spanning from everything from

the actual components and their readiness all the way to a CNCF AI conformance readiness. want to acquire one of those recipes, what ACRE does is provide both the CLI and a red self-hosted rest API that you can deploy in your own data center. And when you use ACRE to create a recipe, you you have to provide any number of variables. The more values the more inputs you

provide into the recipe, the more optimized the recipe is going to be for specific use case. So you can specify the actual service, the accelerator type or GPU type. You can specify the intent with training versus inference, the OS types and these are some representative values in here, but the output of that is going to be a recipe that is either written to a file system or

to OCI image. And because ACRE encodes the entire graph of that knowledge inside of the binary, you can actually query that graph in real time in your CI/CD pipeline without connection to the outside world. Once you have that recipe which represents the desired state inside of the cluster, you can actually generate a bundle. The bundle is just set of artifacts that are going to be used to

materialize that recipe inside of the cluster. So ACRE comes with number of deployers supported already, currently Argo CD and helm. Other ones are coming very soon. And what you when you deploy when you create a bundle, you provide the information with the recipe and the desired output. And because bundle is before because ACRE is also itself self-attested during the build time, we can attest each one of

those bundles during the actual generation time. What that means that later down the pipeline once you have persisted the the bundle in your GitLab repository or GitHub repository, you can later months later actually validate the bundle and its accuracy that wasn't compromised all the way with a provenance down to the specific version of ACRE from which it was generated. once you deploy the bundle that we've created

using the recipe into the cluster, regardless which way you've done that, ACRE provides number of validators that can be used to validate that that recipe on the cluster. We talked about the readiness of components. What that can also do is validate the conformance of that cluster against a CNCF AI conformance program. And it will provide you actually an option to the artifacts that are required to submit

a PR to CNCF to to get a conformance certification. Cool. So we seeded ACRE with recipes for H100 and GB200 accelerators across EKS and GKE, but the work Nvidia is doing here is just the start. And we have a long pipeline of different capabilities and recipes that we plan to add, but we've also made this open to everybody. And we're excited to work with hyperscalers, AI clouds,

developers, and users to add new recipes and new capabilities into ACRE across all types of different environments. And so I've been really excited to see in the last week since we've opened ACRE up, we've seen contributions from both Microsoft Azure and Oracle. And we're really excited to work with you to add many more recipes and capabilities into ACRE over time. So go ahead and check out the

project. It's at github.com/nvidia/acre and have a great KubeCon. >> [applause]