Continuous AI Conformance: The kOp... Arnaud Meukam, Janet Kuo, Justin Santa Barbara, Ciprian Hacman

This talk discusses the Kubernetes AI Conformance Program and its implementation using Kops, presented by contributors from Google and Microsoft. The speakers elaborate on the new requirements for Kubernetes in handling AI workloads, highlighting the inadequacies of traditional Kubernetes conformance for these CPU and resource-intensive tasks. They emphasize the necessity of establishing standards for AI capabilities, which include scheduling, orchestration, and security. The session details how Kops operates as an open-source tool for managing Kubernetes clusters, and how it is involved in AI conformance testing through automated processes. Overall, the talk aims to promote a community-driven approach to Kubernetes conformance that addresses the rapid evolution of AI technologies.

Hello everybody, thank you for joining us. We'll get started. I'm sure people will continue to trickle in. My name is Justin Santa Barbara. I have been working with Kubernetes for more than 10 years now since 1.0 type days. I'm a chaos maintainer sig cluster life cycle lead I guess and we're going to talk today about AI conformance and in particular how we did it with Kops and

some of the lessons that are more broadly applicable to the ecosystem. Oh, and I work at Google. I should always mention that. Arno, do you want to go next? So hi, my name is Arno Murky. I'm I'm the sig chair and technical lead for the Kubernetes infrastructure group which basically cover any infrastructure problem for the community. Also release manager and as of today I'm open source engineer

so which means I've been working for the Kubernetes project for a very very long time and happy to be here to talk about the AI conformance. Hi, my name is Ciprian. I'm a software engineer at Microsoft working on Kubernetes and the platform engineering. I've been a contributor and maintainer in Kubernetes projects for the last 8 and I'm happy to continue with that for a long time hopefully.

Hi everyone, I'm Janet and work at Google. You probably see me on stage this morning about AI conformance. Thank you. I'm I'm also working at Kubernetes for over 10 years. I know Justin for a long time. Very happy to be here to present the AI conformance and Kops. So first on what are the gaps that we that we need AI conformance. So Kubernetes was originally designed for

stateless and CPU bound web services. With AI taking off, being so popular now, we have new requirements. There's special CPU intensive workloads, a lot of different um requirements around scheduling, topology, networking. There's a lot of friction and fragmentation resulting from that. And a lot of the vendors are going in with their own solutions. This creates fragmentation. So, we realized that Kubernetes conformance is not enough. We used

to say you can write once and run it anywhere, but it doesn't apply to AI workloads. AI workloads are so important, and this is moving so fast. So, we need a conformance program that's faster than Kubernetes conformance. Um there's also um we also want to provide common capabilities and for the APIs for AI workloads to be able to run on. And then, so we announced the Kubernetes

AI Conformance Program at KubeCon NA last year. So, we set the standards for what capabilities that platform must provide to run AI in a reliable and easy way. So, these are the pillars of current AI conformance. We have the pillars around DRA and accelerator. There's observabilities about accelerator metrics. There's also scheduling, orchestration, things like gang scheduling and auto scaling. We care about those. And there's also networking

and the gateway and inference gateway is what I mentioned earlier in the keynote today. And there's also security, and then finally, we need to be able to support deployment of complex AI operator. So, we started a program, and there are a bunch of platforms already certified. But, they are mostly commercial offerings. So, how about we have a fully open source community-driven distribution that can also meet the

standard? We can make sure we are This can keep us honest. So, let's talk about Kops. Okay. so, what is Kops? Kops Mhm? Kops stands for Kubernetes operations. It's an open source project for creating, managing, and upgrading Kubernetes clusters. Has been around for a long time, basically since Kubernetes 1.0. Uh and is one of the easy ways to install Kubernetes with an open source tool. we consider

it production-grade, uh especially for cloud infrastructure. Um it's most of the time associated with AWS, but it has support for GCE, Azure, Hetzner, OpenStack, and some other cloud providers with some others coming soon. We use Kops also for Kubernetes conformance. Uh having an open source Kubernetes distribution that can run conformance keeps us Uh we don't want Kubernetes to be consulting uh etc. We We want We don't

want it to be the domain of managed solutions, so only cloud providers with big teams and big budgets can run it. Uh so, that's why we've been running Kubernetes conformance since the beginning of the program. Also, we've been involved with various other efforts to standardize uh setups, like it's used for scalability tests. Uh we've been running it up to 5,000 nodes. So, this is where the scalability

tests run these days for Google Cloud and AWS. It's in the open, so anybody can can see the results. we've been hitting a lot of the problems that people are seeing uh when they're running when they're trying to scale up their clusters, and we've worked with uh the cloud providers to improve things. We hope to be also one of the first OS open source distributions that will

run Uh we actually have Justin that will talk about that. Okay. So, this is a screenshot of uh how conformance uh tests look like for us. Uh we are running them for all supported Kubernetes uh releases all the time, once or three times a day, depending on how new the distribution is. And we've just added AI conformance to Okay. Why is it important to have AI conformance

in the open? AI conformance Kubernetes distributions as I was saying should not be the domain of cloud providers. So that's what we're trying to do and healthy open source makes managed Kubernetes is also better. Thank you. Yes. So let's talk a little bit about the AI conformance effort specifically within I think there is no one that has enough resources to do everything they want to do, right?

And Kubernetes is or Kubernetes is is basically a volunteer project with, you know, a lot of corporate sponsors. Kops is a volunteer project within that that has, you know, much less sponsorship. So we really have to lean in very heavily into automated testing. And I think that's actually a great thing. One of the things we want to do is avoid, you know, creating an effort where we

have to do a lot of boilerplate work on every PR or on every release, whatever it is. And we wanted to include the AI conformance program in that. So the automated testing is something we've leaned into very heavily here. And I think that applies also, you know, to anyone that has an in-house Kubernetes distribution, whatever it might be, even for the vendors, right? For the AI conformance

program, the same thing. Like we want to avoid toil. We want to be as efficient as we can. There is a little challenge in this in that the the Kubernetes conform There are two big conformance programs. There's the Kubernetes conformance program, which was defined by tests. And so the that was very clear as to exactly what we needed to do in order to achieve testing and of

conformance. For the AI conformance program, it was very much about capabilities and outcomes. Like, can I run AI training on a high-performance cluster? Do you want to Do you want a water? You okay? Okay, good. Um and uh so that makes it harder to do automated testing. Uh we have to figure out, you know, what should we test? Uh because it is not defined by an explicit

test. And so what we've decided to do instead is effectively we test common approaches instead. So, if you are using the NVIDIA GPU operator to manage your DRA configuration, then we have a test that will trigger on that and look for the NVIDIA-specific uh annotations, validate that the DRA with the NVIDIA GPU operator is is working correctly, all of those sorts of things. And so if you

pass one of our tests because you use, say, the NVIDIA GPU operator and we have written that test, then you can tick that box very easily. You get an automated submission for that particular criteria. However, there are other ways to pass the AI conformance program. You do not have to use the NVIDIA GPU operator. You can use a different accelerator, a different vendor, a different software stack.

Um so if you fail the tests or don't pass the test, that doesn't mean you fail the AI conformance program. It just means you have to go through the slow path. And so what we do is we create like a a fast path that enables people to do the We hope people will follow the easy way uh to AI conformance. So we encourage you to use these

things, but we don't require it. And I think that that That's the Janet has done a wonderful job here. Like, it's a very clever, sneaky approach to like getting agreement on behaviors without requiring like specific implementations. So I I hope we can use that same trick for other conformance programs in future as well. And so what we did in Chaos is we basically created one of these

fast path tests for every single requirement of the AI conformance program. Um We chose a way to configure Kops that, you know, it includes the Nvidia GPU operator. It runs, I think in this case, on AWS. Um that defined there's a script for that. It defines, you know, how to set up an AI conformance Kops cluster. It runs those tests and we have written tests that like

trigger on each of these decisions. So, I think it's Nvidia GPU operator, like the Prometheus stack, a bunch of those sorts of things. Um I also want to give a a shout out to Gartner. I don't know if anyone here is from Gartner, but uh Gartner beat us to the punch. Congratulations. They they are an open source Kubernetes distribution and they were the first ones to be

AI conformance and, you know, we certainly looked at how how they did it. They did a similar approach um for a lot of the stuff they did there. Um so, great great to see that as The way it works for Kops is uh you know, all the Kubernetes testing goes through Prow. or the vast majority of Kubernetes testing goes through Prow. Um it's we spend several million

dollars a year just running tests for Kubernetes projects. Anytime you send a PR to any Kubernetes project, uh you know, all those test runs are going uh through Prow. They're running on real infrastructure um with credits donated to the CNCF by AWS, GCP, Azure, Oracle, I think Nvidia this morning, many others. You know, so lots of lots of people do that and it is an an easy

way for us to get uh visibility and signal again without requiring using the most of the resources we do have, which are cloud credits and not necessarily a ton of people. And so, like you can look at the dashboard and see, you know, the conformance results, which are running every couple of hours uh for Kops. So, uh we can run them on a PR or we can

run them on and we run them every couple of hours. And we're able to take those results. the tests actually spit out the YAML test results in the way the AI conformance program expects. We've even created some automation to submit that automatically upstream to the AI conformance project or subproject of Kubernetes. And the AI conformance project actually has some approval automation that they run as well, which

looks for that YAML file. And so we're all building this this automated pipeline for for approval of of AI conformance together. So, what basically I would say the the beginning of basically what we're doing is basically, like I said, we care about transparency and we care about And in order to do that, we basically define opening tests to make it happen because we want the AI conformance

program has got behaviors. So, in order to make it happen, we had to write our own tests. Basically, that's run uh periodically to make sure we basically are AI conformant. But in order to avoid fragmentation between everyone doing that, we basically want to upstream those tests and like converge that to a single framework of tests to make sure that everyone leverage the same tooling for AI conformance.

So, there's no discrepancy between the different doing that. And that's useful for everyone willing to do that, whether you do it in an internal private cloud or using a public cloud provider, we hope it will basically provide the same guarantee about the AI conformance Like Justin said, it's about automation. We basically want to like automate the maximum to secure conformance verification. So, there's no it's not human

error prone in the sense like, "Okay, we have like something running. We have a pull request merged introducing a new test, and that's it. And that's how we basically define the reference configuration for everyone. So, any new I would say cloud provider, or anyone interested to basically do that, we just look at what was done through the those tests, and basically make sure, okay, this is the

reference configuration I can set up and basically run the ultimate tests. And the audience is basically for to be AI conformant. AI conformant means you have AI workloads running on Kubernetes, whether it's reference inference, sorry, or training, or distillation or fine-tuning. You basically want to be able to leverage tooling and reference architecture in order to make it happen. So, our everyone from people having sovereign cloud distribution

or existing hyperscalers to make it So, AI conformances or will hopefully always grow in the sense that we're going to integrate more tests, more requirements, more complicated use case in order because AI itself is so big and going so fast that we have to make sure we keep the same level of transparence amongst everyone basically willing to do deep learning or machine learning in their infrastructure. So,

the idea is basically for Chaos to uh always align with whatever new behavior is required for the next I would say need of AI. We can We can from focus on uh chat GPT-1, which was about basically how businesses can do training to now how businesses are doing agentic AI. And that's basically come with basically a lot of new requirement complex requirement different combination and the idea

is like you scale up as a as a pilot effort to make sure that the air we have like one tooling share amongst everyone and everyone means the audience in in need of air conformance so they use that one tooling that can do basically what's needed for air also at the same time make sure we basically introduce new scenarios of user story like basically how I do

at scale like we talk about five 5,000 nodes for just communities why not do that using chaos to do also that for AI conformance like how you do inference how you run I would say thousand of AI agent inside the same cluster how you guarantee that or make the promise of that for anyone willing to do that so the idea is basically make sure we basically follow

the track of AI requirements based on what's the business need in order to make sure that we have the same level of I would say transparency and guarantee about basically the promise we want to do for existing user chaos user or future people willing to leverage chaos so that's that's basically like we're going to see over time in the AI conformance program more requirement we come in

the requirement over time and we always want to align with that and with that if anyone's interest we basically online you can uh talk to subproject AI conformance or talk also to seek less life cycle plus life cycle wish on chaos basically if you want to be involved in that thank you everyone for coming and if you have any questions feel free to come to the mic.

I'll also be around after this for questions if not. Yeah. Oh, there's questions. You taking? I'm taking. I think. Hi. So, probably in the community there are Hi. Um We know there are several cluster life cycle management tool other than cops. So, is there any activity uh that other integrate a conformance testing or a conformance cluster setup very handy way or something? uh we hope so. I'm

I'm a sig cluster life cycle lead so we'll certainly be encouraging that. Um one of the things we're trying to do with our testing here is is create that test suite so that other like if cluster API for example wants to add support they can run the test suite and uh Janet is responsible for a a centralized test suite. We're trying to get the test suite that

we wrote for cops basically upstream to that. So, cluster API or whoever it is will be able to use that centralized test suite as will vendors. and we're going to reduce the barrier that way. Um I think that please express your desire to those providers to uh to to support it but I it is it is it's actually really good. I think one of the one of

the unsung virtues of the AI conformance program is that it tests um behaviors in a way that we've sort of stopped testing in Kubernetes you can pass Kubernetes conformance even if you don't support volumes or real persistent volumes for example. Uh the AI conformance, you know, if is is much more behavioral, as we talked about, and it is much harder to pass it without actually being able

to do real AI. So, I think that is a it is a great thing for the projects, all the projects, including cops, to uh like go through the exercise of making sure that all these pieces, when you put them together, actually work. And yes, you can do AI because even if if you're not using AI, you probably want cluster auto scaler and a working HPA and all

these things that, you know, it is good to verify continuously that we are able to deliver, whether you want AI or not. Okay, thank you.