How Will Customized Kubernetes Distributions Work... Michael M, Joel S, Bridget K, Jesse B & Bowei D

This panel discusses the potential of customized Kubernetes distributions and their implications for users and enterprises. The speakers, all from notable tech companies, explore the need for standardization across cloud platforms, emphasizing that users often seek a consistent installation experience regardless of the cloud service being used. They address the importance of Kubernetes testing, especially as it relates to cloud provider integrations, and stress that any distribution should maintain conformance and facilitate ease of customization. The conversation also touches on the possibility of community-driven distributions that serve specific workloads or environments, reflecting the diverse requirements of users who may favor optional features. Lastly, they consider how these distributions might balance upstream Kubernetes purity with tailored solutions, pondering whether a standardized approach could coexist with opinionated versions offered by major vendors.

Welcome everyone. Uh this is our panel about customized Kubernetes distributions and how they might help you. Um my name is Michael McHune. I'm a software engineer at Red Hat. Uh I one of the co-chairs of SIG cloud provider and I've been working on uh uh oh got notifications coming up. Thank you. Thank you John. Um, yeah, I work on a lot of cloud provider stuff, autoscaling and

cluster infrastructure. And so this is kind of a problem that I've been thinking about a lot. >> Uh, hi, I'm Joel. I'm also a software engineer at Red Hat. I'm also a SIG cloud tech lead. Um, yeah, I've been doing lots of cloud stuff for the last sort of five, six years with with Red Hat and the SIG Cloud Group. >> I'm Jesse Butler. I'm a product

manager in the EKS team. My voice is shot, I apologize. Um, I'm yeah at AWS now since 2020. Um, I've been in the CNCF and open container space for uh since the start pretty much and uh happy to be here. I'm also a maintainer of the uh the SIGs sub project Crow which is why I'm here. Uh but yeah, thanks. All right. I'm Bridget Cromhout and I

work at Microsoft as a product manager in upstream open source and uh I'm also a co-chair of SIG cloud provider and have been active in a number of other projects in CNCF and you know other SIGs and Kubernetes and uh I'm really excited about our collaboration here. >> I am Bowie. You might have seen me in other contexts as the SIG network uh chair and I'm sort

of representing Walter but actually I've done a lot of work in the GCP cloud provider G um I work on GKE. >> Okay. So I think you know many people in this room or in this community are familiar with uh Linux distributions right like Auntu, Debian, Fedora whatever and those experiences bring with them you know certain implicit uh I guess user experiences right there's installation there's graphical

interfaces that are different there are package management you know these type of things and although there are some commercially available distribution ions of Kubernetes. There isn't really yet a a customizable community curated way to do Kubernetes distributions. I'm curious, what might a Kubernetes distribution look like? What what features might it include out of the box? And what user experiences might you expect if this were a thing?

>> I'll take this one. It's interesting to talk about distributions because we you know you're thinking like well it's like you know it's going to be different but actually I think one of the key things for Kubernetes distribution at least for me is to have as much standardization as possible amongst all the different platforms that you're deploying on and I think one of the key things is

like we want to make sure for example the upstream Kubernetes testing works on all platforms and in a very uniform way and I think um I know that cloud provider extraction happened a couple years ago but like we really want want to make sure that coverage is maintained even though the cloud provider has been abstracted away. >> Yeah. And I think part of that too, we pulled

out some of the testing some of the testing scripts related from that that were very cloud provider specific. So yeah. >> Yeah, >> I I agree that um we don't want to lose the benefits of standardized testing. I also think that while we talk about um customization, the reality is a lot of enduser companies in general are looking for something that will work. And so I think

it's incumbent upon us to make sure that the testing and all the validation and just making sure that there's still conformance etc still exists. even if we're putting things um to the community and saying go ahead create something we can't be irresponsible and say create something yolo it into production good luck have fun hope it works like we can't do that >> yeah for me that's that's

kind of the tie-in why I find this really interesting as a topic because we you know I work on EKS uh it's a big managed product and you have big production clusters that you run in it but you don't always need an EKS cluster you don't always need a GKE cluster right you you might just want to spin something up in a in a pipeline you might

want to just test something really quick, something on your laptop or on a Raspberry Pi or whatever it is. Um, if you if it matters to you, it would be nice to know that the bits, you know, align to the source at a specific tag and there's tooling that supports it and there's documentation that tells you what it is. So, to me, that's kind of the root

of what a DRO is. It's like bits, tools, docs. Everything around that is just icing. Um, and I think that's sort of one of the the thrusts of like I don't I don't want customers to have to kind of pull things together, but it's also not something where they're saying, "Hey, I want to buy this product." It's not. So, it's like supporting the community um around organizing

something like a DRO feels like a really good step for the community um beyond just testing. >> So, I was going to touch on like the installation experience because I think that's something that So, historically, I used to uh be the team lead of the Open Shift cloud team. So we integrate with seven different platforms and you would not believe how different they all are. Like for

me the the key thing here is we build something that is consistent and where I can do the same command and I can go to any of these cloud providers the longtail that we support in SIG cloud and it's just the same experience on each one so that as an end user I can just be like right well I did this on GCP the other day I'm

going to do exactly the same thing on AWS and it just works >> and if it doesn't work Joe will open an issue on the cloud provider repo and get it fixed. So you mentioned like an installer right and you know and that's a complicated thing and that's a big a big thing to consume but you know when I think about this kind of selfishly in the

in the perspective of testing and whatnot you know one of the things that I want is like you know I want to say can I have an AWS distribution and know that the cloud controller managers are in place and the storage you know controllers are in place or could I have a GCP you know distribution that has the GCP you know cloud controllers in place. And so

kind of in that context, what would you expect in terms of, you know, impinated defaults or primitives that would come from a community distribution and do you think those would be different or would they be modular or or what would that look like from a community perspective? >> I can give it a shot. So I think one really good thing that Kubernetes has is the conformance, right?

So even if you for example on your platform have specific opinionated things and you must for cloud provider integration we still have right this like big swath of functionality that's going to be tested by conformance and is going to be common and portable right so it's like even if you have those custom bits for your particular cloud provider that's okay >> yeah and building on what Bowie

is saying about conformance uh we're also now in the space of Kubernetes AI make sure that we >> put the AI in there. >> Well, first AI mentioned. Okay. Um but we got to make sure that um we continue to pay attention to not accidentally building something that'll make it more difficult for Kubernetes to meet today's new workloads. >> Yeah, I um kind of again I think

that's exactly right. And and when I look at think about tooling and I think about why you would have a DRO to begin with, let's talk about agents for a second. So I could just point Claude code or Kira or whatever at some bits and say you don't even need docs. You already know what you're doing. YOLO. And then I spend like 180,000 tokens before I even

get to the point where I'm like, oh, I need an API key. Right? So there is a certain um element of efficiency that would come through something where you're like, hey, point it at this tag and you get that cluster wherever on any cloud provider, on any substrate, right? I think then when you get to well what does it look like on GCP versus AWS then you

get into sort of the opinionated things like we might say well you need autoscaler on one because you of course you're going to need it you need carpenter on another because of course you'd need it and you sort of say um you know like where you land matters yeah so I I think that that's another part of it too is like you can have these opinionated things

that just become kind of the default in that environment which we do have for Linux right >> right in spades Yeah. >> So there's a couple things I want to tease apart there, right? So like first, you know, we talked a little bit about testing and and kind of that effort the cloud the cloud provider um you know extraction was one of the things that we did

and and we pulled out all the tests and and looking at those tests they were very bespoke, right? And they and they very much had to know what platform they were running on. And so one thing is is kind of like making these tests generic so that we could run them anywhere, right? But I think another interesting thing that we're talking about here is like a purpose-built

Kubernetes right so you talk about agents and it's like what you know what would become bundled automatically that might you know >> so the way I'm thinking about this kind of problem is that we there are lots of commercially available distributions I'm not going to name any but obviously you've all seen these and they come with everything out of the box that gets you a cluster whereas

if you're going to build like cube the hardware and you run it yourself you've got basically nothing but there are some elements you need CSI CRI CCM probably um I don't think we'll ever agree on like here's a specific set that I want. There are too many different use cases for cube. So I it kind of led me to feeling like those commercial distributions are like the

full fat version of Coke and maybe we need like a diet or a light version of our distribution so that people can have their own like choice of like I want the absolute bare minimum zero >> and then there's a full fat dro as well. >> I really want a glass of Coke that has fat in it. But and on that same line then would we see

would we see something really extreme like uh the the gen two of Kubernetes where it's like you know I'm going to build this whole thing from source on up and I'm going to pick every option and bundle it together to to deploy in my cluster you know it's kind of going in the same way right >> yeah and there'll obviously be the full fat agentic version as

well >> right exactly exactly so I guess you know when we start to talk about these you purpose-built Kubernetes or or or customized distributions that would be for a specific you know kind of usage and I think this is really important in given today and and where we are and what we're talking about but like digital sovereignty has become a really big topic and what what could

we do or what how could this open the ecosystem to start creating things um you know I think we were talking like how open claw has really taken over because of its ability to kind of bring it into your house and run it your own. Would we see something like that in a Kubernetes distribution? I can speak to that like given I mean looking at our users

um on GCP that I know the best um we do see that people have for example if they have very specific requirements they're going to take a look at the you know what is in cloud provider and then for specific things talking about opinionated options they will replace it with their own specialized thing and create like a custom distribution. So we see that a lot like for

those who have very specific requirements for example sovereignty like they will basically put this together and making that easy for users I think is very key. Yeah, I would say that um in Azure we definitely work with many European customers who are looking for you know sovereign cloud and they're looking for uh not just um maybe the code runs in Europe but also are the maintainers European

and so I think that if we end up with um like we want to make sure just I think back to when Kubernetes started and there were you know maybe a few people at a company on the stage that could actually cut a release and the community has done a really good job of allowing the release team to be in multiple geos across the world um not

dependent on any one employer, not dependent on any one country. And I think that that's the sort of thing that we're going to need to push for in this space too is any kind of um distribution out there needs to have uh the choice has to be in a specific geo to be able to do what they want to do without people in another geo being able

to stop them which is hopefully what open source is about >> yeah absolutely and I you know I think also it's interesting to me from the perspective of you know when I look at technologies like uh Azure stackhub for example right which is is a very kind of onrem solution that someone you know could bring into their organization and that's the kind of place where you know

there might find there might be value in saying okay if you're going to if you're going to take this Azure stack hub and build some sort of you know internal structure on it is there a distribution then that's like built specifically for that right all right let me look up the next one all right so now that we're talking about all these customizations and everything right And

we we have tools like Helm now that can it's kind of like a package manager in a way for Kubernetes. But how can we make it easier for the people who would use these things to make their multicloud journeys the way they want to? How how would we build a distribution system, you know, without having to invent an installer and another package manager and some sort of

interface? Like how how would we make this customization possible for people? >> I hope you're not talking about the 15th standard. I am absolutely not. >> But I mean there are plenty of patterns for deploying cube already, right? Like so uh spoiler I also work in the cluster API community. Um cluster API for example is one kind of relatively uniform way of deploying cube across many different

platforms. The add-ons and the kind of like stuff that goes on top they have their own way of doing that cluster resource set and you can kind of say well I want to deploy this across all of these. Maybe that's something we would go down. Maybe we would have like Helm in the clusters. Again, it's one of those things of like how how many different installers can

we bake into one distribution? Preferably not many. >> All of them. >> Well, and can we even agree on what installation means? Right. >> Installing Linux is one thing, but installing Kubernetes is something different. I do think like there's probably like two different problems because going back to testing I think we do need in the upstream probably to converge on some small set right because that is

like a it's like that is probably one distro in some sense abstract sense so that we can kind of do the conformance. Yeah, I think like if I think back to Linux distros and when we kind of first started assembling these, we had that same question and I think the community answered it or you know whoever came up with uh you know Fedora and Debian and you

had then re and AP and there's like two different packaging. You start seeing between the semantics of these dros. Oh, that's what makes them different. And then I think um you know users choose what works for them. you know, I I I particularly like the the RPM packaging system and I package my software that way. So, I'm going to use a DRO that uses RPM and then

we have yum and then we have dandified yum that comes along and you have all of these iterations of that thing and it leads um you know from everything from re for commercial applications all the way down to like silver blue and like interesting things all the way down to rawhide if you want to pick up nightly builds but there is a consistency there. So I think

in Kubernetes we do have these decisions to make right you do have people that write a single binary and they spin up things you know on bare metal and you have people that like to use Helm and continue to use Helm and expand that community. You have um other people that are leaning into IA and Kubernetes with things like crossplane or ACKC as like maybe using Crow.

So I think you you might actually have the community decide how many distros you have because it's going to be based on that flavor, right? It's not like the source will be different. The bits definitely shouldn't be different. The docs probably aren't different except around tooling. So you have like tooling and packaging and if you think about Linux distros, that's sort of that's sort of the difference,

right? And then you have all these cool things like Arch Linux and all these other things that have I've been left behind. Um but like I think that that is what we would see is eventually you have yeah this core thing as you're saying is sort of like this foundational thing and everything else goes from it. >> Yeah. Like because the if you look at the cloud

provider repos it's like yes it's like an artifact but it's also a collection of best practices right and like for for example some users I know like they do look at that and then they basically they have they may have like a large team that's actually doing their own dro but they will look at that to kind of build their stuff >> like as a starting point.

>> Yes. Right. >> Right. Because if you're going to run if you're going to run Kubernetes on a specific platform, you don't want to have you don't want to be missing features, right? Like at the most simple level like uh you know load balancer type services, right? You don't want to install on one platform and have those and then install the other one and it's not working,

right? And having that similarity across them. >> And not to over abuse like the OS DRO comparison, but that's kind of like, oh, I'm on this architecture versus that architecture, so I need this kernel build, that kernel build. I have these um you know hardware add-ons and extensions. So I need these certain drivers, right? >> Apparently my laptop is doing something weird. Do I Do I need

to turn notifications off? >> It's trying to sleep. >> Oh, it's trying to sleep. Me, too. >> Um >> we >> Yes. Yeah. And I'm failing terribly. Um so, you know, we're talking about kind of these options and and features and whatnot. And I think and when we look at the testing world, we kind of know what we want, right? I want to be able to install

it and have CI built on top of it and kind of know what what's what, right? But as we start to congregate around the idea of like common ways to do these things, how on the rails would you expect that to be? >> I'll answer short because I've talked enough, but I think honestly it's that's the end of the sentence I said before where it's like, yeah,

how how adventurous are you and how push button do you want it? That's another flavor of a DRO, right? Is it Arch Linux or is it real? >> Right. And I think the Arch Linux comparison is really interesting and I know we are overusing the Linux Kubernetes, you know, kind of comparison. But I heard, you know, someone said this many years ago about, you know, Kubernetes is

like the operating system for the data center or something like that, right? And it it can be viewed that way. And I think in that in that vein, Arch, you know, I was listening to two colleagues have a conversation yesterday about which package manager they were using for Arch. And it was two package managers I had never heard of before. And and one of them one of

them um you know my colleague Josephine she was like yeah this is this is a Rust implementation that I got into maintaining and it's you know it's six months or a year old or whatever. And I would love to see that kind of excitement in Kubernetes, right? Where people are being able to do this. And if there were like an arch Linux of Kubernetes, like what you

know, it's like you might expect that almost, right? And this is on the opposite end of the kind of minimalist idea that you're talking about, right? >> Yeah. I mean, some of this conversation going back and forth about things like, oh, would we have helm and stuff? And I think that's definitely diet level. That's not in the zero level of my my distribution, >> right? Um but

yeah I the problem is there are so many different standards. There are so many different ways to install the software like uh while I was thinking about that operator life cycle manager came into my head as well like there are so many different opinions that we are I think we do need to set like a standard and a best practice like B was just saying like people

look at the cloud provider repos and I think one of the issues we have there is that they might look at the way it works in GCP and let's say they want to go look at AWS as well or Azure. I don't think we're consistent even between the cloud provider repos on how we set those best practices out >> or just the layout of the documentation >>

the layout of the docs the layout of the repo any of the layout like the discoverability as an end user looking at those different things I have to rediscover each cloud >> the release cadences like >> well and so the inspiration around testing right as this is kind of like is Kubernetes just Kubernetes are we only testing the core pieces of Kubernetes or do we need to

test these integrations that necessarily happen on different clouds. Uh >> oh, here there be dragons when you're making the test the test grid matrix of infinite complexity. >> Yeah, absolutely. >> With all capital letters we do we need man pages. >> Coup pages. I don't know. >> Does Kubernetes need man pages? I' well like portable documentation like one format. It's in the bundle. >> I mean some

people are using agents for this though, right? something they can communicate using natural language or >> All right, we're closing in on the last 10 minutes here. So, I would if anyone has questions, I would invite you to please come up to the microphone and we'll make one available. Um, and if not, we'll just keep going. Any questions? All right. >> Surprising. >> All right. >> Oh,

John's gone. >> Oh, thank you. I was here to heckle. Now, >> so so okay, you mentioned conformance. Um, conformance is >> might be asking you >> a fairly I mean it's a it is a like it's a minimal set, right? It's a it's a right it's it's not intended to do be an exhaustive set of tests. It also only tests mandatory GA features. So arbback for

instance isn't part of conformance because not everybody you know it's not as it's an optional feature like how in a H we talked many years ago about like creating optional conformance profiles that like so that you could test arbback so that like a district could say hey if you if I if you implement arbback it should be done the same way across different kubernetes. Have you thought

about reviving that kind of idea or is it more like you want to because also a lot of the conformance tests test the APIs without testing all the semantics of the APIs like like what is your depth of testing you would suggest for for this kind of um distributions? >> I think that we probably all have an opinion on this. I will say that when you start

looking at conformance and you're talking about musts and shoulds and to your point about optional things, I think the instead of mandating, I think it's a lot more important to be very clear about letting people have a programmatic way to query what's actually there. And if they can get answers, maybe they won't be the answers they like, but at least they can make the right decision. Um,

but if they can't get answers and it's an ambiguous situation, that's I think where people are going to be pretty upset about not being able to safely choose to integrate something. So, I think this is quite an interesting question because I was um I hope my boss doesn't watch this. I was having an interesting conversation with my boss a couple of weeks ago. Um, he is he's

all over AI at the moment and he's been very excited by being able to leverage Claude, bearing in mind he hasn't written a line of code in 10 years. Um, I'm just looking around all the red hatters. Please do not talk to Chris. he spent a weekend uh getting Claude to rewrite Kubernetes in Rust and he came in on the Monday morning and I had a chat

with him and he was like it passed The fact that he could do that right in a weekend is kind of like impressive but also scary. But it kind of highlighted to me was like wow our performance suite is easy right that we're clearly missing things. And I did I mean I did say to him did you actually try anything? And he was oh no I didn't

try anything but it passed conformance. So yeah unless we want rostanetes coming out of claude next week I think we should probably be looking into extending our performance suites. >> There there is a certain element of of compliance and conformance that relate to like is the shape of the shadow cast on the wall correct? And then there is how deep do you go down that rabbit hole?

And I think you're pointing to that as saying, yeah, well, if you've if you've enabled an optional feature, is it is it like a free-for-all? No, it shouldn't be. Right. So, I think I think you do have that option of kind of revisiting how we think of that. And, you know, potentially saying um where the the musts, shoulds, coulds, and you know, wills, like all of those

words, nouns and verbs come together to say like if this feature is enabled, which is optional, then it must pass. You can't have a should pass. It should have a must pass, right? And then the depth is like maybe you have different you know varying depths of of of implementation that you that you um >> I will make just an interesting analogy that is exactly the gateway

API extended feature definition. So >> I did it >> there there is some precedence on that. >> First of all, I love a good Plato's cave an you know analogy. So thank you for >> You just wiggle the laptop. >> Oh. All right. Sorry. Um, now I've lost my train of thought, Dave. >> Yeah. Well, I mean, now go on. I I was I was thinking about

something else and then the laptop interrupted me. So, >> um, well, no. Okay, I caught it. So, thinking about conformance, right? And and for me, like I'm coming at this from this very hardcore testing perspective, right? because I'm I'm remiss to see that we're regressing on these tests that we used to have because they require require a cloud provider integration and you know as a community we

acknowledge the complexity of this and we decided to extract everything from the Kubernetes code that had do you know we designed CSI a long time ago and that that's been great and now we're doing it for the CCMs and we're starting to come into these challenges right and so when I look at the conformance especially on these you know featur features, right? We'll go back to services

that provide a load balancer behind them, right? We're building uh a testing interface now that can be used by all the cloud providers where when they build their tests, you know, they can kind of answer these questions. It's like, do you do this? You know, do you provide load balancing service? And then we can kind of, you know, run the test or not. And I think the

long tale of this because there used to be many more networking tests that kind of tested some of this functionality that the cloud go ahead. No, no, I mean you know you know they had to be done >> service does actually test conformance >> yeah the the conformance suite does test the services and the load balancing but you know the I think the thing is is that

it the long tale of this in my mind is that as we build these tests out more and more and we build these interfaces then we'll naturally start to learn almost which distributions would be able to do which things right because we could look at what they turn on and what they don't turn on and then we could you know we could we could see what what

conformance they support or what what features they support at the least. So hopefully not too tangential but um the you said the testing framework you're working on has this sort of if how is that discovery done because we have the one of the things we're doing with DRA is there's there's all these different drivers and we want to do some some sort of uniform testing across those

drivers. It's sort of like gateway API, right? Like and and we'll have there are certain features we're like, well, if if this if it's appropriate, if it makes sense for this device to publish information about which PCIe bus it's on, then we want to check that you're doing that. But like, you know, there are devices where that's not appropriate, right? It doesn't even make sense. So, we

can't just test it across the board, right? So, like I I'd be interested in talking more about like what how your framework is doing that and how we might make use of >> that. Yeah, that would land in the optional optional enablement, optional integration, optional feature, like whatever it is, you've got to allow for that if enabled then must, right? Then the question is, do you have

variable specs? Do you have some, you know, could you do a deep compliance check and then also just for one thing just check the length of the shadow, right? Maybe that's just good enough like does it exist good enough? So I think you've got to make those decisions and that's where we get to the sort of use use case by use case and also maybe potentially the

environment that you're running it in. >> you're going to have environmental inputs on that as well. >> And and but to Bridget's point like is that discoverable? Like so could could I as a as an ex I want to deploy my application against this distribution. Is there some file I can read to make sure the distribution can meet the criteria? I >> I think that's probably something

that we'll need to figure out. And I was having a conversation with someone about this in the hallway the other day and and and they were talking about the notion of yeah it would there be a way for me to look at the nodes in my cluster and know that they were you know I have this distribution installed or something right and I think that as we

figure that out we're we're going to kind of learn that you know from from the perspective of the testing this is all being done like programmatically through interfaces and everything right so it and I'll I'll just go deep into the code for a second here if we look at the old KK the way we used to have these tests in Like for example the node life cycling

test you know if you have a CCM installed and the instance goes away the CCM will detect that and then you know it will mark the node so that you know it's gone right and if you looked at the way the tests were there were script files that said like okay call out to G-Cloud and tell it to kill this instance right and okay we we don't

want that there so it in the new way of looking that that what we've been doing at Red Hat you know with one of my colleagues is extracting the the tests from KK putting them into the cloud cloud provider repo which is you know kind of the root framework for where CCMs can be built from and setting up a test end in there and then we have

an interface and that interface enumerates all the different kinds of tests we're doing and then each individual like concrete implementation of a cloud provider when they want to build their endto-end tests they can you know include the core cloud provider as a dependency which they're probably doing anyways and they can use that interface to communicate about which test they run. And then when we run the test

harness, and this is where it's going to get really complicated. When we run the test harness in a future where we have a distribution, CI says, "All right, I want an OpenStack distribution." Okay, boom. You've got an OpenStack distribution with a cluster. I want to build the OpenStack CCM test. So I take the OpenStack CCM, I build it with the main of cloud provider, and I deploy

this on a cluster that has the main of Kubernetes. And then I run the test binary, right? And it will during the test binary, it will enumerate what features are available and what features aren't available. Now it and again I'm really just handwaving and you know doing my Steve Jobs uh you know whatever like reality distortion field in the future right maybe there's a way to take

the information that comes out of those tests and synchronize it with the way we're doing the distributions so that once the tests are run we have a like a machine interpretable way to say this is what it has you know >> there's a question >> was there one more question yeah I think we're we're over time can we take one >> Um, Kubernetes um is made to

be flexible and extendable. Um, and every major vendor has a opinionated Kubernetes version. And if we design a standardized um distribution, however you want to call it, where should we draw the line between upstream purity and opinionated um Kubernetes or should we get rid of a opinionated Kubernetes? Um if yes, aren't you killing your own business model? >> That's a great question. So I I can just

speak a little bit like I mean there are a lot of users who don't want to make the choices, right? I don't know if you ever like had to remodel your house or something and then you realize that there's like a million choices that someone has to make and that one person is you. And like I think for that the managed stuff is like great but of

course there are also these users that they really need to customize very highly customize specific things. Yeah. That's when they need to kind of assemble it themselves. >> Yeah. And even for the you know the larger managed products that I know of there is a a substrate of Kubernetes there that is common. Um that's what the conformance is. So the stuff where we get customized is really

making it work in a particular environment on a particular hyperscaler. Um and so I think that's yeah we we get down to sort of the optional features being things that maybe should be up to you. Um but I think that that the root of conformance for um for a project like this this this large is just sort of a responsibility that we carry. >> And keep in

mind too that it's certainly possible to take the images from um you know a managed service and then customize those people do and run that with their own extra uh requirements. So I think that having the ability to check and see exactly what this is meeting is a really important component. >> Yeah. I think and to the question too like this kind of blends in with what

Joel was saying about if we can come up with like a really minimal way to talk about a distribution, right? Maybe that becomes the foundational building block then that other things can build on top of and and maybe it would actually it would actually be a you know cost-saving measure for the different uh vendors who put out their own distributions to say well if we agree to

use this minimal building block we kind of know what the foundation is for everything right so and then we can collaborate together like we've been doing on Kubernetes to solve some of these other problems like how do you do install population, you know, those kind of things. And I think I think we're pretty over time now. So, thank you all for coming out and listening to us

pratt on about this. Really appreciate it.