Amplifying End User Voices: Platform Architects on the... Rajas K, Zach S, Kevin K, Elias T & Dawn C

This panel discussion focuses on the challenges and feedback experienced by platform architects working with Kubernetes in the cloud-native ecosystem. The speakers share insights into current user demands, emphasizing the need for improved node abstraction, scheduling capabilities, and simplicity within Kubernetes. Dawn Chen from Google highlights the complexities related to node abstraction, particularly in the context of AI workloads. Kevin Klues from Nvidia discusses the necessity for gang scheduling of pods and integrated support for GPU workloads, while Zach Shepherd from VMware emphasizes the recurring user desire for streamlined operations. The panel also explores the balance between core Kubernetes features and community-driven extensions, alongside emerging trends like confidential computing and the complexities introduced by various workload requirements.

Thank you for making it to this panel discussion. Uh a bit of a context. Um this is to gather around and have a forum for platform architects who deal with Kubernetes users day in day out to amplify their voices and talk about their feedback and how that translates into features, bug fixes, projects, um and it involvement of um cloud native ecosystem. So, I am uh Rajesh. I

am a software engineer at Broadcom. I work on VMware vSphere Kubernetes service. And also uh I'm part of the CNCF technical advisory group for workloads foundation. Uh I would encourage my panelists to give a brief introduction about yourself and then also tell us very crisply uh what is that one feature or capability that your users asking for that you haven't yet seen in the cloud native ecosystem.

Dawn, do you want to go first? Uh thanks. My name is Dawn Chen from Google. Uh I've been involved in Kubernetes even before the Kubernetes inception. So, I'm more focused on the data center node. So, this is why a lot of customer ask us now emerging problem. It is node abstraction. Uh so, even and especially in this AI emerging uh uh year, and the node abstraction is

completely broken. So, how are we going to make it automated those kind of works? That's the top problem uh users ask for me. Uh my name is Kevin Klues. I'm at Nvidia. I'm one of the original architects of the DRA feature and the the implementer of the DRA driver for GPUs. I'm also a sig node maintainer. Um in terms of uh uh, features that people are asking

for for our stuff, you know, I I think it's no secret now that the node is no longer the unit of compute. You have to think at rack scale and multi-node scale for getting the best performance you can. Um, and in addition to just general GPU features that we push out in our DRA driver, I think the the the thing that people are asking for the most

is having the ability to gang schedule pods and make sure that all pods come up as a group rather than being scheduled individually. And things like Kai scheduler make this possible, but we want integrated support in uh, Kubernetes itself to make this possible. And there's six schedulers work six scheduling is working on this right now. Uh, and we're looking forward to integrating that with a lot of

the other stuff that's going on. Hi, I'm Zach Shepherd. I'm the lead architect for vSphere Kubernetes service I think maybe the most recurring piece of feedback I hear from end users is a desire for simplicity. Uh, all of us are fortunate enough to be here at KubeCon learning for a whole week. Uh, I think a lot of end users don't get that experience and it's really hard

to keep up with what's going on and how to get the most out of their Kubernetes clusters. Uh, I'm Ellis Tarn. I'm a principal software engineer at Amazon EKS. Uh, I've been in the Kubernetes ecosystem for six years-ish now. Um, created the Carpenter project, worked on EKS capabilities, EKS Auto, Crow. And I think you're you're in for a bad time if you don't want to hear about

AI. The I have the same observation as Kevin as we we've kind of moved from this single kind of single pod to to node relationship where applications were very small, very fungible. And we're moving to this new world where AI applications span multiple nodes. Uh, many pods consuming many GPUs in concert and that's broken a lot of Kubernetes primitives, um, like eviction and and life cycle management,

roll out, auto scaling, scheduling. Um, so that's that's I think one of the core focus areas right now Uh, uh for Cool. I think there are a couple of areas over here so we can dive deep into each one of them. So, starting off with the feedback to feature pipeline. Um, DRA, Carpenter, a couple of things that were called out right now. These came up as projects

mainly because users said that something was broken. And now that they are in the hands of the users, how do you see the adoption? How has that changed? Does that surprise you from how it was initially designed? Kevin, do you want to go take that one? >> Sure, yeah. Um, yeah, two things you mentioned there. One being Carpenter and two being how we how things have changed

since we first envisioned things. DRA went through a lot of uh changes over the time from when we first conceived it. Uh and so we now kind of you know, what we what we call classic DRA now is the initial inception of what we looked at at DRA and it had this, you know, the the idea was that you have this third-party vendor vendor-specific controller that handles

all of the scheduling, all of the allocation of your devices, but it was this black box that um you couldn't really peer into. Every vendor could write their own. It was very very flexible, but it meant that things like auto scaling really suffered because now the auto scaler doesn't have a view into what this third-party controller is doing in order to kind of automate and discover what

uh new nodes could come online to serve the resources that that third-party controller would then um be able to allocate resources for. Um and so, yeah, that's probably the biggest change in how we initially saw DRA coming into being. Um uh and now everything is integrated in tree and there's a very transparent way of advertising devices, having the scheduler consume that, do its but I've heard recently

that even what we're doing now isn't necessarily ideal thing for auto scalers. And so, I I think you probably have some some more thoughts on that. Yeah, and my my friend Jason in the audience who's implementing DRA for Carpenter, uh, he probably has the most thoughts on it. just for for maybe to do a quick baseline on what we're talking about cuz I don't know if everybody's

familiar with with some of these requirements. I think the biggest, most burning one is is inference. It's that there's many many companies trying to do inference, many companies who have GPUs and have other accelerators, and are trying to, you know, shoehorn, or not shoehorn, but leverage Kubernetes as the orchestration platform for coordinating those workloads. And these workloads are unique in that they, let's say you have a

uh uh uh inference application, and that model isn't going to fit just on a single GPU. In order for inference to work, you that might need to be across a bunch of GPUs. So, you might have eight GPUs on a single host, and you might actually need eight of those hosts, each with eight GPUs, to fulfill a single inference request. So, we're talking about 64 GPUs all

responsible for a single response, and they all need to be interconnected with low latency network. They all need to be coordinated together. They need to be scheduled as close in a data center to each other. And so, that's that's kind of the the fundamental paradigm shift, at least, that's been like uh at the node level, making sure that all the the devices are available, that they're allocated

uh correctly, that they have the right networking hardware that can interconnect them. And then, at the scheduling layer and node life cycle layer, making sure that the Kubernetes primitives for application life cycle work on those work on those uh those 64 GPUs or those eight pods across eight hosts at the same time. Right. So, I'm I'm seeing that there is a need for the right separation between

all of these layers, especially uh node that you called out. And Don, you also mentioned node abstraction being one of uh the concerns. So, Kubernetes started with something that was built for say stateless web services and now it's supporting AI workloads and um HPC and whatnot. So, uh with all of the configuration surface grown in because of the stuff that has happened in the node, how how

do you see what belongs in core Kubernetes and what uh you know, the rest of the ecosystem? What's What does that separation look like? Yeah, thanks for that. This is great question. Um since I'm involved with Kubernetes before the inception, um I just please remember after Kubernetes by design from day one, actually we don't plan to build off the monolithic uh system to for everyone, right? So,

from by design from day one, it is it is extensible. So, um so we try to actively proactively define what this belong to core, what is belong to the uh to the ecosystem. We try to draw the line, uh keep changing that way because you are along with the Kubernetes. If we look at the history and the Kubernetes had to have a lot of development, a lot

of API, right? So, there's the container runtime API which it is make us to survive this uh transition from Docker to the Kubernetes orchestration, right? So, the same things and uh same things under the under the CSI for storage, right? So, Kubernetes don't need uh care and Kubernetes don't need uh care about the intermediate about the kernel, how to start off the C group, how to make

the C group, and how to start all the details. Just need need the container told the container runtime and such as the container D or other thing and they say, "Oh, make this container And also same times and Kubernetes don't need to know the detail about the underneath overlay, storage, all those kind of things, right? So, the same thing goes to scheduler. So, we build off the

scheduling framework so the people can plug in and to build their own efficient uh scheduling uh plug in and plug into those kind of the their Kubernetes orchestration. 2 years ago, when we realized of the Kubernetes, a lot of existing principle and uh cannot support not sufficient to uh AI workload and HPC workload. So, we propose like new of environment, right? So, we progress how we Kubernetes

evolve Kubernetes to support the uh HPC workload and also the AI workload. And the same with like the framework like such as the Ray and the Slurm. So, and we we identify what it is the pod life cycle or workload life cycle we need to evolve. So, there's the all the gang scheduling and also all those kind of the uh pod life cycle evolvement and in-place pod

resizing uh and um and auto swap, all those kind of things is being development. So, if you ask me to draw a line here and uh I I just recently share with many uh engineer in the community uh it is the mechanism Kubernetes mechanism this part belong to the Kubernetes core and uh uh but policy policies belong to the ecosystem, right? So, the differentiator so the our

ecosystem and our vendors can build off the differentiator around of the core Kubernetes. That's kind of what I draw. Sounds good. Zach, do you want to capitalize on this and talk about the configuration aspect? You also mentioned that making things simpler is what you're looking for. Yeah, thanks. I think when I look at the ecosystem, one place that's done a really good job of sort of separating

what's core from uh what's added by other users, whether it's other members of the community, operators, or end users of the system. I feel like Cluster API has done a really good job. Uh and for those who aren't familiar, Cluster API sort of has three different extensibility models. You can have uh providers, such as an infrastructure provider that integrates uh Cluster API with a particular uh cloud

vendor or cloud technology. Um and those are often maintained by other parts of the community. At the other end of the spectrum, you can have a cluster class, which is used to encapsulate a set of configuration options. And that lets a user, end user, express a set of preferences and encapsulate those so that they don't need to repeat themselves cluster by cluster by cluster. And that's kind

of a a different way to move uh thoughts and opinions out of the core. And then in between, you've got the ability to create uh what the community calls runtime extensions. And those are ways to influence how Cluster API is going to achieve its uh desired state. These could be adding life cycle hooks to, say, uh upgrade an add-on as a part of upgrading your cluster. Or

it could be uh an in-place update hook to be able to perform uh some configuration changes without a roll out. And so I think by having the sort of breadth of extensibility options to fulfill these different use cases, it's really been able to build sort of a thriving system where whether you're another developer in the community or an end user, you're able to to use it to

do what you need with kind of the core staying uh universal. Right. Um I think a couple of uh aspects came over Extensibility, configuration. Uh you mentioned the uh scheduler aspects as well. A couple of these were pointed out in one of the white papers that we are working on in technical advisory group of workloads foundation for scheduling challenges for AI workloads. So, uh that's also another

area if folks are interested uh to look at. Moving on, does all of you all have involved with have been involved with like user feedback and um escalations and stuff like that. So, when when an incident uh reveals a gap in in the platform, uh how do you decide whether the fix is supposed to be something to be done locally, it is is it supposed to be

going into the platform, or in core Kubernetes, or in in the wider ecosystem? Um Kevin, do you want to take it in? Yeah, sure. I mean, things go wrong all the time with GPUs. I'm sure everyone's very familiar with this, and it's always a constant struggle of do we fix this in place, roll out a one-off patch, do we fork some project make a change in it,

you know, give that to our customers, or do we, you know, have this upstream first mindset? And at least all the components that my team in Nvidia works on, we definitely always have this upstream first mindset. Anything we build, we plan to upstream. Um whether that's in our own components or in core Kubernetes itself. Um I think anyone that watched the keynote this morning, you saw that

we're now donating our DRA driver for GPUs to the CNCF so that, you know, people So, it's not just us in control of this, so that if something breaks, we're not the only ones that need to be responsible for going in and fixing as fast as possible. If one customer environment says, "Hey, you know, this is a problem. We need this fix." They can now work with

the community to get that in there without feeling like Nvidia might say no, or cuz now it's an open governance model where the community decides, and not just one company. Yep. Um Zach, what what is the outer approach to incidents? Sure. Yeah, so I this is uh probably going to sound familiar to a lot of folks in the audience. When something goes wrong, you do a retrospective.

You try to figure out really, truly, what was the root cause? And, uh, at least on our team, what we do is we try to break that down into uh, what are the you know, what's the root cause? How do we correct that true root cause? But, also, what are the other contributing factors? And what actions can we take to either prevent or mitigate similar issues in

the future? Uh, and then, of course, when you have a a series of incidents, you start to look for patterns. of course, when we find bugs, the starting point is to contribute the fix upstream. Sometimes, though, waiting for an upstream project to ship a release, backport the fix to previous branches, ship patch releases, sometimes our users can't wait for that. And so, it's sort of a an

upstream-first model, but a cherry-picked downstream when needed. Um, and then, when we start to see patterns that affect the rest of the community, you look for ways to build a tool that helps everyone. And so, uh, two examples of that recently are etcd diagnosis and etcd recovery. If you've ever dealt with a broken etcd cluster, uh, you can use these tools to help identify what's wrong and

how to fix it. And the idea here is to build reusable tools so that everybody in the community can benefit. we've seen, uh, overall concerns with stuff like scheduling of different workloads, uh, node abstraction, or, um, making things simpler, like an upstream-first approach to contributing to projects, um, having separation in layers and stuff like that. And while you expect the ideal state of a platform team to

be, you know, having all of these config management in place, observability, uh, the reality is some sort of a a bit of a distinction from there wherein there's configuration drift, security issues, things like um version sprawl and stuff like that. So, What does this gap signify between the ideal state and uh the the the gaps that uh the ideal state and what is the current state of

enterprise teams? Like, what does that gap signify? I think this is kind of like the existing from the day one. Since I mentioned that earlier, right? I said that we never want to build the Kubernetes as the monolithic for everything. And uh And so, this is why every there's the extensive part. Then, we have like a multiple implementation. And in reality, right? So, there's a lot of

When you When customer When you start to talk about in production, talk about end-to-end to support their workload, they always have the struggle with what kind of the solution they should uh they should choosing, right? So, how that going to What's the performance? All those kind of things. So, end up it is for platform team, even we try to build something we keep missing that. We build

something and run anywhere. But in reality, it is we build something tone everywhere. And that's kind of a light mile for the platform teams, right? So, that's We recognize this problem from day one. And And we The problem is we have to allow those kind of extension and a lot of innovation built around those instances. But at the same time, open source community, we are proactively identify

what it is reference approach and how do we support for each given of the workload like the branch work uh the batch workload and uh inference workload, uh analytical workload. What it is the reference architecture we can build on top of that. What it is the benchmark. What is the conformance test. And we build all around those kind of things. And hopefully that can uh help and

address a lot of end users' problem. All right. Uh Alice, do you want to Yeah, just just on that point like there's there's some things that are kind of necessarily different between providers like the bindings from you know, this abstraction layer to this vendor's implementation of that abstraction. And then there's other pieces that are just don't necessarily belong in core yet because they're not mature enough yet.

And there's there's kind of this this soup of of configurations and options and ways to put all the Kubernetes pieces together. Maybe we can rely most on what's in core Kubernetes itself. And where different vendors or different Kubernetes distributions or different platform teams or different companies are assembling product projects of of different maturity levels and of different vendors into their platform together. And that that in and

of itself is often a very large challenge. You have you know, maybe many of you are part of these platform teams at different companies or you're part of vendors that are trying to take on you know, a slice of that operational complexity and trying to vend something that works and composes all these primitives together. Um this is this is one of the things I'm excited about Krew

for as well as a way to kind of draw draw boundaries around these these integrations and and kind of package them. I think there's other tools in this space like Terraform and Helm. Um but Krew is kind of another example of a project that's that's not quite yet in Kubernetes core that we think maybe this should eventually land in core. Though right now it's just in a

SIG and is also kind of part of that how do we how do we solve these problems, mature them, get com- comfortable with them, factor out the provider specific implementations as we need to and and mature and and kind of graduate the rest. So, GitOps and adoption of different projects like separation of concerns would be the themes as what I'm gathering from here. I I mentioned the

scheduling challenges white paper a while back and some of the challenges that we saw were around cost efficiency and resource utilization and what is the unit of scheduling and how that differs from different workloads. Apart from that, there are some other challenges that exist in the ecosystem right now and there are upcoming areas that are coming up like confidential computing and stuff like that. So, where where

do you all see if I was you know, going to open the panel, like where do you all see the platform ecosystem or the cloud native ecosystem going towards in the next two years? Like what sort of projects do you see getting developed in the community to address some of the challenges that you've pointed out? Um you mentioned cloud confidential computing. I mean, that's a big one

that at least in video is already investing heavily in cuz we see these and especially with all these agents that are suddenly going to be, you know, wild on the internet, things like confidential computing and whatnot will will be going to become more and more important going forward, I think. Um always challenge as the as the wealth of the Uber technique at Google. So, but the for

me at this moment and emerge of the challenge, it is the I will pick up one. I will call the how to support the this uh disaggregated of the inference workload, right? So, today we uh basically schedule the inference request as the a Uh I can of we we all can see that the version and if we really want to achieve of the time to the first

token latency and also improve of the balance of the cost efficiency and utilization and performance all this together and I can see that within next two years and we are going to change it to schedule the inference phase instead of entire of the request. That's fundamentally like earlier I mentioned that no obstruction layer, right? So then you have the different type of the node and you have

to abstract something for different like some is for computer density, some is for memory bandwidth and all those kind of things and to balance those kind. So that's kind of what I'm currently facing and try to thinking about, yeah. Anyone else would like to take a stab on it? I'm excited for more agentic primitives in the Kubernetes ecosystem. So sandboxes is maybe one, but I think in

the same way that platform engineering is is kind of a it's the art of delivering guardrails to your organization for building applications. I think it's it's kind of naturally extends into the world of agents where organizations are trying to adopt generative AI and they look to their platform engineering teams to tell them how to do that and the best way to deliver those primitives to those to

that audience is through Kubernetes. So I'm excited to keep an eye on that space. proposal yesterday for agentic Yesterday I just I think it is it is it is yesterday. I just proposal something how to introduce new primitives to support agentic work nodes. Yeah. Yeah, look at the Kubernetes enhancement request. Yeah, there. Yeah, when when I think about the next two years, I mean obviously all of

that is is going to be a lot. And so I think maybe one of the biggest challenges for for platform teams is going to be keeping up with the pace of innovation. How do you stay current? How do you stay informed? How do you keep your your infrastructure constantly evolving? Um so I'm going to be really excited about projects in that So things that make simpler and

then address these challenges. I think that there have been uh a bunch of themes that we've explored today. Uh the challenges being around the different types of workloads that are coming in, the complexity of the ecosystem, the emerging areas that are coming up and then how would you address them? You would have a separation of layers. Node abstraction was called out. Uh building on top of fundamentals,

the uh that was called out uh I think at this point I would like to open it up to the audience and then see if we have any questions. Uh we have a microphone in the front. So if you have any questions, you could line up. Okay, I I don't think we have any questions, but uh If you were to pay down tech debt based on how

the ecosystem is right now, where would you start? I would rewrite the kubelet. It's that simple. Can you give us a why on that one? What's what what makes you hate the kubelet? It's not that I hate the kubelet. There's just the it's It does its job, but anytime you want to add something new, it's a battle to go in and figure out the right all the

right places that need to get changed in order to add any new feature because it's kind of been organically grown over the course of what, 12 years? Um, and now any new feature you need to add to it, you really have to have a mental model of how the entire cube that works cuz the code is I wouldn't call it spaghetti code, but it is all over

the place. Any other areas that anyone of you all redesign towards the evolution towards Kubernetes 2.0? I'm going to ignore the Kubernetes 2.0 part of that. I I feel like one area that I'd love to revisit is API machinery. I feel like the difference in behavior between built-in types and CRDs makes a lot of historical sense, but I I I feel like it leads to bifurcation in

a lot of code paths. That makes sense. Uh Cool. Uh these were some of the areas that were called out in the discussion today. So, if you're interested um cluster API got called out, work tag workloads foundation, working group device management is where most of the DRA work is happening, signod, uh crow, carpenter, these projects. Uh Zach also called out at CD diagnosis and recovery, so feel

free to get involved. And um scan the QR code and leave us some feedback on based on how you felt about this discussion. Thank you for making it And thank you for having the discussion.