SIG Apps Updates: Building the Next Generation of Kubernetes Workloads... Maciej Szulik & Janet Kuo

This talk discusses advancements in Kubernetes workloads, focusing on the Special Interest Group (SIG) apps and their recent developments. The speakers, Janet and Mache from Google, highlight new beta features introduced in Kubernetes 1.35, including mechanisms for stateful sets and deployment pod termination. They explore enhancements aimed at improving AI standards within Kubernetes, reflecting on feedback from users and the importance of community contributions. The presentation covers job modifications for long-running processes and the role of the batch working group in refining Kubernetes functionality for machine learning tasks. Additionally, they introduce new projects like the MCP life cycle operator that manage the life cycle of server workloads, alongside a showcase of the agent sandbox project which facilitates the creation of isolated environments for development.

Hi everyone, welcome to the SAPS talk. We are going to talk about building the next generation of Kubernetes workloads together. So what is SIG apps? We focus on the application use case and then we own the core workflow APIs including the deployment stful set jobs and chrome jobs and demon set etc. And we also develop the application level tools and standards. And we are also exploring more

uh Kubernetes native standards for AI given that AI is so popular now. And then before I move on, let's introduce ourselves. I'm Janet. I'm from Google. I'm one of the lead of SAPS. >> Uh my name is Mache. I'm also co-leading um SIG apps with Janet. Sadly, Kennet is the third uh person in charge that you should be chasing if you want to get stuff added within

the our preview. Uh Canon couldn't be with us. Um but yeah, that's the the three of us. >> So, we have a few new beta features in 135. There's a maximum available for staple sets. It's for faster and more flexible row out. It improves the roll out speed. It was enabled by default before, but because we discovered some edge cases that's not covered in the feature yet.

So, this is disabled by feature uh disabled by default for now. And then huge shout out to HA. >> Yeah. who helped implement this feature. >> Yeah. And disablement is like a fresh thing because we found a problem literally like a week prior to CubeCon. We started look looking into it. Thanks to Philillip for noticing that there were some additional edge cases and after discussion back and

forth about the direction we decided that um we know that there is a problem and we want to make sure that all the edge cases are covered. There are some significant uh problems that we will have to figure out. We will be rolling back uh 136 will have it disabled off. The next uh point uh point version of 135 will also have it disabled. So if you

are relying on those features being on, you'll have to make sure that the feature gate behind it will be uh will need to be enabled. Our hope is that if that uh that the work that we're currently um ongoing, we will be able to address all the po possible problems and and then within like 137 and 138 we'll be able to move this forward. Uh but that

also shows how important for us is the feedback that we're getting from you the users. uh because it was actually uh some person showing up on our slack like listen this is the problem that I were seeing in a max and available in a stateful set and we started going back and forth they only after only after they described us what the problem is how they approached

it we were able to nail down where uh where the problem is how to fix it and then eventually uh move forward with it so uh big shout out also to the people I can't remember their name uh but they they do reach out to us and we were able thanks to their uh feedback address the problems. >> All right, the next beta feature is about terminating

pods in deployments. We are adding a new status field for terminating replicas so that you can see which um how many pods are being terminated. And then this is part of the the other cap the deployment pod replacement policy for being able to tell uh whether you want the termination to complete or you want it to roll out during the termination and then also again huge shout

out to Philillip for helping with this cap. Yeah, especially that if um if you recall our previous discussion about this topic, uh you probably noticed that we've been talking about port replacement policies for I would say two years by now more or less uh only recently when we started to moving certain pieces of that functionality uh forward because we realized that the terminating replicas in deployments and

the replica sets we can actually promote that particular feature and make it available by default for everyone. And in parallel, we will be figuring out how to properly address uh the pod deployment uh the pod replacement policy and deployments. There's a lot of discussion that we're currently having around the implementation uh because on one hand the implementation that Philip put together is great. The downside is it

is changing pretty significantly how both the replica set and the deployment controllers are working which has its own downside because we are not 100% sure that uh something is not broken in a similar way how I was just talking about the mox unavailable in stateful set so we're trying to figure out the best path forward uh but the split allowed us to move one and the other

in different uh at different pace Yeah. So, Phillip is actually here with us today. Let's give him a >> We need more testing. So, >> yes, we need more tests. Uh >> interested in contributing. >> Yeah, definitely. >> And that and that applies equally for stateful sets, replica set, deployment sets. We want to make sure uh that all the edge cases and this is what we're Philip

and I were working for that fix for the u moxen available uh the code changes is like 20 lines of code but I'm pretty sure we are close to 200 lines of just tests uh but that only shows how much behind the stateful sets because if you remember stateful set has been in maybe not necessarily maintenance mode but it they were like nothing has been happening in

the stateful set world. We have a couple of additional changes that we want to implement, but getting those tests and those edge cases resolved allowed us to review the controller and also ensure that certain missing bits are being covered and are being built on. And testing and just adding tests is probably one of the easiest way for you to contribute to cube and to the sig apps

because changing controllers is very hard and I recognize that. Uh but adding tasks and slowly making your way through is definitely um something that is viable and something that we very much appreciate. Okay, I'm taking uh I'm taking over. Um so when we wrote originally originally jobs we figure out that there will be certain fields within the job resource that will say okay once the job starts

running you cannot uh modify certain fields that was um uh parallelism was one of the first that we allowed eventually modifying but you could not modify the number of completions you couldn't modify the pod template you couldn't modify the resources but as We started working with the batch working group. Who knows what the batch working group does? Okay, so batch working group is basically uh one of

the working groups that is uh that was spun out of the sig apps. they are um figuring out where Kubernetes is missing certain functionalities uh to allow running various ML and I'm going to purposely do ML and AI uh things on cube because they are handling anything from um HPC workloads all kinds of training to all the AIS currently that are happening so a lot of the

in uh improvements that were happening within the the the job that I'll be talking about a little bit more over the next couple of more minutes. A lot of those work is actually was uh based on the feedback that they started talking with other users who are for example using Kubernetes to train. Um so one of their requests was basically I want to actually modify uh the

job to be able to fit it or adjust its performance because some of those jobs might be long running sometimes for a week or even longer uh training jobs and over time certain uh possibilities appear that they could modify and so we came up with oh okay uh we don't want to allow uh fully randomly modifying because suddenly then you're having a nonunified job where it did

something with certain constraints and then suddenly you're um you're doing it differently. So what we figured out was what uh we will allow you to pause well in in a job's case it is called suspend the job and only then when we will actually terminate all the path that has been started by then will allow you to modify certain fields in this particular case this is talking

about the CPU and the GPU and the memory uh to more efficiently use the the the platform capabilities that you have available for your training up. So that's one thing. Uh additional changes that has been slowly rolling from the batch working group was adding um the back of limit per index. So if you've been using jobs in the the original fashion how it was originally wrote uh

you probably tried something that uh you probably noticed that in the spec in the top level spec field there is a backoff limit which by default is six which basically means if my pod in a job fails I'm going to retry uh retry that that particular pod six times but the number or the setting is a global one if your job is running I don't know 10,000

100,000 pods and you set it to whatever the number you can come up with whether it'll be 10% of that or something like that. It is not uniformly spread across the entire job. And if you're running an index job uh where each pod um has a unique index and it will be retrieded within its own context kind of like similar to what state flow set does. there

you can have the ability to say I want to have the ability to specify like the back off within that uh within that particular uh index. So in a normal case if you're going to if one pod will exhaust all the entire backoff limit uh budget the entire job will be considered a failed. Whereas with the index you can only modify and say oh yeah this one

particular index is a failed one because it reached um the backoff uh budget but all the other are free to continue working and in combination with uh job success and completion policy where you can say oh uh for my index job I'm considering this a success if it reaches like 99% of executions you can uh you can entirely ignore that one uh failed pod. So those bits

combined together allow you to more efficiently uh use Kubernetes uh for various training jobs or large scale uh job executions. Um we talked about the um the pod replacement policy for uh pod deploy uh pod replacement policy for deployments. Um and interestingly enough a similar functionality and that's what one of the reasons why we started talking about it for deployment. Uh we implemented something similar for jobs.

Uh the reason for that is in some cases we can identify a patterns of failures um within jobs and then be able to tell whether that failure is a fat a fatal failure and it doesn't make sense to re to retry it over and over again or it's something that is only a temporary intermittent uh problem that I don't know whether there will be an infrastructure problem

whether there will be networking temporarily down or something like that and you know that okay with my pod and you can encode additional logic within your pot if it exits with I don't know um exit code one you know you can retry the error if it exit uh with like 100 you know you can say yeah it just doesn't make sense I'm going to uh bail that

in the index it it doesn't make sense to uh to consume the same resources over and over again so that in the combination with uh the previous things that we were talking about gives the the job authors uh a lot more flexibility around running those jobs especially that like I said uh we're talking about jobs that are running on large scale over multiple days or sometimes even

uh weeks uh period of time you can think of if you want to uh use the AI uh terminology uh for training when you're running a particular job over like a month or so uh but it doesn't have to be AI specifically I think that's the last one that again came from um from the batch working group. Uh has anyone heard about Q a project? Perfect. Great.

That's uh I'm very glad to hear. Uh big shout out to Miho and Kevin. Uh both of them are authors and maintainers of the Q project. For those that haven't heard, Q is basically a mechanism that is built on top of uh on top of Kubernetes which provides queuing primitives and um and fair sharing mechanism which allows you to effectively use the resources that you have within

your cluster for uh those longunning uh work uh workloads as well as sharing uh their resources. So for example, if your team is using X amount of resources, but team B uh for the next I don't know weekend, let's say, will will have its resources unused. Through uh Q uh configuration, you're able to reuse the mechanism that are available for uh for reuse. And through this way

you are able to um actually request more than normally you would uh your your quota would be available but only if the team on the other end explicitly says so that they can share their resources. So anyway, long story short, um Q uh and Miha with Kevin came to us one day and they asked that they would like to be able to replicate jobs into the uh

clusters and they would like to have an ability to have like a centrally managed cluster where only a state of the job is being presented but the actual executions will be um will be replicated to worker um machines, worker clusters specifically. And so the worker clusters would be reporting uh partial state for the progress of the job. And in normal cases uh the the cluster the the

central hub normally job controller would start acting on the job and in their case they wanted to use the uh the job in the in the central cluster to only represent the summary status of all the worker uh jobs that are being executed in all the subclusters. So for that me uh for that to happen they needed a mechanism within the job that would say that there

is an external uh manager of this particular job and the default job controller that exists in a cube controller manager should not touch at all the that particular job. Um I know the mechanism sounds extremely simple because most of you if you will touch it or you will play with it will only know about the the annotation that you're setting or the field spec manage by and

it will uh say something other than the default job controller. But behind the scenes what actually happened is we went through the entire um job uh status specification and to make sure that every other job controller behaves similar to the default job controller. We put a lot of guard rails around the status resource to make sure that if someone is um is managing the controller it should

uh and user can expect the same status being reported. So it can't happen that the job suddenly is suspended and then not or it's changing the statuses in in in nondefault way. So we had to encode the entire validation in the API server to make sure that both the job controller uh along the way we figured out that there are some inconsistency within the default job controller

that we had to fix. Uh so yeah it looks simple but it wasn't. >> Uh okay I'm speaking more but cool there there's um there's a a couple more topics that we have still in progress. Um, stateful sets, like I mentioned before, stateful sets is a topic that we're slowly bringing to our attention after it's been abandoned for a little while. Again, shout out to uh to

Heba for her work. Um we've we've been talking about for a while how to resolve the problem of stuck deployments uh stuck stateful set deployments and we've had a couple of ideas uh around uh the parallel is one result the parallel pod management that was added a couple releases back but we recently started talking about maybe we should just um do something similar what a regular deployment

has which is a recreate strategy that should handle the majority of the cases and the majority of the situation that we are aware of and so we started working on it. Um we have the cap ready. Uh sadly we were not able to put together the implementation. The implementation landed shortly before the freeze but uh we didn't uh we didn't manage to get it through all the

reviews on time. uh so we will continue working with this topic and very hopefully I'm very hopeful that uh 137 is when we will be able to introduce on top of that the the problems that I've uh that I've mentioned uh before around the stateful set controller implementation uh we also realized that the clean a little bit of cleanup in the code is required to help with

implementing additional features like the uh the um Yeah. Uh, and that's the that's the one that we were talking about terminating pots and deployments. Yeah. So, okay, that's And I'm going to give it a back to >> Okay. Now, let's talk about the soft projects in Sikaps. So, this is agent sandbox that was announced at coupon na. This is a uh simple controller for creating an easy

sandbox using a pod for running use cases like a a agent runtimes. It provides an isolated stateful and singleton workloads. Um and you can scan the QR code to see uh the content and let me go back to >> Should I hold your mic? >> Oh yeah, sure. Thanks. Yeah. So um so basically uh we can go to the GitHub repo here to see what's on on

to see the content of this repo. So basically we have the sandbox CRD as the core that creates a a pod under the one only one pod per uh sandbox and also a service for it. So you can connect to it and there's also an extension layer for creating things like uh basically a warm pool of pods for you to create a sandbox in a faster uh

bootstrapping phase. So basically you can create a template describe your sandbox and then you can create a warm pool that will create a s uh create pods out of the sandbox template and when you want to create a sandbox you can use the sandbox clam to clam a resource from a pool and then I'm going to do a demo uh >> yeah not this is it bigger

there. >> Can you see the the letters? Is it good enough? >> Oh, okay. Yeah. So, basically, um I can show you the um Okay, let me do this. Do this. Okay. All right. So basically this is what the um the schema looks like. Basically you can define the pop template in a sandbox. It's pretty simple. And then you can also define things like volume c template

over here. So basically this sandbox creates a open call into into the sandbox. So on and if I just and so I have a name space for demo today and it has nothing in it. So I'm going to deploy this on the sandbox in the name space. Okay. then I should be able to see resources being created in here. But it takes time for the pot to

start. So I'm going to just show you the uh sandbox that already deployed earlier. So basically I have the um open clock running right now and then I can basically exact into the uh sandbox to to do things like I want to on board to open claw and then I can run this command inside it and I can do anything I want without worrying about this is

going to you know put a security risk on my own machine. So this is a use case of agent sandbox and um I can also um do a port forwarding of the basically this is following this example of open claw. So basically I can do port forwarding and if I open this link I can just uh access openclaw using the UI as well. So um that's the

gist of the project and then some people also use it for creating a developer on on like environment for either coding or just create a a sandbox for them their developers or end users to use. And there's also another sub project which is pretty new. They just started a month ago. This is called MCP life cycle operator. It's an operator for managing the life cycle of MCP

servers in the cluster. So I was told that they're going to uh create a release soon, maybe this week or next week. Yeah. So is pretty new. If you're interested, feel free to look into both projects. Uh and then um finally this is our and bi-weekly meeting and where you can find >> Yeah, those are Slack, mailing list where you can find and when everything when we're

meeting we're meeting every other uh Monday. Uh we have about five more minutes if I remember correctly. Yes, I I just got a sign. Um, there's a mic over here. I can run uh through the audience. If people have any questions, we're more than happy to answer them. If not, uh, we'll be hanging around after the session for a couple more minutes and we're ready to answer

folks questions. I'm guessing people are shy to ask questions. So yeah, we'll be Yeah, go ahead. >> Thanks. Um uh I don't know if things like inference pool uh is that counted as a different type of app that falls under your domain or that's a different SIG that handles that kind of workload. Um there is a serving working group that handles all the inference use cases if

I uh because it not only affects the just the workload running but there are also use cases for the networking layer and making sure that you're efficiently using the networking. So they are very closely working with the uh with the SIG network. they are also in touch with us with regards to making sure that the workloads API eventually if there are some additions required they will reach

out to us uh so it's more like a a a working group thing that is handling that particular um use cases but yeah about the new update strategy of recruit for stable that what is the use case just to avoid some uh stuck in deployment? Yeah. So originally a lot of our users and that was a conscious decision that we've made early on when we were creating

stateful set is that stateful sets were targeted at your workloads which are uh which are meant to ensure that the data that you're uh providing through state flow set is always like the most important part. So uh we initially when we wrote the controller uh the first version we said if a controller gets stuck for whatever reason whether that will be a temporary network issue whether a

pod will be will have a wrong um specification or something like that that we cannot uh that means there's something wrong going on and for us to protect your data that you have behind the stateful set we don't want it and we will require a manual intervention from a person uh to figure out what's going on and eventually move forward over time we learn that that's not

always the case that people are using stateful sets for um use cases uh oftentimes just because stateful sets guarantees you the naming of the pod uh similar to what I was talking with index uh jobs people you were using that so we introduced the parallel uh pod management which allows you to roll more than just one because normally stateful set will only run one at a time

in ordered manner. Uh so the recreate is basically to address that because we'll we will destroy the entire uh stateful set and only then start normally creating pots depending on whatever the uh the the rolling policy is whether that's ordered or uh or parallel. So yeah roughly so uh thank you very much. We're we're given like red signs like stop stop talking. Thank