The KEP Lifecycle: How the Release Team Guides Enhanceme... Kat C, Sreeram V, Rayan D & Subhasmita S

This talk covers the Kubernetes Enhancement Proposals (KEPs) lifecycle and their significance within the Kubernetes release process. The speakers, active members of the Kubernetes community, explain that KEPs are designed to guide the introduction of new features and deprecations, consisting of a structured document called kep.yaml that contains critical metadata. They discuss the role of the Kubernetes release team, which coordinates efforts globally to ensure new Kubernetes versions are released every 15 weeks. A detailed overview of the KEP submission process from a contributor's perspective is provided, highlighting the importance of collaboration with Special Interest Groups (SIGs) and the necessity of passing the Production Readiness Review (PRR). The presentation also emphasizes updates in the enhancements tracking process, ensuring adherence to new deadlines for improved efficiency in release cycles.

Uh, hey folks, uh, thanks for joining us here today for our talk, uh, the Kep lifecycle. How to release, how the release team guides, uh, enhancements to stability. Uh, before that, uh, a quick round of introduction. My name is Ryan Das. Uh, I work at, uh, DigitalOcean as a senior platform engineer. I am also, involved with Sig Release, uh, release team specifically since, uh, last 2 years.

And, uh, this release I'm, release leads at, uh, shadow. And, uh, yeah. Hello, everybody. Uh, my name is Kat Cosgrove. I am the head of developer advocacy at Minimus, but more relevantly for this, I am a member of the Kubernetes steering committee. I own the Kubernetes release team subproject, and I'm a technical lead for Sig Docs. Hey, everyone. Uh, my name is Sriram Machiraju. I also work

for DigitalOcean as a senior software engineer in their, uh, managed Kubernetes team. I've also been involved with the upstream Kubernetes community for a while now. I am part of, uh, the 1.36 release team as the release branch management lead, and I'm also involved in, uh, Sig Node. Hi, everyone. Uh, my name is Subhash Mehta. I have been around the release team for a quite a long bit,

like since version 1.26. And currently, I'm, uh, working as a shadow there, like for enhancement sub team. And I've been involved in Kubernetes for quite a while, uh, since my LFX mentorship days working around, uh, Sig Cappy, uh, Cappy, then CapG subprojects. So, yeah. Yeah, thanks, everyone. So, today, uh, we'll go over, what Keps are, uh, why we need them, and, uh, we'll go over, uh, the

caps uh, point of view of I mean I mean caps cap cycle from release team's point of view as well as uh, contributors point of view and uh, some updates from the uh, enhancements uh, team. yeah, uh, what is a cap? So, cap is basically uh, Kubernetes uh, enhancement proposal uh, using which uh, I mean new features and uh, deprecations get into uh, new releases. And uh,

it consists of multiple uh, I it consists of a design document called uh, a cap.yaml. It uh, which is basically uh, the metadata of the cap. And uh, everything related to the uh, enhancements reside inside k/enhancements repo. And uh, let's go over uh, Kubernetes release team. So, uh, the release team basically handles everything uh, end to end to release uh, Kubernetes every uh, uh, every cycle. We

are a group of people from all over the globe and uh, we collaborate and uh, bring a new version of Kubernetes to the world every uh, uh, 15 weeks. So, let's go over the history of caps. Uh, so, it initially got introduced uh, by introduced in uh, 2017 by Caleb Miles. Uh, caps are basically inspired by uh, Rust uh, RFC which is uh, request for comment and

Python's uh, PEP uh, Python enhancement proposal. So, uh, Kubernetes let's talk about Kubernetes SIGs SIGs before we move forward. So, a Kubernetes SIG is basically uh, special interest group uh, which uh are dedicated teams uh teams for multiple uh components of Kubernetes like uh SIG API machinery is for uh uh Q API server, uh SIG CLI is for uh kubectl, customize, etc. So, we fall under uh

the uh SIG release uh I mean special interest group and we are a subproject of SIG release. Uh and we consist of multiple subteams as well. Uh we'll go over the uh the uh team uh in the later part of the slides. So, KEPs today uh So, KEPs today uh is managed by the enhancement subproject of uh SIG architecture. Uh we have a meeting every 2 weeks

at 10 uh p.m. uh Pacific time. And uh as I said, all the KEPs are uh K/enhancements repo. You can go there and check uh check about all the KEPs that are there. So, uh KEP template. So, if you are proposing a new KEP, uh you have to fill the KEP template and get it reviewed by your SIG leads and uh uh there are a lot of

uh uh things to uh I mean fill uh it goes through multiple review steps and then it comes to us for tracking. And uh So, this is the uh GitHub repo of uh K/enhancements. Uh you can go there to check uh any KEP that uh uh that uh there is. And then uh you can find it in the issue section. And this is an example of a

KEP uh raised by Sri Ram. Uh yeah, uh you can go over the cap number and the I mean cap issue in the case less enhancements repo and then uh check it out. Now, Shriram will take you through the caps from contributors perspective. So, over to you. Yeah. So, Ryan just showed us how like what caps are and how they're used to track new features in Kubernetes.

So, now we're going to look at caps from the perspective of contributors. So, if you are trying to add a new feature to Kubernetes, let's say change the API spec or something like that. And if you want to get that change merged into Kubernetes, you can't just go and open a PR and expect that to be like reviewed by maintainers. So, that's where the cap process comes

in and what we do is uh if you have an idea or a change that you would like to make in Kubernetes, you would uh first figure out which SIG the change would come under. So, let's say you want to change something in the the container runtime interface API. So, that would be that would fall under SIG node. So, you'd probably uh attend these meetings of SIG

node and discuss your idea with the SIG. So, each cap is supposed to be sponsored by a SIG and it'll be approved by the SIG uh leads basically. So, once you have uh discussed your idea with the SIG uh maintainers, you would go ahead and create the issue that we just saw. So, once that is done, uh the release team would track or add this cap to

a release cycle and for that you would need a uh opted in label from the SIG maintainer. And at this point, uh your cap is added to a release, but now you need to actually do the work. So, you would open a PR to the Kubernetes/enhancements repository in GitHub. And in this PR you would add a couple of things. Uh the first one is the kep.yaml file

that uh Ryan was talking about earlier, which has all the metadata about all the uh about the KEP uh basically which version the KEP is uh targeting uh for which stage. So, if you if you create a KEP right now, we are uh like almost halfway into the 1.36 uh release cycle. So, if you want to propose a change to the 1.37 release cycle, you can open

a KEP and your kep.yaml would have the uh milestone as 1.37, and uh you'd be targeting alpha, which is the the cycle that all the new features have to go through. First, you will uh release the feature as an alpha uh stage, where it'll be turned off by default, and you would eventually graduate it to uh generally available. So, once you open this PR, you would uh

need uh two things mainly. The first is a kep.yaml, which has the uh metadata, and the the actual readme file, where we have the KEP doc, where you propose the change. And uh you can basically clone the template that Ryan showed earlier, and uh fill in all the questions. And once you have that, you'll get it reviewed by uh the SIG maintainers, basically. And along with this,

we also need a uh as part of the KEP template, we also have something called the PRR questionnaire. So, PRR is production readiness review, and these are a bunch of questions related to different uh aspects of the feature, like uh how would you roll back a feature if something goes wrong. Uh what all changes are necessary uh for rolling back this uh feature in production, and what

are the kinds of metrics that you're planning to introduce, uh in order to make sure that the the feature is working as expected, and so on. So, the KEP, uh even if you get your KEP added to the cycle, uh it needs to pass the PRR review, and uh it needs to get that approval uh before it can be uh before you can uh open the code

uh changes as a PR. So, we have the links uh the slides where you can find more about the PRR uh process. And once you have filled all of these questions and opened your PR, submit it to the uh So, basically your kept needs to be reviewed by the SIG maintainers and also another set of PRR maintainers. And once both of these are uh approved, you can

uh the the release team would take it from there and then they they would like track the kept in the cycle. And now you have to actually do the work to uh get the code changes merged. And uh once the So, throughout the release process we have different stages. First is the PRR freeze where you'd get the approvals and the enhancements freeze is after you get the

approval, you have to get the PR merged in the Kubernetes enhancements repository. And then we have a code freeze where you have to uh which is the deadline before which you have to merge all your code changes to the Kubernetes Kubernetes repository. And then we have a docs and test freeze by the time Sorry. Uh by which time you have to merge all the doc changes if

you you have any. Yeah, now uh I'll hand it over to Subhashmita. Okay. So, uh I'll take on from here. So, once the uh contributor has started like uh they have filed a KEP, uh they need to follow a step process of 15-week step process where the KEP is being tracked. This is the entire release cycle. So, roughly about uh like Kubernetes releases three times a year.

Uh that's almost like 15 weeks uh each where we have one to two weeks of break between each release uh since there might be some delays and we need to uh accommodate for that. So, um if I give you a bigger picture of what it looks like in in the work uh in the week zero to four the release team gets All of the enhancement proposals that

are being that are new, that are in the process already, that are graduating to a new milestone, are getting tracked. we are collecting all of the enhancements. We are, uh, opening PRs. Uh, we're doing all of the admin work there. And then, by week four, uh, we start tracking all of the existing caps or that are already in in this milestone that are that need to be,

uh, for the PRR freeze. Uh, as as Sriram mentioned that we would be, uh, going over each of these PRR requirements and we check each of the caps. Uh, do they abide by it or not? And, uh, we'll talk more about the PRR, uh, freeze and deadlines later on. And then, we move on to the enhancements freeze where, uh, all the necessary requirements that needs to be

declared, uh, by the specific SIG and the contributors that this feature is getting added for this release, has to be finalized by PRR freeze, enhancement freeze, sorry. And then, we move over to code freeze where all of the changes, all the requirements like related to PR, related to, uh, the code, the documentation, all the tests, and everything needs to be, uh, already in the merge ready state

or either merged to the KK repository, that is the main And by week 15, we release, uh, the version and we just open, uh, we are open for any bug fixes and then, most probably, maybe cherry picks. That's all. So, this this is what the entire life cycle it looks like, from, uh, bird's point of view. But, uh, if I zoom in, so, week zero is where,

uh, we start creating our enhancements tracking board. Then, we find all of the issues that have graduated to stable and we check if their milestone has been changed to stable or is it like still sitting on waiting on new features that need to be added. So, we make sure their milestones match what they're targeting. And then we check if uh if the cap has already been implemented

like in GA and it's still there, the issue has not been closed. So, we we remove all of that uh uh confusion so that new caps can come in and they are tracked properly. Then uh we also make sure that any old caps, all of the lead opted in label is removed because once we track any new caps with the lead opted in label, it gets into

the loop and uh the prow needs to do their job in tracking automatically all of the new caps in that board. So, we have to make sure any new cap that are targeting a new release has their fleet opted in label uh by the SIG or or the contributor who's in charge of that cap. Then by week first and until week four, we send uh an announcement

email in the Kubernetes dev mailing list for call for enhancements, um track the existing caps, and verify all of the caps have a PRs uh of have PRs associated for their PRR readiness review. um yeah, for the PRR readiness review, PRR review, it's a set of question as Sriram mentioned. So, every cap author needs to fill that uh and and submit. And it gets reviewed by a

set of authors and set of reviewers. So, it makes sure there's no confusion, there's no re-addition of features that's already existing. It also makes sure it's like if something goes wrong in that feature, how do they roll back? Or uh is it a new feature? How do they turn it on? Like it's it's uh more of a general information for the existing users and users and what

to expect from this cap. So, it's quite And it also has information on who is going to review it, who are the specific authors. So, it's it's it sets clear boundaries, and it also like uh sets the stage for later on, so there's no confusion when we are uh close to the So, okay. By the week of PRR freeze, uh once we hit the deadline, uh previously

it was a soft deadline, but now since I think from version 1.35, we have a PRR freeze as a hard code uh hard requirement, where we uh create a pull request, the release uh lead, enhancement release lead uh creates a pull request to uh stop all of the jobs from automating and uh adding new caps to the board, because then we want to track only the caps

that have already opted in and are already in the process of tracking it properly. Um this is to just make sure that any new features that are getting added to the board or for that particular release are not um getting sidetracked, are not delayed. So, in the process, we are not delaying the release process entirely. So, yeah, that that makes sure that we are on time early,

and we catch any errors or any any uh bugs that could come up later in the process. So, we we we have got time, buffer time to do Now, uh then comes enhancement freeze, I believe, after a week. So, after 1 and 1/2 or 1 week, we have the enhancement freeze. We notify all of the authors or the contributors who are working on the active caps. So,

we make sure uh in the enhancement freeze, any new requirements are declared, the PRR uh cap file is being merged, and it's it has been approved by PRR approver. And and we are ready to uh just it basically makes sure that the feature that they're targeting is implementable, and is uh in in place for the release uh upcoming release. Now, the after post enhancements, we make sure

any new caps or any caps that did not go through all of the rules are removed strictly. If they still want to have some time, if they still want to get this feature included, they have to go through an exception process. So, where the release team actually monitors the cap the exception request, they see if this this actually needs to be included, if this feature is critical,

or if it's a dependent feature on the existing feature that's already tracked. So, we take decisions based on that. And also we monitor the issues that are risk for the code freeze that's coming next. Um before the code freeze, all of the changes that are required in docs, in code, and in in the tests needs to be either in the merge ready state, or they should have

the approved or uh LGTM labels labels on it, so that it's ready for the code freeze uh coming up. We give a buffer of around like 1 to 2 weeks before code freeze, so there's ample of time to get all of the code implemented, and uh the features can get tracked properly. Now, for the week of code freeze, all of our shadows, like enhancement shadows, reach out

to the assigned caps. Um they make sure everything is tracked, uh proper reminder is given with time. And uh the com It also gives time to the coms lead to develop major themes for the official community's blog post, and and the docs team to actually uh track and include in any other changes that are required in the docs to be included. After that, we have a code

freeze party. We also have announcements freeze party. It's not actually a party, though we have been wanting to uh get some drinks after that. So, what what we do here, we as a team assemble, and we uh go through all of the caps that are at risk that did not make the freeze freeze cut. So, we make sure everything is tracked. We go through all the included

PRs, all any other included documentation, everything that is being assigned to that cap that needs to be tracked. And we track it. We also make sure we reach out to the authors and see if this can be included in the release. If so, they need to create an exception for it. If not, then we remove it from the milestone, and they have to like target it for

the next release. So, we have being very strict about it now because it later on it does not delays the entire release cycle. Okay, I'll I'll that's about all about release kept from the kept process. I'll hand it over to Cat. Hello. All right. Updates to the enhancements tracking process. Has anybody here ever been on the release team before or like is aware already of how the

release process works? Okay. So, a long long time ago, and by a long long time ago, I mean like 2 and 1/2 years ago only, we used to run the entire enhancements tracking process out of a Google sheet. Okay, it was nightmarish, and we required two more sub teams, three more sub teams than we have now to run the release because of that. It was just a

huge amount of um tedium involved. But, we're trying to be constantly improving, so we've made a lot of changes over the years to the way we track enhancements. Um and one of those, as already mentioned, is that the PRR freeze is a hard deadline now. I know that we don't like deadlines, and that we don't like being told that something is inflexible, but um history with making

changes to the release team does show us that it's better to have hard deadlines than soft ones. Um, KEP authors also always have to have at least a PR opened. It doesn't have to be merged, but it has to be opened with the PRR questionnaire already filled out and the kep.yaml updated. We're pretty inflexible on that one, too. don't care if it's merged or not. It does

have to exist. Uh, if that KEP does not meet this requirement, we will remove the KEP from the milestone um, during PRR freeze. And we don't we don't like doing that. It doesn't feel good cuz we know y'all work hard to open these enhancements. We know you work hard to get them going, but um, we deal with so many now that we can't really afford to allow

a ton of wiggle room on that. Um, so I'm going to go into why we did this. With PRR freeze, um, that is a hard deadline now and owned by the release team now when it used to be a soft deadline owned by the production readiness subproject of SIG architecture. Um, it was actually their request that we do this. They didn't feel, um, equipped or empowered to

enforce a deadline on a release team item when they are not part of the release team themselves. They don't come to the release team meetings. It's not like the the branch manager gets loaned to us from release management. So, branch management does have a direct relationship with the release team. They come to the meetings. PRR does not. So, there was a lot of miscommunication. There were several

issues with enhancements getting kicked from the release super late because we noticed, like, well after the freeze that they hadn't actually undergone production readiness review or they were using a PRR template that was like 2 years out of date. So, it was missing a whole bunch of questions that, like, today we require answers to. And so even though, yes, there was more wiggle room on due dates

back then, it ultimately was significantly worse for both for us, the release team, for the production readiness team, and for cap It was just like bad for everybody. So, I'm sorry that we have another hard deadline to give y'all, but the good news is you can file an exception request in advance. You don't have to wait until you're late. We would actually prefer that you file it

in advance. It's so easy. It is an email sent to us and the owning sig with like four questions. It's It'll take you like 30 seconds to do, I promise. And you can send those to us in advance. And if you do that in advance, we're way more likely to approve your exception request, and you have nothing to worry about. Like generally, if you're only asking for

like one to three days, it is so entirely whatever. We just need to know that you're working on it. that's why we gave you another hard deadline. It feels like every time I do one of these sig release update talks, I'm announcing another deadline. Last time that was docs freeze. Really sorry about that. Um but we I I promise it's made the process easier for everybody. We've

gone from getting like an average of 40-ish caps through to the final release to now we're looking at like 60-ish on a regular basis. So, it it is more efficient. It is better for y'all. It is better for us. It is better for our users. So, there's there's kind of not really a downside here even though it's makes people cranky initially. If you've never been on the

release team before, now's the time. Now's the time. So, the release team has a shadow program. It is the only part of the Kubernetes project that is just like an open application to come participate in Kubernetes, and you get automatic org membership if you get selected for the Um we have a very, very structured contributor ladder, uh very exhaustive handbooks for each of the different roles on

the team. It's a It's a pretty significant on-ramp for new contributors. I will warn that it is very much like drinking from a firehose joining the Kubernetes release team. I don't think it's necessarily the best way to contribute to Kubernetes for the first time, but it sure is the fastest. we get about 200-ish applicants per cycle. So, do not feel bad if you do not get selected.

Just keep applying. Most people do not get in the first time. So, uh if you're somebody we've seen around in other SIGs, that is better. That makes it more likely that we will select you. And nobody is too highly ranked in this project to serve on the Kubernetes release team. We occasionally have people with SIG chairs come do a stint on the release team. The SIG Docs

leadership have all rotated through the release team within the last couple So, if you are a SIG chair or a tech lead or you're on a governing body for another CNCF project, I promise you are not too cool to come see how we crank out Kubernetes three times a year. So, um if you have questions about that, you can reach out or do it in the Q&A,

I guess. All right. So, that is how a cap works. It's That's the life cycle of an enhancement from the contributors view, from the release team's view. They are fairly simple little things. Like, it's just a document, but it allows us to keep track of a lot of different moving parts in a centralized way that makes sense to a lot of different people at the same time,

even though they're coming from like very different perspectives, cuz SIG Node doesn't care about the same things that the release team does, and SIG Docs doesn't, and SIG Security doesn't, but they can all look at the way the enhancements tracking board is functioning and get a feel for how well something is going or how janky something is and that's helpful when we're all volunteers trying to get

this thing out the door for free so other people can make money off of us. But yeah, caps are super important to the kubernetes ecosystem. They track all of the updates coming into a current release. That includes deprecations and removals. Which also triggers a mid-cycle blog if we have one of those. Usually we only care about ones that are in KK but every once in a while

we archive a critical piece of kubernetes infrastructure called ingress engine x and it has to come in your part of the kubernetes release cycle. These all live in K enhancements so you can go scroll them at your leisure. And just gives us a lot of structure for how features are added. We've got some previous kubecon talks on this topic cuz every once in a while people forget

and also we change it a lot. And everything about the way sig release and the release team in general operates is documented. We love documentation. The CNCF makes us turn over slides so these links are clickable. And you will be able to uh get into all of our docs but if you just go to um github.com/kubernetes/sig-release you can see all of the release team Questions? What's up?

There's a mic There's a mic right there if you want or I'll just repeat what you say into the mic. Just because I'm interested. How did it end up that you use Google Sheets instead of just having it in GitHub in the first place? Uh, for the recording, the question is how did it end up that you use Google GitHub in the first place? Um, because the

way the release team works and tracks enhancements pre-dates the existence of GitHub's issue tracker. Yeah, hit it. So, I had a question about the kep.yaml that was mentioned in the slide. Is there any automation that's kind of behind that and does that drive things like the enhancements board um, that you're using? Yeah, so uh, we have a contributor website called kubernetes.dev uh, where we have a page

for all of the caps and you can like filter them based on the SIG and everything. So, we use the kep.yaml for that. So, it reads all of the kep.yaml reads all the data from all the kep.yaml for all the different caps and uh, like presents it to you in a nice easy to grok format so you can filter caps and filter based on the contributor or

the SIG or uh, the date. Something like that. We definitely would want more automation uh, tracking the caps itself. So, right now we don't have any automation in place that checks if the cap template is up-to-date. So, if you open a PR today adding a new cap or updating an existing cap, somebody has to go through the cap document and figure out if it is missing any

of the new PRR questions. We have been trying to like PRR questionnaire and the cap template Uh, but that is still something that's not done. So, that is Uh, something if folks are interested to contribute, uh, folks can look into. After the building of the tracking board, um, that automation entirely happens based off of labels. So, it looks for, uh, milestone, for instance, 136, um, and then

it looks for lead opted in. And then it throws it on the board and imports a whole bunch of data about it based on the rest of the labels. Some of it has to be updated manually. A lot of the docs stuff does. Um, cuz the calling it the enhancements tracking board is not entirely accurate anymore. It The entire release gets run out of that tracking board.

Um, docs runs out of that tracking board. Um, the only I think the only team that doesn't have to touch it is release signal, right? Yeah. Everybody else has their little fingies in something that we just call the enhancements tracking board when it it is basically just the release board. Thank you. You're so welcome. Any other questions? We're out of time? All right. Later.