What's New With Kubectl and Kustomize … and How You... Marly S, Yugo K, Eddie Z, Maciej S & Arda G

This talk discusses the updates and improvements in the Kubernetes CLI tooling, particularly focusing on the features and capabilities of 'kubectl' and 'kustomize'. The speaker introduces new functionalities, bug fixes, and user feedback processes in the latest versions, such as enhancements to resource customization, interactive delete features for safety, and the introduction of a new configuration file, QRC, for better user experience. The session highlights the importance of backwards compatibility and community involvement in shaping the future of these tools. Additional topics include plans for implementing multiple kubeconfig support, addressing long-standing feature requests, and the evolution of YAML handling within Kubernetes tooling.

Hello. Thank you all for joining us uh for what's new with cube control and customize and how you can help or like I like to say cub ectal which is the official pronunciation. Uh my name is Eddie Zaneski. I'm joined by a slew of awesome people. Marley Salazar from Integral Ad Science, Yugo Kobayashi from SV Institutions, Artaguchu from Red Hat, and Mache Shoulick from Defense Unicorns. Uh,

anyone familiar with what SIG CLI is? Oh, that's great. I love explaining this. So, SIGs are the special interest groups of the Kubernetes project that own different parts of the Kubernetes codebase and project itself. And so, SIGs generally have most of the power in terms of direction and steering for their part of the project. And our SIG owns the CLI tooling. So, customize cube control, uh, a

bunch of other libraries and SDKs that you may or not may not be using through a bunch of the tools you use. Uh, but we meet every Wednesday at 9:00 a.m. Pacific time, whether we're doing a bug scrub or we're doing a general SIG meeting where we have an agenda and kind of talk through backlogs and issues. >> 6 p.m. >> 6 pm uh in Europe, Central

Europe time. Uh, so please come join us. Uh we're super friendly and uh yeah, it's a great place to get involved contributing back to this Kubernetes project. So I'm going to hand it off to Yugo to talk through customize. Uh hello, I'm maintainer of customize. customiz what is it? Customize tool that's allow you to customize resource files. Uh latest version of customize is B581. Uh what's happened

after last QCon anyway? Uh we introduced propagation name set cor inherit function in 580 but that's contained a regression bug related name set proagation for child names uh child customization y. Uh this issue happen because the implementation is uh propagated to the name space set in the parent name space to all child customization. Uh for example it pes applied from a different file in a child name

child customization an empconer uh that is fixed in B 581. Please see detail uh issue number 6044 and please use a new customization. I please use new customiz customiz the binary and uh propagation names current to her that customization stop that function is stopped to name hops to overrite. Uh since some user rel on our previous behavior, we prompt to add an option to the previous behavior.

Uh please see detail for issue number 6058. Uh for example, Helm chameace is not set for all resource problem. >> Okay. Um if you've ever worked with um cube cuddle you probably work with like run into issues like I want to do this thing by default. Uh the biggest heat that we got heated debate about was around uh delete. Uh who broke their production by doing cube

cut delete all. Cool. There's a a couple of them. Um uh I was trying to implement something that got closed. uh we got different options around oh we want to prevent people from doing keep cut delete all all but at the end of the day uh we decided that we among everything else the most important thing is we need to keep the backwards compatibility and that's an

interesting one because um after we implemented QRC and when I was doing recently my testing uh we recommend certain defaults we recommend people use the interactive delete just to make In a similar way how you when you're doing pseudo or when you're a root, you have the interactive delete by default. I have it on my um as a default for QRC which is basically a mechanism that

we currently have for tweaking the default for cube cuddle. So uh I was testing a PR and it turned out that uh the entire test started failing and I was like in panic mode like did I break uh did I break something? It turned out the my QRC C which was super smart and asked for the interactive delete prevented the E2E from actually removing certain things. Uh

so if you're uh if you're going to ask why we have the QRC that is the main reason and why we are so strict around the backwards compatibility is just to make sure that situation like mine and that was only within my environment does not leak into your production. uh because trust me, you don't want to be called at like 3:00 am and on Sunday morning. Um

so yes, QRC uh you can basically introduce any fancy things, tweaks. Uh currently uh it gives superpowers. Um it's currently in beta. We are leaving ourselves a little bit more time to gather feedback to expand the possibilities. It provides you with currently with the ability to define aliases to tweak the default values like I mentioned the interactive delete uh being the default or serverside apply being the

default. All those things are currently there. We recently added a credential plug-in allow list so that you can say oh there's this plugin that I trust it can be run or I don't trust anything else other than like GCP or whatever O client exists for a particular thing. That's another uh layer of protection that you can implement with uh with a QRC. Um yeah uh the art

recently added and uh Peter uh qrc command so you don't have to manually type what the qrc should look like. Uh there's a bunch of command qrc um specifically unsurprisingly the name uh which help you to view your qrc file and eventually uh tweak the the settings. Uh k uh anyone try that one cool um if you look at it it's not that bad and it actually

resolves a ton of problem. Kayam is a subset of yaml. uh if you look at it and uh squint that's kind of like a combination of YAML and JSON. Uh if you're curious uh grab us after this presentation I can walk you through why we decided to do Cayaml. Uh but uh at the core it solves ton of problems that we've been experiencing over the past 10

years uh around YAML limitations. Uh I haven't seen any problems aside from one bug. So very likely we're going to promote it to um to stable next uh release speed deprecated almost a decade ago. So some modern clients do not support it and consequently people cannot use cube control at hx exact and port forward. So thanks to shan now we have switched to websockets. This is in

beta and basically you don't have to do anything as long as your cube control and your cluster is newer versions. Uh the underlying protocol has changed to websockets from speedy. Um this uh is in beta and it is planned to be G8 in uh 137. Yes. And while we are cleaning our cap dashboard, we see that these two caps actually they are in the code and they

can be considered as a stable and we officially uh mark them as stable. >> Okay, >> deprecations >> uh cleaning up some of the legacy stuff uh is pretty important in the m maintenance of a project. So we have deprecated officially by setting the new default for the cube kettle debug the debug command. Uh we have set it to general instead of the legacy profile um because

that's that's we would prefer that you all use that going forward. Um also related to QRC uh we have deprecated the preferences field uh just because nobody used it for anything. Um originally it was meant to be kind of like serving the function that QRC now now serves and then just due to lack of support uh KUI and KRM functions have also been uh archived and marked

as deprecated. >> Um, multiple condition support in cube control weights is a long-standing feature request and we usually recommended using the subsequent cube control weight commands to achieve it. But this usage has some defects like for example resource version may change. So basically you can you may uh wait for multiple instances of the same resource. So this is not ideal. We decided to add uh multiple conditions

supporting cube control weight as builtin. It is available in one uh 36. You can use it. 76 is like this. Uh aside from the create and delete conditions, all the conditions will be ended to each other in an sequential order. So order matters. But create and delete conditions are special ones because uh you cannot wait for any condition before the creation of uh the resource. So cube

control weight searches for the create condition and if it finds regardless of the order it will be weighted first. Cube control waits searches for the delete condition and if it finds deletion condition will be waited last and all the rest of the conditions will be inserted in between. Feel free to test it and uh give us a feedback in case there is any issues and problems. >>

Awesome. So quick summary of where we are. Uh and then we got ton of time for questions and discussion which is what our favorite part of this is. Uh, is anyone using cube control in the room? Can we see hands? Okay, I'd expect almost everybody. What about customize? It's a good number of hands. Is anyone using QRC yet? Is this the first time anyone's hearing about QRC?

Yeah, we got to do a better job of that. Uh, cube RC was our attempt to kind of introduce default behavior. So, the cube control is as old as a Kubernetes project itself, right? you build a Kubernetes and then you need to build a CLI tool to work with the the Kubernetes. Uh and so it's really hard for us to make behavioral changes to the tooling that

everyone's built pipelines and scripts around for years. And so you have to kind of opt into new behavior. And so that's where QRC came from as a way for us to introduce behavior options for you as users that you can opt into things that we just can't break in your pipelines. delete. Interactive delete is one of those options that's really powerful and you you all should probably

like opt into it and we should do a better job of advertising and explaining it's there, right? But the idea is to hey, I want to be prompted before I delete a bunch of resources and to know what's actually going to be deleted, right? And so that's what interactive delete came in. And the only way we can give that to you is by introducing a new config

file that you can opt into. Uh so some of the other things we're thinking through are multiple cube config support. We have some of this now like cube config view set all that stuff. It's kind of really hard to work with especially when you're working with a bunch of different clusters and contexts. So we want to solve a problem here but we need feedback and and what

does a solution to this problem look like for y'all. Uh so I'm just seeding some topics for discussion here. Uh we need a new command that's like apply that is server side apply by default. We've gone round and round on all sorts of different names for this. We're really bad at naming things. Uh so we need to build this at some point. So if you have a

good idea of what that could be called or how you want it to work, let us know. Uh JSON path in uh cube control is like partially in fully to the spec, but the J the JSON path spec is like really small in general like it doesn't have a length function built in a bunch of stuff that you depend on for other JSON path implementations. And so

we're thinking about ways that we can give that functionality to you by switching to things like cell or some other tooling out there. So if you use JSON path or it's lacking something or you like cell or you hate cell definitely let us know. We want to work on uh CRI native copy. So right now cube control copy depends on a tar binary being in your in

your container or in your host. Uh there's been tons of CVEes over the year that we're afraid to touch any of that code. So we really just want to rip it out and replace with something that's actually secure by going through the CRI itself. So this is something that we're hoping to make a bunch of progress on this year. So, if you're interested in any of that

work, let us know. Uh, and the last thing I want to talk about is, you know, we added these headers a while ago that got mentioned. Um, these will send audit headers to your server so you can kind of keep track of like what CLI commands your developers are using. Uh, we'd love to get that data from you somehow to find out how people are using the

CRI, right? I'd love to come up here at the next CubeCon and show you a graph of like delete is the most common command by this percentage. uh I need a way that you can give me that data so that you we can anonymize it and you can trust and upload it to the project so we can go through it and kind of share it in an

anonymous way. So we kind of just need an idea of how you'd be comfortable giving us that data so we can aggregate it. So uh feel free to chime in and help us out there. Uh and then without further ado again that's our meeting times. You should totally join us. Uh these are some of my best friends that are around the world and it's truly a global

project. you know, we have we have Poland, the US, Turkey, and Japan. Uh, and then if you do the survey, they let us keep doing these maintainer sessions. So, give us good reviews on that. Uh, and with that, we love to open the floor to questions. We've got a mic here, and we've got another mic we can pass around. So, yeah, just give us your thoughts, give

us your dump. We'd love for this to be super informal and, uh, talk to us about what you love or hate about the tooling that you guys use. So, please, someone open it up. I know I know you've got complaints and burning questions. Thank you. Thank you. Anyone here? Okay. Uh, regarding the telemetry, I think it's really easy for some passwords or secrets to slip in and

uh, I would be afraid to use that, but I think the most realistic option right now is to just use the user shell history and export the cubectl commands from that or maybe partner with Arduino or something like that. >> Yeah. Yeah. This is this would go into audit logs. Uh, you you opt into this in the first place to turn this on. Um, obviously you could

see the intent of why we'd want to like think through some of the usage and see what commands people, you know, are using the most, >> interestingly enough, um, a lot of the the flags, I might be wrong to some, but um, we have a lot of the flags marked already as containing secrets and those are internally within cube cuddle process set that such that they will

not be locked. uh because we know that those uh may contain sensitive data and so we're not passing if I'm not mistaken because that that was a cap that we worked on very long time ago. We didn't even pass the actual values of the flax. We're only passing um the flax themselves because we less care about the actual data that are being passed to flax but more

about which parts are being used. Good question. >> Uh what about prune in apply? I mean it's been like marked as problematic for forever but there's no succeeding feature. I think >> I have a promise from Justin who is uh who started working on this particular topic with Katrina a couple years back. Uh we are aware uh Justin has been moved back and forth through uh a

couple of topics. He promised me on Sunday during the maintainer summit that he will return to this and he uh and he wants to push this through. Uh I think I spoke with someone that was interested in also this particular topic. So I have a commitment from two people to work on it. One of them being the author. Uh if you're interested in in helping with that

effort, uh I'm more than happy to to get you in touch with Jasten and slowly but surely we can uh move this >> Yeah, thanks. Um, what's the motivation behind making a serverside apply command instead of just included in in the QRC like the other stuff stuff? >> You want me to do this? So, we normally have a slide that we didn't pull up. Do do you

have that a slide of the imperative versus declarative stuff? So, >> apply is the only way to do declarative resource management in Kubernetes, right? all of the other commands that you use like think like roll out for example right roll out updates the field in your resource it no longer matches the desired state in your config right uh we server side apply is important because of the

um the last applied annotation so if you ever look at a resource that you deploy in Kubernetes there's an annotation that gets added that is like the last state that was applied and that's how it calculates the diff and so any sort of like things that you change or any other resources or operators that change that there can be tons of drift. And so apply being the

default is the desired behavior that that resolves a whole bunch of issues. But it's one of those things where we can't break current compatibility and unexpected behavior. There there is also uh surprisingly from very long time one of the uh founders of of uh of Kubernetes project from Tim Hawin an idea and an issue open about I want to keep cut apply to die. Um so given

what we said with regards to backwards compatibility that's not going to happen ever. Uh but ideally if there will be a new command that we can point people to rather than oh you need to tweak your QRC and then if you move it somewhere you forgot a bit or something like that. while if we have a command and we will do a better job of of advertising

that instead of the QRC it'll just be easier because it's harder to convince people to oh and by the way you have to run cube cut server side true and if we say cube cuttle whatever the new name of the apply will be because we had weird ideas uh we literally went through the synonyms of the apply. Uh, also apply is like not quite um what it

does and we're trying to figure out like we know what it what we know what it should be doing. We just don't know the name. So it's like this close to have it done. If you have great ideas around naming actuate we we considered it actuate at some point in time. >> Actuate. Yeah. >> Doom. Doom. I I >> you did talk about actuate. >> Actuate was

high on the list. Doomed was not on that list, but that's a reasonable suggestion. We definitely will go through that one. >> Yeah. And this is also not just like making a new command that sets that flag to true. This is our chance to kind of redesign the experience and make it so that hey, this is a this is the command that you use to put resources

into Kubernetes to you know to have the declared state uh the declared state. So yeah, this we need to think through like what this should actually look like after the 10 years that we've spent working with the ply and fighting through it. So yeah, any feedback? Again, we're here for tons of feedback and uh we need all the help we can get. >> Get it. Um yeah,

I I'm have a question for the multiple cube configs because that's exactly something where I build a a plugin for cube cuddle. Um, yeah, just do you already have ideas how you would do this or should I just come afterwards to you and we have a talk or something like that? >> Yeah, I mean I I haven't seen your plugin, but like we'd be open to upstreaming

plugins that we find is the right behavior, right? So, we just know it's something we need to solve for. Uh, and if you've built the answer to that, like that'd be something we'd be happy to upstream. So, come join us. Come hang out at our meetings. And yeah, >> also a plugin is a great way to like try and get functionality you might want in Qtle and

then prove that people actually want to use it and then we can upstream that kind of code into Cube Kettle. That's like a path to getting features into it. >> Also, it's like one of the the simplest and the fastest way to for you to iterate on a particular new command because if something lands in a cubecado, the release cycle is like three times a year. If

you're a plug-in, you basically can release every week. >> So, something I love about um customize is vase or variables. And something I hate about customize is the deprecation of vase, especially uh for cases where um the replacements don't really work. So, maybe you can say something about the future plans in this area. sorry. Uh what's the problem of refreshment? I think the problem with replacements is

um that you can't um like in the last layer inject um information into the first layer like um I think the typical example is um setting the host names for ingress for example. I think that doesn't really work. >> Also, it's quite verbose and messy and not fun at all. >> Can we talk after? Sorry. Yeah, that's perfectly a valid answer as well. Okay, I see you.

You can give me the time like people figure out that I'm going to be running back and forth through the entire room. Uh kind of almost on a similar topic. Um Helm chart generator in customize. I know that's I think a really nice way to do some of those more arbitrary mutations that you might want with replacements, but replacements a little more locked down. What's kind of

the plan for the Helm chart inflator? Is it going to be considered going forward a really strong first class feature? Are we going to get like private off support or is it considered maybe potentially not the most first class? uh I yeah I think uh her chart in customize em just a bit too. All right. Can we talk? Sorry. No worries. >> Are there any other questions?

>> We got about five minute a little less than five minutes left. >> You're literally doing this on purpose. >> This is what happens when you get elected to steering. Actually, >> I am steering the boat from one end to the other. So are there any plans to support arbitrary sub resources in cube cuttle other than scale and status especially sub resources that has the usual crowd

operations. So I'm not talking about logo or port forward but the simple ones. >> Uh the simple answer is no because um in the API we don't support it. the the if you're creating an a CRD uh only status and scale are supported. You can't create their your own unless you're creating your own uh API server. So that is the reason why we're not thinking about supporting

additional sub resources. Probably got time for one more. >> you're killing me. >> one question. One last question. Anything. >> I have a question for y'all. What is currently stopping you from joining us in helping to build and shape your tools? Is it your boss wouldn't give you time to to work on open source? Is that the answer for most folks? Go ahead. >> Okay. Maybe start

a war. Uh, cube CCL, cube cuddle, cube control. Which one? >> No, our logo is officially a cuddlefish. So, I mean, >> is cube CCL valid? One of my co-workers says it's not. >> Everything's valid here, man. Like, >> yeah, everything is valid. Yeah, completely. I mean, we went with the cuttlefish just because it was nice to say cube cuddle. Uh but uh if you hear even

the longtime cube maintainers are using all three of them cube cuddle, >> and cobact yeah. >> I'm not using that one. That's mouthful. Uh I'm gonna stick with the old ones. Thank you very much everyone. Uh we'll stick around if you have any other questions or you want to talk. Thank