Stop Flying Blind: Real-Time, Enforceable Cluster-Wide Quotas with Ky... Mariam Fahmy & Adam Crowder

This talk focuses on implementing dynamic cluster-wide resource quotas in Kubernetes using the tools Crow and Kyverno. The speakers, Adam Crowder and Mariam Fahmy, discuss the challenges of managing resource quotas in a multi-tenant Kubernetes cluster where various teams work within different namespaces. They explain that while Kubernetes has a native resource quota feature, it lacks the capability to enforce a cluster-wide quota across various namespaces. The solution involves using Crow to create a cluster-scoped resource quota and manage the associated state and policies, while Kyverno acts as a policy engine to validate and enforce these quotas. By defining resource graph definitions and employing cell expressions, the setup allows for automatic monitoring and enforcement, providing both visibility and control over resource usage in a dynamic environment.

Cool, thanks everyone for coming. We're going to talk about implementing dynamic cluster-wide resource quotas with Crow and Kyverno. But before we get into that, we'll start with some introductions. So hey, I'm Adam Crowder. I'm a software engineer at AWS. Specifically, I work on EKS, which is our Kubernetes offering. Worked on a bunch of stuff there over the years, but nowadays I spend most of my time trying

to make Kubernetes easier to use both within Amazon and externally as well. Yeah, Miriam. Hi everyone. I'm Mariam Fahmy. I work as a software engineer at Cloudflare and I have been working in Kyverno for 2 years and a half and I'm one of the Kyverno maintainers. That's it. Awesome. Let's get into it. Yeah, just a second. The screens doesn't show our slides. Someone Oh, perfect. All right.

So for this talk, we're going to walk through this theoretical scenario here where you're a platform engineer, you're running a Kubernetes cluster, right? For your team of developers. And specifically, you're running a multi-tenant Kubernetes cluster. So we've got multiple teams operating within the cluster and each team as well has multiple different namespaces. So multi multi-tenant multi-namespace. And what we'd like to do is essentially enforce like a

cluster-wide resource uh against essentially each team across namespaces. So if you're not familiar, Kubernetes has a native built-in resource called resource quotas. It's a namespace resource where you can essentially define like CPU and memory limits and stuff like that. And then as pods come up in that namespace, Kubernetes can enforce that we're not essentially violating those those So those are namespaced. And so if we're taking a

look at a solution here, how we can build something that essentially enforces a cluster-wide quota using these namespaced, we run into a few challenges here. So let's say we're just orchestrating these individual namespace resource quotas. So first of all, we don't really have a native way to aggregate across all the different namespaces. We have to keep track of things like which namespaces are being used, where the

resource quotas exist, which teams are associated with them, etc. And you know, we can write scripts and whatnot to query for these, but essentially there's no essentially native resource for this, right? Aside from that, we don't really have a good way to enforce budgets across namespaces. So if we want our developer teams to be able to tune their own resource quotas, we don't really have a higher-level

way of allowing that while enforcing a cluster-wide quota. So we could manage all of the resource quotas within the cluster ourselves, but then we have to have some sort of solution for our developer teams to come back to us and either ask them us to change it or we have to build some sort of automated solution or what have you, right? And there are you're probably thinking

in your head right now a bunch of different solutions for this and of course there are, right? We could just do this manually, we could write a bunch of scripts, we could maybe do this, you know, quote unquote properly and build like a Kubernetes controller for this that you know, is monitoring and managing these resources for us. And that's all possible, but all those solutions essentially require

like a bunch of manual effort or like maintaining this like big effort like Kubernetes operator controller. And so we can do a little bit better But yeah, really the three core things that we're looking for in this solution are one, visibility. We want a centralized place where we can view all of the the resource quotas across the whole cluster. We want enforcement. So we don't we want

to allow our developers to be able to go and change their resource quotas within their namespaces while not violating like a cluster-scoped policy. And we want this to be automatic. So obviously no manual scripts, we don't want to keep track of things manually. We want to be able to automatically update whenever we see new namespaces and stuff like that. So how are we going to build this?

So we've got a couple tools as the title of the talk might imply, right? We're going to use Crow and Kyverno. And we'll talk more about these tools in just a second, but at a super high level, we're going to be able to use Crow to create that cluster-scoped resource and then also manage and kind of gather state from all the resource quotas from within the cluster.

And then we're going to be able to use Kyverno to actually be able to enforce these quotas and specifically allowing these teams to edit resource quotas without necessarily going over what we provision for them. So let's talk a little bit about Crow for a second. So what is Crow? If you haven't [clears throat] heard of it, it's a fairly recent project that we started and launched that

actually we recently donated to Kubernetes SIG a few months ago. So we're still actively developing developing it. On the surface, it's looks kind of like a way for you to essentially define a set of resources and basically create an interface for it. Basically Crow create a CRD for it. Define a set of resources with an RGD, a resource graph definition. You can have templating for what resources

are included in this and then you can stamp out essentially copies of these these templates with a CRD itself. So with Crow, it's a little it has a lot of overlap with something like Helm except it's a little different because it's actually an active operator, right? So one of the main differences is Crow is continuously running in your cluster and monitoring all of the resources that you're

creating and it's exposing its interface right through an in-cluster CRD. So it's kind of a declarative way to do this resource management inside of Kubernetes while still allowing you to have like templating and all the things that you might want for for declaring a set of resources. But yeah, I'll pass it off to Mariam here to talk about Kyverno Okay, so what's Kyverno? In short, Kyverno is

a policy engine for Kubernetes. To put it simply, you can use it to validate, mutate, and generate resources. And other than that, you can use it to apply it for like, you know, write Kyverno policies to evaluate for example Docker files, Terraform files, any JSON manifests in short. But for the sake of this session, we will only talk about how to use Kyverno inside Kubernetes cluster. let's

say that you have a use case where you need to enforce that every pod must have an environment label. The easiest way to do so is to write a Kyverno validating policy that matches pods and in your validation logic, you use cell expressions to say that every pod must have this environment label, right? And the way it works is so straightforward. Whenever a user creates a new

pod, this request goes to the API server and then the API server will forward the request to the Kyverno because Kyverno is deployed as an admission controller. And once the request hits Kyverno, Kyverno will start fetching the existing policies and see if the pod matches any of these policies. And if yes, then Kyverno will start will start and evaluate the Kyverno policy against the pod and it

will simply check if the pod violates the policy or not. And depends on the result, it will return either reject or accept to the API server. So if the pod violates the policy, this means that it it doesn't have the environment label on the pod. So it will reject the request, the pod will never be created. Otherwise, it will be allowed. Okay, so now let's have a

look at how we can use both Kyverno and Crow to achieve both visibility and enforcement for the So everything starts with you as a platform admin. Instead of creating and managing dozens of Kubernetes resources, you simply create one single resource per team, which is the cluster resource quota. And we use Crow to help us generating the custom resource definition of this cluster resource quota. And as we

will see later, this cluster resource quota will have a status object that will act as our live dashboard. So you simply it's like a way to just show how much every team is using across the cluster, right? And then once you create the cluster resource quota for each team, Crow will manage and generate two resources for us. First of all, it will generate a watcher for all

the resource quotas associated with that team just to keep an eye for every new resource quota created and then update our live dashboard as we will see later. And the second resource is the Kyverno validating policy. And here Kyverno acts as a security guard for each team. So it will simply block any resource quota that will go over the budget. So by the end of it, you

will have achieved both visibility and enforcement. So Crow will act as our So Crow will like do our calculations and will it will show us like the live dashboard and by that we can see like how cluster like the total usage and cetera. And then we use Crow to act as the security guard to block any resource quota that goes over the Okay, so now let's see

how we can define the cluster resource quota. First of all, we create a resource graph definition and it will simply it will simply contain two parts. The first part is the schema and this is where we define the cluster resource quota. As we see as we see here, we say that each cluster resource quota must have a team name. And then we say that the cluster resource

quota will have a budget field. And this is where we set the hard limits for both CPU and memory requests and limits. And then finally, we say that status. And this is where we show our where we show the total usage for each team. So crew will do the calculations using cell expressions. And then it will simply like this is like a way to say to crew,

please fetch all the resource quotas associated with that team, some sum them up automatically, and then show show the result under the total usage. So now that we have the cluster resource quota created, um crew of course will create the custom resource definition of us. Now we need a way to tell crew, please whenever a cluster resource quota created for the team, automatically manage two resources, the

quota watcher and the validating policy. So how we do that? Uh we, you know, in the in the resource graph definition, as I mentioned, there are two parts. The schema part and this is where we define the schema of the cluster resource quota. And the second part is the resources field. This is where we define both the watcher and the validating policy. As you can see, we

say that we have a team quota and here on and within it we use the external the external reference field where we say, please please create a watcher for all the resource quotas related to that And then we say uh the second resource that crew must manage is the validating policy. So here we say we will have a validating policy that match all the resource quotas related

to that team. And then inside the validation logic, we uh please calculate all the existing uh resource quotas, sum them up add them to the new request and see if this exceeds the budget or not. to put it simply, the cluster whenever a new cluster resource quota is created, crew will manage both the watcher and the validating policy. And whenever a new resource quota is created, this

request goes to the API server, and then the API server will forward it to the Kyverno, and Kyverno will see if the resource if this new resource quota matches any of the existing policies, and it will do the calculations for us. So it will simply list all the existing resource quotas, add them up, and then try to check the new request and see if you know, the

result will exceed the budget or not. If it exceeds the budget, then Kyverno will reject the resource quota from the beginning. So it will not be created at all. And finally, so this is the summary. Like whenever we have a new team, you simply create a one resource, the cluster resource quota. This cluster resource quota have the status where you can see the total usage of of

every team. And internally, crew will create for us a validating policy. This is our security guard. It will enforce the budget for every team. And it will also create a watcher for us. So it will simply watch any new resource quotas, and then add them automatically inside the cluster resource quota. So in the end you have this live dashboard where you can see every team how much

every team is using across the cluster. to get the live dashboard, you just run Cube Cattle get cluster resource quota where you can see the total usage for And then if there is if there is resource quota that's is about to be created and this exceeds the budget, then as you can see here, Kyverno will block it. So it doesn't go And finally, this this solution is

not just for one team. You can simply create cluster resource quota whenever you have a new team. Uh so at the end, you manage the cluster resource quotas, and you write you just create the resource graph definition only once. So the first the first manifest that we have showed is just created the for for the first one for the for the first time. And then whenever there

is a new team, you just create the cluster resource quotas where it allow you to show the the total usage of every team, and automatically it will create both the validating policy and the watcher for the resource quotas, and that's it. So now we see that we see a demo from Adam. Cool. Yeah. Let's take a look. Cool. So when explaining kind of at the core of

this, right, is this resource graph definition. So when explaining these, I kind of like to start from the the end state here. So as Miriam was just talking about, this is going to be our cluster resource quota kind of interface that we've got. And this is an example of kind of what we'll be applying to enforce this this cluster resource quota against this production team. So as

you can see, we're interacting with just a cluster scoped CRD here with no name space. We've defined some budgets and whatnot. Um and so now I'm going to jump over to kind of the RGD here and kind of show how we do this. So um as Miriam mentioned, uh RGDs kind of have two different parts. So first, they have a schema, and this is because crew exposes

its functionality by creating and managing CRDs. So when we create a resource graph definition, crew's going to go create a CRD in the cluster that then we can use. Um and so this first part, the schema, is basically going through and defining what that CRD looks like. So uh what we've done with crew here, if we take a look at the schema, so we can see we've

defined the name here and cluster scope. And uh we have a spec here. Um so normally when you define a CRD, you have to essentially define an open API schema. Um what we've done for crew is we built something called simple schema. So as you can see here, it's just a little bit of an easier more human readable way to kind of define the spec here. So

we've defined three fields that we want as the the interface here is team name, which is a string and we have set to required. Uh a description, which has a default and is is optional. And then a budget, which for the purposes of this demo is just an untyped string map, but we could type this out if we wanted to. Uh so we can also have crew

essentially fill in custom statuses for instances of this CRD once it's created. So that's if we look down here in the status section, we can see the cell expressions here, and I'll jump back to that in just a second. The important thing to note right now is that all these cell expressions are using this quotas variable. So I'm going to jump down to the second part of

the RGDs, which are the actual resources. So crew kind of has two different types of resources. So there are the typical resources, right, where you would just define a Kubernetes resource and you say, whenever there's an instance of this CRD created, I want you to go create and manage this underlying Kubernetes resource. Um there's also essentially read-only resources. So we can tell crew, hey, I actually want

you to go find a bunch of stuff in the cluster and make it available to me through like for through like a variable for cell So [clears throat] actually that's what we're doing here with this first So we have this ID quotas here. And because we've we have external ref here, this is telling crew, hey, I want you to look at and find all these resources, but

I don't want you to touch And so in this external ref, we have essentially we're telling it, hey, go look for all the resource quotas which match this label selector, which for the purposes of of this implementation we're saying is a label based on the team name. And so that will now make all of that data available to us for templating. So if we scroll back up

now to the status, we can actually see here. So that's actually what we're using here in the status. Crew is going to go spin up going to go list and watch for any changes to resource quotas that match into the status here. So we can see here for quota count, it's just listing how how many quotas that crew is looking at for this team. Um and for

the CPU and memory here, we have some expressions which are basically just iterating over these quotas and pulling out some specific field that we want to look at and basically adding them all together. So crew is going to do the work of actually going and finding all of those resource quotas throughout the cluster based on a label and allow us to essentially do templating based off of

them. Cool. And so now we'll look at the other type of resource here. So the other thing that we have to do is we have to wire up Kyverno. So we have to set up that validating policy whenever this is used. So the second thing we have here is a typical Kubernetes resource. So we just have template defined here. And in this case, we're making a validating

policy with Kyverno. We've also got a crew feature here so that we can actually tell crew, hey, go ahead and exclude this resource entirely if for whatever reason we created a quota that didn't have any CPU or memory Um but in the typical case, crew will go ahead and create this So if we take a look at it, we can do templating within the policy, and essentially

this is where we're defining that Kyverno logic that that Miriam talked about. So if we take a look here, um we've got essentially a rule here telling Kyverno when it should use this policy essentially. We're saying, hey, go ahead and look for any sort of create or update action to any resource quotas, and specifically ones that that also match that label selector that we mentioned earlier. Then

if we continue scrolling down, we also have some cell expressions in Kyverno to basically make some variables available to it. It's looking at things like the CPU and memory of those objects and essentially making them available as as variables so that then we can actually go and use them in our expressions here. So, these are the actual expressions that enforce that validation logic. Um and basically what

we're doing here is we're saying, "Hey, if the the changed CPU or memory plus the current usage is beyond the amount that we have defined in that CRD, go ahead and deny the request and return this this custom error message." So, let's go ahead and apply this and take a look at what that looks like So, I'll go ahead and apply the RGD. And so, now what's

happened is KubeRow has taken it and it's basically now exposed a CRD for us. [snorts] So, it's created a CRD so we can do this here. And we can take a look. If I scroll up here, we can see that it's created this custom resource definition. It's applied some labels to it that KubeRow does. And we can see that it's a cluster-scoped cluster resource quotas. That's the

interface that we wanted. Uh and we can see that it's created this open API schema that was based on our original kind of like simple schema up there. And essentially this is available for use. So, let's go ahead and try that. So, instance that I alluded to earlier. So, let's say we want to apply this resource wide quota to this production team. So, we'll just go ahead

and now use the CRD itself and apply And it's gone ahead and created it. So, now if we take a look, we can take a look at that the instance that we just created If we scroll up, we can see again some KubeRow labels. We can see the spec is essentially just what we've applied, regular CRD stuff. And we can scroll down to the status. KubeRow applies

some standard conditions. And we can also see those custom conditions that we defined here. So, quota count and the CPU and requests for for uh CPU and limits for memory and um and requests and all that. And right now they're set to zero cuz we essentially don't have any resource quotas in the cluster yet. So, now let's say I am the the team that owns production and

I'm going to go ahead and apply a resource quota to one of my namespaces here. So, this is just a the built-in Kubernetes resource quota. So, I'm going to go ahead and apply that. And this will enforce these quotas in this one particular namespace. And now let's go back and look at that So, we can see that KubeRow essentially picked up the fact that this other resource

quota was created with this production label. And it's gone ahead and said, "Okay, I'm now tracking one resource quota against this cluster resource." Uh this one individual resource quota against the cluster resource quota. And we can see here in the limits, it's just values that are essentially just matching the one resource quota cuz we only have one applied now. Um so, let's go over and say we

want to apply another resource quota to a different namespace, but for the same team here. So, it's still got that team label production. And we go and apply that. That works just like normal. And we can take back a look and we can see the status again has been updated. So, now it's tracking two. We can see the quotas here are essentially a sum of the two

quotas put together. And now let's go ahead and edit one of these. So, let's say I'm the production team and I say, "Oh, I actually need to request 20 CPU in this first production namespace." So, I'm going to go ahead and make that edit. And try and save that. And when I do that, we can see here that the request was actually denied because of that Kyverno

validating policy here. And we can see Kyverno the webhook denied the request. And our custom error message here, the team budget's been exceeded for production. CPU requests will bring the total to 21 with this edit, but our budget's only 10. So, essentially we've successfully denied that request. And now this this team with this uh this production team can essentially edit and mutate the resource request, but only

within the limit that we've defined on that cluster resource quota. Um so, yeah. Go back here quick. So, kind of key takeaways from this. Essentially we were able to use both KubeRow and Kyverno to essentially build what amounts to like a basic Kubernetes operator and do implementation enforcement of these these policies all without having actually write any specific operator code or our own any separate components. We

were able to just utilize a few lines of YAML and cell to essentially implement this. we're able to essentially implement these higher level interfaces for these functionalities through CRDs that KubeRow creates for us. And more importantly too, this pattern's now scalable. We can go and reapply additional cluster resource quotas to enforce it against other teams. And now all we're managing is just through this one native Kubernetes

interface. And then we could integrate that into whatever else we wanted. Um so, yeah. I've got some QR codes up here. So, the resources for the demo repo are on the left there. And then the GitHub for KubeRow and Kyverno are there as well. Um but yeah, that's pretty much it. We can take any questions as well, but other than that, yeah, thanks for coming. >> [applause]

>> Okay. Nice. I have a question. How would you prevent teams stealing quota from each other? Cuz if they put the label on the Yeah, definitely. So, like obviously there's a label here and we're kind of trusting teams to apply it correctly. Funny enough, we could use other Kyverno policies to enforce like labels and we don't really have great attribute-based you know, our back built into Kubernetes,

but there's lots of ways and lots of other projects you can use to do But yeah, this particular demo would rely on essentially you trusting the teams to supply the right labels or having some other system, right, that's applying those. Thanks very much for the presentation. It was a really interesting use case to showcase both KubeRow and Kyverno. I was wondering how KubeRow as a project is

going. Is it far off from 1.0? How's yeah, how's that progressing? Yeah, KubeRow's moving super fast right now. We're really excited for it. Actually the some of the features I use in this demo were only with 0.9 release that we released like on Tuesday. It's it's got a ton of momentum behind it. 1.0, I don't know that we have like a public timeline like right yet, but

I can tell you and and you can go look at the repo yourself and go look at all the change logs. It's super actively developed. We have I think like 18 open call them CREPs, enhancement proposals for KubeRow that are being reviewed and and we're working on. So, it's it's very very rapidly developing for sure. Um I guess my question about 1.0 was like in terms of

like stability and and and maturity. Like so, now that it's out of just AWS, Google, and Azure and as part of the seek, that's obviously a good sign. But yeah, like is it considered from a Kubernetes perspective something that's like GA and production-ready or not quite yet? Yeah, I'd say like the the functionality that's there is is quite stable. We have a lot of people already using

it in production. And we're still pre-1.0, so there may be a couple of breaking changes that happen. We're kind of making sure that those all line up with like a 1.0 release. But it you know, the functionality that's there is is is stable. We've load tested it with like millions of managing millions of resources at once. Like we're we're pretty happy with where it's at and I

would say, you know, it's still a pre-GA project, but you know, I would say using it in production, like there's tons of people doing it. And so, I wouldn't say like you shouldn't use it. It's going to fall over or anything like that. Yeah. Amazing. Thanks so much. Yeah, thank you. If no one else has a question, I have another one. Obviously Kyverno is like much more

powerful, but for this simple use case, couldn't you just use the built-in Kubernetes admission controller for this? Yeah, you can use the built-in yeah. Cool. Thanks. It's just that Kyverno provides like extra functionalities, but but for the sake of this demo, so you can either use KubeRow to get the total usage of the of the team and then use it in the cell expressions or you can

just use the built-in not the built-in, use the cell libraries provided by Kyverno to fetch the existing resource quotas. yeah, you have two options. You can use the validating admission policy and use the KubeRow total usage to calculate it and get the current usage and then do the addition and whether to, you know, check if it exceeds the budget or not. But in case you want to

use you know, in case you want to use another mechanism to get the existing resource quotas, you have to use the Kyverno validating policies because we have the functionalities to get Kubernetes resources like doing an API call within the policy itself. So, Okay, great. Thank you. Thank you. Cool. Thanks, everyone.