Policy Engines for Kubernetes: Picking One Without Losing Your Mind - Nabarun Pal, Broadcom

This talk discusses the complexities of policy enforcement in Kubernetes and the available solutions within the cloud-native ecosystem. The speaker highlights the importance of compliance requirements, cost control, and operational sanity when implementing policies, emphasizing that effective policy enforcement goes beyond setting rules on paper. He explores various policy engines, including Kubernetes native validating admission policies, Kyverno, and Open Policy Agent (OPA). The speaker provides insights into the trade-offs and considerations for each option, such as ease of adoption, resource footprint, and the need for mutation capabilities. Ultimately, he advises starting with simpler policies and scaling up complexity as necessary.

Welcome everyone. Uh, thank you for making it to one of the last talks on the first day of the conference. I'm happy seeing like so many people around. Um, I'm Nabarun. Today I'm going to talk about something that sounds simple but gets complicated really fast. Policy enforcement in Kubernetes. So, can you all raise your hand? Oh, even before I ask the question, we have some hands. did

you ever have a pod running as root in production that should not have been that way? Huh. And how many of you, keep your hands up, and how many of you figured it out during an incident and not actually because of a policy violation? Oh, not many. So, you all know about policy. Why are you in this talk then? Okay. Let's still go ahead. Uh, oh. Seems

like I have been going through slides. Um, so we're going to look at a framework and a few of the solutions that are available in the cloud native ecosystem to do policy enforcement enforcement in Kubernetes. And I hope by the end of the session you have some idea about like what solution to choose in what scenario and how you can migrate between the solutions like when you

grow or when you scale down um, your infrastructure. policy enforcement does sound very simple but it is not. It only sounds simple on paper. Um, I'm sure like some of these points might close hit to home. Uh, 42 teams um, analogy is my favorite. Um, we feel like things might have been solved but often what happens is different teams have a different way to handle policy and

they might result in like different things or you might have different products running on the same infrastructure which might have their own opinions on doing policy enforcement. Uh it also looks very trivial until you try to do it at scale. Let's say if you have like 10,000 policies um which I hope you don't have any single cluster, but there are instances where you are running multi-tenant apps

and you can run into those kind of pitfalls. So let's talk some real world scenarios on like why you are here. Compliance requirements. Uh let's face it. SOC 2, HIPAA, uh PCI, GDPR, we all uh might have gone into things um in those directions. Now, whenever you have an audit going, there's a difference between we have in policy as an answer and you telling and showing your

auditor the exact audit log on what was audited, when it was audited, and the decisions that led to the audit. That gives the actual value. The second answer actually passes the audits and that is what auditors need, the audit trail or the audit log and not just, "Hey, this is what we are doing." They want like, "What has happened after you implemented your audit logging?" Cost control.

Um numbers are entirely hypothetical, just made up by me. Um can be different dependent on what you are actually running. Um before enabling any sort of policies on resources, your uh teams, dependent teams, might be running workloads with infinite resources or not have resource limits requests or uh anything of that sort defined. This result in like real money savings, real cost control, and again, these are all

numbers made up just to prove a point. And then operational sanity. Like Did you run into a pod crash because of no readiness probe? Or whom kills? These are all scenarios which can happen in reality. When you have policies, those conversations are not happening at runtime or when the failures happen, but they are happening when you are actually deploying the workload, when they are going through your

seed CICD pipeline and you are doing a PR review of the deployment. That is when things happen. And yes, um on-call engineers actually do sleep better with policies. Um I have data on this. It's my own sleep tracker. Um There might be people here who have been on on-call and have been paged really late at night because of such incidents, but these are all avoidable. So, let's

talk about some of the parameters that actually matter while selecting a policy engine. The policy language and the expressiveness with which a policy is defined in your infrastructure. And I I think I would argue that this is the most important thing when we are talking about like creating policies on And it And how how long does it actually take for anyone on your team to get started

into a policy engine? Um The answer may be drastically different different depending on Do they need to learn like 2 days of YAML or do they need 2 to 4 weeks of training on some other policy engine language? And it can make a day and night difference on adoption. Operational over overhead. I would actually want you all to screenshot this. Actually, don't don't take a screen grab

right now. Anyways, I'll post this as a hand handout notes in on Sched which you can check out. But But, about all these questions. How many new components are you deploying in your infrastructure just to do policy enforcement? What happens when the policy engine is unavailable? What happens to your workloads? What's happened What happens to your deployment pipelines? How do you upgrade without breaking existing policies? How

What is the memory and CPU footprint? What are the resource usage? How do I back up back up your uh policy state? How do you do DR? Those kind of scenarios actually make a lot of sense when choosing a policy engine. I think these questions should be asked when you choose the engine and not when you actually have an incident. Things like upgrade, HA, DR. So, day-two

things actually should be thought on a day zero and decision should be made accordingly. Next up is failure modes. when we talk about webhooks in the Kubernetes space, there's always a question on what is the failure mode. What if the webhook is not available? Do you fail? Do you ignore? Those kind of decisions are really important. My typical recommendation is start with ignore, prove that the engine

is stable, it works, and then go into the blocking mode. We'll get into like what those engines do and how do you do those How do you set these flags in each of these engines? you also need to take care of the global timeout trap. In Kube API server, there is a 30-second limit total on all webhook requests that can be made as part of a request-response

flow. And this is not just for one single webhook. will be total of all webhooks if you have something which injects some uh basically which mutates the uh manifest, inject something, or if you have like other custom webhooks that you use to define something else or do some other decisions. All of those add up, and it should be less than 30 seconds. Otherwise, you are uh you

are running into trouble in production. Then, integration in ecosystem. Can you validate these policies before they even hit your cluster? Ideally, everything should be left shifted when we talk about policy. Let's say if you are using Argo CD. You are using GitHub's workflows. You need your policy engine to really play well with that model so that you can validate policies, you can maybe dry run them, give

your developer some tools so that they can validate the policies ahead in time before they even get to the cluster. Auditability, obviously we talked about compliance requirements. These are not really hypothetical scenarios. Auditors will ask you, "Show me all the resources that validated policy XYZ in the last 90 days." Or can you prove that no privileged pods ran in the production namespace? Having an audit log really

gives you that power to say like, "Hey, nothing went wrong in my cluster." This is where the 847 blocked violations evidence I mentioned earlier actually comes from. If the policy engine can't answer these questions, it's really futile when we talk about that dimension. Now, let's look at some of the main choices. One of them is Kubernetes native. How many you of you have heard about validating admission

policy? Have like played with it? Nice. Good good amount of people who have tried the native way out. it's nothing but common expression language. It's baked into Kubernetes API server itself. You don't need to run anything additional. That's the power of validating admission policy. No new webhooks, no external dependencies, no network calls. Although there are some gotchas that we'll come into you have to keep those in

mind while you vet Kubernetes native validating admission policies. Here the failure mode story is completely different. Instead of asking, "What if my webhook pod crashes?" you are asking, "What if my API server crashes?" Well, we run into a lot more different problems when we think in that failure domain. But it is also a much rarer event that affects everything in your cluster anyway. Now, let's go through

an example. So, we talked about blocking privileged pods and we'll take this example a lot of times in this talk. When you do validating admission policy, you essentially have two different objects. One which defines the actual policy, the other one binds the policy to, let's say, namespace. So, you can gradually get rolled into Kubernetes native policies by opting in namespaces or opting in different kinds of resources

by using like match expressions. And it's up to you that how gradually you roll out. You start enforcing things on a staging namespace, verify that nothing breaks, then expand your scope into production namespaces. And you don't need to flip the switch entirely on day one. Now, what Cell can do? It's basically specked out for safety and performance. It is intentionally non-Turing complete It's guaranteed to terminate, that's

why we use we use it in Kubernetes. No infinite loops, no runaway expressions, and worst case complexity is also predictable. Although we need to give it some bells and whistles to make the whole cost computation predictable. We'll talk about it in the caveats of using uh web. Now, what it cannot do. It cannot look at your all your resources in the It does It doesn't have capacity

for like complex data operations. It doesn't have capacity for HTTP calls. Uh mutation is conditions applied. Um mutation Mutating admission policy is coming as GA as part of Kubernetes 1.36 in a few weeks. Uh so, you get to use mutation as well as part of the native uh technology. You do You also don't have support for image verification. We have been seeing a lot of supply chain

attacks with respect to uh mutating tags and uh non-verifiable uh images. All those things are very important and unfortunately, Kubernetes native web doesn't have capacity for that, but other solutions that we talk about do have those. Now, cost-based restrictions. I was talking about cost a lot. it is one of the other things that surprises people in production. CEL estimates the computational cost of your expression before even

executing it. So, when you apply the validating admission policy, it actually computes the cost and tries to store it in memory that what will be the run time cost when it tries to actually execute the expression on an object. Now, complex operations like this, if you do like, can I if can you check if any of my annotations actually starts with allowed?" Now, if you have an

unbounded number of annotations or if you don't have a maximum length set on any field in your CRD, you are going to run into those gotchas. So, the what is the solution? In any of your properties, you define like max properties or max length if it's a string type. This helps the cell computation predictably store the cost computation and give it a bounded value. Otherwise, you are

again running into unbounded issues. Now, when to use native policies? Here's a very simple model. Use native app for simple checks, field validation, label requirements, naming conventions. For mutating mutating admission comes out with closing the gap for straightforward use other straightforward use cases on mutation like adding resource requests if nothing has been specified. And when you need additional complexity like image verification, you bring in other engines.

So, you grow gradually. I believe most clusters and most teams should start from here because there are zero additional components to manage. There are zero webhooks. It's zero additional webhooks to make available. It's just Kubernetes. Now, when not to use native policies? And most importantly, it's either or an or discussion. You can add Kvern or Open Policy Agent Gatekeeper later and your native policies will still be

working. So, think of use cases like you need to use mutation, but then again mutation is coming. You need to verify image signatures. We already talked a lot about. You need complex logic which requires external data. There are other solutions which take care of it. Now, I was talking about mutating admission policy. It's coming in GA. So, you can basically add object metadata like labels, annotations, or

other fields based on like what the object actually has. So, do look out for Kubernetes 1.36. There are some more interesting stuff coming in later versions of Kubernetes which takes away some part of the admission pipeline, little left shift towards authorization. There There are There is work going on on conditional our back which exposes some part of the object in the authorization pipeline. So, you can write

authorization uh web hooks or authorization uh policies which will help you figure out something based on the user, what who the user is, what privileges do the user have, as well as the object. So, uh that will come in as alpha as in 1.37. So, do try out the feature and give the community feedback. Uh come to the KKR repo. File an issue. Search if an issue

is already there. Uh give feedback. Now, coming to the next one, Kyverno. Well, it's worth looking at because it's just YAML. It doesn't have much learning curve to get started compared to native app or compared to other things because your team is already writing YAML. As I said, no no new language to learn, no new compilation step like we'll see some other policy engines. No separate testing

framework. If you can write Kubernetes, you can also write uh Kyverno policies. Now, the architecture is also fairly simple. You just have one uh controller. You just have one pod running a running as a web hook and inside that there are processes which take care of different parts of the life cycle. So, Kyverno has basically four core policy types. First one is validate. Uh same similar concept

like this validating policy basically checks whether the the label is the team label is set in a workload or not. And if it is set like if it is set to a non non-zero length value. It's quite simple. And then you will notice that validation action actions is set to deny here. What I actually recommend people is to start with audit, get logs, and see try out

whether the policy works and how many uh workloads actually fail. And one good thing with Kyverno is like it supports background scanning. So, it will in the background also keep uh checking for compliance and keep generating reports. So, you actually can gradually grow into uh putting policy enforcements on your infrastructure. The next type is mutate. It basically enforce it you can do default enforcement like I was

mentioning resource requests, which helps you keep your infrastructure usage reasonably sane. The There is one very big practical impact is that developers don't need to remember how to set or where to set resource requests. This will be done automatically for them. Not very suggested um in in the context of verbosity, but it still helps you put some safeguards. The third type is generate. So, Kyverno has a

very interesting thing that it can help you generate resources on the fly based on other requests. For example, if you create a new name space, it can take a network policy from another name space and can generate that in the name space that you just created. What it helps is for example, you want to deny all requests by default. And you don't want to create a new

one or you want to ensure that it's created even before some somewhere down the some somewhere down the pipeline your bootstrap process would be creating this, but it will make sure that it's actually created when the name space is created. And good thing about this is uh there is something called synchronize. So, if your source object changes, it also synchronizes the objects across all the namespaces or

all the constraints that it had created already. It's enormously powerful. Which which we can see here. And then the fourth part is image verification. We talked about a key flaw of native app is that it can't do image So, let's say you say that any any image that you get has to be signed using cosign like which matches gcr.io/myorg has to be signed using cosign using a

specific subject and a issuer. This utilizes the cosign machinery to make sure that your images are actually signed and it can also check for like digest. It can also check like whether digest have mutated or not. So, you actually know like what bytes Background scanning. We I talked briefly about it. Now, the key benefit to background scanning is let's say you already have a lot of workloads

running in your cluster. many many other solutions will just give enforcement at creation time or update time. But what about the resources that you might not intend to change? You don't need to necessarily change resources just to enforce policy requirements. Kyverno helps you by doing background checks and it will say like what validate validation action it should do. Again, I suggest like starting with audit. when you

implement background scanning, you get a full compliance picture even without like creating any resources or touching those resources. And it's not just for like new resources created. Uh, another good thing is uh, uh whenever there are failures, everything gets logged into the policy report. This is like a standard API from the CNCF policy working group. Uh it's not very Kyverno specific. Gatekeeper also writes to the same

format. So, when I talk about Gatekeeper, the same holds true for Gatekeeper as well. So, you can actually build like compliance dashboards by looking at all these resources, parsing them to your liking. Um and then you can basically show your auditor. I think there's a also tool called policy reporter, which turns all these reports into neat dashboards. Now, let's look at production considerations. These are all just

some suggestions. It might depend on like how big your infrastructure is. For example, like I really suggest everyone to run like three replicas at least for high availability. In case one pod goes down, leader election will take care of like getting another uh pod as a leader, which will keep like doing the same things. You And another consideration is like start with uh failure mode ignore. What

it does is So, these are all like the Helm chart values for Kyverno. If you deploy using a different way, you might have a different deployment topology and different way to specify these values. What this does is it proves that your monitoring is in place. You can get more reports, and then eventually you graduate to fail. Uh Kyverno also caches resource data. as your infrastructure grows, the

requirements of resources will also increase. So, you need to keep monitoring Kyverno's actual resource usage to figure out what what or if you need to change these values and update the Kyverno deployment. Um and I hope like you don't get to see like OOM kills at 3:00 a.m., and then you eventually get paged because of the solution that you did not want to get paged. so, here

are just some goods and trade-offs. The Just YAML strength also has a limit. For example, complex policies with many nested conditions might get a little difficult to implement. Kiverno did move out from uh a model where they used a single cluster policy resource. They deprecated it and then moved into all these four types of policies to make the API more I say versatile. Um and yes, one

limitation is Kiverno is Kubernetes only. You can't use it to validate, let's say, Terraform plans or evaluate CICD configurations. Uh for many orgs, that might be fine, this is true for only if you are solving the Kubernetes policy problem. But let's say if you want to deploy like policy enforcement across your stack, across your infrastructure, then you might need to look at things like OPA. Which brings

me to Gatekeeper or OPA or Open Policy Agent. >> [snorts] >> Now, before I show you any Rego, I need to clear something. Uh because I see this confusion consistently across the space. OPA and Gatekeeper are not the same thing. OPA or Open Policy Agent is a general-purpose engine. It speaks Rego, which is the policy language framework And it is not Kubernetes specific. Gatekeeper is actually the

Kubernetes integration. It provides CRD-based policy management, the constraint framework, audit capabilities, and all those bells and whistles. It's built on top of OPA, but it's a separate project essentially. Uh if you are on managed cloud providers, you might already be using uh Open Policy Agent. It's just that you might not be oblivious to it. Uh what does Rego uh Rego look like? It looks strange when you

first see it. Uh, different from the imperative languages that we have seen earlier. But, let's see what is the core of this It computes the set of provided labels. It computes the set of required labels. It tries to find what are the missing labels. And then, if only the count of missing is zero, it generates a message. And then get it gets bubbled up as part of

the audit policy audit uh, pass or So, if you now read it again in a verbose format, you can really understand that it's not very uh, difficult or it's not very odd. It's just that we have been looking at things imperatively all this while. It's actually quite elegant once you have a deep look into it. Now, the key mental model for Rego is every statement in a

rule body must be true for the rule to fire. Uh, it's it's an implicit and between every line. And the violation rule builds a set. You get one entry per violation found. Now, teams that end up investing in learning Rego, they are able to make more expressive policies imaginable. The ceiling of imagination is much higher compared to like cell or just YAML-based validations of Kyverno. Now, Gatekeeper,

just like native apps, uses a two-layer system. Uh, policy library maintainers write things like constraint template, which ends up creating a Kubernetes CRD, uh, like K8s required labels. And then, you can write a constraint, in this case like the K8s required labels resource, which instantiates a template with specific parameters. That these namespaces, these labels should be there. On these namespaces, for pods, these labels should always be

What this enables is like a central security team can maintain the library of constraint templates, whereas individual platform teams can instantiate them with their own parameters, which gives them a little bit of separation of concerns like different teams can maintain their own spheres of influence, and the constraint is the local configuration. So, you get centralized governance, but with distributed control. So, there is a sense of left

shift ideology along with giving more control to your platform teams. One very powerful thing with Gatekeeper or let's say Rego is it can sync cluster data for cross resource validation, but there are some caveats. So, in this So, in this in this example, we're saying like sync namespaces and ingress objects, and we use this later to say like check if the ingress host name conflicts with an

existing ingresses. And this is what this is how you can write the Rego Looks a little complicated compared to all that we have seen with respect to cell, but if you see both solve like different use cases. This solves a very drastically different use case than what validating admission policy was doing. That was scoped to one particular object, whereas this looks cross object. I talked about some

caveat. The caveat is memory usage. So, the more resources that you sync, the more number of resources that you have for each object, the memory overhead of Gatekeeper will keep on increasing. So, you have to be very aware of resource usage when you use this feature. Now, the power of Open Policy Agent is it's beyond Kubernetes. It works everywhere. And here's why some organizations choose OPA even

when their Kubernetes policies are relatively simple or their requirements might be really simple because it works everywhere. It is the same Rego language. It's the same evaluation engine. It's just your Kubernetes admission policies get added on top. And quick show of hands, like who here uses Open Tofu or Terraform? Oh, I expected a lot more. Who has custom authorization logic in microservices? Like building your own authz

layer in your services? Okay, less people. Um who has multiple cloud providers? Okay, quite a few are on hybrid cloud. Those are all the cases where having a unified policy language actually pays off. And Open Policy Agent is the only possible solution that I see right now which can deliver that. Now, production considerations. Um resource footprint, again, as I mentioned, you have to be diligent about what

how many policies you are trying to create, how many resources you are trying to uh sync as like the cross-resource scanning. And then learning curve. So, basic constraints might take a couple of days. Uh custom templates, maybe a few weeks. Advanced data replication, maybe a month [snorts] or a couple of weeks. These are all like anecdotal evidence. It depends how complex or how advanced your team >>

So, you have to take care of all these considerations when you choose uh OPA. Now, my honest recommendation is if your policy needs are Kubernetes only and your policies are relatively straightforward like label requirements or security context checks or resource limits, Open Policy Agent is probably more than you need. Uh the investment is real when you choose Open Policy Agent and if you're not getting the cross-tech

cross-tech benefit, it probably will not pay off over time. Uh the sweet spot is actually when you are cutting across like infra layers and different types of infrastructure, which is when this really pays off. The last one that I really want to talk about, although we are running a lot on lot out on time, I'll just quickly go through. Uh Keywording Keywording takes a very favorite approach

of the uh using WASM. It is very shiny. It gives you a unified interface or a language flexibility to define uh policies like in Rust or TinyGo or anything which uh compiles to uh WASM. Then one key benefit is distribution. It allows you to distribute policies as uh OCI images. Then you can use those OCI images in a sandboxed environment to run your policy uh executions. Language

choice is another thing which is better uh in this situation. The limitation comes to like compilation complexity. Like you have to take care of like what language are you writing in? Does your team have expertise in those language? Basically the benefits become disadvantages sometimes. Then the debugging experience. You have to keep on like compiling, pushing, pulling, checking whether your policy works or not. Then the ecosystem maturity.

Uh the WASM ecosystem is still evolving. Uh it has evolved quite an extent, but it is still evolving. I believe like Keywording will become more mature over time. it's one of the most specialized tools of this four. It's the right choice when your team already has systems programming experience, you have high security requirements that genuinely benefit from WASM isolation, and you're actually willing to invest the time

in building that pipeline. But, if you're evaluating If you're starting from scratch, then other options are more worthwhile to look at. That said, watch this space of Kyverno, things might get better over let's skip the comparison. You can see the handout notes we've talked about a lot. Let's come to this one. This is important, which is probably why you are So, this is a very basic decision

tree. Things might get complicated, but this helps you answer a few key questions. First, do you need mutation or generation? If you need to modify resources at admission time or create resources when other resources are created, that takes you towards Kyverno or Within that branch, do you need image verifi- image signature verification? If yes, then Kyverno. If no, then the answer is not very clear. if you

if you're having complex logic, then go to Gatekeeper or Kyverno, otherwise. Now, if you don't need mutation and you're and simplicity is your top priority, then use natives. Or if you need cross-platform or cross-resource checks, then choose between Gatekeeper or Kyverno. Now, for real-world recommendations, there are some scenarios like if it's a platform team at a startup, go with Kyverno. Easy to start, easy to learn. If

you're in enterprise multi-cloud, going over like cutting across layers, see if Gatekeeper makes sense for you. If you're a security-first fintech, maybe use a dual-layer approach where you have most of your simpler things at native layer, and then you have Kyverno for advanced use cases. Then you have greenfield, uh small teams who are trying to get into policy Native app, try it, grow eventually. Then there are

migration paths. We are already out of time. You can see these in the handout notes. There is nothing much. These are just real life guidelines on how do you migrate between each of these Now, some lessons that don't fit in marketing materials. Start in audit mode. Don't go full blown >> block right away. Start with audit. Make sure everything is fine. Then only tighten things down. Exception

management. So, once you keep on like auditing policies, you will find that a lot of things need exceptions. Like for example, if you run Istio, you would need it to be in the host network mode. You would need the ports to be privileged. So, you would need to create an exception for those. Webhook timeout tuning, we already talked a lot about it. You have to tune that

as well. You have to tune the timeout. For most use cases, I don't think more than 5 seconds is needed. I I have not yet seen like more complex policies which require more what do you say, timeout of webhook. Unless like there are problems like your actual policy controller is facing The next up is resource limits. As you grow with as you grow your infrastructure, your policy

engines resource requirements will also grow. So, keep tuning them. Then policy testing in CI/CD. So, try to test things not in production, but as part of your pull request workflow. We don't want anyone to wake up post midnight or even at 11:00 p.m. for such incidents. a break glass procedure. For example, like what if if actually have an incident? Don't try to debug things during the incident.

Try to see if you can spin down things. Try to see if you can uh disable policy enforcement on a particular section of your cluster just to debug just to make sure like your users are not facing a downtime. And then debug and then get back to your work. Closing notes. Um policy engines and uh treat them with the same rigor as your ingress controller or service

mesh. Uh don't think like uh replica count one is enough. Treat them as equal as other infra services on your infrastructure. There's no universal best choice. Whatever I spoke here, things might not be the same for you. Please look at all the advantages and disadvantages and the requirements of your infra and then make a choice. Start simple. Grow as you need. If needed, start with native, then

grow out to Kyverno. Then if you're trying to run into multiple stacks of your infra, then go to Gatekeeper. Always audit mode first and then plan for exceptions. Having said that, I did not expect that it would run out so long. but I am happy to take any questions like offline because I don't think anyways like this will go into recording the later parts. I'll be here

in the hallway taking your questions on anything. Thank you all for coming. >> [applause]