Advanced Kyverno Patterns : Automating Platform Security and Ope... Frank Jogeleit & Johannes Sonner

This talk presents advanced patterns for using Kyverno, a policy engine designed for Kubernetes. The speakers, Frank and Johannes, discuss Kyverno’s capabilities as an admission controller that validates, mutates, and generates Kubernetes resources. They highlight its reporting system, which helps manage existing workloads and compliance over time, along with features such as the ability to enforce policies across different environments seamlessly. They illustrate practical use cases, including automating access controls in ephemeral environments and integrating infrastructure-as-code tools like Crossplane and Pulumi with Kyverno. Additionally, the speakers emphasize the importance of viewing Kyverno as a comprehensive tool for automation, rather than solely a security solution, and outline upcoming features in its roadmap.

Very welcome to our Kyverno maintainer track today about advanced Kyverno patterns automating platform security and operations. Um My name is Frank. I'm a Kyverno maintainer for, I think, almost 2 years now. I'm also the creator of the subproject policy reporter if someone of you using it. And Yeah, I'm Johannes. I work for Deutsche Telekom and I'm a platform engineering lead there. So, let's take a short look

on our agenda today. So, at first I do a short introduction what Kyverno is, what it does, and what makes it different from alternatives. So, who of you knows Kyverno or using it? Yeah, really quite a lot. That's >> Who who use it uses it in production? Okay. Really great. So, then I keep the introduction short. So, most of you know Kyverno as an admission controller. Means

it's registering inside your Kubernetes cluster as an validating webhook as well as a mutating webhook. So, it intercepts your admission requests and is able to operate on your admission reviews. So, you can validate resources and you can mutate resources. Um it's also possible to generate resources and the background controller enables also background work. So, it's also able to operate on already existing workloads if you want to

apply policies on your already running applications in your already existing clusters. So, but you may know that Kubernetes already has some built-in policies which are called validating admission policies and mutating admission policies. So, why won't you choose Kyverno over it? So, there are some key features Kyverno brings you on top of this already existing solutions. At one we have a really uh good reporting system which makes

it possible to validate, as I mentioned already, existing workloads. So, you can see if you have an application running for months, how it applies to your new policy. And if you want to um make your already existing workloads compliant over time, you are able to do so because you can use the reporting system to show what needs to be changed to comply against your uh policies. We

also, as mentioned, supporting generation, which means you can, for example, auto generate um uh limits and request quotas for new created namespaces. You can synchronize uh central image pull request secret across other namespaces you want to using it as well without the need of copying manually from one namespace to another. And we are providing uh cleaner policies to remove no longer needed resources based on a label

or expressions. We also have some similarities which makes it very easy for you to switch from the inbuilt policies to our policy structs. So, the API is very similar. We are using cell expressions. And we also extending this cell uh with our own libraries to make it even more to make it possible to apply even more use cases like API resource fetching during a policy execution, doing

HTTP requests to internal or external services, um to pass service accounts for easy access, and to um make it make it uh possible to do image verifications yeah, as well in your validation pipeline. So, yeah, as mentioned, these are the full life cycle uh which Kyverno enables you validating resources, mutation, so change a a resource before it's really applied in your cluster, the generation one, the cleanup

resourcing with deleting policies, and the verification part with tools like Cosign, Notary, In-Toto, and so on. So, you can really sure that the image you're loading is the image you want to load in your cluster. yeah. Um as I mentioned, you might know Today, it's not limited to this. So, nowadays, we also providing the possibility to using it for any JSON payload. So, as an CLI tool,

you can it run in your CI/CD pipeline, and um validating, for example, Dockerfiles, Terraform, or any other um file you might be able to convert to a JSON payload. We also have a really new pure project, which is called the Kyverno Auth-C server. Uh it allows you to configure it as an Envoy plugin, and also uh validate your Envoy requests with the same or very similar uh

API uh like the Kyverno Kubernetes validating policies. And we also have a functionality in it to validate HTTP requests, which makes it a solution for, for example, MCP services. And yeah. Um why should you in the end use Kyverno? As I mentioned, it's really easy to learn for people who know Kubernetes already. We are using YAML, as you all do in your daily job, and we are

using cell as an expression language um to write our policies. We're providing the full life cycle of your resources from the validating to generating, mutating, and deleting part. We have an integrate integrated reporting system with subprojects to make this reporting really visible to your teams. We have fine-grained exception, um which means that you can separate the um the application of a policy to a resource if you

don't want to change the policy each time you want to exclude a resource. You have a dedicated possibility to exclude a workload from a policy, and it's really built for Kubernetes, so it's really easy for people who know it uh who know Kubernetes. And now I take over to Johannes. Thanks you so much. Um I have a little bit more or a few more demographics questions about

the audience before we go into the use cases, and that is first of all, uh you heard Frank, uh Kubernetes pretty uh versatile. Who is using Kyverno already for anything that is not at all related to security? Okay, yeah, that's not a lot a lot of people. That's That's great because um later we will see some use cases that are not related to security at all where

we use Kyverno. Um and we I hope we can give you with those use cases a little bit more of an idea what other patterns exist and what you can do with it, especially as a platform engineer. So, shortly about myself, I work for Deutsche Telekom. I don't know if you know Deutsche Telekom. It al- always depends on the on the audience. Um you might also know

T-Mobile instead, which is part of And we are a very very big telecommunications provider, uh lifestyle company. We have fixed line, mobile line, um mobile tariffs. We have Magenta TV, for example, for streaming uh of of videos, movies, whatever, and the upcoming World Cup in Germany. Around the world, we are divided into 50 um or we provide services in 50 different countries, and we have 260 million

customers around the world, which is pretty pretty much. Yeah. Myself, I'm a platform engineering lead for the B2C Germany division. That means on our platform, we develop microservices, or we develop a platform for microservices that run the German B2C core. That means, I don't know who has a German contract. Maybe I think a lot of people from Germany are here. So, if you open the Deutsche Telekom

app, and you would check your mobile contracts or something, all your requests go through our platform. Or if you go to the online shop there, then our platform serves all the requests that you that you issue. We have a lot of services there, 500 microservices. We're doing a lot of retirement of old systems. And it's about 40 teams. So, what I want to say is there's a

lot of standardization standardization going on. There's a lot of need for very fast development of services. We need to be very reliable on the same time, and we need to give people really as much self-service as possible on the on the platform to make sure that we can keep up with the speed of digitalization. The first So, let's jump into the first use case. The first use

case I want to um show to you is when a when a service wants to develop a feature, or a team wants to develop a new feature, they usually spin up something that is called an ephemeral environment on our cluster. We have a lot of shared infrastructure because it's just more efficient. Yeah, you can, of course, like I mean, a cluster is the ultimate isolation itself, but

it's not feasible to um to provide a cluster for every service or for every team. Yeah? So, you need to isolate it in some way using our back. But, the big question is, how do you do that if a namespace must be on demand? Because um you need to create a role, a role binding, and everything in this namespace. Yeah, you can do it via Helm or

something, but it's not not a very good approach because it's not very dynamic, and you need to control it in some way. the solution here is to use Kyverno, actually. We label our namespaces. I will show it in a second how we do it. Um so, that the users can decide or the teams can decide on their own based on the labels in the namespaces who gets

access to this namespace. Like, they can decide, for example, I don't know, team B, team XYC, user XYC, whatever, will also get access to this namespace because they need to test or they work together closely, they develop the same services, I don't know, something like that. Um everything is possible. That's something we learned. Like, that's the first learning I can tell you if you build a platform,

Our users will do everything. So, uh let's have a look at this diagram. I know it's pretty intimidating, but bear with me. I'll try to I try to explain in detail. on the left-hand side, on the very left-hand side, you can see that the user is integrated with some kind of role into our authorization concept. For us, it's um AWS IAM. Could be anything. Uh if you're

a if you're a um user for a long time of Kubernetes, it could be Pinniped or any other option you have that basically puts a user into an into an RBAC group as a subject in his ID token. But, what do you do then? A namespace is created, and how the namespace is created doesn't matter. We use Argo CD, but it doesn't matter. And the teams can

set specific labels on the name space. In this case, for example, this name space is labeled with team alpha EU, team.de. Dot is our project name, so don't worry. And EU is the region. We divide every role by region. But the important part is the team alpha over here. And the second important part is the developer over here. So with this label, the team effectively decides that

team alpha will get the developer role in the newly created name With the second label they decide that team alpha in region non-EU will also get the same role. Or in this case, the QA role. With the third label, they decide that person Bob will get the CPU role in the same name Kyverno will check these labels on creation and update of the name space, and then

create the according role binding in the name space that matches the mentioned role with a cluster role. And if you match a cluster role with a regular role binding, a name spaced one, the actions and the permissions from the cluster role become name spaced. So what you the the big advantage you have here is that you give access to the teams about what roles they want to

assign to automatically by managing the name space, and with GitOps, they can implement any four eyes principle or whatever they want on these labels. Because that's just Git in the end, yeah? And on the other hand, as an administrator, you have the big advantage of controlling what cluster roles exist in the first place, and what they can do. So you can effectively avoid, I don't know, people

having destructive operation permissions, for example. So this This pretty straightforward, and the of course the nice thing that Kyverno does here is that it's completely Kubernetes native. Everything happens in Kubernetes with Kubernetes and with Kyverno creating the necessary resources or updating the necessary A second use case we have a lot. I will show two different examples about is that we have infrastructure as code providers that don't

100% fit our needs. This is a specific example of Crossplane. We're using Crossplane to do different things in our platform including one specific use case where we have an ALB that exposes a specific service to the internet. This service needs a certificate and the certificate is issued by an internal certificate authorization of Deutsche Telekom which is Telesec. It's very public and it's very well-known trusted CA. And

we do this with cert-manager. So, cert-manager gets a certificate from from Telesec in the cluster on demand. Everything is automated. This certificate needs to go into this AWS certificate manager outside of the How do you do that now? With Crossplane you can automate it but there is a catch and this little catch keeps you from doing it 100% automated. And that is you can in Crossplane in

a certificate resource which you can use to use to create a certificate in ACM you can put a secret reference to read the private key from a secret that is created by cert-manager as I said but you cannot put a secret reference to read the whole certificate chain from the secret because that's a string. So, here you're blocked. You need to manually update it in some way.

Or Kyverno to the rescue because there is a very nice um little part of Kyverno that can read from the Kubernetes API and Kyverno will then read the certificate from cert-manager for the and update the string in the crossplane in the crossplane resource. You have effectively avoided implementing any custom script, I don't know, forking the crossplane provider in any way. You have avoided writing your own provider.

You have avoided effectively everything that will give you a headache later on because this is one Kyverno um policy one Kyverno mutating policy that you can add to your infrastructure as code repository at the same place where this ALB certificate manager certificate everything is is happening. Just throw in this additional policy and your case is closed hopefully forever. I mean, I can one word of warning it

will be so easily solved that you will forget about checking if the certificate will support your use case in the future I honestly don't know, maybe they do I haven't checked. Just works. And um a very similar use case we have in the same in the same fashion that is Pulumi operator. Is anyone using Pulumi? Anyone familiar with Pulumi? Okay, so I will explain. Pulumi is a

the second popular alternative to Terraform. Uh it's an infrastructure as code tool. It works pretty much the same but you can use any or a different choice of programming languages and um it has some other features in comparison. So if you don't know it, check it out. It's very it serves us very well. And uh Pulumi has an operator and you can imagine it very similar to

cross to Terraform as I said. You basically define a so-called stack. A stack is a Kubernetes resource and the points to a Git resource where your infrastructure's code definition is. And then the operator runs something like Terraform apply, but in Pulumi it's called Pulumi up, and it's exactly the same thing, basically. So, it starts going through your code and then creating a plan and then updating the

infrastructure resources. Pulumi's When we started using Pulumi, it was only supporting TypeScript. Now, it's supporting a lot of other languages as well, Go language And we started in TypeScript and yeah, that's where we are. if you're familiar with TypeScript, you know that TypeScript uses node volumes. That means every time you create or install a TypeScript project, you will have to download all these dependencies into a folder

called node node volumes, and this is a pretty long process. And the Pulumi operator doesn't support to store these volumes um for longer. Like, it spins up a operate a a pod to run your to run your stack, and then it will dismiss the pod and everything is gone. So, what we do here is if you create a Pulumi will intercept this creation, and it will inject

it will first of all, it will create a volume, and then it will inject this volume definition into the stack definition, so that Pulumi will in future use this volume as a caching mechanism. So, it will in the first run, it will build the dependencies, and then the next time you run the same thing, the same um project, basically, you will have all these dependencies already cached

in this volume, and NPM will take care of recreating the them if if something is wrong or if they are outdated. But again, here, we are basically using a mutating policy that edits the stack on the fly, on demand, if necessary. Injects a volume and makes just speeds up your process tremendously. You don't have to build this thing anymore. You don't have to wait for all the

dependencies to download. how is this possible? Because it's all Kubernetes objects. It's all well-known objects. You know exactly where a volume goes in a in a deployment. The stack it's a deployment in the end. It's a that runs and you can inject a volume into it just as into anything else. flexible enough to do that with a couple of policies. I will not go into detail about

the policies because we only have 30 minutes and if you have questions, you can come to me or ask me or you can ask AI because it can do it as well. It's not difficult. It's just not intuitive. That's the main thing, I think. Um so, this is a this is a second use case and it's not just about TypeScript and caches. It's also about common configuration.

For example, you can inject configuration to scale down um your stack operator pods if it they are not needed. You can inject labels and everything. So, it's just about mutating Kubernetes native resources in the end. this mental model, I think, is very important that you that you don't need to edit something manually or don't need to configure something or write a script or something. Kyverno can do

basically anything as long as it's Kubernetes based in this case. So, last but not least, we have guardrails and um reporting or compliance reporting. So, if you have a couple of services, I mean, maybe if you have 10 teams, it's still fine. You know what every team is doing. If you have 15 teams, I think it will be difficult and if you have 40, 50, hundreds of

teams, then no one knows what the heck is going on, I can tell you. And um um thanks to Kyverno in the first place and policy reporter in the second place, uh we have a pretty good option to use Kyverno's abilities to also create very, very extensive reporting, cloud-native That means um Kyverno creates uh reports for every policy you have. And um as Franco already said, you

can run policies in the background on the one hand, and you also can run policies in a non-enforcing mode, in an audit mode. So, Kyverno will not take any actions and just assess the state, and then report them. And with policy reporter, it's a it's a small deployment in in your cluster. expose these metrics to your observability stack. every policy you write will create metrics in your

observability stack. Be careful with your cardinalities, obviously, yeah. Um just have a look at it, but usually it's or for us it's not a problem. And with all the with all the flexibility you have with um the expression languages and with the Kyverno implementation, create a tier of different basically compliance reporting policies. So, you can create policies of different categories that people can opt in, opt out,

depending on the use case. There are very practical use cases, for example, for us, um how many teams have implemented Argo rollouts for uh gradual rollout for canary deployment, but some services don't need it. Some services, I don't know, are loaders, jobs, something like that, that don't need this, and they need to be able to opt out of this KPI in some way. And they can easily

do it with some labels, so they can mark the service as not relevant for this tier of reporting, for example, or for the specific compliance report. um with with us being able to use Kyverno policies as regular Helm um deployment, for example, or as regular Kubernetes uh resources, we are able to very easily distribute them and distribute them based on specific configuration parameters in different clusters, for

example. For us, we run roll them out via Helm chart in GitOps, but you can do whatever way you want. Yeah, it doesn't matter too much. And um yeah, the outcome is that you have a set of pretty nice predefined dashboards already that you can give to your business and they and you can tell them, "Please stop chasing me if that and that service is reliable or

not." Because if business asks you again, "Oh, does this service have proper probes implemented? If this Is this service already doing canary release?" then you can say, "Open Grafana, check yourself." Um you can see it there. And you can um basically filter Grafana for different categories there for your policies um and you can easily export it wherever you want. I think there is also a dashboard included,

right? In policy reporter that you can use out of the box if you don't want to use Grafana or But for us, obviously, it's it's Grafana. So, last but not least from my end, what are the lessons learned for us? I think it's super important to see Kyverno not as a security tool only. Let's see, yeah. Today in the morning, I was talking to someone and they

were saying, "Yeah, we use Kyverno, but it's in the security department." Doesn't make any sense. You saw some use cases where you can use Kyverno as a holistic infrastructure automation tool. It does everything you need it needs to do based on what happens in your cluster already. You can avoid things like custom implementations, custom controllers. I have seen people doing wild things. Yeah? And it could have

been a Kyverno in the end. And it's well supported, it's patched, it's it's awesome. The second thing is enforcement is good, but policies and reporting help you as well. Like it's just as important. Yeah? Enforcing things is one thing, but telling people what they should be doing and assessing the risk of different and different applications, how compliant they are, is giving you a lot long-term in terms

of reliability. If you would I don't know who's running a lot of services, but what would you give if you would know today if all of your service have proper health health checks implemented? I don't I don't know. With Kyverno, you can come closer to the answer of this question. Yeah, sorry. And last but not least, we prefer a little bit of duct tape. We sometimes refer

to Kyverno as the ultimate duct tape, yeah? Uh we prefer a little bit of duct tape over custom implementations because if I have a custom controller, if I have custom scripts, if I have I don't know, something like an admission controller in Kubernetes, it it can be horrible. Like if I write my own admission controller in Kubernetes, probably the cluster will be down 24/7. So, yeah. These

were some insights into how we use it, and I think um now let's check the roadmap as well, what is coming next. Exciting things coming next with Kyverno. then afterwards we have might have some time for a few questions. Maybe. Yeah, thanks, Johannes. Um yeah, let's see what is on our roadmap. So, the next release, 1.80 uh 18 is uh yeah, comes soon. Um we Yeah, um

working on improving our CLI a lot. So, there are currently use cases which you simulate on a CLI level when you want to do it on a CICD pipeline. We are actively working on making this possible. We have a dedicated LFX mentorship program from the CNCF focusing on this one. We also want to bring these features in our playground. When you don't know it yet, it's a

really good way I hope you can read it a bit to check out Kyverno without having a cluster, without installing anything everywhere. You can just write your policy, you can add some test resources and start playing around with it and writing your own cell expressions. It's available publicly under playground.kyverno.io. We're supporting different versions. If you have an bit older one, you can also try this one out.

Yeah, other than that, I'm working on the CLI. We are shipping our reporting system, our standards to a dedicated organization which is called openreports.io. So, tools like mentioned policy reporter as well as the open reports CRD we are using in Kyverno will be in a dedicated organization soon and will be managed independently. Um this makes it It's already possible, but it makes yeah, more general use case

possible. Other tools like Kubewarden or Falco Sidekick already already working with the CRD and we are hoping that it will be a yeah, general standard within the Kubernetes ecosystem for even more tools. Um yeah. That's for the roadmap. Yeah, one important thing maybe in 1.20 it's planned currently planned to remove the cluster policy CID which is marked in the last release as deprecated. So, if you're still

using it, we would recommend to yeah, start slowly to rewrite them in our new cell-based um policy CRDs. If you write new policies, you should already use this new CID. And yeah, you still have a bit time, but you should slowly start. Um yeah, other than that, maybe real quick, this is the mentioned dashboard from policy reporter which is a yeah, alternative to a really large monitoring

stack. So, if you want to try the project out and want to spin up a a dashboard, um this would be the easier alternative instead of installing a Prometheus, Grafana, and so on. that's it so far. If you have questions later on, we still have our booth today. You can reach out to us. You can get a demo if you're really new to the project. You can

ask questions. Otherwise, we are really active in the Kubernetes and as well as in the C NCF Slack workspace. So, you can already contact us contact us um if you want anything from us. Otherwise, thanks a lot, and we now have not really time for Q&A. Thank you so much for coming.