Evolving Policy Management with Agentic AI: Kyverno MCP and Kagent for... Shuting Zhao & Dahu Kuang

This talk focuses on managing Kubernetes clusters with an emphasis on using AI-driven automation to streamline policy management. The speakers introduce Open Cloud and Kyverno, highlighting how these tools can address challenges faced by platform engineers, such as the verification gap and security compliance across multiple clusters. They discuss the evolution of a solution that facilitates intelligent orchestration and simplifies querying cluster statuses and policy evaluations. The talk includes a live demo where the speakers showcase the installation of Kyverno in different clusters using Open Cloud as an AI assistant to manage policies efficiently. The session concludes with a discussion on security considerations and the roadmap for future enhancements to Kyverno.

Uh, welcome everyone. Welcome to the session. I didn't expect this many people to attend in the last day all afternoon of the conference, but welcome. Um, you might have seen that our session originally titled this Cabana MP MCP server and K agent, but as we're you know, build out the solution and the demo for this stage, the project evolved into something much more fun. And uh, we've

moved beyond just connecting clusters to actually orchestrating with them in with intelligence and that evolution is open cloud. And if you came here to learn how to manage policies across hundreds of clusters without losing your mind you're in exactly the right place. All right. So, before we dive into the details for today, uh, quick introduction of our speakers. I'm Shuting Zhao, a Cabana co-author and maintainer. Uh,

working at Namada, which is the company behind Cabana as a staff engineer. Hi everyone. I'm Daohu Wang from Alibaba Cloud. And I'm working on the continuous security and we are also the friends and the users of Cabana community and then we have also met a lot of requirements from our user. Thank you. Um, all right. Uh, before we dive into the solution, dive into the how, I

wanted to spend a few minutes to talk about the challenges uh, that we as platform engineer facing everyday, right? So, imagine like in your daily work as an we no longer deal with the just one cluster, right? We're dealing with many clusters with different versions, different across different zones, with different rules, but our two makes this difficult. And right now if I ask you, you know, are

all my production cluster ready? You have to jump between different screens and type a lot of commands uh including Coop Cuddle uh just to figure out, you know, to access the logs, the events, and all the uh reporting system to see hey whether uh your cluster is, you know, security compliance or not. And this is a lot boring, right? And a lot of manual work, of course.

We we want to be more efficient, automated than that. And what if you have a like assistant uh while you're holding your morning coffee and ask a simple question like you know, uh just give me like the production review on my clusters. And you can then have a report returned it back to you and then you can focusing on those clusters that need some uh to be

taken care of. And the next challenge uh is what I called verification gap. This is the silent killer of security, right? Um I remember there was the the other day at KubeCon on this Monday earlier a guy just showed up and ask, "Hey Shu Teng, what is the best way to test our policies actually working?" I mean it's simple, right? You can use CLI or the playground

if you don't have access to the cluster or if you have a local kind cluster, just try to break something and validate that validate that. But the guy said, "Hey, hey, this not my use case. I already crafted 20 policies uh across 100 clusters in my organization at what and I want I want to verify that daily if your policy engine is actually functioning as expected." what

if um your GitOps pipeline shows green? Does that really mean that you have a truly safe cluster? Or uh imagine, you know, do you need to manually deploy a bad resource to break the violation to break the policy and see the violations uh to manually verify something is working as expected. Again, imagine if we can just ask a simple question like show me the last five blocked

requests in my production cluster or the last five violations generated from my policy report, you'll know that for sure in the last one of few 24 hours there is something bad happened to your cluster and the policy kicks in and blocks that. And finally, this is not a daily challenge, but uh we as a maintainer are facing it every day. Um so, you know, in fact, uh

there was a guy asking a question stopping at the booth and asking, um I have Argo CD plugged into my CI/CD pipeline and it tries to schedule something to the cluster, but well, Kyverno is trying to mutate something on the fly and these two applications fight against each other and they, you know, they bring the nightmare to the cluster. And he he came to us and asked,

what is the you know, uh solution to this? And we have this well documented uh on our website, but obviously, this is a skill set that you have to know. Otherwise, you need to like jump into to 10 different browsers and looking for the solutions, but what what if those policy skills or the or your skill set is shared in a common brain, right? You it's like

acts like a pool. Whenever you have a question, your assistant come to the pool and fetch for the corresponding questions or the scenarios and then it just figure it out, right? You don't um what is happening behind the thing. You just care that, okay, your cluster is functioning, everything is working. And then imagine something like just installed the pod security standard policy to my cluster and give

me the report without doing the you know hand deploy of the Kyverno policies chart without doing the coup cattle get all the policy report and see the violations etc. You just type that question plug that sorry feed to that your prompt message and then bump get the result back. Right? So we've spent some time looking at the overhead the verification gaps and the knowledge silos. And now

this is the most interesting part a let's talk about the solution. You know, I want to quickly walk through the architecture of today's setup to beat this a gen tech system. We use this three core components and Kyverno come as the your fundamental tool set which you know is CNCF standard we all know and trust. I'm I assume you already know something about policy management something about

Kyverno which just got graduated so some sign shows that we're we can be trust and you don't want your AI assistant just randomly mess up your cluster right? So there comes the open cloud right? You needs the brand to instruct your underlying to say hey do this for me at what time and how to do that and just give me the desired result. And finally we have

this shared brain I call that app store it's like whenever you're looking for something imagine you're working out and every day and you want to record some data of your heart rate of your running distance you just look up browse the app store and find the best app suit for you right? So the policy scale or the scale set just work like that. It knows how to

install Kyverno, knows how to troubleshoot Kyverno, how to show the violations and making sure everything's working as Okay. And when you put them together, you get a system that is as safe as the traditional policy engine, but also as smart and fast as an AI agent. A little deep dive into what is Kyverno. We have been working on, you know, splitting the typical cluster policy into this

new cell-based policy types, which is common expression language. It's also supported um by Kubernetes validating admission policy as well as mutating policy to be able to manage your full life cycle of your resources. What that meant is, um you know, Kyverno runs in the cluster as an admission controller. It does all the validation check for your incoming resource to check your security compliance. It does the image

verification stuff to make sure you your supply chain is is secure. Um to the automation of, you know, auto injecting the self sidecar or mutating the labels, auto provision, the resource generation. The typical use case of this is the when you want to create a namespace, it auto creates the RBA resources, default network policies, etc. to your newly created namespace to the final deletion of your dangling

and stale resources. And we also uh come with the native support for reporting, as I mentioned. If you want to check the policy evaluation result, the report is your best option to check those historical data. It's just It comes with another CRD called uh backed by the open reports uh project and we also have this opt-in exception if you want to exempt a single user or a

single workflow to bypass the policy. And in the end, I want to highlight that Kyverno was built for Kubernetes as a native Kubernetes native policy engine, but now it runs everywhere. Yep, here is the like overview of the five new type of policies based on common the highlight here is that with the same declarative pattern of using a Kubernetes CRD, like here I'm showing the validating policy,

you are able to apply Kyverno anywhere including your Kubernetes clusters to any JSON payload or we have the Envoy plugin. You are able to use it to verify authorization flow and and apply Kyverno policies to the HTTP request. And now, you know, just a brief overview of Open Cloud. In our setup, Kyverno acts as our shield and Open Cloud acts as the brain to instruct how how

Kyverno works, right? It's an AI native orchestration layer that understands the intent behind your words. So, it's actually translating your natural language into the Kubernetes action. It knows how how to manage or switch between your remote cluster, your local cluster. It knows how to operate and orchestrate your underlying tool including Kyverno, Helm, and existing CLIs. And in the end, all you have to do is plug that

into your Slack and start prompting message to it. And finally, uh final piece of this puzzle is Klaw Hub. It's somewhere we want to store all the skills that shares, you know, this policy governance flow. Um instead of having, you know, that knowledge trapped in a expert expert, sorry, expertise of your uh senior engineer, we package that into a um pool that can be shared um across

different users and uh use cases. And here is a quick uh overview of the Sorry, example of the skill that we are showing today, which is about showing me all the violations that are created uh in my cluster. Um as you can see, it's just describe uh like the use case when you want to use the skill using the natural language. It defines some preconditions of when

to run that and how to run that. And everything is just simple as that. With that, you don't need to, you know, type in those Kubecost command anymore, uh switch between clusters, and troubleshoot, or, you know, it just return all the results as violations. And there's this nice little comparison table. Um you know, it's the comparison between the traditional manual way of verifying stuff to the automated

flow using agent take workflow. Everything comes to the to the end as a simple question. Check my cluster, check my policy engine, give me the result, etc. And with that, let me hand over to Dahu. He's going to dive into the demo setup and give you the demo session. Thank you, Ashutosh. Firstly, I I want I want to walk you through how we actually use Kyverno in

the real world. And if you're trying to manage kimono policies across multiple clusters, you really have three choices. Uh the first one is doing the manually. Uh you know, just logging into the cluster and use a kubectl. And it's fine if you have only two or three clusters. Uh but honestly, it's not scale. And one typo, maybe you're in the trouble. Uh the second one is use

Komodor. Uh this is a solid enterprise choice and it's great for stable production environment where you need this strict control. but uh in fact, it's not smart. the second the the last thing is use the AI agent such like the Open Cluster. And this is where the things get interesting. Uh we're talking about AI agent that can actually see what's happening and decide what to do and

uh fix things automatically. And then it's fully close the loop and it's fast and smart. But because it's so powerful, uh we need we we need it really careful about the security. Uh so, we need a rules. Uh for example, use the manual for testing and the Komodor for steady and the compliance. And uh Open Cluster when you need the speed and the intelligence to handle some

complex kimono operations work. So, before the demo, let's take a quicker look at the demo architecture first. Uh we use Open Cluster as an AI assistant to manage the kimono policy across multi-clusters. here's my setup. Uh I have got two uh separate VMs and each running a kind of Kubernetes cluster. And the one control plane and the two nodes. And uh on one node, I have deployed

the Open Cluster gateway. And uh one of the cool things about Git Cloud Open Cloud is it can integrate with any chat apps you have installed and you can send text anywhere. So, let's begin our demo. Okay, you know the Open Cloud is one of the hottest AI agent open source project now and it can integrate many chat apps and we choose Slack for our chat bot.

And here is my two clusters. Both of them one control plane and the two workers. And you can use the Open Cloud node round to join one cluster as an Open Cloud node to the Open Cloud gateway. And on the Open Cloud console, you can check the node status. You can see the can class two is joining as a node. Okay, you can also use the Open

Cloud node status to check the node is paired and connected. And then you can also directly talk to my chat bot and then let him check the status of the class one. Okay, it will return the status of my class one including the name space, the pods, and the summary of my class one. And then I will use the Open Cloud CLI to check the status of

my remote And here's the interesting part you need approve this command running on the remote side. And after I approve the it will return the uh, cluster two status and you can also uh, talk to the chatbot to let him return the remote cluster status. You You also approve this. And this is the status of my remote And now we will deploy the Kimono in both of

my cluster. Open cloud is beginning the work. A lot of tool is working. And the in first time it will use the helm chart. It will use the helm install to try to install Kimono on my remote cluster. And here I forgot I didn't install the helm CLI on my uh, remote cluster. Don't worry, the open cloud will auto fall back to use the kubectl apply to

continue the installation. Okay, let's check the status. Now the cluster one has installed Kimono successfully. And it is still installing Kimono on the And open cloud open cloud will continue waiting for the status and continue querying the status of the remote cluster. Okay, now you can see both of my cluster uh, Kimono. Let's check it. Okay, in my remote cluster all of the Kimono pods has started.

Okay, now I will try to install some uh, Kimono skills from the GitHub repo. Uh, during the installation it will firstly check the current skill status on my open cloud VM. And then try to clone the skills from the uh GitHub repository. And then verify the local skill And then it will do the same things on And this also need your approval. Uh we will install several

Euro Kivano skills such as install policy uh use the Kivano CLI to audit your clusters security status and then show the Okay, after some time we can see all of my cluster has installed the Kivano skills. And then the report will show the skill details. And then let's check on my VM. We go to the open claw workspace skills directory and then we can see there are

six Kivano skills I have just installed. And here is one of them's markdown. Okay, in the next demo we will firstly deploy some privileged testing pod on my local class. And then continue use the Kivano CLI uh to scan the uh my local cluster default namespace the security status. And it can check that any pod that does not comply with the pod security standards. You can see

the open Claw. It's begin the scanning. Perfect. It It create a comprehensive report. Including the violations by type, the pod right pod resource. And you can see the privilege configuration that does not not comply with your uh pod security standards. And also some recommends. And you can also uh get the report on the Open Claw console. And then I will try to deploy some RBA related policies

on my remote cluster. Here I don't need input the policy template because I have predefined it in in my skills. Open Claw will auto find the this policies side and try to deploy them on And now the four RBA related policies has deployed on my remote cluster. And it generates the report. Uh let's check it on my remote cluster. You can see the four Kimono validating policy

has generated. Okay, let's continue. We will deploy some uh Kimono CRD deletion protection uh validating policy on both of my And this time I input the template and the let's the open cloud try to have deploy the CRD and some testing CR instance. For the remote side, we need approve the command. And the open cloud began the working. Besides the policy, it will also creating the CRD

and some testing CR Okay, you can see it's work finished and generates a report. And then it will have us generate a testing card so we can easily test it uh by manual. Let's try to do it. You can see CRD is generated the database CRD just created. And you can see some CR instance that's the open cloud created. And the policy. And we try to delete

the critical DB instance and then try to find the kind and the Kyverno engine have to block the And last we will try to deploy Kyverno mutating policy and the uh try and uh tell the open cloud have deploy a testing pod and we'll find if the mutating policy work. Okay, the open cloud began the work. Firstly, it will create the mutating policy on both of my

cluster. And then it will have us to deploy in the testing pod. >> It will auto check if the pod is ready and it if you have a security context. Okay, so work is finished and it will Let's check it in my local cluster. Uh this is the testing pod I just created and let's see this template. Okay. You can see the run as non security context

is auto patched by And in my remote cluster, okay, the run as non is also patched. And this is a mutating policy the open car just created. Okay, that's my demo. Okay, we we have showed the open close power and the smart. Uh this time we need to have a serious talk. Please stay alert to the security risk. You know, bring a AI agent to your Kubernetes

cluster is just like give a robot the keys to your kingdom. If you don't care for it, it will go wrong fast. And recently reported, there's many security experts highlight some scary things such as the identity uh identity confusion, the supply chain attack, and the exposed pod, and the prompt prompt injection. So, we need a bottom line. Uh treat the AI agent as an untrusted code and

never just assume it's safe. Uh so, how we can sleep well at night if we're using the AI agent. And here's our top security best practice. For example, the isolation, the identity, lockdown your network, and run it in sandbox, and be careful your skills, and keep the human in the loop is very important. And audit everywhere and check the log. To summary, Kyverno give us the rules

and open cloud give us the brain to mind the damage skill. Uh but the power need a strong safety net. So that we can use the AI agent to build the future of the community governance. All right. Looking ahead, I want to talk about our Kyverno roadmap. Early this year, we launched Kyverno SDK, which you can access the core engine programmatically, and internally Kyverno all the Kyverno

components including the controllers, the CLI, and the playground are using the same SDK as we published. So it's guaranteed that you have the consistent and unified result while you're doing the policy evaluation. And currently we're working towards release 1.18. We want to provide this unified Kyverno experience across Kyverno CLI and playground. By that, what that means is that, you know, we have this this Kyverno policy runs

anywhere. We want this dedicated tool to be able to apply all types of Kyverno policies including the authentication authorization policies. So you're able to test that in a unified manner. And we're also migrating all the reporting component to you know, mentioned open reports project. And looking ahead, we're working continuously working on the advanced feature and enhancement of our features including CLI, the exceptions, and some of the

reporting mechanism. And something I wanted you to keep in mind is that we're, you know, since cluster policy is deprecated earlier this year, we're targeting to remove that by the end of towards this year, by the end of October before KubeCon North America, we'll remove the old cluster policy type. And if you haven't get started migrating a policy, I think you'll better consider that soon. Okay. So,

we're open source project, everything is about community. Welcome, you know, you're welcome to join to our community for any kinds of contributions. If you're not on the coding side, uh feel free to send policy samples, um you know, enhance our documentations, and most welcome uh contribute to your skill set of uh policy management. And we just got graduated, by the way. I hope you enjoyed the cupcake

uh earlier this week. All right. Uh I think we're right on time, but let's just quickly see if there's any questions, or else we can close the Thank you, everyone.