Cert-manager – Project Update: Beyond 2026 - Tim Ramlot & Maël Valais, Palo Alto Networks

This talk focuses on cert-manager, a widely used Kubernetes controller for managing X.509 certificates. The speakers discuss the growth of the cert-manager community, highlighted by its graduate status within the Cloud Native Computing Foundation. They explain the functionalities of cert-manager, such as issuing certificates in Kubernetes clusters, alongside introducing trust manager, a controller for distributing trust across workloads. Recent updates include enhanced Helm chart support, security improvements against supply chain attacks, and optimizations for the Gateway API. Future initiatives aim to migrate certain issuers, optimize the ACME issuer, and introduce new features to improve certificate renewal processes.

Okay, welcome everyone. Welcome to our presentation. So, Milan and I, we will be talking today about cert-manager, about what we have done in the last year, and what our vision is for the future. You can uh download the presentation also using the QR codes. There are a few links in the presentation that are useful, and through that uh link through the PDF, you can actually also get

access to these links. All right, let's maybe start with a little show of hands. Who knows what cert-manager is? Okay, great. The opposite. Who is uh Who is using cert-manager in production? Okay, almost everyone. Great. So, basically cert-manager um for those who maybe not know it, it's a group of people. It's a community. It's lots of people that are very interested in X.509 certificates in the Kubernetes

world. And what we do is we we try to create controllers, all kinds of Kubernetes uh things that are very useful for our users. And we publish them on GitHub, so that they can be used by almost a million people per day. And this group of people that are very interested in this, and they're that are writing codes, we're with almost 500 Um and of course, we

also have our own structure. We have like maintainers, contributors, and so on. Um and all of this has been recognized by the CNCF. And so, we are a CNCF graduated project. the most um popular controller that we make is also called cert-manager. And this controller basically issues certificates for you. So, in your Kubernetes cluster, you can ask cert-manager to issue a certificate for you so that in

your workloads you can actually use that certificate. So your workload can be a bots, an ingress, a gateway, a service mesh. Cert manager when it sees that you requested that certificate it will give you a certificate by asking a CA to um issue a certificate for you. So I think a lot of you will uh know Let's Encrypt. That's one of the CAs, one of the most

popular ones. But you can also um issue private CAs using Vault or uh CyberArk. So that's our most popular controller. But we also have some other projects that we're working on. One of them is trust manager. And trust manager is this other controller which is responsible for distributing trust across all of your workloads in a Kubernetes cluster. You can define what uh certificates to trust by using

a set of secrets, config maps You can also tell it to trust the um Mozilla public CAs. And then trust manager will kind of combine all those certificates into one trust bundle and distribute this bundle across all your namespaces in config maps or secrets. Um and then they can be mounted in your workloads in a bots, for example. Okay, so Maël will now continue with the um

updates for this year. Yeah. Thank you. Where is my microphone? Thank you. So one big milestone that we had and Tim already talked about is the graduation of the project. Graduation means structuring the project. And one of the steps is to get a proper governance in place. Uh which is represented by a file called governance.md. What I I am super proud of announcing today is not new.

It has been already 2 weeks since then, but we now have a new maintainer, um Hammant Joshi, uh coming from the US, uh working at Walmart uh over there. And it is so it is our 10th maintainer. It is awesome uh because one of the biggest concerns that we have seen uh coming from the TOC when we were uh discussing graduation was that the lack of diversity

of people. Most of the people there on this list uh actually join from or come from the same company. Although it says Apple, Diagrid, Weave Research, all of these people actually come from Jetstack. Uh So, but but the Eric and Hammant do not come from this uh original company. This is a show of diversity. It shows the the um long-term viab- viability of And I all that's

uh we're all already 5 minutes. I'll add something really quick. Something I heard I hear all the time is why isn't cert-manager built into Kubernetes? Why do I have to install something else? I still don't have an answer to that, and we haven't even started talking about like like to the Kubernetes team. It it is not something we've doing. But the fact that it is a graduated

project and that we have a healthy number of maintainers that review PRs is a show that you can trust and you can rely on this project. Where's uh I'll use that. Okay, I'll talk a bit of about the two releases last releases releases we did. Cert-manager we have now support for listener sets. I'll talk about that just after it's a very important thing. We have a very

great improvement for helm charts uh for people using custom registries. Other private zones is a new feature for genasol one. And we also fixed a few a few things like uh big certificates uh that you can now use. And uh Tim will talk about a few things that are happening very important things that are happening in the background like service side apply. And for trust managers, we

also improved the helm chart user experience and fixed a few things like being able to skip non-CA certificates whenever you're dumping a big list of certificates but has some leaf certificates in them. And some things happening uh with the CRD itself. Okay, for one thing I'm super excited about is this I know it's very small but still you can now give a single flag to helm when

you install cert-manager to replace the key.io registry. If you are using artifactory or you're playing with cert-manager with custom images it has been in the past super painful. You had to replace all of these values everywhere. Not fun and this is a partly it is entirely our our fault uh very early on we chose to use the repository field in a weird way, but this is fixed

and it is fixed not only in cert-manager. So, the new field is called called image Please use it. It's great in 1.20. And it works for all the other projects. You already know this kind of field in all the other projects I know. but this this is finally coming here and thank you thank you Tim and Felix, a colleague of Tim of us, uh for working really

hard on this. Now, the second thing I'm I'm mostly excited about, I've been working on this for 2 years, uh the Ingress Nginx deprecation. Uh you may have heard on Tuesday, if you went to the Gateway API presentation, that uh I think on stage they clicked the archive button. I wasn't there, but I've I've seen a recording. it means something important. It means that many people are

now looking at the Kubernetes blog post, "Oh, they're telling me Gateway API is might be a good way forward while I'm migrating away from Ingress Nginx." And Cert-Manager has a role in this because obviously, or not obviously, many people are using the Cert-Manager annotation on the Ingress resource. The pro The problem we found the pro the major problem we had with Cert-Manager that this Gateway API integration

dates back to 2021. That's So, we didn't know anything back then. And we didn't know specifically that when someone who is using the Ingress resource wants to migrate the entire cluster to Gateway API, they'll have a problem. And this is the that that represents this. And I'll I'll I'll explain really really really quick because I think so many people are affected by this. Uh you you might

decide plea If you feel free to stay on the using the Ingress resource if you feel like it. It's going to be supported forever. Do not worry. Or you can migrate to traffic or engine x ingress from F5. But don't feel free to to be up don't don't feel obligated. Gateway API for context is very good for platform teams. It's mainly an improvement for them for for

for you. For for you, sorry. You are you are the audience. for context, if you were trying to migrate from ingress to Gateway API today, and your controller your the your controller is controlling is is uh the represents an IP address, a load balancer, an appliance, something expensive, your developer who had the ability to set the set my orientation, the TLS host the TLS yeah, TLS host.

So, they would they they are able to configure their own certificate and the HTTP routes. It was awesome. Self-service. Very nice. But moving on to Gateway API, it's Oh, sorry. Uh it's problematic. It They have to talk to cluster operators every time they want to do the same thing as what they did. They the TLS block is now on the Gateway object and the annotation is also

on the Gateway object. It's not good. And we realized that way too late. But fortunately, uh the Gateway API maintainers proposed I don't know who did it. Uh someone proposed this new Oh, sorry. This new No. This new resource listener set, and we support this resource. This resource is nice because you can let developers create it uh update it instead of editing the Gateway. The Gateway is

sensitive. Let's not have the bots touch them touch it. But the license that can and they developers are back at being able to self create self-service self-deploy everything. There is some some quirks like the fact that they can also choose the port that isn't great. And there are other shortcomings, but we're still in the working with the gateway API team to iron to find better ways of

doing this. I'll move on from this because otherwise Tim will kill me. And moving on to Tim's uh Yeah, thank you. security. Um so cert-manager is all about X.509 certificates and certificates um It's not just certificates, but also secrets, the private keys linked to the certificates. And that means that it needs to be secure the solution. So, not only at runtime, but also our environments, our systems

have to be secure cuz if we end up releasing some malicious version of cert-manager and you install it, that's a big problem. So, we have quite a few measures in place to try to kind of automate some of these things, try to put in some uh checks to make sure that our project stays secure. Um so, there are two two main groups of attacks that we um

defend against. First of all, we try to defend against supply chain attacks. And a few measures that we have in place are trust on first use. Um we also have very fast security updates. So, when a dependency has a new release with a security bug, we try to upgrade as quick as possible. And then also in general, we just try to avoid um to have too many

dependencies. We try to reduce the amount of dependencies so that we're less sus- susceptible to all kinds of issues um downstream. Or upstream. then we also try to detect um defend against direct attacks. code scanning, linting. Uh we also all our releases are fully automatic. Uh they're done in pipelines, so it's not like some developers do doing the release from uh laptop and they can modify the

binary creating the release. Uh we also have very regular security audits, and we have a good governance structure in place in case there's some contributor or maintainer who um turns out to be malicious. One of the big challenges we came across when trying to um apply these security measures is that we have many different repositories in um we talked already about cert-manager and trust-manager, but we have

other um controllers. and these controllers, they all needs to adhere to this best practice security uh stuff that I just explained. So, the way that we do this is we centralize a bunch of different automations and scripts and so on into this makefile-modules repository. Uh so, we use makefiles generally for kind of all the the scripts that we have. And these makefile-modules, they are uh reusable across

these projects. So, here we define our best practices, we define our tools, our linters, and that kind of thing. And then we have this um GitHub action to synchronize these modules into these um different repositories that we have so that they all have the same logic. Additionally, we also use Renovate to update our uh dependencies. in more detail now, the supply chain attack, uh the way that

we protect against them is by protecting our um binaries that we download or even the source code that we download. We try to trust on first use. So, that means that whenever we update a dependency, figure out what the digest is of that dependency. That fully kind of locks down what the the downloaded what the downloaded binary or the downloaded source code is. And then we store

that somewhere in our Git repository such that later when a developer want to um release pipeline downloads it, we can verify it hasn't changed. We do that uh for all the tooling, all the binaries that we download. We do it in Go, which has it kind of built-in. We also do it for our GitHub Actions and our OCI images. We pull them uh using their digest. So,

then Renovate we use for uh security updates. Um in the centralized repository, we manage a centralized version for Go, for our images, for our tools. And then in all of the individual repositories, we update our Go um also using Renovate. And sometimes we're even able to upgrade a dependency before some of the vendors have picked up um on a CVE, for So, to reduce the amount of

um unmaintained and unnecessary dependencies, we use very minimal base images. So, we use Apko to build a minimal base image and we kind of build um our controller images on top of that using Ko. Also, in Renovate we have this this check to make sure that none of our dependencies are abandoned so that they didn't have any updates in a long time. Then for direct attacks, we

have code scanning and linting in place. So we use Go lang CI. Go lang CI lint which is a linter that kind of checks all kind of um things across the code base. It's really cool tool. If you're not using it, you should definitely start to do so. We have have our configuration for Go lang CI lint also centralized. Um we also use Go vuln check which

is a vulnerability scanner that makes that scans our codes. Um and it kind of only um warns us about CVEs that are in code bots that we use. We also have this Helm tool linter which is a linter that we built ourselves to um make sure that our Helm files are well uh created. So we find out if there are any references to uh values that we

or the other way around. We have boiler suite which is a tool we also did created to verify that the license headers are in place in the um Go files and other files that we use. Um and then Eric, one of the maintainers, is also working now on introducing Cube API linter which is a linter to enforce some best practices on our CRDs uh for all the

different projects. Like I said before, we have fully automatic release pipelines and we sign our releases using the token of the pipeline so it's impossible uh for someone to kind of modify these releases or do them locally. Uh then the signature won't match. you can check out our governance structure using the link in the slides. Uh we have the set of maintainers that I'm already talked about.

And then we also have very frequent audits. So, in 20 23 we did an audit, in 25 we did an audit. And I can probably announce now that we have another audits or a security gardening guide that we published um together with control plane. And you can check it out on the link below. Um this is very new, so it's definitely very interesting to look at this.

a um security audit, it's also looking at kind of how to harden your uh cert-manager setup uh together with like a chat model, for All right. Then, for the future, what we think um are some cool things that we want to still uh do for cert-manager and things that are currently ongoing. I created a short list here. Um first of all, in cert-manager we have some entry

issuers, so we have external issuers and entry issuers. And generally, I think the consensus has been to somewhere in the future move all these entry issuers out of the cert-manager project. We created this issue library and that you can use to create an issuer, and we'd like to kind of use that to uh move these issuers out. We're also working on this webhook cert-lib, which super experimental

still, to um make it easier for our webhooks to obtain certificates, and that would also be reusable for other controllers. Um I would and that would not have the dependency on cert-manager on the cert-manager cert-manager project per se. So, it would be uh perfect for uh dependencies and that Um we're also trying to improve the ACME issuer um performance-wise. I think there's a lot we can improve

still. Try to make it async. Also, last item there is to support the Acme RE specific or the extension, which basically is an extension that tells you when the certificate should be renewed, ideally. Um other things we think are quite cool are adding like image digest to the helm charts, migrating cert-manager fully to server-side apply. And something that has been already merged and will probably be part

of the next cert-manager release is the renewal window supports. Um this is a new feature that lets you decide in what windows during the day you want um allow renewal. So, that way, for example, you can say, "I only want the certificates to be renewed um during office hours, for example, so that someone is there when things go wrong." All right. Thank you for listening. Um you

can find the cert-manager projects on GitHub. We are here at KubeCon or we were here at KubeCon. If you have any questions, you can also uh join one of our meetings or ask questions on Slack. The links are in slides. Thank you.