Exploring NRI for Automated CA Trust Injection - Tsuzuki Tsuchiya & Kento Kubo, LY Corporation

This talk discusses the NRI (Node Resource Interface) and its application in automating the trust injection of certificates within container environments. The speakers, engineers from LY Corporation in Japan, elaborate on the challenges faced in injecting private CA certificates into containers, especially when dealing with different Linux distributions and programming language requirements. They present their implementation of an NRI plugin that simplifies certificate management during the container startup process. By automatically detecting the operating system and runtime, the NRI plugin configures the necessary environment for applications such as Node.js and Python. The approach aims to reduce complexity by allowing users to enable CA injection with a single pod annotation, thereby enhancing the user experience for developers interacting with Kubernetes.

Thank you for joining us today. Uh I'm Tuzukit. Today we are going to talk about NRI N resource interface and how it can help us to automate an operational toy by showing a case that automates uh the SI trust injection. My name is Suzuit from Corporation in Japan. I'm I work on the development and operation of internal platforms. My name is Kent Kubo, a software engineer at

LY Corporation. I'm also a developer of internal command platform. Uh these are these are today's topics. First, let me explain our private crowd and private CA to share the context. Next, I describe the challenges to inject SH certificates into containers. I also describe uh existing approaches and their problems here. Then I introduce NRI node resource interface and our solution with NRI. Lastly I compare it with existing

ones. Uh first let me explain our private cloud. Our private cloud is running on multiple on premises data centers. It uses OpenStack for managing VMs and uses Ascend for IM identity access management. Ascend is a CNCF sandbox project for service to service authentication and authorization with ARB. And we are part of Kubernetes as a service team to provide a single tenant Kubernetes Kubernetes clusters to internal developers.

We operate more than 1,000 kubernetes clusters powered by cluster API and security is also important to also important to us. We have the internal PKI to manage the private CA for the flexible certificate management. PKI is one of the fundamental technologies for their trust architecture. With a private CA we can enforce security policies and enable mutual TRS with asends and issue server certificates for the internal services.

It means not exposed to the internet. So external users cannot trust the certificates signed by the private CA but internal users can internal users in my company can trust them because the private CA certificates are installed in our laptops and servers. But installing the private CS certificates everywhere isn't simple uh at the scale of our crowd and when especially using containers. I think everyone here has run

into this kind of error that the client cannot trust trust the server. This is caused because uh the CS certificates aren't installed in your images. So if you want to install the public CS certificates, you can install them from the public package manager like uh after get install CS certificates command. But if you want to install the private CS certificates, you need some tricks. Let me let

me explain them in more detail. The first approach is writing Docker files. With the multi-stage build, you can create the the bundle of CS certificates by joining public ones and private ones. And with this approach, uh users need to maintain image repository, CI pipelines and Docker files even even if you want to just use public images from Docker Hub or GHCR. Another approach is using in containers.

Uh in in containers, you can create uh the share bundle like writing docker files and share it to the main containers using emptyd volumes. But this approach requires users to maintain a bit complex pot templates or uh mutating web hook to inject in and in containers can increase time taken for starting containers slightly and rust approach is mounting static files. In this approach, uh users need to

prepare the C b Certificate bundle in advance somewhere. Uh we usually store the certificates in config maps uh config maps secrets host passath or cluster trust bundle. Cluster trust bundle is a kubernetes standard uh cluster scoped object to store and distribute CS certificates in the cluster. Uh it's still disabled by default yet. So you need to enable it with the feature gate. Yeah. But this approach users

can simply uh mount mount CS certificates to ports. Yeah. So this looks very good. But we still have some issues. The issue is that the location trust store exists differ across Linux distributions. on Debian or Ubuntu based distributions uh it's stored in ATC SSL s certificates GRT file but on red hot enterprise nex based distributions it's stored in ATC pkits sets folder and the CA boundary is

created from multiple CA certificates uh they are saved in some directories the command to update the CA bundle from them also defy across distributions and in addition uh the some libraries and ranges don't rely on the OS trust store for examp uh go uses just OS trust store so you don't need any additional steps but Java has its own trust store so users need to import the

private C certificates into its own trust store by executing this command and some libraries for Py Python or NodeJS requires an environment variable to specify the additional share So to sum up the discussion so far, some approaches exist to inject SI certificates into containers, but they don't perform OSAware or language aware So I we thought we happy if the automatic C trust injection mechanism exists that adapt

to the differences of uh best images, languages and libraries. So we implemented it with NRI. NRI is node resource interface which is a plugin API for high level container runtimes that write external plugins inject f to contain life cycles and can customize the resource management. NRI has three key points. It can intercept container life cycle events such as before and after creating containers, starting containers and so

And it can mutate container configurations like environment variables, mounts, devices, arguments and so on. And NRI can work with not only contain but also with cryo and from content 2.0 it's uh it's enabled by default and nRI plugin is executed as a standalone process. It communicates with the high level container runtime via the Unix domain sockets. It gets events from the container runtimes and returns the modified

spec or validation results to the container runtimes. So we implemented an NRI plugin to uh automate the sheer trust injection during the container startup process. The program automatically detects the OS and runes in the cont system and then it appends on appends the CS certificates to the files in the content file system based on the result of the detection step and if needed it automatically modifies environment

variables for NodeJS, NodeJS or Python. The advantages of this approach is users don't need to care the differences of base images and runes and this can be enabled with just a single port annotation. So let me show you a demo here. In this demo we'll show you how the user experience will be. First I create a CA certificate with the server certificate from the CA public and

next apply uh HTTPS server with uh with the server certificate. Yeah. And the HTTPS server is decent on this uh HTTPS server service. So we can connect to v this So let's try to connect to this server. Yeah, this is a very simple port template port manifest. It uses the Debian image with car command installed and car car tries to connect to the http https server but

the private CA isn't installed yet so the command will hey yeah let's take it I'm applying the manifest and after a while I'm checking uh the logs of the Yeah. Uh, it prints the error message because the CS certificates aren't present in the container file system. So, let's resolve this with the NRI plugin. So, I need to set up the NRI plugin. First I create the config

map to store uh the private CS certificate and then I apply the NRI pluging as a demo set with uh required manifest such as arbox resources. Okay. So yeah, now the NRI plugin is running as a demo set. So this cluster only has a one node. So the desired replica is just one. So yeah, next I apply the new port manifest. The differences between old one and

new one is just uh two points. the name of the pot resource and the annotation to enable the injection. Okay. Uh I apply the new manifest and after a while again I check the logs of the port. Okay. And now the client can be able to trust the server certificate with just a portation. That's all for the demo. All right, let's go under the hood. If you

are interested, please scan this QR code to access our repository. Okay, our implementation consists of three steps. the create container hook in the NRI program, the create runtime hook in the OCI hook and the M modifier as the container entry point. I'll explain each step in detail from the next slide. Okay, let's start with the NRI program. Our program subscribes to the NRIs create container events. When

the event occurs, our program first reads port annotations to check if the feature should be enabled. Next, we have to read files in the container file system to perform OS and language aware CA injection. However, NR API doesn't expose where the container routs exist in the host file So we delegated this role to the OCI hook and the amp modifier. At this stage we don't modify files

yet. We just add an OCI hook and mount the MP modifier to be executed as the container The next step is the OCI hook. OCI hook is a standard extension mechanism defined by the OCI runtime specification. It is implemented by low-level container runtimes such as lansi siran and yoki. While the N while the NRI is the extension point for high level container runtimes, OA hook is for

low-level From several hook points, we picked up the create runtime hook. In the create runtime hook, we can access the container file system from host. So we detect the container environment by reading files such as etc release or executables like node command for nodejs. After the detection, it updates or places CS certificates to inject the private CA. We don't set environment variables here because it is difficult

to modify container specs in OAI hook. The final step is the MB modifier as the The end modifier sets the environment based on the detected container environments like node extra search for NodeJS. After that the MP modifier calls exfree system call to run the original entry The application then starts as normal but with the private CA trusted. Now the private CA injection is completed. Okay. Let's compare

our approach with First users don't need to manage Docker files to just install C certificates. Also, port template is still simple and the delay before starting containers is short enough. In addition, our approach can perform OS However, our approach requires node level privileges because it modifies files in the OCI hook step. To migrate this risk, we are planning to introduce a sandbox mechanism to restricted file access.

At the end, let's look at issues and future work for our implementation. First, environment variables are not modified on cubectl exec because the m modifier is executed as The environment variables cannot be passed to the process by cubic exec. Another issue is that our implementation is not available for the read only We are trying to resolve this issue by adding bind mounts for replacement files in the

OC hook step. Finally, we plan to integrate with cross trust bundle which is proposed in cap 3257. Cross trust bundle is a standard way to distribute CA certificates to work across in Kubernetes introduced from the budon 1.33. This integration will make CA management much more flexible. Before closing, let me recap our session. Our NR based approach enables OS and language a C cluster injection with a single

port annotation. It suggests that the NRI can be a useful extension point for broader automation use cases. Also, we are aiming for deep integration with Kub Kubernetes standards including cluster trust bundles. That's all for our presentation. Thank you very much. >> Okay, we are happy to take questions. The microphone is over there. uh we have a lot of new developers onboarding to Kubernetes due to cloud migration

things and whatever else and certificates is something we have a lot of and a lot of developers struggle with especially because there's so many permutations um one thing I just saw when looking through the repository do you support Java yet? Um because I saw NodeJS and Python as processors but not Java yet. So what are the things you support, plan to support and whatever else? >> Uh

so you want to use it for it for job >> I'm sorry, what's job? at work but like we have a lot of developers onboarding to Kubernetes that um struggle with the Kubernet with certificates because we have it a bit strange at my work and you the overview you had with the issues is exactly what we're running into with various teams taking various approaches but looking at

the code from CA injector you mostly just support NodeJS and Python at the moment right >> okay I see I see >> what do you support now what do you plan to support and whatever else Okay. >> Uh yeah, we are going to support uh Java as well. Uh but it's not implemented yet because it has its own trust store. So uh we need we need another

approach to uh modifi modify C certificates trust store. Yeah. So we it's in progress. Yeah. But and uh yeah, I'd like to support other uh languages and libraries. Yeah. but still in progress. So, we accept your requests and yeah, let's collaborate. Thank you. >> Thanks a lot for the answer. >> Yeah, thank you. >> Um, currently you're in this project C injector. I see that you use

it for injecting additional CAS. Is there a plan or way for it to be only on I would say replacing completely the the CAS with only the one that you you want to trust. So basically bypassing the CA certificates that are embedded into the image. >> Yeah. Yeah. That's that that's very good good point. Yeah. Uh it's not implemented yet. Yes. So I think uh there's many

such use cases. So yeah, we are planning to apply some options to enable uh the mode for the fug. >> thank you. >> Okay, thank you. >> Thank you for the talk. Uh I have a question regarding uh environment variables in cube cuttle exec. Uh does the same apply to cube cuttle debug because that creates the ephemeral container but I don't know if it ejects the same

events. Ah, sorry I'm not sure about cubectlia debac but it's running as an container new container so I think it's it it will be injected but not tested yet. >> Thank you. Have you guys uh considered using uh the container D API and your NRI plugin to get the root FS path because that would uh remove you'll be able to inject the Nvars and NRI then and

do all the modifications at that level instead of needing the like exact kind of dance you guys are doing in the OCI hooks. Um have you looked at that Mocha's uh we do something similar with create container to get the path um for the root FS? >> I'm sorry. >> Yeah. So you said there was no good way to get the root FS path in your NRI

plugin. If you talk to containerd directly which is kind of a weird >> uh >> paradigm you can ask >> and derive the root FS path and then you would be able to remove your OCI hook and your uh MVARS would be added to the spec. So your cube cuttle exact >> would just work like you get the n bars there as well. >> that's right. >>

Have you looked at that or no? >> Yeah, but in the the create container of ni uh we cannot get the container ID yet. So it's difficult to uh access look up the container file system in the snapshot file system or such. So >> yes I think we can integrate with just the continent API but so NRI can be used for multiple container runtimes including cryo. So

yeah I'd like to that's true. Yeah both run times. >> Thanks. >> Yeah thank you.