What's Coming Next in Containerd 2.3? - Mike Brown, IBM & Krisztian Litkey, Intel

This talk focuses on developments within containerd, a popular container runtime, featuring key updates and the introduction of the Node Resource Interface (NRI). The speaker, an experienced maintainer, discusses the implementation of new features such as GPU vendor auto-detection, improvements in container lifecycle management, and enhancements in snapshotter support. The NRI is emphasized as a mechanism that allows custom plugins to extend containerd's functionalities in a flexible way, enabling finer control over resource management and enabling integrations with various container orchestration tools, including Kubernetes. The session also highlights the importance of maintaining compatibility with Kubernetes releases and the need for contributions from the community to support ongoing development.

Hello everybody. Thank you for joining us. I'm one of the containerd core maintainers. Been working on it since the beginning. I I did the cry containerd plugin with lantaloo also known as random way back when and been working on it ever since. I'm a sign member. We work with the Kubernetes team, work with Docker, we work with Rancher, you know, pretty much anybody who wants to have

a client or is trying to extend the, you know, containerd end up working with them or somebody else, one of the other maintainers. I don't think I see Phil or Airauda. They should be here today. Um, hopefully at at the end of the session, we'll have questions and maybe we can answer most of them. Okay. And we have other contributors in the audience. Raise your hand if

you've contributed to containerd. Okay. All right. I'll be looking in this d. There's a curse. Thank you, sir. Found them. Uh, so we're a CNCF graduated project since 2019. As I say, the we have a lot of public repos. Everything from snapshoters uh to nerdctl and nerd bucks. quite quite a few very interesting projects. We'll be talking about NRI later on which is a a plug-in model

that works with both containerd and cryo. All right. Uh everything in containerd we we try to be uh not opinionated and to allow other tools other clients as well as run times to be able to work with us and extend uh containerd. So as such it's become very popular and is probably in most of the clouds today. We we support uh remote snapshoters. Some of the new

stuff that's you know has been added over the last few years. Um and we've got more new snapshers. Earos for example. There's Wom runtimes uh has been added as as far as having a new type of runtime. CDI uh we we had a container orchestrated device workg groupoup that came up with the CDI CDI plan and now CDI is supported by DRA controllers. So it's been a

few years but it's now enabled by default inside of the 1.7 and above uh containerds. Okay. NRI nextG plugins. That's what uh Christian's going to be covering. Um but it's key to recognize here that we're you know we're gonna also support performance management of the pods and containers. Okay. Uh anything VMs and anything that gets executed will have some new new ways to configure them using the

NRI. Uh Kubernetes pod sandboxes. Uh the original sandbox that we supported in of course in containerd was the pod sandbox through the cry api and there's now now different ways to support pod sandboxes. For example, kata containers does it. Okay. Kubernetes pod sand we did other sandbox types and new CRD controllers. Yep. So there I don't if you've been looking at the Genti or K agent uh

agent sandboxes they're with at a CRD level they have they've reused the term right so they they have their own sandboxes which from their perspective they configure a pod and then deploy it and it's got special configuration but for the most part it's just a pod. So mo most of these other types of agent sandboxes, they have new isolation requirements that we're going to have to implement.

And again, we we believe we can do that with the NRI extension. Again, we did the NRI so that we could do fancy things being unopinionated and Nvidia there's a a bunch of people writing their own uh new NRI. Next release status. Uh, who's on 17 still or six? Okay. Yeah, figures. Um, we're we're moving to a new LTS in 2.3. We're finally going to have an

LTS to replace. Um, and we're still supporting the existing LTS, the 1.7 LTS, which is our last one, right? Um, in the 17 series, we it was scheduled to go out of support and it's being supported through uh the end of September by Samuel Karp. Thank you, and Chris Hines. Okay, which are both Google people. And what that should probably give you a hint is um if

you need one of our releases to be in LTS or you need our the support to be extended beyond then you just you know we need we need help uh please donate uh some maintainers. Okay. Um and we just happen to have two maintainers on containerd that uh said yeah yeah we'll we can do that because GCP doesn't want to move up. You get the idea right?

Uh 2.0 0 it's was out you know a while back and and now it's end of life already. Uh 2.1 support is active 2.2 support is active. Next chart. So here here's some new news. Okay. Our release cadence for containerd is going to become like not unlike the kubernetes release cadence. We're going to ship three a one of those will probably be a new LTS. Okay. Um,

our LTS's will be our stable releases with less experimental features being added. Okay. It also uh whether it's LTS or not defines whether or not we can bring back uh new features in into a a particular release that you might need and and Kubernetes, right? Um with the nonLTS releases, we don't backport, we don't cherrypick a whole lot of things and certainly no new features, okay, which

is what you expect from older releases. uh on the process and tech debt uh line what what's going on um we have issues right with keeping up with what Kubernetes wants to do because our cycles weren't in sync Kubernetes would have an API and the CRI and we would not be able to get up to it yet because we didn't have a release coming in the same

cycle again that's why we've done that Um I don't know if you know what streaming is um but there are a set of APIs in Kubernetes uh that do the exacts um probes port forwarding okay that those sort of things and for those APIs it's our relationship with the kublet changes we actually do an import of the kublet code into containerd so that when we run a

a container and we execute the string streaming or you we do the exec then we have to actually use some code from the kublet. So it's a little so the good the good news in that is a lot of the signode member teams uh they they work with us uh you know very tightly to make sure that we're keeping these things in sync. If they want to

do performance improvements like speedy uh in the streaming code then then we it's done for both but again we have to keep up in cycle with Kubernetes right time 10 minutes oh I'm not time okay I'll go faster this is a list of the features that we have um that have been added added as a standard in in 2.3 GPU vendor auto detection support there's been a

lot of people are wanting to find out what what GPU is being reported uh RFS layers new the alternative file system layout uh to the current one which was TARS just every blob is a layer is a you know is a tar file and ERS changes that right you'll you can actually push an ARFS file system format you know images and layers at at will now within

containerd okay we talked about tech caps link to a list of caps here okay next just showing you uh you know a set of the extension points that we have for running uh you know run times. We even have one for FreeBSD that Samuel also did. Uh he's done a good job that and everybody's excited at free in FreeBSD land. Um of course we have run hcs

and windows. Uh we support crun run C whichever and you know runj which we have a lot of people running runtimes. They use our shim API and it's a lot of fun. Yeah. Nice. This just a list of the our regular snapshoters. Notefs has become a one of our regularly supported, you know, premier file systems that we expect people to be using and and we have these

remote snapshers like block overlay very very and and Sochi Sochi is pretty cool to go look at. Uh, Phil Estes, I don't know if he's in the audience yet, but he he has the team that's doing doing the associ projects. We have a lot of them. So, if you're interested in contributing, uh, you want to work on the nextG stuff, you know, pick pick one of these,

uh, talk to me afterwards and and we'll get you on board, uh, to working with these experimental things. Okay. >> Thank you, Mike. So, so I have a few remain remaining slides about NRI. Uh, so basically NRI status updates. Uh, but I will start with a recap because I suspect that there are many quite many people in the room who is not quite familiar yet with what

NRI is and what it can do and then I have a few slides about recent developments that we have been working on uh in NRI up till this point and what we are focusing on now. So what NRI is? So the NRI is the node resource interface uh which is a common extension mechanism for CRI compatible runtimes. We have two of those uh or two major ones

cryo and containerde. So it's a common extension mechanism because it's supported by both. It has been present in uh container day from 1.7 and from cryo if I remember correctly 1.26. It's an extension mechanism because you can use it to plug in extra custom logic uh of your own choice. So this logic is pluggable. So you do not need to recompile the runtime. It's not built in.

Uh instead it's implemented using external binaries called them plugins. And these plugins uh communicate with the runtime using a TTRPC based protocol over Unix domain sockets. uh and as such this logic of yours if you wish to plug it in it can be uh plugged in updated and plugged out independently from the runtime itself simply by starting up external plug-in binaries updating them and stopping them. So

this custom logic that you can implement using NRI is portable because uh if you write a plug-in regardless of whether they are using it with cryo container it should work identically s bucks of course. Uh so there are some gaps sometimes but we try to close them pretty fast. It's it's mostly related to the release of container and cry and how we integrate the latest and into

those. So what these custom pieces of logic can do uh so in short uh you can hook your plugins into pod and container life cycle events and you can do limited changes uh to some container attributes uh in a controlled fashion and so we'll see shortly what these changes are. So under the hood how NRI works. So if if you want to understand how NRI works, we

have to take a look at how basically the runtime itself works. So this is a vast oversimplification and it's only uh from the point of view what is interesting for NRI. But basically uh when you create a pod so the orchestration layer first organizes everything so that the port lands on a cluster node on the cluster node your uh cluster node agent the cublet or assigns it

to a node. So the cluster node agent the cublet picks it up uh takes the pot spec and uses the CRI protocol. So the container runtime interface to talk to the runtime uh run a pod and then start creating containers in that pod. So what the runtime is essentially doing it is uh taking the container request from the CI protocol and its own configuration and based on

these it creates uh a formal description of how the container should look like. So it's a sort of recipe which describes uh things like what command should be run in the container uh what environment variables should be set what mounts and devices should be injected uh name spaces and so forth. So all the details of the container. So this is called the OCI spec. Uh and this

OCI spec is then passed on uh to a runtime handler uh which is basically a low-level runtime such as run C, C run, cuta containers. There are also others. Uh and then this low-level runtime is taking care of the dirty details of creating and setting up the container. And this is where NRI comes in. So the essence of NRI is that at this point when the pristine

initial OCI spec has been created, so NRI basically intercepts it uh calls out all of the registered NRI plugins, collects any adjustments. So changes basically to container uh attributes which the plugins would like to do. then checks the collected adjustments uh to detect if there is any uh accidental conflicts. So which the plugins are doing without knowing that they are changing stepping on each other's store. >>

Uh if there is any then flags it as an error prevents container creation if there are no such conflicts from that point on it applies the changes of course to the OCI spec. This is the final OCI aspect and from this point on container creation goes on like without an array. Uh but because container life cycle is a little bit more complex than just creation. So container

is first created then it started it can be updated eventually it is stopped and finally removed. So there are these additional life cycle events that NI allows you to hook your plugins into. So what can your plugins then actually do? So as we saw they can hook into these life cycle events. So for pod you have the corresponding ones the actual CR event. So you have ramp

sandbox update pot sandbox stop pot sandbox and remove pot sandbox. For containers we have creation start update stop remove. But we also have post variants of uh create start and update. So these post variants basically allow the so in the case of multiple plugins they allow the other but the last plugins to find out what was the final state of the container it was created or to

which it was updated >> next slide back one So uh and in so in connection with these some of these events uh basically plugins can can request as we talked earlier about so it they can request changes to container attributes. So this the biggest set of changes can be done during container creation because by nature containers are mostly immutable once they have been created but there is

a small subset of the container attributes mostly related to >> resources. So compute resources and other resource management attributes which can be subsequently updated and NRI works the same way. So initial adjustment you can is the largest scope adjustment which covers uh most of the parameters. Uh so these include annotations, mounts, devices, environment variables or CI hooks but they also also cover these resource management attributes like

which CPU sets a container is allowed to run on uh scheduling parameters, memory limits, uh which memory controllers you can allocate memory from and all the unified CP2 attributes. okay so addition so the in addition to this so some parameters can be updated in connection with other events than just container creation. So these are the container update in which case NRI plugins can also do other updates

and container stop. So when you stop a container the remaining containers which are still running uh can be updated by n plugins. So these were the initial set of attributes that we had controls for but lately we have been adding several new ones. Let's see what these are. so we have on the functional scope front we have added controls for uh Linux scheduling attributes, Linux IO priorization,

name Linux name spaces, sec policy, uh posics r limits uh well bunch of other things sail parameters. So additionally uh we added support for injecting uh network interfaces to the container. So basically moving existing interface from the host side to the container side and to do richer device injection that it is possible with with plain uh device nodes using CDI. So the container device interface that Mike

mentioned earlier. So on the NRA infrastructure side uh we introduced what we call pluggable validation to provide more administrative control over adjustments. So basically with pluggable validation you can plug in a validator plugin which is a special type of plug-in uh and in this NRI event processing phase. So there is a new phase once NRI has made the adjustments to the initial OCI spec. So it calls

out these validator plugins and basically a validator plug each validator plug-in can either accept or reject uh the changes which were requested by other NRI plugins. So if any of the validators rejects the adjustment then the container creation fails. If none of them rejects then it proceeds as normally and these are pluggable. So we have a default one that we implemented uh which basically has some functionality

but because these are just NI plugins like like the rest. So if you need something else that we have not implemented that then you can roll your own and tailor it for your cluster cluster nodes and then have it exactly work as you as you would like. Uh so our built-in default validator has a number of configurability options. So basically it is able it it is mostly

tailored for locking down sort of controversial security related uh controls that you can adjust via NRI. So these include OCI hook injection sec policy mutations namespace mutations adjustment to CCTL these type of things. Uh so additionally uh there is support for declaring which plugins must be pre present in the system for container creation to succeed. So if you have some really important ones which always need to

be there otherwise your containers will end up starting up in a in a way wrong state then you can declare these as required ones and if any of those is missing uh container creation will be prevented. uh to avoid this bootstrap chicken and egg problems where you are rolling out your validator plugins also as basically containerized workloads. So we we have support for tolerations. So you you

can uh you can annotate your containers whether they can tolerate any missing required plugins and which ones they can tolerate. So this this helps basically bootstrapping the static pots and then the required plugins themselves. So we also have new plug-in types. So redhead contributed vess plug-in support. So we can load and run plugins uh which are compiled into vess binaries. So these are strictly sandboxed but if

you use such a plug-in then you do not incure the RPC overhead of TTRPC communication. So this might be sometimes a good tradeoff if your plug-in logic is such that you do not need any external input like you do not need to introspect how your hardware topology looks like none of that. So if all the input which is present in an NRI request uh and all the

metadata about which is which are present about pots and uh containers. So if that's enough for your plugins logic's decision making then this can be actually a good trade-off. Uh when we implemented the default validator plugin we added support for built-in plugins. So it's we do not use it for anything else but if you are building your own container the cryo version and you have some plugin

which is semantically a required plug-in but you would not like to run it as an external one then in principle you have the choice that you could compile it into the binary that you have then it's always available but then this of course comes down with some difficulties for upgrading so you need to upgrade it to upgrade such a plug-in you need to upgrade basically your runtime

binary version. Yeah. >> Yeah. So current work, what are we focusing in NRI now? So, so all of our current efforts are for reaching a first uh API stable release of NRI. So what we call a 1.0. So from 1.0 once we reach it, we'll have the stability promise for binary and source backward compatibility. So after that when we roll out the new release of NRI and

the runtimes. So your existing compiled external plugins should work without recompilation. Uh and if you recompile them then they should not require any source level for any wire protocol changes that we might do for instance some optimizations on the wire. So the recompilation of a plug-in should should be enough to bring in those benefits that we are adding to the protocol. Right. >> So we also have

some work to do on the manageability and observability front. So we are adding an related metrics exposing them from the runtimes things like how many plugins you have, plugin latency, how many requests they have processed, whether they how many are failed, how many succeeded these kind of things and audit logging. So basically if NRI plugins are present which attributes they touched of what containers and what the

changes requested changes were. So something goes wrong then you can go through the audit trail and check whether it was because of some of your plugins. So some other things that we are working on on the infrastructure side. So one big thing is authenticated plugins. So the primary goal for authenticate plugins is again to provide more finer grain validation. Uh so basically what we do with that

is that uh authentication only establishes reliable identity for a plug-in and then in the built-in validator we added support for that that you can assign basically a role to an identity and then you can assign a role specific configuration. So basically that means that you will be able to do things like that that by default you lock everything down. So if you have an unauthenticated plug-in it

will not be able to do any security sensitive changes. But if you have your own plugins which you have thoroughly reviewed and you are sure that it's fine for them to do these changes then for those you can open up the configuration so that they are allowed to do this. >> Right? So removing the trust of containerd down into the plugins as well supporting the security context

and from your pods. Okay. we have one feature request that we received just a week ago which we have already considered implementing earlier but we didn't need ourselves so it's not there and this is something that we might consider for the 1.0 time frame that's hooking in NRI plugins to container XX. So for instance for probes when you are execing into a container creating a new process

then there are some cases where it would be beneficial if n plugins would know about it and could do some alteration of the environment where the probe is running in. So environment variables >> let's see you may not know it but when you do a probe in kubernetes that actually does an exec into the container which is a new init proc. It's not the old init proc

it's another one. So you'll have command and this other probe exec running at the same time. >> And then finally what we have left is improving documentation and then basically reorganizing our tests so that that the end to end test we could share better between container and cryo because now there's a lot of duplication there. So that's it. So questions please come up to the mic. Thank

you man. All right, Alexander, you get to ask. No, you're good. You're good. >> All right. Certainly, somebody wants to know something hard to answer. No. >> Yes. >> I can't easily get out, so I'll just try and yell it out. I was looking at the repo, but there's extensibility in all different layers of the stack and it seems like some of this like as a cluster

administrator if I want to apply uniform policy like I could also do that with a mutating web hook. >> That's right. >> Yeah. So, so I would say that this mutating web hook question comes up a lot when somebody is first time about Okay. So the question was that that there are other sort of >> other ways >> other ways to basically >> yeah other way to

basically contite whatever >> and for instance web hooks this web hook one comes up very often and the major difference between a web hook and naira >> so a web so in a web hook you can look at one container at a time and you can do make changes to those container so that container but once you made your changes you cannot regret it there is no

way to do anything else and with NRI what happens is that so when you are adjusting the container at the same time you can make updates so these more limited resource level changes to other existing containers and when a container is stopped so it's going away then for the remaining ones you can also do this so this allows you basically also to plug in sort of so

resource management naira plugins which are replacing basically what you have now in the so you can disable what you have in the cublet so the CPU manager topology manager and so forth and you can implement something which is more flexible and more tailored for your cluster your needs if you have some special needs using an NRI plugin. So this is the major difference between web hooks and

NRI plugins. Yeah, there's been so many solutions that needed to be done um at the lower runtime level when after the crates happened before the start and as you mentioned before the stop there just things that need to happen and you that you can't do in a web hook because they don't even know it's exiting yet, right? Um so what we had was a situation where people

were doing proxies at the cry level, right? Cry proxies and >> for instance us for instance, Intel. So, we decided, you know, we need an interface that teams like Intel, Nvidia, whatever, can can come in and extend and it's turned out quite a few people are doing this and need it. Um, you can't do everything at a web hook >> and actually so the so the the

great improvement with NRI over those CRI solutions is that so in at CRI level, you have very few knobs exposed to you, right? But at the OCI spec level and at the NI level, you have a huge amount. So all those you can control and it's not really feasible to expose all them you know to the orchestration because I mean complexity would be just unmanageable and it's

not usually not necessary. >> So yes, it's yet another plug-in model but it's one that's supported by cryo and containerd already. Right. >> And actually docker is working on >> and docker. Yeah. Yeah. Right. Right. Right. Yeah. Yeah. Okay, that's a good question. Yes, >> thank you for the nice presentation. My feeling is if I can >> so the the question is if I'm mutating at the

DRA level in our new CRDs, right? Then how can we make sure that the you know these plugins aren't doing the wrong things now right or how do who's in control this is why we're supporting these validators right in restrictor basically restrictors that'll say no no that that plugin can't either can't run or can't do the adjustments that it would like to because those fields are locked

down and we're working with the kublet team to make sure that we you know we take the spec in we know which ones your setting and we can use annotations or whatnot, right, to to turn things off.