Is the Agent in the Room with Us Right Now? - Nick Rutigliano & Andrew Halaney, Netflix

In this talk, Andrew Helani and Nico Trigiano from Netflix discuss Titus, their internal container platform that operates on Kubernetes but is not Kubernetes-aware. They explain how they handle non-standard container workloads and ensure resource isolation on their data plane. The speakers elaborate on various techniques for pod creation, resource management, and network isolation. They also highlight the challenges faced with user namespaces and the inotify system, detailing a recent incident and the subsequent fixes they implemented upstream. The presentation emphasizes the importance of user namespaces for security and resource limits, along with innovative networking solutions like their custom CNI, It Man, which enforces unique network identities and security groups for pods.

Thanks for coming to our talk, last one for the conference here, so I appreciate you hanging around. Um, is the agent in the room with us right now? You guys look like humans, so I think we're good to go. Um, I'm Andrew Helani, I'm a senior software engineer here at Netflix. I'm Nico Trigiano, I'm a senior SRE here at Netflix. Um, we work on the compute runtime

team, which is responsible for our Kubernetes or Titus data plane and the base OS for Netflix. Uh, so what are we going to talk about today? Um, we're going to explain briefly what Titus is, it's our container platform internally. We're going to talk how our container workloads aren't really standard Kubernetes workloads, they're not Kubernetes aware at all, um, and that has some kind of weird expectations, um,

but we do use Kubernetes under the hood. And so we want to talk about things we do on the data plane on the host to isolate um, pods from other pods. So, you know, things like noisy neighbor problems or if a container was to escape isolation, how we could limit the blast radius from that context. Uh, so first, what is Titus? It's basically a simple API, um,

internally that we use. So you would say like here's my container image, I want this much resource, um, like, you know, four CPUs, let's say, um, and under the hood we make it happen. Um, it's got a really long history, um, started out on Mesos, uh, then we kind of shifted to like a super thin pod spec using a virtual kubelet and Docker. We eventually kind of

flushed out this pod spec more and more and found that, you know, using virtual kubelet and Docker on our own data plane was quite limiting. Uh, so the last year we have migrated to using kubelet and containerd with a bunch of plugins, um, to kind of keep the expectations we had prior in the same isolation. Um, so our workloads, uh, really kind of look like a VM

to our users. Uh, that's the world they came from originally and they still expect that kind of behavior. Um, so we do weird things like we inject SSH among other processes into the containers pit name space directly. Um, we let them think they're running as root at least. Um, we run system D in a lot of these containers. I'd say probably the majority. Um, they can bring

their own storage mediums and we attach those in kind of non-standard ways due to the history of um, our platform. And then we have I think pretty good resource isolation between nodes. We run some latency sensitive applications in a some multi-tenant environment and typically we don't have a whole lot of issues. Oh. So yeah, if you want to understand how you can do resource isolation, um, you

kind of need to know where you can do resource isolation. Um, and obviously there's a million things you can do uh, at a higher level like the off and off C stuff within the control plane and our back and cluster isolation or name space as a boundary. We're only going to talk about the data plane. Um, and from what that means is that most of it happens

during pod creation. so for this it means from a kubelet and container runtime perspective, we think of uh, pod creation in sort of four major steps spanning the runtime and image services and these are where we can sort of do sort of fancy isolation things. Um, run pod sandbox, pull image, create container, and start container. Um, some of these are configurable either via your pod spec that

you actually give to the Kubernetes API or they're configured via how you set up your container D or your kubelet. Um, and we use a couple reasons few major ones. The first one is NRI. Uh, this stands for node resource interface. This is sort of um, allows you to plug in domain specific logic into OCI runtimes like container D or cri-o. And these are effectively um, life

cycle hooks at different stages like run pod sandbox, create container. Um, and these are useful for container adjustments. Think like mounts, devices, environment. Then we use OCI runtime spec hooks. These are actual hooks during uh, the container life cycle. Um and these also have a There's a whole range of the different points um in time available to you. The difference with these is these side effects happen

within sort of the containers environment or inside the containers uh runtime namespace. And then finally, we have an OCI wrapper. So, we actually intercept all calls to our runC. Um and instead of specific hooks, we can just do full edits to the actual OCI bundle um and just full OCI spec changes. So, this is kind of a pod life cycle trace, kind of showing what happens from

start to finish. Uh sync pod, if you're not aware, is kind of the entry point for pod life cycle it within the kubelet. Um and from here, we go down through uh into our container runtime and into our NRI endpoints and our CNI. And these are sort of all the places we're going to talk about today where we do something uh bespoke to, you know, isolate agents

or just handle multi-tenant a multi-tenant environment. So, the first one is run pod sandbox. Um So, you might be familiar with this. This is sort of the baseline environment for the pod. This is all the things you need to set up before you can even create your workload. Um things like creating your shared namespaces, uh setting up your C group hierarchy, set up your networking, um create

the pause container to sort of act as the home for all these resources, so that when the pod is deleted, you have something to sort of garbage collect under. Which leads to sort of the first thing we do here. Uh yeah, so the first thing we do is we enable user namespaces. So, you kind of have, like, I guess, three options for what you can do with

your containers with respect to what user they actually run on on the host. Uh you can let them run as real root, but if they escape their isolation, you know, they get out of their mount namespace somehow, they are root user and can do whatever. Um you could isolate it uh so far as to say we don't let anyone run as root, but that can make it

kind of hard to run certain workloads, and if you want to do uh like rootful things, uh that's not possible. Um so, user namespaces is a really nice middle ground. Um each pod is going to think it's running as a root user inside its user name space. It's going to get full Linux system capabilities. It's going to be able to even mount some file systems. But under

the hood it's not running as root user. So how Kubernetes does this is you just set pod.spec that host users to false and it kind of goes from there. So kubelet will assign each pod its own unique user range. So kind of a hand-waving example here, but say pod one is you running as user 100,000 to like 165,000. Pod two is going to get another user range,

say 200,000 to 265,000. Within the container 100,000 is going to look like user zero or root. So that's nice. Another thing to keep in mind is user name spaces are nested. So technically a container can make more user name spaces. It can form a whole tree. And there's a bit of an ownership concept here. I'm a big fan of man pages and in kind of a user

name space zealot I guess at this point. I like this man page a lot. You should read it. So Kubernetes, you know, all these pods should be isolated with with respect to anything user related with user name spaces enabled. But recently we kind of ran into an incident that I thought was kind of fun. So we're going to dive into Where this was kind of broken and

we fixed it upstream. So we discovered this with a inotify based workload. So if you don't know what inotify is, it's a file system API for watching for events like was something deleted under the current directory. You don't really need any fancy privileges to do this. So like you don't actually need full system capabilities. And it has if you read the man page for that, it has

per real user ID limits. So the way you would interpret that is since each pod is running as its own unique user range, they should not be able to affect each other. but we wouldn't be talking about it if that was the case. So imagine you're in my shoes, you get paged in the middle of the night, um your nodes are unhealthy, you're having a hard time

running a bunch of pods, and you're trying to figure out what's going on. Um I hop on a node and I look at the journal control and I see that systemd is failing to set up I notify for various services it's trying to spawn, which is kind of critical for our Um and you can see it gets this no space left on disk error. Um and that's

what I notify will return when you're exhausting the limits I mentioned prior. So I'm trying to figure out what's going on. Did we push some kind of bad change? There's nothing that comes to mind. Uh I'm Googling around and reading the man pages further for how can you track down how these limits are being used? There's not really a good way, so I end up just doing

a uh proc crawl saying like for every PID on the whole system, let's look at all their FDs and let's count the I notify watches and find who's using all these. Um so I find this PID, it's like 1 million something and it's using 500,000 of the I notify watches on the system. So he's clearly, you know, the bad actor I look a little closer at him

and it's actually one of our users. It's a pod and it's not running as root, but somehow it's affecting systemd on the host, which is running as root. Um so that's strange. I, you know, kill the pod, they can deal with it. Um and uh the system gets healthy, so I know that that is my problem. I contact them, they stop doing what they're doing, and uh

I go back to bed for the evening. Um but, you know, in the morning, um like kind of nervous cuz now I realize someone can just page me in the middle of the night whenever they want. All I have to do is kind of write a I notify stress test and I'm out of luck. Um so I'm trying to figure out why the root user was affected

by a pod running in user name space. They should be isolated, that's the point of this. And you can't really find any good docs on this. Um there is a kernel doc which I kind of found. It doesn't mention I notify specifically, it mentions a larger class of user limits, um but this falls into that. So, you read the source code for the syscalls and you can

find that there's actually two more things at play here. Uh so, this is how I view the setup that how the kernel does its accounting. So, the top red box here is the initial user namespace on the host. Um and you can imagine that that bubble up there is the root user doing whatever. So, systemd doing inotify, for example. Um the bottom box is a pod. Let's

call it pod one. Um and you have two processes in there. Um for this we really only need one process to explain this. Um so, they have their, you know, root user or on the host it would be user 100,000 in the bottom left. Um you can also see that there's two files I introduce. Um there's a max user watches file under proc sys user. Um these

are user namespace scoped. So, if you read it in one user namespace, you're going to get a different value than in a different user namespace. They're scoped to the namespace. How the accounting works for all user limits, not just inotify, um is if say the container was to set up an inotify, first the kernel would check the local max user watches and say, "Is this user over

the 50,000 count?" If no, it will continue up the tree, all the way up the tree to the root user. So, containerd is the owning namespace of all of our pods and it's running as root. So, effectively what's happening here is that despite running under user namespaces, our pods are able to kind of use the host the root namespace the root users' resources. And uh all pods

are now sharing this. So, this is not great. Um so, what can we do about it? I kind of actually foreshadowed here. Uh you see this 50,000 number here? We could arbitrarily limit each pod. Um so, we could say you don't you don't really need 500,000 inotify watches. That's ridiculous. Um 50,000's good enough and I'm not going to get pages anymore. Uh, so that's what we did.

We needed to make a run C patch so you can actually set that file up. Uh, but with that in place, we can kind of see how that looks in action. Um, so here's the config.json that backs your container. So, container D produces this, passes it off to run C. Run C actually creates the rest of the container. Um, and in it, you can see the sysctl

user.max_inotify_watches is 50,000. Um, so now, with the patch we have, run C will honor that and actually apply it. So, we're in the container here, and you can see that, uh, there's 50,000 Um, it's limited 50,000. If I cat the file, and then I run a bit of a i notify stress test, um, and you can see we can only get 50,000 i notify watches applied. So,

if you were to on the host, you could still do i notify, you have plenty of headroom. Uh, the container can't modify this either. Run C mounts this in it as read-only. Um, so they can't really escape. Still, that kind of sucks. It's, uh, like a a totally a band-aid, and it breaks the idea of user name spaces limit, uh, being unique to each pod. Like, now

there's a shared limit still. So, we're going to fix that. Um, so we went up to container D and worked with some people upstream, um, and got a solution in place where container D will switch users to the container user, create the user name space, and therefore the container user is now owning that name space. So, the billing should all be, um, uh, correct now in that

each pod is fully isolated, and it's also isolated from the host. now we're on the host of one of my systems with this patch applied. We have a bad agent at the top here. I go ahead and show that he is indeed using 500,000 i notifies. Um, I'm going to go ahead and show that, uh, I am actually on the host. So, I compare the user name

spaces here. The i node file kind of shows you that these are indeed different user name spaces. Um, and then I'm also going to run a little program that uh, does an IO control to say who owns the username space of the container. And you can see that it's now this big UID 1147994112. So, it's no longer root. Prior, that would have been the case. Um, I'm

going to pretend to be system D and use a bunch of i-notifies on the host while the container's still using 500,000. Um, I run my little program again and I can use almost 500,000 something i-notifies before the root user is out of uh resources. So, this is great and we've kind of helped fix this class of user limit problems across the board um, upstream in containerd. So,

if you use pod spec host users, you're you're no longer going to have this problem as of, I think, 2.2, 2.3 in Cool. So, the next thing we set up, um, within run pod sandbox, as you know, is sort of setting up the pod network. Um, and this is done via CNIs, as you all know. so, we have our namespace and all the containers of a pod

must be routable, so they have to, you know, sort of share the same network namespace. Um, and in our cloud environment, um, and with our needs, we couldn't basically bring in an off-the-shelf uh CNI. So, we wrote it man. Not to be confused with It Man featuring Donnie Yen, but I would highly recommend it. It is on Netflix. Um, It Man stands for IP manager. Um, and

our main unique CNI requirements were that pods can bring their own security groups uh and a much stronger network isolation model. So, we're going to go through this quick cuz we got a lot to talk about, but uh the core architecture of It Man involves this sort of trunk branch ENI setup. Um, trunk ENIs are attached to hosts as L2 IPVLAN interfaces on the physical NIC. When

a pod lands on a host, uh as you can see on like the far right, um, a branch ENI is attached to the trunk and it gets to bring its own security group. It even gets to exist in its own subnet and even its own VPC. Um, a branch then gets its IPVLAN uh interface moved into the containers network name space as eth0 and then when traffic

leaves the container it's tagged by its branch ENI's MAC address and VLAN tagged. So it's still sharing the same underlying NIC but it actually looks like it's on a different L2 domain. The other interesting piece with IP man is how we do network resources. So network resources for us are sort of first class requests and limits on the pods back. And we do this using TC with

EDT sort of the algorithm earliest departure time. So we can actually set sort of a limit of the bandwidth every pod can use. An EBPF program classifies each packet by its sort of container which is a combination of like the VLAN ID plus IP address plus TC class. And then we stamp each packet with its This replaced our HTB classification which is like a previous model we

had for doing this but this had a bunch of bottlenecks in the kernel. If you want to know more about this specifically how we enforce this bandwidth using EDT there's much better talks than I can ever give. I would definitely look those up NetDev has some of them. The important piece with this is that as compared to sort of stock Kubernetes where you can kind of get

network as a resource with some bandwidth plugins and I think even some of the newer CNI support this. You know, we can enforce a hard both a quality of service and a limit. So your agent deciding to run iperf and drown your node is just not a thing that can happen here. Going one step further when we talk about the trunk branch ENI setup, interesting isolation semantics.

So again because we can bring our own compared to network policy where sort of enforcement happens on the host this happens kind of at the hardware hypervisor level by our cloud provider. And additionally because the pods themselves have unique security groups as compared to sort of a security group Uh, traffic leaving a node um, sort of in the standard case, you know, two pods could access the

same resource if it was gated by that security group, whereas that's just not a possibility for us. So, they they always sort of had this unique network identity. additionally, as I mentioned before, because the branch ENA uh, MAC address or branch ENA is added, um, separately, pods on the same host can be belong to different networks, uh, be in different VPCs. Pod A can access the internet,

pod B might be internal only. Um, so we have this really strong networking boundary. but also because of this, we actually have kind of a caveat here, where kubelet and pods actually can't route to one another directly. For kubelet to talk to a pod, it has to actually leave the leave the NIC and traverse our cloud fabric, and then come back and qualify for all the security

groups. So, for example, in our pod spec, we actually don't support like network probes. Uh, the next stage here is pull image. There's no real extensions in NRI or the OCI hooks that you can do for this, but there's some configuration options that are interesting, as well as uh, performance improvements uh, around username spaces and mount name spaces that I'd like to go into today. Um, so

first thing is volatile overlay FS. Um, so by default, when you use containerd or probably grao as well, I assume, um, you use the overlay FS snapshotter. So, every root or every uh, container is going to be running its ephemeral root FS on a shared disk on your host. Um, so this is a shared resource, right? Um, and it's an ephemeral root FS. Uh, so sync semantics

don't really make sense for containers in this context. Uh, like if you do a sync and you want to guarantee it's flushed out to disk, why bother? Because the the whole thing's ephemeral. Um, so we had a kernel patch for a long time, uh, to stub out sync on overlay FS, but not even a modern Linux is kind of a stretch, actually. Uh, as of a while

ago now, overlay FS has had a volatile mount option. Containerd has configuration options for this, so you can go in your container d config under overlay FS snapshotter and flip this on. And I would recommend everyone does that. I'm surprised it's not the default. Uh further to enable user namespaces, you kind of have a bit of a situation with how we deal with images. I'd like to

go into that some. Um so a reminder that container d uh is running as root. When you push your OCI image up, you know, it's a bunch of layers with a bunch of files and they'll have, you know, root will own some files, foo bar will own another file. Uh container d is going to respect that, so you know, it's going to pull it down as root

and all the files that are supposed to be owned by root will be owned by root. But your pod is running as user 100,000, let's say, due to user namespaces. So it cannot access this uh this file. The permissions are all wrong. So we need to do some kind of transformation. Um historically, there was a slow chown option. Uh I think that was mainly to get user

namespace development off the ground and was never really considered as a uh We got too far here. Oh, no, we're good. Um never considered as like a production use case, but what that would do is chown every file for every pod. So it go, you know, you'd have let's say three layers. It would chown each layer to pod for pod one do uh you like, you know,

if if something was owned by root, um it would chown it 100,000 up, so the root in the user namespace could access it. Um this doesn't really scale well because you'd have to do this for every pod since the pod's UIDs are unique. Um so you're duplicating files on your file system and it's no longer really getting the benefits of an overlay FS where you can share

these layers if they're, you know, the same base image. Uh so modern Linux supports bind mount ID maps and that's kind of what I show below. Um so you've pulled your image on the left. It's made up of all these layers. Let's say you have three layers in your base image and for simplification, let's say they're all by root. Let's say pod one's 100 user 100,000 again.

What you'll do is you'll make a bind mount, set the ID map attribute, so it shifts it 100,000 users up, And now you have a view of the file system where user 100,000 is owning all these files. So, the pod can access this fine, and uh the other better benefit of this is since the UIDs are unique, um pod two, let's say it's running as 200,000, if

it escapes, it couldn't access through these bind mounts, either, because the user file permissions are all different. But, uh this is actually still like not good enough, really, if you're running containers at scale. Um we were hitting a lot of problems with when you started a bunch of containers at once, and you had a decent number of layers, uh the mount API kind of just choked systemd

out. You can see a systemd issue here. There's like a bazillion of them where systemd is uh people are complaining about Kubernetes and the amount of mounts um that it makes on the host, and systemd is saying that's a bad practice to begin with, and that there's no good kernel APIs for systemd to use to like get around performance issues due to Um I also kind of

opened a get up issue or a containerd issue about this, and said, "Can we do maybe a little bit less mounting?" Work with uh Rodrigo, who's probably somewhere in the room here, um to help kind of improve this. We came up with this solution where, instead of doing all this per layer, we would just find the parent directory of uh all the layers, bind mount that once,

and then build everything through that. So, this goes from like a n by n operation to an n by or o by n operation. So, it speeds things up a lot. Um we've been playing with the overlay FS snapshotter, too, which is a kind of a fun new snapshotter in containerd, get some good performance out of that. Um and we added user name space to support to

that, as well, in the same manner. Um and then this is like the slide I care the most about in this presentation, although it's starting to deviate further from the uh the topic at hand. Um this is future mount name space uh kind of improvements. Um there's a wish list from the UAPI group about kernel features, which is pretty cool, and it tracks where they are in

their life cycle. Um, and then I have a couple issues that are near and dear to me. Uh, number one, Kubernetes makes a lot of intermediate or a lot of mounts, um, and systemd doesn't handle this very well. Um, so historically, what would happen is, uh, systemd would get a Udev event saying something changed, and then the only way it could figure out what changed was it

would scan the whole mount table. So, this takes up a bunch of locks, and the systemd spends a ton of time doing this, especially in that like per-layer mount you're scaling 100 containers at once. That's a lot of mounts to operate on. Um, there's a new API from fanotify to tell you exactly what changed. Uh, so they no longer have to scan the whole Uh, that's something

that I would like to help get upstreamed. Um, containerd is still using the old mount style API for a lot of the snapshotters. Um, and this requires like physical paths on the mount namespace to be mounted, so, uh, you can't use file descriptors. Um, so like if you need to create intermediate mounts, uh, you have to actually mount them, and that costs, uh, like you have to

get a lot of spin locks in the kernel to do that. Uh, the more modern open tree API lets you operate on file descriptors directly, so you no longer have to do these intermediate mounts. You can create the final mount using the file descriptors. Um, and finally, uh, runc could benefit from some new APIs as well. Uh, when you create a container, you unshare the mount namespace

from the host, so you create a new mount namespace. In doing so, you inherit every mount from the host, so all your Kubernetes mounts come along with it. runc then does a pivot root to the the container's rootfs, so now the rootfs looks like it should in the container, but you still have these old mounts around. And they're not supposed to see these mounts, right? So, you

have to unmount all this stuff, and it's a total huge waste of time. Um, there's a new open tree namespace flag that should let you kind of just jump straight to this directly, um, and I'm excited to see where that goes. So, now we're in create container, the next stage sort of in the CRI life cycle. We have our pod sandbox set up. We have our sort

of image reconstructed. We can now actually create like our actual workload. And in this stage we'll hit most of our extension points. We'll hit NRI OCI hooks, our run C wrapper, and a lot happens here. Um So we start with container D turning your pod spec into our OCI spec. We hit our NRI life cycle hook on create Our create container NRI hook updates our OCI config

to add an OCI hook. Container D will then trigger a run C create. This will hit our OCI run C Our wrapper will execute. It will also make modifications. This will then in turn call run C. And as run C is executing, it will call our create runtime hook. That was a mouthful. Why do we do this? Seems like a lot of modifications to just creating a

A big thing for us is process coordination. So as reminded before, you know, our containers at Netflix are really special. We have a lot of processes running alongside our user And additionally, we don't use CSI. Volumes and data are not set up beforehand as they would be in like a volume manager inside a kubelet. So our our creation has now two added goals in that processes shouldn't

start until volumes are ready, which we have to do now. And processes need some additional coordination to work with, you know, all of our sidecars injected. And we do this using a system called tini, which is just init backwards. And we inject this into all of our containers. And this is kind of one of the main functions of our wrapper of our And that what it does

is it'll hijack the entry point of the container that's provided and replace it with tini as well as injecting a socket into the container so that tini can listen on the socket and know when to start. And we'll talk about this a bit later like when we actually do start the But you're probably again wondering why. We've kind of created a check that doesn't really exist, which

is kind of a post create container pre-start container checkpoint preventing like time of check time of use issues. And this is important because we attach storage in sort of this unorthodox way and we have to sort of synchronize with our sidecar processes. The other thing that we do here is we kind of dynamically compose our seccomp and app armor profiles for all of our So again, Titus

users don't interact with the Kubernetes API, they interact with the Titus UI or the Titus API. And they'll declare sort of a high-level capability that we allow them to use such as like, "Hey, I'm going to run an agent." And then we can basically dynamically select the correct profile that has sort of the security model that we want or the security posture we want. You're probably again

thinking, "Why are you doing this in a run C wrapper?" NRI traditionally didn't support this. I think it supports this in 2.2, but that didn't exist at the time. a webhook could also work, but this doesn't quite have the failure mode we want. We're part of the runtime team and we wanted this as a runtime invariant. and it just gave us more control as we already control

the agents themselves. Um additionally, a lot of this didn't exist by the time that we were creating this. So, you know, in comparison to stock Kubernetes, if a seccomp profile is not part of your pod spec prior to 1.27, you had no syscall filtering. Um at 1.27, they added seccomp default. similarly with app armor profile, I think before 1.30, this was just an annotation. Um now, they're

sort of added changes you get the container runtime default one. But again, these didn't exist by We also do some resource limit settings. So, by default, you kind of enable or you inherit your container runtime's resource limits. So, you're kind of prone to like fork bombs or like someone exhausting all the open FDs that the system can really practically handle. So, we set this a little lower.

There is a a static option pod max pids that is a little different than setting the our limit directly, but it's still similar um and something that we are considering using. Also now there's a cap that's very very recent uh 5758 that lets you add these kind of limits on a security We also do CPU isolation. So all our resources have uh resource limits set in our

pods um and we basically had come up with a strategy that became what CPU manager static is. So we have two tiers of pods. We have a higher tier pod that's more latency sensitive and we confine those to certain CPUs on the host. It's hyper thread aware. I think CPU manager is not, but that's the only real difference. Um otherwise all our lower effort pods run in

a shared pool and they just get wall clock time. So if you ask for two CPUs, you get two CPUs, but you don't know what CPUs you're going to be on and you might get some more cash misses. We uh do more strict storage isolation. This is something we've been rolling out. Um so there's a Cgroup option called IO max Kubernetes doesn't really have a concept for

this as far as I know uh where we we're actually setting the throughput that your pod can do to the root FS. So reminder all the overlays are on one backing disk. So all the pods are sharing this backing disk and it's got of course resource or throughput limits. Uh so this prevents uh disk noisy neighbor issues and makes sure that you know that rogue agent isn't

starving the disk. Um next stage is post create container. Uh this is really only used for things that we need the root FS to be set up for. Um in our example uh we set up an XFS quota for this. So um kubelet by default if you have a firm rule storage limits will kind of monitor your pod by polling and kill it if it sees you've

gone over your firm rule storage limits. That doesn't really look like a VM behavior and that's what our users were used to and as well it's I don't know polling kind of stinks. You know, if you can enforce it in the file system itself, it's better. Uh so we set this up per pod um and that way uh we get you know space when you run out

of space and uh you can do like a df-h and actually see the full size of your file system instead of the whole All right. So, now we're in start container. We've actually created our container. We have a functional environment set up. Um again, remember we've blocked the actual entry point to be tini, which is our exact our exact. Um so, when we trigger start container, we're

just starting tini and tini's just listening on a socket waiting for coordination. So, then we get into our post start container uh NRI hook. Um and we have to do things here because we still need our user workload to claim PID 1 because with the user workload dies, we want it to properly reap the pod. Um but we also want to now take this moment where we

haven't started our workload yet to set up our side cars. Um this is for sort of uh cross container sort of storage uh so, with setting up uh our workloads here and setting up our side cars, we have sort of this unique semantic where some of our side car processes need configuration from our user's container. Um so, we want our user workload to to be paused while

the side cars such as like our service mesh will read application config, correctly configure itself, start up, connect to the service mesh, and then we can start the workload. Amongst other things. So, uh I see we're close. Um so, once everything is ready, like all of our side cars are ready cuz they've read container configuration, uh then we can attach our storage directly into the mount name

space. Um again, sort of we want to do this so we have a unified view of the file system for all of our containers. Um as compared to stock where if you have multiple containers in a pod, they just kind of all try to start at the same And then finally, once storage and side cars are ready, we can signal to tini to tell our user workload

to start and the pod is now ready. Um final thing with this, as compared to CSI, uh we never stage volumes on the host. Um they're only bind mounted uh directly into the container's mount name space, and there's no reach high loop side effects because of this. There's no intermediate step before they're moved. Yeah, here's a recap of all the things. You should use user namespaces, enable

volatile, put resource limits on all your stuff. I think that's like the the easy you can do this at home, no problem. And then if you want to go crazier, you can extend things with NRI and OCI. Um thanks for listening. I appreciate Feedback, questions. Yeah.