CRI-O: Faster Pulls, Better Metrics, and a Future Beyond Images - Sohan Kunkerkar & Ayato Tokubi

This session discusses CRI-O, a container runtime interface specifically designed for Kubernetes, focusing on its architecture, features, and recent updates. The speakers, Sohan Goke and Irt Tokobee, explain the significance of CRI-O in managing OCI-compliant images, supporting Kubernetes updates, and its status as a graduated CNCF project. They cover several important topics including CRI list streaming, CRI stats for improved metrics collection, and additional artifact stores to enhance image management across computational nodes. The introduction of enhanced security features like configurable TLS and OpenVax vulnerability reporting is also highlighted. Furthermore, the session delves into peer-to-peer image distribution through the Spiegel project, showcasing its potential to optimize bandwidth usage in environments requiring large image transfers.

Good afternoon, everyone. Thank you all for joining us today. Uh I know it's the last day of KubeCon, and I hope you had enjoyed some interesting sessions. Uh and we are going to make this session also engaging. And we have a lot of things to cover in this presentation, so let's get started. Let me Let me introduce myself. I'm Sohan Goke I work at Red Hat as

a senior software engineer. I'm one of the CRI-O maintainers and actively involved in Signode activities. I'm also uh contributing to Queue, which is uh queuing system for batch workloads, and I'm the reviewer of that project. With me we have Irt Tokobee. He's also the co-maintainer for CRI-O and actively involved with Signode. So, here is the agenda for today's talk. We're going to quickly cover the introduction to

CRI-O for those who are new. Then we'll talk about the CRI-O updates including the Signode that we're working in the upstream side. Then towards the end we'll see how uh peer-to-peer image distribution fits into CRI-O ecosystem followed by a short demo, and towards the end we'll open it for question answers. All right. Can I get a quick show of hands how many of you know about CRI-O?

Cool. So, I'll probably quickly cover what is CRI-O. Uh it's as you see the acronym CRI, it's container runtime interface. O came from the open uh container initiative. So, it it is specifically designed for Kubernetes. It implements Kubernetes container runtime interface and supports OCI compliant images, registries, and containers. So, in a nutshell you can say CRI-O spins up the pod, creates the containers, and dumps the log

where it should be. So, in CRI-O we believe uh simplicity balanced approach where we try to make sure we get some of the features which are battle tested both in upstream Kubernetes as well as the CRI-O side. And uh CRI-O release version goes lockstep with Kubernetes ones, so you'll always see Kubernetes releases uh and the changes in the latest CRI-O release. And last but not the least,

but very important, Cryo is a CNCF graduated project. So, it's maturity depends with the wider adoption across both OpenShift and Kubernetes. So, with this I'll hand it over to Edu, who will cover some of the Cryo updates. Hello everyone. Uh as you probably already know, um Cryo talks to Kubelet to know like what containers and what pods are running on that node. And I'm going to talk

about some uh Kubernetes upstream related uh features in Cryo, and also some like Cryo uh specific uh features in my part. So, first one is uh CRI list streaming. So, in some cases, if you run uh high churn cron job uh clusters or CI/CD platforms with many short-lived you may have hit a wall where the Kubelet simply cannot list containers on uh node anymore. Not a timeout,

a complete failure. The RPC returns an error, and the Kubelet has no visibility into what's running on the node. Here's why that happens. The CRI list operations, list containers, list pod sandboxes, and their stats equivalents are unary RPCs. The runtime collects all the results, serializes them into single one protobuf message, and sends the entire response back to the Kubelet in one shot. gRPC has a hard promises

has limit, currently 16 MB, defined in the Kubelet. It becomes a problem when there are so many resources. When a no When a node hosts 10,000s or more containers or pods, the serialized message uh exceeds that limit, and the RPC fails entirely. The limit has already been raised twice from 4 MB to 8 MB to 16 MB. But, raising it again would just be kicking the can

down the road. This proposal also sees this by introducing server-side streaming RPCs, uh such as streaming containers, uh stream pulse and boxes, stream images, and a few stats related RPCs. Instead of packing everything into a single protobuf message, the runtime sends multiple messages over a stream, and each contain a chunk of items. Each method accepts the same filter parameters as its unary counterpart. To be clear about

what this does and does not solve, streaming RPCs the gRPC message size limit. It does not reduce memory usage because the kubelet currently collects all streamed items into a single list before processing them by design. So, the full result set is still held in memory. The benefit is that list operations no longer fail on high density nodes. The backwards compatibility story is clean. The kubelet attempts the

streaming RPC first. If the runtime returns unimplemented, the kubelet falls back to the unary RPC. So, you don't care about the um order of the update kubelet and the container runtime. Uh this feature is entering alpha in Kubernetes 1.16, and we are expecting CRI-O to have that uh feature in 1.16 as well. The next one is also related to a Kubernetes feature, and not more of a

Kubernetes, but yeah, it's related related to uh CRI-O and container Uh this one is CRI stats. So, this proposes collecting container and photo statistics through the container runtime interface rather than through cAdvisor. To understand why this matters, let's look at how stats collection works The kubelet relies on cAdvisor, which is is directly into the kubelet binary. cAdvisor reads cgroup files, scrapes proc, and maintains its own in-memory

data structures to track CPU, memory, file system, and network usage for every container on the node. The container runtime also tracks much of this same information independently because it needs it for its own operations. So, you end up with two components on the node, both reading the same kind of interfaces, both maintaining their own state, and occasionally disagreeing on the numbers. That's not a great architecture. The

idea behind CRI-Stats is straightforward. The kubelet asks the runtime for stats via the CRI CRI RPCs. And the runtime responds with the data it collects. This eliminates the duplication and moves stats collection closer to the component that money actually manages the containers. Where do things stand? CRI-Stats was supposed to graduate to beta in Kubernetes 1.16, but it remains in alpha. There's a reason for that. On the

container runtime side, we implemented the required metrics for graduation, but the performance testing revealed issues that are not on that are not on the runtime side. The benchmarks showed no performance improvement in kubelet even though it uses CRI-Stats. When we dug into the results, we found that the kubelet still has the cAdvisor code path running even when this feature CRI-Stats is enabled. That cAdvisor housekeeping code contributes

additional resource usage on top of the CRI-Stats. you're effectively paying the cost of both. These aren't a fundamental design problems. They are issues in the kubelet that need to be addressed before the feature can move to beta. The cAdvisor code path needs to be removed from the kubelet, and that that work is planned right now. The long term direction is clear. Stats should come from the runtime,

not from a separate monitoring component compiled into the kubelet. And this feature is purely a CRI-O feature. Uh the additional artifact stores. So, up until now, CRI-O stored all OCI artifacts in a single hardcoded path under the graph root, specifically graph root artifacts. That works fine for small scale use, but it becomes a limitation when you need to place artifacts on different storage backends or share them

across nodes. In 1.35, we added a new configuration option called additional artifact stores under the CRI-O runtime section. This lets you define a list of additional directories where CRI-O will look for OCI artifacts. These stores are read-only and are checked in order before falling back to the default location. If an artifact is found in one of the additional stores, CRI-O uses it directly without copying it into

the default path. Why does this matter? A few potential scenarios. First, machine learning workloads. Machine learning models distributed as OCI artifacts can be tens of gigabytes. You may want those on dedicated high throughput storage like an NVMe volume, a network attached file system, rather than on the same disk as your container images. With additional artifact stores, you point CRI-O at the storage and models are available without

duplication. Second, uh pre-populated caches. In environments where you control the node image, uh think immutable infrastructure or customer OS builds, you can bake artifacts into the image at a known path. Every node boots with those artifacts already available, no pull required. This is useful for air-gapped um where nodes may not have access to a registry at all. Uh, third is uh, shared storage across nodes. If multiple

nodes mount the same read-only file system, say an NFS export or cloud file store, you probably did once and every node in the cluster has access. This reduces bandwidth, speeds up pod startup, and simplifies artifact distribution. The idea of what can be achieved by this is quite similar to the persistent but it can be used in the environment where the persistent volume can be used or and

the control plane doesn't need to prepare the volume during the scheduling the pod. The implementation is straightforward. Cryo iterates through the configured stores in order, checks if the requested artifact exists, and return it it if found. So, there's no merging or overlay logic. It's a simple lookup chain. This shipped in Cryo 1.35. This is currently an experimental feature, and the behavior is subject to change in future

releases, but uh, we welcome any feedback, and the syntax and behavior may evolve based on the feedbacks. And I'm going to talk about some uh, minor security updates. The first one is configurable TLS. Um, Cryo's metrics endpoint and streaming endpoint that is uh, used for exec and port forwarding is uh, HTTP endpoint, and previously used hardcoded TLS settings. As of 1.35, you can now configure the minimum

TLS version and the set of allowed um, cipher suites. This is relevant for environments with strict compliance requirements, uh, FedRAMP, PCI DSS, or internal security policies that mandate specific TLS configurations. You can, for example, enforce TLS 1.3 only or like TLS 1.2 with cipher suites approved by your security team. This is also relevant for post-quantum cryptography PQC. As PQC cipher suites land in Go's TLS stack, operators

can enable them through configuration. The configuration is in the cryo API section TLS mean version and TLS cipher suites and applies to both the streaming server and the metric server. endpoint. The second one is OpenVax ports. We'll start providing OpenVax vulnerability reports for cryo when releasing new versions. This uses Go vulnerability check under the hood to scan the Go binary and its dependencies against the Go vulnerability

database. Then outputs the results in the OpenVax format. OpenVax is a standardized way to communicate vulnerability status. Whether a report is CV actually affects a given artifact, whether it's been fixed or whether it's uh not applicable. This is useful for supply chain security workflows if your organization tracks software composition and vulnerability status across your infrastructure. Uh both of these uh currently live improvements, they don't change cryo's

core behavior but they make it easier to operate cryo in environments with specific security and compliance requirements. Okay, so next I'm going to pass it. Thanks, Ido. Uh one last signal update before we move to peer-to-peer stuff. All right, so let's talk about memory QOS. So, I want to discuss about this problem because uh many of you uh doesn't know about this problem when it exists in

the first place. It's a gap. So, imagine a use case where you have a node with eight gigs of RAM and three pods are running. The first is the guaranteed pod which requests for two gigs and consumes one gig which is still within the control. Pod B is the burstable pod which request of one gigs and utilizes one which is still under control. And pod C is

the best effort one which doesn't request anything but ends up eating three gigs of memory. Now, when the memory pressure hits which pod you think will evict first or dies first? In the wildest imagination we think about pod C because it's the best effort But here is the deal. Like Kubernetes sends one hint to kernel for saying OOM score and kubelet says OOM score low for the

guaranteed pods and higher for the best effort one. So, when kernel tries to OOM kill a pod, it will look for that OOM score and based on that it will die it will basically kill the pod. So, in this case is best effort. But OOM kill is like the last resort for the kernel. So, before that kernel tries to reclaim for reclaiming the memory it can basically

evict the page cache or it can do node swap. So, kernel doesn't know about the hint in that case and request memory request.memory is never communicated to kernel at all because it's a scheduler hint. So, there's a possibility where kernel can't reclaim the guaranteed pod's memory because irrespective of its QoS class. So, reclaim is the problem here. It not only spikes the latency but also degrades the

entire performance. Even if your pod is not getting OOM killed, there's an issue where it might get the degraded performance and whenever you try to access it, it might go into page defaulting access. what memory QoS is doing it is actually closing the gap saying the kernel we have this memory, don't reclaim it. And that is where I'm going to the cap which we reintroduced it. And

I write now in V1 Alpha 3. So, there's a history behind this cap. It was first introduced in Kubernetes 1.22 where we used to set memory.min for all the pod memory base And the idea behind that is to bridge the gap between the scheduler and the However, we realized while calculating the throttle value, that is memory.high the calculation was a bit off because we were setting the

throttling factor as 0.8 which was more aggressive than it was required. So, we made that change, make it work. But then in 1.27, we hit another issue which was related to kernel which is live lock. So, what it does is like, you know, when kernel when container approaches memory but it's not when it reaches to memory.high, kernel will try to throttle that and it will try to

bring that down the usage bring that down. container will actually go into throttle reclaim loop where um container allocates more memory than kernel could actually reclaim it. So, this fix was actually introduced with kernel 5.9. So, before that you will definitely see that live lock issue. And that's where we again re-targeted that to 1.26 with alpha version and we changed the way we are defining the memory

protection. So, previously we used to use memory.min for all pods. Now, we are going to use memory.min only for the guaranteed one and we are going to use the QOS class level protection. So, what does that mean for everyone? So, guaranteed pod will have memory.min because at the end of the day we you request equal to limit for that. So, the entire requested memory is protected. We

don't need memory memory.hot for that. For burstable, we are using memory.low for the soft protection, which indicates that the kernel will reclaim at some point of time. But, um that is like a trade-off that we set we don't want to overcome it on um memory protection budget that we allocate. And last thing is the best effort. We don't request, so there's no protection So, here is the

Cgroup hierarchy in the observability part that I'm going to discuss. So, when we have done with the code changes, there was a uh issue that was raised. And there, uh if you see like the hierarchy, you have Qpods uh like the bin root Cgroup. Inside that, you have main pod that is guaranteed, and you have slices like the burstable and the best effort slices. So, for guaranteed

QoS class, we set the memory.min, which includes both a burstable as well as uh the guaranteed request. And somebody raised that request, why we are actually inflating uh the memory protection. So, here's the answer for that. Uh when you talk about from the kernel docs, it says when you think about the Cgroup hierarchy, you look for the Cgroup protection for your child that is dependent on parents

uh Cgroup thing. And uh you need to have that cushion in place where uh memory.min should accommodate both uh guaranteed and the burstable request under that. Otherwise, it will squeeze in some of the memory from memory from guaranteed pod to the burstable one in order to uh make sure that it will have enough coverage for the Cgroup protection for the children's. So, we addressed that uh in

alpha. And as I said for best effort, there's no protection. We also added a bunch of Prometheus metrics uh to understand how users will use hard and the soft So, while doing this, we realized we are mostly talking about memory protection. So, what happens for the memory memory throttling, you know? So, in kernel decides that with memory events the usage goes beyond memory.hot and it is memory.max,

we can actually see the counter there. that feature is still not there and we are thinking about creating a different cap for release 1.37, which will not only solve this particular feature, but also has some other use cases including the eviction policies. while having this feature development, we also encountered use case where the unified field under update container resources was not updated with cryo. So, what it

does, it basically sends C group away to fill to runtime. So, in this case would be memory.low and high. And um So, think about this in-place pod resize, right? When you do that at the uh your memory request changes. Kubelet recomputes the protection, but it was never conveyed to cryo because we are not setting the unified field. So, this fix was addressed in 1.36 and hopefully you'll

get that sooner. All right. So, yeah, so enough about memory. Let's talk about something which is interesting. Let's talk about peer-to-peer image Um you can see on the slide the traditional image is like 200 MB. As you do some complex stuff related to AI, the images are getting bigger and bulkier. PyTorch is 8 gigs. Then LLM model is roughly between 20 to 50 gigs. Now, you can

do the math here. You know, you have 100 node GPU clusters and you're pulling 20 gigs of model. That is like close to 2 terabytes of egress. That is for deployment and for every rollout of the model. So, um you can also question me like, "Okay, recently we uh GA'd OCI OCI image volume feature with 1.36 it is GA'd a feature where you can mount OCI artifacts

as a read-only volumes in in your pods. But, that gave actually solves you how to use use case and not how how to distribute images across different nodes. You will still end up hitting the image registry across different nodes to pull that image. And that is where Spiegel comes into picture. Spiegel is a stateless local OCI cluster image mirror registry that allows you to replicate the image

across uh different nodes with the help of the libp2p protocol uh to do peer-to-peer distribution. So, how it works it it's it's a daemon set that runs across different nodes. And it's so there's no central repo. It's all decentralized. So, if the pod dies you need to just spin it up and then again they calculate the layers and kind of distribute across different nodes. So, this might

not be a good kind of a high availability registry, but it's certainly a good um use case for solving some of the AI issues that we have. uh talking about Spiegel Spiegel what it does it like when you pull the image Spiegel intercepts that using its distributed hash table. It stores the layers. And when another node doesn't have the it's pull like it's image pull gets intercepted

by Spiegel. It looks for the respective blobs. It fetches from local from local network rather than going to egress and that's where we are saving 90% of the bandwidth. Now coming back to integrating Spiegel and CRIO. Spiegel was predominantly a develop looking into consideration the container this architecture. Probably for CRIO we had to do certain So if you think like for CRIO we are using container storage

as the library to store image layers. And we don't store compressed layers on the layer system. So we introduce a new layer called block cache. So we'll keep the same flow. It's just that the reader will have split into the two things. The first would be the normal flow where CRIO will pull the image, put into the overlay system, will have the overlay layers and second would

be a block cache layer. It will store all the compressed layers that we have. And it will also track the meta metadata.json which will have the blobs and from where it's actually pulling from. This work is still work in progress. There are a bunch of things we are discussing but I want to just kind of give you this overview like how it looks like. I'll show you

the demo as well. Yeah. So I'm going to use the script to make it faster. Just one one disclaimer I'm deploying uh Spiegel as a binary and not using helm chart. So yeah. have the cluster configuration. Where I'm going to use the local binary that we are with all the changes that I discussed before. I'll go ahead and create a cluster with that config. I think it

might take some time. I'll just fast forward. So, yeah, the cluster is up and running. Let's export the cube config. I think the pod is running. All right, so we have all the nodes that we need. Now, here is the thing, like we are defining the following thing under cryo config. We'll be setting the image content cache directory. So, that is like a user specific knob that

we had given it to users. So, they can define where they want to put their cache blocks. And there's one more option to auto reload registry. So, if the registry is stale, cryo can reload that and you need to also define the time through which it will reload the registry and will pull from the latest And we also, as part of the cryo's architecture, we also need

to provide the mirror registry configuration. So, I set it to localhost 500. And I'm going to copy those two files the worker nodes, worker zero and worker one. >> Copy and paste on. So, I'll go ahead and restart cryo to pick those configuration. I will also do the similar for block cache config on the worker one. Now, I'm going to copy the Spiegel binary that I built

locally for to both the workers. And I'm going to get the IP for worker zero. Because that is going to be that is going to be my seed um node through which I'm going to distribute amongst other workers. So, I'm going to go ahead and start Spiegel. There are a bunch of uh flags that you need to set to make it work. I'm going to do the

similar for worker, but this time I'm going to connect that to worker zero. So, now P2P network is up. Now, I'm going to pull an image from It's going to be from the docker.io. So, we have that image. Let's see if we see the layers. So, we can see the content is there. And layers are also present. Now, I'm going to pull the image from worker one.

But that image will be pulled from the local network and not from the egress. So, if you could see, it already pulled the And let's see if um it is done via local network or not. So, I'm checking the journal D logs. Yeah. You can see in the logs that it used local host 500 to pull that image. All right. So, um I want to leave you

with one food of food for thought. Uh in Kubernetes nowadays, um there's a hot topic about aware scheduling. uh we were just thinking, what if scheduler knows about identifies the actual node where the image is already exist. So, you can think about an example where you have a pod with uh 45 gigs of model that you want to spin it up, but you already identify that pod

you already identify the pod before you uh scheduler could actually put that on a particular node. Um we can think of it as a like, you know, uh you have 100 uh 100 deployments. Each is pull each pulling the same image 100 times. With P2P, without P2P, with P2P, one image pull and 99 peer-to-peer. But with uh P2P aware scheduling, you only pull once. And uh but

at the right time and at the right right node, if the disk size is full, you still need uh you are still uh reducing the count for P2P thing like uh instead of doing 99 times you can do it only once and you can find out the node and probably place your pod there. We are still evaluating different uh mechanisms. Uh there's one called you can write

a scheduler plugin, but we need to understand um uh how it what it takes for the GPU nodes and the resource constraint. We're also thinking about uh because Spiegel has locality API through which you understand the blob and layers. So, probably um and there's also a plugin uh with kube scheduler called image locality. We might extend that. We're still discussing uh some of the options. So, feel

free to uh kind of discuss that and we're going to add that in the future road map as And if you have any questions, reach out to us on GitHub and Slack. We're always And with this here's the QR code. Um give us a feedback. Let us know what you want to hear from Priorside and yeah, thank you all.