Beyond Image Pull-Time: Ensuring Runtime Integrit... Toddy Mladenov, Flora Taagen & Dallas Delaney

This talk covers the Notary Project's approach to improving container image security through image layer signing. The speakers, Flora and Dallas, discuss the existing signing process where images are signed at the manifest level, allowing for potential post-pull tampering of individual image layers. They highlight the need for continuous protection and integrity verification at both runtime and rest, particularly in multi-tenant environments. The proposed solution involves extending the Notary Project to implement per-layer image signing using DM Verity, a Linux kernel feature providing block-level read-only protection. The session includes a demonstration of the signing process, runtime verification, and the implications of their enhanced security measures against potential attacks.

Uh I am Tony. I'm one of the maintainers of notary project and today we have a interesting topic uh that will be delivered by Flora and Dallas. Flora and Dallas are new contributors to the project and they'll talk about uh uh image layer signing. So looking forward to that. Uh very quickly for folks who are not familiar with notary project what notary project is it's um a

project in CNCF which is for uh supply chain security. Uh we have two sub kind of projects or tools under notary project. One is the notation CLI that is used for signing OCI artifacts any kind of artifacts like container images sbombs helm charts. Uh you can use the CLI. There are also libraries uh that you can use if you need to do more custom integration into your

tooling. And the second project is called Ratify which is external data provider for uh OPAI gatekeeper and can be used for admission control. Uh you can verify images that are signed with uh notation CLI and uh also you can use them to verify images that are signed with uh cosign or six door infrastructure. uh notary project is used by uh various companies. Uh we are from Microsoft

but also AWS is using it. Alibaba cloud uh you can see a bunch of logos there. So uh without any further ado I'll pass it to Flora and Dallas so they can go to more interesting part of the presentation. Thank you. >> Perfect. Thanks Toddy. Um so to begin we want to walk through what the existing signing process looks like for signing containers with the notary project

today. Um it all starts with the image build phase. So this is when you are building a container image. Maybe you're using tools like docker or github actions. And at this stage your container image has not been signed. Then we enter the image signing phase. This is when the notary project actually steps in to sign your image manifest digest. Uh this creates a cryptographic signature that will

bind the signer's identity to the image manifest itself. Then that signature gets pushed. It gets stored as a separate OCI referral artifact um in your container registry alongside the container image. Um, and then finally we enter the pull time verification step. So when someone wants to pull their image down to the host, the signature gets verified against the image manifest digest to confirm that the image hasn't

been modified, it hasn't been tampered with, um, and it's coming from a trusted signer. So you might have noticed that the notary project today is signing at the manifest level and that signature is getting verified at poll time. So basically during distribution ensuring that that container is originating from a trusted signer. However, that protection and that verification stops at pull time today. So once an image gets

pulled, its individual image layers are downloaded. they persist on disk and they get cached for reuse and today those cached layers are not continuously reverified. Um and this is why manifest signing by itself can fall short because there's this window where post-pole tampering of individual image layers could occur without detection. And so this introduces a security risk where a attacker or a malicious actor, you know, if

given or gaining uh access to the host could be able to modify one of those cached image layers. Um, and they could do so after that initial pull because verification is only happening at pull time and those changes could potentially go unnoticed. And this problem intensifies in multi-tenant environments where layer sharing across multiple containers and applications is common. Um so in short manifest signing by itself isn't

able to provide continuous uh protection and integrity at runtime and at rest for individual container image layers. Um and so now we're going to walk through a example attack scenario. Um you know there's multiple ways that an attacker could exploit this gap. Um but this example kind of illustrates our core problem. So we start with the initial state. So this is when a container image has been

signed. The image manifest signature is verified using notation. And this confirms that our container image has come from a trusted signer and it hasn't been modified on distribution. Um, next is the compromise stage. So in this hypothetical scenario, let's say that an attacker has gained right access to the host. Then we move to the injection stage. So after the image is pulled those individual container image layers

um are persist on disk and are cached on disk and if an attacker has host level access they can modify one of those cacheed layers. So let's say in this example scenario that they inject a malicious binary into one of those cache layers. Well next is the bypass stage. So when the container you know runs or restarts the modified layer will get mounted directly from disk um

because no additional verification um is occurring beyond the verification at pull time. And finally that reaches execution where a malicious binary is allowed to execute even though that image had been previously authenticated. So now we're going to show just a very short demo of that potential attack scenario. Um basically walking through the steps that that we just discussed. Um we already have like a running terminal instance

here. Um and the first thing that we're going to show is that we do have a signed container image that's been successfully verified using the notary project. that image is going to get pulled down onto the host and we'll see that in addition to the image those individual image layers also get pulled um and they get cached on on disk and then in this hypothetical scenario let's

say you know someone has write access and they modify one of the binaries in a cache layer the container image is run and since the image hasn't been repulled, it'll reuse one of those tampered layers. The kernel will mount that tampered directly from disk. Um, and ultimately this results in tampered code executing even though the container image was previously authenticated during distribution. So before we dive too

deep into our solution, there are a couple of terms that'll be relevant that I just want to take a take a moment to explain in detail. The first one that we've already talked a bit about is OCI image layer. So OCI container images are composed of multiple layers and each layer essentially represents a file system change set. So things like adding files or modifying binaries or installing

packages for example. Um and each layer gets stored as a blob in a OCI compliant registry. Um and then gets downloaded individually when an image gets pulled. Um each layer is cached on disk and often reused across multiple images and multiple containers. Um, and then finally, each layer is identified by a content address digest. So you can see an example with the Shaw 256 hashes in the

in the container image diagram there. Um, and those are computed from the actual components of each layer. And then the other term that'll be relevant is device mapper varity, otherwise known as DM verarity. Uh, DMver is actually a Linux kernel feature. um and it provides block level read only protection for data that is stored on disk. Um its job essentially is to ensure that data being read

from disk has not been modified and has not been tampered with. Um conceptually DMver works uh with a Merkel tree of cryptographic hashes over the data. Um and you see an illustration of this on the slide. Um basically a individual contained container image layer gets partitioned into fixedsiz data blocks. Uh each of those blocks gets hashed and then those hashes get recursively hashed um till there is

like a a single value at the top. Um and that single value is called the DMver root hash. Um and importantly the DMver root hash uniquely represents all of the contents in that image layer. So if any of the data in the layer gets modified even by a single bit that DMverity root hash will also change accordingly to reflect that modification. Um so how does DMverity relate

to and enable runtime verification? Um well at runtime when a container executes the kernel will actually check every read operation against the expected DM root hash from that container image layer. If the expected and the derived values match, that operation is allowed. Um, but if they differ, the read is actually blocked. Uh, because that data can't be verified against the trusted hash. Um, so putting OCI image

layers and DM and the notary project together is how we arrive at our solution. Um so to mitigate these postpoletime attack vectors we are extending the notary project with per layer image signing uh backed by DM verarity to provide continuous integrity uh both at runtime and at rest. And this solution has two parts. There's a buildtime component and a runtime component. Um and I'll start by walking

us through the buildtime implementation. So we began with a standard OCI container image layer um or container image comprised of multiple individual layers and for each OCI image layer we will generate a corresponding EFS image. Um EOS EFS stands for enhanced readonly file system. Uh what this step does is it converts a OCI image layer from a sequential stream of data to a ERS image where data

is stored as addressable blocks to be used with DMverity. Uh next as we saw in the previous slide we will generate those DMverity Merkel trees for every image resulting in a unique DMver root hash for each image. Um and then each of those hashes will get signed using the signer's private key. Um the resulting signature as well as the signer certificate are packaged into a PKCS7 envelope.

Uh PKCS7 is a kind of data structure or format that the Linux kernel um can actually consume um for verification. Um and then the certificate chains back to a trusted certificate authority already present in the kernel keyring. Um and then that DMver root hash and its corresponding PKCS7 envelope um get stored um as OCI registry artifacts. They get pushed to an OCI registry attached to the image

as well as the existing U manifest signature. Um and ultimately what we see here is that our signing scope has increased. So when uh previously we were signing only at the image manifest layer with these buildtime changes, we've increased the scope to sign not just at the image manifest layer but also for individual container images as well. Um and I'm going to pass it off to Dallas

now uh to walk us through a demo. Um, so we're going to look at some of the build steps that Flora just mentioned. So in this next demo, we'll see um we're going to run the notation sign command with our new DM varity argument. So we're going to see um us creating those EFS images. We're going to push it to a container registry and we'll also actually

inspect that content to see what we actually pushed. So we'll run this command and we'll probably see a lot of output and then we're going to use ORAS to inspect what the actual result of this is. So Orus in the next step should show us that we've actually have a PKCS7 artifact along with a regular manifest artifact attached to that container image in our registry. So you

can see those are the two artifacts and we'll inspect the first one now. So for the manifest artifact, we see there is one layer and we have a digest there, but we also have a signature which is the thumbrint entry there um in the bottom. The next artifact we're inspecting is this new one that shows the root hash and the PKCS7 envelope. So we're going to see

an array of And in this image, we see three different um layers for this container image. So we'll see a root hash, a PKCS7 envelope, and a digest. And that's going to exist for every layer that we have in this particular So if you go to any container registry, this is what you would essentially see. So we've looked at the buildtime solution. So let's look at some

of the runtime. So the first thing we're going to do is we're going to take these new artifacts and enable a container runtime like containerd to not only pull the container image but also the new OCI So, we're going to enhance or extend the ERO FS um containerd snapshot to actually detect the new artifacts. The next thing we'll do is have the container runtime send this new

PKCS7 envelope to the kernel. The PKCS7 envelope is basically a data structure where you have a certificate and a signature. So we'll pass that data structure to the kernel and the kernel knows how to decode that and it will take the certificate from that PKCS7 envelope and use a certificate authority from the kernel keyring and validate that the certificate is actually valid and coming from a trusted

source. Once it knows that the certificate changed to a certificate authority in the colonel key ring, it's then going to look at the signature in that envelope. It'll use the public key in the certificate to make sure that the root hash signature is actually what is expected. So there are two different layers of authentication happening in this process. Once uh the kernel verifies that the actual um

certificate is valid and that the signature is authentic, the container runtime will then go ahead and set up DMverity. And again, um the container runtime is going to pass the root hash to the DM for each of those container image And it's also going to make sure that um every single read when we're running our container image, every read for a particular block in that container image

um matches the expected root hash for that same layer. So I'm oversimplifying this a bit, but that's the high level how this works. So if we look at the table below, you can see the current approach versus this new um validation approach. So we've extended the signature scope from just the image manifest to the image manifest in all of the individual image layers. We've extended the verification

timing from pull time to also pull time and And we've extended the integrity coverage from the digest to the digest in every single layer in that container So the next demo we'll show is just um I'll just be in a terminal. Um I'm going to run the regular notation verify commands and we'll again modify one of our container image blocks. And now that we have the DM

verarity solution, um we should see that we're not able to modify anything that's cached on our container So the first thing we'll do is before we actually pull a container image, let's run notation verify and make sure that it's actually coming from an authentic source. Okay, this the container image is actually um from some somewhere we expect. Let's go ahead and pull it. All right, we've pulled

the container image. Our container runtime is using the IO FS um snapshot. So, let's inspect some of the layers. All right, we have three image layers in our container image. So, now let's go ahead and write a bite, just a random bite into the first block of those FS image layers. So, now let's go ahead and try to run the image with that modified content. All right,

we see a lot of debug logs. Um, we see some journal control logs for DM verarity and you can see it's it's showing that we actually detected that tampering and that second log from the bottom for DM shows that DMverity detected that block zero is corrupted. So we what we did here is we pulled the container image on the container host. We modified that image after we've

already pulled it down and then we try and run the image after verifying with notation verify but DMverity actually saw that it was tampered with on the container host and refuses to actually run that image. So let's just look at the endto-end flow now. The top part shows the build steps and the bottom part shows kind of the runtime steps here. So the first thing we'll do

is just build our container image like we regularly do. Um the next thing we'll do is we'll sign it. So, we'll not only sign the image manifest, but we'll go ahead and sign all of the layers um and generate PKCS7 data structures to insert our certificates and our signatures in those envelopes. Next, we'll go ahead and package all of that metadata into OCI refer artifacts. So instead

of just pushing our image, we're now pushing um another artifact attached to that image. And our container runtimes are configured to actually detect these new artifacts that are attached now. So we'll go ahead and publish that to any registry we want. And then when we start on the bottom side, we'll see that we run our first verify command. So this is something that you can already do.

We'll verify the image manifest in our container registry before we pull any Once that's successful and we know we're pulling from an authentic source, we'll go ahead and pull the image. In the second verify step, we'll actually have our container runtime um unpack that PKCS7 data structure check the certificates and check the signatures and make sure everything is good. Once that's fine, we'll configure DM verarity and

actually run the container So what that gives us is verification before we pull, verification before we run and code integrity while we run. So let's look at one more demo that shows just the good case where we configure everything correctly and we're not tampering with anything just to see the differences. So the first thing it we'll do again is run our notation verify against our image digest.

All right, it looks like it's coming from a trusted source. We'll go ahead and pull this image. All right, we pull the image. Now we have our ER FS um blobs cached on our container host. We're inspecting the layers. So those are all of the container image layers that are cached on the host. We're not going to do any modifications, but we'll just go ahead and run

the container. All right, you can see it's running now and it has DMver protection. So even if you know after we start the container and we modify something on the cache on disk, it's still not going to run anything that gets tampered with. So once you start the container, um there's no way to tamper with any of the data on it. So just tying this in um

tying everything together, what are the overall benefits here? Well, we have kernel level protection. So we're not only relying on user space anymore. DMverity is a feature in the Linux kernel. So this is something that is being extended and we have continuous DM runtime enforcement. We have tamper prevention. So instead of having this gap where you pull a container image and then you run it, you you're

not able to modify anything from that container image while it's sitting there on your container host. There's low overhead for this approach. DMver is a pretty popular tool. I believe it's used in Chrome OS and Android. So it all it's already used in billions of devices. And it only hashes data blocks from your image that are actually being read from when you're running your container image and

nothing else. And it'll also cache verified data blocks so you're not rerunning the same validation every single time you run some kind of process on your container. And containers can't opt out. So once you configure this and you run your container, there's no going back. So you can't if so for some reason something gets compromised there's no way to disable the embarr on on that running container

anymore. So with this approach we've essentially reduced the attack surface um by adding just more levels of integrity. So we have it you know at the signature level at the certificate level and at pull time. So that's pretty much um all the slides that we had. We have some QR codes here that just show some of the approaches, some of the implementation details and some ways to

contact notary and contribute. Thanks. Thank you very much. Thank you very much everyone. If you have any questions, we have couple of minutes. So there are microphones here or you can approach us at the podium and we can talk about that. But uh feel free. >> Hi, thank you for the presentation. Um I think this is a very helpful thing. Um so what you presented happens below

the container runtime interface, right? Um so how do do I plug this into my Kubernetes uh cluster? Does it rely on a specific runtime or does it work with any runtime? >> So this this is going to use the EOFS um um snapshot. So you'll take your containerd config and configure efs >> I believe. Uh I'm not a containerd maintainer. Um but I believe the eofs snapshot

is going to be available in the next release. >> Mhm. And this particular DMver feature, um, there's a PR out. It's almost fully approved. I think we need one more approval, but it should be available in one of the upcoming containerd releases. >> Awesome. Thank you. >> Hi. Uh, thank you for the presentation. Uh, in the demo, um, the images are stored, um, like owned by root.

So I guess like this technology is supposed to be used in the like rootless environment. Is that correct or like is there any attack scenario? >> Um this should be able to be used in any >> Yeah. So I mean like if the attacker can change the image files then it should have like root root access in that host. It can have root access to the o

to the host. You can modify the files but it won't be able to um like misconfigure DMverity once you already um start the container. Additionally, once you have your snapshot, it's pulling from a registry and it's checking your certificate that you've pushed to that registry and comparing it with the kernel key rings. So you're going to you're going to basically authenticate what you've pulled against the kernel

keyring certificate or certificate authority. >> Okay. Thank you. Thank you.