Enforce VIP Access Only Through Node Attestation - Alice Frosi & Jakob Naucke, Red Hat

This talk focuses on node attestation in the context of confidential computing, presented by Alicia Prosy and Yako Naucke from Red Hat. The speakers define confidential computing and its importance in protecting data in use, addressing a previously unguarded area in data security. They emphasize the need for secure environments, particularly in trusted execution clusters, which rely on specific technologies like AMD SEV-SNP, Intel TDX, or ARM. Remote attestation is discussed as a method for verifying that a machine operates within a secure environment by using trusted hardware and attestation tokens. The presentation also covers practical implementation details involving the ignition boot configuration, clevis for disk encryption, and key management using the Trusty attestation server. Lastly, a demo illustrates the functionality and security advantages of deploying confidential VMs on Kubernetes, showcasing the resilience against memory dumping attacks when using confidential computing.

Okay, so hi everyone. My name is Alicia Prosy and with me there is Yako Naucke. We are both from Red Hat and today we're going to talk about um uh node attestation in confidential computing. So, uh let's imagine your cluster as an exclusive VIP uh club uh where you have inside your more critical and important information and data and you want to uh guarantee that only allowed

entity has access to it. So, outside there are all um malicious uh embedded actor like malware, malicious uh cloud administrator, and in more general entity that should not have allowed to access your data and try to sneak in. So, as the bouncer control passes an ID, uh our proxy is responsible to um access to your data. So, during this presentation, we are going to walk you through

how we protect the system, how we uh um manage the guest list and the VIP list, and uh no other machine can join the cluster uh and keeping at the same time the party secure and not slowing down. why and how this is possible? So, this is possible thanks to confidential computing and here you have the definition from the confidential computing consortium. So, there are exa- there

are two parts of this definition that are important. So, first of all, confidential computing help us to um protect the data that are in use. So, this was uh a hole uh we were missing this in security. Uh traditionally, we we we have well-established technique to protect data at rest and data in transit, but we were missing the data that were compute used by the CPU during

the computation. So, this is the first part and benefit of confidential computing. The second is that it guarantee that your environment where your code is running is actually secure and confidential. So, this of course opens the door of all the use cases that deals with regulated data. And in more general data that are critical like health insurance data or in in general anything you want to keep

secure. in our presentation, we are going to focus especially on confidential clusters. We call we call them trusted execution cluster. So, the boundary from what is inside the confidential environment is the cluster itself and outside is the rest like for example the cloud providers. So, this implied that each node in the cluster need to run on a confidential environment. So, it needs to use either AMD SEV-SNP

or Intel TDX or ARM. So, we put our trust in the cluster administrator but not in the cloud provider. So, use case for this is for example migration to public cloud but keeping at the same time the data But it's also valid for on-premise use cases like for example if you want to do data segregation among your organization, this is also a valid use case. And last

but not least, it gives you proof that the environment where you are running is confidential. how do we prove the environment is confidential? So, this is through a process that is called attestation, and we are going to call it remote attestation because there is a third party that is going to validate if the environment is confidential. So, we start from the machine that are some trusted hardware

that we uh that has a key. This key is going to sign something called quote that contains a list of measurement in general is some hashes. Um so, then the machine send the quote a remote attestation server. Uh first, the attestation server need to validate that the quote is coming from a trusted machine by authenticating and validating the key. and then it compares the measurement that are

included into the quote with some reference values. So, if all the checks are green, when it's sent back is is a so-called attestation token. This is also signed by the key of the trusted attestation server, and then the machine can use this token in order to release an and prove that he has his passes to enter into the party. how do we do this in practice? Um

the hardware that we use into the machine is the TPM that is base for trusted platform modules. Probably many of you are already familiar with that. And uh this is the trusted hardware that create and sign the quote. the remote attestation server is implemented by Trusty. This is a software that's included in confidential containers and already part of CNCF. And then we have a set of policies

uh that includes the attestation policy that defines uh when an attestation is successful. And then a resource policy that basically defines when a resource is unlocked. So, how does it looks like uh very quickly? So, the machine starts when it boots, uh it sends the portal uh to uh the trusty server. It contains, of course, the measurement. Then the remote attestation compares those with the database of

reference values. If this is positive, send back attestation token. And then the machine can use this attestation token in order to release the secret. And in our case, this is going to be the Lux key that is going to be used uh in order to re-encrypt the root disk. So, this is basically the last step when it gets the secret. Uh so, there are various challenges. The

entire process is very complex, and we want to automate this. Um there is, for example, the coordination uh of multiple nodes. Um there is also the calculation of the reference values. When also um a machine gets updated, you need also to update the reference values as well. And here is where it comes into place our operator that we call trusted execution cluster operator and certain machine component

that uh Yakumo is going to explain us. Thank you, Alice. Um yeah, so Alice broke down some of the uh concepts uh that we use for testing uh nodes in Um and I'd like to get a bit into the the practice uh of what we use um for the nodes, um for the machine uh that joins a cluster, um these are mostly three things. Um it's the

ignition boot configuration system. Um, it's a clevis pin for trusty. Uh, this this attestation and the trusty attester to to talk to trusty. So, um, some brief notes about these three. Uh, ignition um, a boot configuration system uh, introduced by by CoreOS. Um, and what it lets you do is it lets you have a con- have a config and merge that with something else that you fetch

from a resource. For example, this one fetches uh, from a registration server. This is a part of trusted execution clusters. Um, and it fetches um, an extra ignition config to be merged. and the certificate and everything. Um, and that config is going to specify uh, disk encryption. Um, and that specification is one for clevis. Um, and specifically for a trusty pin of clevis. And that's this configuration

for the root volume. clevis um, uh, is a project to basically encrypt a disk with anything that isn't a password or a key file, but something else. You can have uh, like a tang server or uh, local TPM values. Um, and for trusted execution clusters, um, we created a novel clevis pin um, for trusty. For this for this attestation system. Um, and then a config provided by

this registration server of trusted execution clusters would say get a key from this URL, the certificate, get it at this path and use this in it data. In it data is is a concept of Trust that you can use to give your to give your notes an identity basically. We achieve this by by extending a PCR and that ensures that a node that passes attestation per se

cannot pose as another node later because the because the PCR was modified immutably and then potentially this config might also contain attestation key registration if the platform uh requires that. So, um this this clevis pin within uh encrypt the the root volume using this configuration and to retrieve the key that would tie into Trusty Attester um attestation system that I'll get into in more detail in a

minute uh introduced by confidential containers where each pod is a confidential VM here. Every node is a confidential VM. but um has has a reuse potential and we do uh use it to uh to attest this node basically again talk to this key brokering service get this path uh for for a key and set this init data. So, on the uh Trusty side um on the on

the server side um Trusty fundamentally is is a key broker service. That's where you get your key, uh an attestation service where you verify what you are, and uh reference values uh reference values to uh against what you what you'd expect uh from a from a node uh that you trust um and the attestation service uh utilizes an attestation policy. Um, for example, this one says, "Do

not trust executables in the in the trust vector, uh, except if the reference value for PCR 4, uh, is what you would expect, uh, from the reference values, um, for, um, would measure bootloader, but you could also or you should also be measuring 14, uh, for shim or especially 11, uh, for UKI, but this is just an example. Um, what what creates additional difficulty for these reference

values is, um, that if you update the node a kernel an update to kernel and bootloader changes the reference values for two boots, um, for the boot thereafter and the the one boot thereafter because they they update, uh, separately and this this this changes your TPM values, um, so these these values must be must be managed, um, but, uh, for registering, for registering a node, um, it'll

it'll also talk, uh, to the key broker service and that has a has a resource policy to for each node that passes attestation also, define what what secrets, um, are allowed. For example, this one says, "Status must be affirming, uh, which means that attestation passed basically, and the evidence must support, uh, the claim that the UUID, um, of the node, uh, that was set earlier, um, matches

the the resource path." So, this this UUID will be, um, stored in in PCR 8, uh, PCR bank number 8, um creating a reversible um mutation, and that's how a node can't pose as another node later. Um but then for for a node to uh to register, um when a new node um is to be attested, uh it'll have this this ignition config that I showed earlier

to register with a registration server. It's a component of that'll create a resource um a machine resource uh for the operator, um and then that operator will set the reference values per per UUID that was that was registered uh to trusty and uh set the secret, um and registration server will return an ignition config that includes this this Clevis config. And from here on out, it's the

same if it's a new node to be attested or an existing one. Um ignition will execute the Clevis pin with this configuration. That'll extend run trusty attester, um and trusty attester will attest uh to trusty, and that'll return a LUKS key uh to encrypt the disk with. these reference values, um they not only depend on the UUID you set, but also on the on the guest OS,

on its on its components like like kernel or or UKI ideally. Um and like I said, they must be computed for all combinations of boot loader and kernel that you that you might expect. So, the cluster admin may approve an image um using a custom resource um that we that we define call uh approved image, um and that'll use the compute PCRs uh project that we also

introduced um to compute the expected PCRs uh for this approved image, and that'll that'll store in Trusty um and that'll be on a on trusted node. And this this approved image um for trusted execution clusters is a boot C image. Um so that's a bootable container image um that's not only includes applications, but includes a a kernel and an in its system. and um that's uh that's

what uh TEC relies on. Um and another detail I want to mention is that uh the operator and the components I uh presented are all uh written in Rust uh using uh using Kubevirt for Kubernetes interaction. Um but uh with that said, uh Aliki is going to provide a little demo. So, now we're going to see uh confidential um computing in in action. So, uh in order

to use uh and create a confidential VM, we're going to use Kubevirt. Uh so, for those of you uh that are not familiar with Kubevirt, uh it basically launch guest the QEMU guest in a pod. So, here you can see already uh the definition of uh virtual machine. And the important part is uh the launch of security that enable SMP. Uh it's still a work in progress.

So, uh the secure boot needs to be disabled, unfortunately, but it's it's going to be supported in the future. And then we have a TPM device. So, I need to squeeze the demo in 5 minutes. So, we're going to um skip certain things, but uh if you want to have uh a deeper look, I uploaded the video in the in the page of the presentation. So, please

take a look there if you are more interested in looking each each each steps. So, we are going to create the virtual machine. Uh this is going to be starting in a pod. And on the left you can see the component that are deployed by the our operator. Important are the registration server that we talked early, the operator itself, and the trusted deployment. So, we can monitor

the booting process on the right with a serial console. And at a certain point we will see that we are trying to create the Lux volume disk. And this is the point in time where we are going to where the attestation will take place. yeah, you can see already that there is a create Lux. And if we monitor the trusted Lux, we will see at a certain

point all the Lux of the attestation. um this is going to be triggered by the key we spin that Jacob was talking about. And important piece is the last one where we got the secret from the path that we have showed previously. So, in this way the virtual machine can get the secret from the trusted attester trusted server and encrypt the root disk. And then it's going

basically to boot um as a standard VM. can speed up this a little. So, we can see now that we have been able to boot a Fedora CoreOS VM. this has uh is a real SEV-SNP machine. We can see that from the kernel messages. It does also a Kubernetes node inside. It is going to be useful for the demo, the second part of the demo. it has

also an encrypted disk with Lux and this is was encrypted using the key that we retrieve from the trusted server. If you're interested in the attestation policy and the resource policy and the demo also show them, but I will skip for timing. So, in the second part of the demo, we are going to see why confidential computing is so important. So, we are going to deploy two

VMs. A regular one and a trusted one. So, we can see for example from the demo VM that's on the left that it's not an SEV SMB machine. It's just a regular VM. Also in this case, we have Kubernetes installed on this VM. It's exactly as the confidential VM. And we want to try to deploy an application. It's a very simple and stupid application written in Python,

but it's going basically to print on standard output some string that we are going to provide. Uh so, we can simply deploy this. There are YAMLs in in GitHub repo. So, if you're interested in replicate this, it's all public. So, we can see simply that the app is up and running right now. We need to expose it on the host in order to do a curl a

post request with the super secret password. This is going to be the secret that we want to try to leak. Um so, we can see that the server um print that on standard output. so, now we are going to create a memory dump with mem with QEMU feature. Uh I'm going to speed up this a little bit. Uh but basically, this create a dump of the memory

of the guest that we can scan in order to try to leak the secret. Uh so, I created this uh uh Kubernetes job in order to scan the dump of the that it's uh located on a PVC. And then we can see, for example, uh that if we monitor the logs of this pod, at a certain point, we're going to be able to see uh the the

password uh the string that we have requested with the curl post post request. this is why confidential computing is so important because for a standard VM, um the the memory is basically in clear. So, we are going basically to replicate the same on a confidential VM. So, we can see again from the serial console uh that this is a SEV-SNP machine. It's the one that we have

attested previously. And again, this is uh the Kubernetes node. We're going to uh deploy exactly the same application. And uh we're going to do again the same curl post request. But the important thing is now uh that we create a memory dump from from this And we try to scan it as we did it before. And in this case, we won't be able to find anything because

the memory is encrypted so your your string basically your secret is is protected so nothing was found. Okay. this was a quick demo how on Cuba confidential VM and on confidential computing in general. What to be done next? We are currently focusing on Azure but plan to support various cloud provider like GCP and AWS. Also the support of Cuba right now we are using it just for

a development. But it's of course one of the use cases is if you have bare metal host. And then we need to protect we need to move to UK. I'm just moving very quickly if somebody is interested in one of the points please ask on the Q&A. So right now we have just shown that we can protect the Lux key for encrypted the root disk but something

that we want also to protect is the ignition configuration and in general the cube config. So in this way we can act as a secret and a node will be allowed to join the cluster only if pass We basically want to protect the CSR with attestation while the node is joining. So we need also to solve the bootstrapping problem. Right now we are using basically a external

cluster in order to attest the the nodes but we have been focusing only on worker nodes and the next next phase we are going to attest the bootstrap and the control plane as well. And last but not least, we would like also to move uh the operator in cluster. So, uh if there is something that you need to retain from this presentation, it's the important role of

the operator is the orchestrator of the confidential Uh it deployed uh the the secure infrastructure like trusted security policy. And it also maintain uh the reference values and it deals also with updates. So, here you have some reference also with the demo. Um if you want to dig into certain details. And that was our presentation. Uh please give feedback uh with the QR code. And now there

is the Q&A. Does anyone have any question for us? Uh there are some mics here. to my understanding, Trusty is like a bank server but then uh without instead of sending just a regular key, you send your measurements and you get back your Lux uh decryption decryption key for your Lux volume, right? Um so, I had a question about that. Um I thought you said the measurements

were the so, the PCR banks have all those hashes and those were signed before sending Yes. to Trusty. But how is that how is that system able to sign it I don't see where the key is stored on the machine to actually do a signature of the hash. >> Is the signature of the Is the key of the TPM. Is it Is it just the hash that

is stored in PCR or is it Is that signed before sending to trust It's signed before sending to trustee. And how do you store the key that do to be able to do the signature? It's on the TPM the key. And how do you get it there? So there is a public key that the trustee can get from the machine and is going to validate that has

been signed with the private key of that TPM. Right, but the public key needs to be placed on the TPM first before you can measure. Okay. have you insight in how that is stored? I'm sorry. Sorry, can you can you repeat the question? So so So you have just the hash is not signed, right? Just a hash on the PCR isn't signed. No, the quote is signed

entirely by the TPM key. Okay. Okay, I'm not sure I don't understand, but thanks at least for the answer. Do you have something to add or just to add there is there is an attestation key registration that if the platform requires it like like if if you're on on bare metal and not using Azure vTPM, that that attestation key is is trusted on first use registered with

uh relying party. But you are trusting the public key of the TPM of that particular instance and then everything is signed with the private key of the TPM and that's where you can guarantee the authentication of and then that the and then you can compare the measurement. Any other question? Yes. Hi, thank you for the presentation. Um when you get a confidential VM in the cloud, then

the cloud provider necessarily also contributes a few software components to the VM as TEE. And you get measurements for these also. Um, but how can we have trust in these components like the firmware or a para-visor or a management agent type things? uh, the also the TPM is provided by the cloud provider. Uh, So, the firmware you can also use your own. Uh, there is also a

possibility to bring your own firmware. But the thing is that everything is measured. So, everything that is measured is what you expect to basically. but typically the root of trust is in the CPU. Right? The Intel or or the CPU. You need to trust the CPU vendor. >> Yeah. And so, you get measurements um, for that code running in your trusted execution environment that is controlled >>

how can we trust this confidential VM as a whole if we attest only the code we put in there, but there is also further code in there that we do not control? And we get opaque measurements unless we have some means to meaningfully verify them. Um, so, yeah, this is the you need to trust some something in the infrastructure, but the attestation part is done by our

operator. So, you are not trusting for example the attestation provided by by Azure. But, uh, yeah, the thing is everything can be measured, but certain parts are given by the the cloud the cloud provider. So, yeah, that that part needs to be needs to be trusted. Okay. Thank you. Anyone else? Yeah, thank you for this talk. I was wondering about the remote at the station server. I

just Would you like host it on prem? is it the provider by the cloud providers using? Uh so, right now we are not providing any, but it needs to be running a trusted environment. So, it's a little bit of chicken and egg problem. Um so, right now it can be deployed in a small cluster that doesn't have many resources, but that environment need to be trusted. So,

either it's a again a confidential VM, but you need to attest that first. So, the the first environment, the cluster zero need to be need to be a trusted environment. It doesn't necessarily need to be confidential, but it's it can be hosted on prem or from from some entity that you that you trust. Otherwise, you cannot bring the the entire infrastructure. Interesting. Yeah, so the chicken and

egg problem. >> Sorry. Hard, I guess. Maybe you like you need run something to be at the state the attestation. Yeah, this is going It's everything very fresh and very new. So, it's going to develop and hopefully next CubeCon we will have more uh more detail. Yeah. Awesome. Thank you. I guess we're running out of time. >> Yeah, we're slightly over, but I think can close here.

Okay, if anyone has questions, please open issue in the repository or we are going to be around at KubeCon. Many thanks for your attention.