From Static Tokens to Attestation: The Evoluti... Ciprian H, Jack Francis M, Josephine P, & Justin B

This talk focuses on the challenges associated with secure node joining in Kubernetes. The speakers, who are engineers from Microsoft, Google, and Red Hat, discuss their experiences and the ongoing debate within the Kubernetes community regarding node security. They highlight various vulnerabilities that have emerged from configurations, particularly regarding metadata services and user data scripts that can lead to unauthorized access and arbitrary code execution. The discussion also touches on node attestation, which ensures that nodes can be verified as legitimate participants in a cluster, and the importance of standardizing practices across different cloud providers to enhance security. They express the need for a balance between user experience and security measures, advocating for a stronger focus on attestation and a unified approach in securing the Kubernetes environment.

So we will discuss about secure node joining obviously in the context of Kubernetes. Um and uh we hope to give you some insights onto that and hopefully have some plans for the future. Uh with that in mind uh let's start with the present introductions. Uh my name is uh Cyprian. Uh I'm a software engineer at Microsoft working on platform engineering. I'm also an open-source contributor and maintainer

for about the last eight years. Uh and I'm happy to continue with that for some time more. Thank you. Uh hi, I'm Justin Santa Barbara. I am a software engineer at Google. I've been working on the Kubernetes project for about uh 10ish years and have failed to solve this problem which makes me one of the least qualified people in this room to be talking on this topic.

So I hope there are many questions that you will grill us on. >> I'm Jack Francis. I'm also a software engineer at Microsoft. Been working on Kubernetes basically as long as these folks. Also similarly guilty of not solving this problem before but looking forward to this discussion amongst all of us. >> Yeah, I'm Josie. I work for Red Hat and uh um I uh maintain the uh

Carpenter implementation for u IBM cloud and touch on different sik autoscaling projects every now and then. >> And uh I'm Michael McHugh. I'm also a software engineer at Red Hat. Uh I've been working on Kubernetes kind of in similar space and previously on OpenStack doing more stuff and I don't think we solved this on OpenStack either. So, I kind of feel like Justin here. >> Yeah. So,

I I think we all agree that uh it's something that hasn't been solved. We've been trying to figure out who was supposed to solve it. Um sadly, no one said yes. So u but we've been having this uh debate at least us as maintainers for some time because we also have projects like Gaops cluster API and CVS and other people complaining about security. Uh, and we want

to make this a good experience for everybody and secure, right? We want clusters to be secure, but we've seen our fair share of uh CVS and uh issues reported. Um so what are memorable things from your point of view as maintainers uh with regards to issues about insecure >> oh sorry but sorry >> sure >> I'm not sure I want to go first here this is like

confessing all of our sins um I I think there have been many there So they're all basically the same vulnerability. The we put too much uh information into the metadata service, the instance metadata service, and that information is sufficient to enable relatively what would normally be a simple container escape, uh can be escalated into much more severe things like a cluster ownership type thing. And so what

we've what I've spent the past 10 years doing is gradually whittling away at uh putting less and less into that metadata service having fewer and fewer node permissions. But we we are very much not there yet. I feel like we've done some stuff in in the chaops project, but other pe other projects have done different stuff and I think it is slightly scary that I'm sure everyone

has the same story with a different light, but I don't know if anyone wants >> um yeah, I I have a similar story which is also related to arbback or if you don't lock down certain resources and you can uh because customers or end users will frequently request some uh some silly features to be added. decided to do whatever they want to do with their clusters like

um running rancher on top of like bare uh IBM cloud instances and to do that they use this user user data script and uh if you don't lock that down properly then you can just have arbitrary code execution as root on those new nodes and uh it's not good. Yeah, one thing that has definitely happened over the years many times is uh it's less of the the

secure breakout type situation but more real realizing that you need to roll your searchs on your cluster and um so lots and lots of self harm in the interest of trying to achieve security you know under some urgent circumstances. So setting things up not only to be able to join securely but also to make them you know robust for day two is is a huge huge challenge.

Yeah, I think and talking about kind of experiences and stories and whatnot and you know we mentioned cluster API and KOPS and Carpenter um you know we're seeing users want to create clusters that are ever more complicated. They want to have cluster API for machine management but then they also want to add in Carpenter to get the best perhaps price optimizations they can on the cloud they're

working on. And in scenarios like this, what we're seeing is now there are multiple ways for nodes to join the cluster, which means that we're extending the attack surface for anyone who might do this. And we don't have a unified way of talking about where does the attestation, where does the secure boot happen within a Kubernetes cluster. >> Um I also feel that there is another problem.

People are trying to secure their clusters without actually knowing what they're doing. They are complicating their clusters in such ways and figuring out later that what they did was wrong. Like they thought it was secure. They didn't know this. Oh, you I should have done it this other way. So, I would say it's more like keep it simple unless you're expert. Don't try to reimplement something. >>

I mean, it's funny you say that because I was, you know, to prepare for this, I was reviewing some of the internal materials uh that we have at Red Hat where we have customers kind of asking for this. And I'm I'm not going to leak any confidential information. At least I hope I don't. Um, but to what Cyprien was saying, yeah, I think one of the one

of the highlighted lines on one of the slides I saw was this should be transparent and easy for users. And I was like, okay, isn't that always the magic, you know? >> So, I wonder just turn off. >> I wonder if uh anyone would care to define what notation means and why it matters, why it might not matter. >> I'll take it. I'll take Okay. So, >>

so node additation to me means that a node is able to self-identify to some other resource and that can that information can be trusted. So, the node is who the node says it is. Now why does that matter in a uh node registration um scenario? So Mike you sort of mentioned the hybrid scenario which is becoming more and more common. So if you are running a managed

Kubernetes service and you're in control of the entire stack you're control you you know you're implementing the control plane uh you're in control of how nodes join that control plane uh things are a little bit more the the surface area is more finite and and known to be able to implement things securely. But if we're introducing hybrid and and you know especially in an environment now where

there's a lot of inf infrastructure scarcity it might be advantageous to uh offer a control plane service but then welcome nodes to join from other uh infrastructure environments. So then node at astation becomes uh more valuable. Um if we bring this down a layer to how node registration actually works in Kubernetes with um bootstrap token uh admission or other secure uh facilities into the node registration process.

What does node at astation add to that? Does anybody >> I think also like we're talking about what is node testation too and I think at one level there's the hardware aspect of this right which is like did I get the instance that I asked for? Is this actually part of the cloud that I'm working on? Uh, but then there's also the software layer too, right? And

and we're seeing, you know, we already have cloud in it and ignition as two of the most popular ways to start nodes within Kubernetes. And now there's also a layer of is this the right software that's running on this node be you know is the hardware what I wanted and is this also the software that I expected too. And I think that that kind of plays into

this and there's multiple layers this needs to happen in. Maybe I can talk a little bit about how how we do node attestation stuff uh within the context of uh IBM cloud and carpenter. Um so the the way we solved this for for the time being and we had a a funny discussion yesterday about this what what is the bare minimum for for node at astation and

where does it start to actually get fancy. Um and uh the the bare minimum kind of implementation we worked on was to use also the the metadata service to retrieve the initial identity and we also have a token service as part of the operator to do then the the verification. So it uh generates the bootstrap token, hands it to the node as it's spinning up as part

of cloud init and then uh it computes this uh the the initial cube config that the cublet then starts with and uh creates the the CSR for it. So if that's happening and if the bootstrap token is uh received through a secure privilege channel that included you know during that process add astation was a requirement and that requirement was fulfilled. Can we just ignore node at astation

once the joining process takes place? Is the token itself sufficient to implicitly suggest at a station? >> Um I I think it is kind of the bare minimum. I think in an ideal world we would have additional steps to it and this already has different different layers to it where you uh verify these uh different tokens different certificates at different levels of the operator. Um but yeah

ideally we would be going towards what Mike kind of described um to have the also the the secure boot at the station part and and all that. So in terms of the bare minimum, does anybody want to talk about how you can do the bare minimum well and how maybe where maybe you're not doing it so well? >> I'll take an opposing view. I'm not sure there

is a bare minimum that we can do consistently across the clouds. I think that what we've done with the we we introduced the join token uh as a as a convenience to create a better user experience and it is much easier to use but I I would say that it is not even good enough across the board and and you should you can maybe make it acceptably

safe in certain circumstances when you're minting a join token for each new machine that joins the cluster but that is a relatively rare configuration. And I don't think it's the majority of nodes. And the thing which gives me fear is that you know that sounds like a great approach that you're using but it is going to be different on AWS and on Azure and on GCP and

there is no standard in the Kubernetes project where we look at these different approaches and we're like hey Justin did you mess this up royally on this particular platform and uh it would make me much more comfortable if we effectively came together and came up with something that we thought was a bare minim minimum that we could maybe get behind. And we were talking about that a

little earlier too about like you know the reason this has been a hard problem to solve in the Kubernetes world is because we've been focused at everything that happens after the machine boots and after Kubernetes after the Kublet has already joined the cluster and everything but there's we now need to look at this and and do we need to as a community or how can we as

a community start to define these discrete states so that we have a way to do this type of attestation. Um, as you said, there is a big variation between what cloud providers offer in terms of security. You have those that just give you a VM with nothing and you have those larger cloud providers that actually offers you TPM, attested metadata or various other services that could help

you identify correctly a VM like is that supposed to be part of my cluster or not? But this is very diverse. There's no like everybody kind of standardized on user data like 169 254 something and yeah everybody can get that everywhere including all the secrets that people put in it. Um but no one standardize on something simple to say hey it give me some signed proof that

this machine is who supposed it is that I can give it to someone else and it can validate. Sure, it's not perfect. You don't you someone can take it and use it, right? But it's more than hey, I'm up and trust me, my ID is this and I'm your friend. And also I'm sorry to talk so much but but also if we're going to create a solution

around this or if we're going to create workflows around this we need to make space for different users demands because one of the things you know that we see from our customers at Red Hat is you know many people are trying to create their own type of platform engineering or or their own type of managed service for their internal users and we need to make sure that

there are options so that if someone says you know for my whatever ever my industry is, whatever the auditing requirements I have, I need to have an external attestation server that is not attached to the Kubernetes cluster. Maybe it's on a separate network, you know, whatever it needs to be done. We need to be able to support that. And we also need to be able to support

kind of the basic behavior that we have today or the minimum we have today. I guess I just wanted to uh say we have about 14 minutes left. If folks want to join in this discussion, feel free to line up there now. Um, we'll keep talking. Uh, but if folks come up then, okay, here we go. >> Talking. >> Thank you so much. >> Please. >> So,

thank you. I'm very happy that you directly opened with the metadata server and you almost got the IP IP right. This is a pet peeve of mine. It's it's so insecure that we uh use a token, a shared secret to join a cluster. Um, I've been playing around with virtualization and the one thing that I uh ran into is the the virtual smart card. We all use

certificates for Cublet to talk to the to the admin server. I'm we have Microsoft, we have Google and IBM on stage. Why don't we as an industry start providing um a certificate to the machine to keep the secrets off the machine? Use it as a signing device. it it directly gives you the identity because the platform proves your identity by giving you a valid certificate. >> I

think that would be that would be great. I mean I think like I'm not representing GCP but like GCP did go with a virtual TPM and I think that TPM is >> a great example for physical hardware and it's a nice model. I think other clouds went other way, but I will um say thank you to our friends at Microsoft who are pushing TPM requirements through their

OSH. So I keep hoping that that will be the driving force that makes the the TPM the or certificate even just a certificate just like ubiquitous. >> Thank you. >> We were kind of talking about this earlier too. We were we were having a little discussion about you know this earlier and and it's specifically the join token right and and how that might be distributed in a

better way um or in a more secure way let's say and this is again where we got back into the distinction and kind of the differences of what does it look like at the low level to start an instance and you know like from Red Hat's perspective we use a core OSbas based operating system which is based on OS tree and that looks a little different when

we boot with ignition from when a cloud init instance starts. And so one of the things we were talking about, you know, a design that that is kind of like circulating at Red Hat is the notion that okay, what if encrypted layers could be used on the OS tree so that at boot time, you know, a token could be generated, encrypted, and put in there. And then

during the boot of the instance, the instance calls out to an external service to say, can I have the key to unlock this so that I can get the joining token? And at that point there's an external service that's doing another layer of attestation on what's happening. So that's kind of another look at this and just kind of another way to >> yeah it's it's still a

shared secret to me. It's you give it into user space and you you started like perfectly it it's the attack factor. It's if your uh control plane gets compromised they deploy a container they extract those tokens and they run with it >> and you will have a big bill. Well, in this case, you would have an external attestation server that you would have to also you would

have you would have to somehow v, you know, >> KMS. >> Yeah, exactly. Break that to get the key to unlock it. So, yeah, I guess it's just adding more steps to the attestation, you know, kind of flow. >> Yeah, thank you. I I'll be working on this. So, let's compete. That's perfect because we we were trying to frame the panel a little bit as like we'll

discuss the status quo and solve the problems at some point. >> Yeah. Well, there is a sig in Kubernetes sigoth would would be this the right sig to engage for for that. So what you were mentioning um at some level you can get this sort of information on VMs. I think I've seen it in uh Azure as a tested metadata. uh AWS similar naming and I think

uh GCP also has a similar thing probably via TPM but anyway it is possible to get this sort of identification sure is not aert per machine but if you want you can do it with TPM the problem is how you validate it that part seems to be still to to vary between clouds, right? You need to to get public key or something to validate that whatever came

to your side is the right thing. Uh a tested metadata is signed with the cloud provider certificate. So it's fairly easy for anyone to just get the certificate from somewhere in the cloud and just say, "Oh, okay. It came from this machine. Hi. Um, in my opinion, you already uh spoke about the you already touched the in my opinion perfect solution which is uh using a secure

element uh trusted boot and uh that could eliminate the need for the join token completely. So you monitor the uh the boot process, you create a the certificate that uh provides evidence through a third party and the whole authorization happens in the control plane. I mean all we would need is some standardized way to uh to create uh virtualized secure elements across the clouds and do the

secure boot. Right. >> Did I >> I think that's I think that's right. I think we we what we are hoping for is that we get that that virtualized secure element across all the clouds and it's not just a matter of the big three as it were or the three that are big four I should say but it is uh big N but it is that there

are many other clouds as well and you know like physical hardware yes typically has a a TPM but there are many clouds that have different capabilities and I think one of the things we want to be really careful of is um yes let's get that but let's also figure out that when we have that that we don't mess it up like we don't enable replay attacks or

whatever it is and so yes let's work with sig off make sure they audited it make sure that we figure out the whole pipeline put as much pressure as we can on the longtail to support it but maybe also find some other way that enables the clouds that might not have all that support on day one to get better and provide them a mechanism to to get

to where we want to be >> I'm sure the others will follow I mean you can you declare all the uh the metadata service legacy and uh >> a conformance program for uh for >> Yeah. So Justin, are we talking about so we do it in Kubernet? We we update the control plane, we update the node registration project uh process and then we simply dare the cloud

providers not to use it. >> you heard it here first. There we go. you know that that's a great question too and this is kind of what what we were talking about in terms of coming to a standard that sits at the level below where Kubernetes is looking right now so that yeah if we could standardize on something and maybe it starts with the big N and

then it becomes popular and they see okay this is an open standard that we've put together and yeah others will start to participate that that's kind of the hope right >> all right so first of I've got a confession. I wrote a proposal many years ago about standardizing and never did it. Uh so sitting in the cluster API repo unimplemented. Um I've got a question for Justin

actually because I was under the impression that chaos was mostly solved this problem at least for AWS. I'm wondering what challenge >> just confirm that publicly. Um so we we have implemented a uh sort of additional handshake. We do not use the joint token uh as part of this whittling down of of the permissions. Um the there are at least two problems with this. Uh the first

big problem is that it's different on AWS than it is on GCP and different from Azure and and so on. And the second problem is so there is n implementations. The second problem is that that implementation is not shared amongst the uh the other installation tooling. So the only people that have really audited that code are you know a handful of of kind people. Uh so that

that scares me like it I would be much more comfortable saying we'd solve the problem if uh it had gone through auditing by sig and you know the various people and I I think there are definitely clouds where we do the best we can best we can um like basically a call back type mechanism where we assume the IP addresses as the IP address is as secure

as you're going to get um or the ability to listen on a on a port on an to get um but I I I'm sure we can do better there and hopefully raise the bar a little bit as well. >> I see. I thank you for your discussion. Um I think we can come up with ways to solve this with virtual TPMS because we can find a

way to have the hypervisor control plane inject something based on the class control. How would we solve this for real hardware and many clouds provide physical hardware and we can't easily influence the TPM in there from external. Is that still a point? >> I think you can. >> I I think you can because in VMs it's easy. The control plane is done by >> the people on

stage. >> But but even um in hardware somebody provisions it for you. It's mostly automated and before they give you access to the machine they can install certificates in the TPM. >> Okay. that I don't share. >> Just wanted to say that you might be right, but you're kind of talking about two ends of the spectrum. One is the big cloud providers with the means to do

it and the other is like bare metal where you can do whatever you want and you're on your own. Basically, the problem is that you have many many in the middle where you get a cheap VPS that uh everybody loves. you know how many like this kind of kubernetes running on such clouds right and they don't give you TPM they don't give you pretty much anything all

you get is an instance uh an IP address and instance metadata because they kind of were forced to add it I I wonder if node joining is that much of a problem for bare metal anyways if you if you colo your hardware then you already kind of track your inventory to begin with and for me the question is more like why do you need more at a

stationation I think it's a good point like it's the dynamic question right like if you're if you have fixed hardware it's less of a problem I do think that maybe there's an underlying issue which is you know we can solve the basic uh like is this VM or machine allowed to join at all but then we'd also love to solve. Hey, like is it running the OS

I think it's running? Is it patched? Is it up to date? Like did it is there a bootloadader attack or whatever it might be? Like we'd love to solve all those other problems. And yes, if we're able to get the TPM uh going, it certainly is a very attractive uh set of guarant or set of promises that we can we can get. >> We probably have time

for one more question. Just FYI, we're we're we're down to a couple minutes left here. >> Yes. Uh thank you for the discussion. Uh it's it's probably not a proper good question. Uh but uh the accomm standard now has implemented or defined the device at test 01 challenge which sort of standardizes um uh client uh device certificates which are um devicebound and it struck me while you

were talking that this is probably maybe a good entry point for having a standardized way of uh at least provisioning certificates but it still leaves a lot of the sort of the attestation parts of the deployment. um uh left out of the entire specification, but that's at least worked by Google to at least get client certificates and it can probably be used for um servers as well.

>> That wasn't really a question. Sorry, but it's more a point. >> Do we want to take one more question then? Yeah, we can take one more. Yeah, come on back up. >> Um yeah, the problem is not if you have collocation that's is not a problem. You join your 10 nodes, you're done. But my my idea is more like you mentioned cheaps, heta or any where

you can just buy physical servers. You can't you can of course run some kind of provisioning mechanism on them. So your provider could do that but that's again a scaling problem and and let's let's call it an um effort problem. Most people don't care that much about this topic to have to really implement it. Um we discussed together that what was part of the panel description. Um

these days we have uh cluster API carpenter which behave a bit differently than we used to have autoscaling groups and other concepts similar you get a node claim or something similar machine in cluster API. So basically you want something you expect that specific thing to come to you. I would say that this is a huge step forward uh at least from visibility point of view like if

you just accept shortlive the joining only for a short time it's improbable that something else will join your cluster. Right? you you have a onetoone node claim to the instances that appear with that specific hardware I'm going to steal 15 seconds. I think something you said there was really good like it's about the effort, right? I think whatever solution we come up with uh it should be

very low effort for the consumer and I think you know we chose something that was low effort for the implement i.e us in the form of those bootstrap tokens. And maybe it's time we, you know, figured out something that was a little harder to implement, but was just as easy to to consume, but maybe more secure at the same time. And >> yeah. All right. Yeah. I

mean I you know I was just going to say to what you were saying too part of this is like making sure that whatever we do it's open enough so that if someone is if someone is putting an instance or a VPS wherever they're doing it that that they can get their you know trusted solution in the mix right so wants to close >> yeah okay yeah

and I guess part of the discussion was also that whatever solution we want to uh work on that we have um pack most of it into the core and then have thin wrappers for the cloud providers in the end to make sure we don't have that much sprawl. Yeah. and ideally some sort of documentation with recommendation for it's >> nice to have >> for people like not

doing it blindly.