Bob and Alice Revisited: Understanding Encryption in Kubernetes - Jackie Maertens & Mitch Connors

This talk discusses encryption standards in Kubernetes, emphasizing the importance of secure communication through identity, authorization, and encryption. The speakers, Mitch Connors and Jackie Martins, explore key concepts including symmetric and asymmetric encryption, transport layer security (TLS), and mutual TLS (MTLS). They illustrate security goals using the classic scenario of Alice and Bob, detailing how messages can be secured against eavesdropping and impersonation. The session also highlights the challenges of certificate management and the use of service meshes in Kubernetes to enable secure communications. The speakers conclude by stressing the integral relationship between encryption and strong identity systems for effective security.

Well, good morning and welcome everyone. Uh, happy to talk with you all about encryption standards in Kubernetes starting from the ground up. Now, when Jackie and I proposed this talk, which we've proposed a few times, so was it like a year ago that we put together the proposal or so? >> Yeah. >> I had no idea how relevant the talk was going to be uh for me.

And that's uh because just yesterday I wake up and I have a text message on my phone from a few hours ago. I'm 8 hours out of my home time zone. And uh the text message is from a number I've never received a message from before, of course, and one of those five-digit numbers. Uh it says that, "Thank you for transferring your 401k, your retirement account, uh

out. It's now closed. And if this was in an error, please click this link." The text message came in while I was sleeping. So uh who clicks the link? >> Okay. I have some things to sell you after the talk. You should come meet over here. >> I feel like they're not all being honest. It doesn't even say what company it came from. Uh, and and with

working at different companies over the years, I've had a few different retirement accounts. So, uh, it's not out of the ordinary. Like, I'm married and my spouse manages most of our finances. So, maybe, but we really didn't talk about that in advance. And of course, in the US, it's the middle of the night and they're asleep. For me, what I really needed to know was what is

the identity of the person who sent this message? Was this really from a company that I have a financial relationship with? And can I safely reply to them and know who I'm talking to that I'm not giving away financial information and details to a stranger on the internet? Uh, and while encryption might not be the first thing that comes to mind for this, I think I hope

by the end of the talk, you'll see how encryption and identity, authorization, and authentication are related as an unbreakable chain. how they're all interdependent on one another and necessary both in our everyday communications. That would be really useful. If someone comes up with a great standard for text message encryption, let me know. I'd like to use it from now on. Uh but also as we build applications

and in Kubernetes. So my name is Mitch Connors. I've been a maintainer of the ISTO project for about seven years now. Uh but I also my day job is I'm a product manager for AKS where I manage networking. And uh I'll take I'll give it to Jackie to take us away with the rest of it. >> Yeah. So, hi, I'm Jackie Martins. I'm a software engineer also

at Microsoft. Um, I'm an ISTO security maintainer and I am our co-lead of the product security working group where we review any CVES or security reports that come in. All right. So, we're going to be talking again about the basic story. It's the name of the talk. So, we're going to be talking about Alice and Bob. So, they were introduced in uh 1978 in an academic research

paper. Um, but they've been with us for a long time. So Alice wants to send a message to Bob and this is a secret message. Um, and this is across a hostile network. And so Eve, the girl in the pink, like I guess me, um, represents this hostile network. Uh, and she can do three things. She can listen to the messages you're sending, she can modify them,

or she can impersonate either Bob or Alice. So that gives us our like three security goals for our secure communication. We have confidentiality. Can Eve read the messages that we're sending? Integrity. Can Eve change the messages that we're sending? And then authentication. Is Alice really communicating with Bob? Is Bob really communicating with Alice? Okay. So through the lens of Alice and Bob, we are going to be

examining uh the tools that serve our security goals. So the first are our key primitives. These are our asymmetric and symmetric keys. And then our secure connections that utilize these key primitives like TLS and MTLS. And then Mitch is going to take it over and he's going to talk about how these tools map to their to applications within Kubernetes. Okay. So first we're going to look at

symmetric keys. So in uh symmetric encryption, this means that Bob and Alice share one secret key. So the same key is doing both the encryption and the decryption. Um symmetric key encryption is fast. Um which me which is why it's used for encrypting the majority of bulk data. Um to send an encrypted message to Bob, Alice will encrypt the message with this symmetric key. So when Bob

gets the message, he'll use that same key to decrypt the message. So Eve, oh, she's so upset. she uh can't read this encrypted message because she doesn't have this symmetric key. Um so the again the symmetric key is very important. So it must be somewhat unpredictable. It can't it has to be generated from some amount of randomness not something that's predictable. So the main challenge with symmetric

keys is like you have to distribute them and you have to do so securely, right? If someone gets the key then that kind of takes down your whole network because it can be used both encrypt and decrypt. Um so the key distribution is one of the primary challenges with symmetric keys but it definitely has its place. Um okay next is the asymmetric crypto. So this is using

two related keys. You have your public key and your private key. In this case it's going to be Bob's public and private key. So and it's designed so that deriving the private key from the public key is um impossible. Uh so how these key pairs are generated I'm not going to go into the details that's uh above above the level of this talk um but the main

thing that you should know is the private key should remain private which makes sense based on the naming and the public key is meant to be shared and distributed. Um so in this example Alice has a copy of Bob's public key and Bob has his private key which makes sense it's private. So when Alice wants to send her encrypted message to Bob, she can sign her message

using this public key. Um and then Bob can decrypt the message with his private key. Yeah. And since the private key is securely stored and theoretically Bob is the only one with this private key, um Eve can't decrypt the message. So she's So one of the main challenges is uh you would think, okay, this sounds great. We have some kind of semblance of privacy here. we have

a private key and like the the public key is meant to be public. So you think okay that solves some of the problems of symmetric keys but it's not so great for bulk or most encryption because it's actually quite slow because it's mathematically complex and computationally heavy. So if you're encrypting a large amount of data this actually takes some time um sometime it's still relatively fast but

it does does matter. Okay. So, beyond encryption, uh, asymmetric keys can be used for establishing trust and in the authentication of the identities of both of the parties exchanging messages. So, it can be used for Alice and Bob and establishing trust between them. So, now we're kind of getting into the uh area of our security goals. So, in this example, Alice wants to send and ensure that

her message um that she's receiving came from Bob and not Eve. like she doesn't want to be texting uh with someone uh like you know who's drained your 401k. She wants me to make sure she's texting with someone she trusts like Bob. Um yeah. So uh yeah on their own asymmetric keys aren't quite enough to do this right because there's actually no identity attached to the key.

There's you know you don't really know that this public key really belongs to the specific person. So, in this example, Bob can sign his message using his private key. Um, it's his way of saying, "I'm the only one with this private key." So, if you have my public key, you know it's me. Um, so yeah, anyone with Bob's public key, right, because that's shared. Um, can decrypt

this message and that's fine theoretically, right? Um, the goal is to prove Bob is Bob and not encrypt. So, that's actually wrong. Um, there's no way to know that that public key really belongs to Bob. Um, Eve could theoretically intercept this message, uh, decrypt it, um, using the public key, right? Because it's public. So, she uses Bob's public key and then re-encrypts it with her own private

key. So, this is that yellow key. Um, if Alice has Eve's public key, then she can also decrypt the message. Great. but she has no way of knowing that she used Eve's public key rather than Bob. So she really still doesn't know that this message now came from Eve instead of So the missing piece of all of this is identity which is where certificates come in. So

a certificate is a digital document uh representing a uh user computer service device. In this case it's representing Bob who looks uh pretty dashing right there. Um, and it contains the public key uh for the subject. So, it has Bob's public key. Um, it notably does not contain his private key. That kind of would defeat the purpose. Um, it has the subject, which is the identity that

the certificate belongs to. And then it has the issuer. Um, this is someone that you, I guess in this case, Alice, trusts that has signed and verified that the public key does in fact belong to Bob. And so again, this is not a complete list of what's in a certificate, but for the sake of our presentation, this is the list that makes the most sense. Um, and

this group of data is called like the to be signed data. That's how we would group this. And so the next component of the certificate is the issuer's signature. Um, so this enables Alice to verify that the certificate wasn't forged and came from an issuer that she trusts. um it's signed by the issuer's private key. So again, still using asymmetric keys, it's signed by the issuers's private

key and enables Alice since she has access to this well-known public key for this issuer. This is typically what you'd have in your web browser. You'd have the public keys of well-known um well-known issuers and she can use this to verify that the issuer is the one who created this So and you might say, okay, great. What is this signature itself actually? So it's the encrypted hash

of this tobsigned data set. Um so this enables Alice to verify that Eve didn't modify any part of this to be signed data um and swap it with her own identity. So if Eve had modified the certificate or the 2B signed data, uh the decrypted hash would no longer match the two beigned data um and Alice would know she's no longer talking to Bob. This isn't a

valid certificate. This isn't uh the 401k or government guys. I don't US government guys. Who knows? Okay. So, this is where this is where TLS comes in. This is the first kind of secure communication that we're going to be talking about. Um and it is the TLS. It stands for transport layer security. and it is a secure communication protocol that utilizes certificates and keys to secure communication

over a network. Um so it provides authentication, encryption and data integrity. Um it utilizes asymmetric keys um for both uh with certificates for authentication. So yeah, we're grouping certificates um with our asymmetric keys and symmetric keys for our actual data encryption because remember symmetric keys are a lot more it's a lot faster than using asymmetric keys for the actual data encryption. Okay, so now we're going to

break down what the actual TLS protocol looks like at a high level. Um so here we have negotiation. This is the first step. So per you know to start communicating you have to say hello. So Alice and Bob, they say hello to each other and they agree on how they're going to be communicating. So they establish their communication parameters. It's a lot there's a lot more to

this than that, but at a high level, they're agreeing on how they're going to talk to each other. Okay. Then comes authentication. Um, so Alice wants to verify Bob's identity. Um, and so she validates his certificate with the well-known and trusted issuers public key. So great, at that point, she's verified. Okay, Bob is who he says he is. Great. Um, [clears throat] okay. Yeah, issuer's public key.

Okay, the next step is the key derivation. So, Alice and Bob work together and they create a shared secret um, which they then use when generating their session keys, which you might be familiar with. And these session keys are symmetric keys. So then lastly, they can be begin using their session um keys to encrypt the actual messages. And then to ensure like the integrity of those messages

themselves, um I'm not going to go into exactly how they do that, but Alice and Bob can use like some sort of integrity tag with their messages um to ensure that the data isn't modified in transit. Oh, yeah. And Alice, you know, because you've done such a great job with your TLS uh protocol, Alice can't um uh can't read your encrypted messages. So, one of the challenge

or a challenge with TLS is right, we're verifying that Bob is who he is, but Bob also has uh needs and he cares about who he's talking to. So, he would also like to know that Alice is who she says she is. Um, so right now we're only verifying our server or Bob's identity. So what in Kubernetes where this might show up, where you might see TLS

is between the API server, um, CD, C, uh, Kublet, web hooks, and various other clients. Okay, so now let's bring in MTLS. This takes TLS um, a step further. Um, it is mutual authentication. So both Alice and Bob authenticate uh each other using their certificates. So you can see they both exchange their certificates. Um it verifies that both sets or both identities um have the correct private

keys. And it's it's not just that Bob's identity is valid. It also matters that Alice's identity is valid. Um the protocol it looks the same as TLS except you're doing another certificate exchange. Um one of the oh one of the challenges of this is it's it's more complex right you have to now manage the distribution of all these certificates it's not something you uh you know let

your web browser the um servers handle you clients now also need to provide their certificates and um your servers or Bob needs to understand those certificates and be able to validate them so it's a bit more complex and in Kubernetes where this can show up is in service meshes that can help you achieve this mutual TLS. Okay, so great. We've just reviewed the key primitives and some

secure communications that can help us obtain our SEC security security goals for confidentiality, integrity, and um authentication. Okay. But the world is no longer as simple as when Bob and Alice met and began communicating. Um so Alice has many friends. She doesn't just talk to Bob. Uh Alice also communicates to different uh friend groups. Some of these groups are closer to her than others um both physically

and emotionally. And Alice doesn't want to secure all of her messages the same way. She maybe cares more about how she talks to one friend group versus the other. Um and Kubernetes is really just like that. Uh you know, communication isn't simple or just onetoone. Uh the the threats and those hostile uh hostile networks are varied. you know, you make some enemies along the way. >> Yeah.

So, when we start talking about Kubernetes, this picture gets even more complicated. It's not just that Alice has lots of friends. It that it's that Alice and Bob have lots of clones of one another, right? When we talk about running multiple instances of a process, uh we might need all of those well, we do need all of those instances to share a cryptographic identity. It doesn't make

sense for pods in one deployment to have unique identities in most cases. Uh so we've got to figure out how do those clones all get the same certificate that they're able to pass around to prove their identity and maybe that makes sense on one node or even in one cluster but what if it's globally distributed how do we get those identities distributed to remote clusters etc. Uh

and then of course we get a complexity of items. In this case we're going to start talking about Bob and Alice uh both as workloads or pods. They could be servers. Uh within Eve we're going to be talking about our compromised network. This could be an adverse actor who's gained some privileged access to one container uh in the cluster or perhaps at the host level some sort

of exploit. [snorts] And we're going to watch packets and requests go back and forth. Uh and of course talk we're going to be talking about using spiffy ids in lie of where we talked about certificates before. We're not going to take a deep dive into the spiffy protocol except to say spiffy is the way that certificates tend to be issued within the kubernetes ecosystem. All right so

let's take a look at how this looks with the most common encryption standard used across kubernetes wireguard and IPSec. So in this case we have two cryptographic identities uh for node one and node two within a kubernetes cluster. uh they're going to use the same X509 certificates to represent themselves. They're going to use effectively something similar to mutual TLS to establish trust. The the specifics are a

little bit different, but for most intents and purposes, this is going to be similar. And they're going to use it to establish an encrypted tunnel. Uh once they've got that encrypted tunnel, any pod on node one can speak over that encrypted tunnel to any pod on node two. Uh, and so here we see Alice uh saying hello to pod Bob and everything checks out. But what if

there's more than one pod running on node one? How many of you run one pod nodes in production frequently? Good, good, good. Okay, we're making some progress here. We've got one guy in the back. We should talk. That sounds interesting. [snorts] Uh, you've got more than one pod, right? Uh, so really what you verified here is that the originator was in node one and that the terminator

was in node two. But which pod was it on node one? Which pod on node two can be a little bit vague. Uh we don't really have cryptographic identity endto end from one pod to the other. And there's a variety of ways that this can be exploited. Uh particularly most of the tools that provide wireguard and IP sec encryption such as psyllium calico other CNIs uh they

tend to enforce network policy. This is how you say who gets to talk to who. Uh, right. Just having certificates that check out between identities and saying, "Okay, as long as the certificate checks out, you just talk and trust whoever they say they are would be like Alice and Bob exchanging all that information and then Bob picking up the phone and saying, "Yeah, who's this?" Right? Uh,

network policy effectively does that because it's got to use IP addresses to validate where the traffic came from. So if we look back to wireguard particularly in the psyllium case the psyllium agent uh is going to ask is this Alice's IP address on the connection uh and in this case we see it's actually pod Eve over on the left hand side uh but she's found a way

to impersonate an IP address. That's a fairly low threshold attack. It's a trivial thing to do within Linux networking to impersonate another IP address. So Eve, because she lives on the same node as Alice, is now able to communicate securely and encryptedly to Bob in a trusted fashion with WireGuard impersonation. So what we've got here is great for protecting our traffic from things outside of the node,

outside of Kubernetes. There's plenty of threat models that come from outside and this will be strong protection. But I think we can level up a little bit. Let's take a look at how service meshes tend to implement automatic MTLS. Uh and when I say service meshes, I'm an ISTO maintainer. I'm going to talk about what I know uh but there are other service meshes out there that

do this linkerd and others that you can have a look at. Um in particular uses a node level damon set called the z tunnel that is actually able to enter the pod network namespace uh intercept all of the traffic and send it over an MTLS authenticated tunnel. And the mtls in this case rather than being node identities of node one and node 2 are going to be

spiffy certificates that represent pod one and pod two. they can only be given to those pods and that's all you uh distributed using Kubernetes service accounts for authentication. So it's a secure chain of trust from one end to the other. They establish that tr that tunnel uh pod Alice is able to send whatever traffic she wants to to pod Bob. And in all of these cases the

pods think they're talking plain text HTTPS. The pods don't know anything about the encryption. It's all happening at the networking layer underneath them. Uh there's no real added application complexity. But in this case, we're actually able to cryptographically verify not just what node, actually not at all what node the traffic came from, but specifically the pod service account that this traffic originated from and the pod service

account that the traffic terminated on. Uh the reason we don't include node or even cluster information in these certificates is because if you're running a multicluster mesh, you want to fail traffic over, you don't need that to switch identities and say fail from cluster one identity to cluster two identity. you simply want to have service account as the only meaningful uh secure identity within Kubernetes and so

that's how mutual TLS works with ISTTO and then uh in terms of whether we allow or deny the traffic once again all we've done is validate that this is a certificate it's come from spiffy and is valid the next step is ISTTO uses authorization policy and unlike network policy which uses labels which translate to IP addresses to check what traffic is going where uh authorization policy actually

uses the negotiated identity on the wire. Uh so if we look again at our attempt to impersonate here we have pod Eve trying to impersonate Alice again. Uh however we've got an MTLS failure. Uh she's saying she's Alice but she's presenting the certificate of Eve. That's not going to check out. Um other well let's see other models. I guess I didn't write Z tunnel interception. Uh so

likewise if uh Eve is not permitted to talk to Bob Zunnel is just going to go ahead and block that altogether even though she has a good identity and a good certificate. It's a certificate that we've said by by a policy cannot have this communication and so we're going to go ahead and block it. Now you might have heard that recently the Psyllium team has added MTLS

support. This is a huge development. We're very happy with them. Uh we on the ISTO team have poked at them a little bit over the years about the lack of MTLS support and the need to level up their security. Uh, and it's great to know that our colleagues, we have colleagues at Microsoft actually working on Psyllium. Uh, they've been paying attention. Uh, the latest Psyllium release includes

MTLS support and it actually does so using the ISTO Z tunnel. Jackie, I think I recognize that pull request icon up there. [snorts] Uh, >> oh yeah. >> So, it's been a collaboration between Jackie's ISTTO upstream team and the Psyllium team to get Zunnel support added insium. So, the same utility that provides that MTLS security in ISTSTEO can now do so insium as well. Let's take a

little bit of a look at how this works. So, we're going to rather than using node identities, we're using pod identities to negotiate trust between Alice and Bob. Uh, and what is verified? Well, it's the same thing as verified as an IST, right? Uh, we know that the traffic has come from the Alice pod. We know that the traffic has gone to the Bob pod. We don't

know anything about nodes or clusters, which is a best practice. However, we have one last problem. Uh, and this is sort of a last mile for Psyllium. They've done great improvement in MTLS, but we're encouraging them to take it one step further. Psyllium is still using network policies to evaluate whether or not the traffic should be allowed or denied. And as we mentioned, those network policies really

boil down to IP addresses like firewall rules. So once again, it's as though Alice and Bob have done all this work to exchange their cryptographic identities over MTLS. And once again, Bob picks up the phone and says, "Hi, who is this?" And if the person on the other end of the phone says it's Eve or or says they're Alice, even though they're Eve, as long as that

IP address checks out, think of that like your caller ID on your phone, we're good to go. We're going to go ahead and have communication as Alice and Bob and Eve has been able to inject herself into the conversation. So, we're really happy with the progress that Psyllium MTLS has made. It's definitely a massive step forward for encryption standards across the psyllium ecosystem and we're encouraging our

colleagues at Psyllium to stay with it. Uh to make sure that they bring their policy in alignment. I can tell there's a number there, but I can't see what number it is. Hopefully, it's not. Okay, five. Thank you. Uh to bring their policy in alignment with their encryption standards. All right. A few things I'd like you to take away from this talk. Oh, and by the way,

uh Calico is also repackaging uh for encryption as well. I haven't been able to do a deep dive of exactly how they relate uh and how those work out. Maybe we'll be able to do a sequel with Bob and Alice in Calico Land uh at the next CubeCon. Things to take away. Uh when we talk about encryption, we really need to be talking about more than strictly

encryption, right? [snorts] Just encryption is not enough to verify communication. Uh encryption, authentication, and authorization are all vital pieces of a chain of establishing trust between any two systems. uh be that a text messaging system, your way of interacting with your colleagues, or the way that your Kubernetes pods interact over the I hope that one of the things you've learned here is that encryption is for everyone

in every circumstance. Our communications are better when they're cryptographically verified, when I have confidence of who's sending me text messages at 4:00 in the morning. uh and they're best for your pods uh for your application traffic on Kubernetes can benefit from this. But strong encryption always requires strong identity. Those things are intimately connected and we need to make sure that we're not trying to build strong encryption

systems on weak identity systems. Uh and lastly, we're really proud to say that the whole world whether you're using Calico or you're using Psyllium or you're using ISTTSTO, the world seems to be standardizing on ISTTO Zunnel for MTLS support. We're very proud of our accomplishments there. we'd be happy to talk to you about how to use the Z tunnel for your encryption needs. Coming back to Bob

and Alice, uh, now they're able to communicate securely. They're able to be sure who they're communicating with, even when they're communicating with clones of one another. But when they get that text at 4:00 in the morning, they can be confident that they should click that link because they know exactly who it came from and the person who sent it knows exactly who it's going to. Thanks a

lot. Thanks for joining us on this journey. And uh, I think we have time for questions.