From “It Works!” to “It’s Secure!”: Hardening Your First Kub... Paul Zerdilas-Herrera & Leon Schulze

This talk focuses on the importance of securing Kubernetes environments by illustrating a scenario involving a junior developer named Dave. The speaker emphasizes that while Dave successfully deployed his first application, he overlooked critical security measures, leading to potential vulnerabilities and chaos within his cloud environment. They explain how attackers exploit weaknesses in Kubernetes, often using common attack vectors such as command injection and unprotected API access. The session highlights practical security measures, including limiting the attack surface, implementing role-based access control, and ensuring that workloads run with minimal privileges. Ultimately, the speakers encourage developers to be proactive about security and utilize available tools and best practices.

Good, good morning everyone. Thank you very much for being here. Um, it's actually a pleasure to see so many people here. We're definitely not expecting that. Uh, so thank you for being here. Um, second day of CubeCon. The weather is not really helping us, but you know, let's make the most out of it. Let's enjoy it. My name is Paul. I work for Nutanix as a solution

engineer. I specialize in our Kubernetes platform, NKP, Nutanics Platform. And today I have here with me my friend Leon. Yeah. Hi guys and girls of course and everyone else. Uh my name is Leon. I'm working for Palo Alto Networks as a sec ops guy. Uh domain consultant it's called. So literally I'm more coming out more of the out of the security world. And yeah so before we

dive into the actual uh presentation just want to take a minute to give you a bit of context of why we're here. Um so all of this started like a year and a half ago. Um, I was super excited. Uh, we're having a couple of beers with uh my friend Leon and I'm like, "Hey, you know, this Kubernetes is fun. I'm deploying clusters, exposing applications, breaking clusters.

Uh, this is fun." And as a good German friend of mine, he was like, "Hey, Paul, is this even secure? Did you even think about security when deploying your stuff?" Right. And I'm like, "I have no idea. I have never thought of it." And so we said, okay, let's actually work on this and try to share some of our findings with all of you guys. Uh we're

going to do this in a very simple way to make this easy. Um and we have a story. We've divided this in three acts. Uh from it works to secure. So let's get started. All right. Just as a short disclaimer, we're providing you also some statistics during the session. These statistics are coming from different vendors, different research institutes. So the numbers we provide you are not made

up. Okay. All right. So I want to introduce you to someone. His name is Dave. Uh Dave is early in his career, junior developer. Um just started at a small startup. Let's say a fintech startup. Uh just got his badge, his notebook, some company merchandise. And Dave is dreaming big, right? He's >> like like all of us >> like all of us on day one. >> So

um yeah he's full of ambition but zero idea what's coming. >> So Wednesday morning I don't know if you can relate to that probably most of you can his manager walks in and is like hey emergency man I need something from you build me something right so in this case some internal support tool um with a tight deadline right today is Wednesday you have until Friday. >>

Friday. >> Friday man. Okay. Wow. >> So Dave got a little problem because Dave, like we said, he's kind of new. He has no clue about really deploying something with Kubernetes and even less about how to secure it, right? He's not thinking about it. So Dave does what everyone of you are uh doing right now. It's literally looking up stuff, right? Tutorials, YouTube videos using his LLM

of his choice. Uh I'm coming from a time where we used a lot of Stack Overflow, but yeah. So finally after a couple of days, two days, some hard work, uh literally Dave got up to speed and he finally is able to deploy his first cluster, right? >> First application. >> His first application. Yeah. Um so literally he is spinning up uh the the cluster his app

trying now of course what's going on testing it out. As you can see some basic app testing it now checking out basic functionalities and uh it's not connected to prod yet right so he did his job it's a fintech so they have the regulations their QA process but Dave is happy his mission is accomplished and uh his team can access right he's sharing the IP with his

boss's team mission accomplished so it's weekend time right Dave is happy having his snacks gaming yeah going into weekend. So, this is where act one ends, right? And this is where, as you can see from the screen and you can understand, things are about to get messy. Uh, while Dave was gaming and enjoying his weekend, his cluster was wide open and someone is about to find out

things about it, right? So, let's go into act number two. Monday morning, he's super excited. He's excited to go to work. He's excited to uh be congratulated by his teammate, his manager about learning Kubernetes so fast, about deploying his first app so fast. Instead, total chaos. He starts re receiving emails from finance about bill is exploding in the public cloud. He receives an email uh from security

red alert. What have you done? And then he starts sweating. He's stressed. What's that mean? Have I done something wrong? Um, he sees there's some names of ports that he cannot even understand what things are running on his environment. Uh, so what did happened over the weekend? Leon, can can you help us uh and explain us what happened during the >> Yeah, sure. So, basically um I'm

not sure how familiar we guys are with network security. Um, maybe you can raise your hand. Who of you knows like the basic network security stuff? That's great. But some of you don't. So we go like just quick recap, right? So internet is like a house of uh street full of houses, right? Each house have doors, windows. So each house has an address, IP address, doors, windows,

ports, right? Something where you can enter. So 24/7 now people are running around the street checking out which houses are open and which are not, right? what they can find out. And in reality, these are tools, right? Attackers scan the internet, the whole IPv4 and IPv6 range space uh multiple times per day. Uh by the way, we good guys do the same and we sell it as

a tech surface management, but that's a different story. Uh so basically uh at some point you find an entry hole, right? You find a way to get in. Sometimes it's even advertised as, hey, come in guys. Uh something is is going on here. And of course, what do you do? you enter it and you try to figure out what's inside. So basically in our attack scenario here,

uh what you saw in the demo video before was uh literally Dave exposing his app through a load balancer. Um there are scenarios in environments where this is not the worst thing to do, but in public cloud maybe you uh some of you people know that already uh having that aligned with a vulnerable web app because Dave is still a junior. It didn't go through Q&A yet.

>> So wait, that's a bad thing. That's a thing that only Dave could do or is that a common thing? >> No, Paul, I have to say uh it's a common thing. 33% of all intrusions start with that as the initial attack vector just leaving the door wide open. >> Wow. Okay. >> Yeah. >> A third. >> So basically uh for the guys uh to give more

some background information here uh Showdown is a search engine for bad and good guys, good people and bad people. And literally Kubernetes uh is widely used. You know that right? And as you can see just looking up for Kubernetes we directly have like a huge amount of results that tells us hey there is something there's a house you can probably try to enter. So basically our attacker

after understanding that he can enter uh he's doing now uh more like a deep scan to figure out okay what's running there right how like which windows which doors are open. So he's using NMAP. It's a network scanner to figure out what services are running uh behind an IP address. And basically he does the same what Dave does, right? He's check uh testing out, hey, this web

app, what is it doing? And he's also fastly finding out, hey, okay, it's not connected to prot yet. So is there a different way I can misuse this web app? So there is a um literally a um tactic or technique called command injection. Literally Dave, like I said, is a junior. He didn't uh uh check that this is uh secured in regards to this attack technique. And

uh basically, it's really fast for the attacker, the bad guys to figure out, okay, I'm running somewhere uh that could help us to uh gain like to to achieve our final mine, right? >> Yeah, it's like a gold mine. Definitely. Okay, so the attacker clearly uh has access to that initial container. But the question is now how would this attack attacker move from one compromised container and

get access to the whole cluster? How he will be able to move and be in charge of the whole Kubernetes cluster? Right? So we need to understand that inside every single Kubernetes um pot kubernetes will create automatically um a hidden compartment a hidden uh path in that file here and inside there there's a token right what does this token token do essentially it's like the pod's digital

passport it's how essentially the pod provides um to the kubernetes API that hey you know I'm a service so trust me so Paul wait a minute uh how many of us really Leave the key under the doormat. >> Way more than you guys think. Around 70% of us don't think about it. We just leave it there. It just exists there. >> [ __ ] That's a big percentage, right?

So, let's have a look on how he would do that in real life. So, clearly hackers are not maturers. They know what they're doing. They're not guessing. they know exactly what that file that directory uh and they go immediately and look for it because once they get this token they can execute that token they can just copy the string they need they can they don't need even

need to sit on this u website they can just open another command and start the real thing right so now the question is what will happen once they've got this string >> yeah I mean the key of the house is stolen now what would you people do Now if you have the key right you would probably try to use that key if you have a build a

huge building of different apartments and figure out where you can enter which rooms which apartments can you and can you enter in this in this building so attackers they do the same right uh they don't have a single goal usually they have like multiple goals when they try to access your environments and basically what they do is now figure out okay do I have some secrets laying

around some treasure some crown jewels I can use in different attacks later on, right? Data exfiltration is something they usually like to do. Uh in Kubernetes, secrets, I don't know if you're familiar, they are not encrypted. Usually, they're just encoded and it's super easy to decode them. So, basically, after they uh figured out what's running, what's there, they also want to gain some money, right? Money is

always good. So, let's deploy some crypto miner and hide it as system monitor. So, there we are. Right now, attacker is in. Dave is sleeping. Not bad. Oh, bad for him uh the Dave, but not uh bad for the attacker. Absolutely. So, yeah, here in the demo you see that as well. Uh literally, uh usually what the attacker does when they have the web app, they would

now open up a reverse shell. So, they don't need to use uh Dave's crappy app anymore. Um, but literally what we do here is because it's so vulnerable, uh, he just runs his commands inside of the web app, deploys his crypto miners, figuring out, okay, what secrets, uh, do I have here? And, uh, probably then checks out later how to use them in different attack steps. So,

yeah, deploying the crypto miner takes some time. Here we go. Deployed as a demon set. And um yeah, I think uh also validating that it's up and running. And the next day, so after the weekend, Dave is realizing something is no bueno, right? Stuff is running where it shouldn't run. What's that? And also has been hidden uh under a name that is not that doesn't sound suspicious.

Yeah. Right. System monitor. You wouldn't expect that to be uh a cryptomining port, right? >> Exactly. So looking at that video, you probably think, well, clearly Dave is dumb, right? That would never happen to me. in reality, especially when you're getting started and you start learning about complex technologies, complex architectures, um, there's a lot of things to learn, a lot of things, a lot of concepts. So

think about like Kubernetes as a is not a simple thing to learn. you probably have to learn, you know, what's a container, what's a YAML, what's a port, what's a service, what's an ingress. Um, and how our brain works is we just think that, you know, I want to make this work. I want to make this work to tell my manager, hey, I did it to my

colleagues, yeah, I did it. So, we just usually default to the fastest path. And most of the times the fastest path is to use the defaults. And what we've seen is that especially in Kubernetes default doesn't mean that it's the safest option. And we don't do that because we are lazy. We do that because we are overwhelmed. Overwhelmed with this with a lot of technologies, a lot

of things probably a time short time frame. Um so this is just not only blaming Dave. It's all of us to be aware that what we are we're learning. We're excited. But sometimes you know it's good to also take our time and make sure that what we're doing it's also safe and an interesting stat like 89% of the companies deploying clusters right now um this year they

have been hit in one way or another and this is crazy and by hit we don't mean that there was a data breach or something but the attacker at least came in at some point and tried to figure out what's going on in your environment right >> and the worst part here is that it's within minutes. So this is important that when we're deploying it, make sure

from the beginning is secure. So let's have a look on how we can make this secure. >> Finally, we are all there for uh we're all here for that, right Paul? >> Yeah, let's let's have a look. So first of all, back to security 101. Uh security is a spectrum. It's not binary. It's not a yes or a no. It's not a on and off, right? You're

never going to be 100% secure. No, you're not. You're just going to make it harder for the attackers to come in and harder and more expensive. And usually what they do, they hopefully they move to the next target and they leave you alone, right? The attackers advantage as the defenders, the good guys, we have to protect everything. The bad guys, they just need to find one hole,

one vulnerable app, one Dave that will do an initial mistake and they're in. Yes, it is unfair, but life is unfair. Leon, so we have to be smart about it. And third, probably the most important thing, you just don't rely on one wall, on one strategy, on one defense line. What happens if they actually get the token? You need to make sure that you have a second,

a third line of defense. So that could be role-based access, that could be network policy. We're going to dive a bit deeper into that, but think about layered approach. Assume they will break in. How are we going to stop them once they are in? So yeah, now let's get a bit more practical. So three steps here. First one, limit the attack surface, right? Close the doors, the

windows that are open and you don't need to be open, right? Uh don't expose your apps. Uh if not necessary, restrict API access. There is the principle of lease privilege, right? So only provide the permissions to services. uh or service accounts where they really needed and there is a principle of role based access control right use roles and assign permissions to these roles. So that stolen token

uh that had cluster admin privileges because they've went the easy way, right? Having admin privileges, developing stuff is always good and fast. Yeah, no bueno in pro. Uh an app like the app of Dave, it never needed it. And the third point, contain the workload to really limit the the possibility of impact, right? Or the level of impact. So the service ran inside uh as root, not

necessary. uh it was able to write to disk also that you can prevent right >> quite basic right thank you uh Leon I think it's also good to keep reminding ourselves on how to do the basic things right um so Le give us a strategy thank you for that the first thing we're going to see is how we're going to make sure to close the door right

um so you see the red box there in the YAML so by default some distributions allow this anonymous authenticator authentication right So this will let anyone talk to your API without having an identity. So we need to set this false. We need to make sure that you've got an identity to be able to speak with um the our API. Um for example in our distribution in the

NKP we make sure that we've done this for you and we secure it out of the box. Second, build a fence network isolation as as Leon mentioned before. Um, you shouldn't be exposing your environment without any real reason. So, try to keep it as uh private as you can. That's obvious, makes sense. Create private clusters, IPO list, make sure that you just give um the right people

the right access to get into your cluster. And third, um you've built a fence, you also need to know what's inside the fence. So using um projects open source projects that we're going to want going to uh see in a bit about scanning your environment understanding before you deploy anything if that that thing could be vulnerable. so we lock the door uh but what if the bad

actors still figure out like uh where's the key right? Uh so for me uh being like a secops guy and not a developer not a cloud native uh expert when I started to learn about Kubernetes and how insecure it is sometimes by default I was almost crying right um but then good news is Kubernetes provides you all the tools all the guides all the steps to make

it more secure and in our case with the key under the doormat it's one line you can use right Automount service account token equals false. That could have um fixed the issue they've had. >> Absolutely. Now, let's quickly go and and have a view and understand of the rolebased access. Why did the the hacker move from a compromised application pod into uh getting access to the whole

cluster? So, what did um Dave do wrong? So, he created a service. That's not bad, right? Uh, okay. He have a funny name. Uh, he put his own name as a service account. So, that was the identity. So, essentially the identity doesn't have any power yet. The problem here was that he just bind this to the actual cluster. So, once this port was compromised and got that

token, that token was allowed to um deploy things in the rest of the cluster and that was too much. So, how should he have done that? create a service probably not put your name as Dave uh but we create a service uh a service account then create a role so this role here if we see the verbs um you can only list right so it's a read

only list and get you cannot deploy anything because that application didn't have to deploy anything um so limit really what the pod needs to do and then bind this right so if you get um hacked We make sure that this gets we cannot propagate we cannot move out of that name space we cannot just escalate our privileges right so we contain this in in in a way

okay so we lock the door that's good we remove the keys laying around somewhere also good we fix the IDs so now what if they still get in um literally the next steps we provide you uh think of it like uh giving your Kubernetes environment like your container like a digital straight checker right like in a siteyard. Uh so literally rule number one no route always good

right we can uh tell Kubernetes to do that so run as nonroot and you just run uh the services inside as a regular user. So also make sure that read only root file system is true. Yeah you're looking your containers disk. You don't want anyone to be writing any new data uh in your own storage. So essentially you don't want um the hacker to be able to

download any malware um save any malicious scripts or modify your code uh because you don't really need the application doesn't really need to write anything into the file system. So even if they get access to the pod it's at that end for their attack. Then privilege escalation sounds always awful. So we don't want that. So there's one line again allow privilege escalation equals false. And uh we

tell like yeah the the bad guy comes in and he finds a trick to become admin tries to escalate privileges and we literally say or the cluster says nope you cannot do that right also good and last one oh you're too fast >> sorry it was yeah that um and last one because I think we had enough of yaml um um drop all the capabilities so we

strip away essentially all of the default Linux privileges that most containers actually don't need. So, you're probably going to be wondering, Paul, are you crazy? You want me to go and check every single port, every single YAML and make sure those um those lines are there? It's like obviously not. Dave had a life, you've got a life, you know, we need to make sure we are as

sufficient as possible when doing this. So, let's see what is the tool set, the toolkit that Dave could actually leverage. Um, obviously, we've got a lot of projects here in CNCO system. We've got thousands of projects that we can leverage to make our life a bit easier. Dave was just learning so he was not aware of it but we are. So let's have a look what we

could use. There are many open source projects. We're just mentioning some of them that we actually used. Um the first thing we mentioned this briefly make sure that you check for abilities before even deploying anything before even the application leaves the developer laptop. Second, uh once you want to implement all of those YAML files that we just saw, like we need to make sure that if that

YAML is not secure, the class is not going to deploy it. Third, once it's up and running, we need to make sure that if there's a vulnerability, if there's an anomaly, um if the hacker wants to deploy a shell, this gets stopped. It gets flagged and we get told, hey, where are you going? Where are you doing? And and and last one, it's it's it's also really

important to have a benchmark to understand what's going on. Um make sure that we can check and be like, hey, I'm compliant. Everything is working fine. Um I I'm trying my best to be secure. Right now, um if we're talking about enterprise solutions and we want a production ready way of deploying our cluster, make sure it's um secure, we can have uh cortex cloud integrated. We have

NKP deploying uh secure clusters out of the box. But that's enough. Let's give us a bit of context about that fixes that story clearly. Yeah. >> But have you heard about anything real uh along those lines a little? >> Yeah. Basically how we came up with Dave's story is um you saw a really like vulnerable app now, right? and a really blunt guy exposing his first web

app there to the internet but it's literally more real than than you think. So there was I don't know if you have heard about it who have heard about react to shell. Oh not many. Okay so uh you can access the slide deck via PDF later and click on this link. Maybe it's interesting for you. So basically um it was a vulnerability that affected containerized environments um

for uh for for React applications and when it got uh the zero day got um disclosed attackers like the scanning immediately took place right so after it got uh disclosed we saw within minutes that people scan for vulnerabilities and try to exploit it. So basically now we talked about this defense and death and layered approach. So if you have multiple walls that really are implemented in your

environment right then you are already doing a great job but there were of course uh customer environments that didn't follow best practices or maybe that where policies weren't up to date whatever which is also fine because I always say security is learning to walk it's never running right uh so basically um defense and death really helps a lot to kind of prevent the bad things become even

worse. >> So, um I hope it kind of helps you to understand uh why we even told you this Dave story and um so summarizing this a bit um we came up with uh a quote really quick to close the session. Um so as you can hear in this ecosystem which is evolving really quick with so many open source uh um projects there are two different types

of people. the people from who learn from mistakes our mistakes or other people mistakes and the people who don't and they pay for the consequences right so let's be number one and and last one just to close uh we thought we found this meme which was last night very funny uh so just be aware that default doesn't mean secure right um and with that thank you very

much thank you for your attention thank you for being here uh thank you Nice.