Why Security of Kubernetes Comes Down to Linux Security - Marina Moore, Edera

This talk by Marina focuses on Kubernetes security and its fundamental connection to Linux security. She introduces the concept of container escapes, which occur when the intended boundaries of containers are breached, leading to unauthorized access to host resources. Through a historical lens, she explains the evolution of containers from virtual machines to modern implementations utilizing namespaces and cgroups within the Linux kernel. The speaker emphasizes that understanding Linux security is essential for comprehending container security, as vulnerabilities often arise from the kernel's handling of resources. The session includes a demo showcasing how containers interact with host system information, ultimately linking the significance of sound Linux security practices with container management.

Hi everybody. I think we're just about at time. I know people are still walking in, but we will get go get going. Hi everyone. I'm Marina. I'm going to talk to you today about a bit about Kubernetes security and why that actually means Linux security. So a little bit about me. Um I'm Marina. I'm the head of research at ADA, a company that builds a harden runtime

for containers. Um and I'm also very involved in the CNCF. one of the co-chairs of the CNCF tag security and compliance which is a technical advisory group that advises projects on security and compliance. Um I'm also a maintainer of the CNCF graduated project tough or the update framework and I'm coming to you from New York. So let's get into it. What are we going to talk about

today? So I want to talk a bit about um container scapes to kind of frame the talk then go into the history of containers. I think that really helps us understand what containers do today. um to understand kind of where they come from, what they do. Then we're going to dive in a little bit into croups and namespaces, have a bit of a demo, and then wrap

it up. So, let's get into this. Um just, you know, I know you're all at KubeCon, but just in case you didn't know, this is this these are containers. Um this is a very simplified view of some containers running on top of a node. Um what you have really and and this is kind of the point here is you have um a bunch of containers that all

sit on top of the shared a shared kernel with all the hardware and all those components underneath and the kernel kind of mediates access to all of those components. There's a bunch of mechanisms in place um kind of between and around these containers things like namespaces croups set comp etc that kind of provide this sort of barrier so that you when you're writing your application can pretend

your container is kind of its own space. But fundamentally what we have is a is a bunch of containers that are acting kind of like processes on a kernel which is responsible for interfacing with all of that hardware. So what is a container escape? So a container escape is when that boundary we were just talking about um is violated. So when um those assumptions that we make

about these different containers being their own you know independent little areas are um are no longer true. And this enables unauthorized operations on the host system from processes running in a container. This includes things like code execution, privilege escalation, um access to the host file system or you know file system of things in other containers, information disclosure and data tampering. And these are all kind of the

issues that happen when we can no longer like make that assumption about So to illustrate kind of why why I'm talking about this, I'm going to go through a couple of examples of container escape vulnerabilities from the past couple of years just to prove that this is a real thing that does happen. Um so here's one. This is um a CV which was a vulnerability that allows

containers to mount the host file system into build containers. So the container then has readrite access to the host file system which causes a container escape. In this one, we have this is a vulnerability in the NVIDIA container toolkit that exploits a time of check time of use vulnerability so that a specially crafted container can gain access to the host file system. This one we have leaky

vessels. This one, this one had a name. Um, it's a container escape in runc and buildkit that enables an attacker to gain access to the file system on the host operating more. This one's a vulnerability in Docker Desktop that lets Linux containers access the Docker engine API and execute privileged commands um including control of other containers and sometimes the host file system in in certain cases. And

finally, another one in Nvidia container toolkit that exploits um a flaw in hooks used to initialize containers leading to privilege escalation. Really not trying to call out any of these projects. I think my point here is more that this is this is a thing that happens to different people because this is a hard thing to do to kind of maintain this container boundary. And really the point

is that this kind of boundary that we've made around containers as I'm going to try and convince you of today is not really a security boundary. It's a mechanism used to control resource usage. And this is kind of why it was first created. And we've kind of used it so much that we've come to think of it as a security boundary. That's not really what it was

first created to do. So to try and convince you of this, we'll go into a bit of history. So first we have some kind of prehistory. Um before we started talking about containers. Um well I guess before this we you know we had computers. We had processes running on computers and operating systems. And then people wanted to start sharing these computers between multiple different users. And one

of the first ways people did this was with virtual machines. The basic idea you want to isolate different processes and different users. Bring a whole operating system, right? Have and this was really common um really even in the early days of of computers because computers were big and expensive and so like universities would have them and have what they would call like terminals which are basically virtual

machines into this big computer. And then so when the idea of containers came along and we'll talk about more of the history in a sec is the is this idea of operating system level virtualization. So instead of virtualization of the entire operating system, you do virtualization inside the operating system. So you get kind of like that picture we had at the beginning, right? Where the containers are

all sharing the same operating system. This is great for um performance, right? Because you don't need to bring a whole operating system and it was very convenient and so folks really liked it. Um and this kind of promise of a shared kernel but process isolation, right? So this can be really cool. But how did we get here? So, a lot of places you could start this history.

I think that the origin of this idea of OS virtualization kind of started with the chroot or chirut, I'm not sure how people say that. Um, command in Linux. Um, this was introdu introduced into actually Unix. Um, that's how old it was in 1979. Um, it's a command and um system call that changes the apparent root directory for the current process and its children. So there's some

connection actually between this command and early uses of the term jailbreak um to refer to processes that bypass this command and kind of break out of the the the chroot jail. Um so that's kind of where this idea and that kind of later led to the idea of a container escape which I think is kind of interesting. But anyway, the chroot command doesn't prevent tampering by root

and doesn't prevent a chroot user from creating node devices mounting other parts of the file system on them. it really only focuses on file system isolation um without thinking about things like system call and network functionality. Um but those are kind of the first thing that had this this idea of um operating system Moving forward a few years um FreeBSD jail um allowed system administrators to partition

a FreeBSD system into independent systems which they called jails. And this was first introduced in 1999. And these jails were a lot more like virtual machines than what you think of as a container today. Again, we're a little bit before the the main idea of a container. Um, HGL provided a virtual environment with its own file system, its own processes, its own user accounts, unlike again containers

today. There's a lot of limitations here that were addressed in the future text that we're going to talk about. For example, you can't mount or unmount file systems to a jail or access device nodes. um these but this was one of the first ones that was actually designed to separate customer workloads which is kind of Next we have Solaris containers. This one has a a naming confusion

problem. Um but anyway this combines resource control with boundary separation provided by what they call zones and this is the naming confusion. So this um the idea of Solaris contain so containers were first released in 2004 and this provided just the resource management and then there were Solaris zones released in 2005 um and then together they became known again as Solaris containers in until uh 2011 and

then they switched back to Solaris zones. So let's just say they're the same thing. Um call it containers, call it zones. Um these containers had the ability to assign storage and resources to a container. Um they didn't have a privilege mode though and so some operations that require operating system privilege are not actually possible in um Solaris containers. Um like today's containers they did run on the

host operating system and allowed both resource assignment and boundary mechanisms. So it's trying to get closer to our kind of modern idea of containers and this I think was the first use of the word container for this. I could be wrong about that. Sorry. Um, and we have OpenV, Open Virtuoso, um, which lets a server run multiple operating system instances for different users. It's a little bit

closer to the VM idea. Again, it also allows all guests, um, to use the same Linux kernel with kind of separate views on files, users, processes, etc. I provided resource management to prevent resource hogging, which is something that, you know, is very common in today's containers, right? You don't want noisy neighbors, etc. Um, and originally this actually used chroot to share the file system that later introduced

kind of an actual separate file system per container. yeah, and this was released as patches to Linux kernel, but it was never actually merged into the official Linux kernel. So, it's kind of its own kind of independent point in time. Next, we have Linux containers, which were first first came out in 2008. Um, this was a year after the release of Croups. um which I think was

another big moment for kind of what we think of as containers. Um these Linux containers combine croups which limit and prioritize resources um with namespaces which provide kind of a separate view of the operating system environment. Mount name spaces are from all the way back previously in this chart in 2002. Um but other namespace types were added in uh 2006 and onward. Um that really increased the

usefulness of namespaces to do what we we we think of today as like making them useful for containers. um like the kind of process isolation, user isolation and this kind of combination of croups and namespaces in my opinion is what kind of defines the current era of containers. This is really the beginning of that. Um and this project is still alive today. This current um work on

it such as Incas I think is how you say that which provides kind of more modern cloud functionality on top of the Linux container system. And finally for this short history um we have 20 2015 um where so docker actually originally used Linux containers as its execution driver but then in 2015 they deprecated the use of Linux containers in favor of lib container um which worked with

Linux namespaces and croups but also with capabilities security profiles network interfaces again this is going to start to sound familiar to folks in the industry today and there was a lot of industry standardization around docker and lib container And this is kind of when we started to see this ecosystem around container management. This is really close to when Kubernetes was released etc. Um and this conference today

kind of points to the success of this whole endeavor. Um lib container was later actually split into run C and container day and containerd in 2016. So you're actually still using sort of the code from from that today. So where did that all lead us to? There's a lot of history. So we have operating system virtualization with name spaces, cgroups, capabilities, security profiles, network interfaces and more

to kind of really provide this separate view for our containers. What we haven't changed is that we have basically processes running on a shared And so this is where I'm going to make this argument which is that um container security is basically comes down to Linux security. And the reason is that all of these different features we talked about on this previous slide, name spaces, cgroups, etc.

are actually features of the Linux kernel that we're using to kind of create this view of containers. Um, and so when something goes wrong, um, container escapes can happen because if if anything goes wrong in the Linux kernel either with these mechanisms in the kernel or with something else, that's when you start to see container escapes and stuff happen starting to happen in your processes, right? Right?

So even though we've built all these abstractions and ways to build these containers over the years, at the end of the day, um you know, we still have this shared kernel underneath it all, and it's important to kind of understand what that is and how this is where um all these issues can stem from because as we talked about before, these container stapes comes from things like

file system access, which is on that that shared kernel, violations of namespaces, croups, etc., which again are features of that kernel. Network access which goes through the kernel. I don't know if you're noticing a theme here. Privilege command again privileged commands are executed by the operating system in the kernel. Um and all this is again what what Linux security is. So if you're thinking about Linux security,

you're thinking about all of these things which means that you know understanding container security means understanding these same things as understanding Linux security. So I think that there's like opportunities more for for conversion here. Okay, so moving on. So the scope of name spaces in croups. So these are in the Linux kernel which means that they can actually only isolate resources in user space because of you

know the way the design of the stack is. You can really only isolate things kind of above you in the stack. Um anything in the kernel itself has has permission to bypass these protections and this is again where a lot of these container escapes stem from. So while croups can provide networking allocation, it doesn't actually provide like security isolation in the kernel for the networking that just

does that in user space. And we see this sometimes with things like um confused namespace assignment for file descriptors which um can bypass the isolation that's supposed to be provided by by those namespaces. Okay. Yes, I think we're doing good on time. I have a bit of a demo. I'm going to try to switch to mirror mode to get this to work. We'll see if that works

well. Um, let's see. Okay, I think I need to make this a lot bigger. How do I Why would it not let me zoom in? Sorry. One second. Okay, let's see. Does that look big enough to folks if I do it there? Okay, cool. Sorry, I can't I can't really see that screen. So, let me know if it's like tiny and at the bottom. Okay, so let's

do that to get to the top. And then I recorded this so I didn't have to try and use conference Wi-Fi, but this is a real recording. Um, okay. So, we have this is on my virtual machines because I didn't want to do it on Mac OS. We have um I'm going to build this is just an unprivileged Docker container running Alpine. Really straightforward to try and

just show that we're actually in a Linux kernel here. So I'm just to show a couple of um things. The first one is the so in /prock you have a bunch of info that you can see. Again this is inside an unprivileged container about the host system outside. Um so this is the CPU info. I it's really long so I I made it short so hopefully you

can see it. Um the um so you can see that you know you can tell what kind of processor you're on. That one's a little less interesting. This one's really interesting. You can actually see the memory info of the of the the host system. And this is kind of interesting for kind of watching what's happening elsewhere. Modules. I think I have another better view of the modules

from slis there which I spelled wrong in my recording. watch there it is. There there it is properly. And what you can see here I'm going to pause for a sec is that you can tell kind of things about the the host system. This was my dev machine and so you can tell that it has like some Zen drivers installed from um other work I do on

this machine. Um, yeah, so you can see into the again stuff in the in the host kernel. One of the things that I found really interesting here that kind of shows that this is the external kernel is that you can see how long this kernel has been running. And so we just spun up this container, right, like two minutes ago. But as you can see, it has

like a really long U event sequence. So it's been running for a long time. So you can see like into the outside part of the kernel. This one here's a little interesting too. You can see again this is on the host system. You can't actually you don't have permission in an unprivileged container to read all the memory but you can watch it change right? You can see

how this gets updated over time. Um this is a kind of fun one. Um this is actually kind of has to do with more more with an implementation detail of Docker than with um containers in general. But um if you have this again this is an unprivileged Docker container. So, what we're going to do is we're going to use TCP dump to watch the network traffic um

on this computer. And then you won't see it here, but I had another window open that was but just running a ping command. You can actually see it show up here. Even though that was outside of the container, you can still see the the kind of network traffic again because it's all sharing the same underlying network. For some reason, this was really slow when I recorded this

um a couple days ago. It was much faster when I did it at home because it must have been the Wi-Fi issue. But anyway, you'll see it in a second. Um, but yeah, the point of this, and this isn't really like I think demo is a strong word for what this is, is just kind of looking at how much of the Linux kernel. Yeah, there's the the

commands. This is again that was from a ping on the host system that you can see here from inside the container. Um, yeah, demo is a strong word. The the point here is really just that um you when you poke around inside your container, you can really get a better feel for how much of the host operating system is really impacting your container is around in your

container and how that all um I don't know how it all works together. I think I think this is a lot of fun. I think this is a great way to learn how your systems work is to go around just see what you can see, see what you can learn on your own. Um, let me switch this back hopefully to this. Oh yes, that worked. That's so

great. Okay. Um, okay. So now that is what we have in containers, right? We have this this idea of um process based isolation inside the kernel, but the kernel is still in there, right? You can kind of if you look around, you can kind of see that it's still still in there. And so this kind of brings us to the what I'm going to call the post

history of containers. I know it's still we still use containers but anyway it's like postmodern it's actually just current. Um so the u these is technologies that are being used today um with or instead of containers to kind of um try and give an even stronger view of that kind of sec kind of create that security boundary right to try and kind of pretend even more that

we're kind of in a container on is is kind of its own So a couple of these that I find interesting. So we have multi- kernel which treats each core as its own machine and then kind of communicates between the cores with message passing. It's a little bit I mean more removed from the idea of containers but kind of does a similar idea. Um behind the scenes

we have sisbox which is a container runtime again compatible with um tools you use that treats each container as a virtual machine using a sorry using a form of rootless kubernetes. And so again, this is like works with Kubernetes but kind of moves a little bit more towards that idea of VMs. And this is kind of something I find really interesting about this kind of post history

of containers is that in the prehistory we started with virtual machines. We have these containers, we've built up this idea of containers over time and then we build it up so much that it kind of becomes like virtual machines again, right? So we're kind of coming full circle on this idea. Um yeah other ones we have we have G Visor a container sandboxing that does what they

call user space system with it's like a user space kernel which is actually just user space system call filtering that adds some additional isolation of of that layer um user name spaces this is actually in Kubernetes itself um was recently added to let a container run processes as root without giving it root permissions on the host this is like a really big step forward to kind of

making that container look like its own little system and I also think It's a really interesting call back to chroot right at the beginning of this history which enables running more powerful applications without actually giving the the container root on the host. Then we have things like microVMs, things like kata containers and firecracker that let you run containers inside the VM. So you can get an ecosystem

of containers but kind of this isolation model from VMs, right? Going back to this idea of like a whole kernel to run your um process. And finally, we have web assembly or WOM, which is a binary format designed for compatibility across systems, similar to the um compatibility of containers, but instead of like running your packaging your application into a container, you can pile it into WOM. So,

kind of a similar idea there. And it's kind again, it's kind of part of this this post-histry trend, I would say, of moving towards um stronger boundaries around containers. So, what does this all mean to you for you in in this audience today? Um, I think one thing I would say is just to really encourage anyone who wants to understand containers and container security to learn more

about that underlying Linux kernel. How much of it you can see from inside your container, how much and and how that kind of understanding can help you really not just with security but also with things like performance because again that that that underlying kernel is what is helping you interface with the CPU with the memory. So if you're having issues with performance of those things, it's nice

to understand how that pass through is happening, what's underneath these abstractions that we're using. Um and this also helps you understand prevent you know noisy neighbors and and you know containers getting access to more resources than they should have. Um and the other thing that this means is that you know operating system virtualization has been built up over over several decades. Um and we can keep making

it better right we can keep improving on this idea of containers how they work inside the Linux kernel and how um we can use them kind of securely and usefully. And finally, I just want to bring this back to kind of some more modern ideas, I guess, to see kind of how this relates to to you today. Um, and this idea of kind of AI agent sandboxing.

So, there's a lot of folks in the world today, maybe all of you in this room, um, have been playing around with this idea of autonomous AI agents. These are um, AI systems based on these LLMs that um, are able to execute tasks independently. And this is very powerful, right? So if you um you know they can go off and they can do things without you having

to constantly watch them. But this means that they're able to run independently and you know they can mess stuff up while you're you know off going for walking your dog. And so it's nice it's really important that we can kind of not just sandbox them but really understand what can these AI agents do if we try and sandbox them inside a container. I think that's where this

becomes really important because they're non-deterministic. Um, and we've seen all there's there's a couple of really cool um like demos that people have done where they set AI agents against each other to try and like attack other agents in other containers and stuff like this. And it really helps kind of show what the what a system that's able to kind of play around with all these boundaries

is able to do. And so when sandboxing these AI agents, I think it's really important to think about these things. And this is I think one of the reasons that all of this is like especially relevant today. So anyway, thank you all for coming. Um, I have the wrong name on there. I'm going to skip to that. Skip that slide. Um, and um, I think we have

a couple minutes for questions. And so there's a mic right over there if folks have questions. And yes, thank