Hack Me If You Can: Learning Kubernetes Security Through a Role-P... Aoi Takahashi & Keita Mochizuki

This talk explores Kubernetes security through an engaging role-play format, highlighting the journey of building and securing an e-commerce application called KubeCon Shopping. The speakers, Aoi Takahashi and Keita Masuzaki, emphasize the importance of securing Kubernetes environments against potential attacks, particularly those exploiting vulnerabilities in applications built with modern frameworks like Next.js and React. They illustrate how to manage Kubernetes secrets with HashiCorp Vault and discuss the critical steps for securing container images, including scanning for vulnerabilities and employing multi-stage builds to minimize attack surfaces. The presentation also covers the role of AI in both aiding development and potentially introducing security risks, along with different methodologies, such as the Pod Security Admission framework, to implement best security practices in Kubernetes. The session aims to enhance awareness of container security and promote proactive measures against hacking attempts.

Hello everyone. Welcome to our show Hack me if you can. Uh learning Kubernetes security through a role play battle. My name is Aoi Takahashi. I am a site reliability engineer from Recruit. Uh I sometimes contribute to Kubernetes docs which I translate the document to Japanese and I came from Japan. And I live with two dogs. I have published a professional publication about uh learning Kubernetes for beginners

and I also self-published a manga book for also for beginners, too. Hello everyone. My name is Keita Masuzaki. I'm a software engineer at NTT Data Japan Corporation and my responsibility is um platform development, technical research and offerings. Besides my work, I published some Japanese books on learning Kubernetes security from risks, um >> [clears throat] >> manage Kubernetes secrets using HashiCorp Vault. Um I sometimes contribute to upstream

Kubernetes and I'm a member of both Kubernetes and Kubernetes SIGs member. Uh recently, I have been contributing to the development of Kubecolor dev up command. Today, thank you so much for coming to our presentation. Let's start. Well, for the target of the session is people who know the basics of Kubernetes. We will not explain what a pod is, what a service is, etc. And people who use

Kubernetes as a platform, build the application container image and deploy it as a pod in Kubernetes. And for beginners in Kubernetes and container security, we will explain the fundamentals of Kubernetes and container security. So, our presentation consists of three chapters. And these are the three characters which we will play today. Engineer, I as an engineer and he as a hacker and there's a hero named Mochi. And

everyone, you are all part of my developing team and let's work together to drive this development forward. And there's one disclaimer. Do not attempt to use AI for malicious attacks. So, all right. Let's start the developing and I am going to build an e-commerce website and I will start developing using AI. uh hello AI, please create an e-commerce web application called KubeCon Shopping. AI, let's go. Okay.

A few moments later. Okay. We did it. Our e-commerce site is now made completely so quickly. Yay. And I've created a pull request for this code and everyone, please approve it. Well, you're saying you can't approve it without reviewing the code. Well, it's fine because an AI wrote it and it's smarter than me. So, everyone, thank you for the approval. Let's merge the PR. Merge. So, I

created the code and next what we need to do is creating the container image and push it. So, hello AI, please build a container image for this web application. Once the build is complete, please push it to the container registry. Yeah, okay. Thank you. And after publishing the container image, we will deploy the manifest to the Kubernetes cluster. Uh please deploy a pod to Kubernetes using our

web application container image and make it accessible from the internet. Okay, I've done. Okay, thank you. So, this is the Dockerfile AI made. Well, I don't know what's written, but it's okay, I guess. And these are the manifest. Okay. So, I created a pull request for the this code and everyone, thanks for the approval. I will merge the PR. Okay, production deploy succeeded. So easy. Yay. Now

we can sell our merchandise and books on the service. And also, to show our gratitude to our friend Inductor, we'll be selling a special drink uh featuring his icon illustration. hm? Oh. I know a little about Kubernetes, so I could understand about the AI AI architecture AI made. The architecture is as shown in the slide. The application is deployed as pods on each node which includes actual

application container. And finally, each pods are exposed via service resource. Well, nice. It's so easy because the AI wrote it all. So, maybe we all can go for a vacation. Nice job, everyone. Hello, I'm a super hacker. Attacking server targets is my pleasure. So, what should I attack today? Oh, I find a new website. Oh, it seems to be an e-commerce service. I bought it. Um let's

check out it. Uh I used to do research by myself, but now I can do I can do it with the help of AI. So, let's ask AI to conduct an investigation. Okay. Find out what kind of web service it is. Um determine what program languages and frameworks are used. Hello, I am AI. This web service appears to be a website offering e-commerce services. Additionally, the following

technologies are utilized in the web service, Next.js and React. Awesome. Hm. Come to think of it, uh I recall hearing about a vulnerability related to um React a while ago. So, let's investigate it, A critical vulnerability identified as CVE-2025-5102 has been discovered in Next.js and React server components in 2025. Yeah, great. Um CVE, Common Vulnerabilities and Exposures, is a globally unique ID assigned to a publicly disclosed

security vulnerability. Um by exploiting this vulnerability, um remote code execution of arbitrary command can be allowed. And the score expresses severity of vulnerability um from 0 to 10. And the score of this vulnerability is Oh, sorry. Uh the score of this vulnerability is 10. uh which means more urgent. Okay, let's investigate whether this vulnerability may exist in the web service. This web service uses Next.js with App

Router and has React server components enabled. Although we cannot verify the patch status, there's a strong possibility that contains vulnerabilities. I see. Uh it might be worth trying. Okay, the investigation is completed. It's my turn. Okay, so let's attack it. First, let's prepare the tool for attacking. Of course, AI prompt make it easier and faster. Okay, I created a tool using Python to exploit the vulnerability. Do

you want to actually execute the attack? Please tell me which command you want to execute. Yeah, awesome. Uh AI created tool for me. Um it seems that the tool could be used to attack the web service essentially by exploiting vulnerabilities. Um next, let's actually try attacking. So, this time, I try to execute the host name command as arbitrary command um remotely using the tool with the help

of Okay, running Python. Uh done. Host name is KubeCon Shopping 5 F F R Z. Okay, uh it seems to have worked well. Now, I find that arbitrary command can be executed remotely uh inside the environment where application is running by exploiting the vulnerabilities. Like this. Okay, I do further more. I try to investigate the hosting environment and identify whether any other vulnerability um exists by using

the um remote code execution mechanism. Okay, it's highly likely this is running as a pod in a Kubernetes environment. It's running as a privileged container and the executing user is root. Okay, I see. According to the answer from AI, this service seems to be running at port on Kubernetes with strong privileges. So, it may be possible to interfere with the node from container. This means I can

expand our attack surface. So, let's try. From this container execute shutdown on that node. Let's go. Okay, using exploit.py, we will run the following commands inside the container and sent third {hyphen} t one {hyphen} a shutdown. Okay. Done. Successfully executed shutdown command for from container to node. Excellent. Everything is working well. Um this is the overview of attack. First, I prepare the tool for attack. I exploit

the pi and by using it first step one I can execute arbitrary command inside by exploiting it. >> And uh step two by using the mechanism of uh remote code execution, I can also on the node. Okay? Finally, I succeed to gain the ability to execute arbitrary command on that Uh this means I can perform various operations on the node. For example, um executing sensitive command on

the node, um stealing secret information from the node, um executing a malicious uh workload on the node, and attacking other pods and so on. Okay, the attack was successful. Say, we did it in Japanese. Yatta ze. >> [applause] >> Oh, no. I was attacked. Someone help me. We've been attacked by a hacker. Everyone, we need to call the hero to defend ourselves from the attacker. Let's call

Mochi. Mochi! Hi there. I'm a hero. >> Now switch my role. I'm not attacker. Now I'm hero, okay? Hey Mochi, thanks for coming and we've been attacked. Oh, it Oh. Oh, it seems your container security measures were insufficient. So, I'll show you how to improve your container. Okay. Um this time we divide our focuses into two area. First, uh container images and second, container itself configures with

Kubernetes manifest, Uh first, let's take a look at the container image. In this case, um the critical vulnerabilities was included in container application. And by exploiting this, attacker can execute arbitrary command inside the container, okay? So, actually the container also contain many components such as a command and two library and so on, uh which makes it easier to attack. Uh after the attacker compromise the container, he

perform various operations uh inside the container by using them, Wait. What? But the AI generated the application and container image. Can AI really introduce these vulnerabilities? That's a good point. Even AI can make mistakes. It's really important not to just blindly trust the result generated by AI, okay? So, especially in this case, relatively new vulnerability may not be recognized and implemented in application by AI. So, um

that is something that we should pay attention to. Hmm, I see. So, we need to be aware of two things. One, detect and remove vulnerabilities and two, keep the container image small to reduce attack surface. Exactly. First, let's take a look at vulnerabilities. Um there are some several points where uh vulnerability may exist. For example, application including code, library and so on. Uh second, um packages which

are added during the building process image building process. Um third, um base I see, but I can't pay much pay much attention to all of it. Okay, then uh scanning tool will help There are several container container uh scanning tool. Um they can help us to detect vulnerability in container image automatically. For example, um Trivy, um Kube-escape Um this is example of scanning container image uh using

Trivy. Oh, that's cool. The vulnerability that caused the attack was detected. So, it seems that we can use the scanning tool in our CI. Yeah, scanning tool uh certainly useful, but keep it in your mind. They may not detect all For example, vulnerability implemented by yourself into the application code may not be detected. Hmm. So, we need to be careful about the application code written by AI.

Exactly. Other point is to keep your container image small as possible. Okay, so you mean the large container images typically contain a lot of tools and are more likely to be exploited to attack, right? Yes, that's right. Uh looking at where Dockerfile, a relatively large container image used as base image. And the container image generated from this Dockerfile became so large. Generally, uh image large container image

contain many components like um some match command and a library and so So, it uh attacker to execute uh several operations inside the container. Okay, so how can we make the container small? Yeah, okay. A common approach is a multi-stage build uh using a minimal container image as the base image. Multi-stage build is a technique for uh reducing the size of container image. Um this is a

way to divide the uh container image building process into a multiple stages. Uh in this case, we divided we divide image image building process into two stage, okay? uh finally, um this smallest image is used as the minimal container image. And this smallest image contain only the essential components needed to run application and don't include unnecessary components like bash. Okay, so let's compare the size of regular

container image with container image based on distroless. Wow, the size is totally different. >> [snorts] >> Okay. So, it's not just small size. This means there are fewer components in the container image. So, let's try running bash in a distroless-based container. It will fail. Okay, so the distroless image doesn't even include bash. That means hackers can't execute any command even the container was infiltrated. But just think

about it. We sometimes use kubectl exec and when we need to inspect the containers. So, what can we do? We can't do it as usual. So, how should we do? Okay, I'll introduce you to the magic command kubectl debug. Um this command allow you to inspect the container with which does not contain uh shell or debug utilities. For example, inspecting the container processes, um file systems and

so on, like that. Okay, looks good. But what's the difference between kubectl exec? Like it looks very same. Yeah, okay. When you execute kubectl debug command, ephemeral container for debugging is added to pod, okay? Um and by choosing a container image for ephemeral container which contains shell or appropriate debugging tool, you can inspect the application container via ephemeral container, okay? Wow, awesome. Next, about container. Before we

go further, let's understand what container actually is. Isn't a container supposed to be an isolated environment like a virtual machine? I don't know why the hacker could attack our containers. Yeah, uh container is isolated process on Linux. Uh for example, in regular Linux environment, you can see all the running process on Linux host by using for example a ps command. But inside the container, you can only

see the process inside the container, right? Yes, I know Additionally, process running as container have limited privileges and restricted capabilities. For example, if you try to run sensitive commands like uh for example, uh shutdown command inside container, uh it may fail with a permission error. Okay, I understand that a vulnerability in the container image was exploited to infiltrate the container. But I can't understand why it was

possible to compromise the node from inside the container. As you said, containers are isolated environments, right? Okay. So, let's take a look at the manifest you used to deploy your pod on Kubernetes. Okay. It looks like there are host PID and privileged settings. Yeah. Uh these settings allow the attacker to interfere with the node from What is that? Yeah. Uh container is isolated process, as I said.

But, um these uh these settings include uh sorry, um these settings reduce isolation of container. Uh first, host PID um makes all processes on the container host to be visible from container. Um privilege to privilege true um grant all permission against the host to uh the process running as container. So, a container with these vulnerable settings allowed an attacker to compromise the host from the container. Yeah,

you're right. Of course, there are special case where these settings are necessary. Uh however, uh it's usually not needed when you use a you you want to run a typical application as a pod. Wait. Why was that setting included? Well, I didn't pay much attention what AI made. Yeah, as I said before, uh even AI can make mistakes. Uh however, it's unlikely that um the AI would

have added these settings on its own. Uh so, uh it's possible that uh a malicious prompt was injected somewhere. So, where can malicious prompts be embedded? Today, we won't go into much detail because today's main uh topic is container and Kubernetes security. Uh but, AI agent face various threats. For example, the uh AI agent itself may have uh been tampered prompt. And in other case, the uh

AI agent extensions such as skills or uh MCP servers um may be tampered with. okay. Uh and uh what's top 10 for agentic application contains a lot of information. So, if you are interested in, uh you should check it out. Okay. So, skill is a markdown file. I don't know. I know that, which contains system prompts. So, you mean if you copy a vulnerable skill from outside

our laptop inside the laptops, the agents might get exploited. And let's talk about MCP servers later. So, you need to be aware of security when using AI agents and related tools. So, let's take a look at how to secure container. When you deploy container as a pod, it's important to understand what configuration make your container secure or more insecure. So, there there are many various uh practice

related to this. So, this time, I'll share the pod security standard um published as Kubernetes official documentation. Pod security admission Pod security standard define three different profiles depending on the security level. And each profile contain uh detailed recommendation setting. And this uh for example, the minimum uh required setting, for example, um privilege privilege and host PID should be false uh defined in the baseline profile. I see.

It's important to understand this to know which settings are safe and which are not. I'll introduce um two measures to secure uh container configuration. Scanning uh Kubernetes manifest and policy-based admission control. Um each means uh each measure is to check the configuration to of pod, but they are applied at different time. Uh first, there are various scanning tools to detect insecure container Oh, Trivy and Kube-bench can

be used to scan Kubernetes manifest, too. Okay. Uh this is an example of a scanning uh Kubernetes manifest using The setting privilege true is detected as severity high. So, with this result, we can notice insecure configurations before deploying the pod. Uh second measure is admission control, which is a mechanism to intercept the request to Kubernetes. Especially, um policy-based admission control um prevent non-compliant resources such as pod

from being deployed according to predefined policy. So, there are two types of control, validation and mutation. Yeah. Uh in case of body uh sorry, uh assume that non-compliant pod is to be deployed. In case of validation, um it will be prohibited and pod will not be deployed. And on the other hand, in case of uh mutation, uh non-compliant settings will be overwritten and pods will be deployed.

Um there are various methods to achieve Especially, pod security admission is a simple way to validate a pod configuration whether it comply with pod security standard. So, it's good for first step. Um but, uh if you want to customize your policy or implement a mutation logic, uh you need to choose other method. Here, we have explained the most common security measures, but there are other measures to

consider. For example, role-based access control RBAC is a method of granting permission to Kubernetes resource based on the role. Um by minimizing permission, it help it help to mitigate the impact of attacking. Um next, uh network policy is a method of uh controlling traffic flow by setting appropriate rule, unauthorized traffic um flow will be blocked. Uh for more information look at Kubernetes security. Um please see the

official Kubernetes security checklist. Since we were attacked using AI, can we defend ourselves using AI, too? AI, too? Yeah. Yes, we can. There are various way to do this uh but by AI. For example, um detecting insecure setting and generate policies and detect and analyze suspicious behavior um uh security alert triage and so on. Okay. So, let's look at detect and fix pod by AI in the

remaining time. Previously, we need to run a scanning tool to detect the vulnerabilities, then understand the details and fix them. But now, do you mean we can let AI to do it for us? Yeah, exactly. One of the uh powerful method is connecting MCP server to AI agent. Okay. I know about what MCP server is. Uh that extends AI agent capabilities by providing external tools. That means,

by executing tools implemented on MCP servers, AI agent can actually perform the actions. Furthermore, AI agent can use the results as context to understand the situation more accurately accurately. And with the MCP servers, we can empower the AI to improve our security, right? Exactly. For But, uh for your reference, it might be possible to achieve uh this without using MCP server. For example, uh combining uh skills

and CLI. Um but, one of the benefits of MCP server is uh their high usability. Uh we can implement the MCP server by ourselves, but there are several um MCP server as an open source or proprietary software. By utilizing these feature, uh AI can understand more accurately what mitigation are needed and implement them. Uh for example, by connecting Trivy MCP server to AI agent, AI agent can

execute Trivy to detect vulnerability and misconfiguration from Dockerfile, container image, and Kubernetes And AI agent understands the result and try to remove vulnerability or fix misconfiguration. Well, that seems AI can make me easy to scan along using AI when developing. Victory! >> So, >> uh thank you for uh coming to our show, and please use this QR code to send our feedbacks. And maybe we can take

a picture because we had won the hacker. Yay!