Tutorial: Attack Defense: Leverage eBPF To Reveal Attack Flows... Constanze R, Markus G & Teodor P

This workshop focuses on a gamified approach to understanding and utilizing eBPF technology in defensive and offensive strategies within a Kubernetes environment. The speaker discusses various tools and strategies for both attackers and defenders, emphasizing the importance of community-driven methodologies. Participants engage in hands-on exercises using eBPF-enabled tools like Tetragon and Pixie to simulate attacks and defenses, exploring techniques such as anomaly detection and reverse DNS scans. The speaker details how adversaries can exploit misconfigured service accounts to gain unauthorized access and execute commands, highlighting the nuances of Kubernetes security and the necessity for robust detection mechanisms. This session illustrates practical implementations and challenges in maintaining cluster security, offering insights into real-world scenarios.

Ah, hey people. So, why um people are still coming in? Welcome to the CBPF attack defense gamified workshop where we will let you attack each other or defend your systems hopefully with this very very serious uh spell book uh as a manual to help you. Um yeah, so first let me introduce the people that are with me on stage. We have Marcus Gillinger as the monster by

the way. actually one of the nicest people I know and this is uh the author of EVPF Turpp. This is the Adopnik who will be covering the defense in the second part of the course and and I myself I'm constants. I'm a CNCF ambassador and a total EVPF fan girl. And yeah, with that being said, I'm going to walk you through the logistics a little bit because

that's going to take some time. And then we will give you options for playing in pairs, playing single and also uh just to read along if you have issues with the Wi-Fi or you know as things may happen or the battery drains. I heard there are no adapters unfortunately. So with that being said um Marcus will guide the screen. Um so we are playing this fully hosted

so you should not be able not have to set up anything locally. This means also we have a little bit of Wi-Fi dependency. So yeah, that's just that. Um, in order to get a seat, um, the first thing is in the schedule as well as spelled here as well as on the QR code on the first page of this leaflet there is sends you to a training

course. The active player needs to or later on if you want to play it uh needs to have a GitHub user in order to sign up. There's no purchase necessary. This is completely community sponsored uh not sponsored. This is communitydriven. Um yes. Uh one thing please uh given all the conference situation, please be patient. Do not rage click on buttons. This might uh yeah we will all

have a good experience if we have a little bit of patience. Thank you. So with that being said, please please join and then once you have joined, press start and then there will be a boot process uh showing up. Um that looks like can you can you show what the playground looks like please? the start lesson already. Yeah. Okay. >> Okay. This is >> Yeah. >> It

works for >> All right. We might have to switch. We have two more backup laptops. Um but okay. No, that's >> It was just the Wi-Fi. >> It was just the Wi-Fi. So this is what it looks like when it boots. And um so we will do some exp. So um you will actually be using production. So actually if you're here to write uh C code in

order to compile and um get you know some EBPF out of the kernel that's the wrong workshop. So, we're going to actually show you how you can use different strategies of using EVPF in production ready tools, how they differ, how you can see an entire attack um chain down the road, and how different strategies work differently and as a as a defender, how you deal with noise,

etc. So, this is why we'll also show you the um the full installers um so that you can actually see the settings. uh we have chosen the settings to be lightweight and not noisy and so you can actually um later play along or replicate this on your own systems at home. So during the time that we're booting um so I would like to actually mention that we

have the so so we are working with tools that we did not invent. We're just using them all of us and they're the real maintainers or some of the actual inventors here. So in order of mention there is Mah, come here. There's Mah for for Tetragon. Um then we have Yes, Matias. Come on. Then we have the lead maintainer of Cubecape. Uh if you have so also

if you and M um for for inspector gadget which you will meet in the most complicated um and quasim uh for inspector gadget. So this also gives you a unique opportunity to actually talk with the people who invented this. So thank you so much for for doing all this great work. Um so again this is communitydriven um and this is a great community. Let's get started. So

thank you so much all of you. Thank you. >> Perfect. >> Suring it up. >> Yeah. So now we're going to um show you a little bit how this lab works. Um so the first thing that we're going to run is the first installer um which is actually node agent for which is a part of cubecape. This will take a while. So what you see Marcus do

now is he just simply copy paste it right like blindly copy paste. So that should work. So we also encourage you to not modify the commands in the first page of this uh work walk through because we cannot yet help you. So later in the course uh and in the in the workshop you're very very welcome to experiment with this but while we're doing the the the

sort of the explanations we encourage to do it onetoone. Okay we have a conference Wi-Fi situation going on here. Um exactly. So copy paste and what will happen eventually is that this um bluish uh blinking button is going to turn green. Um so the other functionality is like how do you co-play and how do you use these labs? So I'm actually going to do the defense on

Marcus' laptop but if I was playing with him as you are playing maybe in pairs. So there are two functionalities you just saw that you can share ports with your friends. You can use this also to to debug together or to co-develop. Um in this particular case, we will give two roles. Um so you choose if you have the offensive role or you choose you have the

defensive role and you can catch the 8080 port for the attack monster. This is going to be symbolized by this net here. so you have to ensure like you enter the port 8080 here and ensure it's public. So toggle this switch and then you can expose it and you will see this URL down here and this URL then you can share with your partner. So one like

I will just open it locally here. Uh but so the defender then will see the full screen of the tutorial whereas the uh attacker will just see the the attack UI itself and you of course you saw it. You can toggle the light and dark theme. So by default it will be dark but you can choose whatever. >> do this. >> So you will also be able

to share a terminal. So that means so so the button pressing on the UI if you have shared the run UI. Um you can both press buttons. I should say here run is the only thing that is stateful. So the attacker tool is stateful. So clicking random buttons there may lead to undesired consequences. Um whereas the other stuff is stateless. So and the commands are item potent.

So um you can and now we're also going to show you how to share the terminal and that means that your cubectl whatsoever will also be shared. So you the only thing you don't have the text uh the split text. So um basically this other button up here you can basically create the URL to your dev machine and share this terminal then as well. So so this

way both can access both And here there's also you might notice you have a dev machine and a control plane. So the dev machine has helm k9s the usual stuff installed whereas the cplane does not. However, if you want wanted to debug that the ebpf probes are attaching etc etc you would be going to the cplane which is so this is a very very squeeze kubernetes as

you probably noticed because we're running everything on the cplane. Um but this is also if you had a full cluster you would share the the actual Kubernetes nodes with the other person to uh do some debugging on. Great. Now >> node agent is ready. >> That is fantastic. >> So we can apply the application profile afterwards. >> So here what you see is a manifest that you

might want to come back to later on when we're talking about noise. Um for now let's just be mentioned that there is a application profile a CRD that we're applying that will become uh relevant later on and this is where to find it is in the first gathering hall. Now we're going to do another tool and Dom Delano uh did not make it in person because he's

preparing his own talk. Um that is Pixie. Pixie is actually not a security tool, but it's very very ebpf native and its installer um takes a while and this is also because it has two components. There is the cockpit component which actually runs in San Francisco um for which you were asked to do a email sign off for which you need a token. Um do I need

to uh MFA this? >> Yes. Okay, right. >> Oh, does work with >> Okay, so this crashed. We're totally not sharing accounts. So, so here we have two parts. So we have a demon set that runs in your nodes on each Kubernetes node and it has metadata components that will distribute the query that you run in the UI or you will later see that there is a

CLI um that communicates with this metadata um components and it pushes down the um the query to the node these node modules Pixie edge modules that are being installed here. So the this is the only tool where we now have a full production ready setup. You will have the UI where you could be monitoring I don't know what is the limit many many clusters at once and

connect different clusters all with their individual edge node edge modules to them. And the interesting part about this Pixie architecture is that the data actually remains on the nodes. It's node local and it's only manifested in the UI if you actually query it. So if you uh as always query select star from star that will have exactly that effect that a lot of data is transferred but

otherwise the data stays per default 24 hours cached on your individual nodes and this is why we have a lot of intermediate components that are going to be installed here. You might see a warning pop up that the kernel headers are not there. That is again for size reasons that we took them off. Uh since we are not live compiling uh BPF, this is a feature that

Pixie also has. You can live uh compile just in time compile. Um then you would need the headers. We are not using that feature right now. And this is why we where you can ignore the headers being missing. So um and now coming actually in reverse order of usage to um the probably I'm I don't know I'm now going to make enemies the probably most famous tool

um with with tetragonon here having the installer you can open a new shell here the dropdown and again def machine you want to use because otherwise you don't have helm uh installed now we have almost everything ready, >> right? So, let's see. So, for both tools, we also have this uh hint box here. So, when those are ready and they turn green, then we can proceed. In

the meanwhile, while those tools are being installed, um we have like one special part which is our entrance uh to this dungeon um which we can also apply already. So, we will provision this and wait until it spawns up. And in the meantime I can I can give you a very quick tour and explanation how uh run works. So um we saw just now all the defense

of defender tools but um for emulation or adversary emulation we need something as well and run is basically the name is inspired by the Norse goddess of uh the sea and she's associated with all sorts of sea calamities. So like that's a very good fit for Kubernetes especially when it comes to the dark side of it. And specifically here um what you should know or what helps

to know is that all the TTPs in there or all the techniques in there um basically uh adhere to the migra attack framework. So you're not like you don't need to know that but it's basically uh a classification of all the adversary behaviors observed in in the wild. So it's based on real data and it's basically a classification of these behaviors in tactics uh techniques and procedures

and what we use is basically just uh the grouping by the tactic. So we have some sort of orientation around this and yeah all that you should take away from that is basically that um it's effectively um what vulnerabilities and CVS are is the same with attack and adversary behavior and we can access the adversary part or the the UI um using the dedicated tab or the

tab you shared with your partner and it's a very simple straightforward UI basically you have a graph in the center which you can if You click on it, you can pan around, you can zoom like later on it will get fuller. Um you can click on individual nodes and then to the left you have the armory which is basically the collection of all the techniques that are

available at this point in time. Um yeah like if you click on a node you see on the top right hand corner some information that is already available and known to run herself. Um and the core design principle is effectively that it's intentionally restricting what you know because an attacker doesn't know your environment. So the uh adversary emulation tool shouldn't like magically know all this stuff as

well. Then throughout the the session and you will see it later on if you need to debug something or want to see what kind of ex uh uh behaviors were executed you can toggle to the flow part and there you will see basically the attack flow um rolling out what you did in which order and what succeeded and what failed and we will come back to that

later on. Right. So here's a a screenshot what it may look like later on um throughout our adventure. Let me check real quick what the state of the tools are. Yeah, wonderful. >> Good to go. >> Okay, so the last point of logistics is basically the dungeon map. So, um what lies ahead of us is basically um a dungeon with several rooms or modules. We are currently

in the uh first room in the gathering where we set up everything and all the the installation and tooling. And then we will proceed to the initial access the gate. Um and yeah we have for every room we have an attack part and an adversary part. So I will basically in uh be the attack part and then my partners will try to see what I did in

that room and we will have this dynamic back and forth and we will proceed throughout the rooms. Um we hope to get definitely to the the fifth room and the sixth one depending on how we how good we are on time. We will uh cover this or that's up to you as an exercise for you to explore. Right. So with this and all the tools set, we

can actually now click on the Kubernetes node and start with the initial access. Let me go over here. So like for all the room part basically we also have a description of all the steps to take in case you get lost. You can just follow along there. And if I may say there's often a um attack manual if the run UI works, do not use it. Just

completely ignore it. >> Right? So uh depending on when you uh spawn the pot, you either have to wait a little bit or it will become available. But there you should select the entry hall pot and you connect to it. So what happens here is run creates a a connection to this part within that cluster. So it's basically all nested. The outside is the cluster itself. Within

there you have the name space and here is the pod. And if there is such a red color then it basically means it's compromised. It's under the control of RAN and we will see throughout the the entire adversary emulation but all actions we take in RAN will basically originate from RN herself or from this node and then propagate through the network we build up and now with

this initial access nothing happened but we can do our first activity. Um so this was purely informational that there is a linkage but we can do our first step by just quering and um inferring where we are in the environment. Let's see. Yeah. So you saw on the top right corner that the information we got now from the command updated and yeah we learned we are uh

the username root which is not much but we can see if the defender can already have a takeaway of this or yeah um so this is sort of like a really bad scenario that I lost my cubectl cube config somewhere. it got somehow dispersed on GitHub maybe or somewhere else um or baked into a image an OCI image somebody untargeted found it and now they have you

know scanned the internet and cubectl uh yeah exec let's see what we see and uh Marcus just did ID as the command and it's now my job to check well uh how could I find this and so we now have always this apprentice path and champions path and that doesn't necessarily mean that the commands are harder. It's just more that you it requires more background knowledge to

necessarily fully understand what's going on. So, we will start these rooms. They're going to be increasingly difficult and there's going to be a lot coming up in room five. You'll have a lot of pots and you will have discovered a lot. So, we're going to start at the absolute basics and you can build on this. You can always come back to this and repeat it slightly modify

it or just take off the the things that you applied with cubectl apply. We're going to start with our first um defensive mechanism which is a tracing policy in tetragonon. So this is an ebpf workshop without writing any C code. Right? So the first abstraction we're using here is a CRD called tracing policy. And what we're hooking into here is a very low-level object like it's a

sys call. It's it's a krop that we're using. Um and when you see here what we're writing down it's like what we're going against a k probe. we have the the name of the SIS call and then we actually need to know what arguments we need to to give this uh tracing policy what binaries we're selecting and um what values of the return arguments we need to

match. So the the idea here is so we're going to get data out of the kernel. It's going to tell us something. We're not writing um any C code. So this is basically the the compilation and the um hooking safely into the kernel is abstracted from us. But unless you happen to know this kernel strct, uh it might not be the most obvious thing to guess that

this is exactly the correct way to apply this. So by now we're going to do our um blind uh copy pasting here at this point. And so you take the the deaf machine at this point and you apply this tracing policy and this is just going to log into standard out. So what we're hooking here in this uh so what binaries we're actually um making active this

sys call hooking is um the binaries that are in this array here which are different forms of shells and what I want to show here is first off that if we uh oh yeah um yeah that we don't see anything because yeah what am I looking at I'm looking at bashes and I am have um uh put this tracing policy as active after the attack happened. So

now I am going to actually repeat the attack which is always challenging for me. Um so I'm going to again click on this part say get ID. This is exactly the same as as Marcus just did. Um execute it again and now I have a JSON document being rendered by JQ. Um and what I see here is that this is the name of the policy. I see

the architecture etc. I see that yeah the architecture here of the sys call this is implemented in x85 uh sys calls are implemented differently in the different CPUs. Um I have these two arguments being returned to me and also interestingly enough I can scroll up a little bit into the parent uh history and I can actually find out how was this how did this attacker implement. Just

seeing if we can increase the >> Oh yeah. How is the readability on this? Actually, could we get the the lights dimmed a little bit? Is that No, we cannot. Sorry. Thank you. Thank you for the feedback. >> Um >> that's not implemented right to increase. >> Uh if you do this locally, um uh I will just remove jq uh without the color. So if I just

remove this, do this again, then it's in white at least. So um so what you can see up here is now that the way the it was implemented was um okay I'm going to let you find this on your own shell. So I see minus C ID here. So user ben shell minus C. Um and this is actually quite interesting to find out like how exactly did

the attacker achieve this because if you did now the same attack with cubectlxec um minus minus bash so pod namespace minus minus bash this would not work. So in this case what we find out how is this actually how did did Ryan actually implement this and another thing I want to show you if you think like well this is really cool so I have a suspicious pod

maybe identified um I want to always actually see what they're doing in the in the bash or in the in the shells what is running there so you could come up with the idea well I am going to simply um copy paste no almost one sec foreign keyboard. There's still a till in there. Yeah. So if I was now to do actually a cubec oopsie cubectl xc

um minus end engine and it's called entry hall minus minus bash sorry Almost. Yeah. So, you can now actually see and if I had done this in a in a screen. Well, actually, I'll do this now real quick. It's better to see this side to side. So I'm now here logged in into the bash and I can now say I can basically um read the base 64

and uh encoded argument that is being returned by the sys call to myself. And now this could seem like this is a very very nice method of watching attackers type in the shell right this is a high density information piece. Um, yes it is. But now just imagine if you were hooking a standard out for every shell on every container in your cluster. I recommend to not

try this out right now um but at a later time. So um for now and in a later room we will um go through an installer and I encourage you to then again have this shell open and watch what the installer captures in in that open shell and you will see in clear text exactly what the attacker is doing. Now, however, I'm going to show you a

different type of detecting cubectl xcac that had nothing to do with shells. So, here we hooked um the cis write calls on different types of shells and returned the file descriptors. Um this is a very very high um information output type of detection. I can go a totally different route. I can say well there is the cubectl exec implementation that is a streaming implementation where cubectl checks

okay you have authorization talks to the API server API server talks to cublet cublet talks to the CRI and I get a streaming back and so the container runtime and well so that means there must be a lot of network traffic going on can I actually see a cubectl uh exec command in the network and the answer is yes. And here now you are uh confronted with

a totally different abstraction to ebpf uh which is a panda like language which pixie implements where we can now get um let me just do this bim. So I will now actually show you the the solution here while I type. Um this is the query and in the query we are um there is a placeholder called path that you are supposed to edit correctly and I will

do this real quick. >> Yeah absolutely >> idea that we watch or that we do it ourselves because it goes pretty quickly. >> I will show you this room once and right after I click like show you what it does then it's your turn. >> Sure. >> Okay cool. Thank you for the question. Um, if you already know the tools, it tends to be overwhelming a little

if you've never seen any of these languages before, but if you know them already, you can absolutely go ahead. Um, where was I? So, here we are watching. This is one of the longest rooms by the way, apart from So, and if I run this now and didn't type something wrong and the network Yes. So, I did So, Marcus did I think one and I did uh

I repeated it twice and then I did the other cubectl with the bash that is still open. So, you see those four cubectl exact commands that Marcus and I just ran and in a totally different uh way of detecting it. So this is a very low data volume output with very low information context. Um and at this point I would ask you to all go to this

step and we're going to bring in much more context later on. So please now uh repeat the steps and try to find what the other one did and and see how you can do this both in Tetragonon as well as in Pixie. Thank you. Marcos. Marcos. Um, do I have Pixie UI open? >> Okay. No, I don't. Maybe not. Let me just do this. >> Yeah, it

looks good. Um, we should continue now. Feel it got display. Oh no, I have to Ah, damn it. Wait, you know, this is different. This is I'm not going to do I can do this. I'll just do I'll do exactly this explanation again and then I'll hand over to you for the next attempt. Okie dokie. So um you just ran pixel in the pixel query language but

you actually also have the UI which I told you about. So if you were a let's say a defender had a sock and you wanted to monitor many many clusters. You see here I have an actual GKE in this dropdown and I have our Kubernetes admin from the lab connected. I could also say like well give me the status of my entire cluster and then I want

to run this exact query as well here. I can go to the scratchpad um and I can put exactly that query I just had. Um if there are any errors you just completely delete everything that is in this uh visc uh situation. It will populate itself by default and you can also run it here. So depending on what kind of person you are um and also how

many clusters you're monitoring this might be a much more easy way to and you can also save these scripts here um to control many clusters at large against for example unwanted or unscheduled exec events. So with that being said we're going to continue now in the attack. So just keep in mind, so this is the way that we query in this case the HTTP events of Pixie,

which is one of the approximately 15 protocols that are implemented. We're going to see a couple more later. Um, and you can always query the H anything that is HTTP related from this table, ideally with filters. With that, over to the next room. All right, then on to room two. And that's where basically run like now we basically just have access to a port and in this

case we know it's a Kubernetes part but it's not always the case. So it could be that the attacker lands in a any machine through an external exploit but um then the most important part is initially to learn everything you can about the environment so you can uh uh execute the right techniques for further escalations. So when you select the the node now you see there are

a lot more techniques available at your disposal and you can um select a few of the available discovery um actions over here we already saw the get user ID but what's also very common is to just check the environment variables for example so if you execute that you will see that on the one hand it updated the information which binaries are available um but we also see

which environment variables are here and we see like it's clear that it's a Kubernetes environment which is already a good tell for any attacker because usually in Kubernetes environments you have a service account token mounted into the part by default. So every practitioner knows this but so also do attackers and it's very it's not uncommon that there are misconfiguration associated to it. So what we do now

is we want to read it and access it and we have a dedicated uh technique there under the tactic credential access. We execute it and we read um the token at the canonical uh path and we see that the graph updated. We we have now a node and the service account token itself and this shows like if we click on the service account token in the graph

we can expand the information. So this is basically the decoded chop token from the service account and we see that it's not just a token itself but we learn something about the environment as well. We learn what node it is running on and its UU ID, the pot um where it's basically mounted into and the service account itself, which can already be very useful from an attacker's

perspective. So over to you. Can you already detect first off, of course, we know that we shouldn't always be mounting service account tokens, but if they were mounted um and used um it should only be that part in a certain context or a certain process in a certain context that should be using that service account token on a regular basis. So, maybe you remember that I said

there was this application profile earlier that we blindly copy pasted. Now could be the time to to check what we actually allowed there. What we're now going to look into is node agent. This is another demon set that can do three types of detections. And the one that we're looking at now is the type of anomaly detection at least theoretically. Mhm. You did you did the cut,

right? >> Yeah. Okay. >> Credential access. If you want to redo >> Yeah, I'm redoing it. Let me just check. Theoretically, okay, I'm going to debug this real quick because theoretically, what should have happened um is that at the time of the um of the access to the service account token, you should have an unexpected service account token used. And that's an anomaly because that was not

previously recorded. that was not in that process. And at this point, we're going to um so yeah, the next step that the attacker is going to do is going to be detected by a different um rule, but I'm just going to use what is already in the in the logs there. Not going to do it live. >> Mhm. So, I will proceed um with the attack. Let's

go over there. So, we have the token, but um we need to know like what we can do with it. And that's usually like everyone using Kubernetes knows uh cubectl and we can there's another technique there check token permission and by default it it uses cubectl and we can execute it and we see that there is an error that's because uh cubectl is not installed takes a

bit due to latency. At this point, you can also check that um tetragon probe that we had earlier if you still have it open. That should be pretty noisy at this point. So, what we need to do to fix this error is actually install cubectl because by default cubectl is not installed and oh, there's a connectivity issue. >> Did we lose network? H. Let's retry it. It's

bit slow. So let's retry it. So yeah, cubectl >> something filled. Something wasn't filled out. >> No, it's fine. Um, cubectl is not found. So we installed it now. Um there is under execution there's the install cubectl command. We can run it. Takes a bit to download the binary. So what happens is it uses curl and installs cubectl in a dedicated environment. So under binaries you should

see now cubectl in the /temp cube uh cubectl directory. And if we go to the service account token again and check the permissions we use the same procedure again. It should work now. So we see under the can section that we have a few permissions but like anyone uh who who's ever checked the permissions like those are pretty much useless. We can't just check what permissions we

have which is none but yeah can we or can the dependent already make use of this type of activity we showed? >> Absolutely. So um if your installation of all the components we might have skipped something I'll check this in a second. Um if Cubscape actually has all the components installed, what you can now see is that it um so we saw earlier again the um unexpected

access to the service account token. Again we used the twice um the service account token to talk to the API but also we have something else. So we have an unexpected process again an anomaly detection because cubectl was definitely not part of the original whatever engine x file that was cubectl has no place there but also we have the network connection. So we are also seeing that

there is a is an outbound network connection from a container shell. So this is a so-called signature rule. Basically, whenever there's from a shell to the Kubernetes API, this is something that is very likely fishy that we're seeing here. And the we have this very interesting detection which is also a signature rule which is called the drifted process. So what here is happening under the hood is

the um the gadget is checking if the binary that is executed is actually in the lower there or in the upper there of that container because if it's in the lower there that is a read only part of the container that was shipped at uh you know when you installed it that was probably intended for usage but if you install if you're using a binary that you

dropped in later in that container when it was already spun up it's in the upper there. So if you're executing from upper there it's very likely something that you dropped which is exactly what had happened and at this point um so yeah to summarize so we have three ways of of basically detecting we have signature rules we have um secret rules basically sensitive locations in the file

system anything on etc or the known locations of service account tokens that we can always watch there are rules that you can write about where those are in your specific kubernetes because RKE2 has for example a different location than a standard vanilla Kubernetes and then you have so-called um signatures whether that's a crypto miner or here the drifted process which we can always we basically always know

that this is fishy unless it's explicitly allow listed in the rules and with this being said I'm going to hand over to you while I uh check why this is not working life. >> So, >> ah, I know why >> it wasn't applied. >> It wasn't applied. >> So, we we actually didn't apply the profile. That's why the anomalies didn't work. So, yeah, classic. So yeah, if

it doesn't work, check if you do get application profiles minus a. we need to reboot the pod >> because the application profile hadn't been applied and it's stateful. >> was too quick with >> No, we scrolled down after the node agent installer and then we went to Pixie whatever. I think we just forgot to apply the the profile. Um, so if I reboot it now, it's run.

You need to retrace your steps real quick, right? Yeah. Okay. Okay. So, um, is this now this a deployment? Yeah. Yep. So if you want to check if that anomaly rule is actually working, you can always check in the in the logs for container has userdefined profile. If that's there, the anomaly is going to work. >> don't let the defender uh the attacker set up the defense

controls. All right. Um, let's reset it real quick because we changed the part under the hood. So, let me retrace the steps. So, and in case that happened to you too, you need to um um reboot the container to which this profile belongs in order to make it have effect. token installation. All right. So from an attack point of view, um this part now is kind of

a dead end. So we explored it. We saw that the environment variables has no real secrets in there. The token mounted in there is not really useful as well. So what else we can do is actually try to explore uh the neighborhood of the pot. And for that like we already have our age-old tool um which is N mapap and we can also explore this of course

in an adversarial setting. Um by default it's not installed. So instead of for us to basically make use of it we can install it and just click on the install package technique and the end mapap if I can type correctly and then execute the command. It takes a few seconds then it will be downloaded and once it's successful under like under binaries you should have the end

map listed over here switch the left side. >> Oh thank you >> no worries. >> Right. So um now when we click back on the on the part itself under discovery we have a new technique which was unlocked by installing N mapap now. So we can click on it and we will see that we need a cider a subnet which we want to scan and here we

have a variable in here um which is basically either we try any arbitrary IP or we can use another technique to actually check which IP the pot has which coincidentally there is one get called get local IP address. we execute this one. Um, we see we have our IP now. And if I open the technique again, we see it's basically prefilled. And then I can start the

scan. And this will uh depending on the the size of the network, it depends. It can be like up to 30 40 seconds, So this this type of activity is also traceable. Like if you have any of the previous security controls in place, for example, Pixie or whatnot, this could also be detected um when this type of of noisy activity happens or when this like end map

is installed. So, it's done now. And as soon as it's done, we see the entire cluster like it's populated with a lot more uh findings in here. So my recommendation is not everything is relevant for this tutorial. At the lower right corner, you have the option to filter out a few name spaces. So I would just um exclude everything except uh dungeon and observability um which is

and of course keep the cluster on like the cubernetes admin at kubernetes needs to remain as well. Um but this gives us a nicer cleaner view of the surroundings and we see now we have another namespace available which we can reach from end mapap but with n map can we detect this already? Yeah. Test >> test. >> Perfect. So like this is getting harder and harder, right?

And we are exploring more of the cluster. So um N map is obviously sort of like a like the oldfashioned way of exploring the whole environment that we're in and um ideally we would detect this and also defend against this. So for this um like um learn learning like what we learned in room one and room two is actually what kind of binaries were executed we like

the drift detection etc and from these we can also deduct what kind of end mapap scan was actually executed because you have UDP scans you have ICMP scans and you can do like also TCP which is exactly what we're doing right like what we did so what we did was in our end map scan. Our workload, the attacker workload was performing a full TCP handshake with the

with the workloads that was that it was able to find and we can obviously detect that using Pixie. So, Pixie detects it um like this. It will detect it. What is this? Yeah. Okay. No worries. Not going to work. But we have a backup. So what actually happened was that it performed a full TCP handshake with whichever workload it has found. But what's interesting actually here is

that if it performed only a TCP handshake, then why do we actually see certain IPs resolved? So because we have lied a bit not really a lot but the end map by default in itself when it finds a certain IP it also automatically tries to resolve it. In this case since we are in the Kubernetes cluster it sends a query to the cube DNS bot. So um

here I haven't shown you the full output but they basically this output is telling you like all the connections and the different amount of data that was sent from the entry hall pot in the dungeon name space to to whichever workload we have other in the cluster. And this obviously is a bit weird, right? Like if our workload supposed to only communicate, let's say, with two or

three other pots, why why is it doing that, right? So that obviously raises a concern. So Marcus is now Nap is old. It's more like a dinosaur, right? And now we go on to the more advanced technique reverse DNS. so yeah with uh end mapap like you have to install the binary it is noisy everyone knows it and knows how to detect it but also like cubernetes

is a complex system it uses DNS under the hood and there is the alternative technique of the reverse DNS scan. So what you can do given again that we know which what the IP of the pod is we can just iterate through all the IP addresses for the given name space and try to resolve those and there are plenty of uh tools available in the Linux ecosystem

but what we use for this technique is get end hosts which is basically using uh the network system switch or whatever it's called uh and tries to resolve the IP address to the corresponding uh domain name. It also takes a little bit. So we will see this the result. Yeah. Right away. So remember um I filtered out a few of the name spaces. But what we saw

just now is that we saw we we see now a tad more of the machines or containers we than we saw before. And that is because Argo CD which are the new pods we found are actually behind the network policy. But with reverse DNS basically they are not touching the the target machines. they're just talking to the DNS server knows if or tries to know if this

IP address has any canonical DNS name and the server happily replies with it. So this is also a very good way to learn about um other systems in the network even though like you are not really supposed to talk to them directly. So, so that's interesting, right? Networking policies were in place, but the attacker was still able to gain like insight that there are also some other

pods, right? So, that's that's not ideal, right? And ideally, we would also detect this. And in the champion path, what we are doing is we're using Coupcape uh which has these pretty cool alerts that not only like we saw in room one and room room two detect drift detection but also it is able to detect whenever a certain workload performs DNS query which is something it has

never performed before. And this is obviously triggers an alert, right? And if we were to see what the uh what the cubecape was able to observe, obviously we right now see don't see anything. But if we perform the reverse DNS scan suddenly we will start seeing nothing. Okay. So let's work around this and get you moving. So ideally you would see that but what we guys see

at as what you guys should see as an output is basically not only DNS requests but also any other binaries that was executed uh that basically triggered that DNS query and that's okay. I was talking about DNS queries. And if you want to only focus on the what the attacker was actually trying to find reach through the DNS protocol, you have some more um commands uh that

you can try below uh that will only filter you out the alerts for the DNS. And whoever is bored uh and wants to tinker around, compare this noise to tetragonon uh compares this output to tetragonon. Figure out who is more noisy, who is more helpful for you and maybe like if you're super into it, give it a try and try to deploy network policies uh and block

such things that the attacker cannot find anything. But note as we move on, try not to destroy your playground and maybe revert what you have created. So thank you. And yeah, now now you have let's say five to 10 minutes to give it a go at these things. Uh and yeah, It keeps >> so by the way so what we seeing is here is that the application

profile that we we cannot bind it it's for some reason our lab is always keep on keeps on deleting it's getting garbage collected all the time this is why we cannot do the anomaly detection yeah um we can still keep on attacking though so I have no mercy for the defenders um we can continue with the uh next part >> uh which is basically like making use

of what we learned now in this environment. So we saw that we have observability um which is like yeah a vip coded observability application um which is basically one of the dangers of uh what happens if you do use the new advanced technologies carelessly and we will uh try to exploit what we can and we see in there we have a radius uh pod and we have

a technique called exploit radis cve. So either we can explore if the version matches or we just try it and give it a shot which we do. So um I execute it and I expect it to fail if holds up. Yeah. So no it's timed out again. Reload the shell real quick. >> Yeah, but why would you vibe code an observability tool? I mean like they're not

expensive or anything. And they did, >> Well, yeah, that's the big question, but so let's execute this again. And yeah, no, still not connecting. The CVE, by the way, that you see here is 2022. And this is actually literally something that that was inspired by literally vibe coding something stupidly and seeing that both it you know often at least my claw does this to me it pulls

in a kind 127 kubernetes 127 that's not the most recent and like for example a reddist that was stable a while ago because it probably got trained on some old-fashioned uh stack overflow posts and so this is actually how we were inspired to to vibe code this because very very often if you let Claude or whoever is your um pull in defaults you don't check it there

might be really outdated versions and of course if you then have end mapap with the version scan n mapap also has lua plugins that can actually recognize well this version isn't just that version but it's also vulnerable and can even do an exploit and this is why we uh said yeah there there are these all these enterprise tools and these LinkedIn posts about yeah I'm just going

to vibe code my own stuff now right um and and this is what can literally happen that it pulls in these wonderful versions because they're discussed often because they had high profile vulnerabilities on them. So what you didn't see just now um while Constansa was explaining about the motivation behind that is that the command failed because rad CLI is not installed on this machine which is well

not unsurprising but like nothing stops us um from installing yet another tool which we saw earlier and uh this is basically a Ubuntu machine. So we can uh install radius using radius tools. so installing it will take a little bit but once it's there um we can retry the attack uh and we can confirm that it is now installed correctly under user bin ready CLI. Um, so

we go back to the radius pot, try to exploit it again. And what should happen now is that we create a a relation between our current entry hole pot and the target radius pot >> maybe >> maybe thinking about it. >> So yeah um maybe in advance I can explain how the the how it works technically. So every command is executed or originates from run itself. We

basically wrap everything in cubectl uh like in the cubectl exec command send this to the entry hall and there from the entry hall basically the actual commands how to exploit radius is uh encoded and this will execute then the corresponding command. Let's reload. Maybe it's just not catching the update. So hopefully it works fast on your end. >> So in in the meantime, what we actually when

we do detection engineering, what we use these command uh chains for is to detect to basically check in our tests if the different ways that we've configured the rules or that we thought that our ABPF probes work actually work for the different TTPs as they can be implemented differently. And so here we're just wanting to give you a visual way of of trying or if you're demonstrating

for example the relevance of a TTP like how much damage you can do in the next room you'll see cluster takeover you always saw that these service account tokens that we had earlier they had almost no permissions right this was all harmless by itself but this chain eventually is going to lead to complete chaos so using the setup can be used to atte um is that implemented

that do do I detect it actually in that specific version that I have installed and of course also to give let uh executives give me money by demonstrating the full impact uh of a potential attack that comes from chaining just a few relatively harmless things that we given you here in this attack chain >> I can't you can just work off the UI otherwise >> I'm just

checking if the part is still up but that's the agent that's radis it's not available yeah >> oh we have lost a lot of pots >> yeah so >> Mhm. So, got to clean up a bit the infrastructure. Yeah. >> Yeah. So that this backfired a bit. We're not going to use Pixie anymore in this workshop. So we're just going to clean it up and make some

room for other workloads. And >> well, we were going to well so actually the the detection for this Reddit CLI was going to be um pixie if you had any form of database in there. Um some databas uses a clear text protocol so that you could also tap differently. you wouldn't need eBPF necessarily for it. But what um eBPF can do, it can hook um the TLS

libraries. So that means any incluster traffic that is encrypted, it can um it can hook into the symbols such that you are getting the the um the traffic before it is encrypted or after it was decrypted. So it's not decrypting it. It's just taking that the clean the plain text parts of the network and then it can also have a protocol inference for example for database protocols

that are not plain text such as Kafka simply be uh taught how to dec basically decode them and give you in the UI you have these queries. Earlier we we queried HTTP. You saw that you can also query your Reddis protocol, Cassandra, Kafka. And for for detection engineering, if this lateral movement that is now being attempted were to work, you can basically watch for strings that are

typically happening in SQL injection attacks. You can grab for um tip like for example this typical um this is a fileless exploit that is sent over the wire. This is not something that should be sent to a database. Um so that you can basically see what is your query typically uh look like and if you have injections in there you can use um these protocol tracers to

have filters and detect if somebody is injecting data or exfiltrating data because otherwise especially in encrypted outbound data it's extremely hard to find what the attacker actually smuggled in there and this is why having ebpf on the wire in between the sockets is extremely valuable because you can a see the unencrypted the traffic and b also resolve the database protocols that are often for humans otherwise not

readable. So this is just a comparison if you did TCP dump versus this. >> All right, so I cleaned it up a little bit so we find the right instance again. So now we have basically the Thank you for the elaboration. Um we do have now this relationship between the the entry hall and the radius pod and whenever we send any commands so now we can execute

commands on this pod. So whenever we do that we basically wrap it first in the cube ctl x command and then we wrap it in the uh uh syntax or the the wrapper for the exploit itself and then we can execute commands here and we can do our previous procedure where we basically check the service account token. We see that it uh has the observability agent mounted

and we can go ahead and check this token and see that if we check the permissions we have those uh fancy get node proxy permissions along with a few others which are very common to see um for observability tools. And the thing is with these uh permissions like maybe some of you uh watched the news back in January there was quite some uh interesting findings surrounding this

because those are all supposed to be uh read only permissions but um as it was uncovered uh back in January that you can do a lot more with this but first we will try to detect or let's skip the >> no no we can't do the detection this is what I talked through but but here to note is so this is why we vibe coded an observability

rule Not just because we want to um acknowledge the presence of some friends here. No, but it's also because um observability tools typically have high permissions in the cluster. And this is a typical one. And this is not a CVE permission type. This is a this they really actually need that permission. It's a get permission. It looks harmless. And this is the one that they're going to

now use to do something else. So the detection as would have happened in Pixie, which we just uninstalled to save the cluster. Um but if it's still working go to the reddus tab um or query the reddis um dropdown and you will see the entire attack in plain text that was injected by ran >> right. So with the uh pots or the get node proxy permissions like

what we can do now is talk to the node itself and ask it what parts uh it contains. So if we execute that um we will in a moment get an update from the cubelet itself. So yeah so here it is um and it tells us there are a lot more ports than we saw with either end mapap or our DNS. So we learned something new again.

And what we see now is that Argo CD is used in this environment. So we saw a few instances of that already but um anyone knowing Argo CD like it it is a GitHubs approach. So the idea is that it has a lot of permissions um and it's basically watching your git repository for any changes and manifests those changes in your cubernetes cluster and for this it

requires a lot of permission that's why it's like behind the network policies and it is hardened but the thing now with this type of uh permission or with this well it's not an exploit um but this loophole um is that with the permissions to read node proxy an attacker is actually able to execute commands using the cublet itself. Um we saw we query which parts are on

there. So we know all the information we need for the attack which is the node name, the pod name and the container name itself. And with this information we can ask the cublet nicely to just execute something for us. So for this we do need a dedicated file which is now called run ws. We just installed it and we will uh download it from a specific uh

path and drop it into our temporary folder. And what should happen in a moment is that the paths update. Yeah. So we see that now our port can perform cublet xc commands on the node itself and this basically then forwards um all the commands to all the other parts on the same node. So with this we effectively have now remote code execution on all the parts on

all the nodes. Um like we have just one node but this would work with any other node as well. Um yeah and we can now like we already have set our eyes on Argo CD and it is an open source project so we can analyze what how it's configured what are useful uh components and service accounts and one that stands out to us is actually the Argo

CD application controller. Um and here we can focus it. We can click on it again and ask again what's the service account uh mounted to it. And it takes a bit but it should read it in a moment. So what's happening here while it's executing it? It's again starting from run packaging the command for qctl to be executed on the entry hall. The entry hall then has

the wrapper for uh the runws. how this basically uh formulates or wraps the the command which we want to finally execute on the Aroscd application controller. So you see it's getting messy real quickly but the pot then sends first the command to the cublet and the cublet forwards it then to the pot. So yeah it's a quite a mouthful. Um in our trace flow you can see

how the real command looks like in practice. So this is basically after the cubectl or this is what is being executed on the entry hall pod and here we see that this was the binary executed this is the crafted URL we have with the right token permissions that we are allowed to use that and this is very like dreadful to to write by hand so it's very

natural that you create a dedicated malware for this and we used runws currently but in practice if there's an adversary they don't really want to share their secrets how they built the the malware and that's why they usually don't try to drop it directly on the file system but make it somehow hidden and we have that v uh version as well which is called drop fileless runws

so what this now does is basically the same binary it's also downloaded locally but it is not creating a new file on the file system and we see that now that it's under binaries it's uh if I find it Yeah, here. So, runws is now under proc and then an arbitrary number fd3. And this is basically now there's a new process running on the system itself which

has a file descriptor open to that location which is just a link but it was never really written to the file system and when this process is killed the file is also gone. So this basically makes it impossible for uh defenders to analyze the binaries and see how they were able to to get this attack going. So now it's up to you to actually detect this stuff.

So that's that's perfect. So what we actually like we we saw two two scary things actually here. And if it wasn't clear and like just to make sure we're not like making any workarounds actually any service account that has nodes/proxy get permission can execute commands on any bot actually container if I'm precise in your cluster that's a fact and if an attacker gets hands on it he

basically has control over your whole this was brought up by Graham in January And there was literally no solution for it neither like how to detect it etc. So what we did was we came up with a gadget actually an EVPF program that is able to detect the this attack actually we identified the unique signature it has and are able to alert based on it. So let

me let us in the apprentice pad we're going to run our custom gadget. So let's get this thing running. what our gadget actually does it it hooks onto the kublet binary at specific offsets just when the encrypted traffic is received by the kublet and encrypted. So we are using EBPFU probes that are attaching at that specific offsets where just at the right time where the traffic is

being uh unencrypted and we see basically everything that is going on. You see a lot of stuff right now going on here but what's essentially the interesting part is if we repeat the attack. So we go to the Argo CD uh application controller and we can just call let's say get user ID and if I go now to here if I scroll above so obviously a lot

of things going on. Voila. So what's the point here? What we see here is basically a get http request to the exec endpoint and how like this is kcon and a lot of you know actually how kubernetes roll back access works. So what happens is you give permissions on resources but on that resources let's say pods you configure verbs get create delete right but how is like

how does the let's say kublet in this case actually determines whether you are allowed to do something or not it does this by mapping the initial http request to any of the verts so in this case this is a get request so whenever there is a request to the Kublet Kublet API checks whether you have nodes/proxy resource uh and then since the initial HTTP request is a

get request it checks whether the service account has a get verb on that resource and since it has it it lets lets it through it lets it through to exe actually successfully execute an attack on the exec endpoint which is quite scary right because in get request we always think about it as read only thingy, right? And like usually the post is the one that is actually

doing something doing something executing something. So again this gadget like um not only we can see this is a get request to the exit endpoint but what you guys also see above is the actual token that was utilized to perform that attack. So you can identify which service account token was utilized. So then you can deduct which was compromised and basically just back trace to all the

things that we were showing you guys in the different rooms. And Marcus also mentioned that there are uh there are and yeah and by the way like whoever's um knows how the communication channels are established but basically what happens here is the websocket protocol works in a way that the initial request is an HTTP get request which then upgrades it to a websocket birectional communication channel where

you can then like run like where the std inn and std out are like used so you can run different things in your container etc. So what Marcus said that also that attackers can get a bit more trickier, a bit more smarter like uh they also can execute the fileless uh binaries right he mentioned fileless websocket binary which is actually if I would show you the code

internally the websocket binary is actually used to perform that uh attack. So in the champion path if you will check you actually also have a script that detects whether there are any fileless binaries on your system. How does it do that? It basically scans through your /proc folder and uh for all the P ids it checks what was the executable that run that process and it checks

whether beside that executable name there is a delete marker. It's like actually written beside there is like a word like this which means in general whoever is not familiar with files and missed it before imagine a process installing a package opening it basically getting a file descriptor to it and then in another process let's say deletes that file but since this initial process has opened it it

has it in the memory it can still utilize it and as soon as it will drop its reference to it, the file will also be gone uh will be also gone from the system. But until that it can use it. It's a fileless malware. So if you guys came that far, congrats. But this is actually really uh the let's say the pinnacle of the difficulty. And yeah,

give it a try. Raise hands. Happy to help. Are we doing something? So, thank you so much for attending. So, we're going to the the seats that you got um you'll have them for 24 more hours so you can so that the full um access to all the labs and anyway every day you have one hour free and also the read access is going to stay there.

Um I just want to thank the people that uh actually powered the VMs that we had. The reason why some of the storage got lost is actually uh we had our node that we were running on got overloaded somewhere at 4:30. Anyway, um the universities that are behind this and Ivan Vilichko who uh did the um the VMs, a huge shout out, he actually left CubeCon to

monitor the system behind the scenes. Um, and the other thing is, um, yeah, if you want any copies to do this walkthrough to help you how this configuration all works, we have some materials that you can pick up at the post, um, project pavilion. And please talk to the maintainers if you have any questions. Um, and repeat this at home. Thank you so much for staying with

us. Hope you enjoyed it. Thank you.