Detect, Decide, Defend: Building Cloud Native Security That Fights Back - Matthias Bertschy, ARMO

This talk focuses on enhancing security frameworks for Kubernetes by improving incident detection and response through a connected understanding of various signals. The speaker, Matias from Armo, discusses the limitations of traditional monitoring tools that operate in silos, creating a barrier to comprehending security incidents across layers. He emphasizes the importance of establishing a baseline behavior for cloud-native workloads, which can be captured in an application profile. By analyzing deviations from this baseline, Matias introduces a framework called CADR, which stands for Cloud Native Application Detect and Response, aimed at correlating different security signals to create coherent incident narratives. The session also highlights RemOps, a declarative response mechanism that facilitates automatic remediation actions in Kubernetes environments, ultimately enhancing the security posture by learning from incidents and preventing future attacks.

Welcome to this talk. So when you are uh on call and your your day starts like this, you know it's going to be fun. So let's take an example. A change detection workload in your EKS cluster just spawned a reverse shell. Oh my god, what happened? So the problem uh with the security in Kubernetes is that it's layered. So you have a web application firewall and it

sees an injection. You have an ebpf uh probe that sees a process execution. your Kubernetes sees some API calls and then you have some cloud security that sees some IM activity. Problem is when you get a security incident, it doesn't respect the layers. So we have four tools, four different alerts, but nobody understand what's going on or what happened. So I'm here to show you a framework

on how to make it better. My name is Matias. I'm a developer for the a company called Armo. We do Kubernetes security. I'm a CNCF maintainer of three projects, Cubscape, Pro, and Inspector Gadget. And I've been contributing to Kubernetes since 2016. If you used startup probes, that's some of my work. And if you ever wondered why the native sidecars they are in it containers, that's also my

fault. So we can talk together after if you don't like it. Um so the real problem is we don't lack signal. We don't we we have enough signals but we lack a connected understanding as I said before like each layer sees its own information. So SQL injection we have like a Python 3 that has spawned a bash communities API as I said the cloud IM and if

you take all these incident it could be just noise but if you connect the dots you understand that it's a real attack. So how do we move from disconnected events to a coherent incident story? So first of all we need to define what normal means you know so every cloud native workload has a predictable behavior uh that's what we call the baseline behavior it includes the expected

processes uh the network behavior what are the system calls which files are accessed which files are open executed what airbach uh rules are being used. if if you know what the normal for your application looks like uh the detection can be really improved. So we should build this baseline and it can be derived from Kubernetes metadata. It could be derived from EVPF probes signals and ideally we

should like store them in a cloud native format. So I suggest we call this an application profile and let's make it a CRD. what happens when the runtime behavior breaks that baseline when it's out of the normal? So let's take back the the attack. this is a sort of a visualization of this attack. Okay. Um I will go into some details. So the f the first deviation

as you see there was an HTTP post. There was like a crafted payload and it tried to inject some code that was executed inside the container. So ideally we should have a way of looking at it and detecting it from the HTTP payload. Okay. Then what happens? So this first uh execution starts another process and then another process. We have a shell, another Python, another bash whatever.

Then some commands are executed it and at the end we learn what the system is with you name. So we can follow this in our system. Let's see. Okay. P ID first, second P ID. What is the command line? And then this this wasn't just like a suspicious string actually. It was a workload that was executing some behavior that is not normal. We never saw this in

the application profile. uh some of the C calls like exec connect were not part of the profile and also the fact that an application starts another web handler that's not even normal. So like we can even like follow this the stack trace if you want. So as as I said before a simple signal a simple deviation is not an incident it's just some noise let's see okay

we take those different noises and I will explain what I call CADR so we all know what ADR is right and now let's let's add a C because we are in the CNCF so a cloud native application detect and response. Um, so it's like it's a sort of framework where we try to to connect the dots and and we try to make sense of these different signals.

Okay. So come back to the to what happened an HTTP we see some process tree and cys calls being called. Um then some some manipulation in the different kubernetes uh resources and finally something happened in in AWS. Okay. So this is not adding more telemetry. We don't add we already have all the sensors that we need. What we are trying to add is like bringing together all

those signals and making an understanding. So instead of having like four different elements that we are not sure what happened, if we can like just have one that we are really confident, we can explain the whole attack and what is very interesting that we will see later we can respond to it. So this is what I want to to show you. So now if we try to

to explain it like in in in English it would be something like this. Uh initial compromise the different the credentials access malicious code tool installation whatever. Okay. If we know the deviation we understand the attack story but what should we do next? Okay. So now I'm coming back to the application profile and explain what it should be. So what we want it to be is not just

being an observation but we want it to be more of a a contract. So we want to to capture the baseline behavior of your application. the allowed process, the allowed C calls, what are the network destinations, which file system parts you can access, what are the security posture expectations. This is not just machine learning magic. This is a declaration based on facts. So we we we learned

this profile, we saved it, we can even sign it and and then this is not just hallucination or derived from the code. This is like reality. So what I mean is that if we the the baseline, we add the deviation and the correlation, we have high confident in this detection. This is how we apply declarative behavior behavioral understanding. And since we have like a high confidence into

this detection, we can do what most teams are afraid of. which is what this talk was all about what we called remups remediation operations so this is something we've been willing to do for years right um sRes wanted to do automatic remediation but they were always too afraid to have false positives but if the detection is contextual and workload aware then automating the response becomes safer. So

what is remops? Um it's a declarative response using Kubernetes native constructs. So it could include rolling back the deployment. We could patch the network policy to isolate the pod. Once once this is done, we could rotate the service account token because maybe it it the attacker had it already. We could further restrict the the existing P by applying a second profile. We could also update the application

profile if it was a false positive or not. And we can also trigger GitHubs to like just redeploy or reconcile our Now if we want to come back to this second profile, if I hope everybody knows what it is, but it's it's a way to restrict the the Cisco uh surface. So it basically limits the capabilities of the attackers and it's Kubernetes native. It's just a bit

difficult to deploy on every node but let's focus on this. So now if we come back to an incident so all the things in in black we know already but now what could we could add to it is our response. So once we know what happened and we have like a full attack graph we could start the remops. So we patch the network policy, we isolate the

pod so we can still observe it but it can no longer phone home. We change the account token that I said and then once we uh we have seen what it does or even automatically we could like just roll back the deployment apply a second profile and then make sure that this does not happen again in the future. this could be automated. This could be super fast.

So yeah, injection to containment 7 seconds. So now with the last step that we had in green, it's not just reacting, but it's also reinforcement. So we learn something and we reinforce your workload. So it's not just responding to an incident, it's also strangling the workload. we make it like more difficult to to hack again. I can come back to the architecture and explain again. CADR what

is it? So we observe by using the different tools we already have. So, eBPF, Cubscape or the different APIs of Kubernetes. Then we create a profile. We save it. We have our baseline. We have our contract for the the behaviors. Then when something happens, we can correlate it, create a graph, understand what And finally when we are confident this is a real attack we can remediate it

with remops applying the second profiles to make it like more strength in the future use githubs this is a pattern this is not a product I mean you could use different tools from the CNCF to to do this what I'm trying to sell you is an architecture for cloud native defense built with open components we can even say like cloud native systems are declarative. So why not

detection and response cannot be declarative too, right? So I'd like to come back to the CNCF now why it's important why we are proposing this in the CNCF because it it's based on open components. I would like to to sh to to to show that it's like composible. There is no proprietary enforcement plane, no blackbox, just open signals, intelligent automation and common sense. So this is the

idea we are pitching remops and I hope different opensource projects and the industry will will take it and that we will improve our automatic remediation of incidents. Okay, that was faster than expected, but at least we have a lot of time for questions, please. So I could ask yes please. >> Um what tools do did you suggest for correlation? I didn't get it. >> So >> for

for correlating the layers the events in the layers. >> Yes. So those tool don't currently exist. That's that's more a call for the for the industry, right? >> We need this. We want this. I mean the CIS admin inside me wants this. I don't want to wake up at 3:00 in the morning and have to deal with this. So, it doesn't quite exist. We we have like

some components that exist. >> Uh like in the cubecape project which I'm a maintainer of, we we already have the application profile. We we can do like alerting on the runtime but we are lacking this intelligence. >> But that's that's some thoughts, right? >> Yes. And another question if it's possible. >> Sure. >> uh in another talk I uh learned about ESBOPs. So software bill of behaviors.

>> Uh how is that different from your application profiles uh that you uh defined? >> That's a very good question. So uh yes Bob is is the result of uh the cubescape project teaming up with constants and and the K8 storm center and and the idea is to have so application profiles they are really specific. They are specific to an environment. They are specific to a configuration

and to a particular workload revision. If you take an application profile from GKE and deploy it on EKS, you will get false positives because the account uh service account tokens, they are not in the same path or some addresses are different for the load balancer or and and and the idea behind SBOB is to take the application profile and make it more generic but in a safe

way. and and the initial idea is to force the vendors uh with the CRA, the European uh uh law to not only give Sbombs but also give SBOS and instead of of doing the learning you could just apply them and and use the existing tool and enforce the the detection. So what is the K test setter doing is trying to automatize the the generalization of application profiles

and make them safe which means that safe of false positives but also safe of false negative by testing them in different environments with real attacks and see that we we catch them. thank you. That's a very good question. And and also something that we do with the sbobs is we we want to sign them so that we have the full tracibility. >> Do you want correlation um

for um taking the action or is it separate? So um do you need to uh to trigger all rules that would be correlated to take the remediation action? >> Yes. So exactly the the goal would be to have a layer of aggregating these different incidents in in a way that that you can create create the graph that I showed before or just and and and then when

you have the confidence like doing an automatic remediation. Yes. Um, but wouldn't you need some low-level remediations because in your case, maybe if there wasn't some IM action, your remediation wouldn't happen. Yeah. Um, so I was trying to give some examples of what remediations we could do from a Kubernetes clusters, but uh, yeah, low level could be needed. Yes, in some cases. >> Thank you. So who

who is a vendor and who is interested in implementing this? I want that and and I can work with you Okay. No more questions. Yes. No. Maybe you have something. Is there some of parts that we maybe can create a project from? >> Uh sorry, what do you mean? Uh >> I mean that uh which tools that already existing in SE landscape we can uh uh use

to create such let's say system/platform. >> Mhm. you know, besides Cubscape. >> Yeah. I I don't know. We We should have a look, but I mean, it would be cool to have a sort of of a working group and and let's let's brainstorm together and see what we can do. But, uh, so I'm I'm actually replacing my two friends, Rot and G, who couldn't come to this

talk, so to make this talk. So, I just took it like last week. I don't know what they had in mind, but uh it's more a call to action like uh >> at least create tinf channel in Slack. >> Yeah, we could. Yes. Yes, definitely. Thank you. Yeah.