In Falco's Nest: The Evolution of Cloud Native Runtime Security - Iacopo Rozzo & Aldo Lacuku

This talk by Jako Parazzo and Aldo Lutaku focuses on the evolution of Falco, an industry standard for runtime security in container-native ecosystems. They celebrate the journey of Falco from its open-source inception in 2016, through its donation to CNCF, to its graduation in 2024. The speakers explain Falco's role in providing security visibility using kernel-level monitoring with capabilities like eBPF instrumentation or legacy kernel modules. Recent updates introduced features such as capture recording for forensic analysis and performance enhancements through the Drop Enter initiative. Further developments discussed include the Falco operator for improved Kubernetes integration, enhancements in configuration support, and upcoming features such as multi-threading and integration with AI coding tools.

Hello CubeCon. Welcome to the Falcon maintainer track. I'm Jako Parazzo and joining me today is Aldo Lutaku from Kong. Uh we are going to uh discuss about the last evolution in the Falco and its ecosystem and we are also going to present uh what's uh coming next. Before moving to the more technical bits of my presentation, let's take a moment to celebrate a decade of detection. Everything

started in uh 2016 when open uh when Falco have been open sourced and in uh 2018 it was donated from CESDIG to the CNCF where it started his maturation process uh that led to the um graduation in 2024 where it reached the highest level of maturity. uh of the uh CNCF program and now we are here in 2026 uh to celebrate one of the industry standard of

uh uh runtime security in container native uh ecosystems. uh for the one that could have uh missed us in the last decade, let's take a moment just uh to uh reply a simple question. What exactly is Falco? Well, uh there is an analogy that we often use to explain uh what is Falco in a nutshell. Uh, Falco can be seen as a security camera as a sec

as a security camera in a building can uh detect um suspicious activity. It provides uh deep visibility into your system and into your uh infrastructure and uh allows to detect security threats in real time. uh we it provides a kernel level visibility either through an ebpf instrumentation or a kernel module for uh legacy uh systems. Maybe uh those uh probes as we call them generate events that

are sent to the Falco detection engine in user space where a set of reuse is matched uh through those events uh generating alerts. Those reuse are expressed with a simple uh YAML based syntax and of course uh we provide a predefined set of rules but you can tweak them you can provide your own it's highly extensible and the extensibility of Falco doesn't end here because it provides

a plug-in system that allows you to uh produce your own set of events or to enrich the existing built-in events with additional information collected either uh like uh to to make an example we can collect metadata from container runtime or Kubernetes and use this information to enrich existing events. Uh now let's take a look to what has been the new features that have been delivered in the

last six months. We had two releases that 0.42 42 uh back in October 2025 and 043 in uh uh January or February uh 2026. So um Falco 042 uh brings a feature called capture recording. It enables uh forensic analysis with Falco. Basically, you can configure your rules with this capture flag that allows to start a capture of the predefined duration uh generating uh what we call what

is called an SAP file. This file can be then uh analyzed with a tool called Stratark. The name could be familiar. It's a sibling uh application to Wireshark and provides the same user experience than uh Wireshark. It has been authored by Gerald Combmes that is the original author of Another uh important um milestone uh basically we had an initiative called the drop enter initiative. This is a

performance related uh feature. Uh basically historically uh Falco had been uh hooking uh into the uh enter and exit path of sys calls generating two events for each sys call. Uh we basically uh realized that we could consolidate all the information into the exit event only. uh and this of course brings an important improvement from a performance standpoint because we uh achieved a lighter kernel instrumentation. uh

the some uh benchmarks revealed like a 20% latency reduction in the instrumented uh programs and as we reduce also the number of events processed in user space we of course reducing the CPU consumption of Falco. A notable consequence of this initiative is that now the event deer field that the the one of you that are familiar with falor rules have been deprecated. Uh there are of course

uh other changes that enter the last two releases. I'm not going into the details, but I invite you to go through the release notes for those two version and check all the stability improvements, bug fixes that have been uh introduced. And now uh let me hand it over to Aldo that will tell us about the Falco operator. Okay. Um if you have been using uh Kubernetes uh

if you have been using Falcon Kubernetes for sure you have been deploying it with Helm chart and it works great uh it deploys Falco it's great for getting started but uh Helm is just a tool so it once the chart is deployed it steps back and uh it doesn't watch for example for rules files it doesn't see if the plugins have drifts or any config map you

are relying on has been accidentally deleted. did. So um that's uh where the that that's the gap that the fal operator closes. Uh it brings uh the fal into the kubernetes native operational model. Um we have uh you you just declare what you want and it will reconcile the state toward the the desired spec. Uh we have introduced the fal operator in 0.41 41 in technical preview

and uh since then we have uh extended the API we have uh built up the ecosystem component support and reconciled and um harden the reconciled logic um with the 0.2 two version. Uh this is the first release that we feel confident to call it uh production ready and um that's it. Going forward uh the fal operator will be the default way on uh running Falcon on Kubernetes.

Now let's have a look at the at the architecture. Um the solution is based on two components and I think it's worth taking a moment why uh to compress rather than one. Um as you know uh Falco runs as a demon set it has to be on every node on your Kubernetes cluster and uh having a centralized operator sitting outside the deployment of Falco uh is just

doesn't work because plugins configs rules files they have to be applied to each instance and that's why the artifact operator that you see here it's um a sidecar it's injected as a sidecar in every Falco pod uh it watches for uh an artifact API group where we there are plugins config rules files and uh it shares a set of uh file system folders using emptyders with falco

as you can see here and it puts those artifacts there. Then Falco the Falco uh container uh will watch for those uh uh files in the file system and will load them accordingly. And um then there is a new another component that is the the fal operator and the fal operator is a component with which users will directly interact with and it watches the instance falco security.dev

API group. Um we have two kinds there. There is the falco kind and the component kind. Well, the falco kind is the let's say it allows you to define how falco will run in your kubernetes cluster and um on top of that there is also the component API. So you as I said before falco runs can run in two different deployment modes. uh the demon set mode

is the let's say the standard runtime security use case that uh you will use in your cluster when you know when you need uh let's say a full uh Cisco level visibility of your infrastructure but there is also another uh deployment mode that is the deployment and it's when you don't need the uh the let's say the node level uh scope but you just are using Falco

with uh plugins so you have a deployment of Falco just with standalone plugins. Think about for example the octa plugin that it will get a stream of uh events from outside and then you can write rules on that. And then as I said there is the component um kind in the instance API group uh which allows to manage all the ecosystem components of falco like uh the

meta collector sidekicks psychic UI and and so on. Uh now let's see what are the major features and features and improvements of the 0.2 uh version. Okay, if anyone of you has been using the technical preview, uh we there was an inline field in the config and rules files where allows users to uh write directly the configuration of rules in the in the CR and it was

just a plain string and as you know writing YAML with proper indentation in a in a YAML string in a string it's it's not easy. So uh in 0.2 two users now can just uh write directly in the spec a structured yaml object and that uh and the API server will provide you validation at apply time and that really matters for application like I don't know for

git um uh githops pipelines where the having the validation at apply time is really and uh now we have also the the OCI artifact API well if you are using the zero 1 version for sure you will need to uh take care of this one because this is a breaking change. So uh the reason behind this is that um there are different ways on pulling the o

the pulling uh artifacts into the operator and the OCI artifact is one end this API was lacking a little bit of uh expression and so in 0.2 two, we made it a little bit more verbose, but it allows also to uh configure scenarios where the registry is a self-hosted one and you have to configure TLS configuration or just an internal one with plain HTTP. Uh then there

is the config map support. What does it mean? Well, the artifacts can we have different ways to uh to configure for example rules files using the rules files artifact. You can get it from an OCI artifact or you in the 0.1 you could just provide it in line. Now we have added the config map supports because we know that a lot of people are uh storing them

their rules files in plain files in plain files in their G repositories and then just through Gittops pipelines uh creating config maps in their clusters and having the possibility to uh just uh reference those uh rules files with a config map is is better. And uh one uh one thing uh worth noting here is that uh the different sources that are OCI artifact config map and the

inline they all can can coexist inside the same uh instance of rules file meaning that you can just pull I don't know the base rules files for example provided by the falco security then you can layer on top of that your custom uh rules and then if you need to tweak them just write an inline uh config uh to override them then I think uh this one

is the let's say my favorite because uh uh you can have uh you know Falco has a lot of ecosystem components that just uh integrate and expand it and uh now thanks to the component kind in the instance API group uh the operator uh can manage uh can manage the components and uh just declaring the the the component CR and you right now we support the meta

collector falco sidekick and the sidekick UI you just set the type there and the operator will take care to deploy them with uh production ready let's say defaults but if you need to customize anything there is a pod template spec field where you can just overwrite any any configuration And this is uh this is what the unified control plane for falco operator uh for falco operator is.

You have one operator uh uh one set of CRDs and uh the entire stack of falco. So uh let's now take a look at uh what's cooking for the next uh Falco release and beyond. Uh let's start with uh BPF iterators. Uh, Falco is uh fetching uh system information by doing uh procs scan uh during the uh initialization phase or uh for state recovery for instance in

case we are dropping events we miss information about some active threads. Uh this is inherently uh and uh is inefficient because uh having to scan the process system means having to open read close tons of small files. Uh this of course is a bottleneck and also is generating a lot of events that go through the um uh Falco engine. So uh we have a feature uh from

the uh fal from the Linux kernel coming from uh version 5.8 eight that comes to the rescue is called BPF iterators is a specific type of EPF program that allows you to uh traverse uh kernel structures as tasks and files. We are uh starting using it from Falco version 0.44. It's the implementation is already almost complete thanks to Leonardo de Javana here in the first row. It's

already in master. So it would be there is very small uncertainty about this uh let's move uh to a big uh architectural uh change in uh that is uh in our uh pipeline. Uh I let's start from uh the reason uh so why uh would we need multi-thread support? Uh when uh Falco was born 10 years ago, uh the model based on a single thread event loop

was perfect for uh the uh machines the system we had at the time. It was simple and blazingly fast. Uh I think to that the the picture uh the that you see uh on the right talks to to itself. Uh modern uh machines uh changed the game. Uh we have a lot of parallelization. uh in a machine with 100 of course uh the risk with this model

is to uh saturate the uh event loop uh and this would result in dropping uh events and reduce the the visibility. Uh I wouldn't spend much time on this side but this is the current uh falco singlethreaded architecture. What is uh important to retain is that um Falco is a stateful engine. As you can see we have uh the thread state in the lipsyn uh layer and

this is the represent probably the main challenge for uh multi- threading. uh we are now uh experimenting we are uh going through a proof of concept based on this architecture. The main uh principles are we are uh partitioning the events by thread group ID. Uh this uh allows basically to have a lo strict consistency locally uh for on each worker where we are processing all the events

for a for a specific process and uh allows to have a very uh light uh synchronization. So we can uh expl uh exploit well the Uh the main problem of this architecture as I already anticipated is uh that we still need to have uh cross partition uh access to some data and uh this if we don't do it right could lead to some uh u state temporal

inconsistencies. Uh so what's the plan? As I said, we are experimenting with this solution at the moment. The plan is to have an experimental multi-threaded version of Falco in the next release. It's not going to be perfect, but we want to have something to iterate on and to avoid analysis paralysis because as you may guess this is a big uh change and we need a base to

iterate on. So let's move now. >> Thanks to you. >> Yeah, thanks. Uh so let's see uh what's next uh on the developer experience uh side. So let's start with the Falco uh LSP. So uh we have now uh some language tooling for Falco. Um some of you probably experienced the uh aring wrote some rules or edited some rules. So this uh is going to help you

uh providing code completion and uh syntax highlighting. Uh at the moment we have a VS code extension but the language uh server is generic. So if you are using another ID with some effort you can adapt it to use it on your preferred ID. Uh thanks to Alessandro Canela here for this effort. uh very useful and now let's switch to uh another exciting experiment. So let's start

uh let's have a a quick demo here. So just to uh provide you the context uh like it or not uh AI coding agent are becoming an important part of the life of a software engineer. uh they are powerful especially if you provide them uh access to your system. So it's important to provide guard rails in this scenario. So uh we basically uh experimented with Falco and

we turned Falco into uh a um basically uh uh guardrail layer for uh AI uh engines. In this uh example, we have at the moment we have support just for uh cloud code. Uh the uh idea is pretty simple. We are uh having a plug-in that is hooking into uh cloud code is uh intercepting the commands the intent of executing commands. It's running uh the real engine

of Falco to uh basically generate a a response. So we it can accept uh reject on the ask for uh confirmation and as you can see uh the the result is here. Uh of course this is uh a tech preview but uh it's easy to install. You can uh test it on Mac OS, on Windows or on Linux. So uh feel free to u play with it.

We we want to have feedbacks. Uh let me come back. So uh we are almost done. Just a few uh community updates. Of course we want to uh enlarge our community. We want to have uh contributions and if uh the incentive we provided is not good enough for you that's here uh another incentive basically uh sysdig to celebrate the 10 years anniversary donated uh 70k to the

and we are going to uh have some funding uh programs for people that he's uh contributing or is willing uh to contribute. Uh the main uh things uh it's going to be um development grants for uh people uh help helping to speed up uh the delivery of cap some new capabilities. contributor steepens for recognized uh cont high level contributors and also a steep end for uh technical

writers. It's always important to have a help to keep our documentation in sync and uh this concludes my talk. So thank you everyone and uh find here all the links uh if you want to join our community you can reach out on Zlack and we have a community call every two weeks on Wednesday. So uh feel free to join us and uh thank you everyone.