Project Lightning Talk: Scaling Kubescape to Thousands of Nodes - Matthias Bertschy, Maintainer

This talk covers the challenges faced by Kubescape when deploying to a large Kubernetes cluster with 5,000 nodes, each generating significant data loads. The speaker explains the initial architecture that led to lock contentions and how traditional storage patterns became inadequate. He describes the evolution of the system, highlighting a shift from naive approaches for data handling to a more sophisticated solution that treats data as time series. This transformation allows for improved scaling and performance by decoupling the write path. The speaker also hints at future enhancements that will optimize resource usage by reducing redundant data learning across nodes.

Hi everybody. I'm Matthias. I'm a maintainer of Kubescape. So, what happens when you are really proud of your project and then you deploy it to a very large cluster? Well, this is what happened when we tried to handle 5,000 nodes and each node sending 10 MB objects for each container running periodically. What you end up with is a thundering herd which I tried to represent here. And

my talk is the story on how we hit a wall where lock contentions from thousands of nodes made the traditional Kubernetes storage patterns obsolete and how we rebuilt the Kubescape architecture to survive it. So, let me talk about Kubescape Kubescape a little bit. So, it's a comprehensive CNCF security platform for Kubernetes. We are an incubating project and we provide continuous scanning and deep eBPF runtime observability. And

this data is used to do to create second profiles, network policies, but also runtime detection using the C language and we can send alerts to alert manager or the syslog. And this depths of of data generates a lot a lot of data. So, and this data we store them in what we call container and the single profiles is easily tens of megabytes. And when you have like

thousands of nodes continuously update this data at massive scale, it creates a problem with the API server logic. So, with such big big objects, we cannot use ETCD. So, we are we have our own aggregated API server and every node agent sends the learned profiles to the storage. And the other components, they are consuming this data via the standard Kubernetes API. Um at first, it was a

very naive approach. We said, "Okay, each node agent keeps in memory the entire CRD, and like every 10 minutes, we just send the full object to persist it using the create method." It doesn't work. The problem is when you have different replicas or even more like demon sets, you have 5,000 node agents sending the same data at the same time, and you have a lot contention, and

you create also a lot of of back and forth traffic, because when you have a conflict, you get back the object that you need to update, you update it, and you send it again. So, it's it's horrible. So, second attempt, we tried the patch method. So, every node agent like for for the the 10 minutes, they generate a JSON patch, and then they send it, and they

rely on the aggregated API server to apply this patch. The problem is this So, this is also a standard method of the API server, but it is also synchronous. So, when you send that patch, suddenly in the storage, you only have 30 seconds to read the big object from the disk, unmarshal it, apply the patch, and save it back. And then if you have also mutexes to

make sure you don't update the same object, it doesn't work. third attempt, that's the current one. We moved the intelligence to the storage layer instead. So, now each node agent sends a unique CRD for each period. We treat them as time series, and then which which means that the API server can just save them to disk immediately, return the acknowledgement, and then the node agent continues. And

then when there is like less CPU utilization, we can just uh asynchronously aggregate them and do the processing that we have to do. So, with this decoupled write path, uh we were allowed it enabled us to to scale much better. then there is probably a a fourth phase that will come in the future since you don't need to learn like the same data a hundred or thousands

of times. So, the storage could tell the others that, "Okay, we need only five learners and the rest can just like wait until we have the results." So, that will come. I'm here until the end of the conference. Uh please come and talk to me. I have a KubeScape kiosk at the project pavilion. I will be also at the Armo booth. And uh yeah, just meet me

and give me ideas. Thank you very much. Aw- awesome. Thank