Project Lightning Talk: Fluent Bit V5 - Eduardo Silva, Maintainer

This talk covers the Fluent Bit project, a telemetry data pipeline that enables data collection from various sources for analysis. The speaker, Eduardo Silva, explains how Fluent Bit allows users to route data for analysis through its architecture of plugins, which include inputs, processors, and outputs. He introduces the new Fluent Bit version 1.10, highlighting support for OpenTelemetry and upcoming features designed to enhance performance and platform security. The session details improvements in metrics processing, such as the new cumulative to delta processor, and demonstrates the advantages of Fluent Bit V5 over previous versions and competitors in terms of CPU and memory efficiency. The speaker encourages audience participation in the project and promotes further engagement at the Fluent kiosk.

How are you guys? All good? I know we are after lunch. We need to get this energy up. So, I will try to this as much informative as possible. My name is Eduardo Silva. You might be familiar with Fluent Bit project or Fluent D. Please raise your hand pretty quickly. If not, I will do a better job explaining what we solve. And here we go. So, telemetry

data or inside observability exist in many forms. And nobody wake up some morning saying, "I want to do telemetry observability." Nobody, right? Because what you want to accomplish is data analysis, right? But in order to accomplish data analysis, you want to move the data that are in your host, in your logs from your application or metrics or traces, and send that to a place where you can

analyze it. And that's what Fluent Bit. Fluent Bit is a telemetry pipeline that allows you to collect all this information through different components that are called inputs. So, how do we collect data from multiple sources? We have the concept of processors where you take this data, sometimes you remove duplicates or obfuscate for security, and then you allow to route this data to the right backends for analysis.

And I say backends because some people might leverage Splunk, others might use Google Stackdriver, or Elasticsearch, and so on. So, Fluent Bit as an agent, as an architecture that is based on plugins. And these plugins I need to implement these different components that we were talking about in in the pipeline. So, we have for inputs, for processors, for outputs, we have for example Lua for the scripting.

We can fetch Kubernetes metadata, and so on. And where Fluent Bit runs, it runs on IoT devices who aren't run Linux, any type of node, BM, bare metal machines, or you can deploy it as a gateway to receive data and aggregate them. And the principles of this project that has been running for 10 years is like it be high performance with low resource usage. The goal, is

you're going to run your your car really, really fast, you want you don't want to waste all your gas. That's what Fluent Bit does and it's fully vendor neutral and it has a whole broad ecosystem support. And today we're going to introduce uh what we are achieving with Fluent Bit B5 that is being released today, hopefully if you see I passes. No, it is all good. It's

all good. So, the first one is all um um Well, let me give you a a brief. We're going to talk about OpenTelemetry, data processing, platform security, and performance in general. So, from interop interoperability for OpenTelemetry, we just updated to support the protocol in its version 1.10 that comes with profiling support in alpha. And we're always trying we're very opinionated how do we handle the protocols. We

always want to be really, really performant. So, having a lower CPU and lower memory overhead is a must for us plus security. From the metrics processing angle, we ship a new processor from metrics for cumulative to delta. So, if you'd like to collect metrics and you want to reduce and just collect the delta between your sample of counters, yeah, you can use this with your new processor

available. We already support selection, but this is like the next step. And in general processors, we implemented a new processor for data analytics that is called topological data analysis. We're going to talk more about that in this Thursday. Uh we implemented a better performance when doing trace sampling. Same as the OpenTelemetry Collector, we can receive traces and do sampling head and tail mode. And also be able

now to process all the auto log attributes. And we re-architected internally how our HTTP server receive connections connections and can scale up to process in all its layers. Plus, in the security aspect, we implemented OAuth 2 in a in a way that is very generic in the agent and any kind of component can use it to receive secrets and sharing them across And from a performance, we

as I said before we care about efficiency, throughput, and actually that ended up being like I'm going to pay more or less money when deploying in the cloud, right? So every optimization that we do, we might save a few cents per CPU cycles. And we did a a previous test, we did a test with Fluent Bit version two, 4.2, with Fluent Bit V5 versus Open Telemetry Collector

latest versions default just to get a baseline. And here the biggest is the better in terms of what was the throughput be able to process OTLP payload. And Fluent Bit V5 well exceeded Fluent Bit 4.2. In a CPU utilization, the purple one or blue one is Fluent Bit And the red is version version five and the collector is the green. So the lower the CPU, of course,

the better same as for memory. And this is CPU as a a raw view. And in general, this is what we can see when we compare one version with the other versus other options in the market. That's what we care about. So I wanted to introduce the project, give you the new updates. We are just on time. So but I would like to invite to everybody to

join or participate in the project, but also visit our Fluent kiosk in project pavilion where you will be able to chat with maintainers, grab your Fluent Bit t-shirt. Every KubeCon we have a new t-shirt for a new release. And also if you have contributed to Fluent Bit, we have some special swags for you as well. And we have two extra presentations, so I'm happy to chat with

you and just say hi, we see around. Thank you so much.