Fluent Bit V5: Pushing the Limits of Observability at Scale - Eduardo Silva

In this talk, Eduardo discusses the advancements in the Fluent Bit project, a telemetry data processing tool that functions as a lightweight agent for log and telemetry data collection. He emphasizes the importance of Fluent Bit's role in handling the increasing volume of telemetry data produced by various environments, including containers and cloud services. The speaker explains the structure of a telemetry pipeline, detailing its input, processing, and output components. Eduardo highlights the latest features introduced in Fluent Bit version 5, such as improved performance metrics, support for OpenTelemetry, and enhancements in processing and routing capabilities. He also discusses the project's vendor-neutrality and the community's contributions, reinforcing Fluent Bit's focus on high performance with low resource usage.

My name is Eduardo and the presentation today is around pushing the limits on the fluent bit project. I assume that some of you know the project. But if you are new or just are getting started on this, please raise your hand so I can Oh, you all if you're using it. Perfect. Okay, we have I need to justify my first slides, right? Because we're going to get

back to the basics. But um well I started this break many years ago. We are from the fluenty team and really happy to have this opportunity to share some knowledge and it's always good to go back and think and analyze why do we have a project right we create stuff we create projects but it's always good to understand why do we have this what type of problem

we are trying to solve. And the biggest challenge today is that telemetry is everywhere, right? Every time we got more deployments, more services, more containers or or metal machines or embedded devices that chips even more telemetry, right? But as I said days before, nobody want to wake up in the morning and say, "Oh, I'm going to do observability." That's boring, right? You want to do data analysis.

Nobody want to manage the agents or taking care of this actually fluent bit is something that you run it and you would like to forget about it right um so telemetry data is everywhere and if data is everywhere means that it's in different places and different formats and if you want to do data analysis which is your final goal you need to have a way to move

this data and centralizes centralize it in a place for data analysis right like a database splank elastic Open search or any solution that is available from a cloud provider. And where fluent bit sits, it's in the middle. And fluent bit basically is like what we call a telemetry pipeline. Right? The moment that you start understanding the lower concept, the best you can have a better implementation. And

a telemetry pipeline pretty much has inputs where data collection happens. You have processors which allows you to modify the data that is being collected. Then you have a routing phase that says hey where this data will go and then you have the outputs which are pretty much connectors for backends like databases or cloud services. Example of inputs for example tail read log files sys log receive data

from open telemetry applications. We can do promeus scraping. We can collect kubernetes events. So it's not just to receive over the network but also from the local system even systemd if you want to process systemd lock logs from your Linux box you can use this type and for processor which allow you to modify the data or enrich it we have Lua scripting so you can have your

Lua scripts in your configuration we have Kubernetes metadata enrichment sampling for traces and we have more than a hundred right but this is the general idea and enroing is like what type of policy or action do we have to decide how to route the data to a different place and outputs are pretty much connectors right so if you're going to send the data to an endpoint that

is based on open telemetry you use the open telemetry plug-in if you're going to send it to Splunk you use the Splunk plugin and so on and this is a pipeline pretty much input output and processing in the middle and why this is important to understand even is very basic Because when you are deploying this in production sometimes you get problems right or the data is not

flowing there's a network error so you can understand okay where the problem could be could be in the network when fluent bit is trying to reach an endpoint or maybe something in the pipeline we have back pressure or we have something going on okay as I said fluent bit is a has a pluggable architecture right that means that you can create and extend it in many many

ways adding different type of e plugins process processors or adding your own output connectors. Companies like Splank, Amazon, Microsoft, all of them has contributed back to the project, right? Being a bendon neutral helps a lot to the community and that's a feature set. But now when this runs, what are the the features that we have as a telemetry pipeline? It's like we support high concurrency. Meaning like

if you want you have lot of capacity in your system and you want to be able to process more data you can use concurrency. We have safety around buffering and buffering with storage in the file system. meaning like if your input going down or you were receiving data influent bit and you had to restart the service you don't lose data that's the goal of course corner cases

always happens in all type of system and also you can extend it as I said with lua with golang with and with zik we are launching the new zik SDK in a few weeks we didn't made it for this cubecom but it's almost coming and where fluent bit runs it's not just kubernetes can be on the edge on a Raspberry Pi or maybe Kubernetes running on the

edge. It can running containers, virtual machines and also can be used as a getaway to receive data and aggregate before sending the data out. And the project is always go to BI and say has been developed with these three principles in mind. High performance, low CPU, low memory, vendor neutral and having a broad ecosystem support. Okay. So, and that bring us to okay what's next for the

project because today we're launching what is called V5 and Fluent is evolving now into a data plane for observability. Historically it was set in as an small lightweight agent on the nodes right that many many years ago the goal was to send data to fluentd and fluent was the main aggregator right and we didn't want to overlap too much but I think that 2026 time to take

back that seat and since fluent bit can scale even more just putting as a full data plane not just a collector but as a processor aggregator where you can do sampling and you can do many smart decisions but keeping our values right our principles of the project it doesn't make sense to build something like some monsters it will be really slow right okay so I imagine that

most of you um are have a strategy or are thinking about open telemetry right how to standardize you know the data collection shipping telemetry data from your own applications and most of the question that we get are like okay where fluent bits connects to open telemetry. Okay. And these two days that's was a a really constant question. That's why I wanted to add more context here. So

open telemetry is a standard a way to define how the data is schematized has a format how this data goes or flow through the network and how applications can create data into this schema and how they can send it through this transport. So the way that fluent bit approach the integration with open telemetry is that we support the schemas of hotel same as we support the schema

of prometheus right and we also speak OTLP which is a open telemetry protocol. So if you have any application that ships data you can have fluid receiving that information doing the transformation and sending that information out. H we support logs, metrics, traces and also with the we're chipping today we also start supporting more formally what is called profiling or profiles which is a new signal that is

still in alpha in open and the way that we approach this is that hotel support for us is mandatory mandatory in terms of if the community on the industry is moving to hotel we need to have we need to be the best citizen into that eosystem system as we have done with syslot with tail and with other protocols in the market. So today you can enjoy all

of this and also we have done in the this version another security updates with the integration of new oid 2 implementation from the processing area we're keeping a new metric processor our metric processors actually were not processing data they were filtering data before we can take metrics and filter them by levels or just select by specific conditions but now the the first processor that actually modified the

data generate a new a new sample is commumulative to delta. If you're using the other collector, you might find that this is something very uh similar in terms of configuration and functionality. Other processors enhancements and is called one of them is for analytics that is called topological data analysis and let me be honest I barely understand how it works. However, we have Hirosh for here from Japan

who did implementations. So he's around and he will have a presentation this on cubecoin Japan right so how to implement this and how this works and the value that delivers so and just the face delivering the news but not the one who did everything here and from a performance perspective we you know that we do sampling for traces like head and tail sampling now trace sampling is

10% faster we have optimized how do we do the buffering how do we reduce the amount of copy of data while we have this window we are correlating in be before doing the sampling of data and that's why it gets faster if you allocate less memory you have less CPU you get better performance another enhancement is like when you take the data and specifically from open telemetry

you have different context you have resource you have scopes you have attributes that are either in the lo records or also in the resource or scopes and Now with our processor to modify the content of traces, metrics and logs. Now you can modify the open telemetry attributes. Talking about processors and transformation, who are you are using the classic configuration mode? Please raise your hand. I assume that

99% moved to YAML already. Oh, that's great. That's great. I thought that the answer was going to be different. Okay, so processors for those who are new are um a new way to deploy filters inside a YAML configuration. And what is the biggest difference? Before you used to get the information, define your collectors, your inputs and say these are the filters and you try to do a

match between T and matches. Oh, this is going to filter this. Now processors are well they have native processor who are faster than filters are attached right away to the input plug-in. For example, if you are data from tail who's reading log files or open telemetry, you can attach a processor that will be just work for that input plug-in and it will generate a final event that

of course get in Q into the pipeline. Now something that we chip like well it was few months ago but it's good to share here is like conditional processing. Sometimes you don't want to process based on tax. You want to process the data based of if the data contains certain pattern like hey we have a key that its value is equal ABC and then you want to

do some conditional processing and you can have multiple processors with a lot of conditionals in the middle. Of course your configuration can grow a lot but if you want to bring your business logic into this this is the way to do it. And same as we have conditional processing, we have routing with conditionals. This is new. I'm sure you have not heard about this. So when you

get the data and you're going to route the data to a destination, you used to set attack and a matching rule between the source and the destination and things that were matching when brought into certain place. Now inside the same input block where you define um the data ingestion point we have the processor here on top. For example we are processing the data always processing happen first

and the output of that can optionally go to a routing section inside the same input block. And you can have conditionals and say if the data contains this pattern or this value route it to a specific and this gives you more granular control of how you move your move the data into Now we have other ways to do it this using the default configuration in fluent bit.

However, h when you have multiple rules, you had you had to do rewrite tagging, creating a new input plug-in, it was more complex. So, doing these type of things is complex in general. However, this solution has simplified how you can accomplish a conditional routing even in a in a better way. platform capabilities. When you have more than an agent, this data pipeline that is receiving data, of

course, you can have hundred or thousand of nodes trying to send the data and fluent bit historically when receiving data over the network specifically over HTTP that was done in just one thread, one worker that worker was capable to handle hundred of connections without any problem. is a synchronous event driven all that you want but I'm sure that all of you even in your phone you have

more than one CPU right so why in the servers were just you know sticking to one CPU we can have multiple threads trying to accomplish and deliver more value so we implemented multi-threading so we sorry we unify the HTTP server API inside fluent bit this is a very internal detail and now we support multi-threading so for example if you have the input plugin that is called splank

that time to receive Splunk data, you can tell it, hey, spawn 10 workers. So now you can scale up even more. And this has been implemented for different plugins like HTTP, Splunk, Elastic Search, Open Telemetry, Prometheus, Remote, right? Now I think that H with this change you might imagine, oh, these guys added ton of code. Yeah, we added ton of code with a lot of help with

AI, but we all AI also we were able to remove more than 4,000 lines of duplicated code because before in order to handle HTTP 1, HTTP2, we we had a lot of redundancy in the API because things were built over time and the the API was never too clean to just reuse something in a in a good way. Now everything has been unified. So you have more

features, you can have multi-threading and we have less complexity in the code base that handle all these futures. Now from our platform capabilities, one request that I don't know we got it many many years ago was the ability to have the internal logger not to just log the internal logging information of fluent bit to a standard output or standard error. Many people told us hey why we

cannot route the logs through the pipeline. So before we can do we had this kind of method FLB info warn and error kind of internal APIs of fluent bit in the code that talk to a logger which another context with an API and ship shipped the data to standard output standard error or just a file. But now in B5, the biggest difference is that we have rearchitected

this in a way that there's no changes in the in the code of fluent bit but just in the logger. But now the logger is able to spawn a different thread. And if you enable the new plug-in that is called fluent bit logs, all the internal logs can be routed through the pipeline. We also have fluent bit metrics. We introduced that some time ago. And what you

can do? Oh, you can do filtering, processing, the same thing as you can do with the other logs. Of course, try not to reingest in a loop your own logs, right? That happens in containers sometimes. the the biggest enterprise thing when adopting um this technology and I'm talking about big banks and financial institution they are really strict about security and how system talks to each other so

you can have a a special secret or you can have a special token to author to authorize how the communication happens but people say hey we need o2 we need an IDP we need we need to have a special called server that works also from a client side but and from a server side. What that means if fluent bit is sending data to another fluent bit which

is behaving like a collector this fluent bit needs to be able to retrieve a token to say hey I'm this agent and I want to communicate to the server. So the O2 server grants a token right and the client this case fluent bit or any other app can use that token inside the request when sending that to fluent bit in the other side fluent bit receive the

request get this token but it's not about to just let it pass it need to go to the server validate this token and say okay you're good to go and this is how the communication happens now this implementation is agnostic inside the the API of fluent embed. So now plugins like HTTP and open telemetry can fully use this in a very um agnostic way. Now we always

talk about this performance CPU memory high throughput. We have done a lot of optimization with SIMD operations faster train sampling I duplicated that. But let's talk about the important part. So if we we did a um a benchmark as how fluent bit can behave from a performance perspective when receiving open telemetry data. So if you compare fluent bit 4.2 we did another eon to just make it

relevant. you will see that here's the biggest difference when you don't have workers or you have one um and fluent bit was enabled with four workers but barely use it two and yeah the collector of course h behind here is like throw put from an open telemetry as receiver and just getting the data accepting the data packaging internally and discarding the data we didn't do any type

of processing because that is another type of benchmark Of course, I always suggest do your own benchmarks, your own use cases with be yours. Don't trust mine, but I'm sharing what our own results. Now, this means like if you were using fluent bit for the two and you are not receiving hotel data, hey, that's fine, keep using it. But if you care more about performance, other type

of optimizations, I encourage you to switch to B5 because you know that we're going to maintain for two for a few months, right? Okay. So more row details. So here we what we have here in the screen is a comparison between fluent bit 4.2 B5 and the collector. This is throughput the ability to receive data in a fraction of time. Right? So you will see that in

the table we had the minimum the maximum the min and the medium. Now, I always say it's not the same to run a cast very very fast than running a car very very fast, making sure that you you optimizing, you know, the fuel for that car or the battery or the energy whatever, right? So, you can run 200 kilometers per hour, but it's about how much gas

you need to achieve that and to be, you know, con constant over time. And that is a reflection also here in CPU and memory usage. Right? So this is fluent bit for that two. Of course it use it less. And this is where you have your pros and cons. A I can use a little bit more of CPU but it can achieve this. You know there's a

trade-off always. And in memory usage it's always similar thing. Of course there's always room for more optimizations. But I think that with B5 we are in the right direction as every year we try to push our boundaries and to be honest our users push our boundaries right. So this is really good news for where are we heading and going and this is CPU usage in a more

row analysis way. The the blue line is fluent bit 4.2 the red is V5 and the green one is the hotel collector. All of them are the latest versions default configuration nothing nothing fancy. So what is interesting is that a fluent bit uh well the default was four workers but barely use it under 150 percentage of CPU that means less than 1.5 CPU in this case. Okay.

So these are are this is how we really care about the development of the project because I'm sure that the amount of data that you're processing today it will be times more that the end of the year or the next year right so I think that with this you will be kind of covered. So, Fluent Bitb has been all about compatibility with open telemetry, giving a better

integration, a better experience, better capabilities for data processing, platform security and performance. So, if you want to learn more even more more, we have another session tomorrow where will be more agnostic where we're going to talk about system calls, multi- threading, buffering, storage, and how H these things works underneath. Sometimes you want to know how this car works in general and sometimes you want to learn how

the engine works and how the operating system provide you certain capabilities that help you to accomplish these type of things. So you are all invited otherwise you can watch it offline when you are back home and I appreciate your time today. Thank you so much. [applause] We have some time for questions. So any question related to this content or in general please here's a microphone feel free

>> maybe just a quick question to the security topic u what is used there for the JWT validation is a lure scripting or is it part of the core >> sorry the question is what is the use >> no no what is what is used for the JWT validation on fluent bit side >> oh for us actually we are agnostic but we happy users who are mostly

use O2 >> but they have the extension with JWT because JWT is an extension RFC723. >> Yeah. How is it validated which with which mechanism on your side? >> Oh the same we use the protocol validation. So we use the same main >> So you expect that the same server is able to generate the token with a signature but you are able to also validate that token

back. >> Now that's clear but in matter of performance uh >> oh we use cache. So in the server side >> when we receive or you have every token has an expiration. >> Yeah. So while you have this there you have a cache for that expiration. We have a hash table and when it's about to expire we renew or we validate. >> Okay. >> Yeah. Thanks. >>

Hey uh great presentation. >> Um I got a question. In what percentage uh fl bit v5 may replace the open telemetry collector in more or less >> yeah I think that we take a different approach on how open telemetry as a project works >> in my vision in my experience I see that open telemetry try to replace everything right in our vision fluent has always been like

a plugandplay solution >> like don't if this is working don't fix If you need an extra future, try this out. Plug it. You have all the connectors. So, we don't look to displace open telemetry. Actually, we integrate with open telemetry. >> But of course, the overlap is with the >> Now, what is the value of the hotel collector for companies that they can easily extend it in

go and create their own distributions. But also that has its own site problems like fragmentation, be behind about updates and you have to maintain one product. Mhm. >> Because if you if you you have a fork, right, there's a intrinsic cost on that. So we try to be an alternative to our users. But most of our users sometimes they replace the collector with fluent bit or they

have fluent bit talking to >> So try to offload certain processing things that are too heavy for the collector. They do it in fluent bit and they send it to the collector. So I think you should take the the approach of what is the best for you for your environment and yeah we integrate with everybody. I think that's the whole thing. >> Sure. Thank you. >> You're

welcome. >> Hi thank you for the uh presentation. Uh perhaps a bit off topic but I have a question on uh observability in general and um there are lots of graphs on usage on memory CPU etc. and sometimes you see spikes and sometimes see a huge drop down. Um but that is not really in context. So do you have any recommendations on how do you address making

it in context where you can see that it's uh in a certain pipeline that you run where there's a certain event attached to a certain spike or a certain drop down. Okay, it's all about so the main question is how I can monitor and know that things are working fine because I I know that you see the drop here. Is that because we stop at the test?

>> Sure that makes sense but in other cases that would be >> yeah it won't be. So every project or for pipelines like hotel collector fluent bit vectors which is from data dog all they have their own internal metrics and usually provide their own graphana dashboard with alerting with rules. So in order to understand where the problem is, it's like first step zero is always monitor every

component that you have right now. If you want to get alerting or understand even in a deeper level, you might try something with EVPF or what cloud provider does and they provide you like a more complete suite with their own tooling to understand what's going on or what type um processing was stuck at some point for something. There are so many use cases but I would say

that internal metrics internal logs it's it's a the primary thing. >> How do you process those metrics and logs? Yeah, you have graphana or you have vendors who provide you a more complete suite for that. We try to focus mostly on ingestion provide the metrics and let the others handle that complexity. >> Okay. Thanks. >> Hi. Uh first uh thank you very much. It's a very nice

project. Uh and if I may I want to describe like kind of difficulty I have in my organization. So what I want to try we get like logs to fluent bit let's say like from sys log >> Then I have like thousand lines script which does like uristics like does parsing some a lot of stuff which is inside fluent bit as a filter. But what I want

to have in the end I use like this like new five arguments uh call because I want to also create like open geometry compatible metadata to to stream it as open telemetry compatible schema essentially. >> Yeah. So from cyswalk yes to audio uh I haven't managed uh I'm doing probably something wrong haven't managed to ask uh fluent bit to actually define resource attributes field so basically group

attributes >> I can could do attributes but not maybe you have some advice or maybe we can discuss it later so as not to occupy time now >> yeah I think that this image kind of represent what you are seeing is like you're collecting the data from thank you from syslo >> but this data doesn't have metadata doesn't have a structure yes absolutely >> well have a

structure but syslo >> yeah exactly >> so there are different ways in fluent bit first one if it's really row you can apply the open telemetry envelope >> which will do is do the enrichment for >> but it will be just kind of the >> inside fluent bit >> okay once you get that infrastructure in place there's a plug-in that is called content modifier here. >> Mhm.

>> This one and content modifier is allows you to modify different components from logs on metric traces. >> And we have a special open telemetry >> To play with the resource attribute, the scope name, scope version, scope attributes or autolog attributes. >> Mhm. Mhm. >> So what you do is you take the data, you apply the envelope and after that you modify by using the special context

because you need to tell the agent where to modify the data. Okay. Okay. Thank you. >> Welcome. >> I will build upon on our fellow practitioner here. Um what you're showing actually is quite good in the right direction. But what if I want to even further change the uh resource attributes? Uh for example, in the war processor, it's a read only type of thing. So you can

get again if I'm using the open telemetry envelope, then I get my structure data into the shape. Then I go into war processor and and there the resource attributes are read only. So I cannot modify them. But maybe I have additional means to put additional context in this open telemetry resource attribute either later in processing pipeline. So in in war processing po point or in exactly this

point where you pointing cat here it's pretty much static what the value I can plug in into the um into the value but I need a flexibility exactly at that point where I can query maybe some portion of the work that is coming and maybe there I can find a a value that makes sense in a resource attribute context of the open telemetry. >> Okay. So, basically

you're looking to enrich the attributes based on some special key from the content. >> Okay. I have a question because we thought about that but I didn't have the answer. Okay. You have a resource. You have five records. How do you decide if you have the same key on all records that you are in to use as an attribute? Which one do you use? Um the concrete

case I I have let's say in Kubernetes environment we have a multi-tenant um system based on name space for whatever and you have a attack um when you read a file you have a T and the T structure you pretty much can guess without going into the cube API server you can pretty much guess the name space the container and the pot just following the tax structure

the concrete problem is I that I have is I can guess that the the name space for >> but I cannot find the place in the overall pipeline where put that name space as a resource attribute early enough in the whole processing pipeline. Oh, okay. I got it. I I understand what you mean because all Okay. Because that data is coming from the locks >> and the

Exactly. And the purpose is if I if I run a huge Kubernetes cluster, I don't want to go to the cube API server and fetch the these attributes from the yeah curing non-stop my cube API server getting this metadata which I can further on. I want pretty much early in the early phase possible of the processing data to be able still to create the resource attributes based

on the information that I already have from the tax system. >> Okay. So this okay so this is no content modifier. It should be the Kubernetes >> I don't want to use Kubernetes filter for performance reasons. >> we need to get this name space somewhere. >> Exact. I have it. I have in the tag when I read the work from the file. >> Oh, you have in

the tag. >> I have the tag. Yeah. >> Okay. So, you need to have record accessor functionality for resource >> Cool. >> Yeah. Let's ship it. >> Yeah, that's that's good. Thanks for the feedback. >> Yeah. Because you guys are I'm I love the project. you are all about high performance things and when you go and think about those kind of high performance concerns or or constraints

these things matter. Yeah. And >> yeah performance and usability it's a mix of those. >> Okay. But thank you. Thank you again. >> No thank you for the feedback. Yeah. Appreciate. >> Hello. Thank you for the presentation. I would like to have a little bit of uh disambig sorry uh clarif clarifying that uh uh in a normal standard EFK stack you have fluentd which is doing the

uh aggregation and you kind of imply that fluent bit with the newest version is doing more and more but still not want to step on the toe of of uh Fluentd, would you still uh or would you actually recommend actually leaving out Fluentd uh from a normal uh or a standard setup and using only fluent uh fluent bit to uh handle somewhat of the aggregation before it

is ingesting into elast elastic? >> I would say that it depends of what type of processing you're doing. If you just had bare fluentd or you have fluentd plus some ruby gems as processors doing some extra things and that will be a difference if just the first case where hey I'm just using as a regator and you have fluentd and fluent bit I think that fluent bit

has evolved times more than fluentd in the latest years and it has more widely used today. Yeah you should replace it if you have a problem with performance or something. Yeah, why not? If it's working, just leave it there. >> All right. I mean, the platform has evolved uh since many many years. So, basically what I get from your words is try it and measure it. >>

Yeah. Um all financial institutions that we work most of them has migrated from fluent to fluent bit even cloud providers Google, Amazon. >> All right, that's kind of answers the question. Thank you. >> Thanks. >> Thank you. >> Um I was wondering for uh version five, is the checklist filter still uh working still available and how are old filters are they being replicating or are they uh

tested and always uh well taken along with the upgrade of the So the main question is how we are doing with the filters when the new versions. >> Yes. And is still the the the checklist for the for the uh the checklist filter is that still available for version five? >> Oh the checklist filter. Sorry we have so many. Yeah. The way that we implemented Okay. The

way that we implemented processors are there's an API to implement native processor who works faster but it's a backwards compatible way for filters. So any filter can be attached as a processor. >> Yeah. We we take a stance of we don't want to break users. We don't want to break configurations or if we have to do it like a very minimal way and we try not to

deprecate plugins in any way. >> Okay. We don't have visibility who's using it. So, but we cannot just do it randomly. >> Okay. Thank you very much. >> Yeah, you're welcome. Thanks. >> Thank Thank you for the presentation. Maybe a very small question. How does Flvid compare to graphana alloy since they're kind of doing the same thing? Good. I mean uh why should I choose fluent bit

over graphan alloy or vice versa? Which scenario is alloy maybe better? I don't know. >> Well, I cannot speak on behalf of Aloy because I'm not a user as you may expect. Um but I can share like okay so when you have a stack like elastic search in their own time they have a slack they have elastic they have file bits >> or if you compare we

have data do with a platform and we have vector because they acquire vector and so on you will always find that these agents are optimized for their own platform. >> Yeah. So if you're going to be married with that platform for many years, that's fine. So I think that it's a good call to use that because you will have less friction. >> However, if you want you're

thinking to have a more bend or neutral approach where you can switch providers any time, you need a bend or neutral solution. >> Makes sense. I guess >> so I would I would say that I'm not telling you which one to use, but it depends of your own. No, actually I was using alloy and you kind of convinced me with uh this talk. So I will just

compare them uh performance-wise first but seems like fl bit is kind of optimized. >> Yeah. But I always say it depends on what solves the job better for you. >> Just compare the memory. [laughter] >> I know. So I have a question about maintaining >> Um you have a lot of PRs and uh you you it seems like you have not enough review capacity. Uh uh meaning

uh there there are a lot of good PRs that I want to have in Fluent Bit. I have one PR myself. Uh uh uh what can I do? What can I do for you >> to make it easier >> uh for you to review it? >> Yeah, that that's a tough question because we have so many PRs. They have a huge backlog >> and and for example,

how the process works is like we take a look at the PRs. So, okay, there's not just PRs, it's like issues and issues are enhancement, feature requests. Oh, this is not working. It's using a two years old version, right? And then we have PRs. So we try to balance between this. Now for the PR to be reviewed I think that the first thing is try to make

sure that the CI passes but also we are having a lot of issues with CI. >> So even before investing time in reviewing a PR okay is it passing the CI that's the first thing. >> Or we also we get for example many contributions that people does not sign off the commits >> and we cannot accept those because of CNCF policies. >> Yeah but >> but how

you can help? Yeah. >> And reach out to us directly. It's like, hey, for example, Hirosh is around here. Where is Hirosh with us? He's a maintainer friend. >> Yeah. So, so the first commit I made, so the first PR I made, he reviewed it and it was merged Sunday, I think. Yeah. Yeah. So, so that that's good. But that was a PR from three months ago,

four months ago. Oh, >> okay. >> So, so that's just great. Um uh but uh yeah. So what is your selection criteria? When when when are you going >> selection criteria is like demand people raising hands on GitHub right >> but but it's quite like you go into like a place where there's issues GitHub you have Slack many things so it's really hard to prioritize. So what

is really important to dedicate the the limited time now if you can come up like hey it's me this is a list of the PRs that has been approved that will make our lifetimes easier because here review PRs or people review PRs but in order to merge them we need somebody to like tell us hey this is ready to go this is ready to go some of

them are really easy to go because there are fixes okay you got a problem and this is a fix >> others are like, "Oh, we have this new fancy future. Why exist? Why it's touching all these components?" So, it depends. But if you can help us to put a pro better process in place, you're welcome. >> Yeah, we need help. That's >> not going to lie. Like,

>> Yeah, we can chat about that actually after here. >> Yeah. Yeah, I will think about this. >> Thank you so much. Yeah. Hi, thanks for the great presentation and I love the product. Uh we use it uh for log shipping from like 15,000 uh clients. It's very great and it's very performant and uh I was going to ask uh in this uh in this conference everybody

is talking about open open telemetry collector and I was a little bit okay did we choose wrong product but with this presentation uh I see that we can use flit for traces and matrixes we didn't uh uh we didn't start sending metrics on traces yet but And I have zero uh experience when regarding to traces. So I got a question that might be >> silly but >>

uh trace generate lots of uh less of lots of data matrix data or no trace >> and uh does fl bit have a kind of filter to drop irrelevant uh traces uh shipping to the uh >> yeah um well in in traces in tracing All this is called sampling. >> Yeah, sampling. But um >> so for sampling we have a head sampling. >> Right. I don't know

if you're familiar with the concept. It's like take a >> decision about certain pattern or trace ID to just drop some of them because tracing is really noisy. It's is really noisy. And I said that trace is not a problem. The problem is who put the trace into the code without thinking about all the thousand traces that exist. >> Right. So, so we have sampling head sampling

and this is Oh, this is the same one. Oh, this is explaining the Oh, somebody update. Oh, I think that Eric the documentation maintainer updated this. Yeah. So, we have head t sampling for that. Now, um how to reduce the stresses even more I think will depend of your criteria. whole you application are instrumented >> but I know that it's a big problem I'm not an expert

on instrumenting application I'm trying to solve the problem okay what can we provide to give you a better user experience to reduce this problem that was generated here >> actually uh if I say uh let's say I think the logs get a uh trace ID uh attribute and let's say if the log uh is another attribute ute which says that everything is okay. This is a successful

thing. So by using this uh log information and the trace id in this one can for example drop all the all the uh span information or trace information already at the fluent bit so that you don't send this >> that would be another selection criteria. Yeah, >> I don't know if we have something like Yeah, this is really simple. If we have this XYZ, just drop it

for traces. I think we don't have it. >> Okay, >> we have a really more complex sampling >> Um, spank cone. Oh, string attribute. Yes, >> it's a string attribute is a condition that you can apply in the sampling rules >> to say that if a specific key >> matches based on a condition certain value, >> you just can sample it or not. Okay, >> we found

it. That's good.