How Telemetry Data Moves: Lessons From Building a High-Performance Open Source Agent - Eduardo Silva

In this talk, Eduardo discusses the fundamentals of data movement within systems, focusing on how data is buffered and processed. He explains the interaction between kernel space and user space, emphasizing the role of system calls in managing data flow. The session also covers data collection and the importance of normalizing data formats for analysis, introducing concepts like MessagePack, a binary format for data representation. Eduardo illustrates how to optimize data flow through concepts like buffering, batching, and the use of event loops, highlighting the importance of multithreading in improving performance. He draws parallels to established projects like Fluentd, detailing their architectural evolution over the years. The talk aims to provide insights into designing efficient data pipelines and understanding the complexities involved in telemetry systems.

Thanks for coming. It's full. I appreciate the time. Who's leaving today? Almost everybody traveling back? Okay, I promise you won't be late to your flight, okay? Would you mind if I take a photo of you guys? This full back. Just raise your hands like There you go. 1 2 3. Awesome. Well, this is almost one of what one was the one of the last session of today

of the this KubeCon. So, I appreciate you being here. And actually it's always a bless that when you get this time is you know that people is around and you have less sessions so you will get more folks. And but it's not about me, it's about you and I appreciate the time to try to to learn about how data moves in a in a real system. We

have been doing this for a while and we always pitch our own projects, our own stuff, but behind every single project there is a lot of engineering a lot of failures, a lot of success and it's always good to share um how do we accomplish things, right? So, that's why we're here. Uh for those who don't know me, my name is Eduardo. There you got my email.

I've been around the fluent and fluent bit ecosystem for almost 10 years. A bit more. And let's get started. So, this is not a product talk. I'm not going to pitch you a solution or something like that, but this is about how data moves uh how data is buffered and got how it gets processed, right? I know that you might have a lot of questions at the

end of the session. If so, please you will have the microphone or we can talk after uh the session in case we don't have that much time. Okay, so the core path. There's something that is called the kernel, the operating system, some something that we call the agent or the user space application. And there's a concept of buffers where we accumulate data and outputs. And all telemetry

data, all telemetry pipelines follow the same pattern. We get a mix of operating system, user space application. We get the data from some place which we process and deliver to an special point. And if you look at any type of product, any open source project, it's the same. We have what we call inputs or sources in wait, how do we retrieve or collect data? Then we got

the concept of processors that we take this data, we do some processing of it and then we ship this information out. Okay? Nothing is too different from that. Okay, we're going to get back now for a second to operating systems one on one. Don't be scared. This is very high level and nobody want to get too much in detail. Who's familiar with kernel space and user space?

Oh, you are. Are you mostly user space or kernel space? Both. We have a kernel developer here? Also, most of you are user space. Okay. So, we play in the unsafe place. We break things, right? Because the kernel cannot break. So, if you take a look at any type of operating system the way to make sure that it works and doesn't break is to add some protection.

So, the kernel itself, the primary job is to abstract the hardware, You just want to type one key, oops, you want to type one key and you want to make sure that after typing that key, that hardware signal get trapped somehow internally, it gets represented in your screen, right? As as a beautiful character which is a bunch of pixels. However, Um, we always play safe in user

space and that's fine. That's where we can break things. What cannot break is the kernel. But understanding how this interact with each other, even if you're not a kernel developer, can give you full advantage on when you are developing a solution or even when you're troubleshooting something. Right? It's pretty normal to to blame some component when something is not working. But actually you have to have a

very realistic view from both component, user space and kernel space. So, the kernel is the one who talks to the hardware. The way that our application talks to the kernel is through what we call system calls, right? There system calls to open connections, to write data over some buffers. There system calls for everything. And our applications sometimes abuse of system calls, right? If you use more most

system calls, sometimes it gets more expensive at a in terms of CPU And actually if your application is generating some data and you want to send that data over the network, it has to go through a system call. They go through the current the right kernel interfaces that interact with the Okay? So, but normally the data that you share from user space to kernel space is not

most of the time a reference of data. Meaning like this data exists here, take it and do something with it. Actually what the system call will do most of the time is generate a copy of that buffer of memory just to play safe before doing any extra operation. Unless you have some serious strategy interaction in the system. And why is this important to understand? Because the moment

that you start moving you need to understand the whole cycle from a very global perspective. Like from your application go through a system call to the kernel space to the hardware and that hardware communicates with other hardware layer, other machine, other virtual machine, whatever it is, right? And go through the same cycle. Basically, this is how two applications can connect. Right? You might create a TCP connection

on one client, one server, but behind the scenes all this data flow is happening. And when we talk about hardware very realistically we talk about CPU cores, memory, disk that we use for storage. So, if you think that you are collecting or generating data, you need to process this data. You might tend to use CPU cores. You might tend to use memory or disk, right? In case

you need some consistency or if you need to generate a copy of that memory in order to do a modification. So, these are our resources when we are developing applications. Of course, today if you are generating a web app, it won't be a problem because most of these things are abstracted. But when it's about to moving data, like um like a system level application, it's not something

that most of the time will have a UI, right? It's something that is running behind the scenes. You You know that it's running, you forget about it. However, that application that you're not seeing, that is running, it's using CPU cores and all these resources. And of course, you don't want You want to make sure that these resources are wisely used. So, data moves from kernel space into

user space through system calls. That's the first thing that I wanted to share with you. Now, if we go back to the main topic of data, right? You don't collect data because you want to collect data, you collect data because you want to do data analysis. But, as you can see, data comes from different sources. It could be a firewall chipping data over UDP. It can be

data that it exists in a log file, or maybe it's a data being generated by systemd in the Linux But, how you collect this information and achieve your goal, which is data analysis? And it's not too straight because you have an environment, you have operating system, you have network, you have you have hardware. So, when you're moving things can go wrong on any place. And of course,

all of us for those who code or by the by code today, it's like we develop things in a way that oh, this is how I want that this will work. And likely will work. However, with experience you learn that you have to prepare for something for the worst case scenario. What would happen with that data when things goes wrong? Network outage, power outage, disk saturation, memory

corruption, or whatever you can think. There are many things that are in the middle that have to be considered. Now, if you're going to do data analysis or you're going to move the data, it's not just to read and write. There's more process involved. And since same as the data comes from a source, this data likely come in a specific format. And source of data B might

have format B, and same as the next one. Actually, this is like a different developers in the same company trying to instrument logging, right? This is the same thing. And so, how do you normalize? How do you make sure that your data has a specific format in a way that you can do your analysis properly. And this is where we start adding more components, right? As uh

data parsing or serialization for encoding. So, you're seeing a new concept here. Are you familiar with MessagePack? Or have you heard about it? Some of you, okay. So, are you familiar with JSON? Yeah. Everybody knows because we can read it, right? But MessagePack is a binary format, okay? And when you read the data and you want to have a some normalization, meaning like I don't care which

language you're speaking. If you're speaking German, Spanish, English, I want everything to be translated at a single unit. For example, Mandarin Chinese. It's just one example. And from there I'm going to do my analysis. But I need to have this way to decode and encode in a different way. And of course, when you convert or transmute one of piece of information in another one, you need to

have a format, a format or a way to structure this information. So, for those who don't MessagePack, MessagePack is like a binary JSON that has been around for 12, 13 years. And basically it's a binary format. Uh in JSON, for example, the same message will take 27 bytes. But in MessagePack will be 18. And there's no compression here. But basically what we are doing is like for

every map that exists, the map has a code. It doesn't have an ASCII character. Right? Same for Boolean values. You don't write true. You have a special and you have a special value that is called, for example, C3 that belongs to And so on. So, it can represent any kind of data types. It's like a JSON. Now, when it's important to have a binary format, it's when

you're going to send the data over the wire. Okay? So, you can have a structure in memory in your program, but that is a structure that's in your memory with pointers, with references. And if you want to have the same representation on the other machine, you need to serialize it. And MessagePack is one format. For those who are familiar with Protobuf, it's kind of similar. The difference

is like MessagePack is a structure, but is a schema-less. Meaning like it doesn't enforce any schema. However, in Protobuf, you declare any schema. That's the biggest difference and change. So, uh for those who are using I do you use Fluentd or Fluent Bit? Just a quick intro. Awesome. Thank you. So, for those you know that you have heard about the concept of chunks. And a chunk is

nothing else that a MessagePack representation of the data that is being collected into the pipeline. That's a simple structure on how do we structure the logs internally in a chunk, right? And that chunk can rely in in the file system. Now, if you have this type of optimization, it's times better than doing it in JSON or other type of um you know, uh schemas or APIs to

encode data. There are many, but we rely in MessagePack, and that's why we are talking about this. Now, and when you are collecting the data, you're doing this processing, you serialize, for example, in our case in MessagePack, and we have this concept of chunks, right? You would like to have this information to be sent out to a destination. Right? But you don't send a MessagePack, for example,

to Splunk. You don't send a MessagePack to Elasticsearch. Actually, who uses Elasticsearch? You might know that what payload receives? MessagePack It expects JSON. So, after you do a normalization, what you do is to take this normalization, my own schema, and convert it back to the schema that is expected by the other endpoint. And when moving the data from one place to the other, you can do it

on this way. Having every chunk or every small message invoke a system call that right I'm simplifying here. Maybe sent will be a a better system call for example, but this is really expensive. So, when you're moving data from one place to the other, remember that we have a lot of data copies between user will always try to do its best to optimize and have a window

to receive the data from user space, but those can be milliseconds. And I'm sure that maybe your user application is not optimized as a kernel and might have some latency. So, this is a really expensive way to move data in general. And I don't know if you know um for those who have assembled an old computer, when you have to put your own video card, your own

network card, or in your servers, you might note that network cards has a concept of buffers. Right? They do a lot of buffering at a hardware level. And that means that they are not sending the data as soon as they get it from the kernel. They just have a a time period, they have a window, they have a buffer, and they send the data because doing multiple

write operations takes a lot of time. Because you have to copy the CPU, there's system calls, the system calls generate latency, and serialization also is expensive. And the way to optimize it optimize this is like doing batching, doing buffering. It's like And this for those who are not familiar with the low-level concepts, this is pretty much like Okay, you have to go to the bank this week.

Next week. After KubeCon. And you have to do five paper works, and you decide to do one on Monday, Tuesday, Wednesday, Thursday, and Friday. It's a waste of time or not? What would you do instead? Do all the paperwork in one day. This is not different. Okay? This is exactly the same. Um or for those who pay the bills every single day, it's always better to just

budget, and this kind of prioritization in in normal life and do it in a single time. Even for focus, the same thing that happened to us with context switching in real life is the same thing that happens with the CPU. Okay? So, make it easier for your product. So, when you're moving data or doing whatever, try to do buffering, try to do batching, and reduce the number

of operations. this was all around concepts, but now we're going to move into what is a data pipeline, and how do we process this information. For example, here we have what we call thread one, which is the main event loop, and the main event loop we're doing buffering. That's mean we are receiving the data, we're buffering internally. Then we have the scheduler that is saying when it's

time to send this data out of the box. And as I say, things can go wrong. That's why we have to handle the tries. Maybe I try to do a a DNS resolution, and something happened. It's not working, failing, and I have to retry. And maybe people like to filter the data before sending this data out. And when the data is ready to be sent, we have

another thread that's in charge of that, same as the thread two, which is in charge to receive data. So, one thread receives data, the other buffer schedule retries, and try to filter, and send the data. This model kind of work, and could be really simple. But, in a world where data is increasing over and over and over every single year, we have to find new ways to

process the data in a more efficient way. Because if you want to add more processing to your data, and you're doing this in your main thread, what would happen? It gets faster or slower? Slower, of course. And what it tried to generate this is not tried. What it's generating is contention. So, even when you are designing your application, you need to try to see where the the

next potential problem could happen. What would be the bottleneck? Okay? Maybe it's not today, but it could be in 2 months. And the solution is always simple. It's like in a in a war, right? It's like divide and conquer. Try to move the concepts to different space to divide the problem in smaller problems and try to see how you can solve in a better So, because here

we're doing the filtering in the main thread, but what would happen if we move this concept as processors instead on a separate thread? That means that we can take advantage of a different CPU. Right? And we're going to reduce the contention that we have in the system. So, when moving data, uh you can have one process, but always consider multi-threading or try to take one simple uh

a specific problem that you have and split it out and find a better solution. Are you familiar with event loops? Kind of kind of. Okay, what is an event? An event. Something that happens, right? um if you want to reach a friend, you have two ways today. Well, you have many, but two main. You can just pick up your phone and make a call. That will be

an event for the other person or you just can wait for being called. Right? So, events in computers at the least when you are coding or creating some type of service are really useful because instead of always trying to do something, you just can wait for the right time to do it. And I'm talking about, for example, if I want to send data and I need to

do a retry, you just don't sleep. You just You can use what is called an event loop and a system where you schedule timers and you say, "Please let me know when this events happens or let me know ping me back in 10 minutes." Right? And what would happen is that your code is going to be called. Now, what is the good thing about event loops? It's

like you can implement asynchronous IO. So, for example, what means asynchronous is like I'm trying to reach out from one machine to the other. And if the other machine is a slow because of network or any type of problem and we cannot communicate fast enough, there are two ways. Or I just sit and wait or I just say, "Oh, this is going to take too long." and

tell somebody, "Let me know when this is ready and I will come back to take this." Okay? That's is the concept of an event loop. An event loop will be like I'm trying to connect or reach out a special server. And maybe the handshake the connection will take a lot of time. What you do, you tell the event loop, "Let's subscribe an event with the kernel that

the kernel tells me when the connection is ready and I will come back here." Now, if you do this inside a separate and also you implement the concept of coroutines. A coroutine is like a lightweight or user space thread, you can do a lot of segmentation in when you're creating code. And this for This is a a common example. What you're seeing here is like a very

high-level overview of an output plugin in Fluent Bit. The first uh the first function that you're seeing then uh FLB upstream connection get, what it does is retrieve a context for a connection to a remote That's it. However, behind the scenes, what happens? Socket creation, connection, DNS resolution, OpenSSL, TLS handshake, and a lot of things. So, there are two ways here. We just can in a normal

way, just try to connect and sit and wait. Or we can do it in an asynchronous way with the help of an event loop. Where this context is running inside a coroutine. And the good thing about the coroutine when the user calls the blue line, the big blue line, what we'll do is to just suspend execution of that function. Suspend ex- ex- ex- execution, sorry. However, the

kernel is already creating the sockets, and there are many events being registered and being tried to be created. And you're always coming back to event loop. And the event loop what we'll do later is to wake up or resume your coroutine, meaning your function. So, for you, the code will be like sequential. Line one, line two, line three, all the lines. However, behind the scenes, there's a

lot of pausing uh connections and things happening behind the scenes. And if you mix all these concepts with multithreading, with coroutines, with event loops, you can move the data at a very, very high scale. Of course, uh there are different type of use cases, and this concept is really good to understand when you're troubleshooting, if you're are in observability space or you need to understand uh why

this is not working, why the data is not here, it's good to start thinking, okay, what are the components of the system, what is the architecture of this product? You don't need to dig into the code, but at least you can get a a high level from the documentation or with the AI about how this is working and what's happening So, telemetry system are about data movement

and data movement defines performance, buffering, defines stability and concurrency, also escalability. Uh for those who are not familiar with the project, I'm going to jump into other slides that we have here. So, this is a Fluentd project and most of the architecture that I was just describing has been implemented over 10 years. Of course, everything everything tried to evolve from one version to the other. It's like,

I don't know, how old are you? And 45? But I started using Apache V1. I don't know if you remember Apache V1, the Apache web server? And my fellow 40s, old fellows 40s? Thank you. So, Apache V1 has a has a mechanism, used to have a mechanism where if you get an HTTP request, do you remember what Apache was doing? Creating a single process per connection. So,

it does a fork. And that's a really expensive um task, even in real life, right? Fork a child. And the thing is that when you have Apache and you have 10 what are you creating? 10 processes. Okay? What happened with Apache V2? What they implemented? Sorry? Oh, they copy Nginx. Nginx. No, I actually you what what was the story because it was Apache 1 Apache V2 started

with threads and that thread well they started creating a thread per connection and then Nginx was born right Nginx started with the same mechanism of event loop asynchronous IO. So you know who has the people in the world around events and asynchronous events? No? Gamers. Every single game that has a gaming emulator right has a very good event loop and when they program the code there's a

ton of things that they need to synchronize. So everything that we're doing here with data movement is not new. Right? But I'm trying to give you the the insight of this. So the project start to move from Apache from processes to threads and now asynchronous sockets same as Nginx does but of course Nginx too over the world. And we do follow a similar architecture in terms when

processing data and moving the data around. Of course there are many many more components and just wanted to give you a high level overview. The project that is based on this architecture is followed by three main principles for the last 10 years and it's be high performance low resource usage have a broad ecosystem support and be vendor neutral which is really really important. So I hope that

this has been helpful for and I was a different type of presentation try had to bring more technical topics to to the table. So thanks so much if you have questions here's the microphone. Thanks. Well that's a bad slide because you know Have a safe flight for everybody who who have to leave. Okay, thank you so much.