How Telemetry Data Moves: Lessons From Building a High-Performance Open Source Agent - Eduardo Silva

This talk, presented by Eduardo Silva from Chronosphere, delves into the intricacies of how telemetry data moves from the kernel to user space, addressing the mechanics involved in data processing at scale. The speaker draws from his experience with Fluent Bit to illustrate how an open source agent can handle billions of events per minute using techniques such as custom user-space serialization, adaptive buffering, and multithreaded I/O orchestration. He discusses how these design decisions relate to Linux primitives, including epoll, async I/O, and zero-copy strategies, ensuring predictable CPU and memory usage. This session offers valuable insights for developers focused on building high-throughput agents, collectors, and streaming engines.