We Deleted Our Observability Stack and Rebuilt It With OTEL: 12 Engine... Yash Sharma & Kunju Perath

This talk features Yash Sharma and Kunju Perath from DigitalOcean as they explore the re-architecting of their observability stack after reaching a critical limit with 5,000 clusters. In response to losing vital audit logs and failing metrics scraping, they adopted OTEL standards and created custom lightweight collectors, enabling them to manage over 20,000 production clusters with zero log loss and complete metrics coverage. The speakers discuss their experience with the upstream OTEL Operator, share insights on their initial missteps that led to overwhelming storage demands, and highlight how they've streamlined operations, reducing their engineering team from 12 to just 4 for managing observability at scale.