Rasmus Selsmark: Processing Millions of Events per Second with Apache Kafka on Kubernetes

In this talk, Rasmus Selsmark from the Data Platform team at Unity discusses their experience in processing millions of events from games using Apache Kafka on Kubernetes. He covers the deployment of multi-zone Kafka clusters with the Strimzi operator, maintains observability with Prometheus, and measures end-to-end latency using KMinion. The speaker also explores using Cruise Control for load balancing, reducing local disk usage with Tiered Storage, and the self-service creation of data pipelines, clusters, and topics, including managing breaking changes and topic versioning. This session provides valuable insights into running large-scale Apache Kafka infrastructure efficiently.