How Many Spark Applications Can Your Etcd Really Handle? - João Soares & João Azevedo, Feedzai

This talk examines the effective integration of Apache Spark on Kubernetes, focusing on the development of a self-service Spark platform. The speakers, João Soares and João Azevedo from Feedzai, discuss their journey transitioning from legacy YARN clusters and the challenges faced, particularly with managing Spark custom resources that can lead to instability in etcd under high workloads. They highlight the significance of open-source tools like Volcano and the Spark Operator from Kubeflow, as well as propose innovative solutions, including the Kubernetes API Aggregation Layer, to enhance storage management in large-scale Spark deployments.