Preventing S3 Cost Storms: Applying Cortex’s Efficiency Lessons to I/O-Heav... A. Fishman-Lichterman

In this session, Ariel Fishman-Lichterman from PointFive discusses strategies to prevent excessive costs associated with S3 storage, using Cortex’s architecture as a guiding framework. The talk emphasizes that treating S3 buckets as if they were traditional filesystems can lead to significant cost overruns due to increased API calls, which can multiply expenses dramatically. The speaker explores common pitfalls, including naive operations like metadata fetches and unbatched writes, that can turn into cost disasters in systems such as Spark and Airflow. Attendees will learn from Cortex’s design choices, like bucket indexes and batched writes, which provide effective solutions to mitigate these challenges in I/O-heavy systems.