Preventing S3 Cost Storms: Applying Cortex’s Efficiency Lessons to I/O-Heav... A. Fishman-Lichterman

This talk discusses the principles of cloud native design and its importance in achieving efficiency in cloud environments. The speaker, Ariel, who is a cloud researcher and contributes to the Cloud Efficiency Hub, illustrates how traditional storage approaches, such as those used in Apache Spark, fail to leverage the advantages of cloud object storage like Amazon S3. Ariel explains that cloud systems are billed based on usage, which necessitates a different architectural approach. He highlights the development of the Spark magic committer, which optimizes state tracking without excessive calls to S3, and contrasts this with Cortex, a scalable multi-tenant backend for Prometheus designed from the ground up for cloud compatibility. Through examples, the talk emphasizes the need for software systems to understand cloud-specific cost structures and coordination to enhance overall efficiency.

We are all here to discuss the cloud native concepts. We know that being cloud native isn't really about where your bytes live. It's about how deeply your design understands the rules of the cloud. When you take all the concepts and run them cloud infrastructure without that understanding, inefficiencies arise. Hi, I'm Ariel. I'm a cloud researcher at point five and a contributor at the cloud efficiency hub. My

day job is to figure out what efficiency really means in the cloud and how engineers can design and operate cloud native efficient systems. Let's talk about S3. With traditional storage, you provision the whole thing, the hardware, the network, the file system. Since it's yours, you don't need to pay an additional fee for the IO operations. Cloud object storage platforms like S3 flips that model entirely. Not only

the bytes stored are billed per consumption, every list, every head, every get is metered separately. Systems that were designed around the old provisioning model bring their operational habits with them and it affects their overall efficiency both in performance and in cost. We'll cover two examples of frameworks, Apache Spark that had to shift its assumptions and CNCF Cortex which shows an ideal model for cloud native storage interactions

design. Let's start with Spark. It's one of the most mature data frameworks we have. For a long time, Spark used S3 itself to track job state when committing changes and the way you track state on a file system is by listing what's there. So, at commit time, Spark would list recursively across the entire output tree to discover what each task had written before it could finalize anything.

Remember, Spark is designed to work with immutable files. Every write creates new files rather than edit editing existing ones. So, one Spark table can easily consist of thousands of small files and the list goes grow with them and the cloud bill grows accordingly. Spark magic committer was developed to solve it. What it did was stop asking S3 what exists. The driver keeps a manifest on what was

written, so commit time requires no discovery at all. The list don't disappear because it is now served by something designed to answer it efficiently. The reason I find this story useful is that Spark wasn't poorly engineered. It was engineered for a different era of storage. Getting to the magic committer took years. Advancements in S3 consistency guarantees and multiple rewrites because you can't just patch your way out

of a foundational assumption. Which brings me to Cortex, the CNCF time series database for Prometheus. Cortex is interesting for the obvious reason. Prometheus is the de facto standard for metrics collection in cloud native environments. It was designed to work with local storage on a single machine. It doesn't scale horizontally and has no long-term retention strategy. Cortex was built to solve exactly that. A scalable, distributed, multi-tenant Prometheus

backend with object storage like S3 as the durable store underneath. It was designed from scratch for the cloud. There are two design choices in Cortex that I think are worth taking away and they're expressions of the same discipline. Cortex doesn't just put an abstraction layer over the object store. It puts a firm, opinionated grip around it. Interacting with the API service directly is a slippery slope. To

make a data system consistent, available, and performant, you need it to answer state questions very often. Each question seems reasonable in isolation, but the compound effect is tons of API calls. Cortex wraps that sharp edged surface so tightly that the application never touches it bare bare handed. The first line of defense is the ingester and don't let the name fool you. It sounds like a simple funnel

that receives data and passes it along, but the ingester is actually a full in-memory database, distributed, sharded, and decentralized. It receives samples, holds them in a local TSDB, and serves queries against them until it flashes to the object store. Without this, every sample would be a put to S3. The API costs alone would be enormous. The second is the compactor and the bucket index. In most databases,

compaction is a background optimization that consolidates multiple small files. But in Cortex, the compactor is load bearing. As part of its scan, it writes a manifest, the bucket index, that tells every other component exactly what blocks exist. Discovery and consolidation are a single operation. The compactor maintains a clean inventory, so the downstream layers never need to fire a single list. The pattern underneath all of this is

what eventually made it into the magic committer. Stop treating the object store as a database you read and write to in a free form and own the metadata coordination and routing in your application layer. So, what do you actually do with this? For every data system you own, ask yourself what it costs per gigabyte, total S3 spend divided by what you're actually storing. It's the most honest

measure of scalability you have and you don't need a FinOps platform to get there. It's a simple query on your billing data. If that number is is 5x or 10x the raw storage rate, there's almost certainly a room to optimize. The second thing is use these patterns as a as lens. The specific techniques are not the issue. They're all responses to the same underlying constraint, which is

that the object store charges for coordination. Every time your system reaches out to discover a state it could have tracked or scatters writes it could have batched or refetches metadata it could have cached, that's the bill accumulating. Once you see that constraint the constraints clearly, the right design choices tend to follow naturally. Everything we looked at today comes back at the same thing. Understanding the environment deeply

enough that the design reflects it, the cost model, the consistency trade-offs, the API surface. With that understanding, you have the right foundation for systematic cloud native efficiency.