Moving Spotify’s Infrastructure Management Up the Stack from Kubebu... Alexander Buck & Tomas Aschan

This talk discusses Spotify's approach to improving infrastructure management through a platform called declarative infra. The speakers explain how this platform enables feature teams to manage their resources using code stored in resource repositories, streamlining the infrastructure provisioning process. They explore challenges such as data duplication in their systems and introduce solutions like Apache Iceberg and BigLake to enhance data handling between Google Cloud Storage and BigQuery. The presentation also highlights the development of new tools like Kube Resource Orchestrator and Kpop, designed to simplify the creation and management of Kubernetes resources for developers. By packaging complex configurations into user-friendly abstractions, Spotify aims to empower both feature and platform teams without overwhelming them with technical complexities.

Hello everybody and welcome to this talk about how Spotify is moving our infrastructure management up the stack. I'm Thomas. And I'm Alex. And we're both very happy that you are taking the time to sit down here with us instead of getting in line early for beers after. Uh we work in a team at Spotify who are building our internal resource management platform, which we call declarative infra.

And building a resource management platform means we provide tooling for feature teams, that is the teams who actually build the Spotify product, to configure their infrastructure in code in their resource repositories, and have that configuration turned into actual cloud resources. But at Spotify, we also have a lot of teams who build our internal developer platform. And some of them build bespoke products that don't have an off-the-shelf

thing that we can just use. And others take care of the usage of our off-the-shelf products across the organization. So, for example, developing best practices for data storage or running cost optimization programs and so on. Our team serves both these personas, the feature teams and the platform teams. And we want platform teams to be able to expose their products to the feature teams so that they can

use a single configuration surface for all of their infrastructure needs, whether it's an whether it's an off-the-shelf product or an internal thing. In other words, we want to provide feature teams with less complex and more powerful abstractions and allow platform teams to evolve their products independently. We are picking up where our colleagues Oliver, who's sitting right here, and Fredrik, who's on parental leave, left off at KubeCon

North America in the fall. Um where they shared our ambition to analyze strategy to manage 1 million resources internally. And we just recently reached that milestone, by the way. Um, show of hands, how many of you have seen that talk either live at KubeCon or on YouTube after? Like three, okay. There is a link here, so you can go watch it, but please don't follow that until

after this talk. Um, in their talk, they recapped the history of infrastructure management at and then highlighted a few problems we were working on solving. And today we're going to zoom in on one of those, namely how do we empower platform teams at Spotify to build powerful, batteries included, infrastructure products and then put them in the hands of feature in ways that let the platform teams focus

on their products rather than on the infrastructure platform itself. We will look at this through the lens of a recently launched internal product. But before we get into what they built, let's have a look at why they built it. Spotify has a lot of data. Every time a user listens to something on Spotify, an event is sent from the client to our data platform. With 750 million

monthly active users, you can imagine how many of these we get. The events are stored and subsequently processed in various data pipelines to power data-driven features such as Discover Weekly and DJ, as well as other business-critical functions like royalty reporting. For batch processing jobs, GCS is a great fit, but it's also useful to be able to run SQL over that data for ad hoc queries and dashboarding

use cases, in which case BigQuery is better a better fit. To enable these use cases, we typically have jobs that copy the data from GCS to BigQuery, but that leads to significant duplication. And it turns out we have quite a lot of 28% of the data that we have in BigQuery is duplicated from GCS. How can we reduce this duplication, but also enable supporting both batch processing

and SQL based use cases? Well, fortunately, there are several products that can help us with this. Apache Iceberg is an open table specification allowing an abstraction between the underlying storage and the processing engine, in our case BigQuery. And Big BigLake, a Google Cloud product, allows interacting allows BigQuery to query over those Iceberg tables. And what that means is that data producer jobs can write to GCS as

normal, and BigQuery can query that data through Iceberg and um BigLake. Solving the duplication problem. So, with that, that's great. How do we set this up? Let's use declarative The feature developer needs to define a handful of resources with tens to hundreds of lines of configuration. Scattered throughout these YAML configurations are properties that need to match up in various places for the system to work. This is

hard to get right, no matter if it's a human authoring the configuration or an AI agent. Not this with AI agents, right? Not only this, but uh we also need an API call to our internally hosted Iceberg catalog to register the Iceberg table. This is not a nice experience for our feature developer. So, the platform team came to ask and asked how we can help them package

all this in a nicer way. And hey, we've actually seen this exact thing before. It's like what we did for Cloud Bigtable Um Bigtable is a Google Cloud product for low-latency, scalable key-value storage that we use a lot for our back-end services at Spotify. So, we have hundreds, maybe thousands of services that use Bigtable. defining a Bigtable instance on the cloud of infra is pretty easy. But,

then it turns out you also need a Bigtable app profile for each region where your service is deployed. And you need IM configuration, and you need auto scalers also per region, and you need a backup schedule, and all in all, this comes out to hundreds of lines of YAML. And lots of places that names and IDs need to line up for things to work. And as our

internal best practices evolve, this means that the configuration needs to change also across all of these services. This situation looks a bit familiar, So, the way that our internal team solved that was to build custom operators for the things that needed internal API calls, like the database auto scaler and the backup schedule. And then wrap it all in a composition operator that they called Spotify This exposed

only the set of features that they wanted feature teams to care about in their storage, and abstracted away everything else as platform concerns. And this turned out to be hugely successful. For example, when they built this, we were using an internally built database auto scaler because the built-in one in Bigtable didn't have all the features we needed But, a while after rolling out Spotify Bigtable, Google actually

fixed their or like evolved their uh auto scaler so that we could use it, and we no longer needed our own. The platform team could then migrate our entire fleet to Google's auto scaler, deprecate the internal one and tear it down, and the feature teams didn't even notice. It just worked. We can just do that again, right? it wasn't all good. Feature teams loved it, but for

our platform developer, they had a lot of work and a lot of learning to do to implement it. They were specialists in storage and used to writing in Java, but now they had to learn Go, Q builder, on trend, understand the interaction with controller runtime, learn how to build status conditions that conform to K status, and in general, build a lot of familiarity with Kubernetes specific concepts

that were very outside their normal domain. In short, the learning curve was steep, and the support that we as platform maintainers were able to provide was limited in terms of giving guidance and education. They had to do the work. It ended up taking two full-time employees eight months to get this into production, and while it's widely used in the organization today, the owning team are no longer

comfortable making changes to it. Can we do better than that? I think we can, yeah. But before we look at how, let's take a step back and try to generalize this problem a little bit. So, there are basically two main types of resources in play here. One of them is abstractions and compositions where you have a lot of resources already available on the platform that you want

to package up in a nicer interface for the user. Designing the schema specifically for your use case and for the best practices you want to encode, so that only the levers that feature teams should care about are available to them. The other type are resources which expo expose cloud through imperative API calls. Those APIs could be bespoke to our platform. We have lots of those, but they

could also be third-party products that we just run internally or even use as a service. And for some of those, for instance, for most Google Cloud products, there are also off-the-shelf operators that we can But for some, we need to build those operators that turn the imperative API calls into declarative resources. Both the Bigtable use case and the Iceberg data deduplication case are examples of how we

need both of this, some imperative stuff stuff that needs to be wrapped and then we need to wrap it all up and package it into a coherent product. And while Spotify Bigtable showed that this is very valuable for the feature teams, it also showed that our developer experience that we provided to the platform teams was not good enough. We need to do better. For abstractions and compositions,

we've rolled out Kube Resource Orchestrator, an open-source project under the Kubernetes umbrella that Jesse shared a bit about this morning in the keynote. It reduces the development surface to writing a resource graph definition, a Kubernetes resource, where the platform developer can focus on how to package to the product, leaving to Kube how to create the underlying resources and the dependencies between them. And in the in addition,

leaving the operational concerns to us as platform maintainers with the operational concerns of running Kube. You can meet read more about Kube on kube.run. And there's a talk on Thursday at 11:00 called Stop Flying Blind. We'll be there for sure. Now, compositions are a powerful mechanism for abstraction, but they depend on the availability of the underlying resources. How do we support the development of these underlying resources

outside of KCC? Well, that's where Kpop comes in. Kubernetes Protobuf operator. Kpop enables platform developers to build operators by implementing a GRPC And Protobuf, they express types that are translated to a generated CRD. They implement a reconcile function in their GRPC service that performs the reconciliation logic. It takes as input the resource and the response is the status. Users can iterate on that operator with a very

tight inner loop by developing against the GRPC service. And it turns out with this pattern, Kubernetes is one of the last things that you users need to think about as they can develop without it to begin with. When they want to deploy it, the Kpop controller will call the GRPC service, take the status, and update the resource in cluster accordingly. Users can also implement GRPC functions for

validate and delete, which corresponds to a validating webhook and finalizer managed by Kpop. And in general, it allows platform teams to focus on their business logic, leaving as many concerns as possible to us as the as the platform uh maintainers and the runtime concerns as well. But, talk is cheap. Let's see Kpop in action. And this is a very live, so pray for me. here we have

an example of a service proto that we can use to generate a Kpop It's a very simple hello world service. It just takes uh greeting thing with the language and whatever and returns a greeting. the handler for this is also very standard. Notice that there's nothing here that knows about Kubernetes. It's just a simple request response uh There's no controller runtime. There's no client go. There's nothing

about any of that stuff. Um you do have to conform to the sort of conventional names of things, but that's about it. And in this handler we have all of the logic needed to return a greeting. So, with this we can define a resource like this and have it reconcile. So, let's look at what that looks like in practice. If we do go run main we have

this listening on localhost. And then we can use the Kpop reconcile CLI function to test this in practice. So, we point this to a file and to our service. And this runs the reconcile function. The function now logged the response it sends just so that we can see what it does. And then the Kpop CLI will also output the resource as Kpop the controller running in the

cluster would store it. So, you can see how the status that you have to return as uh developer is translated into the conventions expected by the Kubernetes ecosystem. But those conventions you don't have to know about. You just give us the enum for is it okay or is it failing or is it in progress and we figure out the rest. Then when you're ready to deploy this

after having iterated and and deciding that your service is doing what you want, when you're ready to deploy this, you just add one of these to the cluster. And this is something that you probably give to us the platform maintainers and you point to your Um and then we will do schema discovery on the like on the GRPC endpoint for that and figure out what your CRD

is like and so on. Uh but of course you might want to iterate on what the CRD is like as well. So you can do Kpop generate CRD and point to your profile. Let's see, where is that? Proto greeting. And then you get the CRD here. So you can locally also iterate on what the CRD will actually look like even though you don't have to write it

and you don't have to care because we generate it for you in production. So let's apply this. Uh samples operator Cool. It is ready. We have our greeting CRD. So let's try to apply that. It worked. Thank you. So um teams who build operators don't actually have to do any operator deployment. They just build their service and we take care of the rest. That's all you need.

Back to our case study. The data platform team was able to build a declarative abstraction for the out-of-band API call to the Iceberg catalog API to register the table. And with that and all the necessary resources available, it was just a matter of being able to package them into an easy-to-use all-things-included product that feature teams could then set up themselves. Using Krate, that meant defining an RGD

that composes the Kpop resource and the required GCP resources with KCC. this meant that the entire complexity of this use case is tucked under a single resource Spotify Iceberg table. And it the platform team is able to accomplish this without writing any lines of Kubernetes operator code. Kpop and Krate, we think are a very powerful combination. We we think it can solve over 90% of the platform

developer use cases within And that's not the end of it. Over the years, it has often been a bit of an uphill battle interacting with platform teams trying to have buy-in for them to extend upon the declarative infra since we've launched Kpop internally, we've seen a massive shift with teams reaching out to us expressing interest to expose their products on the platform with the much uh easier

learning curve. And we think we're on the start of a journey for declarative infra to become a single infrastructure needs at Spotify. oh, sorry. Now that we have Crow and Kpop running in production, making both platform teams and the feature teams happy, where do we go from here? Well, to start with, we will still have a lot of so-called day two problems to solve. Um, the scale

at which we're running this today is nothing compared to what what our ambition is. that will surely bring a bunch of challenges in terms of observability, um, fault tolerance, and so on. We also need to figure out how to do things like versioning both of our own APIs in Kpop to the platform teams, but also how to help them do versioning of their resources that they present

to feature teams. We have a bunch of ideas, but we haven't actually implemented any of that yet. And if this sounds like the first draft of a CFP submission for KubeCon North America, well, let's just say that we've enjoyed talking to you today, and we'd love to do it again. Thank you. Thank you. I think we have quite a bit of time for questions, which is great.

There's a mic up here that you can ask your question in if you want to be heard. Otherwise, we'll repeat it. But before we do that, I have a favor to ask. Today is my son's fourth birthday, and I would love for us to record a uh happy birthday greeting to him before we let this off. So, if you can just say with me, in Swedish, "Grattis

på födelsedagen, Felix." Let's practice that, and then I'll record us all and do Can you say with me? "Grattis på födelsedagen, Felix." That's going to be great. Let me just get my camera in order. He was a little bit disappointed when I left our home, so maybe this can compensate. You want to be in it, Alex? Okay, everybody ready? 3 2 1 Thank you so much. Thank

you. Now, questions. I see a bunch of people coming up here already. That's perfect. Uh thank you. The excited about Kpop. looks like it'd be really helpful for my platform team. Um one very nice thing about Kubernetes APIs is they're really easy to decompose. So, you have a parent resource that creates child resources, and often those child resources do a lot of the work. In Kpop, if

I'm just exposing a GRPC service, do I have a way to decompose what is effectively my operator? Right. Uh I Yeah, go ahead. Kpop is designed for very low-level resource kinds. It's not optimized for when you have uh a composition of multiple resources. So, that's where we think Krew is a great combination. So, we think Kpop for low-level, and then bring it all together with Krew. This

mic's very tall. Um okay. >> I think you can just remove the mic from the stand, too, if you want to hold it. There you go. Thank you. Um great talk. Uh this is less about your talk. I um I was the AWS guy at Heroku, and I was uh first guy to kind of inventory all of our AWS assets and wrangled our CloudTrail data, VPC flow

logs, and all that. So, lots of big data, and the issue with doing that is nobody wants to be the AWS guy. So, I found myself in a situation where I had things that I wanted to talk about, but I needed somebody who knew more about this stuff than me. So, I was very excited to see your talk and I'm going to go watch your talk from

last year. And I was wondering if like, you know, anybody in the audience and yourself like can I send you a LinkedIn connect and just ask some questions? I'd love to talk about maybe like Of course. >> in the DMs. I wasn't sure if Salesforce wants me to share all that in public, but um yeah, it's like just been looking for people who ran into similar problems

of like wrangling infinity basically, cuz when you have that much data, it's really infinity, but sometimes platform engineering doesn't really give you all the tools for that. So, setting up data engineering myself and just dealing with all the teams, Salesforce teams, Heroku teams. Sounds like you had similar challenges and I'd love to kind of learn more. Yeah. >> Let's talk offline. Yeah. Please. Let me help. >>

Here you go. No worries. Thank you. So, uh how would you compare to Helm? Hm. I think um Helm does Helm has a lot of opinions on how you do templating and how you package stuff and how that is rolled out. And it does a lot more than what we're doing here. it also, I think Helm used to do stuff server-side before, but now it's mostly a

client-side thing and then you have different tools that take the Helm chart and actually apply it to the cluster. All of this happens in the cluster. So, you have a crow resource graph definition. It does just a CRD. The RGD is itself is a CRD. So, it's a CR. The when you create your your resource graph definition, it's a custom resource that defines other custom Uh but

all of that happens in the cluster. So, you do that once for the resource type. Uh and then expansion happens in the cluster when someone creates an instance of that new resource that you defined. Um with Helm, you do a lot more work outside of the cluster and then have that applied by Helm or by Argo or by some other GitOps tool, I guess. Um so, I

think that they solve the problems they solve aren't really the same even though Helm also can be used to compose stuff into products, sure. But I think Helm does a lot more that we're not interested in Yeah. There's There's also things that we really like in Crow in terms of being able to report status back to the higher-level object and things like this can make it really

powerful for compositions, I think. Yeah, exactly. All right. Thanks for the talk. I have two questions. One on Crow and the other one on Kpop. What are your thoughts about versioning of CR uh of RGDs in uh Crow? And then yeah, I can repeat the Kpop question when the person Uh yes. So, Thomas briefly mentioned about versioning of CRDs as a day two problem. We We haven't

got there yet. So, we have We haven't got the experience with that yet. But we're very curious as well about that space. Okay. And then um how do you handle auth with Kpop? So, the services must authenticate against the GCP API uh somehow. How do you handle that? In the simple sample that I had here, I very simplified version of the Kpop operator resource. That's actually powered

by Crow. So, in our production environment, we have a resource graph definition which is the Kpop operator where you create one of those and then we create a bunch of things including uh Rback and service accounts and so on. For each instance of the Cape operator, we create a new controller deployment. So, there's like an a new controller for each type of resource that is used that's

that is using Cape operator. And those can of course use different identities that get different access in various systems. So, depending on how you set up access to services in the rest of your platform, you will have to adjust, but the way we did this in our system was just using Crow to expand into the resources we need. And then obviously you can tailor that to your

needs. Hello. Yeah. Hi. Yeah. Well, we've just gotten down a very similar journey to you where we've started implementing Crow as the way of sharing services with various dev teams. And the one thing we've been struggling with is Crow is great for infrastructure, but then when we got to the service specific APIs, it's a nightmare. And all our services are GRPC with Protobuf. And I was thinking,

can we have Cape operator? Like is there any plans to open source that cuz it sounds like it solves our problem perfectly. Mhm. There aren't any official plans to open source it, but we personally would love to do that. We are still doing a lot of internal development of this and we have to I guess figure out how far we want to take that before we can

start thinking properly about open sourcing it, but yes, it's definitely something we are thinking about. If and when and how to do. Watch this space. Hi. Thank you for the talk. Do you think there is some cases where you expect Crow to fall short like the logic of resource group graph definitions are simple and you're when you compare them with like Crossplane and other other tools, do

you Do you there are cases where you will need to still go back to build an operator or do you think I mean, there's definitely cases that might be complex enough that you need to go back to controller runtime or something. Um or whatever framework you want to use. We haven't yet found one. In in our stock, like when we're looking at what people are doing, what

platform teams are doing on our platform, we haven't found something that cannot be solved by Crow and Kpop together. That doesn't mean it doesn't exist. It just means that it's going to be rare enough that when that happens, we can probably just fall back to the low-level implementation and it's going to be fine. The main thing that this buys us, I think, is that for the vast

majority of use cases, the developer experience and the need to know about how Kubernetes works is basically reduced to zero for these And then, of course, if there are teams that want to do specific enough things that this is not good enough, then they probably also know enough about Kubernetes already that building something with controller runtime isn't a big hindrance to them. And if I can add

another one, like just a suggestion, I would be very interested to see how do you manage the uh how do you enable platform, like people creating these abstractions, to roll them out across the fleet? Uh maybe for our next talk. Do you mean how managed to get the feature teams to use them or No, no. How do How do you uh provision the abstractions in the clusters?

All right. It would be also interesting. And maybe I would you bake bake security and governance requirements into them. Of course. Yes, so with both Crow and Kpop, yeah, the Kpop operator resource we have managed in the a repo that syncs to all of our clusters all of the infrastructure resources. And so Kratos the same. Uh the resource graph definitions are in this central config repo. And

what we would love to do in future is to be able to offload the management of the resource graph definitions to be managed by the platform teams directly. Currently we're not in a place where we can do that because Kratos doesn't have isolation of permissions. So, if an RGD has permissions to do something, we don't really want to grant access to another one to do the same

thing. So, the way we get around that at the moment is to gate changes to the RGDs at the moment. But in future we'd love to be able to have isolation to be able to offload the management of those entirely. And this is something that we're thinking about how to contribute back to the Kratos project as well, cuz we know that there are others who are interested

in similar features. So, it's likely that we will try to at least push for it to happen and hopefully contribute to it to happen. Oh, yeah. Hi. Hi everyone. Thanks for the presentation. It was really cool. Uh I've got a question. I'm also part of the platform engineering team and we need to provide uh reusable features in a self-service manner. And we've been uh evaluating something very

similar to what you you demoed, but we decided to go with Crossplane for example. And uh I didn't know about Kratos at all. And I think it's really cool the Kpop uh thing that probably breach some of the the I would say missing features of just Kratos. Have you come across uh Crossplane? And what's your feeling about it? Would it solve the 90% of your problem for

example, if not more? We did evaluate Crossplane was it two three years ago now? Um and then mainly whether we could replace our usage of Kubernetes config connector, which we use for GCP products, with Crossplane, cloud agnostic. Um at that point the main features we were looking for were not there. Uh and then we haven't actually carefully reevaluated that since. So, I can't tell for sure whether

Crossplane has all of the features we need now. Um but at that point it was better for us to stick with the config connector for the GCP stuff and then find other solutions for the other stuff uh basically. Right. Cool. Thank you. Uh really nice talk. Um how do you develop and iterate and test your RGDs? Uh good question. Actually, that was a massive pain point for

the data platform team we talked about, and they were our guinea pigs. They ended up building this massive end-to-end testing framework basically to support it. and I think there's an opportunity for build that as a capability so that for the next thing it's a bit easier. So, and I think another interesting thing is that in Kpop the selling point is they don't have to use Kubernetes at

all. It's one of the last things needing to be done. And then with Krew it's from the opposite end basically. It's very Kubernetes centric. So, there's very different development experiences for these two different abstract uh these different two different uh platform extension patterns. Uh so, yeah. I think um it'll I I think it'd be great for um to to think about how we can improve the Krew

testing support. Yeah. Okay. One thing I will say for Krew though is that it's very lightweight. Like in the demo now I had a very a local kind cluster with the Krew controller running. and the only thing you need for that to to sort of test your RGD and how it expands and and and reconciles and so on is to have the CRDs of the other resources

that you want to expand into available on the same cluster. Which is also a lot easier to get working than having all of those controllers available as well. Because you don't for Crow to work, you don't actually have to have something that reconciles and and like does work on the things you expand into. You just need to make sure that it happens. Uh so for that testing,

setting up a test environment is maybe less of pain than it could be. Thank you. Okay. Uh one last um what do you think about uh the possibility of Crow moving upward to Kubernetes itself? >> Mhm. I know that's an ambition from the maintainers of the Crow project. So yeah, I hope that I think that's a great ambition as well. I'm getting Leo here is telling me

it's beer time. Uh I assume that's why people are filing out as well, but we'll be here if you want to have uh ask us more questions and otherwise find us where there's beer. >> Thank you very much.