API-Driven Infrastructure as Code: Kubernetes APIs as the Con... Florian Hopfensperger & Yury Tsarev

This talk focuses on the implementation of API-driven infrastructure as code using Kubernetes APIs as a communication bridge between teams. The speaker, Florian Hopfensperger, highlights the development of a platform that enables product teams to offer automation products to application teams at Allianz Technology. They demonstrate how the use of Crossplane allows for a unified interface across multiple product teams, facilitating self-service sign-ups for various cloud infrastructure claims. The session details the architecture involved, emphasizing the significance of clear API contracts to manage dependencies and avoid breaking changes. Furthermore, the speaker discusses the importance of contract testing to ensure seamless interactions among various product teams and presents a demo to illustrate how these concepts are applied in practice.

Hello and welcome to our talk, API driven infrastructure as code, Kubernetes APIs as a contact bridge between teams. Now, I know a very long title, but we get into it. My name is Florian Hopfensperger. I work for Allianz Technology as a lead engineer, and I'm happy to share the stage with my dear friend Hi everybody, I'm Yuri from Upbound, and we are creators of Crossplane. So, before

we get started, we have a tradition to make a silly selfie. Silly selfie. Uh nice but silly, silly but nice. The guys are in the audience. So, if you give us yeah, some sign of excitement, that would be great. Yeah, yeah, yeah. Thank you so much. I appreciate it. Um just just an side info, this talk has a reference to a talk which we did last year

in India. So, if you if you want to deep a little bit more dive into the platf into the platform of what we built, check the talk. And at that talk, we showed what the platform is doing. Today, we want to to show you how actually teams are working on this platform. So, a quick recap before we go into. So, I always like to compare our platform

as eBay, okay? So, eBay has on the one hand side some some people who want to sell their products, right? So, we see on the left side, this is in our case the so-called product team. They want to sell their automation products. And on the other side, we have the application team. So, the actual customer who wants to to buy these products, right? And we are in

the middle of this of this circle. We are um part of the the platform team. We operate and develop the platform. So, so clear. And on the left side, the product team, we have not only one product team, we have multiple product teams, obviously, right? And we are only serving internal customers. so we are not exposed to the world wide, only internal. Just let me show you

some result what we achieved the last 4 years since we started building up the platform. So, you can imagine in an in an enterprise company like Allianz, everybody has like their own way to to sign you up, right? So, what we enabled now across all these products, we enabled self self-service sign up and therefore we have around thousand plus control planes which are used by the customers.

So, they are using then the Kubernetes API to to manage their infrastructure. So, and this comes to the second point we offer now a unified interface. Previously, you can imagine when you have 15 20 product teams, everybody has their own interface. For one you need to create a GitHub issue for the other one you need to open a Jira ticket, the other one get emailed teams, right?

Now we have this where we use like um Kubernetes and in the GitHub the customers manage their automation and then Argo CD synchronizes it to the control plane itself. So, we're talking about 20,000 plus infrastructure claims here, which is I guess not a not a so so small number. As I mentioned, 90 plus products managed by around 15 teams. And where we are actually using Crossplane for,

right? So, as you can imagine, all these things is powered by Crossplane and here we are having the Crossplane providers which are really helpful for us to talk to our enterprise APIs. Because you can imagine, we cannot simply go to enter ID and click on create security group. No, we need to follow a certain process, right? So, let me let me show you just a quick overview

of the architecture what we built. So, I I don't want to go so deep into this because it's not 100% connected to this talk, but but still worth to mention it. So, as I mentioned, we have the app team. Hopefully, you see my mouse pointer. Yes, you see it. Um we have the app team which is uh where where they have the Git repository, and then we

have an Argo CD running on a hub cluster, which is then synchronizing their claim on their control plane. So, each customer more or less has their dedicated control plane where they can then manage their infrastructure. And obviously, there are a lot of other components which uh help us to provide the service. Kverno, um Cube step metrics for collecting metrics, doing some reporting, and other things. And on

the left side, we see the product team and the platform team. And yes, they are also using Git and GitOps approach to manage, so to say, the platform itself. So, if a product team wants to deploy a new product, they open a pull request. The pull request will be then deployed. Crossplane rolls it out, and so on. Okay? So, this is like just a high-level overview. Um

so, now coming to the product team again. So, how the product team is actually developing their product? So, we have here, let's say, this process. Um so, first of all, they define an example. When you want to offer an automation project uh product, you need define this API for your customer, right? How it looks like. And don't undere- underestimate the effort you should put into defining this

API. Make it easy, but make it not too complicated, right? Because this is the interface to your customers. If you make it too complex, nobody's using it, or it they automatically will open a lot of issues. Yeah? So, then based on this on this example, they generate the XRD. They can then go into the um test-driven development flow. They define some tests, define the business logic. For

example, they can write down a cross-plane composition as Python. They write the composition, all the things, and then they run the tests, right? And then they can loop over this as long as they like. And then they create the PR. CI is automatically building and testing things, and then they release a canary. Why a canary? You couldn't imagine when you have like 1,000 customers, 1,000 plus customers,

you maybe don't want to target all customers at once. Then you destroy everything, you delete whole Alliance infrastructure. No, you don't want to do this. You maybe only want to try it out in the real world with five customers. So, you do like a canary deployment for all these guys, huh? Then you get the PR merged, CI is publishing and releasing the package. So clear so far?

Now, the hidden challenge, right? How the product teams are working together. As I mention as I can mention, this building the platform was the easy part because it's now already running, we don't need to spend any effort anymore, but the hard part is how these product teams are collaborating together, right? So, we have an unclear ownership boundaries, so who is responsible for what? So, if you share

maybe the same claim, and everybody has their own dedicated fields, this does not work. You have the so-called super composit compositions. So, think about you have a team which is providing landing zone, and the other one is providing DevSecOps services, yeah? And now, this DevSecOps services are mandatory for this landing zone. Do you now force this team to talk to this team, "Hey, please, in the landing

zone claim, please also include here my my configuration." And you have this this tightly coupled thing, right? Which you want to avoid. And yeah, super compositions are really bad to manage. Then breaking changes impacting customers is obvious, right? And undocumented assumptions. What do I really get when I request a landing zone? Do I just get a VPC? Do I get some subnet? What do I actually get

when I order and manage a VPC and landing zone, so to say. So, API contracts became our answer to to solve exactly this challenge. So, what is an API contract in our platform? So, our API contract looks like this. So, you have a a schema, an XRD, right? Define some fields you can use Kubernetes validation to do certain validation rules and so on. You also documentate the

behavior. So, what you get with a landing zone you can use um you can document it. You have obviously some versioning. Kubernetes versioning is a little bit special as you might already know. So, I will have later a take on this. Um and obviously docs. If you have customers or AI agents which are consuming your your infrastructure, then they should have enough docs to understand what actually

is happening. And obviously some tests because you want to have everything tested, not that you break everything. So, let's see how this looks in practice. So, and now bear with me, I did not bring now an XRD with me where you would normally define all these fields because it's unreadable on the screen. I decided to to bring you an ex- to an example, but it should be

put on the XRD on the custom resource definition for Crossplane, right? So, here we have like a custom environment, dev, test, or prod. Can be an enum, right? We have some cost center which should be a string, not a number, right? Things like that where you can have valid uh values like enums, type validation, and And now, maybe maybe the the more interesting stuff here is status

subscription ID. And this is part of my API contract which I have with other product teams, right? We When the customer is applying this we make sure so that owner of the dedicated subscription is making sure that we that this status has a subscription ID which can be consumed later, right? And you put in some documentation like you can use annotations, you describe it what you actually

get. I mean, this is very easy. You can have whatever you you can put whatever you want here, and you have some dependencies, right? If you have any. Can think about the database team which is dependent on a subscription, then you have to dependency of the subscription, right? And versioning is a little bit hard because, you know, breaking changes, you need to have some conversion webhooks which

are converting it. There are really good already passed Kubernetes talks out there which you can read and get known known to it how versioning in Kubernetes works, but thing is, I would say avoid breaking changes if possible. So, now let's let's let's check how it this looks from the provider side. So, the provider wants to consume another provider. Database team wants to consume the thing from the

landing zone team which is then used by a customer, right? So, the challenge here is obviously um we need to deploy this database in some subscription, right? So, here we have the subscription ID or the subscription ref. The problem now um I don't want to let my customer type in the subscription ID, right? He already created this claim as a metadata.name. I want to force him that

he's putting here the metadata.name of his claim, right? Which is already known before he's creating the subscription. Yeah, so the database subscription needs to be transferred from one XR to XR. Easy, same provider, same crossplane provider, easy, right? Both using provider Azure. Provider Azure has the capabilities to just look this up with the so-called selectors or reference field. Easy, done. Job done. No, not so easy maybe.

Cross-provider. Think about an easy example. This database team is using provider SQL. Provider SQL needs to consume the subscription ID for any reason. You relate to as a demo where he's demonstrating something like this. Here the selectors don't work because the provider SQL has no clue about provider Azure. So, what you do is you use the crossplane feature external resources in version one. Now with version two

it's called the required resources. This is like the importer from everything. So, you can import any XRD Sorry, any composite resource, any managed resource, or any with crossplane version two Kubernetes resource, so to say. You can import it, you can read it out, and use this for your automation, right? And it is very nice testable with crossplane render. So, now how an API contract looks like from

a consumer side. So, we are talking about our actual customer, right? So, the customer wants to select a product. Previously, when he was selecting the the landing zone product, he had no clue what he gets. Now with our API contracts, he at least he understands that he gets he or she gets an a landing zone which consists of the following resources because we put the proper a

proper annotation, right? Then he fills out the claim, he creates a PR, and then again, API contract contract testing is in place. We can do automatic validation, right? We can now already in the CI CI say, "Hey, the fields what you are defining, they don't exist." We can already do this at that point, and we can pro- provide additional services like deletion protection because maybe you guys

know what happens when you change the metadata of name field of an Kubernetes resource. And think about this in an infrastructure world, deleting landing zone, building up landing zone, what will happen, right? So, you can already give this additional functionalities. You get the PR merged, Argo CD synchronize the claim, Crossplane is fulfilling the claim. So, how you test actually this contract, right? So, on the one hand

side is um the provider can test it, right? They and provider don't get confused here. The provider in this term is the product team again, not the Crossplane provider, but is providing services, right? So, they can test for schema compliance, they can check if the resources they are promising are rendering out. They can have some backward compatibility, which means are my examples which I created half a

year ago, 2 years ago, are still working, which I begin to this published. they can theoretically check if the SLAs are working, right? And can do everything with cross plane render, for example, and schema validation, at least some parts of it. And on the other hand, what the consumer can test, you you you're now thinking the customer should not test, but yes, he can also test, right?

He can have some um test name spaces in cross plane version two, or in our case, some test control planes, where they just create resources, they update resources, they delete resources, and see what is happening. And they're asking then the the the the product team, "Hey, hey, you said you promised this and this is working." No, it doesn't. And they can actually have also CI, where they

verify all these things, right? And yeah, you can have um we just have like some related tools, which uh can be used for that. Um yeah. So, I would say we are ready for the demo. Where where Yuri provides us now the database, and I would like to pray already the demo gods, because in the in the keynote we saw demo gods were not with her, right?

All right. What I'm going to demonstrate now is uh hard uh part uh that Florian mentioned. I'll run the demo script. So, the setup. Two teams, uh two providers, and no shared language, right? So, in case of infrastructure team, uh they will use provider Azure, and we'll deploy uh flexible servers, isolated networking, and they will actually use the KCL, but we will play the role of application

teams that wants to deploy database for their own applications, and we will use a Python, and uh we will use uh presumably uh mainly provider SQL, and that's a hard part, right? So, uh there is no direct connection, there is no um native selectors between provider SQL and provider Azure. So, how uh to do to integrate that part. And also, we don't want to build any form

of super composition. We don't want to include uh flexible server SQL instance into our app database abstraction. We want to create this loosely coupled uh deployment still with a strong API contract. So, the idea here is that if you work with Crossplane before, you know that the work around here frequently is to use um connection secret, right? But the connection secret is kind of weak contract because

whenever the that is not usually visible, it's not on the top, it's going to it's going to be renamed by intro team every single break for the consumer. And instead of that, we will use this strong API contract on a SQL instance external level, and we will consume the data from status fields, and we will leave the password only for the for the credentials, right? In a

in a connection secret. So, I'm running it live so I can get quickly CubeControl get composite. So, SQL instance is pre-deployed, associated networking, and app database is already there. So, how the integration works? First thing's first. Oh, that's a that's a bummer. That's a serious bummer. How come? Why you we still see the presentation? It should be mirroring, right? Okay. Yeah. Sorry for that, and I'm running

super quickly team infrastructure, right? Team application, uh a shared contract. So, CubeControl get composite, we have our resources deployed. So, the how this integration works, we have a strong API contract in a status field of SQL instance, the one that we consume. And, uh uh on a consumer side, we have uh just a uh in app database. So, e- from a front-end example, if you look at

the app database XR, uh the only thing that consumer should uh provide is a SQL instance ref. The rest will be figured out by our loosely coupled integration. the main contract fields are status, endpoint, and status port, right? And uh everything will work seamlessly uh without direct composition of a SQL instance within uh app database. So, we have a network. we have a SQL instance. Everything is

pre-deployed by the infra team, and we are we are not exposed to the any KCL code of infra team. It's just provided for us by the associated abstraction. We are operating with our app database uh consumer uh instance, and we going to integrate it this uh via this very constrained API So, what going to happen here? So, uh we already have this SQL instance running, and we

have we we have the status uh uh available in runtime, and port is available, port is available. So, we have enough data to integrate. In case of uh app database, there is like so- so-called three-phase reconciliation. In this case, we are going to use a required resources for Merlin known as extra resources, and read this SQL instance uh from the control plane API. If you look at

this uh at the associated code, it's just uh this extra resources in a Python SDK. It is available in every SDK we support. It's a core Crossplane functionality. So, basically, the idea is that we query uh the SQL instances in the same namespace. It's V2-based uh a reference uh setup, and we are getting uh associated SQL instance and its runtime data without uh strong coupling. So, we

uh we will get uh the infrastructure data like reference it by SQL instance name and if endpoint is available, we are going to create this set of resources, right? So, up database is going to be integrated and create this database within a flexible server that is provisioned by totally different team, totally different abstraction. So, final check we already have a consuming secret that is already created by

the up and we can connect to it end to end and hopefully it will work. Yeah, so we read the data from the connection secret of the up database abstraction and MySQL client just connected it and validated that the user has associated grant. So, everything is properly integrated. This is kind of end-to-end test just for the demo as you imagine. Thanks to the strong API contracts, we

can test a lot of this stuff in isolation in so-called composition tests. So, it's kind of unit tests for but for composition logic. a bound composition test pipeline and you can use any open source framework for for that stuff that Florian mentioned like XBehave, SpecFlow, Cucumber frameworks. They're all good stuff. The idea here is that you can create the offline assertions and something is breaking in the

demo, but let's see how it goes. The main idea is that you can avoid running end-to-end test every time most of the stuff locally thanks to the rendering engine that if you most probably use Crossplane render before, we can render the desired stream and create the assertions. we can quickly look at the associated test suite. And the main the main core of it, so we can assert

the desired stream and expect the set of resources. Something doesn't work with the demo setup, and but it's not important. The idea here is that with a with this local test suite, you can cover the 95% of your composition logic. So, I'm personally when I'm building, I'm proceeding locally for most of the logic I cover, I'm writing this TDD style tests. Ooh. And it enables me to

progress like, you know, really 10x. And I leave the end-to-end test to the very end of the pipeline. And that's where the things like chainsaw can create the associated assertion for me. So, this takes much longer on stage than we planned. So, most probably it's a good time to to go back to the presentation. To go back, yeah. The main idea is that you can test stuff

locally, and you can pull this powerful test suite to your CI/CD pipeline, and really postpone the costly end-to-end tests. So, thank you so much. Thank you. So, demo gods were also not with us. Looks like the network is the issue. Let's go back to the presentation. And before we close, I just want to bring some lessons learned here. Um What we learned here when we are using

these API contracts. So, the problem was like the service boundaries and ownership. I mentioned before, right? The super the so-called super compositions. They are really hard to manage. So, now who owns what when it fails? So, the solution for us was like this clear contract boundary that we say, this is your claim, this is the the other claim. You're responsible for what resources are generated with this

claim. So, you put it and the customer itself is obviously also responsible for providing the right inputs, right? The second thing is breaking changes, right? Provider changes something and broke 50 plus teams. So, discovered only production and the rollback was a nightmare. Contract testing plus versioning versioning avoids that. I mentioned versioning is not so super nice in Kubernetes. Um instead of having this so-called dual versioning, with

web with conversion webhooks, you can also go with a new XRD. In our our case, it works very nicely because we know our thousand customers. We can somehow force them that you migrate away from the old version one claim to a version two claim which has a totally different kind or API version, right? But, this works for Um and with that, we have zero production incidents more

or less and unsafe API evolution. So, product teams are defining what they need. Other product teams or customers are consuming it. Um and then documentation drift was also a problem, right? Documentation is already outdated when it's written. This is my takeaway always, right? You write it, already outdated. Um and then you can do documentation as code, so you can put a lot into the annotations or somewhere

beside and you auto generate it based on that. And you can make your documentation testable. When this is also re- rendering out the examples, you can test them, right? This is always accurate, examples are tested, and we have a high adoption rate. So, that's it. I would say thank you. And if you want to have a look at the demo code, use the left your QR code.

And if you want to rate our session, then use the right one. Thank you so much. Thank you. We have three more minutes for questions. If there are any. There is a microphone if somebody wants Just come here. Thank you for the presentation. A quick question. Do Do you have cases where there are like resources that are shared between different clusters, or their life cycle is not

not actually restricted to single cluster? And then, how do you manage that with crossplane state that is kind of bound to a one cluster? We don't have this We don't have this that we have multiple control planes, and they are sharing like But what I could imagine what could work in your in your case is that you, for example, use provider Kubernetes and synchronize it in the

clusters where you need it. Or you use, for example, you can do observer resources like with the management policy observer and read from a data from right from the cloud provider API and connect across the multiple clusters, so it's available. Yeah. Thank you. Thank you. Hey. Hi guys, great talk. Thank you. The pain point about new XRDs or deprecated API versions rings very true. Do you have

any tips for that new XRD? Are you doing things like in the definition allowing for import data from the old XRD or things that make it a little easier? So, if you talk about versioning, right? Yeah. So, we have a very nice blog post written by Nico, our senior principal engineer for Crossplane. The idea is to avoid breaking changes during the design, right? >> Yeah. So, if

you can create an XRD the way that it's evolvable, like for example, instead of single string value, you would write create an arm and for a time being it will be just single element and arm. You design for evolution. So, the idea here to avoid breaking changes as much as possible and just you know, get rid of whole class of problem. So, that's a that's a recommended

password. Okay. What we are what we are doing is for example, we're defining go structs as an API contract, right? Go structs and then we use cube API linter to already do API linting based on that. So, you get already a feeling if you would do something which is not scalable. Okay. This is already a good idea which helps us a lot to improve the API contract

already when we are creating it. Because cube cube API linter would "Don't do this." Got it. Thank you very Thank you very much. Have you migrated to Crossplane V2 already? And what were the things that brought some benefits or what are you looking forward to to get out of it? So, so so we need to migrate to Crossplane version two yet. At the moment at the process

of of migrating it. We are already using version two providers, our main architecture I would not change that much for the time being. The only benefit what we would have then is namespace resources, but we don't need them as we have separate control planes for every customer, right? Thank you. >> But this is on our list, roadmap. Hi, I want to ask about multi-region. So, when you

have a cross plane controller, how do you manage disaster recovery between regions when two clusters need to share some state? Multi-regions. You're talking about two control planes, So, you have you have two separate control planes. One is running in Frankfurt, one is running in Paris. Exactly. And like when you need to for do a disaster recovery for the database. So, like one control plane doesn't work because

you this region is You have a nice talk, right? In I think regarding Yeah, so like but it's a question to you in fact that So, so, so, we are we are just like we are having the control plane, so to say as a snapshot. As an at CD snapshot and we can just restore it on the other one.