Inside Saxo Service Blueprint: Implementing Kubernetes Operators for Legacy Ent... Oskar Kristiansen

This talk discusses the Saxo service blueprint, focusing on modernizing and automating infrastructure and IAM provisioning at Saxo Bank using Kubernetes operators and GitOps. The speaker, Oscar Christensen, outlines the challenges of traditional provisioning processes that required multiple tickets and lacked transparency. This process has been streamlined by allowing developers to express service requirements in a YAML file, leading to an efficient automated provisioning system managed by a platform. The architecture incorporates various tools including Flux and Argo CD for GitOps workflows, while addressing a mix of cloud-native and traditional workloads. Additionally, the use of custom operators for specific integrations enhances flexibility and operational efficiency, ultimately ensuring compliance and governance for service deployment.

Today, I want to speak about the Saxo service blueprint, uh which is what me and my team has been working on for the last 3 years. And it's to use Kubernetes operators and GitOps to modernize and automate the infrastructure and IAM provisioning within Saxo Bank. My name is Oscar Christensen, and I'm a senior enterprise platform engineer at Saxo. So, to begin, I will quickly present Saxo Bank

so we have a context for why this system is beneficial and what led us to take some of the decisions that we did. Saxo Bank was founded in 1992. It's denoted as a systemically important financial institution in Denmark. It manages 118 billion euros in client assets and is spread across which is spread across 1.4 million end clients. uh we work across 12 offices, which together contains 2,432

employees. We have both direct consumer-facing products for investors of different levels as well as partner integrations and APIs. And uh Saxo Bank operates across the world with spread-out offices, institutional clients, and direct clients as well. Financial instruments can be traded through Saxo includes everything from currencies, bonds, mutual funds, margin lending, even crypto. And uh in total, it's more than 71,000 different ones. So, all of this is

just to say that the product offered by Saxo is very complex, and it's under heavy compliance and regulatory demands. And as a more than 30-year-old fintech company and bank, it includes an enormous amount of more or less legacy applications and infrastructure, which must be kept running and compliant. So, that's the business. Uh so, now I can tell you about the engineering problem that we had to solve.

Uh first, we'll look at the old way of provisioning infrastructure in Saxo. Uh a developer has built some new service, and it works, and it fits the requirements, and the test passes, and everything. But before it runs, of course, it can run it, of course, it needs a few things. Needs a server to run on. It needs firewall rules so traffic can reach it. It needs a

service account so it can authenticate to internal systems. It needs a role grant on an SQL database. And it needs DNS and uh TLS certificate so other services can call it and verify against it. So, for this, uh this is like the bare minimum to get some service running in production. And that's already six things across five different infrastructure teams. So, the old way, they would open

six tickets or send send off six emails. And then they would wait because each of those items has a queue, and their request is somewhere in it, and they have no idea where. They can't see whether the AD ticket has been picked up or whether the DBA has started on the role. So, they just wait. And if something goes wrong, then they had to go through the

whole process again. So, there's no authoritative record of what should exist. There just is what exists. And this is in no way a criticism of the teams involved here. They're very good at what they do. The problem we had was the process. So, this is what we set out to fix. So, here's what we built. A developer writes a YAML file describing what the service needs. They

open a PR. It gets auto-assigned to uh domain owners and architects responsible for the things they're asking for. And then it get merged, and from that point on, the platform take care of the rest. It's identity, it's access, its place in the platform, and it's automatically and continuously across all environments. So, that's the contract. One file, one PR, one approval, and then it's done. And uh before

I show you what that file looks like, I want to quickly mention why we're not just saying containerize everything and use standard GitOps because that is a very fair question with modern tooling, and the answer matters for why this system exist at all. So, some of our services uh are very old. They've been running for many years, maybe even 15, 20 years. They are battle-tested, stable, and

deeply integrated into the rest of the estate. The cost and risk of refactoring them to run in containers is simply not justified. Some workloads use IP multicast, which our current Kubernetes network setup doesn't make straightforward to implement. And some traditional software is genuinely difficult to containerize without significant rework that doesn't really provide business value. So, the reality we're left with is a mixed mixed estate. Uh we

have cloud-native workloads on Kubernetes, but we also have traditional workloads on servers that are not going anywhere anytime soon. And both need their dependencies managed. So, now we move on to the YAML. This is what the developer actually writes. Um a name, description, metadata about where and how the service runs. Then uh short identifier, which Kafka topics it consumes, for example, which APIs it provides, and who

is responsible for it. That's it. There's no ticket or knowledge of how AD works or Kafka ACL syntax works. It's just the intent of the developer. And from this, the platform provisions everything it needs. It derives everything it needs to provision. And I'll get into exactly how it does that in a moment, but the point here is the abstraction boundary. The developer owns the what, and the

platform owns the how. uh I just want to mention all these YAML files, they just live in a Git repository, one singular repository with 45 business domain folders and more than 1,000 services with one file per service. And config can be overridden for the different environments, so the same service definition drives dev, test, sim, and live dependencies. So, that's the catalog. It's basically GitOps as you would

GitOps as you would do it with Flux or Argo CD, but for everything around what is deployed in Kubernetes as well as for traditional infrastructure. In order to trust the correctness of the changes proposed to the service catalog, the ver- we have this verifier service that automatically triggers when a PR is created. And it validates structure, uniqueness of short names, API references, and so on. Uh many

such checks are run so the developer can feel safe in what they're asking, and reviewers can also know can sort of judge and approve on the intent and not the uh the syntax syntax correctness. We also run a security approvers verifier to pull in appropriate reviewers depending on the change being proposed. So, if a PR touches role provisioning, for example, it will pull in the people or

groups responsible for those roles to say yes or no, basically. Lifecycle is another such field. A new service starts with lifecycle implementing, and it's a very lightweight review. They get to come into the catalog quickly and provision infrastructure in dev and test. But then to actually deploy things in sim and live, they need to to ask to change lifecycle to production, which is a more heavy review

process where we pull in security, GCC, governance reviewers. And this ensures that they can quickly iterate and test out stuff, but only actual validated, compliant applications can make it to I also want to mention very briefly that the operators that I'm about to describe also consume a second repository, the infrastructure catalog. And this is for reference data such as where traditional infrastructure runs, server collections, database host

names, and things like that. Okay. So, now let's look at the actual automation, how a service becomes So, here is a sample service definitions on the left and all the current operator-based infrastructure provisioning happening behind the scenes on the right. And we will look at a few of the most important and interesting ones. So, there are two stages to make it into Kubernetes to reconcile. The first

is the blueprint engine. This is a long-running service. It runs inside Kubernetes continuously polling service catalog uh Git repository on uh interval. And then on each iteration, it clones the repo, scans the YAML files, applies the environment overlays, and then converts each of them into a Kubernetes custom resource, a blueprint CR. And then it applies those blueprint CRs to the cluster, and from that point on, Kubernetes

owns the desired state. Um the second stage is the blueprint operator, which then reconciles those CRs. On every reconcile, it runs a set of transformers, and it's one per infrastructure domain. So, each transformer inspects the blueprint spec and decides what child CRs to emit. So, it's things like namespace requests, GMSA entities, Kafka ACL requests, and so on. And those uh child CRs are then written to the

cluster. Then, after that, the special uh specialist operators, each watching their own resource type, take over and drive the actual systems. So, I'll walk through what that means in practice with a few examples. So, first of all, the namespace annotation. If the If a YAML service has it, then two things happen. The namespace transformer emits a namespace request CR. The service account transformer emits a service account

CR, and then two separate operators reconcile them. The namespace request operator is the more involved one. On first reconcile, it creates a Git repo from a template with a initial ADO pipeline and a build validation policies called to production and sim environment changes, and that is the tenant's GitOps foundation. So, that is where developers will actually deploy things into Kubernetes realm. On every reconcile, it also creates

a group entity Kubernetes CR, one per environment, which drives the AD operator to create create the corresponding Azure AD access groups in Then it writes the namespace YAML with the AD group object IDs embedded to the central container platform repository via an auto-completing PR. And then we have Flux that picks that up and reconciles the actual Kubernetes namespace to the cluster. At the same time, it also

writes a Helm chart values to a separate elastic metadata repository, which provisions a Kibana space with log sources and data views for every environment. And finally, it raises a PR to the cluster-wide network policy repository, one per environment, where it writes in a Cilium network policy that allows intra-namespace traffic. And when all this is applied, basically the developer is ready to deploy things within their tenant, and

things will actually be allowed to talk to each other, and they can see the logs, and so on. Simultaneously, the service account operator writes a cross-plane user-assigned identity CR per Crossplane creates the Azure managed identity and reports the client ID back in the CR status. The operator then raises one more PR to the central container platform repo, where it embeds the managed identity binding. So, and Flux

propagates it to every cluster, and then the workload identity federation is live. And services can then authenticate against Entra ID, which we use, using their Kubernetes service account. Now, I'll give another example, the group managed service account. So, for services that run on traditional infrastructure, which is Windows servers in our case, not containers, the gMSA transformer reads the short name and cross references with the infrastructure catalog

to find which server collection the server runs on. It then emits a gMSA entity CR, and the AD operator then reconciles that by calling a AD GR endpoint, which is a constrained PowerShell remoting endpoint over WinRM with Kerberos authentication. The GR endpoint runs a command, basically a PowerShell command, against the domain controller and creates or updates the gMSA account. The operator never touches the domain controller directly.

The security segmentation is enforced at the GR Now, I want to discuss the API relationships. Okay, so we'll begin with Kafka ACLs. The Kafka ACL transformer reads the service consumed and provided Kafka API references. It figures out the identity the service is running under. If it's an app registration or managed identity, it gets the client ID from Azure API just by calling it. If it's a gMSA

account, it it checks the gMSA CR and extracts the SAM account name from its status. The Kafka operator then clones the relevant data hub ops Git repository, finds the topic definition YAML files, and appends the client ID or SAM account name to the reader or writer list, depending of it's consumes or provides. And I want to note that Kafka topics are defined as API types and are

also present in the service catalog, as are other dependencies like actual OpenAPI or gRPC APIs, and even databases as Now, let's discuss app registrations and access grants. So, these are provisioned if we see that either the service provides some API for which it needs to provide authority, or if the service uses a gMSA as its identity and therefore needs an app registration to authenticate. So, if the

service declares that it provides a gRPC API, the app registration operator builds an app role manifest from the declared API definition and posts it to an internal endpoint, which creates or updates the Entra ID app registration, which is complete with app roles. On the consuming side, if another service declares that it consumes that gRPC API, an Entra ID role assignment CR is emitted, which specify which managed

identity should be granted which role on which managed identity, and then the platform grants the access once reconciled. Now, for a final example, let's see how we provision network access within Kubernetes for services that declare that they are containerized. So, the cluster-wide network policy transformer reads the namespace annotation, the short name, and everything in consumes and provides APIs. It takes all of that and then it derives

what egress a service should have inside Kubernetes. For Kafka dependencies, it just adds a label to the namespace YAML in the profiles repository, so the Cilium policy allowing access to the broker is applied. For Entra ID dependencies, it's the same thing, just another label, so managed identity outbound calls to Entra ID are allowed. For SQL identity provisioning, it creates a fine-grained policy to allow the specific service

account in the services namespace to access the primary and secondary databases in question. So, I want to quickly note why we don't just use Crossplane or Terraform for all this. First of all, Crossplane is used here, of course, for the Azure managed identities, and we think it's the right tool there. It's a standard cloud resource, standard provider, it's straightforward implementation. But for things like the GR endpoint,

internal Kafka APIs, internal provisioning services, there isn't an out-of-the-box Crossplane provider for these, and writing a Terraform provider for each would be quite significant work with not much reuse in between. Custom operators give us complete freedom to do these integrations, and once we had the shared framework, the core reconciliation loop, the status management, etc., then it's actually quite a small focused piece of code to add a

new operator. I also want to give a note on item potency. So, the operators are designed so that running the same reconcile a thousand times will do exactly the same thing as running it once, given the same environment it's running in. So, for the GitOps parts, before we raise any auto-completed PRs, we always check the actual state, and then of course only create a PR if that

state doesn't represent what the service intends. For the gMSA accounts, the AD operator tracks a revision attribute on each account in the CR status. So, before the calling the GR host, it checks whether the declared set of computer assignments matches the actual state and the of the account, and then only calls the GR endpoint if it doesn't. We have learned the hard way that depending on transitive

properties like, for example, the status on the custom resources to determine if a change should be made or not can cause unforeseen consequences when, for example, we migrated all the custom resources to another cluster, and suddenly learned that once that status is removed, then there actually was some change that should be done, even though the status said it shouldn't. And then then then that happened on all

the custom resources all at once. I also want to discuss the consequences of all of this for disaster recovery. So, as you would assume with any standard GitOps setup, the recovery path is we stand up a fresh cluster, point it at the same Git repositories. The blueprint engine, which is already running as a service, picks it up on the next pull cycle, regenerates all the CRs, and

applies them. And then the operators just start reconciling. So, within one cycle, they drive every integrated system, AD, Kafka, Azure, namespace, and policy Git repos, and so on, back into the desired state. This is especially relevant for us from risk management and compliance viewpoint. Having our internal system hacked is a nightmare scenario, as it's not as simple as just standing the broken resources back up, as you

can't know if bad actors are still messing around inside them. So, having the access controls, network openings, and the fundamental building blocks be auto-provisioned in a separate and isolated environment means it actually becomes realistic for us to recover from potential intrusion to the core I also want to give a note on the impact that this approach has allowed us to have. We are a very small team

with three or four full-time members, depending on how you count. We don't own AD or Kafka or Azure SQL or Elastic or any of those systems. They're all owned by expert teams, which has deep expertise in them. So, by integrating through the interfaces that those teams already maintained, we automated across all these domains without actually taking on the operational burden behind them. We just drive them from

a reconciler. And uh that's why integration flexibility matters beyond the sort of technical convenience. It allows a small team to cover so many uh huge um systems in a large organization by just integrating with what already existed. Everything I've described so far are governed resources. They represent the service place in the enterprise. They need audit trail and they need review. And therefore, they belong in the catalog,

but not everything does. For example, microservice database access is one such example. It's owned entirely by the dev team themselves, and they should get to decide which access is granted at to whom. But um we still wanted to build a solution for them to get the same advantages. So, we provided some uh tooling basically. This Helm chart goes in their own tenant repo that our operator created.

And then each database list the roles the identity needs, and the managed identity identity thereby gets the listed database roles. Uh config map with connection details is created in the namespace. So, when it's rotated or if the primary database goes down, they can immediately reconnect without manual intervention. this is still continuously reconciled by our blueprint operators, but it's just an uh distributed way instead from the tenants

instead of from the central repo. The Helm chart creates uh SQL identity CR. Once reconciled, the operators call the Azure REST API to identify the databases, and then it calls our cloud graph API to create the actual user and assign the roles for the identity. Another example I want to mention of infrastructure provisioning that doesn't belong in the in the central repository is things like DNS, certificate

provisioning and rotation, and load balancing strategies for our services that runs on traditional uh machines on VMs. These uh not governance decisions. They are routing decisions, so they belong with the team that actually owns the Um they specify some metadata that's tied back to the service definition in the Helm chart, the service uh which the application runs on, and which port this service listens to. we leverage

the Kubernetes ecosystem to provide all required resources and actually perform the routing using Envoy. We pre- present a certificate creating using cert-manager and register a DNS uh for the service using external DNS as you would. And then we do the load balancing, health checking, and routing using Envoy and also perform upstream TLS verification. And all this required infrastructure, which traditionally for these types of services would have

been quite a bit of work uh from expert teams to do, they can do themselves with just this short YAML snippet and using the available cloud-native tooling. finally, I want to show how we provide the transparency into the core infrastructure provisioning, which I mentioned initially was missing from the old approach where you just send off an email. Some of you may have noticed that our service definitions

looks familiar, and that's because they follow the backstage service definition format. And it's actually the same for the APIs and even the server collections and database uh definitions as well in the infrastructure catalog. So, that this means that all the components are automatically ingested into our internal backstage instance, which we call The Reach. Uh and here developers can navigate to their service and see information and status

for their provisioning. So, I've gone a little bit uh over time, so I'll try to be quick here. So, this is uh The Reach, and I'm sure you most of you already are familiar with Backstage, so we'll go to the catalog here. we'll go to this one. So, here you can see some of our integration. This service in particular is a Kubernetes service, and of course there's

links to all the relevant things, the source codes and the logs and so on and so forth. So, you can get sort of a at a glance uh feeling of the service and how it's doing. Of course, you can also see all the relations. So, everything that the service is has dependencies to, all APIs, who owns it, even which developers work on it, it's all captured in

its relationship graph. Um under Kubernetes, since this is a Kubernetes service, we have the Headlamp embedded via Backstage plugin, so they can go and monitor their resources directly from here. We also have things like Prometheus, and they can create sealed secrets for themselves and see cystic scans and so on. But really, I wanted to show the blueprints tab here, which is where they can go and actually

see the status of the CRs that I have mentioned. So, they can for example see go and see that the the namespace that they use has been successfully provisioned, and also the sub resources like the AD groups I mentioned have been successfully provisioned. also want to mention, since this is quite a lot of custom functionality, you might say, to help developers navigate this, we have created of

course an AI which follows around, and we call it Eva. So, we can say, "Hi, Eva." And then Eva will respond with all the or at least some of the many uh integrations that we have added into the AI. So, it can actually look up the uh things inside Kubernetes. It can basically use kubectl, but in a very limited way, so it doesn't break anything or read

any secrets or anything. Uh the same with Prometheus, and um the same with the with the blueprint custom resources. So, it can basically help the developers to see the status of their infrastructure in uh sort of live. They can see it immediately. Also, we've given it a bunch of skills, so it's an expert on everything that I just showed you and much more around this blueprint setup.

So, developers and it can also read the tenants and read the service catalog, so it can tell developers in natural language exactly how to get what they want for their service. It also has actually uh it follows you around here in the corner. Even inside Backstage, it can help. You can ask it, like "Uh what is this page I'm on? What can I do here?" And it

can actually see the page. It says we're on the network checker page. um and it has access even to its own source code, Backstage source code, so it knows what is going on here. And it gives the suggestion to actually check the firewalls, so let's say uh yes. Is it open from 1.1.1.2? See, TCP. uh it actually has access to the tools in Backstage as well. So,

it can tell me that this uh opening is closed. I also just want to quickly show I copied in so I didn't have to type it This is uh our namespace infrastructure automation, it's called. If I >> can cover far more ground that the size would otherwise suggest. Thank you for your time. Have a great rest of the conference. Uh I wanted to say ask me any

question you want, but we have 3 seconds left, so we'll end it there. Thank you.