From GitOps to AIOps: Evolving RBI's Kubernetes Platform with Cross... Gabor Horvath & Ewald Überall

In this session, Gabbor War and his colleague Evadar share their experiences transitioning from GitOps to AI Ops at Raiffeisen Bank International, where they developed the Mercury platform for cloud-native applications. The talk focuses on the architecture of the Mercury platform, discussing its multi-tenancy features and how they utilize Crossplane for infrastructure management. The speakers detail the specific isolation layers within their Kubernetes environment and explain their service offerings such as Namespace as a Service and Account as a Service. They also outline the integration of GitHub, Cargo, Argo CD, and Crossplane in their deployment pipelines while addressing the complexities of infrastructure promotion versus application promotion. This includes the risks and challenges they encountered, especially related to resource migrations between Crossplane versions, and how an AI-driven tool could assist in risk analysis of pull requests in their CI/CD process.

All right. Hi, welcome on our session. I think we are just in time. There are still people coming in but I would like to start now. Uh, welcome on our session. My name is Gabbor War and together with my colleague Evadar, we will share our journey how we moved uh from GitOps to AI ops using crossplane and shed cargo. We are working at rifles and bank international

which is one of Austria's leading corporate and investment banks. We have 11 subsidiary banks in central eastern Europe and we have 18.6 million customers with around 42,000 employees. We are working in Mercury team where we develop Mercury platform which is a groupwide platform for building and running cloudnative applications. We are a team of 20 people. Out of 20 people, we have 15 engineers. Approximately 100 plus engineers

are using our platform. Currently, we have thousand plus namespaces across 13 clusters. And we have three service offerings. The first one is namespace as a service where we provide a shared OKD open shift based cluster and customers can uh request their an isolated name space and then they they can run their their containerized applications. The second offering is account as a service where we offer AWS dedicated

secured account. They can deploy their cloud resources which can be consumed from the We also have cluster as a service here. The difference is uh we have other internal platform teams. They can reuse our stack. They can reuse our templates and code. And we already have two of such internal platform customers. Uh in all uh here in this picture you will see uh different building blocks what

we have in our architecture and as you see we have all the building blocks that requires nowadays for a modern platform. In today's talk, we will focus more on GitHubs pipeline and on crossplane related uh self-service libraries. In order to understand what multi-tenency means for Mercury platform, I would like to guide you through on three distinct isolation layers which we have. The first one is the platform

Kubernetes account. This is where we deploy our OKD open shift cluster and internal customers will share the uh compute resources but they don't have full access to the to the cluster. Instead what they get is a isolated namespace and as soon as they uh onboard it and the namespace gets created then Kao will create a default deny network policy and by default there is no ingress there

is no incoming traffic or outgoing traffic. We also have strict ARBback controls where only the active directory membership determines who has access to that given isolated namespace. In order not to break out from this name space, we also deploy Kyu policies and then the rest of the namespace is the place where crossplane resources can be created, claims can be created and other uh Kubernetes manifests are deployed.

And the third layer and not many people are expecting that is that we have AWS and there we have S3 buckets KMS other in this way we can do separate billing we can have separate IM roles and it's a individual blast radius for our customers. So this all works perfectly but it makes promotion much more complex because uh just one promotion like a crossplane resource promotion will

uh travels through all these layers. So that's why you have to think about uh how you uh plan your promotion system. Before I uh explain what we built uh I would like to guide you through on the GitHub stack. So here we have four layers of uh tools. The first one is GitHub enterprise. Here we have three kinds of uh repositories. First one is platform related uh

where we define how our platform looks like. We also have customer related repositories. There our customers can onboard applications and also it will be important later they can trigger crossplay migrations. And the third kind of repositories are the self-service crossplane riaries where we define uh what customers can claim. The second is cargo. And here is the what I would like to highlight is we we have only

a single UI for cargo where our internal users can watch how their warehouses are reconciling the changes, how freights are generated and how it can get the freight autopromoted to the dev environment or after different soak times. how it can traverse through test and prod environments and I would like to remember that we have same pipeline for application promotion and also for infrastructure promotion. So the same

pipeline but there are different consequences if something goes wrong. I will explain this later. The third layer is Argo CD but we don't have one central Argo CD which handles all test production tier clusters but for each uh each uh cluster we have dedicated Argo CD and there is actually an architecture decision behind because cargo controller can only integrate with one uh nearby Argo CD and the

fourth layer is crossplane. We also deploy to every cluster. Pix are these compositions. They get rendered. There will be manage resources which then will be reconciled in the dedicated AWS resource account. Yeah. So when we came up with our solution, we had wrong assumption at the beginning that all promotions are equal. But they are not. If you think about when you make a bad image tag, what

would happen? A pod will crash. There will be a roll back. Kubernetes will uh you can trigger a roll back in in cargo. Kubernetes will reconcile. Pods will recover in second. The blast radius is only one uh pod in one name space and the failure will be visible immediately. like in cargo you will have a broken heart icon and you notice immediately. In contrast, if you have

a bad uh infrastructure related promotion like uh you promote through the stages a bad crossben claim claim then um real AWS resource will be changed behind and the reconilation time depends on what what we have behind like if there is an RBS and there is a migration it can take minutes hours and also if you have a KM S key there is this mandatory deletion policy from

AWS 7 till 30 days and it can happen that you have a resource in the in the in the account and you pay for it but you don't know what's the actual state of it. So yes failures can be silent and wrong as you see there are different risks involved with infrastructure promotion and app promotion and also your test is the gate to prod. So if uh

you have a stack reconciliation on test account this breaks the promotion to product account and also wrong or incomplete cross pen v2 migrations can happen. Uh so if you if you summarize and think back what we have we have multi-enant cluster we have several uh customers we uh 13 customerf facing clusters and you need some requirements for your and this is actually a summary what we would

need. You would need isolated ex execution like a failing infra promotion on test must not affect prod also you would need to set up different list policies which means maybe for prod you should differentiate the in the soak time you should have different tools and gates before uh you approve a promotion and you need a visibility on the on the central cargo obviously you can also deploy

cargo go in each and every cluster but then customers would require um UI access always to remember to which UI to go to check what's going on with and from security aspect this is also not good because this one central uh cargo should not have access to each and every cluster and you do not you will not need to expose the API Kubernetes API on every cluster

So this is a solution what we came up and um this is called cargo sharded topology or agent based architecture and uh here on the picture you can see I'm not sure yes you can see that on we have a central cargo we call it SSD account ST and this central cargo can uh deploy uh to different tiers. It can deploy to test and prod and uh

also for the sharded cargos. If you see here this uh six different uh we deploy a local cargo controller and each of these local cargo controller can talk to the nearby local Argo CD controller. So we also have Argo CD using Argo CD operator and we deploy it to each and every cluster. So in practice what our customer would require they would need to uh specify uh

in their uh stage definition spec.shard. So this is how they map the given stage to a given cluster and then the sharded controller will pick up the promotion. It will execute it and it will sync with the local Argo CD. All right. So we know how Cargo handles the promotion. But what about with the infrastructure what it is promoting? So we we're creating our infrastructure with crossplane.

Um I am a big fan of crossplane. I like crossplane very much. Um the idea of simplifying infrastructure with just a few lines of Yum code uh on the one hand on the other hand to have some bunch of managed resources is something that I like very much. We and our team uh were yep we and our team were making use of this functionality of crossprint already

quite early. So nearly right from the beginning when we were setting up our our um our infrastructure when we were setting up our platform we made use of crossplane. At that point of time of course it was just just crossplane version one where we had the restriction of the that namespace scoped uh setups or that the the managed resources were just visible in the clust in the

globally uh scoped environment. So the the managed resources created out of a claim were just globally scoped whereas our teams were always working in a tightly coupled namespace environment. So I said we have a tightly coupled namespace environment with a a shared cluster uh environment and therefore for the teams it was a little bit cumbersome to get out of the to get the information that they were

applying deploying uh they were just able to see the claim. So what we did was that we tried to make use of the status transformer to give a little bit of an insight what has been applied. We were able make in addition also adding some kind of an resource tracking information a list of the resources the list of the managed resources that are going to be created.

We were feeding it back into the status field to give the customers already a little bit of an insight which kind of resources are going to be created. When a new resource is created, they were able to see, okay, from this 10 managed resources that are typically created, five are already done. The rest are still pending. And therefore, a customer was able to identify at least a

little bit when something got stuck. But still, this was not enough. We did not yet have the possibility that the teams were able to debug any issues inside of their own name spaces. they did not have the possibility to verify something was wrong and they did not have the possibility to identify and to debug any issues. So for us it was a no-brainer that we um moved

to crossplane version two as early as possible. This was really a big fit or a good fit for our setup. The the change from crossbin version two that the managed resources from the globally scope managed resources were moved into namespaced resources. So we in quite early when when the public release was was done, we were moving our infrastructure to crossplane version two and we were just creating

versions and we were just creating managed resources using crossplane version two anymore. However, there were also of course a a number of um um previous resources resources that we had already created with crossplane version one and there is no official way of um doing a migration a migration from the managed resources from the cluster scope managed resources to the namespace scope managed resources. So the question for

us was how to migrate because we don't want to keep the old ones running all the time. We want to get rid of crossplane version one and just focus on the development with crossplane version two. A colleague of mine Marik was investigating this a little bit and he found some quite interesting comments in the crossplane community and this was something that we were then trying to enhance.

We were trying to fit it into our own needs and what we came up was in the end a three-steps approach. The first step that needs to be done is to enhance the claim with one parameter. The parameter is just a simple one. It's just migration equals true. Nothing else. We on our side uh the life manifests in the in Argo CD were then enhanced with an

specific update with a specific status information. The uh information that we're we're adding here is a parameter called migration spec. The migration spec itself includes a list of all the managed resources that belong to exactly this one claim. How do we get this information? We are having a inside of our compositions a bunch of functions are being called and out of the context we're able to get

the list of these So we have a list of the managed resources. we were feeding it back into the migration spec and the teams were able to see which kind of managed resources belong to one claim. Another step that needs to be done is to um for the cluster wide scoped managed resources they need to step back. we were putting them into the observe mode um by

this they were giving up up the the full control were giving up the ownership of the cloud resources. So these are the three uh these are the parts of the first step. The second step is then from the customers themselves they were able or they started to create the XR and u as we were able to give the information which managed resources belong to one claim they

were just simply copy and pasting the migration spec into the XR. It's really just a copy and paste. We prepared everything for them that they just need to copy and paste the information and put it into the XR. From that point on, as soon as this XR is applied, the managed resources are being created. The teams are able to see inside of Argo CD which kind of

building blocks they are using. When they are creating an S3 bucket, they get the information of the key, the policies, the uh logging information and so on. And of course the last part that is needed and this is another cool feature that I like on crossplane. We make use of the import functionality. So the cloud resources themselves somehow need to be controlled again of course and we

made use here of the import functionality of crossplane to get the ownership back into the XR. So from that perspective on it was possible to or from that point on it is possible to control the the managed resources again and take the ownership. The last step is just the cleanup. Of course this is obvious. You need to clean up what you have created and what you don't

need anymore. And we are just cleaning up the the claim. And as the managed resources, the cluster wide scope managed resources are anyway just in an observe mode. There is no risk uh of of uh deleting any cloud resources. Um so this is then just the last step that needs to be done. So from the teams who are working with us these uh steps these three steps

gave them the possibility to migrate without our need so completely autonomous from crossplane version one to crossplane version two and is as we did not touch the cloud resources itself they didn't have to use any maintenance window or anything like this to continue with their work right but we still have the human problem because as Evad mentioned customers will still need to open a PR to trigger

the migration and pull request is actually a YAML. It has namespace labels. It has some feature flags and also all the references to the already existing uh crossplane manage So they can make mistakes in this PR right? They for example have wrong wrong namespace annotation or some copypaste errors or YAML indentation errors and it also does not scale like we from platform operator point of view still

need to review those PRs and uh this can be also time consuming. So we were wondering if AI can help here. uh we were asking ourselves what if that we could come up with an AI agent that is aware of crossplane resources. It is aware of all the application states of Argo CD. So what if it will review the migration PRs? It will compare against the actual

cluster state not just the YAML diff. And we also wanted that it can flag the risk before it entering before the change is entering into the pipeline. And if something still enters the pipeline and it goes wrong, we would like to do Argo CD application failure analysis. And in this way, we can give some hints to our customers to remediate the problem. But it's important to mention

that this AI will not automate the migration. It still requires the human, but it will just surface some risks which otherwise would get unnoticed. So let's uh prepare a live demo and what you will see here is that uh we will mimic we will take our developer head and we will mimic that we are our we are triggering the migration. So we will open a crossplay migration

PR the sharded cargo will pick up the freight that contains this change. We will we will make an intentional mistake a wrong y section we will use for migration spec. We will then see how the automated AIP review will grade the risk of the change and then we will also do an Ago CD MCP analysis and see what went wrong. Then we will fix the PR repromote

and see how the promotion succeed hopefully. >> Yeah, let let the demo cards be with us. So, first let me prepare everything. And I'm just going to refresh the screen that I'm at least not following into some kind of timeouts for this. So, so what we have here is a simple Y as said I like the idea of having the abstraction from crossplane that it simplifies things.

We have here when S3 bucket u the claim that we use for creating an S3 bucket already for the sake of time I have added the migration parameter equals true. So uh this part is already set and um what else do we have? We have our sharded cargo infrastructure with the warehouse and the stage uh where the the warehouse is promoting to. And of course last but

not least the Argo CD application where we have this one claim visible at the moment. Um I said uh inside of a claim uh we that the users were just able to see the claim but not any of these managed resources that are being part of it. So let's have a look at the bottom and make it a little bit bigger. So what we have here is

this additional parameter the migration spec parameter that we added and you see here all the belong to this one claim. We have a KMS key issed for an S3 bucket. We have the allias for the KMS key life cycle rules. And if you wonder why there are log and main when we create an an S3 bucket we just don't create the S3 bucket itself the data bucket

but each bucket itself gets a log bucket automatically as well. So what we need to do is to simply copy and paste this migration spec and add it to the XR. So what we are doing here is we just simply and copy paste the whole configuration from the previous setup. This is not crossplane version one anymore but crossplane version two. The differentiator between those two is in

our case of course you always need to have some kind of a differentiator if it's crossplane version one or two is in our case that the API name has changed. So we renamed our API instead of crossplane. Mercury.rbi.cloud cloud. We just renamed it to mercury.rbi.cloud. Nothing else in the end is necessary. So by applying this, this migration would have done successfully. However, you can't imagine how often

it happened that people were just simply doing a wrong indentation. So putting the migration spec not just a few lines where it should be, but just putting it on the root level. So as we want to do an error by intention, I'm storing this now and pushing it to the repository. So then let's reduce the size of this screen and go back to Argo. So the promotion

has already been started. That's good. So we have a new thread already being created. A promotion is already being triggered and the promotion steps executed already till the level that the pull request is waiting for me. Let's have a look on this pull request. Okay, now the demo gods are there. Uh so typically what we would see here now is uh that this PR is not just

automatically created by cargo but it should give some kind of a risk analysis. It should give some information um what kind of risk you have when you run this PR. Um it gives a context of the context summary of what the pull request should include. Um let's have a look if the rest of the demo still succeeds. So, I'm promoting the PR refresh. no. So, what is

Let's wait. Let's refresh the screen. So, I think I have to go back now to the backup. Uh, let's just wait one more second. But I guess that Let's just refresh once again. Nope. Okay, then we always have a backup. then let's just jump to the place. So this would be the pull request that we would have expected um a pull request with a risk analysis and

a reasoning. So the information why uh what has been added and what has been approved and what is uh what the pull request is part of. Um the next stage would be that yeah we copy the migration species we did already and so the risk analysis that is then taken um should come up here. So we made an error in the we we made an error in

the um approval or in the in the uh um changes in the code that we did and expect now that Argo CD is is syncing as there is an error argo CD is not able to sync but uh is um uh that the sync is hanging or or will not be able to complete. What is then happening in the background is that this is then your part.

>> Yes. So what normally you would see here you know in in the currently in open source cargo you can have integration only with HTTP step. So what we came up with that this HTTP step is uh calling an internal endpoint which is integrated with our LLM and it's integrated with our GitHub MCP for the PR risk analysis and uh we also um return the risk rate

and we also return the reasoning from the LLM what went wrong. So here you here you see that um and now we are after the step of Argo CD update and here the Argo CD diagnosis is ongoing and since it returns a structured JSON output cargo can parse it and can present it here to the end user. What was the what was the issue? So here on

the video you should see So here you see that the migration spec was put to a wrong um wrong yl section and uh it gives you hint how to fix it. Yeah. But important is to understand that here we develop next to cargo a small application which is then integrating with LLM and other tools. >> Okay, that's good. >> So from the demo it's just that the

uh migration spec has then been fixed. Let's move forward to because now we can go fast. The outcome in the end is that um uh where is the part that? So the part in the end should be that we have the not just the one claim visible from crossbin version two but that the teams have the possibility to see now the full list of managed resources inside

of cargo inside of Argo. they have the possibility to see all the resources directly and this is in the end what we wanted to present here on this part of the demo so that the migration itself is typically successful um but yeah yep going back to the slides right so maybe just one one more to go through on it um we made an application this exposes two

end points One is analyze risk, one is the diagnosis endpoint. Uh the risk is called before promotion. It takes as an input the GitHub p request. It will make the PRDF and the metadata from uh GitHub. We use cloudet 4.6 and the output which we return is the risk grade which can be small, medium or high. And this is what we present in the cargo. The other

is diagnosis endpoint. This runs after promotion conditionally only in case there is a failure in Argo CD and it takes the Argo CD app name and all the uh resources in that application and the status and events and based on that it will just return if it's healthy or not and with some remediation steps. Right? So what we would like from you to take with you is

um uh if you want to scale GitHubs then shed cargo is one way to do it in a way that you will not lose your control on the multi-enant environments. Also, you should tra treat treat uh infra promotion a bit differently than uh application promotions uh because there are different uh risks involved as I mentioned before and uh although cosplain uh v2 fixes the tenant visibility problem

for our internet users, the migration is still manual and it requires some custom tooling like you can use AI or any other uh uh techniques to clock the further pro promotion in uh production and what we would like to highlight that uh our experience is that the role of the AI is not really a magic it's just uh a second pair of eyes in the moments when

when we when you as a platform operator are under pressure you oversee some complexity it can really help you to identify issues and how we want to take this further we saw on aropon on Monday that cargo's developer Acuity uh introduced custom promotion steps to better integrate with we can use it to better integrate with LM and other MCP servers. So basically we have more freedom to

uh to integrate with third parties and we also would like to offer uh to our customers as a this promot this promotion analysis and uh PR analysis and Argo City anal diagnosis. We would like to give it as a reusable cluster promotion step in cargo. We would like to sunset crossplane v1 and yes if you are already migrating to v2 at scale we would like to be

interested how you are doing it. Let's have some talk and let's chat about that. Thank you for listening. Um if you have questions there are microphone here. There are microphones, one at least here. And please Thank you.