Rabobank’s Path to Secure, Fast Kubernetes Delivery - Beatrice Forslund & Koshin Verberne, Rabobank

This talk focuses on the secure and efficient delivery of Kubernetes at Rabo Bank. The speakers, members of the container platform team, discuss their journey of managing and delivering Kubernetes clusters while adhering to strict financial cybersecurity regulations. They explain the challenges faced in ensuring security and compliance, and how they established a platform team to streamline operations for developers. By employing a cluster-as-a-service model, they provide dedicated, isolated environments for tenants, ensuring that compliance and security are prioritized from day one. The session also covers tools and processes used, such as Terraform, Argo CD, and Kyverno, which help automate and secure the deployment process.

Good afternoon everyone. Uh thank you for attending our presentation about uh how we deliver uh Kubernetes uh in a secure and uh well very fast uh delivery at the at the Rabo Bank. Uh I am Kosher Fern. >> Uh my name is Petris. >> Yes. So we both work at the container platform team at the Rabo Bank. Uh managing and delivering Kubernetes clusters to uh to tenants

at the bank. So today we will tell you about how what our journey was like to uh delivering Kubernetes uh the decisions and challenges we faced at uh making uh the Kubernetes experience secure and compliant according to the bank's uh regulations and standards and uh how we overcame these challenges and what advice or giveaways we can share with you the audience. And we know it is probably

one of the last presentations some of you will attend. So we try to keep it also a bit light-hearted. Uh little bit about ourselves. So I am a uh platform engineer who previously worked at the AWS platform team for uh about five years. So actually Kubernetes is uh relatively new for me. I started last year with it and uh I've been learning quite a lot uh going

through it and uh also from this event. >> All right. Yeah. I am I started working in open shift um now working with AKS and I'm also a cubestronaut and this is our f first uh talk at cubecon. >> Yes. So uh told you about uh Kubernetes and what we do with Kubernetes. So maybe for the uh international audience who don't know about Raab Bank, uh we're

a Dutch bank uh serving close to 9 million uh customers and uh we also uh operate internationally to uh well fund and finance farmers all over the globe. Uh well and how we do this we have a lot of uh of our infrastructure uh hosted across uh three uh landing zones uh at the bank and uh in recent years also a lot of developers have really enjoyed

building and operating their own Kubernetes clusters especially in cloud for some uh understanding of what the uh landscape looks like. So for on Azure we have about 147 or close to 150 uh Azure Kubernetes uh service clusters uh running and 473 uh Azure container uh apps uh which is underneath also yeah managed Kubernetes. uh second for the uh second provider we're also running on AWS EKS so

elastic kubernetes service with there a little bit over 40 uh EKS clusters running there and uh we also have still our on premise running uh where we have about roughly about 12 open shift clusters and on open shift we started from the beginning with a uh container platform team but uh the landscape in cloud especially is uh very variet so a lot teams there were in the

past allowed to uh create and operate on Kubernetes clusters there but throughout the years as we started up maturing our platform offering we've begun to approach these teams to tell them to migrate and uh also get uh backing from architecture to say like yeah we need to have more restriction around uh uh creating uh clusters uh in cloud. So having a bit of an understanding what uh

the landscape looks like uh why do we need more uh yeah restrictions uh around security for communities. >> All right so just like any other banks we have very strict uh financial cyber security and anti-rime laws. Um and these laws are not coming from anywhere but they are coming from our Dutch national bank and the European Central Bank. uh this means if we miss a control uh

we could have huge regulatory fines and even losing our uh banking license. So our security is our top top priority. Um and we cannot just say oh it's secure but we also need to prove it. Uh we have regular audits uh from uh internal and external regulators who require evidence that we are in control of our security. Everything needs to be documented. Uh risk owners need to

be assigned and if there are any deviations they must go through risk acceptance and be known. Um furthermore uh they will also check. So we are um keeping up with um MFA, modern encryption, threat detection, network segmentation and more. So basically all all of our deployments go through a security compliancy and operational checks that will keep our customers our systems and our banking license safe and that

is why we cannot just ship things to production and our talk with will will be mostly about that. So Kushin will continue. >> Yes. And uh to make sure that not every uh DevOps team within the bank has to reinvent the wheel themselves with regards to uh securing Kubernetes, we've established a container platform team a couple years ago called Qup. So that is uh pretty much the

concept of platform engineering as you all would have uh learned throughout this conference and how platform engineering is handled. uh it's very important to have the relations established and clear boundaries and responsibilities of what the platform team um is responsible for uh versus what a tenant of of of a cluster uh is responsible of. So as a platform team we build the underlying infrastructure. We set up

the networking uh we also uh handle the security and compliance reporting and enforcing security for our clusters. uh and to make sure that the cluster is uh usable as soon as possible, we uh have all the uh well requirement uh permissions and freedom for our tenants to operate their part of the uh part of the software stack. Uh and as a tenant, you pretty much uh can

just get quickly started with kickstarting your project. uh you have the the needed freedom and the capacity to test out and run your applications before you go live uh and innovate. That is pretty much what we try to enable for for our uh consumers. So we built the foundation and you built the future. Uh and what it would look like is the basically the uh the example

of a car. The underlying engine is built and and defined for probably most big car companies that use the same underneath underlying engine or platform. And uh the hood itself, the the chassis is uh can be up to the liking or design of a of a consumer with a different color or whatever they would want. So that nicely fits together into having one operating and fully running

system from day one. So to give you a landing zones look like on for the container platform. Uh here you can see a whole list of solutions and tools we make use of uh within the container platform team. Uh and some of the tools you might recognize is used across all three landing zones. So for example, do we all use the hotel agent to collect uh metrics

and uh logs which we send to uh what we use within a bank as a standardized uh uh monitoring solution called Splunk. So there every developer uh knows like okay this is where I can get my metrics and uh logs from to uh create alerts or to create findings for my application. So that is what we try to achieve across these three landing zones is also a

uh to give a similar experience to our tenants that they don't need to uh face too many challenges or differences across each uh environment. Uh but where it makes sense we do uh deviate sometimes from uh standards across uh landing zones. For example, some of the uh services that that are supported within in in a cloud environment such as for uh vulnerability management. You can see we

make use of Microsoft Defender for cloud on Azure and uh on AWS we use a service called inspector to uh perform uh vulnerability management uh processes. Uh the underlying data is similar in a way. So we can do same reporting for those tools. So from an architecture point of view there where it makes sense we can uh deviate from each other's standards but we do try to

pursue a uh similar um and yes starting from uh sort of from on premise as I explained there we had open shift already uh how did we design the uh the solution across the other two All right. So when we wanted to start out with our platforms um AKS and EKS, we started investigating how we should offer Kubernetes as a platform platform. Uh and we looked at

either giving a namespace or giving a whole cluster. Um we already had Open Shift running in production and they followed giving a namespace approach. uh because they had it working and we wanted to aim at giving the same offering to them all. Uh then we started looking into npace as a service. Um giving a name space a service uh we called it nspace as a service um

which means that all the tenants are running on one cluster uh one share clusters and that means that that they share the same underlying compute resource sources. So it's uh nodes, CPU, memory. Um and because of these because these resources are pulled together instead of split across on many small clusters, it means that the overall usage is a lot more efficient. So instead of having idle capacity,

the shared cluster could smooth out uh usage across everyone, which would lead to lower cost and fewer resources. Um however uh we came to a problem which how can we guarantee that tenant B cannot reach the weather API on a network level because at Java Bank we have very strict isolation uh lot rules and on who can access what. Um, we found basically ways how to do

it with some Kubernetes policies, but we wanted it outside of the cluster and we require basically require fire firewalls rules enforced on the node and network level. So, this was a bit trickier. Um, our goal was to have secure clusters by default and we we need them as soon as possible with as little complexity as possible. So instead to not have to worry about this deep network

level isolation from day one, we decided for something else. So we decided is then for a cluster as a service. So here each tenant will get their own dedicated um which is isolated in its own virtual network. Uh this means if tenant A needs to access the weather API, you can apply that rule specifically to tenant A's network and tenant B cannot longer access it. Uh the

same with private uh private endpoints. Uh now you can create it for one team and one tenant and not be used by another team. Um and this shift to clusters dedicated clusters still provided a strong business value. we can deliver a secure and compliant Kubernetes environment by default and yeah our main priority is security over cost and on EKS they did it a bit differently so cushion

>> yes yeah so from a network topology it is similar so we have a isolation between the uh the VNET on Azure but also on the VPC level on AWS so we included and isolation also on contra plane level. So every tenant that on boards on our uh Kubernetes platform they receive their own they have they have a requirement to have their own AWS account first created

and from within where we uh provision the EKS cluster. So the isolation is also on the control plane that for instance uh tenant A cannot reach the S3 bucket of uh tenant B or that uh databases on for example RDS uh or Dynamo DB cannot be uh accessed by other tenants because you get this isolation on an account level from the start you don't have to write

create very complex uh IM policies or resourcebased policies uh on each resource that is created by your tenant. So that helped us to uh scale quickly and get started uh as early as possible. So having the architecture decisions made and how we want to offer the uh cluster as a service uh we went into designing uh the platform offering and the tools that we uh employ. >>

All right. Right. So we deploy everything with uh um everything info related with Terraform and then inside the cluster we use Argo CD. So now that we have the platform, we wanted everything that we push to production to be secure and tested and that every change no matter how small should meet the same security standards. Um so basically what we did is every pipeline that we have

goes through a code quality scanning task which for example we are using curcube. Um also a vulnerability scanning tool uh for example check marks and then a tool that scans for secrets for example secret scanner and these three tree three task helps us to ensure that everything we do is secure. Furthermore we also use a tool called renovate. Uh this ensures that we are always keeping our

dependencies and base images up to current. Um, when we implemented this new tool, we saw that there were 20 plus new updates and now we still get uh five to 10 new updates weekly, new patches all the time. Um, it was a bit hard to keep up with because we require everything to be tested uh before going to production and this becomes quite a operational um effort

for the team. Um so here it helped us to have a really clear uh structure uh prioritizing the updates. Uh we split it so that one person is responsible for these updates weekly and we apply to uh test development first and if there's we also check if there contains a vulnerability fix then we prioritize that first. Um for continuous deployments we also use Argo CD. Ergo CD

is relatively new inside our bank. Um because our compliancy rules were originally built around traditional pipelines, CI/CD pipelines. Um and there were many discussions how should we make Argo CD compliant with our IT change Um with standard CI/CD pipelines you push a change to production. However, with Argo CD it is pulling from our git repository. And one requirement that we had for Argo CD is that every

change to production needs a change task registered in our IT storage service management tool prior to deployment. So instead of having Argo CD tracking our main branch, we built a lightweight pipeline whose job is to create a a change to our IT sim and then creates a new git tag and this pipeline is then triggered with each merge pull request and Argo CD will watch this new

tag instead of watching our main branch. So this means that we keeping our GitHubs model uh intact while also satisfying our strict government's rules. Um so here we yeah we we really prioritize security but then we try to make it uh useful and this is our platform and now we started having some discussions with tenants which question. >> Yes. So uh now that we have uh established

designed the platform and uh can get started with onboarding tenants uh we uh of course need to focus around the topics of uh governance like what is a tenant allowed to perform within its cluster and which uh API actions is it absolutely not allowed to perform uh while giving them the the needed uh yeah freedom and uh possibilities to uh build and test out new applications or

experiment with uh potentially new custom resources that they might provision. So for this uh we heavily rely on Kivero. Uh actually during this event I also learned so much more about the keyo policies and and now the new migration to cellbased policies. So that will definitely help us out uh refining and improving our uh validating policies in this case. Uh but what Kivero helps us to enable

is to set up a strict boundary of what tenants inside of the cluster are allowed to perform and uh what accepted uh uh deployments or resources that they create uh falls within that uh boundary. But other than that they have pretty much yeah near near admin level uh access to their cluster to their part of the control plane and can create and spin up name spaces or

uh create a deployments either through a pipeline or through their own githubs implementation while always adhering to the uh boundaries we define with Kivero. Uh and in addition uh Kivera also helps us to give feedback to uh our tenants easily through the use of uh reporting uh the policy reporting uh that we can uh deliver to our tenants uh in through our internal developer portal as a

dashboard to report also to their part of the business like hey uh you have uh these uh violations of our uh policies and this is the recommendations you can follow on our confidence page for instance. like with examples of how to become compliant. Uh this this framework really gives a good good uh standard to uh apply and uh grow for our platform. But it comes with its

uh now let's say frustrations from developers who are facing these policy boundaries. Uh well you always as a platform engineer need to consider the desires of your tenants versus uh the security constraints you always want to have enforced for your platform. Uh so it is for us a very uh good learning opportunity to get into dialogue with our tennis like what they require to run their application

be it a common offtheshelf solution or an in-house uh uh app which developers know everything uh about uh well some silly uh stories that we have never heard fortunately uh but if someone wants to pull an unsafe image from Docker Hub uh well this was going to be blocked by Ephero and they can only make use of curated uh container registries that we allow within the bank.

Uh that can help tenants to understand uh which policies are uh enforced and why to uh show them what the necessity is uh at the bank to keep the uh standards and regulations enforced through the whole life cycle of building your application be it in development or production. So this is also a good way to educate tenants and to help them understand how they can build and

develop their applications in a more secure fashion. Uh having uh all of this in place um you can imagine okay what is the uh uh benefit or the result that we gained from offering a managed Kubernetes platform to our tenants. Uh well I can tell you it's has been quite a significant improvement over what uh what was done in the past because uh we've really saved a

lot of hours in uh delivering a production ready cluster from day one to our tenants. As you can see, it is uh reduced from uh 54 days in the past like close to half a year of work uh like a five day work week to uh half a day or four hours only of requesting a cluster, getting it approved, provisioning it and registering it within uh our

IT service management uh solution and then uh gaining the right access to the cluster and with all the permissions in place. This used to be done by a squad all by themselves who were running on uh Kubernetes in in whatever envir landing zone. Uh and that was uh well either very costly or uh insecure because they were not aware of all the security standard and having a

platform team in place definitely helped us at the bank to uh enforce all these standards and keep on improving it. So uh for you to uh see how we offer our uh our platform to our tenants, we have a small demo uh that we can uh show to you. Yeah, this is pre-recorded. So let's see. All right. So this is uh uh it's called Nexa. It's our

um uh what is it? >> internal development. >> Internal development portal. Um, and it's actually built on backstage. And this is for a tenant. You can request a And here I have the option for a open shift and ease. And I'm going to go ahead and press a. >> Yeah, go press it. Come on. >> Do it. >> Good. Okay. Right. So when we request a cluster,

I need to fill in some values. I need to fill in what my application name is. I need to fill in what my squad, who I am. I'm going to call this cluster CubeCon demo. Uh I can pick an Um also I can fill in a entry group name. So this is giving the um the access to which which team I should give access to. Um, furthermore,

I also need to fill in an SPN which is giving access to the pipelines. And then finally, I see an over uh overview and I'm going to go ahead and submit this request. So it's go screen. Then basically what happens in the background is a pipeline is triggered on our site and this is a pipeline that creates a a Terraform values file. So this is all the

values for our uh cluster and it also creates a task on our uh ashure devops portal so that we are notified. We also get an email but I didn't show it. And after that we also get a pipeline to actually start the creation of the cluster and it already started but I want some for ice principle right so I need to approve it and once that pipeline

is done running which takes around an hour. so then we can see I have it in Ashure and I can also go ahead and connect to and then I can also see that it started in Argo CD and that's how we offer Kubernetes to our tenants basically and Koshin will continue with some oh sorry some key >> yeah final key giveaways because uh we don't like to

uh take something from the others but give it to you as a yeah some of our uh lessons learned and experiences that you also might be able to uh take over into as your company. uh what uh what we considered for our choice of the architecture namespace as a service versus cluster as a service for us the uh priority of having high security from day one with

well minimal overhead on our end to uh provision and deliver the cluster versus the cost. Well, that uh that part helped us to choose like which direction to go uh and potentially in the future we can further uh improve things and develop it to perhaps go for namespace as a service with more uh cost efficiency. But this has helped us to make a start quickly and uh

see that we can uh yeah get a quick traction from our tenants. Uh second of all uh the uh tooling the right choosing the right tooling is very important uh for uh man managing the cluster and implementing the right automation. So it is very important to choose the right tools uh that can help you to uh deliver and uh maintain your clusters. Uh and uh last of

all is uh constraints and restrictions of your platform help you to create the right dialogue and the right conversations with your tenants to further learn and understand what the needs of your tenants are and perhaps also look into how you can uh uh learn and take the feedback of your tenants to improve the platform offering uh while uh adhering and keeping the security standards uh at the

same at the same level uh and of course us too we are not uh the only ones who've been working uh on this uh offering so uh we really like to give a great big shout out to all the uh colleagues and our team at Qub. So thank you everyone for uh well attending and listen to our talk and