Ghost in the Platform: How the Dutch Tax Authority Built a Servi... Jerry van Hulst, & Marcel Kerker

This talk delves into the development of the Coastal Platform by the Dutch Tax Authority, known as the Belastingdienst. The speaker outlines the journey that began in 2017, focusing on empowering developers with a container hosting platform based on Kubernetes. They discuss the initial successes with high developer velocity and the subsequent challenges faced, including cognitive load and lack of standardization among diverse teams. To address these issues, they introduced an onboarding process involving a detailed intake questionnaire and the Project as a Service (PaaS), which allows DevOps teams to create complete Kubernetes environments through a single YAML request. The presentation also highlights their ongoing efforts to foster community practices, workshops, and documentation to guide developers in utilizing the platform effectively.

Welcome our presentation about Coastal Platform. But what amazing people here. A lot of people. Do you know that you really are in the same room or the correct room because maybe in other room? No? Okay. Um we talk about our Coastal Platform. We start the journey. Jerry, what's about it? Well, let's start with the beginning. Uh we are from the Dutch Tax Authority or as the Dutch

people might know us as the Belastingdienst, I will refer to us as the Belastingdienst from here. Uh we are part of the Dutch government. We are responsible for the collection of taxes, payment of benefits, and enforcement of customs. We are what some of you may know as the most hated company in the Netherlands. For such a hated company, we do have a lot of employees, over 28,000

people spread over the Netherlands, among which are 4,000 IT personnel. Last year, we collected over 300 billion euros in taxes. That is taxes from uh citizens, businesses, tourists. If you are in the Netherlands, you probably came across us. We have a vast IT landscape with a rich history spanning back over 60 years. But we're going to be focusing on the last 10. We only have 30 minutes.

My name is Jerry van Hulst. I am the product owner for the container hosting platform team. Yes, and my name is Marco Kerker. I am a a consultant at HCS company and also part of the platform team. And also I am a CNCF ambassador. So, um what was what is our mission? Our mission sent here is to empower the Belastingdienst, the tax office, uh, with a platform

and tools to take full ownership of a digital future. This mission is, uh, wrote by us, our product owner manager. And, um, we talk in this talk about how how is our journey. Yuri, what's our journey? Our journey starts in about 2017, where a group of engineers and new technology called Kubernetes and a vision came together. They wanted to create a platform with one thing in mind,

and that is maximum developer autonomy. We would go to that build the platform, they would run it. We started by building and automating the infrastructure, and by doing so, we created the foundation for what is still to this day our container hosting platform. The first years were focused on building the platform, and in about 2019, the customer started joining the platform. Really our early adopter phase, and

everything was going absolutely amazing. Enthusiasm was on high, adoption was growing, and developer velocity was skyrocketing. During the same time, we also founded our enablement team, where Marcel is part of. But, not everything was great. While the platform was in a great space infrastructurally, after the early adopter phase, developers wanting to land on the platform started to struggle. They started to struggle with high cognitive load. It

was very difficult for engineers that had never worked with OpenShift, never worked with Kubernetes, to all of a sudden be required to know our back, Argo CD, network policies. At the same time, developer standardization was nothing not not something we had on the platform. Platform engineers on clusters had to do the same thing. They didn't have any documentation on how to do it. Multiple people were doing

the same thing in a different way. This was painful for some developers and it really stagnated platform growth. At the same time, we also saw that lead times for new teams on the platform was really starting to drag out. What was first landing on the platform starting and deploying your first application in a matter of hours started turning weeks or months from provisioning the cluster to onboarding

your application on the cluster to learning all the new tools. Enablement simply couldn't keep up with demand. This brings us to our problem. How do we reduce the cognitive load for developers and at the same time, how do we increase standardization to speed up those workflows? Yes. So, we start our onboarding journey. Uh first, uh DevOps teams uh must prepare to start uh for our platform. They

They must uh For example, they must uh have an software architecture. They must uh have also um For For example, they must re-platform uh their application. For example, uh a lot of application are used with WebSphere uh WebSphere Liberty. They must turn it over to uh Open Liberty. Or maybe they have a monolith application, they make it micro microservices. Um and we also have a learn uh

platform. They can do a proof of that. Um after that, they must request an intake. Um The We have a intake with a with a lot of questionnaire, almost uh about 50 questions. Questions questions about uh um They have if if they're used in health checks or readiness checks or what about kind of database they are using or what kind of logging uh can application scale or

not. we also in that intake we uh we we talk about our uh tool chains about uh we explain about Argo CD with uh we we explain about uh Tekton and so on. Um and after that we say a go or no-go for the application. They go can go to the non-production environment. on the non-production environment uh after a while they can uh ask a um pass.

Later on we talk about our pass, our project as a service. Um and um and platform engineer platform uh team makes a harbor for it and they can start uh on the platform. Um after a while they go to the pro- production environment, they also must have a a questionary uh lesser questionaries uh questionaries like have they on board on the open uh of the operation bridge

or are they on board on the sock? Um after the a while they can go to the production. Um in the we all ask ask them about uh a moment. we we all ask them about our manifesto. Our our manifesto um is uh uh about 70 points that we they must uh accept. Um For example, if uh if the application is restartable, so we update the application,

we update the platform uh on the while. They must know that the application that the pod can restart from one node to another node. and also one more thing is that I must update the that I never update the container. We If you want a new container, they must redeploy a new one on the platform. And we also ask them if every every environment is the same.

So, test, production, or must be the same. And developers teams are responsible for their own environment. So, don't call us if there is some problem with the application. But our our approach is that we don't don't be that we don't be the police. We just help the customers with their Marcel mentioned earlier the pass operator project as a service. We asked ourselves, what if we could give

DevOps teams a way to create a complete containerized environment with just one request? Well, how do you do that? You build an operator. It's Kubernetes after all. Project as a service is our answer to this question. Our fully open source solution to giving developers a full Kubernetes environment with just one YAML request. As I said, fully open source, a full Kubernetes environment. GitOps as its interface, just

create a pull request, get it approved, and you can start on It's really designed to hit the ground running. Developers should not have to install Argo CD, Keycloak, Tekton. It's all pre-configured, documented, and ready to go for real-world applications. It's everything you need in one YAML. One YAML to rule them all, as you can say. It's everything from a project name to R back, who is your

admin group, namespaces, resource quota, most importantly, the capabilities. Because PaaS is not just a namespace. It's so much more. Pre-configured capabilities ready for workloads such as Argo CD, Keycloak, Grafana, but also the opportunity to build custom capabilities for your applications. These are all automatically provisioned, documented, and ready for use. Developers don't need to install or configure anything. Just go to your PaaS, fill in I want Argo

CD, I want Grafana, maybe you want Keycloak. Just put it in there, provide the uh uh the the secrets you want, ready to go. Marcel, can you show us how that works? The DevOps teams they must make a fork for from our Git. make a their own YAML. They can add different namespaces or they can add and they can ask for about the capabilities we they want

Argo CD, Tekton, and so on. Um if they are ready, then they might they make a pull request to our Git. And then the the YAML is is sent in our Git. Then some magic happens because then we have the validator. The validator uh, checks about a lot of things. Um, it checks if the Git uh, push pull request is a YAML is a kind of YAML.

It checks if the the YAML is valid. Is if the name is correct or that it it check the secrets. Uh, it also checks if the L if the if there's any L query is the L L query right. after the pull request, validate has been done, then sometimes it can be automatically, uh, approved. But, if it is the first time, then two two person would approve

the pull request. And also, um, if um, if somebody, uh, have a new pull request, but have changed some quota, uh, we want to check the quota of on on our cluster, so that must also approve by, uh, some people. Yeah, and that's mainly because we have limited infrastructure and by kind of gatekeeping capacity on the clusters, we prevent people from oversizing their environments just because they're

used to oversizing. after that, uh, if it's approved, uh, automatically, then it merge to our, uh, Git. And then some magic happens. Uh, our R2D sees this, uh, Git repo. He pushes it uh, on he deployed it on our cluster and then everything what is described in the YAML is, uh, is going on of make on our platform. So, then after Well, it takes 2 minutes, I

think. Then, uh, our R2D is a tecton as a uh, all the capabilities we want is on As I mentioned, PaaS is open source. We are firm believers in open source. We massively use open source, the CNCF landscape, of course. Uh we want to give back. So, the PaaS operator is on GitHub. You can follow the link or scan the QR code. They make no pictures of

us, huh? They're allowed. Just blur him. Don't worry, we'll put the QR code back on the screen at the end of the presentation. Also, we have have our start. We call it backs built by It's made by Backstage. It is an ultimate method starting point of our CI/CD. Um it's a powerful Backstage template. Uh in this template, we create complete CI CI/CD CI/CD pipe pipeline uh with

core tools like Tekton, Argo CD, and so on. Um but it's not only used for a cloud-native application, you can also use it for not cloud-aware applications. what I get out of the box. our starter makes uh different repos. It makes a test repo, uh a Java repo, code repo. It makes Tekton repo and uh a deploy repo. Um for example, in the Tekton repo, there our

starter also make uh the pipelines, the pipeline runs, and triggers. So, you have in the in the Tekton you make tasks about SonarQube or uh Triffy, about the build task. Everything is right on in the Tekton uh Git repo after the starter has finished. of course, it makes uh also all our manifests. We uh you need to do the pilot applications. You put that in the in

the deploy repo. And also we have a August today repo and inside that you have all the applications you need for deploy automatically with August your application. And even more, it make also documentations. But more important, it make also our pass. The pass repo is already ready after the Tekton after the pipeline is finished and it also make a real pull request to our repo. Let's show

some hands. Who has seen The Wizard of Oz? You're probably familiar with the yellow brick road. Well, this is similar. This is our golden path. Our opinionated guidance, our way of telling developers this is how you should use the platform. We want to stop people from having to really reinvent the wheel using these white papers, recipes, and ready-to-go solutions. They are split over the application life cycle

starting with day zero, the design phase. Here are strategic architecture and security decisions. Why are we building this application? How are we going to build this application? We offer white papers. We offer guidance for this. Then day one, the deployment of your application, implementing GitOps, and using your pass. We describe how developers should deploy their All the way through day two, operating. Maintaining your application, operating on

OpenShift platform. How should developers scale their applications? White papers about runtime security, right sizing, incident management. It's kind of what we offer in in the And we constantly keep updating these. They're still How many are there now? Over 50, I think. Yes. Yeah, over 50 white papers as of right now. Constantly updated, constant new ones being made. How we empower those 99 plus teams. also a cup

Community of practice. Yeah, community They're mostly our lead developers community of practice. They learn from each other. For example, if one team have a problem, maybe the problem is already solved in the other team. They can connect to each other and they can fast forward with their own problem. We also have a container user group. This is where we as platform team told our our customers about

new releases. We give them most we give presentations. For example, if a if if a new feature we we have made, we expose that to our customers. We have also have container days. The next our next container days is the 7th of April of this year. We talk about we have workshops about Devops. We have workshop about OpenShift itself. We have an external uh speaker from Red

Hat. He told something about test containers this this year. And so on. We do it twice a week twice in the twice a week twice a year. That'd be often. Yeah, that's too often. Yes. And we have a lot of self-paced workshops. We have Tekton workshops about uh with basic uh medium and advanced. we have architecture detectors, we have customized detectors, we have no factor customized uh

all kind of customer all all kind of uh uh self-paced uh courses. Um Uh and of course uh we also point uh our customers to external uh courses uh from Red Hat DO1 uh 88 or uh uh CKA from uh Linux Foundation. and we have one of our accelerator um program. We call that uh gas pedal. Uh in one day we take uh our the our team

take uh take a customer from one day they can build the application, they can deploy the application their own application from one day. They learn a lot of uh that that they own application on the so, what we have now on our platform We have more than 80 teams of our and more than 160 applications. Um of course we don't do it with two of us. We

uh do it with more than 50 40 peoples and most of we is sitting here in here in the in the room. Yes. Yeah. Over 80 teams, 160 applications in production right now. that's not just one OpenShift cluster. We have 78 OpenShift clusters as of right now with many more on the way. On these OpenShift clusters we have 227-ish development passing and 125 It's a lot of

name spaces. They're spread over 78 OpenShift clusters running over 900 worker nodes. Pretty bizarre if you look at this graph. I wanted to show this graph. I made this a little while ago to show platform development over the years. We started measuring the amount of clusters in this graph in 2021. We were already running OpenShift for a couple years then. But it was the furthest I could

go. As you can see, platform growth was very stable up until about November 2023. I arrived at that day. That's correct, Marcel. Everything changed when you arrived. With you, a lot more enablement folks arrived. We really started to take adoption more seriously. Not just build infrastructure, which we still do, we still build great infrastructure, but also focus more on getting customers, getting developers onto the platform. As

you can see, the last years we grew almost exponentially to almost 80 clusters right now. Projections going to over 100 clusters this quarter. So, what's next, Marcel? Oh, yes. Um yes, we want to have more integration with backstage. We We want to expand our our starter our CD uh uh starter. Um oh, we didn't all talk about our golden path. We have more white papers to make

uh to to have the that to our customers. Um we also looking at AI. For example, we now have a POC with uh we have we're using uh chat ops. Uh and our AI is looking at if somebody ask a question, the our AI looks maybe uh he he looks at our documentation and uh make make the the the answer of that question, put it uh back

in uh in our GitOps. Uh also, we are looking at we make uh the customer make some tickets uh uh in the in the service desk. We will also to automate that. Maybe can our our AI answer the question right away. Uh um Yeah, right away. Um and of course, we our users are the important thing that our platform exists. Uh this if they want new features,

they can ask also us and then we can look if we build that new features for them. I'm glad you brought up AI because it wouldn't be a tech conference if you didn't mention it at least once. So, we're reaching the end of this talk, but before we end things, I would like to have a look back at some of the lessons we learned along the way.

The first one I could come up with is that a platform is more than just infrastructure. It's an ecosystem of infrastructure, enablement, and developers getting together and really creating a a platform, not just When you move to scale, standardization is very important. Developers don't need 10 different ways to do something a little bit. They want one way to do something good. lastly, with our support, we build

for self-sufficiency. We want to help people and empower them to really take off and use the platform. Yes. Not just come back for every problem, make them capable of fixing it themselves. Thank you. As promised, the PAS Operator readily available on GitHub for everyone. If there are any questions, we will be here after the presentation. All of our technical guys are here as well. So, if it's

a technical question, please ask