No Shame, Just Pain: How We Migrated Away From Kubernete... Jannis Relakis & Michael Seiwald-McCarty

This talk focuses on the challenges and strategies of migrating from multiple Kubernetes versions to a more streamlined setup at Salonus. The speakers, Janisle and Michael, share their experience of consolidating six different Kubernetes flavors into three, tailored for each cloud provider, while ensuring zero downtime during the process. They provide insight into the complexities of their existing environment, which included 160 clusters across AWS, Azure, and GCP. The discussion covers the design of a target platform, the significance of uniformity, and key tools like a custom service mesh referred to as the 'wormhole' that facilitated seamless application migration. Additionally, they emphasize the importance of automation, structured migration plans, and lessons learned along their journey to improve efficiency and reduce technical debt.

Hello everyone and thank you for joining us in the session. Uh no shame just pain how we migrated away from Kubernetes 116 in brackets in 2025. So yeah um today it's going to be me Janisle. I'm a senior platform engineer at Salonus and today with me on stage is Michael. He's a staff platform engineer but also there are a few other people in the crowd that were

part of this project. So yeah, raise your hands, get the credit that you deserve. All right. Um, okay. So, we are Salonus. Uh, we are the company that helps other companies improve their business processes. Uh, we introduced the term process mining in 2011. And, uh, our product evolved into the process intelligence platform in 2023. And as you can see on the left part of the screen, we

have a bunch of renowned customers and also renowned partners. So why are we standing here? Why are we having this talk? Why am I talking to you? Um the reason is that recently I heard u an interesting stat. 70% of all large scale migrations fail. And in the words of XKCD, I couldn't describe it better so I let the professional do it here. um he's talking about

the proliferation of standards. There are 14 standards. That's ridiculous. We need to consolidate. We need to bring it down to one. What tends to happen is you end up with 15 standards. So, um we were in a similar situation. We had six Kubernetes flavors and we wanted to consolidate them down to three, one for each cloud provider. And the fear was that we going to end up

with nine. And um spoiler alert, that didn't happen. And that ties back to the why. I'm standing here to share how we avoided that from happening. I'm going to share the lessons learned and also the mistakes that we made along the way so that you can avoid them. So before we get into the details of the migration, we want to take a step back and um give

you a little introduction of our Kubernetes setup and also the history and how we accumulated so many flavors which then triggered the migration project in the first place. So in Solonus we have what we call environments which are basically full installations of the Salonus platform and one environment consists of multiple clusters and these environments are created based on certain criteria. One is tenency. So we have shared

environments where we have uh multiple clusters and customers sometimes hundreds of customers but we also have uh dedicated private uh environments for just a single customer. Furthermore, we run on multiple cloud providers depending on the customer preference. Uh so currently we run on AWS, Azure and GCP and data locality is of course also very important which is why we run in 22 different cloud regions right now

and all of this adds up to around about 160 clusters recently. Um now how did we accumulate that many flavors and and uh yeah so we started out in 2017. Initially we used cops uh to create our first environments uh staging and production environments on AWS. One year later we adopted Gardner. Uh we had the additional requirement to run in Azure and also back then Gardner was

really like an innovative concept which running Kubernetes inside Kubernetes and then have lightweight clusters. So that was good improvement. Uh in 2021 we adopted open shift. So we went more into an enterprise security focused uh direction. Also, this was the first time we had professional support for our Kubernetes clusters. And now, there's nothing wrong with any of these choices. Uh they they can be great. The problem

was more that we did not manage to consolidate them and uh we basically kept them around and we also kept some of them um we did not upgrade some of them. And this is because the company grew very fast. the platform team um was was busy with like market expansion and and yeah, we just didn't have enough time for the for the maintenance. Uh so finally in

2023 we decided we need to do all of this from scratch. We need completely new clusters and migrate everything over. Uh and this is when we started developing the target platform. So to reiterate, this is the the picture of our landscape before the migration. We have um yeah many different flavors, outdated versions. Making changes and innovating on on this is very hard because we often had to

do uh things multiple times. Uh there were of course security concerns and we were also completely out of support with a lot of our clusters. And what we wanted to achieve with the migration is a picture that looks more like this. So we have one flavor per cloud provider everything on the same version. Uh also these would behave very similar on a high level even though you

have EKS and AKS and so on. Um they are modern, secure and much easier to maintain than what what we had before. So now just a few words about the the target platform that we started building So our requirements back then were we still needed multi cloud support. So that was that was just a given requirement. We also wanted to make no application changes for this migration

or require no application changes. And some of our guiding principles were were um uniformity. So every cluster needs to look the same and and provides the same Kubernetes capabilities and automation. Of course the building blocks we already u mentioned it managed Kubernetes services. So we did not want to run our control planes ourselves anymore. So we adopted um EKS initially then AKS and and GKE were were

added over time and infrastructure as code. So everything is provisioned using uh very fine grained terraform modules and then we use teragrant on on top of them and some features that we just wanted to highlight here. Um we have we use argos application sets for cluster add-ons. um we gave a big priority to to cluster upgrades from the beginning. So since this started we were already through

seven upgrade cycles and we really want to have all our clusters on the same version. So no no no no surprises there. And a little bit later on we also added Carpenter to to improve the node management and and also help with costsaving efforts. Now this is sort of on a on a very high level what came out of this what we call internally a cluster fleet

um consists of three clusters. We have the main cluster where the majority of the workloads are running. We have one cluster dedicated for machine learning and AI workloads and one cluster dedicated for our query engine which is a a very central component to Solonus. And all of these clusters are connected through the same underlying shared network. And now speaking of networking, we want to now introduce uh

one of the components that helped us to migrate with without downtime, which is what we call the wormhole. So the wormhole is basically a crosscluster service mesh that we built ourselves. So why did we do this? We wanted to be able to migrate the workloads independently of each other. So it was not like one big bang migration with maybe a maintenance window. It was really like no

downtime and workloads can run for a certain time on the old cluster or the new cluster and they they can seamlessly talk to So any solutions that were already available for this maybe you think of selium or something we could not use because we had these old versions we have many different flavors. So this was just not possible and also this allowed us to customize it very

much to our own needs. How does the wormhole work? It's basically a set of envoy proxies with static configuration. It allows us to proxy HTTP traffic between the different clusters. And what we use here is um dynamic forward proxying. It's it's an enway feature where you basically you don't need to configure the the targets. It will just find them dynamically based on on DNS. And this is

what it looks like. So you have on each cluster you have an egress proxy which can simply um forward the the HTTP traffic to the other cluster and all of this is transparent to the application. So they don't need to know when they call service X they don't care if service X is running on the old or the new cluster. It's it's transparent to them. And you

have the ingress proxy on the other side which takes in the traffic. And now to really round this off, uh we want to go through a full migration example based on this wormhole. So initially we have application A running in the old cluster and we want to get it to to the new cluster. How do we get there with zero downtime? So and this is based on

a on a stateless application. So what we do is we simply redeploy the same application A on the new cluster. And now yeah it's it's running there. It's not doing much. It's not receiving any traffic. So the next step uh and where the wormhole comes in is we patch this service object this Kubernetes service uh we patch the selector and point it to the wormhole. So this

is a a zero downtime operation uh where simply the traffic now goes to the to the pot or to the service and then to the pots on the new cluster uh through the wormhole which is the the dashed line here. Now these um pots on the old cluster are are orphaned I mean and and eventually we can uh shut them down. And to complete the picture, this

of course also works the other way around. So application A needs to talk to application C, but this is not migrated yet. And again, the wormhole enable it enables it to talk to these applications. All right. So now we saw how the two clusters can communicate or the applications on the two clusters can communicate with each other. Uh let's have a look at how we actually migrated

the apps from the old cluster to the new and if we were able to treat them all the same way. Um just a side note here all the applications both the ones on the on the old cluster and the ones on the new cluster were able to talk to the data stores. So they were able to communicate with the relational database or with the object storage. All

right. So we have the applications and our goal was to basically classify them or group them into into certain groups based on how we are going to migrate them and based on their characteristics. So we figured out that one big bucket of applications was what we called stateless apps. And these apps you could just scale them up on the target cluster then cut over the traffic over

the wormhole as Michael just showed and then scale them down on the old cluster. Pretty straightforward. Then another bucket that we identified was singletons. These are applications that can only have one replica running either on the old cluster or on the new cluster but not simultaneously. And there the migration procedure was the other way around. First you scale down then you do the cut over and then

you scale up um which introduces a downtime but it's negligible if you do this in quick succession. And then there was a third big group which was the stateful apps and this included for instance our query engine or our ML workloads and some other applications. And for this one we needed to create bespoke migrations. The these needed like special one-off uh migration procedures. Okay. So now we

basically made sure that we know how to migrate each of our apps. Um, and just to give you a little bit of of an idea of what the distribution was, 90% of our apps were stateless. Approximately 1% these were legacy apps were singletons and 9% were stateful Okay, so now we know how to migrate the apps, but let's look at how to migrate an entire environment, right?

And for that we created what we called internally the master plan or dependency diagram basically uh which starts with um the thing at the bottom which is provisioning the new clusters. So this is the first step right and on the top is your target like tearing down the old and somewhere in between you have the workload migrations that I just showed you stateless stateful and so on.

Um and slowly but surely we filled in the gaps. So we made sure for instance that we do the um the or we we we understood that we needed to do the ingress cut over after we had migrated the singletons in the stateless which makes sense but we also had to do some things before we even migrated like migrating or setting up the wormhole or setting up

rabbit MQ federation so that the rabbit MQ messages could be federated between the clusters and some other stuff mostly infrastructure things but this completed the picture. Now we had like a a plan, a dependency diagram, a whole picture of how to migrate a single environment. And this unlocked a few things for us. It unlocked better scheduling. We had a complete picture. So we knew when need when

what needed to happen when. Um it also allowed us to parallelize things uh easier. If you have a plan, then you know how to parallelize activities. It also allowed us to optimize the whole procedure. We figured out that we had bottlenecks at certain points or we figured out that there were too many handovers between teams because not all of these steps were performed by the same team.

All right, so now we saw the macro picture. Let's zoom in into the stateless service migration. And um I said before scale up, cut over, scale down, but the reality is a little bit more complicated than that. And um you can see this uh in this slide. The first thing that we actually did was duplicate the customize overlays. So we use customize to describe the configuration of

our apps on the old and on the new cluster. But we wanted to be able to control these separately and independently from each other. That's why we duplicated the old overlay. And then the next step was adjusting it. H and this is why we needed two of them because they were in the end not identical. Um and why did we need to adjust them? Remember we are

moving from open shift um for instance to GKE AKS and EKS and for instance open shift specific things were not needed in the new system. So a shared volume or security context constraint these are things that don't exist on the new cluster. Therefore, they needed to be removed um and the overlay needed to be adjusted. Or similarly, um we are moving from 116 to 132, 133, 134.

Therefore, we needed to update the manifest so that they're compatible with this newer versions. Then we have the the the scale up, then we have the cut over, then we have the scale down and some other steps. For brevity, I've um I've only included these five. It's important to note here that all of these steps resulted in one or more PRs. Keep that in mind. All right.

So, let's do some math. If we have 10 PRs per app and we also have a 100 apps per environment approximately and we have 40 environments, then we have 40,000 PRs in total just to migrate all our apps. This is insane, right? I mean uh we would have uh if we did all of this manually we would probably be still migrating we would have caused 100 plus

incidents and this would have been very inconsistent. So um we would have accured a lot of technical debt if we did it that way. What was the solution to this? We built migration tooling. We built our CLI that allowed us to migrate these uh these apps automatically. But most importantly also in batches. So not one by one but in batches of 10 or 100 even in some

cases. And um this CLI was able to perform all the actions that I showed you earlier scaling up scaling down cutting over traffic and so on. Uh it went to the clusters or to the cloud providers cloud provider APIs or to Argo or wherever it needed to fetch information whenever it needed it. And the output was um uh the output was always PRs. So the output was

always reviewed by a human uh human in the loop. So that yeah we kept an eye on on the results and the result of all of this was that everything was documented because we had basically a list of PRs that uh were generated by this tool. It is uh repeatable. we can just um rerun the the same commands and it was also consistent because the tool is

just a program with a deterministic output. Therefore, we could rerun them and we would always get the same result and but most importantly it's much faster than doing it manually. So therefore um uh it was important that we introduce this tool. A last point that I want to make want to make here is creating automation also enabled us to um have more people perform these migrations. Initially

it was just a few specialists which was a bottleneck by itself but by having a CLI and by not having to do you know manual steps and following runbooks and one command here one command there we just had a tool that automated this all away which meant that we could democratize these migrations which was a huge relief for me personally. Um all right so now we talked

about the the how and we showed you the what the tooling was and everything. Let's um talk about the results what actually happened during this migration. So here's the timeline and uh you'll see that early 2023 the project started and we mainly focused on the design of the target platform and we also um started designing the migrations stateful stateless and the others that I showed you in

early 2024. We actually started with the migration itself. Um everything was pretty much manual. everything was very very slow and um we actually set a focus on EKS to try to limit the parameters of these Then around early 2024 um we started with AKS but you'll see the the the lines are still pretty flat. So this is when we realized that we really really needed to to

invest in automation. But another factor that made us very slow is what I like to call and internally we started calling the snowflakiness and this is what you can see here is the snowflakiness index. So we realized that within a provider let's take gardener AWS which was one of the old flavors that we had. Um even within that provider the flavor the the environments were different. So

we had for instance environments that were using linkerd but others that were usingto. We had uh environments that were hippa compliant so they had a different sort of hardening. We had environments that were on govcloud that were configured differently and the list goes on and on and on and on. So um as you can imagine this heterogeneity in the environment also slowed us down because we with

each new variation we needed to adjust our tooling we needed to adjust our migration procedures and this slows you down quite a lot. um when we had migrated a significant amount the snowflake in started receding and it's important to remember that this project was not only a modernization project but I think primarily it was a consolidation project that allowed us to have a much more uniform platform.

So now that we walked through this whole migration journey, we want to share some of our most important lessons that that we personally learned out of this. So first, don't change too many things at once. So we really consciously uh kept this sort of a lift and shift migration and not an application refactoring project. So um yeah, we we really kept the the scope um small. For

example, if an application used EFS, that might not have been ideal and we also addressed that separately, but it was not part of the migration. Um beware of scope creeps. So also during the migration, it would often happen to us that somebody comes to us and says, um you're already migrating. Can we not also add this change and this change and that? and we really had to

say no and protect the the timeline of the project. We already talked about snowflakes and I want to highlight here that um the key to scaling a migration project like this is repeatability and snowflakes are kind of the enemy of repeatability because then suddenly your um your proven procedure doesn't work anymore and you have to get back to the to the drawing board. Also when you do

this migration for multiple years even uh the the world does not stop. So other things inside the company are changing and we have to keep up with this changes. Uh just to give a few examples we introduced cargo recently and and selium. Some services were decommissioned so suddenly we did not need to migrate them anymore and new ones were added. So we constantly had to to to

to keep up to date with that. One of our most important lessons is this. Um so every migration had its its own learnings and we would every time improve our runbooks, improve our automations and so on so that the next migration would be a little bit faster, a little bit smoother and then overall we we would uh get to this uh velocity or increase our velocity minimize

handovers. So we already talked that there were many people many teams involved and this also made sense. So we had uh the domain experts that would design a certain migration step but then we tried to really keep these these uh steps sort of self-service so that other teams can execute them and they did not have to wait for a certain team which might be uh busy with

with with other priorities. And last but not least uh try to not cause the same incidents twice. So when you touch every single application in every single cluster, of course, uh there's a lot of potential for things to go wrong and sometimes they did. But um we we really um focused on on making it more reliable over time. And um one thing that that uh helped us

immensely were service level objectives with this. And yeah, so now we we hope that our migration journey and our lessons can help you in your own migration projects. Uh thanks a lot for uh being here with us and we are more than happy to take some questions. Yeah, there's a microphone here in the front. Thank you very much. My favorite talk of the day so far. Uh

I have a question. So you still need to tear down the wormhole. Was there a disruption at that point when you need to tear down the old cluster and change the traffic pattern? >> No, at that point everything was already migrated on the new cluster and also the ingress was migrated on the new cluster. So there was basically no traffic on the >> Was there um um

a performance penalty while the wormhole was working? >> Um not one that we noticed to be honest. I don't think we ever investigated it enough to figure that out because but we did have incidents where we managed to make the wormhole collapse and especially there were cases where both sides were pointing at the wormhole which was a mistake of the configuration that should have never happened. But

basically what would happen is the request would ping pong. So they would go through the wormhole to the one side to the next side and that basically exploded the wormhole at some point. And um and while this was if that happened and that did happen mid migration which means that some services were still on the old cluster and some were on the new that meant that the

connectivity was interrupted and that also caused a few incidents. >> Thank you very much. >> Yeah you're welcome. please come forward. >> Hey there, thanks uh for sharing your experience. I was wondering if you guys have considered to uh open source that migration tool. Looks pretty cool. >> I I mean we could I'm afraid it's just too specific. It's too specific to our the the the way

that our repositories are set up and the way that we designed the migration, but I don't think that >> there is anything we could certainly publish also the patterns and like in a blog post. I the tooling I mean nowadays pretty easy to to implement but as Yiani said it's very specific to Solonus. >> But yeah, we can look into that definitely if if there's interest. >>

Absolutely. Yeah. >> Yeah. Cool. Thank you. >> What happened with gardener and what are you using today to manage the fleet of the clusters on the different hyperscalers? >> Um pretty much uh git repository which with has all our um cluster fleet definitions uh and it's kind of our single source of truth where we then generate all the infrastructure as code for the for the fleets. Uh

but we don't have some like a control plane on top of that manages all the clusters. We we don't have that currently. >> And and what happened with gardener? >> Gardener was one of the cluster flavors we decommissioned. So we moved away from gardener. We migrated. Sorry. >> Why? >> Why? Because we maneuvered ourselves in a position where we couldn't upgrade anymore. and a gardener. The last

gardener clusters that we had were in a very very old version. And you saw the amount of clusters we had. If we were to migrate them all version by version, which is necessary in most cases, this would probably have taken us longer than doing what we did. the EP migrations and everything else was also managed with Terraform and um uh let's say as code uh process >>

uh the the IP migration >> see yes >> yeah yeah we we also had the tooling that allowed us to do that actually yes >> okay thank you we are working on a migration from 1.12 to current on prem So we don't have all the tooling to change GPS and so on. >> Well, the author of the person of the tool is sitting right there with the

red hoodie. You can ask him more questions about that. >> The ideas we are planning are the same as you did. So >> yeah, we're collecting old versions. There was somebody that came up and said we have 118, you have 112. I'm sure >> it's a common problem. >> Yeah. Yeah, it seems to be. Thank you. >> You're welcome. Thank you. >> Thank you for the talk.

It was very fun. Uh I'm curious uh every kind of migration like that as done in multiple steps probably has a certain level of error >> happening at every uh step. How did you error handle basically how did you revert when needed or in ensure that there weren't partial migrations applied? >> Yeah. Um so we always try to fix forward right but our runbooks uh also included

roll back procedures and this was for every major migration the case. Um so we thought ahead and we were like what happens if we can't and so and that happened in a few occasions right we we did break and especially in the beginning we did break environments so we did roll back but we tried to also automate that so in order in order to get out of

that situation quickly or as quickly as possible that's actually exactly the core of the question because I was thinking that a system like temporal for example or something that allows you some job orchestration that handles uh erroring at every step might work pretty well with this like >> Yeah. Yeah, that makes sense. Maybe for the next migration then we can use something like that. >> Fingers crossed

you don't have to. >> Thank you. >> Uh oh. Hi. >> Uh so you gave a few examples for snowflake clusters. >> Yeah. like for example running on GC cloud um having the need for heaper compliance and so on. I'm assuming you still need those after the migration. So how how did that migration then help you to get rid of those snowflakes? >> Um it it didn't

really would. So, so to take HIPPA as an example, we really always aim to make the standard environment already HIPPA compliant and then all clusters still look the same and the environments that don't need HIPPA, they are still theoretically hippo compliant even if they don't need to be. So we if we can we always try to inv uh to to avoid such snowflakes now. >> Yeah. Okay.

for for that it makes sense but like um golf cloud is still an additional flavor right or like I don't know other constraints >> yes you're right go cloud is different but still we try to apply the same principles even in the golf cloud so that the the differences that we do have between the golf cloud and the non-golf cloud are as minimal as possible which was

not the case before kind of like the golf cloud setup was a complete unique thing by its own >> yeah that makes sense And now after you've migrated, how do you ensure that there's no like drift between all the stuff that you've now set up very nicely? Like usually people I don't know use some kind of operator to keep all that stuff uh in sync and so

on. Like now what happens if people like reconfigure their fleets or whatever in a different way? >> I mean we centrally manage the fleet. So it's basically the the Kubernetes platform team which is uh responsible for that and yeah everything has to go through through infrastructurees code and through git. So uh generally there are no like manual manual changes in the and everything is using a template

that has feature flags for different environments. So that is the way that we're trying to like centralize the configuration and make sure that everything stays the same. So basically the same thing is rolled out everywhere. Just some features are enabled in some of the clusters and some features aren't when we don't need them. Thanks. You're welcome. Cool. Are there any more questions? All right, let's go for

lunch. And also rate and leave feedback, please.