Cloud Native Theater | Istio Day: Zero-Downtime Migration from ingress-nginx to Isti... Joe Abellard

In this talk, Joe, a senior software engineer at Bloomberg, discusses the execution of a zero downtime migration from the deprecated Ingress EngineX to a multicluster platform built on the Commada project. He begins by explaining the need for multicluster orchestration to achieve data center redundancy at scale and how this approach evolved at Bloomberg. The speaker details the architecture and features of the Commada platform, emphasizing its role in managing workloads across multiple clusters. He outlines strategies for preserving routing semantics during the migration process, including the use of a unified ingress endpoint to maintain traffic continuity. The session also addresses challenges in managing ingress traffic and observability through Prometheus, concluding with insights on the future integration of the Gateway API for enhanced capabilities.

Okay, welcome everyone. I'm Joe. I work as a senior software engineer at Bloomberg. Um, and in this talk, I'll walk you through how we executed a zero downtime migration from the now retired um Angress EngineX um to our multicluster platform built at top commada um at Bloomberg. Um this is a very timely talk because the ingress engineext repo actually got archived earlier today. So, it's officially dead.

This is our agenda for today. So I'll start by um talking about the need for multicluster orchestration at Bloomberg and how multicluster orchestration has evolved as a key enabler for achieving um data center redundancy at scale. Um then I'll talk about comma and how we built a fully managed platform at top kada to fulfill our use cases for multicluster orchestration. Um and at this point everyone should

have a very good understanding of our managed platform. So then I'll dive into um ingress traffic semantics our use cases for ingress traffic. Um then I'll talk about the migration strategies we explored as well as how we preserve routing semantics as we executed that migration. Um and lastly I'll talk about future um work ahead. So let's get started. So at Bloomberg many of our platform teams run

a top Kubernetes. Um so for a second let's just assume that this Kubernetes cluster is a cluster powering one of our platforms and then in a disaster scenario um it goes down at Bloomberg um data center redundancy to ensure continuity of service in these type of disaster scenarios is mission critical. So this is a huge problem that we have to solve. Um a common pattern for solving

these type of problems is by using multicluster orchestration. Um and the way that works is you start by building multiple clusters that are strategically placed across multiple data centers or failure domains. Um and once you have those clusters set up, you then need a multicluster orchestrator or orchestration layer that can make placement decisions across that fleet of clusters. Um and ideally to avail of the existing Kubernetes

tool chain that the central component in that orchestration layer should be the cube API server. Um, and in that control plane, you should also have a scheduler that at a bare minimum can make intelligent placement decisions across your fleet of clusters. An added bonus would be something like support for automated um, failover in a disaster scenario. Um, and at Bloomberg um, we've converged on the CNCF um,

commada project as our multicluster orchestrator. So I'll walk through how that works and how we've built a fully managed platform at top the comma project. So Kada is a CNCF project for um Kubernetes native multicluster management um that enables you to run your workloads across entire fleets of clusters. Um architecturally a comma setup looks very similar to a Kubernetes cluster setup that all of you are familiar

with. It's basically just a control plane that is joined to a set of clusters as opposed to worker nodes. Um and in that control plane um the central component um here labeled as comma API server is just the standard cube API server which enables seamless integration with the existing Kubernetes tool chain and instead of cubeuler you have comma scheduler responsible for making intelligent placement decisions um across

your fleet of clusters. Now today, if you want a Kubernetes cluster, most likely you're not going to do it the hard way because there's a lot of day one and day two app that comes with setting that up and maintaining it. You would most likely just get one from a managed service offering. Um, and the same is true for multicluster um orchestration and managing multicluster um infrastructure

that comes with even more day one and day two apps. Um and given that we have multiple teams at Bloomberg that want to use Kamada um and that we do not want every team to set up this uh multicluster infra um the hard way we decided to build a fully managed service top commada um to fulfill all all of our use cases for multicluster So over the

next few slides I'll take you over how we did that. Um we start with this standalone topology. Um and in this topology we first build what we call the host or management cluster. Um and then in that host or management cluster we installed the commada operator. Comma operator is one of the components from the commada project. Um and then we use that operator to set up a

base um comma control plane. At this point there's still more work to be done. So we have our own operator um the manage comma operator and that takes care of things like setting up a cluster registration subsystem um to um fully automate member cluster registration for our tenants. Uh but most relevant to this talk is um that it also does all of the plumbing to set up

ingress traffic to the comma API servers. Um and once all that plumbing is done then ingress engineext also running on those host clusters um will take care of the rest. Um at this point we have successfully set up uh an instance come instance for a tenant. So let's walk through what type of failure scenarios we can withstand with this topology. Um with the first one being lost

of a member cluster for that tenant. And if that were to happen, we can still maintain continuity of service as that tenant would still have another member cluster still up and running and workload previously running on the fail cluster could be migrated over to the other cluster um if needed. However, if we lose um the host cluster, then all of the managed services out. And to solve

that we have to make sure that everything including the managed service itself runs in HA which is why we settled on this HA topology. And in this topology instead of using just one host cluster we now have a cohort of host clusters that spend u multiple data centers. And then we take um each coma instance and stretch it across that cohort of host clusters. And that works

because all of the commander control plane components with the exception of the API server run with um leader election. Um we then integrate with our DNSbased um service discovery system to provide one unified ingress endpoint to that stretch control plane. Um and that works because we use the same CA umert um to provision the instance on both host clusters. Um and now in this topology we can

withstand um many different type of failure scenarios with the most severe of them being loss of um entire data center and even if that were to happen service as we would still have everything up and running in the other data center. Okay. So now that we all have a good understanding of how the platform works, I'm going to stop um diving more into ingress traffic semantics. Um

starting with how traffic flows to the edge of our host clusters. Um and again in this HA topology we have a cohort of host clusters that spin multiple data centers. Um and each of those host clusters will then install ingress engineext as well as a service discovery controller. Um and the ingress engineext controller will then start watching ingress resources to to dynamically program EngineX and the service

discovery controller also watching ingress resources will register those service um instances with our DNSbased service discovery system. Um and also in this topology we set things up so that um ingress traffic can either come directly through one of the host clusters or go via the unified ingress endpoint. um for HA compliance. Um and with this topology, if we were to query DNS for traffic going directly through

one of the host clusters, let's say the one in DC1, um we see that there's a few things happening. So the worker nodes for that host cluster are registered um with DNS and the um ingress domain for that host cluster is actually just a wild card came that points to that DNS entry. So at the end of the day we will get assuming that this host cluster

has two worker nodes we will get back um two IPs and traffic will be load balanced of course just two worker nodes and likewise for the cluster in data center 2 assuming that that cluster also has two worker nodes at the end of the day we'll get back two IPs and traffic will also be load balanced across those worker nodes um but if um traffic goes via

the unified ingress endpoint now we will get back um four IPs, two for each um cluster across our cohort of data Um and in the disaster scenario, if we were to lose one of those um host clusters, then instead of getting back four IPs, we will get back only two IPs and all traffic will go um via that host Okay, so that's for I just explained how

traffic is um flows um to the edge of our clusters. So now I'll start talking about um the use cases we have on the manage platform for ingress traffic with the obvious one being of course um managing traffic to the comma API servers and again commat API servers are just standard cube API servers and if you look at the annotations you'll notice that we set up we

set things up for an HTTPS backend with SSL pass to enabled and again traffic can either come um directly via one of the host clusters or via the unified ingress endpoint. So we have two host entries on here. a first class feature of the manage platform is observability. Um and to achieve that we have to script metrics from all the control pin components. Um however some of

those control pin components um such as the comma API server have protected metric endpoints which um introduces challenges. Um so uh to explain um how we solve those challenges I have to first start by providing a bit of context into how our host clusters are set up. So in addition to worker nodes or host clusters have what we call monitor nodes and those monitor nodes don't run

workloads instead they run a fake cublet and that fake cublet registers them with the cluster in addition to fake cublets they also run prometheus servers and those prometheus servers are responsible for scripting metrics from um targets in the cluster um and usually we will set up two monitor nodes in an HA group um where all replicas in that HA group script all of the same metrics for

the cluster. Um and although the fake cublet is used to register those monitor nodes in the cluster um those monitor nodes themselves are actually not part of the cluster's overlay network. Um so to script metrics um Prometheus running on those monitor nodes actually has to go through a systemd de manage engineext metrics proxy um to um uh get the metrics um and all of this infra is

set up out of the box when we build a new cluster but it's still insufficient for solving this um specific use case. So to do that we have to go a little bit further. Um and we do that by running an again I love engineext. We run an engineext metric proxy as part of each control plane. Um then we will annotate the API server pads with the

commonly used Prometheus um annotations as well as some extra annotations that we need um to configure Prometheus so that it knows how to discover um the metrics proxy. Um then at this point um we will use Prometheus Kubernetes service discovery as well as Prometheus reabeling configs to configure the Prometheus instances so that it knows how to reach um the metrics proxy to get the metrics and the

way it works is the Prometheus client will reach out to the engineext metrics proxy and then the engineext metrics proxy will reach out uh we'll reach the API API server pod by setting SNI to the incluster DNS name of the API server while um dialing the pod IP. Um the Prometheus client certificate is signed by the cluster CA. Um the engineext metrics proxy server is also signed

by the cluster CA and the metrics proxy has a client signed by that commada API comma control pin CA that is authorized to script the metrics. So all the traffic is authorized and encrypted end to end. And this is what the angress resource looks like for this use case. And again, we're setting setting things up for an HTTPS back end with SSL pass to. And for this

use case, traffic needs needs to go directly via the ingress for that specific cluster, which is why we only have one host entry. Okay, now that I've explained some of the use cases we have for ingress traffic, I'll talk about exactly how we executed the migration with zero downtime. And to achieve that we explored a few strategies. The first one we can call a side byside strategy

with a cutoff period. And again the way that works is we have multiple host clusters across our multiple data centers. Um and each of those host clusters we have engine X running on host port 443. Um then we will alongside engine X install onvoid on alternative HTT HTTP port 8443 um and then once that's up and running um traffic could either come via either port. So what

we'll do is then allow some cutoff period to reconfigure everything using the old port to use the new um 8443 port. And once that cut off period is done, we can just get rid of engine X. And at this point, the migration is completed and we have Envoy running on all the host clusters. This strategy works, but it comes with the downside of having to reconfigure everything

so that we use the new port as opposed to the old one. So let's see what we can do better. Um, this is the strategy we ended up settling down with. And this one we can call the fizz roll out by DC. Um and again we start with engine X running on host port 443 on all the host clusters and then we'll just take down engine X

on one side. Everything is still cool at this point because of the unified ingress endpoint. So traffic will just go to the other side. Um and then we will install onvoid on this side. So at this point we have Envoid running on hostport 443 on one side and EngineX running on host port 443 on the other side and traffic can go on either side and everything will

still work. Um and then we do the same thing on the other side. So we'll just take down engine X uh we'll bring we'll bring up envoy envoy and at this point um the migration is completed. Um for this migration we had a roll back strategy in place in case anything went wrong but it ended up going well so we ended up not using it. Okay. So

now let's talk about routing some intake. So we basically just set up the gateway um for envoy to listen on hostport 443 with TLS pathway enabled. Um and given that traffic can come in either via the unified ingress domain or the a specific cluster ingress domain. Um we have two host wild cards on here to allow for that. And we do the same thing on both host

clusters. Um and this is what things look like before and after. No more annotations. That's for the first use case. Um and this is for the other use case I I talked about. So let now let's talk about future work ahead. I know what a lot of you are thinking. So why did we not just move to gateway API as opposed to first taking a baby step

to isto? Um and the main reason for that is because the TLS route API that we need um for TLS pass through functionality um was still an experimental API at the time when we did this migration and it's kind of hard to justify doing a migration to an experimental API. um but recently I think about two weeks ago the TLS rod API graduated from the experimental channel

to the stable channel. Um so for future work we have to add support for that API to our service discovery controller and then we can migrate from ETO custom resources like virtual service um to using gateway tomorrow there'll be a very hot talk by one of the tenants actually running on that platform for how to run disaster resilient Trino on multicluster Kubernetes powered by Kada and um

Trino Gateway um and this is your days and there's a lot of parallels for how Trino gateway routes traffic routes traffic to Trino clusters is to how um eastto can be used to route traffic um internally inside your cluster to the different services. So I highly re recommend that all of you go and check it out and um I'll be hanging out at the Kada project booth.

So to learn more about this migration or the comma project or just multicluster orchestration in general, just stop by and I'll be more than happy to talk with you. Thank you.