Disaster Resilient Trino on Kubernetes: Multi-Cluster Setup With Karma... Sung Yun & Antoine Marthey

In this session, the speakers discuss the evolution of their Managed Trino platform designed for high availability and disaster resilience. Initially shared at KubeCon, the previous single cluster architecture faced challenges as data governance became less of a concern while scaling the system became critical. They explain how they transitioned to a multi-cluster deployment model, employing Karmada for orchestration and a Trino Gateway for query routing, resulting in a unified endpoint for users. This new architecture allows service owners and query users to engage with the system without being burdened by the complexities of high availability. The speakers delve into technical implementations of the Trino API server, Karmada, and the Trino Gateway, highlighting how failures are managed seamlessly. They conclude with future plans for further enhancements to improve resource allocation and fault tolerance.

All right. Hello everyone. I'm Sung and I lead the Managed Trino and Iceberg Platform Engineering team at Bloomberg. Okay. Um I'm Antoine and I'm a engineer in the Managed uh Trino and Iceberg team uh at All right. Nice to meet you all. Thanks Thank you all for coming at this late hour today. All right. So, last year at KubeCon, we shared how we built a secure governed

Trino platform on Kubernetes by leveraging Open Policy Agent, a CNCF project. Over the past year, as adoption grew, the next set of challenges became very clear. Data governance was no longer our bottleneck. The real challenge had shifted to building a highly available data platform on top of our Trino clusters. That shift pushed us to evolve the platform from a single cluster governed uh governance-focused architecture to a

multi-cluster deployment-based design where high availability and data and disaster resilience are first-class concerns. So, today I'll set the context for that evolution and Antoine will walk us through how we built uh and designed this next-generation Trino platform. Our Managed Trino platform serves two primary personas. On one side, we have the Trino service owners, which is typically an engineering team that is defining and managing a Trino service,

configuring the catalogs that they want to attach, and the resource limits they want to launch the pods with. On the other side, we have the query user, or the analysts, engineers, or applications that connect to a query endpoint and submit SQL queries. For both personas, the expectations are Trino service owners expect predictable life cycle management, and query users expect a reliable query endpoint. In our previous architecture,

meeting those expectations required those personas to coordinate those failures. The core challenge with Trino is that queries are stateful and tied to a specific coordinator. If that coordinator fails, the query doesn't transparently fail over. It is simply lost. So, availability isn't something we can make perfect within a Trino system. But, what we can do is to make failures predictable and make recovery and failover of a responsibility

of the platform instead of leaving it to the users. Let's briefly recap the single cluster architecture we shared last year. On a high level, Trino service owners express their intent to deploy Trino onto a cluster through an API. And within our platform, that intent is represented at two levels. The Trino service owners interact with a simplified abstraction called Trino service, which captures high-level intent like the catalogs

you want to attach, the resource configurations you And the platform operator materializes that into the actual runtime, coordinator, workers, and networking configurations through a lower-level CRD called the Trino deployment. So, within our model, the platform owns the deployment complexity, while users interact with a simplified API. Once the Trino deployment is running, a query user connects to a single query endpoint exposed by the cluster. Every query request

is evaluated by an OPA server running inside the cluster, which retrieves policy bundles from a central policy store to authorize access. From a security perspective, this model works very well. But in this model, the platform did not yet factor in availability and resilience into the abstraction. Now, we'll see what this means for the two personas. This means that when the cluster is unavailable, whether due to infrastructure

issues or maintenance, the entire data platform was impacted. And that's a challenge that needs to be addressed when you start relying on Trino for production critical workloads. It's important to note that even in this uh phase of architecture, we did provide clusters in two availability zones. So, Trino service owners, if they really wanted to, could deploy their Trino services in multiple zones if they wanted disaster resilience.

That also meant that the query users would be able to continue submitting queries against the Trino pod, even if one of those zones were unavailable. Unfortunately, this meant that the responsibility of achieving high availability and disaster resilience were pushed to these personas. And this is because when a Trino service is reconciled, it exposed a zone-specific endpoint. You can see here that the endpoint is returned as return

returned as tied to the specific zone. So, if a service owner deployed into two separate zones, they would retrieve two separate endpoints. And that meant that DR and HA existed if Trino service owners and query users implemented it in their own applications themselves. So, instead, we'd want a Trino service status that looks a lot more like this, which returns a unified Trino endpoint regardless of which zone

the actual underlying Trino runtimes are launched into. And that meant that our architecture had to evolve. We wanted to evolve from an architecture like this, where each persona operated against zone-specific APIs, right? Into one that conceptually looked like where they interact with a single unified Trino API. And under the hood, multiple availability zones still exist, but failover coordination and high availability should be handled by the platform

and not by these users And in this state, multi-cluster resilience becomes an implementation detail that is fully owned by the So, how did we actually build this? Well, as the title of our talk suggests, with Karmada and with Trino Gateway. Now that we understand the context behind the drivers that actually prompted our platform to evolve, Antoine will now walk through how we designed and built a platform

that abstracts away availability zones and provides high availability and disaster resilience by default. Thank you, Sung. Um okay, so Sung walked you through the problems um tied to a single cluster infrastructure. Uh now let me walk you through how to remedy to this. So, we introduced the Trino API server. So, the Trino API server is a stretched CD backed API that served as the single source of

truth for all the Trino resources. When a user creates a Trino service on our new federated endpoint, the service provider routes the request to one of our cluster. Since they now share the same stretched CD backend, both cluster are aware of the newly created resources. So, we run multiple replicas of our Trino service controller uh for high availability. But only one of them actively reconcile at any

time through Kubernetes leader election mechanism. So, if a leader goes down, um another replica picks up the lease and continues. So, we achieve without duplicate work. Our controller will then like create what we call a Trino deployment CR. And we need to deploy it down to our stack into our Kubernetes cluster that physically runs the Trino pod. And so, to do this, we rely on Karmada. Um

Karmada is an open-source Kubernetes management system that enables you to run your cloud-native application across multiple or cloud without change to your application. Um Karmada is also a CNCF incubation project uh with maintainer from a number of organization uh including Bloomberg. Um so, last year at KubeCon China, um we presented how we use Karmada across the organization. And our Trino platform is one of the key adopter

of this technology. Okay. So, on this slide, um we can see that the Trino API is submit the Trino deployment into Karmada. After defining what the propagation policies are for this Trino deployment CR, Karmada will submit the Trino deployment into the member cluster in order to build the Trino pods. Karmada continuously monitor the state of this of those Trino deployments. And if a Trino deployment in a

member cluster becomes unhealthy, whether it's due to node failures, resource pressure, or any other issue, the controller detects and attempt to recreate the Trino pods. So, Karmada is really the orchestration bridge between our declarative Trino API layer used by the user and the actual multi-cluster one time that runs the So, here's an example of a propagation policy. So, it's a Karmada resource that tells the Karmada control

plane how to distribute Kubernetes resources to member cluster. think of it as like routing rules for multi cluster deployment. So, in that spec we are distributing Trino deployments, CRs, and our approach is to duplicate it across all member cluster. So, all member cluster run Trino pods. So, one important piece of our Karmada integration is uh how we aggregate the status of each cluster. For this, we use

the resource interpreter customization. So, we create a Lua script that teach Karmada how to aggregate the status from the different Trino deployment across Okay, so to sum up Sorry, to sum up the tree the Trino service owner work workflow. So, the user creates a Trino service into the Trino's and submits to the Our controller creates a Trino deployment and submitted to Karmada. And then Karmada fans out

the Trino deployment to the member clusters in order to create the Trino pod. Okay, so now we have Trino pods on each data center. And here comes the query users. Uh and they want to query these clusters to access data. And so for this, this is where we introduced the Trino Gateway. This is how users actually connect and run So Trino Gateway is a load balancer, proxy

server, uh and configurable routing gateway for multiple Trino So it's backed by a database that stores the key information that the gateway needs. Um it doesn't store data, it just store configurations. The Trino Gateway is also exposed through a feeder endpoint, similar to the submission endpoint. Our service provider routes the request to one of the data center Trino Gateway The Trino Gateway pods then intelligently routes the

request to one of the Trino pods. So under the hood, the Trino Gateway handles dynamic back end management. So as Trino uh pods come and go, scale up, scale down, um disaster happen, cluster get upgraded, um the gateway automatically registers and deregisters them. All right, so that that's a great feature. But so how do we integrate uh the Trino Gateway to our stack? And how do we

register Trino services to the Trino Gateway? So when a Trino service is created and its pod uh come up healthy, the Trino service will create a Trino registration CR. This tells the gateway, "Hey, this I'm a new back end and here's my address." So this registration will be stored in the back end database that configures the gateway. So, the Trino gateway pod can access it and start

routing queries to it. All right, so now that's the point of this talk. It's like let's walk through some component failures one by one and see how the system responds to them. So, if there is a gateway failure in one of our data center, our service discovery detect that the gateway in data center two is unhealthy and start routing traffic only to data center one. An important

An important things to see here is that the surviving gateway in data center one can still dispatch queries um to the Trino pod in data center two. Only the gateway pod in data center two is down. So, we only lost the gateway but not the compute behind it. So, now let's say that a Trino pod fails in one of our data center. Now, the Trino gateway detects

it detects that the data center one um pod is unresponsive and automatically routes query um to the Trino pod in the data Once the cluster in data center two recovers, the Trino uh two recovers, the gateway rediscover it and resume distributing queries across both data centers. Okay, so now let's say there is um a Trino API server failure or even worse, like let's say the whole infrastructure

is down, everything burned. Um It's not ideal. but what we're going to do is that we're going to bring back the nodes, we're going to bring back all the pods up, and because the underlying data lives in the external stretch at CD which is independent of of our API server, no state is lost. The API reconnects to at CD, our controller resyncs all the resources, and the

system resumes operation as if nothing happened. So, what are the key benefits for the user? Um for Trino service owner, um they can deploy a single simple manifest definition, and they can do it against um a unified Trino API endpoint. The Trino services automatically recover from cluster failures, and the state is persisted in an external stretch at CD service for rapid recovery. For query users, they also

get a single endpoint um via the Trino gateway. And this endpoint is highly available since query are automatically routed um to the healthy clusters. All right, so we still This is like our our current release, but we still have like a lot of work ahead of us, and and we're planning to extend our dual DC architecture to additional data center uh in order to increase fault tolerance

and uh geographic distribution. we're also planning uh to work on optimizing our resource allocation. So, Trino worker pod are statically defined uh today via an algorithm that we built. Um so, we are working towards uh HPA driven uh scaling that just workers replica based on query load uh query load, CPU utilization, um JVM metrics uh allowing cluster to scale up uh during peak demand and to reclaim

resources during idle period. and that's a discussion we just had, but we also want to start logging some queries uh via the event listeners and to capture and persist user uh query metadata. And lastly, we want to work on um resource group enforcement control, query concurrency limits, and queuing behavior. so, the the idea behind this is to prevent a single heavy query to monopolizing the the cluster

resources. Um yep, that's it for our talk. Uh happy to answer any questions you guys have. Sure. Yeah. Can you speak here? Thank you for the presentation. Could you please uh tell how do you allow updates? Uh I mean, like without down times. Do you update firstly one cluster than another one? And uh second question I'd like to ask uh why do you deploy Trino gateway in

each cluster? Why not to have uh the one Trino gateway? Yeah, that's such a good question. Um I'm so glad that we're thinking along the same lines with the community here. with this current release, we actually leave the uh responsibility of managing blue and green deployment entirely on them. So, there's another layer of duplication they're going to have to own on top of this this deployment because

our current architecture doesn't factor it in. So, if they really want to introduce a change, test it out against the same routing group first in the channel gateway, that's on them. But, we do have an item later this year where we want to look into how we can leverage Karmada propagation policy or other tools that are within the landscape to see if we can do exactly that,

which is to deploy to one side first, uh define some rules to make sure that there is going to be one side that is still active and right before we continue to roll out the change to the other zones as well. Yep. Hey, hello. Can you hear me? Uh I wanted to discuss how you do automated tests of disaster recovery. Do you use any frameworks for this

like Terra test or how you validate that entire disaster recovery works? Could you repeat the question? Sorry. There's a lot of echo. Yeah, sorry. How you ensure that disaster recovery in V and V works? Do you have any automated tests or automated frameworks for this? Yeah. So, we do have um kind of scheduled end-to-end tests and release base end-to-end tests. Uh by your question, did you mean

like when when we bring back up a cluster, we're verifying that uh a zone is still active in the way we need? So, channel gateway actually already has um a baked-in application logic to detect when a specific back-end is healthy or not. So, within channel gateway, it actually has the application logic to understand if a back-end is healthy enough for the queries to be routed back to

it again. That's that's the logic we're relying on for that. Yeah. >> Hello. Next question. How you protect against the Karmada cluster being out? For example, if the cluster Karmada the host Karmada if it's going out the control plane I mean Karmada cluster. If it's going down what would you do? So I I guess the the question is like what if Karmada goes down, right? Yeah, yeah.

So so so Karmada Bloomberg is a managed service. So we do not as a Trino team manage uh directly Karmada. We using them as a service and they provide SLOs on their side. That we using their prod um But do you know how they do it? I'm sorry? Do you know how they do it? How how they do it? Uh that's going to be a great question

for the Karmada folks. They're in the they're in the room by the way. So uh feel free to ask them like uh at the end of this talk. From from your diagram I understood that as a Trino service owner my interaction is with the top cluster with the stretch at TV. But the deployment is actually done by Karmada in multiple clusters. How do you transmit if something

goes wrong with the deployment of the Trino service in one of the clusters? How do you reflect that information all the way up to my interface which is the top cluster? the the the way the flow works is that once it's down there and their statuses our lower script is taking care of the aggregation layer. So then Karmada reflect this in the status of the Trino deployment.

And then our controller as a watch on Karmada API server to every time there is an event that is kicked off um we are we are taking the status of Trino deployment and updating the status of the Trino service so the user can get the status. So you have a controller in the top cluster that watches the That's correct. So if you I can go back to

the to the design here. and I guess where the Trino service controller is here. So we have the Trino service controller there. And this controller uh when we set up the manager, we make it like look for events that are happening on the Karmada uh API server. So if there's a change on the Trino deployment, they will emit an event against the Karmada control plane and our

controller will be watching that the API server and then reconcile. I want to ask uh what uh Trino gateway has uh more than a gateway API of Kubernetes. Yeah, great question. So um Trino failover is a tricky thing. So in from the client side, when you initially submit a query, you you don't have a query ID. And the coordinator that receives it actually returns you the query

ID you're supposed to use for the next page and so forth. So there's a little bit of application logic, right, uh for routing Trino queries to begin with. And then on the other side for you to kind of manage um the draining process of Trino, there is also a little bit of application logic that is baked into Trino gateway that allows you to tag certain backends as

being inactive, which signals um the queries that are currently in process, right, to complete, still be routed to that inactive backend. Uh while the newer queries are prevented from being routed to that one. So that if you have certain um query limit to how long your queries can be, you have a window of time that, you know, if you drain a cluster in this managed through the

Trino gateway, you're guaranteed to not be able not not have to interrupt any of the queries and drain it successfully. So that that was exactly the reason why we decided to go with Trino gateway because it is an open-source project within the Trino community that has developed this application logic. And of course, if you deem that you want to go with an Envoy gateway for instance and

implement that logic yourself, that is absolutely doable as well. I have two questions. Have you considered uh using um something more than just deployments, uh something like Argo CD or even something more um complex than just a deployment with the uh Kubernetes deployment? Um and second question is um how easy is it to expand your compute to three clusters instead of two of them, just in case

two of them is not enough anymore? Um so I guess the the the first question is like um why not integrating Argo into our workflow? Um so there is not a huge need for us to do that. cuz this is a pretty simple workflow. Um this is just like a a controller submitting a a CRD into uh Karmada and then Karmada uh spanning it out onto uh

Kubernetes clusters. like we didn't see the need of integrating with with Argo. Um and then the second question, uh sorry, was around pick from the home good school. Yes, that's correct. Um so it's it would be pretty easy. So um on our side at Bloomberg, we rely on something called CAS, Kubernetes as a So the idea is that we can just like take a spec definition of

a Kubernetes cluster and just uh create uh create it by requesting some machines that would deploy our whole stack. So uh we also have a deployment tool that is like um that is called Bobcat and that help us to um basically just add the the configuration of the cluster and then the whole stack will be deployed on this on that Then the the last step would just

be to uh make sure that our Carmada knows about this this cluster. So when we integrate with managed Carmada, we just need to let them know uh by let them know it's obviously via uh uh their their API to be like, "Hey, I want to register that cluster because now I extended my my my uh geographic distribution. So please register that new cluster in there." And then

you will know that I need to deploy Trino Trino uh Trino deployment into that new zone that will register against the gateway and then the the the two uh workflow for Trino service owner and for query user would be exactly the same. I have another question about this it is in the deployment. Can you expand how it is implemented? From what I see it is basically single

point of failure for an entire deployment. Because it just shifted logically. Sorry, I I I I didn't get that. No, can you expand about stretch at ETCD implementation? How it works? same thing at Bloomberg there is a another managed services which is um, stretched as a server, where basically you you request uh, an allocation and uh, they provide at CD. Uh, and then what we did is

that we have um, a helm chart um, that is installing an API server. So what you see in there, the Trino API server is actually just a pod. And it's a pod that is configurable uh, that act like an API server that where you can enter like the certificate gates to connect to it and where you can reference uh, like a stretched CD endpoint. So the idea

is that this pod will then swap, act like an API server and swap the local uh, at CD just to point to uh, at CD Um, just to answer that a little bit more, there's actually kind of industry parallels uh, just to so that um, we kind of highlight what you can actually use outside of a managed ecosystem. Um, kind API server that can run on Postgres,

right? Where the actual data is replicated across multiple zones is a good parallel for you to be able to adopt if you're thinking about uh, implementing something similar. But the this uh, at CD pod is deployed in both clusters. So you have like two at CDs. Yeah, so we we basically set those two pods to share the same cube config. So to share the same certificate. So

if a request is coming to our service provider, it will go to one of the pod and since they share the same like um, stretched CD backend, both pod are going to be aware uh, about the about this newly uh, newly Um, so they they share the same cube config basically. Yeah, I think just to kind of tie the knot and uh, note on that point, the

key idea there is I different clusters just store and share the um the custom resource persistence into a single source of truth that is still distributed and highly available. Yeah, Postgres and CockroachDB offer a similar approach as long as the Postgres instance itself is actually replicated. All right, I think that was all of the questions. Thank you all so much for coming to the talk so late

in the day. Thank you.