Achieving Resilient Multi-Cluster AI Inference on Kubernetes With Kar... Wei-Cheng Lai & Han-Ju Chen

In this talk at KubeCon Europe, Han Ru Chen and Wei Chen Lin present a practical approach to running AI serving reliably at scale on Kubernetes, addressing challenges such as bursty traffic, GPU scarcity, and regional failure risks. They introduce a two-layer architecture comprising fleet orchestration across clusters and inference serving within each cluster. The session covers fleet orchestration using Karmada, focusing on placement, spreading, overrides, and automated failover strategies to enhance reliability. In the serving layer, Ray Serve's capabilities are highlighted, including request-driven auto-scaling and efficient GPU utilization through features like fractional GPU allocation and model multiplexing. The talk concludes with insights into how Ray, a unified compute framework, integrates with Kubernetes to streamline AI workloads across various environments.

Hello everyone. Um, welcome to KubeCon Europe. My name is Han Ru Chen and my partner, my co-presenter, is Wei Chen Lin. I work at Anyscale and uh Anyscale is the company behind Ray, the open source standard for distributed computing in Python. Uh, due to visa issues, my partner won't be able to give a talk with me in person, but he will record a video to explain the

Commander-related part of the presentation. Today, we will share a practical approach for running AI serving reliably at scale on Kubernetes under uneven GPU capacity and regional failure risks. Here is the plan. We will start with the production challenges, including bursty traffic, strict P95 and P99 latency targets, and GPU scarcity, and explain why these push single-cluster Kubernetes to its limits. We will then present a two-layer blueprint. The

first layer is fleet orchestration across clusters. The second layer is inference serving inside each cluster. Wei Chen and I will split the session. Wei Chen will cover the fleet layer and show how Commander applies policy for placement, replica set spreading, per-cluster overrides, and automated failover. I will cover the serving layer and show how Ray Serve by covering Ray Serve is including request-driven auto scaling, safe upgrades, and

advanced feature Ray Serve provides. By the end, you will have a vendor-neutral reference architecture you can adapt. Now, let's start with the challenge. To start, I want to set a clear baseline. At scale, AI serving has production requirements that are strict stricter than most stateless microservices. First, we need predictable service level objects. That means tight P95 and P99 latency targets, strong availability, and controlled error rates, even

when traffic is bursty. Second, we need elastic capacity and GPU efficiency. Traffic can spike quickly, but scaling serving is not instantaneous. Models need warm-up time, including loading large weights and initializing GPU memory. We need absorb spikes without building large queues, while also avoiding over provisioning and leaving expensive GPUs idle. That makes right sizing and utilization critical. Third, we need regional locality to meet latency budgets and comply

with data residency boundaries. Especially relevant here in Europe, inference often needs to run close to users and within specific regions. Finally, we need safe operations and visibility. We need rollouts that are gated by health signals, clear rollback paths, and model level telemetry such as tail latency, queue depths, and error rates. Many inference requests are long-lived, so upgrades should support graceful draining rather than abrupt termination. The goal

is to meet all of these requirements at the same time. Next, I will explain why a single cluster with standard Kubernetes primitives becomes a bottleneck. Given these production requirements, single cluster Kubernetes and standard primitives fall short in two areas: infrastructure limits and serving orchestration. First is the single cluster boundary. One cluster is one blessed radius. An outage, a control plane incident, or even planned maintenance can disrupt

the entire inference service. GPU capacity also hits cliffs. When a region runs out of GPUs, requests start queuing and latency spikes quickly. A single physical location also can satisfy both tight global latency targets and regional data residency Second is the vanilla Kubernetes gap. Standard service and deployments are not designed for inference serving features such as multi-node and multi-GPU execution. Operationally, inference requests are often long-lived. Without serving

aware draining, a normal HPA scale down or rollout can interrupt in-flight requests and token streams. So, the takeaway is simple. A single cluster is a hard boundary, and primitives do not provide the serving life cycle inference needs. This is why we use a two-layer design, a free layer across clusters, and a serving layer inside each cluster. To address both the single cluster boundary and the vanilla Kubernetes

gap, we separate concerns with a two-layer architecture. At the top is the free layer. Kamada acts as the multi-cluster control plane and use policy to decide where workloads run. How river costs are spread, how cluster-specific overrides are applied, and how failover works when a cluster or region becomes unhealthy. This answers the where. Inside each target cluster is the serving layer. We run KServe via Kuberay service to

provide inference serving capabilities such as request-driven scaling, health gating upgrades, graceful draining, auto scaling, and some advanced feature I'll talk about later. This layer answers the how. Next, we will start with the fleet layer and go through the Karmada patterns: placement, spreading, override, and Hi, everyone. My name is Weichao Lai. I'm a software engineer at Bloomberg, working on the machine learning platform. And an open source contributor

to the Karmada and KubeRay projects. Unfortunately, I'm unable to join the person in join in person today due to a visa issue. So, I've recorded this short introduction. In the next few minutes, I'll briefly share how we build the fleet layer using Karmada. After that, my co-presenter, Henry, will continue the session. Thank you for your understanding. Let's get started. Karmada is Kubernetes native. We can keep standard

APIs, standard manifests, and the same Q-caller workloads. If you can deploy a ray service in one cluster, you do not need a new workload format for merely cluster. This model is decoupled. The ray service YAML stays state stable. And all merely cluster cluster topology and fleet logic is expressed as declarative policies. Through policy, Karmada provides the fleet capabilities we need. Placement and replica spreading, per cluster overrides,

and automatic failover. Ray services is a custom resource, so we use the Karmada resource interpreter to let Karmada understand ray service status, resource requirements, and readiness semantics, so fleet policy can act on it consistently. Now, let's make the fleet layer concrete with the first policy pattern for replica scheduling in Karmada, defining a propagation policy. First is placement. With cluster affinity, we select the clusters that meet latency

or compliance requirements, such as clusters labeled for a specific region. We do this without changing the ray services YAML. Second is spreading. With spread constraints, we require the service to be distributed across multiple clusters or regions. This reduces blast radius and avoids concentrating capacity in a single failure domain. Third is replica scheduling. For ray services, we mostly uh most commonly use duplicated. That means Karmada deploys the

same ray service into each selected cluster, and each cluster runs its own serving stack. This aligns with our two-layer design because scaling and upgrades are handled inside the cluster by ray service, while multi-cluster placement and resilience are handled by Karmada. Karmada also supports divided scheduling when a workload needs a replica split but it is a less common for our ray service setup. Next, once the service is

running in multiple clusters, we need to adapt to differences between clusters. That brings us us to overrides. The override policy is to handle the heterogeneity challenge. In a real fleet, clusters can differ in GPU labels and scheduling constraints, storage defaults, and the container registry they use. If we encode those environments detail inside the ray services YAML, we quickly end up maintaining separate YAML per That increases trade

drift and makes operation harder. Instead, we keep one ray service definition and apply cluster-specific adjustment through an override policy, as shown in the diagram. The base ray service stays the same, and Karmada applies targeted overrides before the resource is applied to each member cluster. Common examples including updating the image registry prefix for a local mirror, selecting the appropriate storage class for a mobile caching, and applying closer

specific labels or environment settings needed for scheduling integration. This keeps the responsibilities clear. The fleet layer adapts environment wiring, and Ray Serve remains the source of truth for the service definition. With placement, spreading, and overrides in place, the remaining fleet topic is failure handling. Next, we'll cover dual-level failover. We use two-level failover triggers provided by Karmada to limit impact. First is cluster-level failover. Karmada monitors member cluster

health. If a cluster becomes not ready or unreachable, it is marked with the corresponding cluster tag. New scheduling to that cluster stops. And after a configurable grace period, affected workloads can be affected and rescheduled onto healthy clusters that still satisfy the placement policy, based on the cluster failover policy. Second is workload-level failover. A cluster can be healthy while a specific Ray Serve is not ready, degraded, or

crash looping. To avoid relying only on cluster health, we integrate Ray Serve with Karmada using a resource interpreter. The interpreter maps Ray Serve status into health signals that policies can act on. So, failover decisions can target the affected workload. That concludes the fleet layer. Next, I'll hand over hand back to Hanru, who will cover Ray Serve served by Ray Serve, including auto scaling, safe upgrades, and advanced

features. Thank you for listening. Now, I'm going to talk about challenges of AI serving in production, and how Ray Serve and KubeRay solved it. So, the first problem is real-world AI serving isn't just one model, tensor in, tensor out. It's pre-processing, multiple models, post-processing, and business logic chained together, each with different resource needs. Traditionally, each step or deployment becomes its own service, its own container, its own

config, its own deploy pipeline, which is very hard to manage. So, in Ray Serve, we have a feature called model composition. Instead of deploying each step as a separate Kubernetes service, Ray Serve lets you compose your entire pipeline as a single Python application with independently scalable components. The next problem is inefficient GPU utilization. Many machine learning models don't need an entire GPU compute and memory capacity. In

traditional serving setups, each model is assigned a dedicated GPU regardless of how much of the model actually use. A model that only needs 30% of a GPU still occupies 100% of it, leaving 70% idle. At scale, leeway translate directly into unnecessary infrastructure cost. And Ray Serve provides a solution. It supports fractional GPU allocation. Instead of dedicating a full GPU for every model, you can specify the exact

fraction each one needs. The model using only 30% of a GPU now requests just 30%. And the remaining capacity is available for other workloads. One configuration change, no specialized hardware. And in the diagram this example, you can see that your GPU utilization can jump from 30% to near 100%. The second problem with inefficient GPU utilization is that modern applications often serve many model variants that share the

same architecture but have different weights. Most receive sparse, unpredictable traffic. Provisioning a dedicated GPU for each is economically invisible. RayServe supports model multiplexing. A single replica can host multiple multiple model variants in GPU memory simultaneously, loading and evicting them on demand by LRU caching. The router knows which replica has which model loaded and directs requests accordingly. So, you can register hundreds of model variants, but only keep

a handful warm at any time. Low-rank adaptation is a natural fit for model multiplexing because all variants share the same base model weights. Only the small adapter weights, typically less than 1% of the base model, need to be swapped in and swapped out. This dramatically reduces both the swap latency and the GPU memory footprint, allowing a single replica to cache far more adapters than full models. The

next problem is scaling inflexibility. Traditional machine learning serving forces you to scale the entire pipeline. It treats CPU processing, GPU inference, and CPU post-processing as a single unit. So, scaling for a CPU bottleneck wastes GPUs, and scaling for a GPU bottleneck wastes CPUs. Making matters worse, uh default Kubernetes HPA scales based on CPU or memory utilization, not request loading. It can't see how many requests are queued

and processing, so it reacts too slowly for bursting inference traffic. RayServe solves this problem by scaling each deployment independently. So, pre-processing inference and each have their own replica counts and auto-scaling policy. Scaling decisions are driven by request queue depths, not CPU utilization. When queued requests exceed the target, replicas scale up within seconds. When traffic drops, they scale back down after a configurable cool down, which leaves no

wasted resource. You can see the right example. When GPU is bottleneck, we can only scale the GPU deployment. The next problem is development and production gap. In traditional machine learning serving, the code you write in development is like a simple Python script. It looks very different from what runs in production. For example, like you need Docker files, Kubernetes manifest YAMLs, service mesh configs, load balancers, monitoring stacks,

forcing team to rebuild and repackage their application for every environment transition. With RayServe, you don't need to rewrite anything. You just write a Python file, a serve.py Python file, test it with one-line Python, a single command line. And when you're ready for production, you wrap it in a Ray service YAML and kubectl apply. Same code, but just same uh just different command and all one line. The

next problem is framework locking. So, traditional model serving tools are tied to a specific machine learning framework or specific cloud provider. For example, like TensorFlow serving for TensorFlow, Torch serving for PyTorch, SageMaker for AWS, and Vertex AI for GCP. When your team adopts a new framework or migrates to a different cloud, the entire serving infrastructure has to be rebuilt. Ray Serve solves this problem. It is framework

agnostic and cloud agnostic. It serves any Python code, PyTorch, scikit-learn, or your own custom logic through the same serving infrastructure. Because it's built on Ray, it deploys wherever Ray runs, any cloud, and even on prem. You can swap models and frameworks, but the serving layer can still stays the same. Right now, I want to spend some time to talk about Ray. Ray Serve is built on top

of the Ray, and Ray is a unified compute framework that turns complex distributed AI workloads into simple Python programs. You can use Ray to run AI workloads on any hardware, any infra, at any scale. Ray was originally developed at UC Berkeley, and Anyscale has been one of its major contributors. And right now, it's part of the PyTorch Foundation. Here are the problems Ray solved. So, building an

AI application end to end requires involving and learning a lot of different frameworks, from data processing, training, auto tuning, to reinforcement learning and serving. Each has its own framework traditionally. Also, every stage of the machine learning life cycle use a different distributed system. So, glue code and operational overhead grow very, very fast. And there is a huge gap between development and production. Code that works on a

laptop needs a lot of rewriting to run on a cluster. And Ray is a solution. Ray provides a unified compute layer for AI workloads. One API scales seamlessly from a laptop to infinite laptop with minimal code change. Ray open source provide five native libraries cover the full machine learning life cycle under a single framework. Ray data for scalable data loading, transformation, and even batch inference. Ray train

for distributed training with fault tolerance. Ray tune for distributed hyperparameter tuning. Ray serve for scalable model serving for online inference. Ray reinforcement library distributed reinforcement learning. And these all libraries share the same foundation, Ray core. Ray runs on Kubernetes, AWS, GCP, Azure, Slurm, Yarn, or on premise. So, you can run Ray at anywhere. Now, I want to spend 30 seconds to teach you how to use Ray.

On the left is a regular Python program, a function and a class. On the right, the same code with one additional decorator, Ray.remote. That's all you need to learn to write a Ray program. After adding the decorator, the function becomes a distributed task and a class becomes a stateful actor. You can call Ray.remote to execute. Ray.get to collect the result. Now, I want to spend some time

to talk about Ray's architecture and we are going to use it later. This is what a Ray cluster looks like. In a very, very high level, it looks very, very similar to Kubernetes. They both are resource orchestrator, but at different layer. Ray is process level orchestration. Kubernetes is container level and Ray schedules Python function and classes. Kubernetes schedules containers. On the left, you have the control plane.

The head node is stores cluster metadata and handles scheduling and auto-scaling decisions. On the right, you have the data plane, the worker nodes. This is where your actual workloads run. And now, let's talk about how to run Ray on Kubernetes. Here is our today's main character, KubeRay. The Ray focus on application layer, and it handles process orchestration. Kubernetes focus on container layer, and it handles container orchestration.

KubeRay bridge release two layers. It's a Kubernetes operator that manage the life cycle of Ray applications on Kubernetes prepare resource to run application, and Ray decides how the application is going to use the KubeRay provides three API, or you can understand as three CRD. Ray cluster for managing Ray cluster we just seen before, and you can run any Ray workloads in a Ray cluster. Ray job for

workloads like distributed training or batch inference. It creates a cluster, runs a job, and cleans up. A Ray service, which is what we'll focus on today. It manage both the cluster and Ray serve application with zero downtime upgrades and high availability. Next, I'm going to talk about key features in Ray service. The first feature is in-place update for serving applications. Imagine you're running ChatGPT 5.3 on a

Ray cluster, and you want to upgrade to ChatGPT 5.4. With in-place update, you don't need to create a new cluster. You simply update the serve config in your Ray service manifest. And that's it. KubeRay detects the change. It sends a HTTP request to the Ray control plane API server, and Ray Serve handles the rolling update internally. All within the same cluster. So, no new cluster, no downtime,

no wasted resource. We just need to update the config and apply it. The second feature is zero downtime upgrade. when you want to upgrade the entire Ray during the zero downtime upgrade process, KubeRay will create a new Ray temporarily and waits for it to become ready. Once the new Ray cluster is ready, the Kubernetes service here will switch the traffic to the new Ray cluster. And finally,

KubeRay will delete the old The third feature I want to talk about is high availability. So, serving AI models in production means you have to dealing with failures. For example, a replica process can crash. A node can goes down. Or even the whole Ray cluster crash. Restart the Ray cluster can take a lot of time. KubeRay plus Ray Serve handles recovery automatically at every layer. Ray Serve

helps data planes replica to restart. KubeRay help control plane and data plane nodes restart in the Kubernetes Let's talk about auto scaling. So, AI inference traffic is bursty, and a service can go from near zero to thousands of requests in seconds. In KubeRay with Ray Serve, auto scaling happens in three layers. Layer one is the Ray Serve auto scaler. It scales serve replicas based on request pressure,

queue size, and ongoing requests, rather than just resource utilization. If the Ray cluster already has a spare capacity, new replicas can start quickly as reactors on existing Ray nodes. Layer two is the Ray auto scaler. So, when the current cluster no longer has enough logical resource to place low Ray replicas, Ray S worker pods, because Ray understand the workloads resource requirements, it can scale the appropriate worker

group. For example, GPU workers for the uh large language model deployment, and CPU workers for pre-processing. Layer three is the Kubernetes cluster If the new Ray worker pods can't be scheduled because the Kubernetes cluster is all of capacity, Kubernetes provisions new nodes. So, these three layers work together. Serve layer scales the application, Ray scales the worker pod, and Kubernetes scales the cluster by adding Kubernetes node. I

want to mention two points spec- specially. Unlike the default Kubernetes HPA setup, where auto scales on CPU or memory utilization, Ray Serve is natively request-aware and scales based on queue depths, and ongoing requests. And if the billing policy is not enough, Ray Serve also lets you define a custom auto scaling And KubeRay, right now I want to talk about its ecosystem. KubeRay integrates Kubernetes ecosystem you already

use. For scheduling, we support Q, including multi-Q for multi-cluster job dispatching, as well as Volcano, Apache Unicorn, Nvidia's Kai scheduler, and the Kubernetes scheduler plugin. Whether you need gas scheduling, priority queues, or GPU sharing, KubeRay works with your best scheduler of choice. For observability, KubeRay exports Ray metrics to Prometheus and ships with pre-built Grafana dashboards. So, you can monitor cluster health, auto scaling, and request latencies through

the same monitoring stack you already have. And if you need to profile a running workload, we support PySpy integration for CPU flame graphs directly from the Ray dashboard. Finally, I want to introduce a new feature in Ray and KubeRay, the history server. Many teams run Ray cluster as ephemeral units. They spin up cluster, run the job, and tear it down. This saves resource cost, but when the

job fails and the cluster is gone, you lose the Ray dashboard, your primary debugging tool. The history server fix this. While a cluster is alive, it automatically collects cluster metadata and logs into persistent storage like S3 or GCS. After a cluster is deleted, the history server restores the dashboard from the data. So, you can debug a terminated Ray cluster as if it was still running with no

Ray cluster resource cost. And the KubeRay history server is mainly designed and developed by Anyscale, Google GKE, and Alibaba ACK. please join the community. Please join our Ray Slack, and here we have a lot of Ray masters from Anyscale can answer your questions. Also, please join KubeRay Slack to ask more questions. They are in CNCF computer channel. Thank you everyone.