The Shared Service Blueprint: A Guide to Multi-Tenancy, Illustrated With KEDA - Aya Igarashi

This talk discusses the challenges of implementing shared services in a multi-tenant Kubernetes cluster, specifically focusing on KEDA as a case study. The speaker, Aya Gadash from Preferred Networks, explains the necessity of multi-tenancy for their platform and the design choices made to achieve namespace isolation. She details how a balance of namespaced and node-level isolation is implemented to separate tenant credentials and contain failures. The session covers multiple Kubernetes constraints related to metrics and authentication flows, culminating in a solution that ensures each tenant maintains its own isolated trust boundary. The importance of network policies is also emphasized to prevent cross-tenant access issues.

Hi everyone. Thank you for joining. I'm Aya Gadash, software engineer at Preferred Networks. Today, I'll be talking about reality of providing shared service in multi-tenant Kubernetes cluster. I'll use Keda as a case study to show how we can achieve namespace tenant isolation even when the service wasn't originally designed for it. Here's our road map for today. First, I'll explain why multi-tenancy is needed for our platform and

why we choose our specific isolation model. Then, we'll dive into Keda and look at why we choose the pertinent deployment model. The main part of this talk on four Kubernetes constraints, which we have to solve. At the end, I'll come back to the full architecture and design lessons. A bit about us. Preferred Networks is a full-stack AI company. We build everything from research model down to custom

AI chips called MNcore. We deliver this computing power through the our Preferred Computing Platform, a Kubernetes-based environment serving both internal and external researchers. PFCP is not an anonymous public service where anyone can sign up and create workloads, but still, we share clusters across different organizations, so multi-tenancy is our business requirement. This gives us two constraints. Tenant credential must stay separated and failure must stay contained. If something

goes wrong, like simple misconfiguration, it should never become another tenant's problem. That's why keeping a small blast radius is key to our design. Here's a major Kubernetes multi-tenancy patterns. Kubernetes multi-tenancy is a spectrum. On the local side, you have a shared control plane with namespaced isolation. As you move to the stronger isolation, you pay more for virtual cluster, dedicated nodes, or physically fully separated clusters. Of course,

there's no single correct answers here. It all depends on your requirements. And for our platform, we combine two levels. Default namespaced isolation for everyone and optional node-level isolation for those who need it. This gives us a balance of flexibility within a single cluster. And this is our actual tenancy model. Each tenant owner root namespace and its owner can manage its child namespaces. We use higher hierarchical namespaces

called HNC to handle that structure and ensure things like resource quota or something like that consistency. As some of you might know, the Kubernetes org HNC project has already been archived, but it's a special for our platform, so we keep developing our folk version. The important point here is our tenant boundary is lived inside a single cluster. This means every shared component where it's a controller and

extension API server or network path must respect the hierarchy. If not, the iteration model breaks. So, why did I choose KEDA for this talk? Because it makes a difficult part of multi-tenancy visible. At PFN, we provide many services to our users. When we build an in-house component, we can design for our own platform from the beginning, but of course KEDA is a open source. It was not

originally designed for us. That makes it a good case study. We have to adapt it our requirement. Also, KEDA touches many boundary at once, both internal Kubernetes cluster and external cloud integrations. Before we dive into the design, let's do a quick recap of how KEDA works. Here's the architecture. KEDA extends horizontal pod auto scaler called HPA using Kubernetes external metrics. When a user creates a resource like

scaled object resource, the KEDA operator reconciles it and creates a new HPA resource. The results point to the KEDA metrics. If the event source needs authentication, KEDA gets the credential from the trigger authentication results. Then the HPA controller periodically gets those metrics and handles the actual scaling. Yeah, you don't need to remember everything here, but just remember that KEDA is a system of multiple components, not just

one controller. This matters because it means that each pieces has its own identity, its own API surface, and its own trust boundaries. So, next, let's think about the Roughly speaking, there are two options. The first option is shared KEDA, installing a single instance and using KEDA building isolation features. The second option is a pertinent KEDA, installing a separate instance for every tenant. At first glance, the shared

model looks very reasonable. It gives you one installation to manage and one upgrade path. On top of that, KEDA object model supports namespace level authentication. if you look at the Kubernetes objects, shared KEDA seems like the better choice. But, finally, we choose pertinent KEDA, and the reason comes down to the authentication flow. Now, take a look at KEDA diagram again and let's look closely how KEDA handles

authentication for event sources. KEDA supports many event sources like Prometheus, Kafka, Redis, and so on. And each one has its own authentication method. In our case, we need AWS managed Prometheus support. So, this diagram show how that integration works. As I mentioned, target authentication is a resource that define an authentication method. And because it's namespace the resource clearly separated between tenants. So, but look at the diagram

to see how the token exchange works. First, KEDA operator use it its own uh short-lived cloud credential in this diagram, green one, and then operator use a key to query the metrics from the source. However, when initiating the exchange process, bootstrap identity in the chain does still belong to the operator process. This is where the actual trust boundary lives. KEDA is designed to be tenant-friendly as configuration

can be scoped per namespace. And our back can isolate the object. If you only look at the object method, the shared KEDA still looks reasonable. But the situation changes depending on your event source. In our case and in any case where the operator needs a federated credential exchange, the bootstrap tokens that start the chain belong to the operator process. This means if the operator is shared, the

trust boundary is also shared. That's why we choose pertinent KEDA to ensure that each tenant has own isolated trust boundary. So, pertinent KEDA model solves that main identity problem, but we still have another four Kubernetes constraints. Cluster-wide singleton constraint, aggregated API authentication, dynamic dynamic namespace scope, and traffic boundaries. So, the list of the talk, I will explain the challenge and then show you the solution we use.

The first challenge is external.metrics.k8s.io. The problem is HPA uses a single cluster-wide API group Even if we install a separate metric server for each tenant, Kubernetes still expects one single front door for all On the left, you can see the API service resource. This is where you register backend with the API server. You can have this API service resource for different API groups, but only one backend

can be served as a same API group. If you try to register a second one, it's simply overwritten. And on the light side, the HPA resource has an excellent type with the metrics name. It has no mechanism to specify which back end should return the metrics. So, even though we install a separated Keda metrics server for each tenant, we cannot register each one as an individual back

end. The solution was to keep a single API service and put a router behind it. And this works naturally because external metrics API path already contains the name space. So, router checks inspects the path, extracts the name name space, and forward request to the light tenant to the back end. No custom needed. It's all in the URL. We implemented using Nginx. As you can see, it's just

a simple location match using a regular expression. This allows the front door to stay cluster wide while the back end stay isolated per tenant. But, once we place proxy in the path, we run into next problem, identity. Before we talk about the failure case, let's look at the normal flow for aggregated API. When the HPA needs metrics, it sends a to the cube API server. The API

server authenticates the request as usual. And then proceeds to Keda metrics Instead of forwarding the original bearer token, the API server sends identity headers like X-Remote-User and X-Remote-Group. The back end, in this case Keda metrics server, must trust the proxy, not just request headers. To do this, the back end server verifies the HP API server's client certificate. In Kubernetes, these default settings, such as CA bounder and

header keys, are distributed via config map named extension API server authentication. And you can see kubectl get in kube-system Once the client authentication is approved, the back end grabs a user info from the headers and makes a call to the API server with a subject access review to see if they are authorized. And now we can see what change when we put our router in the path.

The router needs to see HTTP path to route traffic by namespace. So, the traffic terminates TLS and open a new connection to the back end. From the back end's point of view, the caller is now the router, not the kube API server, but the default certificate is fetched from the config map or client it still expects kube-apiserver When this trust chain breaks, authentication step fails and the

call off falls through system anonymous user groups and the subject access review is denied. Of course, the whole request fails. To fix this, we have to rebuild the trust chain. So, we need these three things. First, the router must present a client certificate when connected connect to upstream to the backend. Second, backend must trust the router CA, not the default kube-apiserver CA. Third, router must preserve the

forwarded X-Remote-Headers without rewriting or dropping them. Since the Keda metrics server is built like a standard custom metrics API we can configure all trust certificates and header keys using standard plugs. So, we just tweak those plugs to match our router settings. Once those three pieces are aligned, the backend can see the original HPA identity again. And also authorization works again. The third problem is namespace scope. Each

per-tenant operator shouldn't watch only namespaces belonging its own This is important for permit performance, but even more important for security because the operator reads secrets for authentication. So, we wanted to keep this scope as narrow as possible. The difficulty is that namespaces are dynamic. Tenant owners can create and remove them at any time. But, KEDA gives us a watch namespace environment variable. But, it's a static comma

separated list. And, this isn't just a KEDA limitation, but most operators use controller runtime library, and the library doesn't allow changing the watch scope after the manager starts. So, we need a way to keep operator scope aligned with the tenant hierarchy without changing KEDA itself. Our solution was very simple wrapper process. We put operator behind a simple namespace reloader. The reloader does Watches tenant namespace, update watch

namespace environment, and it restarts the operator when the namespace set changes. The design constraint was to stay close to the upstream manifests. If we do drift from the original installation settings, the cost of future upgrade grow quickly. So, we keep the changes to KEDA deployment manifest minimal. Just updating image and injecting several environment variables. And, now the watch scope stays narrow and it stays aligned with the

tenant hierarchy even as a name space changes. The last problem is the network As you know, the default Kubernetes has no network isolation between name By default, any port can open to connect to any other port. This could be a security risk for multi-tenancy. For example, if you look at red dotted line in the diagram, an misconfigured or even a malicious port in tenant A can directly

reach tenant B, like metric server, operators, or even user workload. To prevent this, we need to use network policy to block any expected traffic. Then, how exactly should we design and configure those policies? And how should we network boundaries actually follow our tenant boundaries? Our first step was to classify the traffic and then identified allowed past. We grouped our component into specific scopes like tenant system and

tenant wide. For ingress, we allow only known sources like router or metric server. For ingress, we strictly limit traffic to tenant's own name spaces and required to shared services. Everything else, including cross-tenant traffic, is denied. By the way, we use Cilium as an our CNI driver, and it has been such a huge help for us during this process. Because when I tried to verify my network design,

having a visualization tool is a very useful. Like you have find the misconfigured policy and unexpected network flows that we missed in our initial design. Um once we map the traffic path, we create our policy resources. Regarding policy management, Kubernetes building network policy is namespace scope resource. So, this means you have to define policy for every single of course, it's hard to do scale. Kubernetes official cluster

network policy is still alpha and still not yet building. So, instead, we use Cilium cluster wide network policy. This is a much more efficient because it allows us to define rules for an entire organization using namespace levels. This is especially helpful in our environment because we use HNC, so namespaces are often changed. Now, we can pull all pieces back together. Cluster wide, we keep one API server

from door and one shared admission webhook. Per tenant, we run the operator, metric server, and namespace reloader. The routers solve the single API problem. MTRS and header request trust keep the color identity intact. The reloader keeps the operator scope aligned with the tenant name spaces. And network policy makes the traffic boundary for the tenant boundary. So, to wrap things up, I wanted to show some key takeaways.

With an in-house operator, you can design for multi-tenancy from the beginning, but vendor product and open source are different. This usually means we have to write extra glue code or use some operational hacks to make it work. When we evaluate our shared service for our multi-tenant platform, we check First, we look at operations. We need to make sure the service is easy to manage and resource efficient

for our scale. Next, we consider reliability. We have to understand the blast radius and figure out how to protect our tenants from noisy neighborhoods if something goes wrong. Finally, we review security. We check if the service respects our security boundaries like labeling its permission, network path, and authentication methods. Of course, there's no single perfect setting that solves everything. Security is balance between strong isolation and operational cost.

You have to accept trade-off and decide your configuration based on your actual business requirements. Thank you. So, any questions? If not, so wrap it up and feel free to ask me directly. Thank you.