Three Shades of Isolation: A Multi-tenancy Fortress - Braulio Dumba & Paolo Dettori, IBM

This talk introduces a novel approach to multi-tenancy in Kubernetes, focusing on achieving effective isolation among tenants through control plane, data plane, and network isolation. The speakers, Paulo Detorii and Braio, from IBM Research, explain that multi-tenancy allows sharing of cluster resources while ensuring each tenant remains isolated. They outline four key isolation requirements, emphasizing the importance of preventing side channel attacks and ensuring independent management of cluster resources via dedicated API servers for each tenant. They propose leveraging open source technologies such as KubeFlex for control plane management, KubeVirt for running virtual machines alongside containers, and OVN Kubernetes for advanced networking. The session provides insights into a framework that balances resource utilization with security, while demonstrating a practical implementation along with performance benchmarks, which showed minimal latency overhead when employing multi-tenant isolation.

Hello everyone. How's everyone doing? I hope you're having good time in Amsterdam. Thank you. Okay. So my name is Paulo Detorii. I work for IBM research and this is I'm Dumba. So I work for IBM research. >> Okay. So today we are presenting three shade of fisolation and this is really about multi-tenency and our approach to multi-tenency. This combines control plane, data plane and networking isolation using

opensource technologies. First of all, let's talk about the multi-tenency in Kubernetes. What do we mean by multi-tenency? So multi-tenency means sharing a single cluster resources, compute, storage, networking among multiple tenants. What is a tenant? A tenant can be a user, can be a team, can be an application. As Kubernetes adoption becoming more mature, workloads are moving to production. The demand for multi-tenency is growing very rapidly. Organization

want to have efficiency and share infrastructure without compromising on isolation. And there are four key requirements for isolation that we have identified. First of all, we want to have data plane isolation. Tenants should not share compute nodes preventing side channel attacks and resource interference. Second, we have to think about access boundaries, logical segregation through network policies and name spaces. Third, control plane isolation. Each tenant needs their

own API server and control plane to manage cluster scope resources independently. Fourth, we like to provide a user experience that looks like a Kubernetes experience, a vania kubernetes experience. Tenants should be able to manage their own CRDs, cluster roles, and storage classes. Why do we need workload isolation? There are three main motivations. Security is the first and is probably the most important. Without proper isolation, one tenant

could potentially access other tenants pods, deployments, and secrets. We need to enforce both logical and physical segregation between Performance is the second concern. There is always the noisy enable problem to keep in mind in a shared cluster. One tenant highuling throughput or resource intensive workload can degrade performance for others. This especially problematic for latency sensitive AI inferencing workloads. Third, cluster scope resources conflicts in Kubernetes. Objects like

CRDs, cluster roles, storage classes exist at the cluster level and cannot be name spaced. When you have multiple A platforms for example Kajenti for agent orchestration, LLD for distribute LLM inferencing or production VLM production stack they often need different version of CRDs or completed cluster roles strong isolation with SH GPU infrastructure become essential. Let's look at existing models today for The first model is cluster as a

service where each tenant gets a fully dedicated cluster. This is probably the easiest to understand. It provides a stronger isolation but creates also clusters pro also high operational overhead a poor resource utilization as scale managing hundreds of clusters become a significant burden. The second model is namespace as a used by tools like capsule which is a CCF sandbox project. Here tenants share a single cluster but they

are confined to dedicated name spaces. The limitation is that tenant cannot use cluster scope resources like CRDs or cluster roles. There is still the noisy neighbor problem uh because compute is shared airback policy become complex to manage. The third model is control plane as a service. This an example of this is v cluster. Each tenant get a virtual cluster with his own API server and control plane

solving the cluster scope resources problem. However, tenants still share the same data plane meaning they share worker nodes. This lead enable and security boundary issue are resolved at the compute level. Our approach address all three cases simultaneously. And now Brol is going to talk about our approach. >> Yeah, thanks Paulo. To to address this challenge that Paul just mentioned the previous slides, well, we propose a new

approach for multinational isolation. Our approach targets three design goals. The first design goal is easy management. We want a solution that simplifies the multinency management and enforcement strategies for closers admins. a solution that is cost effective with minimal source consumption and lastly a solution that that is uh you uses only open source uh technologies cloud native Furthermore we want a solution that hardness the tenants boundary by

providing three sets of isolation data plane control plane and the network layer. For the network layer, we want to further decouple the control plane and data plane task to enhance security and improve the resiliency of of the to achieve to to build our framework. We use four CNSF opensource technologies that I will describe next and and most of us are aware of these technologies. The first technology

is cube cube flex. It is a sandbox project under the cube stellar umbrella that enables control plane as a service. It provides several several types of control planes such as vloa k3s and others. It allows clauses admins to manage the life cycle of those control planes. The second technology that we use is cubevert. Cubevert is also a CNSF open source project that let us run virtual machines

as kubernet uh pods. It allows running and managing virtual machines alongside containers in the same Uh the next technology is KCS. It is a lightweight uh proje a lightweight uh fully satisfied kubernet distribution uh that is packaged as a single binary under 100 megabytes. Finally we use OVN Kubernetes that is a Kubernetes CNI plugin. It is an open source project that use open virtual network and open

v switch to provide advanced software defined capabilities in a kubernet cluster. In the next slide I will explain how we use these four technologies to build our solution. Uh this this diagram shows the high level review of our framework. At the top we have a hosting kubernet cluster. Uh this cluster runs the follow management components. Cube flex for control plane provision. Cube vert for VM management and

OVN kubernet for network uh management. For each tenant we have a dedicated control plane where we create cube flex control plane of type k3s. Also for each tenant we have a dedicated data plane where we use cube ver to create VMs to be used as the as the data Each tenant also gets its own dedicated user defined network or UDN and therefore ensure that their network traffic

is completely isolated and segment from other tenants. Uh in the next slide I I'll give provide a more detailed overview of our architecture. In this slide for for simplicity we we are just focused on the uh showing the architecture of a single tenant. So for each tenant uh we have cube flex that creates a dedicate control plane of Essentially, it provide us a lightweight kubernet control plane

that run as a process inside a single kubernet in the control plane name space. Cube vert then creates VMs that serve as the tenants worker node. Inside of of each VM, we have case agents that connects connects back to the control plane of the tenant. And this give us the data plane isolation for each tenant by provided hardware level boundary without acquiring dedicated physical uh therefore ensuring

that their network traffic is completely isolate and segmentful from other tenants. We also providing a decouple of the data plane and control plane uh traffic by using two two ODNs. We are using a a userdefined network as as the as the primary UDN for the for the for the data plane traffic and also a cluster user defined network for the control plane traffic and and this and

this traffic spans uh two name spaces and and using uh this this configuration this give us a fully uh for each have a fully network meaning the no traffic from the default network can uh reach a tenants and tenants uh workload and of course it provides also a strong visualation across uh the other tenants in in this picture it zero yeah represents yeah the the primary network

that we use for for the data plane and it one denotes the secondary network that we use for the control plan control plane task. The result of this approach is is a is a cost effective architecture where each tenant experience a fully isolate kubernet cluster with their own control plane uh their own worker nodes and their own uh network all running efficiently and sharing the same physical

infrastructure. In a nutshell, using this approach when a new tenant is on board, a cluster admin you use cube flex to provide a new control plane. Then cube vert to create the VMs with predefined resource configuration CPU, memory and the others. And then use OVN Kubernet to set up the the OD the ODN or network for the specific tenant. From the the tenant perspective, they interact with

the standard uh Kubernet API and they can deploy workloads, create CRDs, define artbacks and the other clusterwide resources all without all or without awareness of the underlying multi-tenency This architecture cleanly separates the concerns. The cluster administrator manage the hosting clusters and the tenant provision while each tenant operates independently in their own environment. This approach also helps to simplify uh the management and enforcement strategies for a close

admin. Like for example, a cluster admin does not need to manage and enforce her source limits for all the name spaces or posts deployed for for each tenant. Instead, those limits and and configuration are only need for the control for the post that are running the control plane and the data plane nodes in the cluster. Furthermore, this architecture helps to minimize the noisy the noise neighbor problem

that Paul described before in in that is typical in a share kubernet cluster because each tenant isolation uh and sourceization is stick to the boundaries of the source given to the cube data plane uh nodes and the and the Next uh I will show uh a demo of of how we build and deploy this this solution. >> But before showing showing the demo, I would like to

to to describe the infrastructure setup that we use to deploy uh our solution. We use a a host control plane kubernet with five worker nodes for for the workloads. Uh we we we use we deploy two distributed inference workloads. Those are LLMD VLM production stack across two In this demo I will show an ex external user sending a request to to to to those tenants workload that

are running in a fully multenant environment. in in this setup I'm highlighting so because we we need to provide the ingress traffic for the workloads that are running uh in the in highly multenous environment we we we use we provide we create a third UDN and the prox pod the prox pod just bridge uh the connectivity between the default network and the ODN uh the audio network

uh with but still maintaining the network isolation by providing external connectivity. uh but but there are different ways of course to to provide this external connectivity for workloads running in a UDN network like you can use uh not port in ingress IP but for for our environment uh that was the best approach given that we have our clusters behind firewalls and be and using uh private IP

IP IP address now Now I'll shoot I'll let me switch to the demo and okay switch screens. Yeah, works. Okay, great. Great. >> Okay. Great. So here we are showing her demo because we have uh the our workload running uh in a remote cluster and today we don't want to be at the merc of the demo's god. Okay. This demo is is divided in two parts. Uh

in in the first uh part I'll show uh the the the infrastructure that we deploy to to build this fully uh maintenance environment. The second part I'll show uh the a workload and how external user can interact with the workload running in a isolate multi-enant environment. First I'll start by showing the uh the cube flex control plane. It it runs in a in a single uh name

space and for for each for each tenant we deploy uh two two control planes that also run in their dedicated name space. Here we show from tenant one and tenant two and this control planes are of type case K case K case K case K case K case K case K case K case K case K case K3S for each of tenants we we create uh two

cubvert VMs uh four here we choose four one per tenant then furthermore we create two uh user defined networks those are for the uh data plane uh traffic. Sim similarly we create two uh two two cluster wide network that those are for the control plane uh traffic. Now let's look close closely one of the configuration of a of a closer end. Here we see that uh uh

we can we use a name space selector where you you define uh uh the the name space that are able to talk to each other using uh this UDN. Here we see that this we show an example for tenant one. We see that the data plane and the control pen and name spaces are part of of this network. Uh here we we we are doing an Azac

just on the control plane for for the tenant one and tenant two just to show that we have a kubernet like environment where where we have the kubert VMs connect to to those control planes for both tenant one and tenant two and this provides the kubernet like environment that each tenant can can use for your your workload. Now let's look at the configuration of one of the

cube vert pods. Uh ODN attach uh annotations for each interfaces that is uh uh that a pod that is in the OD network uh uh is using. So here we will show uh the ODN interfaces uh that our of our cube convert VM is using and and those interfaces and the the correspond IP address are mapped inside of the uh V VM. Yeah, we we we just

quickly log into one of the VMs and we see that we have uh uh three inter three interfaces. Uh it0 is for the prime primary network. E21 is for the control plane network and here we have has we have a third interface that we are using for the external traffic to provide external connectivity. Now in the second part of this demo we are going to show workloads

uh two inference uh distributed inference workloads LLMD and production stack deployed in the tennis environment for for the for the uh screen on the on the top left we see the environment for tenant one and tenant one uh has LLMD deployed for simplicity here we are using LLMD simulator. So meaning we have we are using a GPU a simulator VLM pod and for tenant 2 we have

we deployed the VLM production stack again using a sim simulate VLM pod. Now for the for the screen on on the right, we see uh an external client that is sending first an inference request for for for the workload running on a tenant one environment. And and next uh the same client going to send an inference request for tenant two that is is running on the second

So this is the the end of the demo. So in a nutshell we see how uh just switch back the screen. We see how uh uh can can you we see uh uh two workloads uh V LMD and VLM production stack running uh in in in a tenant uh fully tenant environment that has a control plane data plane and network So next next we want to we

conduct some experiment to try to measure the overhead of providing our our multi-tenency isolation stack. Uh basically the the the key question that we want to answer is what is the latency cost that that we are paying by providing control plane data plane and network isolation. And this diagram shows shows uh uh try to illustrate the the the overhead uh between using uh uh isol environment you

using and cube vert isolation versus no no isolation. The diagram on the left s shows the steps that an external users uh request takes to to reach a workload to that is running in a fully multenate environment. The the diagram on on the right shows the opposite without isolation. We can clearly see that there are additional hopes that any client requests must transverse when using uh these

additional isolation layers for to provide a fully multienant environment for for for the for for experiment. We use LLMD use the simulation again as as the target workload and we use inference perf as the tool to generate the the load in our experiments. uh inference perf is a tool supported by the kubernet community that is used to to benchmark the the polament of LM inference models and

we we we use three stages because we want to to simulate different type of traffic uh with different query per second rate. On the first stage we use 10 seconds uh one square per second sorry. Second stage we use five five five requests per second and on the second uh test we use uh 10 requests per second. In total we generate uh 360 propens uh and and

here the results that we obtain from the latency result that we obtain for all those stages. uh we can see that our linear result shows that the overhead is is is is minimal. For example, if you look at the media, we see that the overhead is on the single digit millisecond range just just to to make sure that to make sure that we have good result with

with a good uh fidelity. We also measure the over the CPU and memory overhead of the cub VMs that we use because we don't want our latency result to be skewed by the over overhead of the worker nodes itself. We are going to still conduct more more experiments to to measure the overhead of our isolation stack across other different types of workloads. Next Paulo going to discuss

the uh ongoing work and next steps. >> Thank you Braio. Uh so we're going to talk about what we are doing next. Um so first of all I just want to mention this work on the console. By the way we have here Andy Anderson that thank you Andy for doing all this great work. This is part of the coupe stellar evolution. And so this is actually a

place basically where you can see different patterns different missions you can actually have visibility and provision things. So I just invite you to you know look at the QR code go and try it out because we are basically working on a pattern like the one we described that you can actually uh somehow try out in this console soon we're going to it in a few in a

couple of weeks probably but for now you can just explore what is there the other thing I want to mention uh who hasn't heard about open claw just raising hand everybody not have heard about open claw okay so I assume everybody knows about this is becoming very viral So I just wanted to mention because we'll be talking about isolation. We know that something like open CL it's

a very cool tool but can do also very unsafe stuff right and so here's the problem open CL actually runs agents commands directly on the host and sandboxing sandbox is typically off by default and so there are plugins that can have full host access with no boundaries they can do pretty dangerous stuff right so for example you could have credentials there you probably know that people has

been buying like crazy make minis you know that kind of sold out just to run this kind of thing, right? So what does it take to run something like this in Kubernetes, right? And so there are approaches today for sandboxing grimson boundaries around and this is pretty much what we want to sort of think about one of the scenario could address something like what I show here.

It's not of course all you need. There are still other pieces that you need around but just know this at least covers some of the problems that you have right for example um on the data plane the agent run inside of VM. So the host is unreachable. On the control plane we have a per tenant API server limits the blast radius. Um and then on the network

side there is UDM that allow to segment the traffic. So this just one example that I think is is cool to think about. Um so uh to wrap up we presented a new approach to Kubernetes multi-tenency. This provides these three layers of isolation we described. These are all based on it's more like a pattern is all based on open source. You probably can use also different open

source project but consider this more like a pattern and we also saw that experimentally that the framework overhead is minimum. So we are talking about singledigit microcond latency. For the next steps we plan to measure overhead across additional workload profiles including real GPU inferencing with larger models and we continue also development of this console that just show to have some patterns there like this. Um some additional

resource I want to provide at the end uh if you want to check out know one is about LLDD that is one of the workload we show there is also another project we're working on is called kajent.io is actually about running agents in a kind of a secure environment with agent identity spire identity and we're doing pretty cool stuff there including sandboxing and some of the things

we describe here. Thank you. Thank you. And I think we open up for questions. There are microphones with numbers. I think there is one two uh there is three on the back. If you have any question, please come to the microphone. um you don't cater or this this model doesn't have any provisioning for uh local discs local attached NVME drives that then are mounted directly in the

uh >> Okay. So you're talking about separation at the level of the the storage. >> So you don't have to go over the network but you have direct IOPS complete full >> IOPS. That would be one way or maybe you can attach uh through that se segmented network if you want to have the isolation is a network drive. >> Hello thanks for the talk. I have a

question. How do you prevent noisy neighbors problem? Because Q is just a pod right? you have they also run on the host environment in your cluster right >> you want to take this one >> uh so here the the the the tenant pod don't don't have access to to the fully resource in the closer so because it's running the workload is run inside of the cube VM

so the the the resource that the tenant can consume is only those that the cluster has been predefined that the VMs for the size of of the V VMs that attri to for the tenants. >> Did I answer that answer your question? >> Thank you. >> Thank you for the presentation. How is this comparable to hosted control planes by Red Hat? >> Actually, this is a pattern

and indeed we work with that technology in the past. So you can consider that if you want to do something just using plain open source technology you can use something like this. But then if you want to have a supported product you know that's probably where you want to go the control planes. Of course the other difference is that if you want like a full open shift

cluster you know you're going to use OEP control plane. If you want more like a lightweight kind of you know control plane this probably something that can you can set up very quickly. >> Okay thank you very Yes, we we we are using cube flex as the host control plane, but of course you can use other technologies as that as the one that you mentioned for example.

>> Welcome. >> Hi, thank you for this talk. Um I saw that you are using only one control plane. how you mitigate host host going down and to be ha >> okay so you're talking about the hosting cluster that is hosting the control plane if that goes down >> your tenant cluster also only from your demo only have one control plane and one API server how you

prevent the >> so I think the idea is um so you're talking about something like HA for example you're talking about the full hosting cluster going just your control plane because for example if your character go down >> okay just the control plane if you don't care about the redundancy uh you're talking about control plane the host control plane right not the host not the cluster that

is hosting right >> no it's a cluster that mean host >> okay so the cluster the host cluster right because I mean there are two scenarios could be that the specific control plane that you are hosting can crash the pod can crash right and then kubernetes is going to restart so for a few for a time it probably unless you're doing HA that is another option right

you could do also HA you could have two actually we have done that in host control plane that is the product >> yeah so just just to so in this demo we show just one control plane just for >> but but this this those control planes have been managed by cube flex so you you can spin up more control planes to to have a but for for

just for the purpose this demo we just just use one but just using one >> generally speaking the power crash and it's going to continue pick it up, right? But it's going to have stand out time. >> Okay. And second question, so you are still have pressure for example on disk because all your VM are on the same host actually and the line storage is the same.

>> Uh so sorry could you repeat your >> So your storage itself is still >> storage >> is still on the same disc from what I can uh understand. So you still have >> so so this going to depend how the cluster me wants to set up the maintenance environment. the cluster me could have different discs on the uh volumes mounting on the cluster that to duplicate

for for each for each tenant just depend on tenants the clos configured environment but yeah for simplicity in this in this talk we we didn't yeah got into much that details >> okay thank you >> thank you >> thank you very much for the presentation this was very nifty setup that you made there I was actually curious um how did you handle the top level Kubernetes cluster,

which uh distribution did you choose? What did you decide to go for regarding the operating system? So that the first layer that runs on bare metal on the machine itself, how did you design that? >> Uh we we we are using uh uh the the vanilla kubernet cluster uh and we are using Ubuntu as as as the the VM. So we we want to uh we don't

want to have any dependency on specific OS or distribution. So, so this pattern you can use even use uh if it's a kubernet cluster even a variation you you that support those technology of technology