How To Break Multi-Tenancy Again and Again ...and What We Can Learn F... Lorin Lehawany & Sven Nobis

This talk focuses on the challenges of securing namespace-based multi-tenancy in Kubernetes environments, presented by security analysts Lorin Lehavani and Sven Nobis from ERNW. They introduce their research aiming to evaluate whether industry best practices are adequate for isolating multi-tenant clusters. The speakers outline three identified exploits, detailing issues with cross-namespace references in custom resource definitions and annotations, which have implications for existing systems like Kubeflow and Traefik. They also discuss a man-in-the-middle vulnerability involving the Istio service mesh, demonstrating how attackers can intercept traffic by exploiting namespace configurations. The session concludes with a proposed methodology for practitioners to assess and mitigate these security risks in their own clusters.

All right. So, welcome everyone uh to our talk. Um today we want to show you how we broke multi-tenancy again and again and what we can learn from it. But, before we start, who we are. So, my name is Lorin Lehavani and this is my colleague Sven Nobis. We both work as a security analyst at the company ERNW, a security consultant company uh located in Germany. And

here in the background, you can see the rest of our team. So, as a security analyst, we work with different customers in uh yeah, different sectors. And we've seen that namespace-based multi-tenancy is quite often used because it can save costs compared to cluster-based multi-tenancy. And what we've also noticed is that hardening such clusters from the security perspective is quite challenging. So, we decided to do a research

with the question, is industry best practice hardening enough to isolate namespace-based multi-tenancy? And we found some problems or multiple problems which are not well studied yet. So, in the first half of our talk, we multi-tenancy in various ways. In the second half of our talk, we want to show you what we can learn from it by introducing our methodology. And lastly, yeah, the conclusion with some takeaways.

So, let's start with the first part part, breaking multi-tenancy. Before we start, what is namespace-based multi-tenancy? Many of you already probably know. Uh here we can see, yeah, a implementation of a name space based multi-tenancy cluster. So, we can see here different tenants, tenant A and tenant B, each have access to their own name space. And all of those tenants share one cluster. And here you can

see the lines, yeah, they represent communication happening within the cluster. So, in the control plane layer, it represents communication for managing the cluster, for example, interacting with the Kubernetes API. And on the data plane, it represents communication between workloads, for example, that have been, yeah, deployed by tenants. So, what we found is, yeah, we could the isolation in such clusters in many And what we've also found

that the best practices do not protect against those problems. So, in our today's talk, we want to show you three exploits that we found. Two of them on the control plane layer and one on the data plane layer. So, let's start with the first exploit. Insecure cross name space references in custom resource definition. But before we start, what is cross name space references? Here we have a

simple where on the one hand we have name space one, name space one have a resource, a custom resource definition resource with the name example. On the other hand, we have another name space in this cluster, name space two, also custom resource definition resource, and what we notice here, it another resource from a different name So, why is this a problem? And to understand that, we present

you now our first real world scenario that we found in Kubeflow, where we could exploit this. So, what is Kubeflow? Probably many of you already know that also. Kubeflow is a CNCF project that offers a machine learning and AI platform on Kubernetes. And here we can see the Kubeflow dashboard. And what we see here is every user has a Kubeflow profile assigned, and uh to every Kubeflow

profile it is assigned a namespace in Kubernetes. Okay? This is a classical namespace So, every user here uh can uh create their own machine learning workloads, for example via notebooks or pipelines, and can execute code within those workloads. click here on connect. And what we see here is that Jupiter notebook of that user. So, let's have a deeper look at the terminal that we have access to

as a And the first thing that we want to do as hackers, okay, who am I in this um in this context and what permissions do I have? So, we do the first thing, kubectl auth who am I, and I see the user have a service account uh assigned with the name default editor. Okay? As next, I want to see what permissions does the service account has.

So, we list those permissions. And we see here some well-known permissions like the uh yeah, config maps or deployments. But what we also see here that the user has also permissions from Istio. Meaning the user can uh create a virtual service here. This is quite interesting. And let's see why. But before we start, a quick recap what this Istio and virtual services. Probably many of you also

already know that, but a quick recap here. Um so here we have an end user who trying to access a service from the outside. And this request first is forwarded to the load balancer. The load balancer, yeah, forwards the request to the Istio gateway. The Istio gateway here exists in the gateway namespace. And the Istio gateway now routes this uh request to the service via virtual services.

And what we notice here is that the virtual service exists in the workload name namespace, different namespace than And uh yeah, the workload namespace here the user has access to. So. Let's go back now to uh our example or our theory from from earlier. And now we apply the Kubeflow resources uh yeah, in the in the theory. So what do we see here now? On the one

hand, we have the Kubeflow namespace. Within this Kubeflow namespace, we have now the Kubeflow gateway. On the other hand, we have the user's namespace, where as we learned just now, the user has permission to create their virtual service. And within the virtual service now, a user can reference the Kubeflow gateway, which exists in a different namespace. Okay, but still, why is this a problem? And to understand

that, let's introduce now our attacker and see how he can how the attacker can So, the attacker uh deploys the following virtual As we learned, yeah, they can uh create uh such a uh such a resource in the cluster. And the attacker now references the Kubeflow Gateway. Then the attacker overrides the favicon from the Kubeflow dashboard to their own chosen favicon, yeah? uh routes the uh request

to their own controlled pod. So, what happens now, you can see at the top the Kubeflow dashboard. And what we notice now is that the favicon has been replaced by the attackers' uh chosen one, yeah? Okay, this is cool. But what is the security impact of it? To understand that, let's have a look now what we just did. Um yeah, the Kubeflow user accesses the Kubeflow dashboard.

And the request is forwarded to the attackers' controlled pod via the Kubeflow gateway, as we just seen, yeah? And what the attacker now can do with this request that um he ha- um that the attacker ha- has in the attacker's pod. The attacker can log those requests, and they have access to the cookies of the Kubeflow user, meaning the attacker can impersonate that Kubeflow user. And the

attacker have complete control over the Kubeflow user session. Meaning they can do whatever they want in the in the name of uh uh that uh Kubeflow user. This also applies to every Kubeflow user accessing the Kubeflow dashboard. Yeah, this is a problem. we did a coordinated disclosure together with Kubeflow and the issue is already fixed by Kubeflow by removing the issue added permission from the service account.

And at this point we would like to thank Kubeflow project maintainers of Kubeflow for fixing this issue in a timely manner and also for the great cooperation with them. We've also learned a lot about Kubeflow. Yeah, so we learned until now that insecure cross name space references in CRDs is a problem. We've seen this in Kubeflow. In Kubeflow well the issue was fixed. However, the issue exists

in various other projects as well. We reference here an excellent research paper which was just recently um released it talked exactly about those kind of problems and they show that this issue exists in a lot of other projects as So, here we are at the end of our first Now, let's move to the second exploit on and show you now how we broke multi-tenancy again. This time

via annotations. Let's go to our familiar cluster from And what we notice here the difference is that referencing resource here is done via annotations and the referencing resource here is a built-in Kubernetes resource. This case the As we learned earlier, this could be yeah, problematic. And to understand the impact of this, let's introduce our second real world that we have found in traffic where we can exploit

this. Traffic as many of you already know is a popular open source HTTP reverse proxy and load balancer. Let's have a quick look how it works. In traffic, so we have a request coming It goes first to the traffic proxy. The traffic proxy forwards the request to the service via ingress. Here we just uh removed the the ingress just for simplicity. And the um service then forwards

the request to the corresponding pod. What we notice here, the service can reference a uh service transport. What is a service transport? A service transport is a custom resource definition from Traefik. And um with that you can configure how Traefik connects to the pod. In our example, via mutual TLS authentication. Still, what is the problem? Where is the impact of this? Now we learned the Let's introduce

now our attacker again. We simplify now the uh victim's namespace to only the relevant components. So, the service transport resource as well as the pod. And introduce our attacker, which exists in their own namespace, and the attacker has permission to create a service. What the attacker do at this point? They reference via annotations the service transport of uh service transport of the victims, And what they do

as next, they forward the request to their control pod, so-called pod. And uh the request is then channeled to The attacker can now utilize the Traefik's proxy to do the mTLS authentication against the victim's pod for them. So, the attacker can use the client secret um which exists in the victim's um uh resource without actually having direct access to it. So, yeah, this is We also did

a coordinated disclosure here together with Traefik. And Traefik is currently fixing this issue. However, they updated their security documentation to raise awareness to their users about this issue, and they recommended using the gateway API. At this point, we would like to thank Traefik as well for for the security update security documentation update, yeah. And also for the great cooperation with them. We learned also a lot about

the Traefik while working with them. At this point, we learned that insecure cross-namespace references and annotation is problematic. We've seen this in Traefik. the impact of this could also go beyond the scope of the cluster. And to understand that, we have here a If you give your tenant the permission to create their own ingress resource, they can reference via annotations, for example here, GCP compute certificate, which

exists on the cloud provider project outside Meaning that your meaning that the tenant can basically define any certificate which exists in the cloud provider project, not only their own. So, yeah, this is also a problem. Yeah, at this point, we are at the end of our second exploit. Now, let's show you how we broke multi-tenancy again and again. And for that, I'll give the talk to Sven.

Thank you very much, Lorraine. And yeah, if you haven't enough from annotation-based problems, just take a look at Ingress Nightmare. There are a lot of more of those kind of we focused until now on the control plane, and now we want to move over to the data plane. And for that, I bring back our friend Istio. And now we're focusing on the service mesh. Here on the

left side, I have a simple example of a service mesh. Uh so, we have our victim's namespace, and there's running a pod with a workload container. And if the workload container wants to access the network, it's every request is intercepted by the Istio sidecar. So, we have sidecar mode in this example, but it would also work for ambient mode. So, the request is intercepted, and this Istio

sidecar decides where to route the traffic. The workload container wants to access example.com. So, yeah. Istio sidecar says that's in the in the internet, and send it to the internet. Everything is fine until now. But, just until our attacker comes into play. So, on the right side, we have our attacker namespace, and he deploys a virtual service. We already know a virtual service, but this is a

special kind of virtual service. Why is that a problem? Let's look into it. We have here a virtual service which is deployed to the service mesh. It's not deployed to a gateway, but to the And the attacker says, "I'm the host example.com." And when the Istio sidecar wants to route traffic now to example.com, he sees, "Ah, there is an virtual service running by the attacker to example.com.

I'll route the traffic to the virtual service instead of the internet." And then the attacker can route the traffic to his own attacker control pod. So, this is a classic man-in-the-middle attack, so he can intercept traffic. There are some limitations. So, we cannot only deploy virtual services on arbitrary hosts in the internet, but also to hosts to destinations in the service mesh. So, we can rewrite destinations

in service mesh. And there are some limitations uh for this kind of attack because if the attacker can route it to his attacker controlled pod, and sometimes he wants to yeah, intercept the traffic to and route the traffic back to the original service to see what the victim is talking to the destination service. And in that we have some limitations. So, the attacker needs to be able

to access the destination service. So, an authorization policy would prevent that the attacker can reach the destination service. And also, the destination service would see that attacker is accessing the destination service, not the victim anymore. So, we uh don't have a full break of mutual TLS authentication an issue. We can just intercept traffic and yeah, look into it and route it to our own service or to

other destination services. But this is a real real problem in some deployments. So, we also did a coordinated disclosure together with Istio. So, the Istio maintainers consider this issue as expected behavioral. It's a preferential experience trade-off because this API square design even before Air g in Kubernetes existed. And if they change it, it would be a breaking change. So, they recommend to use the Gateway API as

a replacement if you use namespace based multi-tenancy. And it's also a thing which we seen as a good yeah, replacement. Gateway API offers a secure way to cross namespace references and also offers a secure way to deploy something to the Um together with Istio, we published a security note and blog post to address this issue. So, everybody who using Istio can address check their cluster and try

to address those issues in their cluster. So, a big thank you also to the Istio project. We had a few discussions with them and also they helped us with the blog post. It was very productive collaboration and we are happy that it's got published just today. Um we learned a lot of during our research, and now we come to the part where we hope that you also

can learn something for it what we can learn from it. We developed a methodology uh to find exactly those kind of issues in your cluster. And for that, I come to our three-step methodology. The first one is the use step. Do I use namespace multi-tenancy? You might think, "Ah, it's easy. I know it." But we have some examples where it might be not so easy to detect

this. And then the assessment part, how do I identify potential weaknesses in my cluster? And the third part, to address them. So, the first part is do I use namespace multi-tenancy? And I have two examples for that. So, where is namespace multi-tenancy commonly found? The one thing you might already think of is like, "Ah, I have multiple teams in my company, and and I give them direct

access to the Kubernetes API, and they can deploy in their namespace applications uh into the same cluster, and they communicate with each other." Um and you typically share a level of trust with your developers. They hopefully don't do nasty stuff, and yeah, sometimes maybe a developer gets compromised, but it's not so so such a big problem because you have also other security measures in place or controls

in place to prevent this kind of attacks in your company. But this is a different kind of multi-tenancy where you don't have full trust, likely. You have multiple actors, often customers, and give them indirect, but sometimes also direct access uh to your cluster. Some one example we've already seen was Kubeflow. You've seen that the users had through the service account, but also the dashboard offers indirect access

to the cluster because you can deploy Kubernetes resources by creating a Again, virtual service, sorry. Not a virtual service. Uh I mean, in this time um a notebook, and uh or a pipeline machine learning pipeline. And then you give then you have some kind of indirect access. But other cases where we have this kind of problem are CI/CD pipelines for instance. You might deploy your production environment

through a CI/CD pipeline and besides this in the same environment also there is a pipeline which runs when somebody does a merge request. Maybe even from the internet if you have a public repository. True we had such problems in the past, very serious problems where somebody can make a pull request and then a pipeline runs and this is the same environment as your production code where where

we sensitive secrets are accessible. And you want clearly want to isolate those kind of things and also another example is scripting capabilities in applications. If you give give the customer the ability to yeah, create reports with some code or if you give them the ability to run workflows, you have they have code execution at some point and you also want to separate those tenants from each other.

This is typically untrusted. You don't trust well, don't fully trust your customers and you also don't trust any random user from the internet. And also as we've seen with this last examples, this namespace multi-tenancy or multi-tenancy workload multi-tenancy is more non-obvious. So it's not so clear that you have it. If you look at the Kubernetes architecture diagram, you only see okay, there is cluster running but you

see that multi-tenancy only if you dig deeper into your application and see ah okay, there is indirect access to my cluster from tenants and customers and I need to separate them. So if I know that I need to separate um I need to identify potential weaknesses. And the first step is to apply hardening. I hope you already have applied industry best practices to your cluster, have something

like a pod security standards running and enforcing them so you have a certain level of security in your cluster. Yeah. If I have this this already done, I can go over to the next step to identify all of my components running in my cluster. This can be components provided by your platform but also components you install in your cluster like Istio on And then ask the question,

which resources are in control or potentially in control of a tenant? And if I know all resources which are in control of a tenant, I can access assess those resources. And there can be two things we have to consider here. The one thing is the control plane interaction. can a customer do on the control plane? Do I give them direct or indirect access to the Kubernetes API

where they can create resources to this that belongs to this component and how do can the customer or the tenant interact with that? And the second part is on the data plane is a little bit harder to assess because you need to ask, okay, what can a tenant do on the data plane with those resources or those components? Because can they interact to to the service of

a component? Can they reach the service in the service mesh or can they interact with the service mesh or even does the component provide something on on the container level and can the tenant interact with this component on the container or on the in the container through the container environment? Yeah, and if I know those interactions, we need to evaluate those interactions. Does one of those interactions

of those affect components or resources outside of your name space, affect other tenants, or affect the platform itself? And for that we can use the CIA trade. So, we can ask, does it affect the confidentiality of another tenant or platform, the integrity, or the ability, for instance, through a denial of service attack? And if I find such interactions, I need to address them. And the first thing

we seen with with traffic and also with Kubeflow, we reported the issue to the vendor and developed together with the vendor a fix for that. And you can just apply the fix, everything is fine. This is pretty easy. But sometimes, like in the Istio case, that's not possible. You It's a breaking change. You won't do a breaking change. And for that, you can, beside using the gateway

API, for instance, also uh use existing admission policy sets. So, if you say, "Okay, I cannot migrate to gateway API yet because it's not uh possible in my environment, you can also um apply an existing policy set. And for instance, if you use Kyverno, Kyverno's an excellent repository of policies, and some of them fix those namespace-based multi-tenancy problems here. But not every policy can be found on

Kyverno. Sometimes you might have um yeah, you might have uh yeah, um uh yeah, sorry. You have to talk to define your own. Sorry. Uh sorry. Uh sorry. We might have um yeah, domain-specific problems, things you you you very specific to your cluster or to your setup to your multi-tenancy. And um for that, you can, for instance, implement is your own admission policies for admission controls. And

here we have back our And um we have we can say, "Okay, I have been uh I do not allow my tenants to deploy something to the service mesh. I also do not allow my tenants to reference something uh in another namespace." So, let's say they can use a a gateway in another namespace, but allow them to use their own gateways or gateways I provide to Um

this can be an option. But if you say, "Ah, no, I my tenants need to deploy something on the service mesh because this is a use case." I have also a second option for this problem and say, "Okay, I disallow to override any host on a gateway. I disallow also arbitrary hosts like example.com or hosts I uh in the service mesh that don't belong to the tenant

and allow only certain part whitelist certain part of host the tenant can use. And this would be a policy you can write to address those kind of problems we showed before. With that, we are also at the end of the methodology and at the end of our talk. And I want to provide you also at the end of our talk some takeaways. With our talk, we wanted

to increase awareness on because it's hard to get right. I hope you've seen it in our talk. And also its presence is everywhere and I hope you've seen this too. And please invest time to assess implications if you use namespace based multi-tenancy and you can use our methodology as a guideline to assess your cluster. And we have here on the right side a QR code to our

methodology. It's provided in much more details than I can show you today. We only have a half an hour. And there are some more examples details about it. And for that, thank you very much for your attention. If you want to talk more about Kubernetes security with us, just reach us after the talk. You can ask questions and if you cannot reach us now, feel free to

send us an email or add us via LinkedIn. We're happy to talk a lot about security security. It's a very interesting topic. We focused our research on and we couldn't also show all of the problems we found. There are also other implications we cannot show today. And we post about every issue we showed blog post on our blog insinyur.net. The issue blog post just released. The other

problems we released once the fix is fully deployed and we got the CVEs. So, thank you again for your attention and have a nice day at KubeCon.