When Multitenancy Goes Wrong: A Deep Dive Into Kcp’s First CVE - Marvin Beckers, ClickHouse

This talk, led by Marvin Beckers, focuses on the first official Common Vulnerabilities and Exposures (CVE) for KCP, a multi-tenant control plane for Kubernetes-like APIs. Beckers explains KCP's architecture, emphasizing its role as a standalone API server that supports Kubernetes resource models. He discusses the concept of virtual workspaces that allow for logical separation of APIs and objects within KCP, enhancing its multi-tenancy capabilities. The speaker details a security vulnerability found in the authorization checks related to API bindings and actions within these workspaces, which could lead to unauthorized access. Beckers outlines the implications of this CVE, the steps taken to rectify the issue, and the importance of having a robust security process in place for open-source projects.

So, thank you very much all for coming. Um, today I want to talk about the first official CVE that we had in a CNCF sandbox project that's called KCP. So, before we get started, my name is Marvin Beckers. I am a maintainer for KCP. I am also a cloud software engineer at Click House. Um, this talk was supposed to be with a former colleague of mine, Marco.

He couldn't make it um like on the last minute. So, I am doing 50% more of this talk than I anticipated. So, bear with me. Um, but we'll get through it. So, um, don't worry. So, maybe before we really get started, I want to ask a question here. Um, who here knows what KCP is? Could you raise your hands, please? Okay. Uh, thank you very much to

everyone else who came who has no idea what KCP is. Um, I have a section for you, um, where I'm going to briefly talk about KCP and try to explain the project in h five to 10 minutes. Um but we'll see because usually this is like a full talk to just explain KCP. Um we'll we'll see but we'll you know give you a rough idea of what

this is. So let's get started with that part. So this is kind of the tagline that I like to um like bring up whenever someone asks what KCP is. KCP is a multi-tenant horizontally scalable control plane for Kubernetes like APIs. And if you're confused what that means, you're probably not alone. It's hard to explain a whole project in just one sentence. Um, but let's focus on some

aspects of what I just showed you. The first one is like Kubernetes like APIs. What do you mean? Um, the very much important information to know is that KCP is not Kubernetes. It's not like a controller or an add-on that you install on top of Kubernetes. It's its own standalone API server that is however based on Kubernetes. So it speaks the Kubernetes resource model. It's also using

a lot of libraries that are also used to build the cube API server, but it's a distinct entity because it speaks the same language as the Kubernetes API. However, you can actually use clients that understand cubes API language, the Kubernetes resource model. You can use clients like that against KCP. So here you can basically see that I just pointed KCP um to sorry I I pointed cubectl

to a KCP instance and the request comes back like it happily tells me there are no config maps in KCP um but it happily served that request to cubectl and responded with there's not nothing there but the client works like it it can connect so what is KCP typically used for what is our use case that we try to position it as as a project. Um, KCP

is a building block, but it's a building block for things like internal developer platform. So, platform engineering has been something that's been going on for quite a few years as a topic. Um, but you can also use it like more internally than that because you still have like a I guess um consumer and uh service provider relationship in an IDP. Um, but you can also use it

to orchestrate like a global ZAS or pairs or whatever kind of solution. Like Kubernetes is um a great control plane, but it's mostly focused on running like specific Kubernetes clusters and specific regions. Maybe they're multi-AZ, but they tend to not be multi-reion. A KCP is a control plan that you can put on top of that to orchestrate things that are running on your different regions, which might

be Kubernetes, which might be something else. You can even make go like one step further. um simply because all the tooling for talking to Kubernetes and its API it already exists. So you can use as I told you about cubectl you can use like an open tofu provider you can use all the different tooling. Um, so instead of building your own bespoke REST API, you might just

expose KCP to your customers and allow them to order through interface that they might already know because they've worked with Kubernetes in the past. And that's kind of extending the IDP idea to uh external customers as well. So that kind of means KCP is a very important component um and a very like security critical one. Um and basically we need to take that topic very seriously and

we do and that's kind of what I want to tell you today about. Before we jump into the actual CVE there's one thing that I need to tell you about um like one of the many concepts in KCP and that is workspaces and virtual workspaces. I told you it's a bit like the Kubernetes API. it speaks the same language but what it adds on top of the

Kubernetes API is for example a multi-tenency um system so basically instead of being one Kubernetes API endpoint that you talk to KCP exposes a like kind of like hundreds or thousands of endpoints that look like they are their own distinct Kubernetes API endpoints. So it's something like where you would be running like hundred or maybe a thousand cube API servers. Maybe they are like lightweight but they

would still be distinct entities. KCP is not like that. It does that with logical separation. So you have one API server, the KCP API server that is capable of serving many different API endpoints. And because KCP is built for sharing APIs, that kind of means um you have a role of an API provider, a role of an API consumer. And these tend to interact across the boundaries

of a workspace. This is where the logical separation is purposely um like broken down a little bit. So an API consumer can consume an API that an API provider is providing and that's mostly in different workspace. So it's a different part of your multi-tenant system that is allowing you to provide an API um to a consumer. So you can basically say okay um consumers are creating Kubernetes

resource model objects across their workspaces and the API provider is expected to reconcile them. This is like a Kubernetes operator just with you know multi-tenency added on top of it. But how does the provider get access to these objects? Like the provider needs to know what is in those objects to be able to reconcile them against an external system whatever that might be. And for that we've

built this concept of virtual workspaces. And virtual workspaces are also kind of cube API servers in the sense that they use a lot of the libraries underneath as well. But in um in comparison to like a real API server, they don't have a storage back end. So they don't have at CD. What they're doing is they have a storage back end implemented that actually connects to KCP

itself. And that way we implement a view-like functionality in virtual workspaces. So instead of seeing everything that is in a workspace, if you go through a virtual workspace, you get a filtered view of all the data that kind of pertains to you. Um, so saying okay, I'm an API provider. I've exposed an API and if I go through the virtual workspace that is matching, I will only

get the objects that I am supposed to reconcile. So this is kind of how we implement multi-tenency for the API Under the hood, we have a couple of like we have a framework to build virtual workspaces and one of them is the dynamic virtual workspace that is something that is built on top of the generic API server library coming from cube directly. There's a couple of interesting

things when you instantiate a dynamic virtual workspace. One of them is that you have a way to parse information from the URL. Um, another one is that and this is kind of the central heart of a virtual workspace. Um, this one here is the API set management and an API definition set is something that I can't get into in all detail, but it provides the virtual workspace

with all the information that is necessary to serve an API. So that means um it not only tells the virtual workspace what the schema is. So like a CD like definition of you know what the object should look like um it also tells it how to find its storage and we provide some libraries to do what I told you earlier not use any real storage but use

the KCP API server as a storage layer. So it's like kind of an interesting like um you had the virtual workspace and that um has a filtered informer down to KCP to provide the information back and then if you are familiar with um what the like um yeah what the flow of a request through a cube API server is um you always have like authentication, authorization, uh

admission um So there's an authorizer that you can prepare and give to the virtual workspace. So it can implement you know yeah authorization decisions and you can also do some admission. So you can do mutation and validation of your request as a separate step step. Um so with all of that said this is kind of the stage. This is kind of what KCP does and we found

a CV in that and I'll tell you how we found that. um, one of the most important thing, and this is kind of where we're trying to, some of this is obvious to be honest, but if you're a maintainer for a project in the CNCF or maybe another open source project or just an internal component, um, you need to stay curious like never trust code that you

written like I don't know three months ago, six months ago, like maybe last week. Um, you never don't ever let down your guard. It's it can always be that you either added this a long time ago that you've added something or that you have added a regression in a recent change. So it's it's something that you always need to be aware of. Um in our specific case

we had this ticket here. So in this ticket we wanted to extend the functionality of virtual workspaces. So before we did this um the these are the kind of objects that um construct the contract between an API provider and an API consumer in KCP. So there's an API export and the API export is something that the API provider creates in their workspace and says okay here are

the schemas of APIs that I want to provide. So let's say we have like in this example we have widgets but let's say you provide databases as a service then you would have a database object database schema that you're exposing through an API export and then there is the consumer part um and that's the API binding. So the consumer in their workspace they create um well this

API binding object which references the API export and also accepts a couple of things like permission claims. So permission claims are an extension of the virtual workspace as well. It's not only about providing new API schemas to user to users. It's also about accessing already pre-existing resources in a workspace. So say for example the database object that I told you about that might have a reference to

a secret. So that means you as a service provider, you also need to be able to read secrets, right? Because otherwise you can't look into that secret um or you can't write that secret, you know, whatever. And what we wanted to do was basically allow API um to have more choice because at the moment at the time of this when this issue was created, the only thing

that you could do was basically give full access to resource. So either you can do everything with a secret or you can do nothing with a secret. So this is what we wanted to add. Simple enough, right? So we wanted to restrict this by verbs. You know this from Kubernetes airbug already. It's, you know, not something that we've come up with. It's just extending uh on a

known airbug mechanism and adding it to what we've built. And then we looked at docs because well you know this thing like the virtual workspace in question was written a while ago. Um so we wanted to first check okay you know what is actually happening. Um so this is kind of the request flow like very like roughly sketched out but when you talk to a virtual workspace

as a service provider you go through authorization you go through admission and then your request gets handled and the request handler was basically responsible for fetching the data from the different workspaces and giving you this aggregated view of the data that you are allowed to access. So doc said okay the authorizer checks API bindings um to basically make sure that the provider is actually allowed to access

this because yeah that's kind of very important thing to do right and we thought great okay so what we're going to do is we're going to extend this existing authorizer because the existing authorizer kind of already did what we wanted it to do but now we need to needed to make it aware of verbs. So like in when the request came in and authorization happened, we also

needed to check okay is the verb that um the API provider is sending to us is that also something that uh had the API consumer has accepted in their API binding. Straightforward, right? Yeah. And the doc said yes. problem is this check didn't exist. So that is when we started to stumble over this and we're like oh no because this check is kind of important as I

just told you it's kind of making sure that the API consumer has accepted that you as an API provider gain access to their data. Yeah, this is how we got the CV basically. So at that point we were like okay there's something wrong here. um we need to fix this, we need to activate our security process. And while KCP had a written security process at the time,

this was kind of the first time we really had something we're like, okay, we need to exercise our approach process to the fullest now. So let's look a little bit more at that CVE. How did that come together? Like how did we investigate and what kind of conclusions did we draw from it? this is kind of the flow that I showed you earlier but what was actually

happening. So there is an authorizer sorry authorizer in that virtual workspace. But the problem is it only checks against the API export. So it only checks on the API provider side. It doesn't check on the API consumer side like it was supposed to do. So you needed access to an API export. there's some additional verbs like like additional checks in place there which kind of like unfortunately

we don't really have time to cover. Um but yeah the authorizer was there it was just doing half of its job really unfortunately. Um so how did we not notice this earlier? This you know sounds not great right? Um there was another mechanism in play here which um kind of buried the ledge on this for quite a long time. So I told you earlier and we can

go back here. the request handler that's coming from an API definition set and that is a scoped view of like that's based on a scoped informer into KCP and what we basically did in KCP was to and like how to support this is um this informer has a label and like a label selector and the label selector is applied by a controller in KCP to objects that

have been claimed or created through an API like it's a a little complex I'll admit, but it's basically um the kind of eventual consistent model of KCP. So, um there's a control in KCP. It looks at all the API bindings and all of these API bindings for all of these API bindings in the second step. It looks at resources that are related to that API binding that

are either created from it or that look like they have been claimed through a permission claim and then it labels that and because of this labeling mechanism it shows up in the storage layer of the virtual workspace at all otherwise it wouldn't see this like you can't list or get objects that you can't that you don't have access to because they are labeled this way. So that

meant when you did a get or list through the virtual workspace um you got only the objects that you were supposed to access because here the API binding was correctly in play. This is what we want like we want from the API from the API consumer side we want to have a check if this is actually has been like claimed or created through it and only then

it's exposed to the API consumer. So what were the applic uh the implications here? So because we didn't have this check um we started to realize okay there's an attack vector here um so anyone can create an AP so no not anyone everyone who has the necessary permissions to create an API export in any workspace can create one um then they gain access to the virtual workspace

endpoint and because this check was missing that meant you could do certain operations through the virtual workspace whether the consumer consumer has um had accepted or like even bound to your API or not and that's a problem obviously. Um the good news is this had some limitations to it. Um I told you about like the most obvious one earlier or just a minute ago like list and

get operations didn't show this. So you did a get you didn't get any objects back because like the storage layer of the virtual workspace was taking care of it. But the oversight that we had in the authorizer meant that create and delete operations were possible and you could create and delete objects into arbitrary workspaces as long as you knew their logical cluster ID. So that's not an

obvious ID. Um I guess you could brute force this, but it's not that you know if you'd only have a very limited access to the KCP instance that you're talking to, you don't know like who's behind these ids and um whose like workspaces there. So there's a certain limitation to this and also you can't claim any objects uh v sorry any resource v permission claims you can

only claim um uh built-in resources like config maps or secrets which you know fair enough that's that's bad enough but you can you cannot read them you can't read a secret with this vulnerability um also we so this is how we did the initial analysis um we used the CVSS scoring mechanism to basically come up with a score. Um we went through that um like we were

like okay this is bad. Um but going by how we understood the mechanism it ended up with a 4.5. So that's you know kind of midle I guess on the CVS scoring. So I'm not sure if anyone has already kind of caught up to what the problem is, but I told you that this permission claim mechanism allows you to claim any built-in resources and then create and

delete them through this vulnerability. Problem is cluster roles and clust bindings are built-in resources. So we ask ourselves okay is there an escalation mechanism that is in this and yeah yeah there was so this means that as long as you had necessary permissions uh sorry yeah necessary permissions to create an API export uh or to access the virtual li I mean yeah you needed that permission um

and you knew a logical cluster ID which again like you could only guess or have like broader permissions to see it um you could create a cluster admin cluster role binding in that and suddenly you had access like full access to the workspace and that meant we had to go back and look at the CVS uh CVSS scoring and said oh no this is like almost a

10. Um, I think the only thing that saved us from a 10 was basically that you need privileges. Like this is not something um that affects a KCP instance that's exposed to the internet. Like you can't with anonymous credentials um go in and um exploit this, but it was you know bad enough. So honestly the fix itself um was rather trivial. So once we knew there was

no authorizer um we extended the authorizer chain. Um I showed you very much earlier where in a dynamic virtual workspace you can do that. Um there we added a new authorizer. It looked at the API bindings. It also looked at the verbs um while we were at it I guess. And um that then allowed us to fix it. So we used GitHub um extensively for this. Um

and this is kind of where we're getting to. Okay. uh what kind of information do we want to share with you know everyone who is maybe um maintaining a project before its first CVE. Um the security advisory feature of GitHub was very helpful to us because we could create a private report. Um we could triage it there. We could add comments. Um we could even start a

private fork that allowed us to work on a branch together. And none of that was, you know, visible because we needed to have a coordinated disclosure of this, obviously. So, um, because this was such a highly scored CV, um, we also pre-announced it. So, we have a mailing list for all of our users, um, and we sent out emails. We also sent out messages on our Slack

channels that we had for the project making sure that everyone or hopefully everyone was aware of that um because they needed to prepare some time in the coming days to apply a patch that we were going to release. Um this pre-announcement didn't include any details. Um, you might have seen this kind of practice from other CNCF projects and it was kind of very important for us. Um,

because we wanted to make sure that people know something is coming and we're not I don't remember when we actually released the patch releases. It was surely not on a Friday, but you know, whenever you kind of drop this, the talk the the clock starts ticking for all your users. So, it's very important to make them aware. um there's some very nice uh like feature set in

security advisories in GitHub. Uh one of them is basically that GitHub itself is its own CNA. So all that we need to do was fill out the report and kind of explain okay this is the vulnerability and GitHub took care of assigning the CV for us. So this was like a very frictionless process for us which was great because well we were focusing on dealing with the

actual thing um and making sure that our users knew and all of that um and GitHub basically took care of the um yeah bureaucratic aspects of this. So very thankful for that actually. what is kind of the point of this talk and this is obviously like a little bit of a well um embarrassing vulnerability in a way um because it's such a thing like we are we

want this multi-tenency API control plane and there's like this central kind of vulnerability in it that can be exploited but the reality is that uh yeah all all code has vulnerabilities we just don't know about some of them and we kind of want to make sure that even if it's embarrassing you go out and do the right thing you're transparent about it, you tell people about it

because this is important like this is how we learn as a community as you know as an engineering discipline and we also want people to be prepared. So if you have never had a CVE in your project be it like CNCF or like otherwise open source you should still have a security process and we were like very lucky in a way we had all of this kind

of prepared we never had to apply it before. Um so we had some learnings from that but it was important to know okay what happens if we do this and we learned you know some new things like the pre-announcement was something that we didn't have in the process I think before but we realized you know it's um I kind of said this before but I do want

to encourage everyone um like most of the projects in the CNCF if not all are already hosted on GitHub GitHub is already a trusted provider for you um so if you need to coordinate on is um try to avoid using I don't know like a Slack channel or I don't know WhatsApp chat or emails try to use GitHub because it allows you to have these conversations and

you know that this information is not disclosed until you're ready to do it. So yeah, GitHub is a very good um platform for this. Um and as I said earlier like we were able to to focus on fixing it on communicating what we needed to communicate and internally we're able to discuss this. So this is like clear recommendation from us use GitHub uh use GitHub security advisories

um if you are ever in this kind of situation. Okay. But what are the things that um we might want to have in the future? So, one of the key problems that we have as an open source project is we kind of don't know who our users are. And that's good. Like that's what open source is supposed to be. Like no one who doesn't want to interact

with us has to interact with us. Um but we also don't have a trusted vendor program. Like Kubernetes has something like this. Um so they can send out more detailed pre-announcements if need be. Um but this list is you know something that you um need to be trusted with. So it's not like something that you know is sent out to everyone. Um so eventually I think what

we need to do is we need to build a vendor program for KCP so that for example if there are hosted KCP versions um we can send out patches um to trusted vendors to apply them as soon as possible. Um these kind of things. And another question that we had is how do we automatically check for this kind of stuff? Um because honestly like once we knew

what was missing it was obvious. Um like but do you build a lint for this? Like how do you know if your authorizer is doing the right thing? You don't really know. Um, one of the things that actually we didn't have in mind back when we had this, uh, vulnerability, um, but that I had in mind now when we started doing this talk is I wonder what

happens if I point an LLM at our docs and ask it to compare it with the actual code because that was our kind of key problem here. Like our doc said there is this check and we trusted ourselves that we wrote that down correctly but we didn't. So this is one of the interesting things that I kind of want to look into. Um can we point an

LM at these two kind of data sources and let it figure out if there are any um yeah uh differences between what is written down and what is actually in the code. Okay, wrapping this up. Um um so yeah uh don't be hesitant about talking about your CVS. This is important. Your users need to know um the community needs to know because this is the only way

that we can learn. And to be honest, um, while this might have applied to KCP in specific here to virtual workspaces, I know that like people are building cube API server kind of based or inspired systems that are not KCP. So if you have this kind of authorizer chain, you always need to double check it. Is it doing what it it's expecting to do? Like honestly, this

applies to REST API as well, like something that is maybe not built from the Kubernetes resource model at all. You need to think about these kind of things and you need to keep thinking about them because you might be adding new features as well. So it's relevant to all of us to be honest. And I know we had a lot of content specific to KCP here because

it was the specific CVE. Um but yeah and with that the last thing that I have to do is thank you all for coming. Um really appreciate I know that not everyone knew about KCP. I hope now you know a little bit more about KCP about our security posture um that we take these things seriously and that we need to and if you want to have more

information there's a couple of QR codes on there um the website the CVE and the talk itself which will have the slides so thank you much for coming and if there are any questions I've been told that over there is a microphone right behind you um and you unfortunately need to step up to that to ask your question so that everyone can hear it. >> Hi, uh

thanks for that insight was really interesting. I would have two questions to the logical cluster idea. Um because you mentioned um an attacker would need to know the logical class ID of the target workspace. If I'm an attacker, I think it's rather impossible to guess. It's like a 15 characters random string. How would I guess that? And how would you get a 9.6 score with that? And

maybe the second question or do you want to >> uh maybe let's talk about that first. It's a it's a like reasonable question. The thing is we don't know if an attacker knows a logical cluster ID or not because you might have granted um the attacker like read only permissions on like the list of workspaces. Um so they might be able to extract that ID. We we

don't quite know to be honest depends on your permission model. Um so while it might not like it might not be the case. We wanted to make sure that the possibility possibility very much exists. So it needs to be taken seriously. >> Okay. And the second question um can you explain a bit more about the virtual workspace URL? So what what is it actually because it's not

a cube like API I think because it involves this multicluster thing which you don't have on the on the workspace um API itself. Um and would it could it be enough to use this one to discover the logical clusters or the workspaces? Um you mean specifically for the attack, right? The discovery. >> Yes. >> Yeah. Okay. No, no, no, no. Makes sense. Um so the thing is

a virtual workspace kind of you saw that there was a custom path like resolver logic in there. So in the end, a virtual workspace can expose multiple endpoints that look like Kubernetes API endpoints similar to how workspaces all look like distinct ones. So the kind of the way that this specific virtual workspace works is there's there's a wild card endpoint. That wild card endpoint will get you

basically um like a like will get you all the objects for the specific API that's been exposed that it will give you the list of all objects created across all workspaces. But here's the kicker. You couldn't use that for discoverability because the list operation was correctly limited by the label selector that was applied. So you never saw the resource that you deleted or created and you couldn't

see like what was basically available um like which workspaces you couldn't enumerate them from there. >> Okay, makes sense. Actually a third question if no one else is queuing. Um about this labeling you you described for the get and list operations. I was wondering why is this necessary and couldn't it be also evaluated on the fly when a request is issued that the admission or the authorization

controller could uh check on the fly if uh authorization is fine. >> Uh yeah that that's that's a valid point. Um I think this is for efficiency reasons because um if you don't have these labels um on the objects you would need to list like so the the pseudo storage back end that the virtual workspace has is like an informer base against KCP. So it will would

have to list all the objects that exist like all of it basically and all the resources as well because you kind of don't know is this like resource owned by the API export or not. um and um that would you know basically eat huge chunks of memory or like processing power every time someone sends a request. So this async labeling mechanism allows us to serve the requests

efficiently. >> Makes sense. Thanks. >> Right. Thank you very much. Um if there are no other questions enjoy I guess I think it's bof crawl cube crawl time now. So thank you all for coming. See you around.