Right-sized Access Control & Pull with Proof - Stanislav Láznička & Lucas Käldström

This talk explores significant enhancements made to Kubernetes image pulling and authorization mechanisms. The speakers first discuss improvements to the authentication experience for pulling private images, introducing service account tokens that enable more flexible credential management. They highlight an issue with the 'if not present' image pull policy, which allows unauthorized access to private images, and present the 'always pull images' admission controller as a partial solution. The discussion then transitions to authorization enhancements that aim to streamline permission management in Kubernetes, enabling more fine-grained access controls through a new 'maybe' response mode. This allows for conditional evaluations that can better align permissions with specific functionalities without compromising security.

And we are off. Hello everybody. Uh I do hope that every and each of you uh read the abstract because if you didn't well we are just two silly men in funny hats. >> Yes. >> My name is Stlaznichka and this is my dear colleague >> Lucas Chastra >> and today we will be uh taking you on a deep dive through some of the uh enhancements that

we are working on in Sig O. Yeah, it's uh well, you made it to the last slot of the the day, so congratulations and hopefully you're not asleep by the end of the presentation. >> We're trying to make you not not suffer too much. Uh all right. So, traditionally, uh these talks went, you know, uh through all the caps that the group was working on and we

would uh speak about like each of the topics a little bit. Uh this time we're making uh something a little bit different. So here you can see that the group was not actually selecting but then there was a lot of work happening. Um and so what we're doing this time is we're going to go on a deep dive to through two of the features that we worked

on. Um so let's jump right into it. Uh changes to image pulling. So uh this is something that my colleague Anisha Shaker and me were working on. Uh and this all started in cube 132 when we decided that it would be nice uh to kind of improve uh the experience or the authentication experience um of pulling images. Right? Up until this point you if you wanted to

pull a private image you pretty much had only several options. Uh you could either use the image pool secrets which is not exactly flexible like you know synchronizing your secrets all around the um all around the cluster or you could use the uh credential providers which also weren't very flexible in in any way or you would just have to rely on the clusterwide uh credentials right and

so this is how the service account tokens for cube blood credential providers was born uh we basically wanted to take the FML tokens that that service account service accounts now have. And uh we we will be we wanted to find a way how the cube uh could use these and send them to credential providers so that these can then do whatever they want with them, right? And

so that's exactly what happened, right? So so the cublet is now able to request an audience scoped uh scoped token service account token and send it to a credential provider. The credential provider can then do whatever it wants. It can either send it back and that means that the image pool will be happening exactly with the service account token or it can do other things. Uh for

example, you know, for cloud providers, uh this could be helpful u to exchange the token for some kind of a cloud provider specific token which then can be used uh to reach um to reach an image somewhere in a repository which is I mean uh image repository which is specific to the cloud. Right? So this is all great except for the one thing uh and that is

that the if not present and never uh image pool policies are not exactly safe. I I will tell you in a minute why that is so uh for now uh just trust me that that is what the insure secret pulled images uh cap is uh is addressing that and that is what the rest of my slides at least will be about right and that is something that

I worked on. So uh let's see at uh what the issue is right imagine you're in a cluster and you've got a pot a which is trying to uh use in a private image right so uh it it is trying to run a container with this private image and so what it does it it uses the pull if not present policy uh and since it knows that

uh the image is private It also specifies uh cred specifies a reference to a secret which contains the credentials uh that can be used to pull this private image. So it sends all this to the cublet. Uh the cublet uh then checks whether it has the image uh present or not. It sees that it does not and it then requests the container runtime to pull uh the

image with the credential from the secret that it read previously. uh if the credentials are valid uh the private image register returns the image uh the container runtime stores it locally and tells the cube that this is this is all good you have the image now and the bot A is able to run. Now the fun thing happens when a pot B comes with an image pool

policy if not present but it does not specify any secret or credentials whatsoever. It sends this request trying to run the same very same image, right? A private image that that we just pulled. It sends this request to the cublet. Cublet sees okay image. I'm going to ask uh the container runtime if I have this image locally. The container runtime returns the image and the cublet just

runs it right because well that's how it works like the image is present and the uh policy pool policy was running not present. So that's that's what it does and that's where the problem is. uh you you were just able to access an image that that was previously pulled and is private without any credentials. So uh there is a solution or was a solution uh to this

problem and that is uh the always pull images uh admission controller. is a mutating uh admission controller which basically just turns all the uh all the pool policies for all the images for for all the pots to uh to always so that the so that the registry is always you know uh in in the game the cubit always tries to pull the image. uh this has several

problems, right? So the first uh is that users must be aware of the issue, right? If this this uh this admission controller is actually not enabled by default. Uh another problem is this kind of puts your workloads in danger uh if your egress or the remote registries uh are not stable enough, right? It also adds uh a bit of a network overhead but that's that's not the

biggest issue when when the image is actually uh present locally on the node only the manifest gets pulled not the full image. But but it's still there. But what also bothers us is you know a lot of unnecessary authentications can be happen in this way right you send in your your credentials somewhere uh to the wild and so as you can see there was quite a lot

of space for improvements and uh so so we started working on this right and uh usually when I when I want want to oh do some kind of a feature or create a feature improve something I start uh with looking at what actually is present, right? And somewhere deep in the uh cublet codebase uh that there exists a uh image manage interface uh which defines one uh

one method that you need to implement in order to satisfy this interface and that is the ensure image exists. uh you can see that uh this method uh contains quite a few arguments and I highlighted some that will be important for us right uh you can see obviously we will need to know which image we are using so that's the requested image we want to know the

pull secrets that can be used to pull the image uh later with the service account improvements we will be also interested in the pot because we we can extract the service account from the pot and pull policy obviously we need to know All right. So we know what the state is uh and we are able to create a plan. Uh the basic principle that uh we are

trying to work with was don't require pulling unless really necessary. Pulling is basically the way how we are verifying the credentials because we don't have any other way. We soon realized that uh we need to build some kind of a store where we would uh record all the all the information about how images were pulled by the cubot, right? And so we decided to build u a

mapping for from from an image to uh to pull credentials that it was pulled with right uh uh so we actually want to p we knew that we want to persist this between cublet restarts uh because you know coming from the principle that we don't want to require pulling as really necessary if if if what restarted we we would have to request require all everybody to u

reverify their credentials, right? We're only quering this kind of database uh in the if not present then ever pool. I think that that kind of makes sense. Um and another thing which is kind of re uh re relevant to the uh to the next point is cublet and the container runtime they run async which means that cublet can easily restart uh while the image is actually being

pulled and that's not great. We we we want to know like which images were being pulled by by the cubat, right? And so we actually needed uh two instead of one, we needed two kind of kinds of records. One that would be created um before uh the pool actually happened. Those are the pooling records and then one uh once once the pool is actually finished and we

see the image. Um and uh this kind of like I said is relevant to the uh the topic of pre-loaded images because uh there's this use case in in Kubernetes where the users are are allowed to load the images onto the node outside of of uh of cublet right and this can be useful if for example you're setting up your control plane this way like in the

case of kind uh which usually has a bunch of images very present right and we wanted to keep uh this uh these workflows and that is also why uh we are trying to distinguish uh which images were being pulled by the cublet and which were not that's that's what the pulling and pulled records the the duality of the records is about and so we also but but

we also wanted to uh allow users to kind to kind of like play around with like how these pulled and pre-pulled images uh should be should be treated right and that's why we came came up with four different policies. Either you don't care about the source of the image and that means that you will just specify the always verify policy which means that uh that every attempt

to access uh an image that we just on the node must be verified. You must have a record of it. Uh or there's the never verify pre-loaded images which is actually the default which wouldn't verify the the images which that are pre-loaded. This uh policy can actually be uh then further restricted by the never verify allow listed images. Right? If you choose that one, you can just

specify a set of u set of images which will never be uh reverified. Uh or you you just don't care uh which can be valid in some use cases and and you just say I I don't care about any of this. Just never try to verify uh the the pool Now, that was a plan. Uh, obviously, you know, uh, stuff gets complicated. This these are actually pictures

from, uh, from the cap. Uh, read it if you if you're interested. But what we ended up with is this kind of uh, user interface where you have uh, where you have oh, oopsie doodle uh, where you have two fields uh, in the cube configuration. One of them is image pull credentials verification. We really tried to come up with better names, but it turns out that it's

really really hard to describe what what this feature is doing in like one s simple word. So yeah, we ended up with this and the second one obviously is the pre-loaded image uh images verification allow list that comes you know with the never verify allow listed images. So it was like the feature if it took six months and five months was on the naming and then one

month >> honestly we spent quite some time trying to figure out like how to name these but yeah uh so so this is this is one of the things that that the users can see and obviously like there there there's also the uh the record store and uh you can you can actually see the records that are being stored about how the images were being pulled in

your cublets uh directory in the image manager subdirectory there's two sub is pulling and pulled. You can see uh in the pulling case that there's actually not that much information, right? Because we don't actually know much about what what's going to happen. But in the pool directory, you can see that you know there there's quite a lot of information like we we've got the image ref which

is like the reference to the image on the node. uh we've got uh Kubernetes service accounts which means that this this particular image was pulled by uh by using the service account credentials with with a service account of this UID name space and name. All right. Uh that's it from me and now uh Lucash will be telling you some exciting things about authorization. >> Yes. Now it's

um going to be interesting to let's put it there. Yeah. So well there in in the very first slide uh we had a lot of like we listed a lot of Kubernetes enhancement proposals that uh the authorization and authentication special interest group in Kubernetes has been looking at. Um this is one of them. U but yeah there's lots of other goodies if you go to the Kubernetes

enhancements repo as well. So one of the core problems that this feature relates to is the kind of emergent behavior of the existing system which kind of requires you to first make an ARBback rule and there you kind of you might be overranting uh if you wanted to say something like you know Lucas should only be able to create gateways when the field uh gateway class name

is well in this case test gateway. Okay, if that's exactly the slice that I want to kind of give the amount of permissions I want to give the user um then I need to first give them a lot of permissions create any type of gateway and make a deny rule that kind of you try to match them and make it you know remove exactly all the unwanted

permissions but this you know by experience can be quite hard and and fragile as well as you add more uh subjects or remove them. So what if we just could you know like express this slice of I I want to do give you exactly these permissions. Um and so that's that's kind of one of the things and now that well AI agents and and these kind of

things are very interesting as well. Um, you might say that it's more important than ever that we have more fine grain controls because sometimes you could maybe trust a human to do some of like not do some bad things, but now you don't have those guarantees anymore. And the same goes for impersonation, which is another Kubernetes enhancement proposal which is pretty similar. um where you might have

more permission. You might have quite a lot of permissions as yourself and sometimes you want want to delegate those permissions to someone else and let them act kind of on your behalf. But maybe you don't want to give them all of the the permissions. So maybe you don't want them the AI agent or whatever to be able to delete for example. So with this constrained impersonation, you

can give them a kind of smaller circle that they can use while they're operating. But um some of these kind of that is shown here is um I kind of went back um to the issues of you know like from 201617 in Kubernetes and and kind of wondered what can we do to kind of you know solve some of these issues that have been long time open

in my master's thesis and then that turned into the cap which we're going to go through. So in short if you remember only one slide uh you know let it be this one um when it talks to the authorizer uh it is kind of asking you know here I have some metadata that I extracted from the URL uh should this you know person be able to proceed

and do their requests and now instead of just you know yes no that's been the case before now you also have a mode which is maybe and uh yeah So that's basically the cap and I guess we can go home now. Um maybe and then something about the object for for example if the type is TLS. So there's a couple of use cases. Uh we won't go

too much into them but for example allow someone to read secrets with a given label. Um allow someone to only you know update when a sensitive field is unchanged. Um or create only object with a you know certain names. This is something that people have told that the gotcha arbback you can't actually specify the resource names in arbback for a create. It won't work. Um for this

reason allow a node agent to only access you know objects referring to them. So filtering by spec node name. Allow a controller to only add or deny their own finalizer. Now a controller could just go rogue and remove all the finalizers and other controllers wouldn't be able to clean up or deny everyone except you know certain admins or similar to change a sensitive field. So this was

basically the other Kubernetes enhancement proposal for DRRA admin access. So a short recap in the Kubernetes rec request flow you have authenticators then you have the authorizer and the authorizer only has metadata. So you know what API group, what name space name, stuff like that. Um and um then for a write request you also have admission control. So this actually gets the object. So the right so

the old and the new object and let you do something and if you have used validating an admission rules that's um what what you can do at this stage or web hooks but this actually doesn't exist for reads. So reads are completely unconstrained. So you can do this today with a web hook or a cell rule which is uh which are both good options but now again

it they are deny only. So you need to do this kind of allow first match and then deny. So what can we do instead of it? We can make a unified policy through partial evaluation. So if I have something like Lucas can create gateways only when test uh class is test gateway. I could write in pseudo code something like this in in kind of cellish form. So

the request verb is create the API group is gateway resources gateways. User is Lucas and then gateway class name is test gateway. So all of these have to hold in order for me to be allowed. Now if I uh but because we don't have the the object available in authorization, we don't want to decode object objects from possibly unauthorized users. Uh we mark the request object as

unknown. So we kind of if I do a cubectl create with a gateway uh we can the authorizer can step through this expression and see that request verb is create yes it is it's a cubectl create API group is correct the resource is correct it's me Lucas um and it's a v1 now let's say uh payload but what what the authorizer cannot know is what is the

request object it doesn't have this information at this point so what it the trick that it does is that it kind of saves this information, puts it into the maybe category and then later we enforce this. So this is called partial evaluation. But what if a hacker or you know like another user you know anyone would do the same cubectl request with the same data. Uh well

it's still a create and it's still the same API group and uh resource but now the username is something different. So now this is actually false. So because this is false now it actually doesn't matter what else is there like no matter whatever the the request object is this expression as a whole will always be false. So we can actually deny this straight away like this this

you know hacker user can never uh become authorized. So we we don't need to decode this object. So um Kubernetes authorization in general allows you to specify multiple authorizers and before this uh it was deni allow deny or no opinion um and allow or deny then you that's what kind of controls what response code you get most likely uh but if no opinion you kind of go

to the next authorizer until you get uh this a response and now we kind of add a new it's a little bit more complicated this but on a high level uh we add a new response uh kind of decision mode which is conditional and um if you have this kind of if you have a possibility of becoming allowed then you proceed to the next stage. So what

is a you know how do you express this maybe thing? Uh well you a condition has an ID uh it has an effect which is which means that if this condition returns true what should happen should we get an allow should we get a deny or should we get a no opinion um well then there's a condition itself which can be you know it's a arbitrary string

um you can if you like to you can express this as cell or cedar or opa or something like that whatever kind of you like as a type um or you can just write something random fu and then uh kubernetes will call you back. So what is happening when you run a request in this new mode and in this PR 10 thou 100,000 something uh of kubernetes

where I implemented this um now we go to the authorization phase we ask the authorizer you know what do you think um and it gives you an unconditional allow an unconditional deny or then a set of conditions if we have conditions we propagate these with the request and we kind of give them back. We have a new admission controller which catches these and gives them back to

the authorizer. So whatever the API the authorizer gave us before we give them back in the second stage. And now you can actually compute this this last part of the the expression and assign is it true or false or or what do you want to do. So and and if this is expressed as cell then we don't even need to call the the authorizer if it's kind

of would be behind an expensive web hook somewhere we can just readily eva evaluate the cell expression inside of the API server just like VIP. So and it uses the exact same code as validating admission policy. So um that's all kind of just a unification of of this. But the the end result is that the authorizer can actually now restrict up front um what is going to

happen later which it couldn't before. So what does it look like? Something like Lucas can only write TLS secrets will now be shown actually when you do a self-s subject access review. Um so this is a cubectl who am I sorry uh can I um there's both of them and if I now ask can I uh create secrets then it will say yes only if type is

um TLS in this case um so now we actually have an encoding of this uh conditional uh authorization and um if I eventually this is not part of the initial cap but what you could do further is also to extend this to reads. Um it would be the second cap basically like there's one for writes there would be one for reads uh and then one to add

kind of a what we in the sigot have as a working name arbback++ um if you'd like to see this please join the sig meetings and you know tell us what should arbback++ do um there's I mean we have lots of issues from users um you know through from throughout the years but um yeah specific speific kind of actionable you know request would be appreciated and and

we can only do it um you know through this user request but at the end of the day um this means that uh instead of having kind of separately for metadata and data and for reads and writes because there's another ke which adds kind of the read part. So instead of having like three different boxes where you would need to understand different formats and stuff, we could

unify it to only one layer. So like you could you could express across all of these dimensions in just whatever is cell cedar um opa or or similar and and provide thus a kind of more unified experience for the end users which I'm I'm sure people uh appreciate. So that's a feature. Um again it will kind of go through a couple of more revisions. Uh the code

that I refer to here would hopefully could merge in the 137 release. Um and then yeah we'll we'll we want community feedback for the upcoming features as well and yeah let us know if if this would be useful to you. All right, questions. >> Perfect. Can you uh can you tell that to the mic or um if Okay. Yeah. So, are there questions about these endeavors? Uh

can you if you can Yep. >> Hi. uh just uh both parts of the presentation really good on on yours the first one you said you built a credential map and you had uh each image and the credential that pulled up and I'm just thinking that credential is it tied to uh some account identity would it change like could later on that identity no longer have the

access to >> uh yeah so Um what we c is actually uh more or less the metadata of in this this case the secret right the credential is stored in a secret. So we so we catch the metadata about the secret and we uh catch the hash of of the content of the secret right and we know like which credentials were were actually being used. Um if

the secret changes uh we consider this be a rotation and so the these are considered to be the same right. Uh and right uh and also if if there's a secret somewhere else in the cluster uh we we also like are able to check that that these are the same. Now your question was if the identity no longer has access to uh the secret we don't actually

uh we don't we don't actually solve this kind of situation. we consider uh consider it since the identity was able to reach the u the repository once this way that that it it should be able to just pull from there. >> Yeah. >> So good question. >> Thank you. >> Other ones I think we have exactly one minute left. >> Sprint to the mic. >> All right.

Yeah. Uh thank you all for your attention. I know it's late. Yeah, thank you very much.