Kube-Oddities - The Quirks That Keep Kubernetes Interesting - Marcus Noble & Márk Sági-Kazár

This talk explores various quirks and oddities associated with Kubernetes, delivered by Marcus Noble and Mark Shagi Kazar, both experienced platform engineers and CNCF ambassadors. They delve into topics such as the role of sidecar containers and their tricky syntax, best practices for referencing container images using SHA values, and the peculiarities with Kubernetes pod hostnames and DNS resolution. The speakers also discuss critical security aspects, including the implications of the token request API and the potential risks of the escalate verb in RBAC configurations. Furthermore, they highlight some operational insights like using static manifests with kubelet to run pods without an API server. The session invites attendees to share their experiences and frustrations related to Kubernetes to foster a collaborative environment for learning.

Good morning everybody. Uh thank you for coming today. We are going to talk to you today about uh cube oddities. Um for those of you that don't know me. I'm that chap at at the top. My name is Marcus Noble. I am a platform engineer at Monzo. Um I'm also a CNCF ambassador and I run a monthly newsletter called cloudnative.now where I do like a roundup of

everything happening in the cloudnative ecosystem within the past month um with KubeCon. This month's going to be a fairly busy one for me I think. So, look out for that if you're interested. Um, and I've now got about I think it's about seven years worth of experience in like the cloud native space over a bunch of different roles, different personas that I've that I've had over my

career. So, um, come across some some very interesting oddities within Kubernetes over that time. I'll pass over to my dear friend Mark. >> Yeah, my name is Mark Shagi Kazar. I'm well, don't try to pronounce it. I'm from Hungary. I'm also a CNCF ambassador and I've been doing uh lot of community organizing and open source projects. Uh I'm also an organizer of KCD Budapest uh which is

going to happen this year again. So if you want to attend or submit any talks, please do. And yeah, my last latest project is uh an online course called Kubernetes the very hard way. So if you want to learn the inner workings of Kubernetes, check it out. So before we get to the good stuff, uh why are we the ones delivering this talk? And uh last year,

I think it was last year in Munich, >> yep, >> where we got talking about our PTSD or the weird things in Kubernetes that you know the things that you always have to look up, the things that you always get wrong for the first time and you have to then fix it. Uh and we thought we we we managed to gather quite a few things. So we

thought we would share these things with you and maybe you can uh share your experience after the talk. But hopefully whether you are a um inexperienced Kubernetes operator or expert or whether you just ran your first uh cube cuttle command this morning. Hopefully we can we can uh share something new with you. >> Just just want to quickly check with you It's pronounced uh coup control, right?

Is that is that correct? >> No, I don't think so. I mean uh if anything, it's cube city, right? >> No, I'm pretty sure it's it's cube cuddle, right? Cube cuddle. >> Anybody? >> What do you think? Cube cuddle. >> Ah, we'll win you over. We'll win you I'll win you over. Right. Okay. Right. So, I'm going to kick off talking about pods. Like, if you touch

Kubernetes, I'm pretty sure you come across pods at some point. are kind of essential in Kubernetes I'd say um but they do have some of their quirks. So I want to first talk to you about sidecar containers. Now sidecar containers are a feature of Kubernetes where you have a a separate container that runs alongside your main application container and it may do some like ancillary ancill that

word uh work for your your application. This may be things like it it's a log it handles logging output or or something like that. These were very popular um oh sorry these were popularized a lot by like ISTTO service mesh and stuff back in the day. Those similar sort of things where they'll do some network service mesh at the side of your main container and they run

all within the same pod. Now yeah um these are now officially supported in Kubernetes as of version 129. They are they are uh on by default. >> Finally >> finally um so you may know we have init containers. These are what run at the startup of your pod. They allow you to like set up the environment. Maybe you need to tweak some IP tables or something fun.

Maybe you need to pull down some data. You've then got your containers which is where your application lives. This is where your actual logic logic runs and all this kind of thing. >> And then we have sidecar containers, >> Sounds good. >> No. Um yeah, no sidecar containers. What we actually have is uh special magic in it containers. Um the way you create a sidecar container is

you use this restart policy always on an init container that is the only value you can have for that restart policy. The the default if you don't have it is just that it doesn't have it and it's not to be confused with the restart policy on the pod level. So this is actually on the init container level. What this means is the pod uh sorry the container

will uh start running at start of the pod and continue running after all the other init containers and then alongside your main application containers. So you don't actually have sidecar containers defined as a thing. You just have these like special init containers that keep running. And I'll be honest, I hate this syntax. I hate that this is the way you have to do it. I wish we

had >> Do you know why they came up with this design? >> I don't. I don't. I if somebody does know like the in-depth reason why let's >> yeah please tell us please tell us >> um next I'm going to talk to you about images and the best practice is to use the image sh so when you're referencing images in your containers you will do something like

engine x colon latest and it will always pull down this uh you know potentially insecure potentially secure uh uh image you don't know what you're going to get until it actually pulls it now you could then do uh a tag based one so you could do you know engine x142 two or whatever it may be, but that can still change that that image can be overwritten in

the uh registry. So, what you really want to do is you want to reference the the SHA kind of like this. It's best practice and it means that the content that you're pulling down is going to be the same each time because the char is based on the contents of the container. If it changes, the char changes. Now, one thing to be aware of is um if

you're using this and you have both the version the the the human readable 125.1 is actually meaningless to Kubernetes. It ignores it and it only uses the SH. So if these become out of sync with one another, you're actually going to be your you know your Kubernetes is going to be pulling 152 down, but your humans that are reading it think, "Oh yeah, is that safe?" You

know, verified 125.1. It's fine. So, if you've got any tools that automatically update these through PRs, Renovate, dependable, whatever it may be, make sure that it's updating both of these versions. So, you're keeping your computers and your humans happy at the same time. >> Have you ever been beaten by this? >> Oh, yeah. Yeah. Yeah. But I want to play a game with you, Mark. Let's play

a game. This game is an incredibly fun game. It's going to be like a new prime time TV show. It's called Spot the Mistake Pod Volume Edition. I am going to show you two pod manifests. There is one mistake on screen and I want you to point it out to me. >> Let's do it. >> Here we go. Where's the mistake? >> Can anybody spot it? >>

Okay. >> Secret. >> Boom. Thank you. Yes. >> I hate this. This catches me out every single time. When you are referencing a secret as a volume, you have to do secrets. Secret name. Whereas with a config map, it is just config map.name. I have to look this up every single time because I can never remember which way around it is. I hate this inconsistency. This is

like the legacy stuff from the early days of Kubernetes that we now can't change. And it's it I cry a little bit inside every time I see this. >> Mark, go and tell us about some networking, please. >> Networking. Well, it's always fun, >> Best topic. Everybody Everybody loves a bit of networking, right? >> All right. So, let's uh there's a little known fact that you can

actually change the host name of a pod. You can also set a subdomain. Cool. So if you do so the pod itself and if you run the host name command it's going to attach the cluster domain and the service and the name space but you can change the host name of the pod itself. >> Cool. So so this means that we can configure what what host name

our pods are addressable on then right Mark? >> Uh not really. >> So it turns out pods themselves can't have DNS assigned. So if you even if you set the host name in the pod, you're not going to be able to talk to that specific pod through that name. Even worse, if you use the pod name itself, that's not going to work either. So pods can't be

addressed. >> directly via DS. >> So we can we can just tell pods they have a host name, but it's not necessarily the host name that they >> basically not going to work outside of the pod. >> Ah, interesting. So how may we contact? >> There is another way. It's kind of a workaround. It's called headless services. When you create a service without a cluster IP, >>

okay, >> it means it's not going to have a stable endpoint, a stable IP. So if you resolve the headless service DNS, it's always going to give you a pod IP. So if you want to address pods directly, that may be something. But the other thing is that the headless services, they attach the pod name in front of the domain name. >> Okay? So you can access

the individual pods via that. Okay. >> So that's a silver lining. >> Okay. That >> all right. Let's let's play another game if if you play with me. >> I'm ready. Are we all ready? >> Yes. >> All right. So there is this thing called DNS policy. You can set the DNS policy for a pod which means you can change how resolve comp is going to look

like within the pod. >> Yeah. >> So it's it can have these four values. These are all valid values. Guess which one is the default. Now Mark, come on. Come on. Come on. Like obviously default has to be the default, right? Default has to be the >> Is it the default? Do you agree? >> Yeah, it's cluster first. >> So default means it's going to fall back

to the nodes resolve comp values. Cluster first is going to be the default where you can actually use the cluster DNS and resolve service names. >> Naming is hard. Yeah, >> naming is hard. >> All right, let's talk about another fun stuff. >> Let's let's let's talk about security. Let's let's let's pick up the pace. Let us pick up the the the energy a little bit and

talk about security. Now, what if I told you that there was a possible way in Kubernetes to create credentials that you cannot rotate or revoke? What you think is that a that a good thing to how you like this, don't you? >> Yeah. This is this is kind of the reaction that you you'd expect. Well, I want to introduce you to the token request API. Now, this

allows you to create a um a token against a service account within your cluster. Now, this may be one of the the, you know, privileged ones within your cluster, such as the node controller uh service account, which has, you know, some decent arbback associated with it. You can do things with node. You can do some things with pods. So, if we create some credentials with with this,

we can be a little bit nasty. Um, the problem is once you've got this, you've then got a JWT all token that cannot be revoked and it cannot be rotated. The only way that you can get rid of this is either wait for it to expire or delete the service account. So, not ideal. This is what we get. So, I've created one of these tokens and I

do it or who am I? And I am, as you can see up there, coupe system node controller. Um, these tokens, as I said, cannot be revoked. So, I have this for at least as long as the TTL is for these tokens. And I can get pods, I can delete pods, I can't create pods, so I can't do a huge amount of damage, but I can delete

nodes. So, I can be a bit chaotic. Um, as I said, the only way to get rid of them is you've got to delete the service account. Now, I don't know about you, but I don't want to delete node controller service account. I don't know what impact that's going to have. Um, I said the alternative is make sure we have some some sensible expiration set on these.

So 24 hours maybe depending on your environment you're in you may even want to shorten that to some really really really short there are some environments where this is very long so be aware of that not great but I want to talk about another thing I want to talk about arbback and I want to talk about arbback escalate so any Kubernetes user that has the permission to

create uh arbback related roles so uh roles or cluster roles they can only create those roles that have the same or less uh permissions that they themselves have. So, it's my permission. It does. I mean, smart people put this together. >> Maybe some in the room. Thank you very much. Um like you don't want to be able to have somebody that can create a cluster role only

that can then create a cluster ro that gives them like full access to everything. That's that's that's not a great idea. Kubernetes says no. when you try to do that. Unless, unless, let me introduce you to the escalate verb. Now, this is a verb that doesn't come around very often and I learned about this from the very very wonderful Rory Mun in one of his fantastic talks.

If you happen to have this, if you happen to have this uh permission here, for example, you have star in your resource star on your verbs for an arbback, you've then got the escalate verb for arback. This means that you can effectively do whatever you want with your new superpowers and give yourself whatever access you possibly want. Be very careful when you're using stars. We should not

be using stars in these as much as we can. Great. Right. I want to talk to you a little bit about admission policies and uh how they interact with arbback and things like this. So you may be well aware that arbback is like an additive permissionbased system. So arbback can give you permissions. It cannot be used to like take away permissions, right? So you can give somebody

the ability to create pods, but you can't use arbback to take away the ability to create pods that match a specific thing. That's where admission policies and admission controllers come in. Now you can have uh validating and mutating admission web hooks and admission policies. Now, now these allow you to be more fine grained in the way that you control all API requests to your cluster. So you

can say this user in this name space can only create or edit things that match this label. For example, you can be really really granular. But there's a blind spot. There is one area where these cannot work. They cannot restrict themselves. You cannot create these policies or web hooks against creation or deletion or editing of policies on web hooks Here is the actual code from Kubernetes. It

just straight up says if it's any of these, skip it. We can't apply it to these. So that means that for very good reason, you can't have people create policies that stop you then changing policies. Like you don't want a bad actor to get in there that creates a validating web hook that then prevents you from changing your validating or mutating web hooks in the future. like

very very good thing to know about, very good thing to have, good to be aware of in case you are trying to do some little clever stuff where you dynamically create these web hooks, you cannot do that. Let's wrap it up with some operations. >> Yeah, let's let's talk about operations. So, I'm pretty sure you all know cubectl get pods. Do you know that there is a

CCTL as well? >> Who used it before? >> Oh, quite a few. Cool. >> Hold on. We mean crackle, right? >> Oh, crap. Yeah. >> Cry cuddle. >> Cry. Cry cuddle. >> Cry cuddle. Come on. >> So, and the thing is like it has a pods command, but it's a little bit different from cubectl get pods. Like the pod, you will know what a pod is in

kubern. >> Surely pods are just pods. >> Well, not really. And the confusing thing about them is they actually look very similar. So if you take a look at uh pod or I should say pod pod sandbox technically they all have labels and metadata and annotations but they're not the same thing. >> So they got they got similar things but like jumbled about a bit. It's not

in the metadata. It's Yeah. Okay. >> So CRI stands for container runtime interface and that's the interface that cublet talks to container runtimes to. Uh and the creek cuddle >> cuddle >> cuddle. It's catching on. It's catching >> Yep. Yep, it's actually the one that you can use uh to talk to this container runtime interface which is a low-level API and uh the concept of pot sandboxes

is the one that connects container runtimes container concept to cubes pot concept basically but they are not the same thing. So if you are >> it's like a it's like a bridge between the two. >> Yeah. Okay. >> Exactly. >> Nice. >> All right. Who who who knew that cubullet can run pods without an API server? Okay, so there is this thing called static manifests where on

each node you can create pod manifests and cubat will run those pods for you without an API server. Those pods cannot be deleted. Basically, you can list the pods, but you can delete them through the API server. And there is a sneaky little feature in there. You can actually hide pods from the API server. >> Um, I'm sorry, what? >> Yeah, you can hide pods running on

worker nodes as static manifests, but they're not going to show up on when you run cubectl. Sorry, cube cuddle. get >> So basically the trick is you create a pod in a namespace that doesn't exist. And the reason you can hide these pods is because cublat tries to create a so-called mirror pod in the API server but it's a best effort thing. So if it can't create

it because the name space doesn't exist, it's not going to create >> Oh. So, I don't want to give you advice, but if you want to use your company's clusters for Bitcoin mining or something like that, this is a great way to >> No, but it's we do not recommend that. >> Yeah, I'm definitely not recommending >> Not recommend that. >> All right. So, what what are

these static manifests good for? did you actually know that Cublet can run like as a small Kubernetes cluster? Well, not a cluster but a single node cluster. So you can, as I said with the static manifest, you can run pods on nodes with cubelet only. It's called cubelet standalone mode. And that's how for example the control plane is set up. >> Makes sense. Um, so you can

run the control plane inside Kubernetes and it's good because you can use the same tools to monitor the control plane without static manifest. You would have like a chicken and egg program like how do you set up the control plane without running pods and how do you run pods without the control plane. So that's what the cubelet standalone is and the static manifests are good for. You

can run or you can't use many features like you can run init containers, you can run or or you can use host features but you can't use secrets or config maps or anything >> So anything that requires the API server you can't. >> Gotcha. So basically you have like a light Kubernetes cluster, single node Kubernetes cluster. Yeah. Kubernetes light. Yeah, I like it. I like it. But

who need who needs clusters these days? Let's just go back to one massive box, right? Yeah, >> I like it. But also, we got a a cublet cuddle that >> cublet has. Yeah, cublet cuddle. >> Cublet cuddle. >> Cublet has its own API that you can talk to. So, if you want to list pods, running pods, you can use cublet cuddle to to list those. >> All

right, I think that's >> cool. We've actually done that a lot quicker than I was expecting. So uh if you were interested in this uh we do have some other things that are similar that that the two of us have done. So I have a very similar talk to this uh called pod deep dive the interesting bits where I I go into a lot more of the

just weird things about pods that kind of have made me frustrated over the over the years. Mark, do you want to uh talk some more about your very hard way? >> Oh yeah. So um as I said before I've created a course called Kubernetes the very hard way. It's available on Exuse Labs. I don't know if you heard about that. It's a hands-on course >> where you

can uh read the course material and run the commands yourself in a browser basically. >> And if you are interested in the slides or the any of the resources related to things we've talked about, you can go to the top QR code and link there. If you have any feedback suggestions or your own quirks and weirdness and stuff, you can either fill out uh the feedback form

that we've got there or contact either me or Mark at various places that we live at on the internet and we would love to hear about the things that kind of make you frustrated over the years with using Kubernetes. I'm sure there are many many many more that you can all tell us about. >> We need to keep this talk interesting. So please >> we need we

need like V1, V2, V3 of this. So let's let's keep this keeps going. And with that I would like to say thank you very much. Thank you.