Operators in Action: Making Kubernetes Work for You - Verena Traub, b'nerd GmbH

This talk explores Kubernetes operators and their practical application through a recent case from Binerd, where the speaker is a cloud consultant with extensive experience in managing clusters. The speaker introduces the concept of operators, highlighting how they extend the Kubernetes API to manage specific applications more effectively. They discuss the challenges of deploying a complex application that is not originally built for Kubernetes, emphasizing the need to handle dependencies and user provisioning dynamically. The presentation includes insights on custom resource definitions, reconciliation logic, and integration with external services necessary for the application’s operation. The speaker also shares lessons learned about modularization and resource cleanup in operator management.

So, it's exactly a quarter past three. I think we can start. There's still some people coming. There is some spots here and there. I mean, you will find one. And first of all, thank you so much for being here. I I mean, that's what I would guess is a packed room. Thank you so much all for your interest in my talk today. I will talk about um

Kubernetes operators and how they can help us. And what I brought is actually a case from um my company. A very recent one. So, actually, it's just I mean, just released, but it was released pretty recent um to production. And you are actually the very very first people hearing about this. Um so, I hope this makes you kind of feeling honored, let's say. I mean, faces don't

show, but okay. Um last Um but before really digging into first things first, maybe who am I? Uh I'm Verena. I'm a former tech and recruiting manager turned uh web developer turned AWS consultant turned cloud consultant working now at Binerd. And we at Binerd, it might surprise you, but uh we love Kubernetes. And we basically we are managed uh service providers, so we um work on clusters

every day. And we deploy a lot of stuff to it. Uh also some apps. And also one very specific app. I won't call the name, but let's say it's the basis of today's um case, and it's a very let's call it special application. Um meaning, it's not really built for Kubernetes to be really honest. You can run uh it on Kubernetes, but it's not exactly built. It

has some very weird specifics uh when it comes to um operating it. So, well, it might not be the ideal application where I would say like, "Okay, cool. Now, let's go for Kubernetes with it." Uh still, we have a client. And that client came to us um and he wanted to uh build a startup, build a new company, and act as a reseller for this one application.

Meaning, he would like to offer to his clients different packages of the app. Meaning, there's a basic package having some sub applications within. There is a premium and an ultimate one. And he wants to get them on Kubernetes because he actually counts with a lot of users. Right? We're not talking about one or two instances per day, but rather 100, maybe. So, a lot, right? That was

his idea. And the Let's say the starting point was basically, okay, we have an app the client is building, which is actually there for onboarding clients, so users. So, a user can sign up and choose, okay, I want to have package premium. I will pay for it, and then off we go. Some magic happens, and instances spin up, right? And the client wasn't really I mean, the

magic could be for for them, it could be anything, right? Ideally, it's an API they can send CRUD requests to, but I mean, in the end, just make it work. Dear B Nerd people. And there we go. Formally, before that client, we basically were using the official Helm chart to deploy the application, and that was cool. We didn't deploy so many from time to time. Then we

did some updates on it. So, it was completely doable manually. this wouldn't work. Speaking of hundreds of instances, we do not really want to do that. Especially as there normally is also some manual configuration you have to do in the last bits. So, no. Absolutely no. Hello, and welcome to the land of Kubernetes operator operators. And as this slide says, I mean, that's basically what we found

out pretty fast. That's for us the the easiest solution to just get this project started. And as the slide also says, I mean, basically operators are way to extend the Kubernetes API, right? So, we can extend it and we can build in our very domain specific, weird, app specific, let's say, knowledge into it, right? That was basically the idea why it makes so much sense for us.

Having a packed room, so maybe we can do some interactive stuff like raising hands for Who of you is working currently with operators? I mean, community built ones. I mean, close to everyone. Who is building their own operator? Some. So, why are you here? No, just kidding. But, I mean, there are still some who haven't worked with operators, at least so far from from a building builders

perspective, let's say. So, maybe we just give a quick glimpse of what it actually is and what is important, blah blah. You know the game. So, what are operators all about? I mean, in the end, we have an operator within our cluster, we have a custom resource, which is basically managed and watched by our operator. So, whenever a user, an app, a you name it, modifies this

custom resource, the operator takes action and takes care that the cluster, like what is on the cluster, reflects what we want to have from our custom resource, right? Simple as that. So, we need three different say components to have a own operator. First one is the custom resource definition. It's basically the new API type you are telling Kubernetes to use. The second one is the custom resource,

we saw it on the slide before. So, this is the desired state. So, the thing, in our case, the app, you want to have within your cluster. And then we have the operator, which is then not surprisingly kind of the heart, because has controller logic, which basically always scans the cluster and reconciles resources when needed. So far so clear, right? Just digging again a bit into it.

Um just a very tiny um example of a custom resource definition for a Hello World app doing nothing except saying Hello World. Um so in the end what we can see, and I hope no that this one is big. Uh we can see we can give um our new custom resource definition a name. Uh we can also give them short names. And in this case we have

the scope name spaced, meaning this custom resource deriving out of this custom resource definition um will live in a name space and can only be used within that name space, right? Um and what kind of enables us um this kind of little snippet is that we can interact with our new resources like we would do with the classic Kubernetes resources, right? We get pods, we can get

our Hello World apps. Right? That's just basically the idea. And then the more interesting part comes, which is the Open API V3 schema which comes out of the box and lets us um define a set of different things. Uh I think most importantly you will have a spec where you can put stuff in. That's what you want to have. That's the desired state, right? So you In

our case we define, okay, we need an image, we need replicas, and it should run on a port. And then we have the status and that's where the operator comes in and in the end this is what is actually happening with in the cluster and what is going on, right? And in our case we define, dear operator, please let us know about available replicas within the cluster.

Then when we want to have our instance, our wonderful Hello World app, we can just create a custom resource saying Hello World app, and we can say, please use this image. I want to have three replicas running on port 8080. Simple as that, we can now generally apply these files to Kubernetes to our cluster, but it will not change Because this the most important thing is missing,

the logic, right? So, Kubernetes knows now there is stuff, but it doesn't know what to do with it. And that's basically now the heart, as I mentioned. That's the reconciliation function and the reconciliation loop. And it's a really mean word when you will talk about it, because I will definitely jump uh fall over it from time to time. But in the end, what happens is always the

same. This is of course a very generic one. Uh but in the end, the operator will check for Okay, what is currently in the cluster? If I have a new custom resource, which is not there yet, I want to ensure there's a deployment, there is a service, there is whatever else needed. And then when it's there, I will say, "Hey, by the way, we now have a

new instance." So far so good? Okay, I see some nodding. All good. So, we now kind of know what we need, right? Um and I think that's also a lot what you can find online, right? Uh but now comes the fun part, like really implementing it. And now comes the interesting part of Okay, we are also not talking just about Kubernetes resources, we're talking about a real

with a lot of specifics, let's say. So, on our road, there are some milestones, which I will now also explain and talk a little bit about. Uh so, the app has dependencies. I'm pretty sure you know a lot of apps which have dependencies, so I think that should be clear that this is something the operator definitely needs to handle. We want to have automatic name and credential

generation, more on that later. I talked about the different profiles, right? Basic, premium, ultimate. This is also some something we should have We have startup time. Many apps have startup time, this one has a long one. This needs to be handled. Um, DNS integration, of course. And then we have a lot of I mean, a lot of but some external services which are not running on Kubernetes

and they will never do. But they interact very closely with the application on Kubernetes. So, this is also something we need to tackle. Let's get started and our hands dirty with handling app dependencies. Um, and as I said, it's not surprising also for any app out in the real wild. Um, there are some dependencies. I just listed a few. There are even more. Uh, but we have

Redis for caching. We have a database. We need persistent volumes. We have ingress. Yada yada, right? There is a lot of things to tackle. So, the question how can we do that? And first thing always is creating a custom resource definition. Surprise. That's a very minimal one. I mean, there is a lot of uh, as you can see, dots and uncommon and stuff because it's way longer

and would not at all match in But what we basically can see when I have a client app which is name spaced again like our Hello World app. And we have um, in our specs, we have things we need, right? Dependencies we need. Database, Redis, S3, admin like admin credentials and such. And ingress. That's the first step. Now, the interesting part is okay, how do we realize

it? In our case, we were thinking okay, of course, we could also write an uh, Postgres operator. We could also write any other operator, but there is so much out in the wild. So, we don't have to invent a real fresh. We just use what there currently is already and implement that into our logic, right? So, in the end, an operator talks to another operator. meaning this

also means we have to wait for stuff, right? So, we can't just initialize our app directly because it's relying relying on okay, we first need a database and the credentials and then we need Redis. is actually pretty easily done, right? So, within our reconciliation logic, we just check is there a database? No, then let's wait. Um and if it's there, awesome. We can continue. We can check

for other dependencies. In the end, when every dependency is ready, the operator can um move on and really start with the installation of the application. Right? Um actually exactly the same is checking for the secrets from other um services we have. Um so, we just check is there any secret within our cluster matching our naming convention we know from that operator. Uh if no, please wait. If

it's there, Naming and credentials. And I'm pretty sure most of you here understand when I say okay, this is nothing we wanted to use this to decide. Right? Um so, our application itself has an admin password. Um it also needs a name for within our cluster and we definitely wanted the operator to take care of it. Again, actually pretty simple. We decided on a high query haiku

kind of kind of thingy mimicking a bit like how Kubernetes name things. So, we would just have a tiny function taking some adjectives, um taking some nouns, adding some suffix and some prefix, and off we go with our wonderful for example, CA gold leaf something. Right? And then we can when um creating the instance within our reconciliation logic, we can just say okay, namespace please be the

name we just created. And the same for the instance, of course. Take that namespace. Take that name. Um the same kind of mechanism we can do for the um password. So, again, we do something randomly, right? Like a random generated password. The difference here is we of course want to store that in a secret, right? So, that's basically in the end the only difference we make here

um that we not just pass the value, like pass the secret um like the password, but we first have a secret, and this secret can then be passed into our application. Good. First milestone. Profiles. So, as mentioned, we have profiles basic, ultimate, premium, um and they come with set of different apps um within the app. Okay, I know it sounds weird, but uh well. And we have

um different backends or different external components. They also might differ between the different profiles. So, generally, we could just say, "Okay, now let's intend extend our custom resource definition we already have, right?" Would be easy. But it's actually not that what we would recommend, right? Um because this actually brings a completely new um let's say logic to the operator, right? It's not so much now what is

needed on a general resource side of thing. Okay, what does the app need? But it's like, "Okay, what the the concrete instance we have will differ, right? And there are some apps which are completely different and managed by the instance or the app itself than by Kubernetes or any other operator. So, that's why kind of different logic, hence new CRD, So, we have another uh custom resource

definition. It's that time it's called client app profile, not surprisingly. It's actually scoped now cluster-wide, right? Because every instance within the cluster should be able to use that profile. And we can put in some defaults. It's a bit dummy content again. I'm sorry for that, but we have uh for example here in our spec, we can say, "Okay, please um we have app one. It includes back

ends and it's um I mean, you can enable or disable it. By default, it's disabled. For example, the basic profile doesn't have it. So, you have to turn it on by default and then you can add all the other apps you can So, our custom resource then is um just naming the apps we have. And if we want to change the default, we put it in there.

If we don't want to change the default, we can just leave it out. Right? And this also enables us to later on um adapt profiles, for example, if there is new needs, if there is another client who likes our operator, and so on. So, it gives a bit more flexibility. Startup time. Many applications need some time to get everything covered, right? In our case, for the basic

profile, it's roughly a minute. For premium or ultimate, it can go up to 3 minutes. And I think you agree that this is I mean, horrible user experience. If you sign up as a new user, you say, "I want to have package premium." And you wait for a minute in the in front of the browser, right? That's just not feasible. So, what we decided here is we

want to involve um to implement um pools, right? So, we have client app pools with pre-warmed instances. You can think of it kind of auto scaling, but just in advance. So, within the cluster, there are always running instances which are pre-configured. So, they don't have that long startup time. And when a client comes, like a user comes, and wants uh wants a new instance, we just take

one out of the pool. We make some minor adaptations. It might take a tiny bit, but then we are good. Right? So, that's the idea of pools. And again, it's a different logic, right? So, we again wouldn't really put it into any of our existing custom resource definitions. So, hi again, new custom resource In that case, it's a pool. It's again It's again scoped cluster wide because

the whole cluster should be able to have And we say, "Okay, how many replicas does this pool has to contain?" Up from zero to, let's say, whatever. But the default, for example, to five. Later we will have three in the examples. And I passed the profile. So, what this gives us in the end then is we have um three CRDs, of course. Surprise. But we will have

then within our cluster, we have three different pools. Premium, basic, ultimate. And all of these pools have three instances, for example, or five. Let's go with five. 5 * 3, so there are 15 instances all the time available for the clients. Or the users. DNS integration. Actually, when I thought about it in the very beginning, I thought, "Okay, this will be a nightmare." I mean, you don't

have to go to the DNS provider and so on. And lucky us, no, it wasn't. Because again, we didn't invent the wheel fresh, but again, we tried to separate, let's say, the domains, right? There is a wonderful project called external DNS. And that's basically, I mean, checking within your cluster if there is what is going on within your ingresses and your services. And if a new host

name pops up, then it just automatically sets up a a record within your or at your DNS provider, right? Pretty simple. And then, no surprise, there is a cert manager within the cluster who can deal with certs. So, first thinking horrible, in the end was pretty simple. So, whenever we have an ingress updated, DNS record will be created by external DNS. We will request a cert. The

cert manager does its usual thing, like checking the DNS challenge, issue the cert, and we have TLS ready. That was really easy. It's also nice when things are like easy like that. So, the only thing actually for our operator / operator code was to just add uh the annotation for the cert manager when we built the ingress, right? When we say create instance, then please ingress, and

then please wonderful cert manager, do your thing, and external DNS will automatically do its thing. Pretty nice. External components components weren't that funny or easy. Um not surprisingly, we knew that from the very beginning. And there is a set of um components we have here. So, again, anonymized, but we have our cluster with our instance, and these dependencies, and then we have a back end one, back

end two, and a custom turn server running on separate VMs, right? And the main thing actually is they interact pretty closely. They link closely. So, the back end needs to know the host name of the instance. Um it gives a password which the instance needs to know, as well as the host name you So, you get the point, right? So, they are linked extremely close, but they

can't run on the same system. Um So, actually for us, the easiest way was again thinking of okay, wouldn't it be fun to have the operator again talk to just somewhere and not just manage it itself? Because again, operator should manage resources within the cluster, and operator should only uh any kind of resource related to our instance itself, right? Um so, it should not really also implement

some VMs somewhere. Um And that's was actually the easiest way to do. Uh in the next slide, I left out the turn server, but I mean, you get the trick. Um So, the idea in the end is we just built a very tiny, simple uh control API in front of our back ends on the VMs. Uh so, the operator then can just post or get or any

card operation Uh run against it and we have custom resources within the cluster, so the operator knows what to do. Right? So, he can take care of the resources within the cluster and he can post and interact with the APIs. So, then it's actually pretty easy also from a controller point of view. Within our reconcile function, we just check for, okay, how many backends do we have?

How many of them should be there? And if one is missing, then please register and do the fancy stuff within the API the control API and if there is one too much, then well, surprise, instance got deleted. So, I also have to deregister my backend. And there we are, right? Finally road trip managed, we have an operator and I know it was much and I I still

see a lot of interested thing faces, so that's good. I didn't lose you completely, but just to recap a little bit. So, what have we built then in total? Right? Um, so we have our operator and he's he, she, it is taking care the first of all of all the pools we have, right? So, we have a basic pool with basic, premium with basic instances, we have

a premium pool with premium and so on. You know and he takes care that there is always in our example three. The second, a user, maybe not the second, but the user on the client's platform tries to onboard and he chooses, okay, I want to have a premium package. So, the curl the API call goes to our operator. Operator knows, ah, awesome. Let's grab one of the

premium instances. I will assign it to that one specific client. I will do the tiny little adaptions, host name configuration, the external resources and then off we go and I will also put on a new instance um in the premium pool so that we are again up with three ones. And the operator No, the um client app can all the time poll for what is with my

instance? What is the instance? Uh but we also do proactively send webhook notifications, so it's not just just a polling thing. Learnings. I mean, I explained a lot um about CRDs and why we did that. So, I think that's really when you think of I want to build my own operator, it's definitely worth it at the very beginning think about, okay, what exactly do I need from

a custom resource definition point perspective? What is the logic? What are the different bits and pieces I need? And does it make sense to have everything in one or rather split it up to several resource definitions? Um it actually brings a lot of benefit to split it up from a logic. Um and also think about, okay, is this something I have to do or is there something

out there, external DNS or Postgres operator, for example, who can do things for me? So, I have operator operator communication, um and I don't need that that aspect at all, right? So, it's really worth invest some time at the very beginning, and I know how projects run. There's addition here and addition there and well, I know, uh but it will save you much time later on um

when you from the beginning try to make a reasonable split of tasks instead of doing that later. It will be just a mess and a nightmare. Also, it helps you with the whole, let's say, code base because what happens when doing it the the logic like the complexity and also your reconcile function and everything adapted to it, it will grow massively, like really intensely. You start with

a few hundred lines of code and then whoop, thousands line of code. So, it's really really uh a lot. In our case, I think you got it, like there's so many dependencies, so many things we have to wait for stuff. We have to create own stuff. Um again, it's actually way easier if you have split uh CRDs to really understand what where is what uh in in

your code. And I know we all love documentation. It's actually why we do IT, but it's actually really really helpful to document from the very very start exactly where is what. And also think at specific points, is there a way out how I can modularize what I'm doing, right? I know that's a not so fun task, but it's definitely worth it and will give will spare you

some headaches. A fun thing actually, cleaning up resources is way more complex than we thought in the very beginning because I mean, in the end delete, right? But there are so many race conditions and we underestimated that actually to be honest because we always thought about yeah, of course we have these race conditions installing the app, but yeah, at a specific point users might want to delete

it. So stupid thing like stupid example like I mean, you can't delete an S3 bucket after you deleted the credentials. Surprise, right? So we we don't want to have orphaned infrastructure bits somewhere in our cluster. So again, make up your mind. Finalizers might become a good friend of yours. Just try to also keep make some bring some love to deleting resources. Testing and debugging operators. I promised

debugging tips. The the bad thing is, I mean, you can do a lot of things like integration tests, you test your code of course, right? Like the basics. You can use some mock API. I mean, in the end the operator talks to an API, right? So the best case to test and try out and play around with operators is please use a real cluster. Mock APIs are

really really limited. Just use a test dev whatever stage fun cluster, but really try it out there because you will always find surprises when doing things because you thought about it, you thought a lot about it. You implemented and then still funny things happen from out of nowhere. So, um use use a test cluster to really try also when it comes to trying new features, please do

it on dev first. Uh things crazy things can happen. Interesting things can happen and normally I mean the idea normally is that when you push a new operator version to your cluster, the old instances are not affected, right? It just is for new ones then. Depending on the logic you have or potential errors. So, if you want to deploy a new feature, please test in dev first

if it will kill all your old instances {slash} if something other weird happens, right? So, that's maybe one aspect to it. It really makes sense to test it real life in a real cluster with a real Kubernetes API. But please in dev. also I mean that was rather the testing part. Debugging Debugging is really hard and debugging again depends on things we really love. Writing logs in

our code. Uh because this is really really really important. Um so, use um clear logging in your code base. Uh use Kubernetes events you can potentially use find uh and update your status really really regularly because in the end if you test in your wonderful dev cluster because you don't test on prod, uh then chances are pretty high something does not start and the operator should tell

you why or at least where it stopped, right? Maybe not why. I'm That's right. But where it's exactly broke, right? So, that's also a responsibility. Really try to um give a lot of updates. Report where you are creation flow. thank you. I hope I could give some glimpse, some insight, some inspiration. Um I know it it was much. I have 1 and 1/2 minutes for questions. There

is a mic somewhere. If you feel shy or just a question is too long for 1 minute 19 seconds, I'm still here up until Thursday lunchtime. So just stay here for a bit, approach me later on. Thank you. There's a question. Awesome. You have 56 seconds. Hi. so I have I'm going to be quick. Uh I've had the opportunity to actually deploy my own operator that I

have myself written. Uh and my question is how did you actually deliver them to the cluster? I tried with Helm. I'm also aware of the operator operator lifecycle management project. But for example, Helm has problems with CRDs specifically, which are core to the operator paradigm. Uh what's Can you give any guidance on this maybe? So to be honest, in our case we are using Flux. So we

have a deployment for the operator and the CRDs and then we just have Flux to manage that within our cluster. For us it made made it easiest. All right. Have you ever had any problems upgrading your CRDs for example this way? No, not as of now. But I mean it's released a few months. So it's not that old. But so far, no. Thank you so much. Thank

you.