OpenAPI Meets Kubernetes: Auto-Generating CRDs and Oper... Sergiusz Urbaniak & Jose Vázquez González

This talk focuses on the innovative approaches to autogenerating operators and managing APIs within Kubernetes. The speakers, Sergio and Jose, from MongoDB, discuss the evolution of Kubernetes from a mere workload scheduler to a robust platform for integrating with external systems, particularly emphasizing operator development. They outline various types of operators, including workload, configuration, and external resource operators, and share insights from their experience with the MongoDB Atlas Kubernetes operator. The discussion introduces tools such as 'OpenAPI to CRD' and 'CRD to Go,' which automate the generation of Custom Resource Definitions and Go types from OpenAPI specifications, thereby enhancing development efficiency. Additionally, they present a unified state machine for managing resource lifecycle and status conditions, showcasing how to create a seamless developer experience when interacting with external APIs through Kubernetes.

Yeah. >> Okay. Can you hear me? All right, that looks good. Um, okay. Welcome everybody. Um, I hope you're enjoying CubeCon. Who Who enjoys CubeCon? Hands up. Awesome. Very cool. Um, welcome to our talk uh autogenerating codds and operators the smart way. Uh, my name is Sergio, which is completely unpronouncable. Just call me search, whatever works for you. I'm a software engineer at MongoDB. >> I'm Jose,

also a software engineer with service. Cool. Um, so what are we trying to solve? I want to see another hands up. Uh, who in this room is still developing operators? That's very good. Cool. Awesome. And who in this audience is integrating those operators into developer platforms? Also very good. Okay, good news. You're in the right room. Um, so what are we trying to solve? Especially those of

you who do operators and who do platform integration? obviously are aware that Kubernetes is not only a workload scheduler anymore, right? I mean that was all the rage when the project was created. Uh but today Kubernetes is obviously also a foundation to build platforms. Um and more formally uh we see more and more demand and more and more things being done uh to connect Kubernetes to external

systems and this is also the stuff that we have been working on for the last two and a half years and obviously the whole notion of the cube resource model and the API centric nature of Kubernetes lets you in a great way um abstract and expose APIs to users in a unified way. Um what we found with the operator that we are implementing that the classic operator

pattern doesn't really work well. And what I what do I mean with operator pattern? I mean the pattern as we used to develop operators or still are that are launching workloads on your clusters. They're like a little bit of a different beast and they they work a little bit like in similar way uh in in different ways. So to make things a little bit more concrete what

are we talking about when I when I speak about different operator types? Obviously the elephant in the room is the workload operators right like the key idea is to run something inside cube something that occupies CPU GPU nowadays obviously memory network um independent if it's stateful or stateless obviously I mean we are from MongoDB so we're particularly interested in stateful applications so we run MongoDB on your

premises um but also obviously machine learning inference workloads on your clusters the other type of operators that we saw over time uh and I see this sort of like of a progression from from left to right in these slides um are configuration operators like these operators don't put any particular workloads on your cluster or don't even like churn but maybe maybe they put churn on the API

server and I see one API machinery maintainer here uh but the key idea is here to write operators that extend the possibilities or the functionality of your communities cluster most famously maybe you know search manager is a great example right I mean this This operator does not put any pods or any any deployments on your cluster, but it creates certificates obviously by leveraging cube APIs and CRDs

or you have operators that do policy enforcement um or for me for me the um orbback controller inside the controller manager that comes with Kubernetes is also a configuration operator or operators that are preparing ingress rules for um network traffic or network network configuration. And the last bit of or like the last type of operators uh is what I refer to or what we refer to as

external resource operators. And especially in the context of internal developer platforms, these became much more sort of like um interesting. Uh and the key idea here is to connect Kubernetes to other systems. Uh the stuff that we are working on, we have also from the MongoDB side a software as a service offering called Atlas. And by using this SAS service, you can launch MongoDB uh instances on

our service. So we expose an API to do so. And we me and Jose have been working on an operator. This whole like MongoDB Atlas Kubernetes operator kind of like um makes sense from the naming perspective that lets you provision those clusters. But obviously other famous examples from other ecosystems like you want to provision an S3 bucket, right? So there are also other things that you can

interact with with external systems just by leveraging operator ideas. And these are just some examples again like the Atlas Kubernetes operator that we are working on. Maybe who of you have ever used or leveraged ACK the AWS controllers for communities. I see some hands up. Okay cool. So that's quite known. Another incarnation of this idea is the Azure service operator and uh who obviously by the nature

of platforms has used crossplane providers. Okay, even more. So, crossplane providers are also fall into this realm of sort of like external resource operators. Um, with the crossplane sort of like ecosystem being the most agnostic one, right? Pro like by being able to provision a resource of any external service type. So, how do you implement such type of operators of like looking at prior art? Um and

one obvious way to do it and this is how crossplane providers do it is sort of like you rep terraform that is like you take an existing ecosystem that lets you provision external resources you build something around it and expose its functionality inside kubernetes. So crossbang created this very interesting project called upjet which essentially wraps terraform and uh the existing terraform provider ecosystem and exposes this functionality

inside cook cluster. Another nice project by the way I love the name Burrito uh has a similar idea. It reps sort of like Terapform providers and lets you like declare uh in in the context of a communities cluster um external resources and burrito like delegates to terraform. So the big pro of this approach obviously like you leverage an existing ecosystem and the terraform providers are there they

are implemented um by wrapping them somehow um you expose this functionality in cube and call it a day but there are some cons with this concept as well uh first of all like terraform providers are really not tailored for cube right like they they live in the realm of terraform resources and the like they are first class citizens to be integrated integrated in ACL obviously the other

point is you must have a terapform form provider in the first place. So if you have like some API in your company or uh maybe such a terraform provider doesn't exist yet. I mean this these frameworks don't help you. And the other thing is like it feels just a little heavyweight right like to wrap like a whole terapform binary you have to maintain the infamous state fault

somehow you have to hide all of this implementation detail from the user. So it feels like a little bit heavy heavyweight from an architectural perspective. The other idea is and this is the idea that we pursued um with what we will be presenting to you is the idea of translating APIs again like if you look at Kubernetes or the way how I look at Kubernetes like Kubernetes

is essentially is an API platform and the and the um ecosystem evolved around custom resource definitions but by the way this is not the only way how to extend Kubernetes but it's the most famous one um to expose new APIs to users. So why instead of like wrapping terraform we do sort of like an API to API translation especially in our context of the service offering from

MongoDB called Atlas. Atlas offers an open API endpoint. So it's kind of natural because maybe hope many of you know that CDs are really nothing more than like hopefully lightweight wrappers around an open API spec. So it it sounded quite natural to us. Okay, we have an open API on this side and an open API on the other side. So how hard can it be to convert

between those API shapes? But the idea obviously can be extended like to other API shapes as well like I had the discussion um quite some time ago with a former colleague to translate similar things to a protobuff endpoints or like to protobuff services. Um the con obviously of this approach is you cannot exist like you cannot leverage existing providers right you cannot like wrap an existing open

API to something translator. Uh so that that was sort of like the downside of it but the approach was so lightweight that we went with it. The question is how do you implement one right like we are developers. So uh I have the open API endp points on one side. I want to expose them as cd. So I download controller runtime. I spin up kind. Uh I'm

warm up VS code and hopefully nowadays like my AI agent and I have from controller runtime this infamous reconcile method. It's like what now? Right like and maybe many of you know like this this reconcile sort of like idiomatic reconciliation loop but this is like very abstract. It's like not very concrete. It still doesn't give me like a concrete recipe how to implement the reconciliation in here

and like who of you in this want to see other hands up has like maintained at least two operators and from operator to operator you saw like completely different code bases. Okay. So you know the pain. >> Yeah. So basically what we found out when we were doing this manually is that it didn't scale. It didn't scale for us. The Atlas team is is quite big. It's

actually several teams doing features for our customers and we were relatively small team for making the uh Kubernetes client side of that. Um yeah, so basically we we were having a lot of feature lag. We couldn't catch up with uh with Atlas features because I mean we couldn't go fast enough. But that was not the only problem because we were doing uh the features one by one.

There were several developers doing several resources at the same time. There was inconsistency at the development level but even worse at the user experience level. the word little inconsistencies in the behavior of of the um uh resources. And then the last point uh pain point is that um many many customers want to do uh IDPS integrated development platforms and that requires a level of flexibility that it

was very expensive for us to provide with our curated approach. So basically what we did is uh as as Arus was mentioning we flipped the the whole workflow on his head. Uh the old way of doing things if you're familiar is you usually if you you work in go is you figure out how you want your CRD go types to look like. Then you tell Cube Builder

to build the the GM the YAML CRDs for you. and then you stop uh coding. But uh what we did is basically we say okay we already have an open API spec which is similar to our CRDs. We just automate that step. So we have a tool that gives us the CRDs in YAML and then we have another tool that gives us the go code already and

then you start hacking on it. And I think you wanted to mention about the prototyping, right? Yeah. So like that's super important because like I I have for quite some time already the opinion that like most of us who start coding an operator and starting like with the ghost trucks as the first thing that we do is sort of like the wrong approach with an APIcentric platform.

It never like clicked in my head why this makes sense. When I have an API platform like Kubernetes is I really want to start with the API first. Like why should I write code to check out how my API will would look like? I want to like craft the API first and then start coding once I'm satisfied with the shape of the API. And this is sort

of like exactly what we also changed here. It's not only like the semantic similarity between the open API and the CRD stuff, like the whole like process of taking the original remote API shape, transforming it into the Kubernetes API shape, and then just make a stop. Take all your stakeholders on board. Talk to them. Talk to the developer platform teams. Talk to the product managers. Talk to

the communities. Is this how the API looks good to you? And then only everybody says yes, that looks good to me. Only then start hacking. And this is sort of like in my opinion even the way how cute builder could or should work in modern times of of sort of like API first approaches. So this is how it looks like live and in color. Um as you

know like we we as engineers are really bad at naming. So like to make this whole feature generation pipeline uh working we invented a couple of components to make this all happen. Um the first one kind of obvious open API to CG. It's a small uh binary. Oh yeah and another distinction is everything that you see in the upper half is sort of like happening at compile

time. So this is like real binaries that execute on your in your make file and everything that is sort of like in the blue is runtime. So these are mostly libraries. So the first thing that happens is you invoke a tool called open API to codd and it does exactly what it does. And by the way, if somebody has a fancy name, please let me know. I'm

still looking for a fancy name. Um it takes an open API specification and converts it into cube cuttle applyable cds. And that's something very important. So what we have like we have pretty rich semantics in this open API to CD configuration that lets you tweak sort of like the target shape. We have a notion of plugins in this open API to CRD tool which lets you tweak

and manipulate the final shape of the API in CD land. Um once you're done with this and this is sort of like the process that you do with your stakeholders. So use only like this one tool to do the final shaping of the API and then once everybody's satisfied with it then you start coding. What we then implemented is a tool called CRD to go, which is

kind of also obvious what it does. And again, I'm looking for inspiration how to call this little thing. It takes an existing COD YAML and creates ghost trucks out of it such that they are reconcilable by your operator, right? And and they are known to your to your uh operator runtime. And by the way, like I I heard from in the conference that there are other uh

also languages being used nowadays to write operators. Obviously, this could be C or D to anything, right? Um the last bit that is sort of like at compile time happening is a tool that we call scaffolder. The scaffolder also looks at the cd yl again no cube builder annotations just the original cdl and scaffolds the whole sort of like reconciliation loops and how it looks like we

will get into a second. So everything that happened until now is sort of like structural in nature, right? We like we converted one API shape to another and we invented three tools behind it. But that still doesn't give us any any notion or any semantics around behavior and these are like um those two additional components in the runtime down there that make this happen. The one is

the concept of a unified state machine. And what this means I will get into a second, but the last final bit is and by the way this was the most creative naming and again like if if George is watching here our director he came up with the name and I find it really awesome. It's called crappy and it's the and it's the abbreviation for custom resource to

API translator library. Okay. So I I love the name. So essentially what what it does is also what what what the name implies. You have a go truck which represents the cood on one side and then you have the open API or protobuff or whatever strct on the other side and this runtime library is converting between the two of those right so let's get into behavior coming

back to the original slide right reconcile what now does this like pictures help you until now not so much right so let's introduce the notion of of a universal state machine because this was something that was the missing bit for the behavioral aspect of of this feature generation pipeline tool and you know like KRM is all about sort of like stateful resources living in the API server.

So having sort of like a unified notion about the life cycle of those resources kind of makes sense. And if you look at the state machine, which is kind of a little silly drawing, like the initial reaction is like, okay, this doesn't look very impressive. And that's that's actually a good thing because when you think about it, every resource in Kubernetes follows exactly these states and straight

transitions, right? When you apply a resource in Kubernetes, no controller has ever seen this resource yet. So this resource is considered to be an initial state. Once a re controller or an operator starts reconciling the thing you could say okay ah I'm aware of this resource now I have to create the representation of this resource in this bespoken remote system be protobuff be it a open API

or soap if you still use these things and then you obviously go through the other states up until whenever the deletion time stamp is is set is the state would be called deletion requested and then you can also trigger off sort of like those longunning operations in your remote system go through a deleting state until you unset the delete uh you unset all the finalizes and let

Kubernetes garbage collect that stuff and we have this stuff sort of like from the importing side as well. So imagine you do not want to create resources on the remote system but you want to import existing ones. So we have a similar approach from the state machine here as well. What we added in the library is also the notion to sort of like tweak the state machine

as well like there are some resources in your external systems who don't know need those ink states right creating updating deleting right you like we refer to those resource types as atomic resource types like once you create it they are there immediately right so you can you can sort of like reduce the the scope of the state machine to what what you need so we are engineers

so how how does it look in code so we have no reconcile method anymore more. But what we have with this universal state machine library is we literally have state handling methods, right? And every state that you saw in the universal state machine has an appropriate handle method. So if the resource has never ever been yet reconciled in in in your operator, handle initial will be called

once you are done with the initial state. So you recognized okay I need to do something on the remote system. You just return uh result next state state state state created with some some text and then you will notify the universal state machine library. Okay, I'm done. I want to transition to the next state. The universal state machine library will persist that information in the status conditions.

And by the way, you don't need to patch status conditions. You just return the next state and the library takes over everything for you. And then you implement those those state handling methods as you wish to do so. And now it becomes a little bit more clearer what to do like you don't have this opaque recon method anymore but instead you just implement a very concrete state

machine. Um this is a very concrete example how open API to COD works. So on the left side of things you have the original open API specification. Um like with the component schemas whoever has seen an open API schema this is nothing nothing surprising. This is sort of like an example of the configuration YAML that gets feeded into the open API to CRD tool. Uh and it's

spitting out a CRD, right? And you can see like with those fields, they are kind of equivalent, right? So a field on the open API side is present in the CD on the right hand side. Um and in the configuration, you obviously configure things like the GVK, the versioning, any additional things uh that are necessary to augment the generation of the CD. For instance, you could mark

some fields as sensitive fields. So these fields become references to secrets automatically. Or you augment some fields to be referenced to other types like to config maps or to other generated resources. And this is taken over by the config yaml. >> Yeah. And the next is CD to go. As you know, you know what it does already. It takes the CRDs the yamos from the the previous

tool and it will give you uh go types for them. So you don't need to do the opposite. You you can go from the CRD. Um and basically it will generate one package for all the whole CRDs which usually are in the same group. And this one is a generic uh is a is one is our own the first generic an open source one. H the others

are the other tools are also uh open source but they are not as generic. This one you can already use if if you want directly. um uh features about that CD2 has the most important one I would say is a smart type naming uh is is the most complex because the translation itself is very simple but actually generating names that don't seem to be generated by code

that look like human uh human picked them is uh is difficult. Um yeah we you can also try to rename and do reservations of names so that you you uh tweak uh the way you want. Um we had a doc ports you we let Q builder do the deep copies for us so we don't have to do that again and we have plugins we have uh apply

configuration annotations so yeah it's pretty complete this one. This is an example of how CRD to go would take the CRD on the left and it will generate code on on the right. There is nothing surprising here. It's more or less what you would expect if you would do it yourself. Uh next one is favorite name crappy which basically is a um two-way translator. It takes client

objects that might be your own CRDs and maybe secrets and other things and it will generate uh it will give you a filled up um API object and the other way around. Um basically this tool had like a two-stage evolution. We started comparing apples and pears at the beginning in our code. We were comparing things from the spec directly to a the API that didn't make much

sense. So we had like a manual translation layer that we had to code uh you know manually and it was too expensive and then we moved to crappy. It's kind of a Marcelin and Marceler and it needs to have the information from the CRDs to be able to do the grouping and ungrouping of of the references like for instance secrets. And here you have an example of

how you would use it when you want to make a call to the API. Before do you do that? You take your your client uh your Kubernetes client object and you translate it to the API request uh yeah request. You do the invocation and then from the response you do the opposite. You convert it back to Kubernetes structure. Um the controller scaffolder is the last generator that

we have is this one is pretty one to one. We we just want to generate some code that is known by us but we want to do it for different types. Uh it also takes the CRDs as input and that's basically the way it knows what it needs to do. It's very custom code for our use case for now. It's open source as well. Um and things

that it does it generates uh some files that are totally autogenerated that you don't need to touch. There are some other files that are uh the ones for the handlers state handlers. Those you need to fill with with your implementation. Uh the indexes we provide indexes out of the box and they work uh directly with a state handler. And we have a sporter. Sportters are when you

need to uh you already have something deployed in your through your API and you want to start managing from Kubernetes, you can support that and you don't need to write it from scratch. Um yeah, this is the way the scaffolder looks. Uh I mean the generator we are using yen here and it's pretty much one to one. You kind of read in the generator what we gener

generate on the other side. Um yeah, the demo >> Jose, can you hold the microphone for >> So, I will. Okay. So, demo or didn't happen, right? Um I'm bluntly honest to you. I have a demo that I uh recorded inhand um just to avoid the anger of the demo gods. So, what you see here on the left side of things is sort of like our MongoDB

Atlas. And this thing exposes the aforementioned Open API endpoints. And on the right hand side of things, you just see a terminal and uh K9s and the operator is running sort of like in the background. So let's let's sort of like Whoops. Let's create a primitive in Atlas. In this case, we create a project. So a project in Atlas is something very very similar like to a

namespace in Kubernetes. It's sort of like a shallow, you know, holder that you can deploy resources in. For instance, MongoDB clusters. And like on the upper right terminal, you see the autogenerated CD. So this is like this was not crafted by hand. This was just fully autogenerated from the open API spec. This is kind group which is sort of like the internal schema name for projects in

atlas. And you fill out the spec like you see this entry property and this org ID name region usage whatever metadata is in there is coming from the original open API specification. You see a version on this spec thing over there. And if you're curious what that is, there was a talk that me and Stefan Shimsky from API machinery gave last year. This is a so-called major

CRD major versioning pattern that is allows us sort of like a seamless hopefully seamless API evolution without breaking existing customers and we can do very very very intrusive things like delete fields and completely restructure uh the shape of CRDs. So uh let's apply this and hopefully what you see here we go like on the lower right hand side is sort of like the universal state machine live

and in color right we applied the group resource to communities the reconciler picked it up and it set the ready condition to true and why is it to true anybody has a guess say it out loud it's created right so so it's created it's available in the remote system. So why shouldn't should it be ready false? Right? But that's that's the sort of like also uh the

important message here is that ready true false um is implied by the state. Uh and obviously uh it's also implied if an error returns from the remote API. But here you see on the left hand side of things in atlas this resource is being created. Um so let's add another primitive in here. Uh obviously we all about um MongoDB clusters. So there is another primitive type in

Atlas called flex cluster. Again like you fill out the body name. Um and what you see here is the group ref. Uh hopefully that references the group that we just created before. And this is another notion that we have in this feature automation pipeline. If you have ever sort of like translated external APIs to to Kubernetes, those external APIs very don't very often don't have the notion

of references between resources. they have them but like implicit via ids or something like this and we do this automated like automated conversion between those sort of like underlying identifiers that make two resources related to each other. So you as a user can just specify a resource reference and call it a day and it's all part of the feature generation pipeline. So let's fast forward like the

cluster should be appearing down there. Um and yeah here we go. Yes. So the flex cluster is ready false. Why is it ready false now? Now now you know why >> it's creating. Yes. Exactly. It's in the creating. It's in the creating uh state, right? So it's still not settled. Uh the reason is pending. Right. So and boop. Right. We see here on the right hand side

um it's already true. Why? Because the resource is settled in the remote system. Also by the way the the reason is also autogenerated from you from the universal state machine and why well the state is created. Uh one interesting thing is uh this output here as well if we will fast forward a little bit in the demo. Yes. One thing that I would like to highlight here

are the status conditions because you see like two status fields. Oh now I can take it. Okay. You see two status fields on the autogenerated resource. one the status conditions that we all know and love or maybe hate. Uh these are the standardized upstream Kubernetes sort of like conditions that you can hook in your Argo CD pipeline or like whatever standardized uh where we have the ready

true false semantics. We have the persistent current state. But what you see down there like with the v 2025 something something are additional status fields that you know come back from the remote API endpoint that have nothing to do like with status conditions which are like rather coar grain in nature. So most famously for us it's very important to be able to connect to a database right

so you have see this connection string down there which is like in the URI syntax and this obviously comes back from the uh remote system. This is not a field that you have to set as a user but this is something that comes back from the remote system and must be displayed in the status conditions then you can read it out uh from your application and connect

uh to the remote system. Um yeah that concludes the demo and coming back to the slides. Yes, disclaimer, all of this is open source, right? So, um you can go to our repository. As Rosie mentioned, some of those tools are still hosted in the Atlas Kubernetes operator uh repository. Uh yeah, those who take pictures, please have a look at the slides because the URLs are abbreviated. Um

most importantly, we want to move most of the tools into the COD to go um uh organization on GitHub. But and this is sort of like um the the message to you and the conclusions and this is where we would love to interact with you like do you think this pipeline is useful to you right because we would love to get feedback from you if this is

something you could leverage for your own operators if you are in the space of connecting external systems if you think this is useful please talk to us this will give us informed sort of like you know this will inform us whether it's worth pursuing like publishing those tools on the CD2CO um repository and in the organization Uh there is a lot of technical details on metadata on

the CRDs around major versioning the CRD specs the v2025 semantics there is a lot of things to rabbit hole much much much further and the 30 minutes are simply not not not good enough to cover all of this but if you have questions lower level please approach us we're here um yeah I hope you enjoyed the show um and happy hacking on new operators Thank you.