Metal3.io’s Path to CNCF Incubation: Governance, Processes, and Comm... Kashif Khan & Dmitry Tantsur

This talk explores how the MetalKube.io project has transitioned from the CNCF sandbox to incubation, emphasizing key aspects of governance, security processes, and community health. The speaker, Kashif Khan, a maintainer of MetalKube.io, along with Dmitry, highlights the challenges of transforming implicit knowledge and informal processes into structured and adaptable systems. They discuss the project's goal to provide a Kubernetes native API for bare metal provisioning and lifecycle management. The session outlines the significance of clear governance in maintaining coherence among various components, and the need for formal processes to ensure project sustainability. Additionally, they detail their comprehensive approach to security, documentation, and community engagement to enhance project visibility and adoption.

Welcome everyone. This talk is actually trying to cover how MetalKube.io project has navigated the path from CNCF sandbox to incubation focusing on governance, security process, and community health. We'll share concrete artifacts and lessons that any project can reuse. We are going to talk about this governance and security process and and security a lot. So, it's idea is not to bore you to death, but then when you

leave you are actually leaving with the correct terminologies that the project should need to survive in the CNCF landscape. few words about us. So, my name is Kashif Khan. I am maintainer of this project called MetalKube.io, but apart from that I wear a couple of hats. So, I am also co-chair of the CNCF technical advisory group for infrastructure and working as an open source architect in the

Ericsson software technology. And my name is Dmitry. I work in the OpenShift team. I have been a maintainer of MetalKube since nearly the very beginning and I have been a maintainer of ironic for more than 11 years. Right. what we ran into was a pretty common situation in the MetalKube.io project. We had a working system, a code base, and multiple companies were contributing. So, from the outside

things were looking really nice and healthy, but internally there was a lot of things which was implicit. The problem with was that if someone new comes in and wants to contribute or adapt the project, they can't actually verify the process as such and I'm talking about in the beginning. So, if they can't verify it, they cannot trust it either. So, the work that we had in our

hand was to turn this implicit process, tribal knowledge, and informal ownership into something structured as well as adaptable and reviewable. That's really the core of this journey that we are going to talk about today. So, before I jump into itself a quick context so everything else makes sense. What the project actually does, it provides Kubernetes native API for bare metal provisioning and life cycle management. The key

idea here is that you have a bunch of servers and then you want to deploy a Kubernetes um cluster on top of it and manage the whole life cycle of the bare metal server in a declarative way. That's what we are trying to solve. So, what traditionally was a very, you know, manual process with lots of scripts and environment specific logic, we try to fill that space

with a declarative model and and provide a way of handling the bare metal server in the Kubernetes space using YAMLs and then we have controllers which are then trying to match the states as we desire as users, for example. So, the system is trying to handle the provisioning and the life cycle declaratively. And then how we do that, it's it's not a task of a single component.

As you can see we have components like ironic which is the real engine which talks with the hardware, but then we have an API for the bare metal server which we call bare metal host and the controller which takes care of reconciling and matching the desired state with the actual state is called the bare metal operator. We also a plugin or provider for cluster API which we

call the cluster API provider MetalKube. We have static IP address manager which we call it the MetalKube's IPAM. And then of course we also put in a lot of effort in components like Open MetalKube specific version of the ironic image itself. And recently we have also worked really hard on making the deployment of ironic inside Kubernetes using the ironic standalone operator. So, as as you can see

each of these components is actually it has its own life cycle and sometimes even have a different organization contributing behind it and different release cadence as well. So, if you don't have a clear structure, this can turn into chaos very quickly. And that's why governance and process becomes necessary, not just becomes necessary just to keep the system coherent. So, at this moment I would like Dmitry to

take us to the early history and how we joined CNCF. Yeah, let's talk briefly about the history of the project. As often happens, the project appeared from urgent needs to handle in this case bare metal machines and of course handle them yesterday. Of course that's cost some corners to be cut when it comes to governance, when it comes to processes and the fact that we were building

on the shoulders of a giant which is OpenStack ironic already very major project at that point did help us with the speed, but also did help us to not to pay attention to certain aspects that we did not urgently needed to. Governance questions, processes we kind of put us put aside at that point because things worked and things worked well. After 1 year several companies were already

contributing to the project and started even experimenting with putting the project into production into their products. And this growth made us realize that not all things work smoothly based on that approach. As a case study for example, I want to tell the history of our quick start guide which is history that is not over yet by the way. We started again with the just basic needs. We

need to run CI, so we need to be able to install our software in a way that works for the CI and we need to develop which is a similar case because we are developers of project we understand it we can um run it pretty quickly. So, we ended up with a mix of Ansible and bash scripts. And then we ended up with separate documentation which was

written for operators kind of, but it quickly became outdated and were actually very rarely updated. Iteration number two was to okay insert XKCD about certain competing standards. We now writing one documentation to cover all the documentations which produced it in another quick start guide which at some point we ended up with three guides to quick starts on MetalKube. Neither of which were actually very well tested. And

this is sort of a situation that can happen when you don't have a clear picture of your project's maintenance path in your mind as you start and you just focus on the code and getting code done and getting code done very quickly. That contributed to our motivation to join CNCF as a to a certain degree disciplined force to make us work on our processes, to make our

processes align with the broader with broader ecosystem is doing, but also of course provide us a neutral governance so that we eliminate this concern about it being single vendor or double vendor project in our case. Open up possibility for interacting with other projects and other projects considering us as a serious partner to work together and of course access to various CNCF resources including this wonderful event, but

also a whole bunch of other programs, services, and so on. So when we realized that sandbox proposal was quickly drafted and I really encourage projects to do it quite early because that makes you sit down and write explicit goals which essentially the goals I just listed align the policies and the processes transfer assets to a neutral organization that will keep ensuring this neutrality and improve visibility in

the CNCF to attract contributors, to attract partners uh We didn't plan on code changes really. Sandboxing does not require you to create a better architecture for your code or solve some of the problems that Kashif is going to talk about. Or, you know, change how it's actually maintained. Nobody is going to come and approve your patches. It's just really creates a focus for you for your further

development. you need things like clear description of the scope, what your project actually does. And that's sometimes hard. You you need to stop expanding the scope and start focusing. You need to decide on your infrastructure and CNCF can help to a certain degree. And of course formalize your governance governance and how the thing is maintained. In our case owners file pretty well known and people recognize it.

And document document document so that people don't guess how you operate or don't have to ask in person. And that successful application opened for us a path to incubation and Kashif is going to talk about that. >> Yeah, thanks a lot Dmitry for the sandboxing process. The the incubation process is is a bit so that the transition is basically that you are not or no longer trying

to prove that the project is working. It's more about proving that the project is sustainable. It's neutral and it's secure and that requires a completely different set of artifacts. And I think we have already seen a bunch of these in today's keynote earlier. So, once you move toward incubation, as I said, the the evaluation is much more structured. CNCF, particularly the technical oversight committee or the people

who are actually reviewing the project for incubation, they look at five main areas. Governance, engineering principles, security community, and the ecosystem. The important thing here is that they like these are not independent. These are very coherent. So, for example, you might have a strong engineering principles established inside your process, but the if if the governance is unclear, it raises the concern about long-term sustainability. And similarly, if

security practices are not well-defined, that also becomes a blocker, regardless of how good the code is. So, the challenge is not just improving in one area, but building a balanced system and a balanced ecosystem in that sense across all of these five bullet points. And that's where most of the work actually happens. So, one of the important structural changes that we made in our project was introducing

a dedicated repository called community. We don't claim that we were the first one doing that, because when we actually did that, there were also other incubating projects, which we actually get the idea from, but then we have tried to make the process a bit better. So, before this, for us, the governance information was sometimes scattered in multiple repos, as you have seen that we have multiple components

as well inside the metalcube.io GitHub org. And some of them processes were even not documented at all. So, that was kind of creating a lot of friction and confi- confusion within the community. If you are a contributor, as I said in the beginning as well, you don't know what to check or what to look for. If you are a reviewer, you cannot get a full picture of

how you can, for example, become a maintainer. So, we are we tried to centralize everything into a single repository, which is the community repo. This includes the governance policies, contributor roles, the type of roadmap that we are trying to look into, the adapters who have used or who are using the project currently in production, and also other organizational information. The key idea here is that this repo

can become source of a single place of of truth when it comes to the organization of the project And anyone can go into this place and and understand how the project operates, and that way it can significantly reduce the effort to evaluate the project as well. So, as I just mentioned in the last slide as well, the processes and the documents that we tried to actually maintain

currently in the community repo are, for example, the code of conduct, the contribution guide, the contribution ladder, that means how you can actually start contributing and eventually move to a level of a maintainer, for example. We have the contributor role role definitions in that, the maintainers guide, the public project roadmap, or where you can find it at least, and then the adapters list as well. By making

all of these things explicit and consistent, we were we are trying to remove the ambiguity. This is important as not only for the contributors, but also think about those who want to really adopt your project, but they are not sure if the project is sustainable or not, or what is the process of going from one step to the other inside the project, are those defined or not.

we also introduced a formal contributing ladder. Initially, the progression in the project was very informal. People were contributing and they were trying to gain, and once we have actually gained trust inside the community, they were gradually receiving more access. But that was working when the team was rather small, but when we started seeing from different organizations eager to come and contribute, then it was kind of not

scaling the process So, and of course, it can also create a perception of bias, because if the process is not transparent, you we we don't know how how the project is handling all of these governance within itself. So, by doing this, introducing the ladder and then having how you can actually achieve through the levels, and what is the promotion we were trying to apply the same access

control principles to the community itself and making it also visible. Then comes that making the community activity visible also, not only to the contributors, but also to people who want to adopt as well as the reviewers of of the CNCF when it comes to technical oversight committee. So, the communication channel is public. ways to reach out for us in public, like we have Slack and mailing list,

and we have weekly meetings, which are also the minutes are published and and viewable in public, as well as the recordings are also uploaded to the YouTube. So, again, the reason here is the transparency that we have when we are discussing and trying to make a decision on features or the processes and how we are actually adding even the feature in features or processes itself, all of

those things are transparent and discussed in public. And then, on top of that, we were also we are also trying to use tools like the LF X insights or clone monitors to provide these objective indicators about the project health in different repositories, for example. So, again, the idea is that anyone can observe how active and healthy the project is. And on the engineering side, we had to

formalize the the practices that we do inside the the project when it comes to the roadmap, the architecture of the project, and the release processes. So, before this or before having this, we were we had releases, of course, but then the process was not again fully defined. And again, if someone is trying to consume your project or the code itself, it it was not certain that what

is the release cadence of different components, and if if we have regular releases when it comes to fixes and new features and CVE fixes even, for example. If you don't know what the next release is coming out or when the next release is coming out, or what it includes, it's very hard to rely on the project itself. So, we also again focused on consistency. Currently, we are

we have a good and automated process where we actually just pull up a PR with the change log, and once the reviews are done, it get merged, then the automation comes into play, and we try to release from there onwards. And we have regular releases, clear documentation about it, and we also try to have be predictable when it comes to the behavior. The goal here is to

make the system reliable, not just technically, but also operationally. And lastly, the security. Well, security is is very important, as we know. It's it's we need to make it more structured, and we need to also define the process how we handle the the processes when it comes to vulnerability, for example, because before we did that, this process again, the responsibilities were very implicit. But then, we tried

to have dedicated security team, trying to handle all the disclosures, for example. We have a security lead, and we have also made sure that if there are like there are some vulnerabilities that you need to report, what is the process correct process of actually having this through a dedicated community channel or communication channel, instead of opening up issues in the GitHub itself. So, this actually ensures that

the security issues are also handled consistently and more importantly, is handled responsibly as well. we have one of the requirements for the incubation is to create a security self-assessment that and also have the security policy formalized. And the problem here is that the security claims achieve, we also have evidence for that. So, you need to show, for example, how the system is designed, where the risks and

the force this forces the project to think systematically about security as We also introduced the security insight file, which is again validated through different automated processes, and again adds another layer of rigor in in the making. And by doing that, the security information is not just documented, but we also try to make sure that it is also enforced. And this is also one of the level of

detail that CNCF expects during incubation. And the last thing that I again want to highlight is the ecosystem. So, although you have these defined processes, and you have the governance and everything, but for incubation, ecosystem is also ecosystem adoption is also very critical. You need to demonstrate that the project is used by multiple independent organization, not only the few people who are contributing, otherwise, it kind of

looks like a single vendor project itself. So, we also maintain a public list of adapters, and also highlight the integration with other projects and systems, like Cluster API or projects like Sylva, for example. This is again an evidence that the project is providing real value beyond its original creators, and it also indicates that the project is likely to sustain over time. before I jump into the the

new process itself, I just wanted to highlight also that having all of these things, but then on top of that, we also needed to show that there is an evidence of what we are trying what we are claiming. So, how many active maintainers do we have? What is the like contributors and from which organization do they belong? So, the important thing again is to show the the

reviewer of of the technical oversight committee that that also not only on the quantitative side also the qualitative side we are trying to make sure that the governance is complete and this is kind of also the piece of evidence or information a reviewer is is looking when they try and and check if the project is incubating incubation ready The process has evolved since the CNCF process I

mean for getting incubated. I'm just going to quickly highlight what is the new process now because we didn't follow the new process which is has been established very recently. But the idea is that you need to have different sorts of self-governance and self-technical review as well as the security review itself. And we tried to map also how MetalKube parts of it in the governance process during the

incubation issue. So basic idea is that the maintainers or there would be one of the sub project TOC sub project reviews people would actually come along and talk with the maintainers and then make sure that you also have the self-governance and self-technical review up to date which is a living document kind of thing and then you show that how the what's the structure of your project and

if you meet the criteria to get incubated or So again the idea is to reduce the ambiguity and speed up the review process in itself. At this moment I would like to ask Dimitri to take us from the learnings that we had. Yeah, let's see what you can learn for your own project from our experience. Things that worked as Kashif mentioned focus all the government issues in

a single place that improves visibility and discoverability tremendously both for adopters and for reviewers. You have ideally to treat governance contributor ladder security processes as first class features not an afterthought as early as possible and as explicitly as possible. It helped us a lot to draft our application and publicly and attract as many people from the community to review it as possible and seek help from our

organizations people who know the process. And it also helped evolve to evolve the application as changes as conditions change. The self-assessment style works well it allows you to take a sober look at yourself as if you were an outsider reviewing your project. What what's the security our architecture works what is the project health overall be honest to yourself because that's how you will be evaluated. And automate

automate automate everything can be automated should be automated for example we collect maintainers from all the sub projects in a single place again for visibility. Of course all security metadata again very helpful to run it in the CI rather than manually. If you were starting today we would definitely introduce governments and all those processes in the community repository early on maybe from the beginning. And also start

designing for security and for being assessed and for self-assessment from day one. We definitely align our CI patterns our release processes with CNCF from the beginning because we know who wants to apply so why not do We would work user first on our documentation on our quick start and not just assume that everyone be comfortable with what CI is comfortable with developers comfortable again visibility are key

words here. But also realistically staged approach works fine. You can start it code first and figure this all out we have done that. You can use sandbox application as power to force you into more compliance and you can use incubation as a health for you and it's success is a successful sign. A practical checklist for you here feel free to take a photo it's mostly repeats the

points for governance community you need to governance documentation at least all key policies and everything you need. You need to develop a contributor ladder and how people become contributors a maintainers guide and you need to have a road map not just for CNCF again and for review. Security make it public make it explicit do self-assessments uh provide security metadata. Document it's very important document release process even

for your internal goals I can tell you people forget how we release we ourselves forget. Use CI to enforce the processes as much as possible don't rely on humans to remember things all the time. Um try maybe to separate code from governance so again people don't have to go and chase what is there and look for different bits of information everywhere. Draft your application keep checking it

against current CNCF criteria collect evidence for things you claim be it adopters or governance or security and put it all in one place. And yeah as a summary CNCF readiness is as much about governance as it about the code or even more. Um we have some successful recipes here like a dedicated community repo or self-assessment processes and do invest in clarity in terms of which roles how

people migrate between these roles what processes look like and And we hope it helps you if you want to leave us feedback please use this QR code and let us know we'd be happy. As a bonus for you we have still have a lot of MetalKube t-shirts they're in the back you can find them just stop by we'll give you some t-shirt. And that's it we have

don't have a lot of time for questions but If you still have some questions feel free to step up on the mic. Yeah please. Hi thanks. Can you elaborate a bit more on CI CD patterns for establishing in CNCF that you mentioned. Yeah so the basic idea in CNCF is to have some sort of like a vendor neutral neutrality when it comes to CI CD both like

when you are introducing new changes in your project how it is first of all formally reviewed and maintained so what is the process of having reviewers and in approvers and then how automated the whole thing is in case of the CI CD as well like for example if you take MetalKube we have things running on prowl we also have Jenkins so idea is to have like all

sort of integration but as well as when you are trying and when we also have GitHub actions for example for automated checking but then as much as possible try to push these things in vendor neutral hardwares CNCF is actually providing these sort of resources to the projects. So reach out to the CNCF service service desk if you are a maintainer or contributor and then you will know

that what sort of resources you are entitled to for your CI CD.