Beyond the Cloud: Managing Baremetal the Kubernetes Way Using Metal... Ádám Rozmán & Nicolas Belouin

This talk focuses on managing bare metal infrastructure through the MetaCube project and its integration with the Silva project as a use case. The speaker introduces MetaCube as a comprehensive lifecycle management tool for bare metal Kubernetes systems, capable of addressing key challenges faced during deployment and maintenance. The architecture of MetaCube is modular and supports multiple layers of abstraction, from multi-cluster management to bare metal host management, using custom resources and templates. The session highlights the importance of IP address management in complex environments and discusses the integration of modern provisioning techniques with tools like Cluster API and Helm charts. Additionally, the speaker elaborates on the Silva project's goals to facilitate cloud software frameworks for telecommunications and edge computing, utilizing open source components.

Welcome everyone. Thank you for joining us in this late stage of the event. I I see that uh many of you joined. I hope you will enjoy our presentation. My name is Adam Rosman. I'm from Ericson, more specifically Ericson software technology >> and I'm Nicolola Bulwa from Souza most specifically the engineering telco team. >> Yeah. Today we will talk about uh beyond the cloud cloud managing bermatal

the kubernetes way using the meta cube project and showing you silva project as a specific use case for meta cube integration. So I'm a one of the maintainer of the meta cube project and I will first walk you through that what is the meta cube project. So the meta cube is a bare metal life cycle manager uh clustering tool and in general an end to end solution

for many of the bare meta kubernetes problems that you might face uh in your career or in your You don't have to really memorize this uh picture. This is an reference architecture of the whole MACQ project. This only shows the active part. So the controllers that manage the life cycle from a very high multicluster level down to the nitty-gritty small uh course to the power cycling of

a physical machine and so on. So you can already see on this uh picture that there is a uh dashed line a dashed line in the middle and everything above that line is about multicluster management. If you don't want multicluster management or clustering of bare meta machines, you don't need that part. You can leave that off. But if you uh need that functionality, you have to take

that part. And the part below the dash line is is uh purely about bare meta machine management, life cycle management, operating system installation, firmware updates, all these all these things that you have to do with a machine. So on the top you can see that we integrate with cluster API everywhere where you see the chevrons not the arrows those are interfaces and because there are interfaces in

so many places this project is very modular and extendable and and you can configure it to your liking and we will see later an example from the sila project what parts they take and what parts they don't take. So on the top as you as as you can see we have cluster API standard Kubernetes component then we have an official provider for the cluster API in the

form of the cluster API meta cube provider that's just an interface implementation we also have uh support for the cluster API IP address manager and we have our own IP address manager that also follows the same conventions as the cluster API uh IP address manager. Why do we have IP address managers? In some cases in in some environments, you can reach your own uh your data center

through some very complicated routing or you have very strict uh security rules or simply you don't have a DHCP server in that uh data center in that specific environment and you cannot bring your own. So you have to handle somehow the IP address management, IP allocation and so on. So we have a Kubernetes native way of doing it. We have the same sort of leasing logic that

you have in a DHCP server just written for uh Kubernetes and then that uh and we have pools that you can set up and then claims are generated against the pools and then the claims are served so or or fulfilled and then you have IP addresses attached to your nodes. This this can grow very large in some of the deployments I work with. Each machine has 14

different network interfaces or even more and they take uh IP addresses for I don't know 20 different pools. So this can go uh grow quite large. And then we have the cluster API metac provider as I said uh that basically provides a mapping between a cappy machine. So the cluster API machine and we have a we have a pair for that called the meta cube machine and

then uh basically we uh we also have then uh further abstractions for for uh templating. We have machine templates. So you can create the same template for a large number of machines and then you can also create data templates. So when you deploy a 100 machines, you don't have to make uh custom uh configuration for each of them. You can basically write templates and then those templates

can even take values from the from the IP address manager or other tools and then build your configuration files for you. So that's that's the clustering part and templating part. And then below that we have the the bare metal uh host management. In the middle we have the bare metal operator. This operator here this provides you a custom resource called the bare metal host. That's the root

of everything. uh that's the minimal custom resource that you have to deploy and and this and and uh one other components is the minimal deployment that you need to present a Kubernetes uh present a physical machine in a Kubernetes space. So you will have a onetoone mapping technically of your physical machine into Kubernetes space and then this bare metal operator is also extendable. It has as you

can see a chevron below it. So that means that we have an interface. It's called the provisioner interface. We provide in the upstream two basic provisioner. One for testing and one uh called the ionic provisioner which basically means that the bare metal operator can talk to the hardware through that uh provisioner. And then uh why ironic? Isn't ironic an openstack project? Yes, but we provide an uh

separate image. Uh we release that image. you can download it from us and that image uh has a standalone version of the ironic. You don't need to install OpenStack or touch OpenStack or do anything with that. You just use our image and all of the auxiliary components that Ionic needs are packed into that image. It is deployed as a podent and you can use it as your

provisioner. If you need multiple of these ironics because you have very large deployments, thousands of physical machines or you have your your uh different machines uh geographically separated very far from each other, then you can use the ionic standalone operator which will help you configure the Ionic because we know that configuring irononic is very complicated. There are hundreds of configuration options but by using the standalone operator

you can help yourself or or your users can help themselves and and that's just a utility tool for managing a lot of ionics. Okay. So we are at the bottom. I talked about the provisioner. So uh what does the provisioner do? Provisioner talks to the baseboard management controller of the machine. What is that? That's a separate little computer that is sitting in every enterprise x86 hardware basically

within your server. It usually has a separate uh network interface and it's usually always on and that provides us functionality or or option for full life cycle management because we can turn on turn off your machine all the time. We can we can have uh BIOS updates, firmware updates. we can uh uh configure the boot boot order and do all these things that generally someone would do

manually or would do would do through the web API of of such a BMC and we can automate all that and uh with that power that we can automate all these things we can also start provisioning meaning installing your operating system and installing all your tooling that you need to to bootstrap a Kubernetes node and join it into the cluster And uh we have two different ways

to do it. There is a managed and an unmanaged uh way. So unmanaged way is basically you just attach your ISO to the machine and you let it go. You don't care about it. Or there is the managed way when you deploy our agent which is a ROM disk. So it loads into the memory of your machine and then it has a more granular way uh of

uh downloading your image, writing it to disk, writing your configuration options to the disk and then uh rebooting your machine. And uh by using our our agent you have extension points and you have better control and better uh uh uh logging and and and you will see where where uh the agent is at at specific state of of the deployment but of course that means that you

also involve the agent. It's up to the user. And uh with that I just want to highlight one other thing. uh we support redfish ISO IDRA and uh IPMI protocols for the BMC and uh that's that's uh one important thing so if you have something like IRMC or something else we cannot work with that and all the machines that we manage has to have a BMC and

then uh last point on this picture is that at the at the this corner you can see that we are working on uh switch management so there will be an obstruction of the physical switches that you have next to your Rex and uh you will be able to manage those also with this bare metal operator and they will have separate CRS. So that's about the architecture. It's

a big uh configurable stack and then uh one thing that you have to consider before you start using meta cube do you want clustering? If you want clustering, of course, you need at least a cluster resource and the KCP template. But if you want to manage physical machines, the minimum that you need is a boot MAC address. Any MAC address that that is uh visible on that

specific machine, we need one. We need the address for your BMC, access to your BMC, credentials for your BMC and the disk image you want to install. And then you can ask yourself all these questions and those will help you uh decides how to uh integrate and how to structure the whole meta cube. And with that let's see a use case. >> Yeah. So basically you must

be asking yourself first what is the project silver because uh this is not something you usually know from just knowing metal cube. Uh so uh project silva is actually a telco initiative uh that is under the umbrella of the uh Linux foundation in Europe and this is this has the this has the goal of uh making a cloud software framework for telco and edge. So that fits

for telco and edge requirements. uh it also has a reference implementation for that and uh the uh members of this uh silver organization are mostly uh carriers, network function vendors, hardware vendors and infrastructure software vendors and providers. uh it's built only on top of open source components and it uses a fully declarative and githups uh approach uh mainly using uh flux. So the main architecture for this

uh the most usual one I would say is to have a central site and then to have multiple edge sites that will be hosted. Uh the central site is hosted either on bare metal or on the cloud whatever and it runs all the uh it runs all the uh management system. So this is where you will have the uh deployment of mal cube. This is where you

will have the deployment of cluster API and it will manage the lots of edge sites you may have uh to run your workload. So here the example workload that is uh is for like uh I think a 5G deployments but uh this is not something we're going to focus on. So we are really going to focus on what how we deploy uh this kind of uh thing

on bare metal. So with metal um soil is really centered around hem charts and using uh cluster API and using flux to uh deploy those hem charts. So you have a really big silver units helm chart that goes where you put all your values to um configure all the components. So uh be it uh cluster API, ML cube provider or other providers if you need other providers.

Uh it will also be where you will have all the uh uh Kubernetes linked provider for cluster API. So currently it supports uh the K2 provider, the Cubedium provider, uh the OKD provider and the canonical um Kubernetes provider if I remember correctly. Um this is also where you will deploy all the needed uh infrastructure components. Uh and then you will have another set of helmshot that will

all uh define a specific like um cluster edge cluster. So this is where you will have the definition of the uh in for bare metal of the bare metal host of the uh different capy uh resources that uh Adam just talked about. Um so now as as I said silver is using metal cube and uh the way it's using metal cube so to answer the questions basically

the list of questions that uh Adam has about what you need to use metal cube. So here most of the uh BMC like are using red fish nowadays and here we're specifically using red fish with virtual media. Uh why virtual media? Because this allows to get rid of the uh need for nail to link between the uh uh management clusters and the edge clusters. And this way

you just need IP connection. It's way easier and it's also ease the deployment of ironic since you don't need to give it access to the host network. Um the uh like I said the bare metal host and all the uh capy uh related uh resources are are deployed are deployed using the uh one of the silver hand shots and uh to if you want to use bare

metal or when you want to deploy the initial management clusters you are going to use a bootstrap cluster that will then gener provision the uh management cluster and then pivot the uh resources into that management cluster. So it becomes self-managed. So if you're using uh if you're using bare metal for your management cluster, it will also be managed using uh metal cube in a self-managed way. So

this results in uh this simplified architecture I would say. So we are deploying the uh capy uh resources, the capy provider for milk cube uh the IP address manager if you need it. Uh this one is optional and uh those three are uh deployed using their uh customize um deployment method. And for the bare metal operator and the uh ironic uh image and pod those are deployed

using a helm shot that is the one provided by soua uh simply because there are currently no upstream metal cube uh chart available. Um then the uh ironic part we see we have way less component than in the previous um in the previous um chart simply because we don't need like the DHCP we don't need uh the u keepd we don't need uh the ipixie uh stuff

because we are not using the this uh pixie boot uh elements we providing an IPA provider that embeds the uh the OS image uh simply to uh ease the air gap deployments that are pretty common in the space. And uh this makes a smaller I would say uh implementation that the ones you can find on all different uh metal cube flavors. But this is what we have

for for for this use case. So I will hand it over back to Adam. >> Yeah, thank you. You will find the QR codes in the following slides. This QR codes points uh to the book. So you can follow along. I don't want to bore any of you with CRS and lines of code or anything. So if you want you can look look it up in the

book. I will just talk about the general process. So let's say you deployed with Silva or you deployed with the uh just with the upstream uh components. First thing that will happen that you have to deploy it to some sort of uh initial cluster we call it an ephemerra cluster or bootstrap cluster it can be a kind cluster a mini cube your existing uh control plane an

existing worker it doesn't matter just a kubernetes environment you deploy that all the components the way you like with h with customize with with a CLI tool with your own code whatever then after that uh of course this gets bootstrapped You check that every port is in in a correct state. Uh you are ready to go. Then the first thing you do is you initialize or register

a BMH uh BMH bata host one or more depending on your use cases. You provide those few information boot address, BMC address, protocol you want to use uh credentials for your BMC and then as as soon as you provided that valid information and and applied the bare meta host two things can happen. If everything was correct, depending on your configuration, there will be either an automatic hardware

inspection where you where the agent will be deployed, inspects your machine, returns to the BMH host, uh all the inspection data that ranges from from block devices, network cards, uh instruction set of your CPU, all these things or uh if you disable this then there won't be any sort of initial initial inspection. You can also bring in bare meta hosts in a provisioned state. Uh we can

talk about that later if someone is interested uh during the questions. Uh then after you have this registered uh host if you register them correctly they will end up in an available state meaning they are available to be picked up for provisioning and then you can do provisioning directly on the bare meta host custom resources by providing the image spec. So there is a spec field image

you fill it out with some uh image address that you have like an Ubuntu image or whatever and then it starts automatically provisioning that machine or you can rely on uh cluster API meta cube and copy and build a cluster and then and then u you will have your your meta cube machines and your meta cube machines uh will be built based on a machine template and

your machine template will will have a host selector portion. Host selector means that that it selects bermata host based on labels. So if you have an available bermata host that has a correct label then it will be automatically picked up and then it will be the base of your capping machine and that will be joining the cluster after the whole provisioning process ends. And after you have

your machines and they have joined the cluster and you you have a working cluster, you still have day two operations you can do as as uh as it was uh mentioned before. Uh you can pivot. So you can move the whole management uh plane of of the meta cube onto the managed cluster. So you can make it self-managed. You can do rolling update, change the update. Uh

you can do remediations. You can uh sideline machines from the cluster. You can swap machines. Uh swap host under the machines. You can do different servicing activities. So we have quite a lot of operations. Basically what cappy supports we support plus a few uh minor things. Then uh let's say you want to upgrade how the upgrade process looks like. The left side is basically what you have

with every uh cluster API upgrade process. So you initiate a rolling update, you have your mock surge. You have all these options. What will happen is that uh your uh cluster will start scaling in scale or outscale and then and then it will start removing a machine and then making another machine. And when this switch happens that old machine is removed on the capy capry level and

the new machine is created then comes in the the special part for the bare meta host management because every time we have to deprovision your machine because we don't know that will that machine be picked for the same uh uh meta cube and capy machine or not. You can specify cleaning methods. So at this point you can wipe the the whole whole all of the disk in

the machine. Wipe just a subset of the disk. Wipe just the disk where you want to install. So there are a lot of housekeeping activities that can happen automatically if you configure. But your bare meta host will go into this deprovision state and then based on the host selector that I mentioned uh the meta cube machine and the copy machine will pick a new one or the

same one. We have options for that too to always pick the same one and then it will do a new provisioning uh cycle on the on the machine and at the end of the process it will join your cluster. One thing to mention I forgot to mention that uh this provisioning process I mentioned that there is a restart at the end and we don't have control over

how your machine will join the cluster. uh we have ability to use the cubeadium bootstrap provider. So your user network and metadata through the cubeadm bootstrap provider I are injected into a config drive and cloud in it or if you use another bootstrap provider the the configuration tool of that provider will pick it up and and join it into the cluster but that's beyond the scope of

bare metal life cycle management. We don't track that. So the time after the reboot and and and before the the node joins the cluster that's your own custom uh flow and then a few advanced topics uh before the end uh because it was mentioned in the submission we have options for firmware settings. So through the BMC if you have ISO Hydra or Redfish you can uh specify

firmware setting you can specify firmware schema where you tell that what is allowed to be changed in your firmware and uh what what limits and values are allowed for your firmware configuration and uh that will be the base of validation for you setting uh the firmware settings through the host firmware setting CR. So these are separate CRs. They all all link to a bermata host. But that's

the way how you can configure a firmware through your BMC. And then we have yet another firmware related uh functionality. This is the firmware updates. This is al this also goes to redfish compatible uh BMC's. You have to have also like the specific hardware model have to support all these uh features simply because all all the hardware and all the generations of hardware are different. But we

have this host firmware components. Uh I forgot to mention you can have multiple firmware schema uh to use as validation based towards the firmware setting and same way you can have multiple host firmware components. These are very simple. So they they just point out that you you have uh you have expect like an initial state of a firmware like my BMC uh uh is has this firmware

and it's on this version that was the like a given version that's the base version and then you can uh file a new host firmware components and uh in the spec you can put update the name of the my firmware components then then uh a link to that firmware and then we will take care of that And this gets uh handled when your machine is in prepar

prepar preparing state. That means uh before right after you register the machine. So when you initially specify the bata host and all these components or if you have a provision machine you have to deprovision it uh and reprovision it to to basically init initial initialize this uh this preparation state. So at the time of restarts or reprovisionings that's the time when we can do this sort of

BMC based uh firmware upgrades. And uh with that uh that's all we wanted to mention. If you are interested about the project you can scan the QR codes they will take you to the respective uh projects. And I think uh we have time five minutes for for questions. Thank you for the presentation. Uh what do you mean by configuration of top of top of rack switches? Do

do you support all types of switches or or do you support uh only layer two or layer three as well? >> I I will show or I will rather go back and then uh I will uh explain. So what will happen is that we will basically adopt OpenStack's generic switch tool and all those uh physical switches that are supported by this project called generic switch. I don't

know the exact brands uh but we will have support for those and you will have uh custom resources that are from userfacing perspective are very generic. They just uh allow you a few options like von ids and all these things. We will extend it with in the future but behind it the hardware support is whatever the generic switch supports. >> So this will support BGP as well.

If I the CNI supports PGP this this will work >> I I uh I actually don't know what are the limitations on generic switch. So I don't want to say but uh I I can just say that check out generic switch if it has support for a protocol or an option then we will eventually support that. >> Okay. Thank you. Hi, thank you for your call um

for your presentation. Um yesterday there was a presentation from another room about uh cluster API in place updates. Yes. >> Uh are you looking into it? >> Yeah. So uh technically there are the new these new features I think in v.12 the chain and the in place chained we support without any uh problem. Technically we support in place also. uh the the way that in place is

happening as far as I understand that uh there is this process of SSH in and doing work with the cubeat and all this thing and upgrading them it's fine but we want to come up with our own hook that kind of uh does a more uh complete way of in place we call it like we have this process right now in the provisioning that we so soal

reimage the disk or or or your disc so we clean and install something on your disc We want to make that seamless with this in place upgrade. So yes, we support it, but we want to go one step further and have our own separate hooks for it. Yes. >> Providing the same flexibility and the same like you want to add just an SSH key. You don't need

to reinstall >> Yes. Yes. Yes. >> Perfect. Thank you. >> Hi there. First of all, thank you for the talk. Very interesting. >> Um, if I understand correctly, you don't have any preference for operating systems, right? So we could technically use it to roll out Open Shift clusters and nodes as well, you know, just booting the core OS layer and then they have their own mechanisms to

bootstrap a >> Yeah, this this meta cube stuff is already uh something that's being sold with open shift. So I I guess there is support for that if you ask Redhead. Uh yeah, you can uh roll out any flavor of Kubernetes. There is there is one important thing that in the upstream we say we only support the Cubed and bootstrap provider. I know that in the Silva

they have K3s bootstrap provider and there are others who use ignition bootstrap provider because it's part of it's it's being sold as or there is an open shift flavor of this I don't think there is some bootstrap provider but what would just help you put open shift on top of this but there is an open shift solution for this problem so >> I know but we're trying

always to stay out of the red products as much as possible so we could use the same mechanisms to roll out open shift rancher and so on >> yeah but theoretically If there is no ready to use way, you can relatively easily do it by making a bootstrap >> Cool. Thank you. >> Yeah. Uh thanks for the presentation. Um just one question about updating the the nodes.

So um are they updated in any order or can we have like workflow updating first? Uh so you can you you have options uh this mock surge option and and uh like roll out uh strategy you can set that so you can go one by one two by two uh the control planes can be done one by one and then the workers can be scaled in any

way you want them. Uh if you want to uh you can also limit which uh BMH is or BA hosts to take with the labels. So if you can play with the host selector uh and we have the node reuse feature. So we we we allow you option to always pick the same node. That's that's kind of the extent of how you can uh handle u the

roll out. >> Okay. If no more question. Thanks. >> Thank you for