Evolving Baremetal-as-a-Service: Secure Multi-Cluster Network... Yushiro Furukawa & Mitsuhiro Tanino

This talk introduces Flower, a next-generation private cloud platform developed by LY Corporation, which focuses on bare metal as a service (BMaaS). The speakers, Yushiro Furukawa and Mitsuhiro Tanino, explain the architecture and key components of Flower, emphasizing the unified API and the control plane built entirely on Kubernetes. They highlight the importance of reliability and scalability, detailing how Flower manages over 3,000 bare metal servers across multiple regions and availability zones. The presentation also delves into the integration of security policies, management of hardware using IPMI, and the challenges of provisioning in a large-scale environment. By transitioning from Ansible to a Kubernetes custom controller pattern, Flower achieves much faster deployment times and lower operational costs.

Welcome to our session. The title is evolving bare metal as a service secure multi cluster networking and secure identity automation so long, okay. And I we are so excited to share what we have built and learned. So and you already had a lunch so maybe getting sleepy but yeah, please listen to our presentation. Okay. Let me introduce myself first. I'm Yushiro Furukawa a senior software engineer at

LY Corporation. I have contributed to open stack neutral firewall as a service and ironic for several years and also contributed the metal cube a bit. And now I'm currently the product owner of the flower bare metal I will explain later what flower is. And yeah, we will talk about the our private cloud named flower today. Okay, hello. I'm Mitsuhiro Tanino also senior software engineer at the LY

I have contributed to the open stack cinder committee and the Kubernetes storage for the several years. And my day-to-day work involves operating operating our hyper scale open stack cluster with Kubernetes cluster and developing the future to enhance private cloud stability or reliability and availability. Thank you. So here is what we will cover today. Uh first we will introduce our private cloud platform and explain what bare metal

as a service means to us. And then we will dive into the three pillars of our bare metal as a service. I think that the second topic is the most important point a most interesting point for our presentation. And the unified API C plan built entirely on Kubernetes and integrated networking and security. And finally, we will wrap up. And today's slide is available on the scale page

and maybe please visit to from the browser and you can download the PDF file. Okay? So, uh before um Flower, so like you will say, "What is LY Corporation?" So, like have you ever heard LY Corporation? Thank you. Thank you. Thank you. Okay. Okay. Okay. You You all know it. Okay, thank you. And so, LY Corporation was formed through the merger of Z Holdings, which include the

Yahoo Japan Corporation and LINE Corporation. Flower is our next-generation private cloud designed to provide unified, scalable, and reliable infrastructure for all our services. It's built on massive OpenStack on Kubernetes, providing a wide range of services from IaaS to PaaS for our developers. In fact, we keep on upgrading the master branch for OpenStack. In this presentation, unfortunately, we don't talk about this topic due to the the time.

So, but if you are interested in that's one, so please check our X presentation of Open Source Summit Japan. So, the our private cloud, Flower, spans multiple regions and availability zones. I think you already knows about like what is the availability zone and what is the region, but like let's revisit in this slide. So, our physical hierarchy has a three levels, uh racks, which is a minimum

unit and includes a mini hypervisors. You know the mini metal racks. And availability zone, which correspond to the physical buildings, and regions, which are geographically separated data centers. We run multiple regions mirror for disaster recovery. If one region goes down, operations continue from the other. Within each region, multiple AZs provide building level fault isolation. This physical topology is fundamental to how we design our software architecture. Our

failure domains and the code correspond to failure domains in the building. We define failure domain as a luck. That's okay. Let's go to the scale of the flower. Then the flower is released around the 2023. And now to give you a sense of the scale as of the June 2025, flower managed over 500 hypervisors and 9,200 virtual machines and more than 3,000 bare metal servers across two

regions and six availability zones. Today's talk focuses specifically on those 3,000 plus bare metal servers and how we manage them at scale using Kubernetes. So, why do we need bare metal as a service at all? And there are four key use cases from our company's point of view. The first one is a AI and ML. So, I think yeah, you already heard about that today or yesterday's

keynote. So, yeah, AI or ML this is a huge demand for the bare metal as a service. And the second is a legacy system migration. Some application have specific hardware requirements or compliance needs that require dedicated hardware. And third, the storage workloads. The bare metal supports flexible high-density storage configuration and direct device access. And fourth, the this is a software with CPU core-based licensing. Using bare metal

with lower core CPUs can significantly reduce license fees. Uh these workloads simply cannot run efficiently on virtual machines. Now, let's dive into the bare metal as a service itself. I'll hand over to Mitsuru. Thank you, Ishiro. Okay, so let me explain about the our three pillars of the bare metal as a services. So, first, unified API. So, we offer one API for both virtual machines and the

bare metal servers for our end users. And second, ship frame built entirely on Kubernetes. This means that all control plane components run on Kubernetes, giving us availability uh high availability across multiple uh integrated uh network and the So, network issue policies uh provided by our in-house virtual private cloud and X.509 certificate uh automatically applied to bare metal servers at the provisioning So, from now, let me explain

these three pillars one by one in more detail. So, first is uh let's start from the So, from the user's perspective, uh provisioning a VM and the provisioning a bare metal server looks identical. So, the command is same openstack server create. The difference is uh the flavor name and also the image name. Here, you can see the uh command line box. So, for a VM, you specify

a flavor like 8 VCPU 32 GB, and for a bare metal server, you specify a metal flavor like 52 PCPU and 256 GB memory. The user uh never needs to think about what's running underneath of these virtual virtual machine or physical However, but under the hood, VM and bare metal are fundamentally different. It uses a different drivers or different boot up mechanism, uh different network integration, and

different power protocol. About power control, sorry. For example, as the the boot up case, VM just boot up from the virtual disk. But for the bare metal case, it requires IPMI pixie boot and installing OS from a scratch into the disk. As for the power control case, VM just uses the hypervisor API, but the bare metal uses the IPMI. It's a very slow operation for the physical

hardware control. So, unified API means same API, same command, same API, but behind the scenes, uh totally different worlds, especially for IaaS layer. So, now let's look at the second pillar. That is the Shipyard built on entirely The entire control plane is Kubernetes native. That is shown in the left side blue box. That is here. So, we uh the Shipyard runs on Kubernetes and it includes OpenStack,

IPMI management, OS installer, and custom controllers in order to deploy the OpenStack itself. So, running on Kubernetes gives us self-healing, uh declarative management, and much AZ resilience for every management component uh for free. Instead, the deep brain shown in the right side uh, consists of the physical servers running on bare metal server management agent and the virtual private cloud agent. These are not actually on the Kubernetes,

but they are fully controlled by and orchestrated by the sheet brain Kubernetes layer. And right side, final one is the external system like asset DB and the DNS. They are integrated in via API and they are automatically updated when bare metal instance is created. So, user can access their server immediately once the provisioning is completed. Let me explain the management layer. So, the management plane itself is

highly available. So, the management Kubernetes cluster is distributed to the across all three AZs within a region. Even if one building uh, fails, the management plane continues operating and the bare metal servers in surviving AZs remain fully manageable. The same sheet brain uh, configuration is deployed in both regions, ensuring our disaster recovery policy. So, as you can see uh, on the right figure, it has two regions

and uh, each region has three AZs. So, the upper one is the region one and the second one is the region two. And in the each region, we have the three So, worker nodes for these Kubernetes clusters are distributed to the three AZs automatically. So, even though one AZ goes down, remaining AZs can keep serving the bare metal services. Okay, now. So, let's dive into the two

key components. One is IPMI management, which I explained in the previous slide uh slide. I uh IPMI is a protocol we use to control physical hardware like the power on, power off, or pixie boot, reboot, etc. The challenge is that IPMI is very slow operation because it operates the actual physical hardware. A single operation may take 30 seconds or more, and sometimes BMC are simply unresponsive due

to the issue of the uh base uh base management controller So, at our scale, hundreds of operations may run at the same time. So, our solution is an async pipeline. The API server responds immediately and never blocks. And jobs are queued in BMC gateways uh deployed per data center. Sorry. Uh sorry. Yeah. Jobs are queued in the Redis streams and uh which decouple the fast API from

the slow hardware. BMC gateways deployed per data center room execute the IPMI actual IPMI commands. And the job fetcher collects uh results asynchronously. Because the API server and the job fetcher run as a Kubernetes deployments, we get self-healing for free by these Next one is OS installer. The OS installer also follows the same Kubernetes patterns. OSI uh the OS the OS installer API continuously watches the desired

OS installation state. A the reconciler detects gaps between the task DB and the actual server state. And automatically creates missing task if it exists. On failure case, the reconciler detects the gap and retries These Kubernetes patterns abstract complex physical OS deployment into standard and resilient API codes. Operator don't need to monitor individual server state. The system automatically cover always by using the Kubernetes pattern. Okay, now let

me explain the scale scalability problem we hit. Our original architecture used Ansible to deploy bare metal server compute manager with one port per server. So, as we explained, so we have uh, more than 3,000 uh, BM servers, which means uh, it requires more than 3,000 port to deploy in the previous architecture. Now we modify and move on to the new new And in the OpenStack, the component

that manages each bare metal server is called Nova compute and it is computer manager that runs on controller plane and communicate with the physical As the uh, more than 6,000 over 6,000 plus servers scale, the scalability problem the scalability problem became clear. uh, deployment time. Ansible tried to execute the steps sequentially and has fundamental limits on parallelization. Deploying more than 6,000 plus server uh, managers server managers

took more than 2 hours in the our previous And second, it caused an inefficiency. With one compute manager port per server, we needed more than 55 plus heavy worker nodes just to run the these ports. And third, operation pain. Every upgrade required a full portal restart, which disrupted our bare metal services and it affected the actual end users. by moving to the Kubernetes custom controllers pattern, all

three problems all three problems were solved. The controller watches the changes and reconciles them in parallel automatically and continuously. Deployment now finishes in a few minutes. And also, 10 plus workers are enough and the self-healing feature of controller significantly reduces our operation cost. So, this is our current new architecture bare metal server bare metal compute manager deployment pattern. So, the solution was very to model every every

bare metal server compute manager as a Kubernetes custom resource. The flow is straightforward. So, first, our operator commits server configuration to Git. the approval manager PR and it triggers the GitHub actions, which automatically generates and applies the custom resource manifest to the Kubernetes. And third, the custom controller watches the new or updated custom resources. the controller performs the deployment for each server in parallel. This includes the

This includes creating Kubernetes resources and also configuring configuring in the OpenStack setting together. So, now let's explain more detail about the our custom resource. So, the custom resource is defined per rack. It means the physical rack in the data center because the physical rack is failure domain we defined. If a rack level hardware failure occurs, we want to that failure boundary to be reflected by the directory

in our Kubernetes resource model. Each rack custom resource contains multiple server manager entries. For each server manager, we declare three things. The host name, which identifies the physical machine, the OpenStack flavor, which describes the hardware specification such as CPU uh, count and the memory. And the OpenStack project, which defines the fit tenant this server belong to. This product structure maps directly to OpenStack host aggregate. Uh, so

the logical grouping in OpenStack always mirrors the physical grouping in the data All of this configuration lives in Git. There is no manual registration process. And when a new server is added, an operator simply adds the uh, entry to the YAML file and opens up a request. Everything else is automated. This gives uh, us a single source of truth and eliminates configuration drift between Git and the

actual running And next, let me explain what the reconciliation loop actually does. For each rack, the controller creates one config map per server. This config map contains the computer manager's configuration for for that specific server. The controller also creates one deployment per rack. This deployment contains multiple computer manager containers, one per server in that rack. Grouping by rack uh means that if a rack fails, only the

deployment for that rack is affected. By using Kubernetes custom controllers pattern, we gained uh three things. First is parallelization. Multiple reconciliation loops run concurrently, so all racks are processed And second, incremental deployment. Only changed resources uh are reconciled. Adding one server does not trigger a full redeployment compared to the previous architecture. And the third, reduce worker footprint. Because multiple computer manager containers share one deployment per rack,

we need less worker nodes than before. So, the previous uh previous architecture, so we needed some around the 55 worker nodes, but in the new architecture, we only need 10 worker uh around the worker nodes to manage the more than 6,000 uh bare metal servers. And these three improvements together are what reduced deployment time from the hours, actually the two hours, to the 10 minutes or so.

Okay, now let's move to the third pillar, integrated network and security. And the subsection, I'd like to show you four major integrations with bare metal Uh first one, ACL settings between Kubernetes clusters. And second is the same as the ACL settings for bare metal server itself. And third is the long user script inside in the bare metal server when booting. And the fourth is the Ascent integration.

Uh of course, this is in booting the bare metal server. So, first of all, in the two And I'll show you ACL settings between As you can see in right figure, our sibling components depends on multiple Kubernetes cluster, and they have dedicated VPCs. Uh our in-house VPCs. Um we should introduce that the VPCs are in-house and automatically applies ACL policies between pods which locate on different Decoupled

architecture ensure the strict security boundaries between logical control and physical execution. The second point is the ACL settings for So, in order to apply ACL policies into uh we we had to develop uh develop our in-house VPC agent, and uh need to install in the bare metal And VPC agent uh ACL policy controller manager on Kubernetes. Uh we are including our in-house VPC agent into OS image.

The VPC agent automatically installed when booting the bare metal server. So, when the booting, uh VPC agent refers ACL policy and apply the policies as Linux NF tables. Then, the VPC agent keep watching the desired status and insert ACL rule into NFtables. So, it's kind of like catching up the desired status the matches the Kubernetes policy, I think. And here is the example of the CLD. Also,

for example, uh let me pick up one rule with a a permit from X to Y. This one. Then we can define any like action. Permit or the deny and priority, protocol, and source IP address range, source source ports. So, it means like uh from this IP address can access to the other this uh destination uh 443, so like HTTPS. Allow HTTPS access from here to destination.

That's it. And the third point and fourth point uh run user script and authentication. So, end user tend to prepare some customized user script and run into an instance when booting. For example, some creating some directories or changing the permission, something like that. And OpenStack already supports this feature by calling metadata API. So, OpenStack Nova provides metadata API and from the like look at the left figure.

So, like virtual machine case. So, virtual machine somehow trying to access to the metadata API and getting the user data. cloud-init. This is a very standard uh basic component to run some like automation. And like first, the VM get the user data from the metadata API and then run the script. and the this is neutral. So, sorry, this is a like not a Kubernetes, but like this

is OpenStack. So, the Neutron components will proxy to the metadata API access from virtual machine by accessing this 169.254.169.2254. This is a standardized like access how to access to the metadata API. Yeah. So, we already have the good reference architecture in the virtual machine side. So, we just made uh our PM management agent, and which include an API proxy inner and API proxy outer. So, you can

see this one. And we use the Nginx to proxy uh same same architecture like the receiving this one and the waiting for the Unix domain socket, and finally access to the And finally, uh this is Auth and So, by uh bare metal instance can get X4 uh X509 certificate when booting. Let me explain the weird abstracted figure on right. So, first, the PM PM management agent called

IID API to issue an IID. So, IID as in instance identity document. And second, Auth and uh CA calls Auth and to get a certificate. And finally, uh Auth and provisions the X uh 509 certificate to the bare metal server. So, yeah, today this is a wrap up. Uh we just introduced our bare metal as a service uh in the private cloud product. And there some like

in integration with a VPC and user script and Auth and So, yeah, uh that's all for the our presentation. Thanks for listening. And the there is a 3 minutes left, so but if you have any questions or like please come to the first mic and like ask some question on Hello. Hello. Two quick questions. So in the first one we talked about the versus what you had

originally. How did you come across the I guess you would have to establish like a service level objective for how quickly you could provision? Cuz in the original model it looked like you were doing a full fan out like if you had 5,000 bare metal requests you would spin up 500 Ansible runners. And in the new model you have it seems like a like a batch dispatching

job where you have something that iterates over the work and executes it. So how did you guys come across um deciding what is right for your organization for that reconciliation service objective? So that's that's the first question. So I'll pause. You're talking about like whole No, from a a bare metal provisioning perspective. Mhm. It looked like your the original problem you were solving is that if you

had like a thousand provisioning requests you would spin up a thousand Ansible jobs. And so your the resources would scale horizontally to accommodate that. But in the new model you're doing using a reconciliation loop. And so I'm curious how you shifted from a let's do it all now to a let's do it eventually model for your business. Okay, so actually is that deployment procedure change is not

creating a PM server itself. The manager computer improvement part. Okay. So initially so that we use the Ansible under to deploy the computer manager for the more than the 6,000. And it take it took the two hours because the Ansible is because they can run step by step and it does uh we try to use the parallelization, but it it is that it didn't Okay. like that

available. So, simply we changed the deployment procedure, not a PM server creation procedure. Computer manager deployment procedure to use the custom resource custom controller. Okay. Are you clear for our situation? You know, makes sense. Okay, thank you very much. Sorry, I'll ask my second question. Okay, okay. Hey, hello. Hello. I wanted to ask if any infrastructure as code tools are used it like Open Toffu or Terra

Grant. And what why not use it? Sorry, um Can you speak a little bit louder? Nice. Why you don't use any infrastructure as code tools like Open Toffu or Terra Grant to provision clusters? Provision clusters? Open Toffu or Like Terra Grant or Terra Form. Uh yeah, that's The simple answer is like the Yeah, this is like brand new private cloud. So, like we have a like old

private cloud and we already have a like OS provisioning tool and IPAM management. So, and we have to reuse the product. So, yeah, honestly, I I will I do like to use the ironic or something like a red fish or something like a brand new architecture, but like yeah, due to the company's reason, so like we already have. So, please reuse it. That's why we we have

to integrate it. So, my question was are you actually using ironic or in your deployment process? Good question. So, like I wanted to use ironic, but like currently, yeah, we are developing the in-house Nova bare metal driver, Nova bare metal driver. I we created and like managing it. Okay, but why aren't you using ironic? Is it for the same reasons as before as Yes, yes, the same

reason. We have the in-house OS installer mechanism or we have the IBM management mechanism which already installed in the data center. So, it is hard to replace all that mechanism into the new. Of course, I raise the hand. So, hey, let's use ironic. But like, okay, you have Oh, you already have. So, okay, that that's why. >> Yeah, because we are we have a similar mechanism to

yours. We're like 10 times smaller than you. And we're looking at moving towards ironic because it would be easier for us just to do it from the same uh API from OpenStack and ironic. >> Yeah, yeah, yeah, yeah, yeah. Okay, thank you. Yeah, thank you. Thank you. Hello. Thanks for the presentation. >> Um I have a similar setup. I'm using um OpenStack. And as the previous question,

I'm I'm using ironic. Oh. Um Cool. But my question is pointing for the auto scaling part. So, I'm I'm setting up the the cluster using cluster Mhm. And I have the auto scaling set. So, the auto scaling applies for VMs, but also for bare metal. So, Kubernetes directly on bare metal. Mhm. Uh my question is goes to the auto scaling. So, auto scaling at the moment for

a VM takes around 30 seconds to spin up. But if it's a bare metal, it's 10 minutes. So, auto scaling if the load comes will request some bare metal to come. So, they will go up. But it takes 10 minutes to go up. So, if the scale down comes next week after 1 minute, I'll lose that machine and comes up again. So, another 10 minutes away. So,

how do you control the flow in auto scaling for the bare metal in order to preserve the 10 minutes scaling and not destroy straight away if the if the peak goes down again. Simply, yeah, that's of course you know that the bare metal installation phase takes 10 minutes or 20 minutes because that yeah, we need to wait that like BIOS booting or something. So, that it takes

a time. But mhm yeah, simple answer. So, currently yeah, we don't support any like auto scale feature for bare metal node itself, bare metal provisioning itself. So maybe you're talking the bare metal instance creation like the auto scale scale out, right? >> Because in Kubernetes usually we use KEDA. So KEDA is able to see the load, we check Prometheus metric and we create the the rate so

we can make it smooth. But on bare metal, on Cluster API, I'm not sure if it's possible to use something like KEDA but wait for Yeah, so maybe if you're talking about like a worker node work uh scale for the bare metal node, so I think that like Metal Cube is like the one of the CNCF project and we just managing the bare metal provisioning with the

Cluster API. So maybe tomorrow Dimitri in the Red Hat will make a presentation about Metal Cube session. So maybe you can go there and maybe you can understand more like clearly about like scaling up point of view, I think. So So yeah, we are now like bare metal provisioning in by using OpenStack Nova side. So yeah, Okay, thank you. Yep. Hey guys, thanks for the talk. Nice

to see OpenStack still around. But the question is about like the networking layer. Bare metal provisioning is also a networking operation. Yeah, during the inspection I have one network, during the provisioning I have the same network and then I have to plug the bare metal into the tenant network. So how do you manage that? How do you manage the switches? I imagine some kind of an L2

plugin but like what's your network situation from the switches? Uh Neutron so yeah, yeah, you know that I mean network flip, right? So yeah, Neutron yeah, previously that network switching like the management network to the business network like when booting. So yeah, in our case uh we are separating the like IPMI network and the business network and they we are using that BGP VGP peering. So, like

the no need to split any the VLAN configuration or something. We don't have care about it. So, like simply you have separated the management layer and business layer and like using the VGP. So, like the when booting the uh bare metal server, then like the VGP peering is established and then automatically connection uh connected. >> Ah, I see. I see. That's pretty cool. And that way you

can also do like hybrid setups where I have virtual machines and bare metal in the same network also, yeah? Mhm. Yeah. All right. Thank you. I think that's almost time. Uh one more one more, okay? Hi. I have a simple question. How did you transition from the old architecture to the new one? How much time and resource did it take? And uh what challenges did you face

during these changes? Uh so, that you are talking about the deployment architecture change, right? >> Yeah. Yeah, so that uh so So, that in our previous cloud, so that we have the more than 6,000 bare metal servers, and always we need to deploy uh more than 2 hours by using the Ansible. And uh yeah, it makes much cost for our operation. So, we decided we decided to

change our operation by using the custom controller. And now the one deployment only takes uh less than 10 minutes. And as I shared, so the number of the worker node become more than 55 to another 10. So, the how can I say? Maybe the 80% of the server resource was decreased by uh our recent uh update.