API is the New SSH: Forging a Zero-Trust VM Platform on Kubernetes - Evangelista Tragni, Devoteam

This talk focuses on the use of KubeVirt, an open-source project by Red Hat, which enables running virtual machines on Kubernetes. The speaker, a lead engineer for DevOps, discusses the adoption of virtualization technologies and how KubeVirt seeks to mitigate vendor lock-in by providing an open-source alternative. He explains the core components of KubeVirt, including virt-api, virt-controller, and virt-handler, and describes how virtual machines are treated as pods within Kubernetes. The session also covers the features of KubeVirt, such as device pass-through, networking considerations, and live migration capabilities. Additionally, the speaker shares practical use cases derived from real-world experiences and addresses some limitations and challenges faced when implementing KubeVirt.

Welcome everyone. Thank you for joining the session. I will I have a little bit of technical information to share okay during this presentation. So my maybe my pacing will be a little bit speedy okay because I want to give time also to ask question and this is our own topic about you know virtual machine and running how to run that. So how to create a VM platform.

So I will start soon okay. If someone is joining please feel free to sit everywhere. I see still some empty chairs here so First of all who I am I'm evangelist at Red Hat lead engineer for DevOps okay. I work currently in Luxembourg okay based but I mean not only working there but involved in this different project also in Germany in this case and today our schedule

will be we will be talking a little bit about virtual machine. Anyone of us knows that we have a big big technology leader in virtualization like right now and this is obviously VMware. I think it would be hard to find someone that will say that VMware is not a technology leader there. But anyone want to try maybe different road reduce some lock-in for some vendor or stuff

like this so another road that could be opened is a having KubeVirt okay open source project of CNCF running this into Kubernetes. So during our session we will talk about of this shift against the open source world and also core capability of KubeVirt. What are also the limitation of KubeVirt some networking choice that we have there and to be honest I I will add some details that

I discovered also yesterday announced by [clears throat] some other project so this will be interesting. So first of all we know a lot of environment have are heavily depending on hypervisor okay and technology that can virtualize our bare metal server. This is why because obviously there are a lot of application this is where we came from. We started with bare metal server and then we applied some

virtualization there developed some virtual machine developed microservices and then adopted also microservices in container. >> [snorts] >> But we still have around maybe I will say 60% overall of the workload is still going on virtual machine. A lot of Kubernetes also cluster run on virtual machine or so this is this means that virtual machine are heavily involved still in our environment and our data center. So what

I what we see to be honest is since a few years there is a increasing adoption of open source okay all around the world and different sector also because before we had some something like a tension when adopting open source. People were a little bit scared to go out from enterprise grade product to maybe some open source that are obviously supported too. But I think this is

changing okay. In fact Kubernetes is everywhere. This is obviously not only the open source one but also the vendor one from different public cloud for different vendor different also located cloud in each region each country. So what I should say is that obviously we want to we are using Kubernetes. We have team skilled over Kubernetes. So if we see possibility to move virtual machine also in Kubernetes

I think this can be helpful too because we will have the same team with highly skilled in Kubernetes that we will we will manage also the virtual machine. So another thing that I want to add is um maybe this started I think that the adoption in Kubernetes of virtual machine started with integrating database server with our stateless application. What we wanted we wanted to bridge this connection

from the Kubernetes cluster to the virtual machine that sometimes were outside the Kubernetes cluster. So from here Red Hat obviously is the main developer and maintainer of KubeVirt. Red Hat developed this concept okay of KubeVirt and integrated this in their solution okay Kubernetes obviously they have also their their own. But what what we are asking right now the biggest question will be for me this is a

racing condition for me. Virtualization for me is not a place where you say I have my data center all virtualized it's a feature. So each platform different platform can give us this feature of virtualizing helps you host what we call virtual machine as an API into Kubernetes. How does this there different concept component of KubeVirt. Obviously KubeVirt is an add-on for your Kubernetes cluster will not be

spinning up your Kubernetes cluster with KubeVirt in it. So like every add-on and product that you install always composite KubeVirt. KubeVirt has three main component when you install it in the in the cluster. First one is virt-api the one that intercept your request okay of creating virtual machine. Second one is called virt-controller the brain that watch for this object okay and manage all the life cycle of

this object this virtual machine object in your cluster. Then there is virt-handler. It's we can say that this is like the ghost of our the deployment of KubeVirt because it's the one that is demon set so each pod pod in each node one pod in each node. And this helps you manage the process of running this virtual machine with libvirt or Qemu driver that need to be

used on the node level. But in the end this is just the brain of a KubeVirt so this is just what you install when you install KubeVirt as a add-on in Kubernetes. But when you try to deploy your a virtual machine what you obtain is a new pod spawned in your cluster. This pod is called virt-launcher. The virt-launcher is a the pod that will in the early

future and this case when this virtual machine is spinning up will host the virtual machine itself. because please remember Kubernetes has different concept for networking and storage. So this pod is act like a a bridge between the virtual machine world and the actual Kubernetes world. So if we want to to give a quick conclusion to this concept I will say in the end the virtual machine in

Kubernetes are just pods okay because what you need to care mainly is the life the pod life itself because when you look at virt-launcher in this case these are the same component that we showed before but in this case explaining how they manage this life cycle. When we have virt-launcher pod and we for example kill this pod kubectl delete pod etc. etc. We are killing the virtual

machine itself too. So this means that as you can understand virtual machine are really really connected to the pod itself to the pod life cycle. Then what I want to share also is this is a YAML okay that you are seeing here. This YAML obviously you know that Kubernetes resource API resource gets deployed with YAML. This is for example a basic YAML of a virtual machine okay.

As you see there is different section. I know that this is a filled slide as some of you will not like that there is too much written. But if you think that on schedule you can download that this will be helpful when you need to reach out after one month for this concept again. So we have different section okay. The first section is where you explain which

API resource are you creating. So meta meta fact version API resource name name space etc. etc. Then there are there is the domain. Domain section is explaining basically we can say the resource for this virtual machine. So as you can see there is some CPU resource declared some memory resource declared that you can have there device declared because obviously virtual machine has virtual hardware. This is as

we are adapted to that and this we need to describe all the disk all the network interface card all the this this concept and this component that need to add to the Then some boot configuration okay. If you want to add some details on how this needs to be booted or some order of disk to describe what's your boot disk in this case. This is the section

where you apply that. And please remember that as in Kubernetes pod you know that you can declare the volume and then mount the volume. This is the same also in virtual machine you declare what is a volume because in the end the pod virt-launcher I came back to the first concept the virt-launcher pod this will mount the persistent volume claim because please remember we are in Kubernetes

so we need to mount the persistent volume claim. And then here I declare the volume for the persistent volume claim claim that needs to be created and then I declare this as a a too on the device and domain This is a simple YAML, obviously based on which operating system and you want to run, there'll be some different configuration, but this is a simple starting point that

could work for that. Also, this is a concept that I think this came from a cluster API if some some of you is familiar for that. When you cleared create some uh um infrastructure with cluster API in each provider that you want, you have the virtual machine object and the virtual machine instance object. This is two different concept. One is the declaration of the object, virtual machine

in this case, that I showed you. The virtual machine instance is the running definition of that virtual machine. So, if I delete the virtual machine instance in this case, I will delete the running execution of my my virtual machine. Okay, this is what I'm saying. Um the virtual machine instance is what is managed by the controller, the vert controller in this case, so managing the life cycle,

as I said before, of this Then, I will show now some a little bit of feature of KubeVirt, okay? And then I will come back to this feature and and have some limitation against this. First of all, obviously, we can have some device pass through, okay? This was also used for ML app, ML ops, and whatever you want to do with the AI in this case in

Kubernetes, because you need to mount in some case GPU there. So, in this case you can have with KubeVirt also inference and having these devices mounted directly under the And this is a first feature. Obviously, when you apply some uh pass through device, you are locking in your resource to that node. Please be aware that this is not a concept that you can be married easily with

Kubernetes. Kubernetes is normally stateless. I need to remove the pod and move that and reschedule that on another node. If I'm making some pass through there, what is happening that I'm binding the pod and the virtual machine to that node. Okay, so this is a little bit uh we can see that also as a constraint Also, storage and network, big concept here. Obviously, I said that the

storage we have persistent volume claim, but you need to choose which kind of storage to use there, okay? Please remember that we are using this as a in Kubernetes with the CSI, container storage interface. So, all the container storage that we have, they're installed, we can use that under the virtual machine as a Networking, big big concept here. So, first of all, virtual machine start as a

bridge, because they mount the IP of the pod that I explained before. So, we have this virtual machine living inside the pod. The pod is is obviously is IP. This IP is mounted to the virtual machine. This is the default bridge IP. If you want additional IP on that virtual machine that you normally can have in virtualization, standard virtualization, this happens with Multus. Multus is an open

source project that can help you have additional IP, so additional interface from different also CNI uh to the same pod. The same apply for virtual machine, because in the end the virtual machine is mounting the same interface of the vert launcher pod, as I said before. So, if this pod has more interface, virtual machine will have more interface. Be wise, Multus is not an easy to manage.

You need some obviously some knowledge behind there how CNI works and how this uh Kubernetes network networking works. So, um this could could lead some to some overhead in maintaining the solution after. And then, another big feature that we have that this comes also from virtualization habit and technology that we have right now, live migration. In some technology this is a uh DRS. For example, for VMware

this is called DRS. It's the feature of moving your virtual machine from one node to the other when it's running without losing any packets. So, imagine you are easy concept, pinging the machine. I move the machine to another node. I don't lose any packet in that ping. This means that the machine is always running. Imagine that not the ping, but my service is running and receiving request

even when I move this, I will not lose any request on my services. Okay, this is a feature that exist in To be honest, this is a little bit uh service in it, we can also have SSH connection. So, we can open How we open SSH easy? We have a service that expose the pod in this case, expose the 22 port of the pod and we can

directly connect to the service and this will mount the SSH service of the virtual So, it follows what is the normal um uh rule to expose this. >> Also, ecosystem integration. This is a good point when all of you, I believe and I hope that all of you is making a little bit of GitOps, okay, where it's working on. If not, you should start doing that because

it's it's already late. So, if if you are moving virtual machine there also in Kubernetes, you can use the same technology and same process that you have for pods. So, continue to use Argo CD, continue to use Flux, continue to use Tekton to make your pipeline and also use cluster API to deploy your provisioner. Imagine that you can have cluster API, it's called there is a a

provider in that called CAPK that is provider for KubeVirt. So, you are deploying infrastructure over a KubeVirt uh provider. Imagine you use KubeVirt as a provider. This is amazing, because you can have uh like some automation that is deploying VM VM on your Kubernetes This is really cool to to adopt. Also, uh some suggestion also this come here from a little bit of experience that you I

will tell you about also during my I can suggest like a few tools also that you can use at OS level hardening. So, first of all is Talos, okay? This is a really nice operating system that has no uh other reduces surface for attack. And then there is also another project that comes also from my friend uh uh Spectro in this case that is Kairos. Kairos is

a really cool project also that helps you ship this operating system image and make some rolling for upgrading. It's really cool. Helps a lot when it's talking about edge solution and edge use case. Also, please remember everything that we can use with KubeVirt that I showed up till now is everything coming from an open source perspective. So, everything that you see in this presentation, everything is open

source, so there is no vendor locking and you are free to adopt, integrate, and maybe if you need some feature code also to help project to evolve in this in this in this aspect, too. Then, regarding the feature, a big big concept I was saying was the networking, as I was saying before. So, when it comes to to Kubernetes, networking concept goes down to CNI. In my

experience, my to be honest, my comparison comes from uh uh just comparing two uh CNI in this case, Cilium and Kube-OVN. Cilium, obviously, is well known, one of the most committed project overall in the in this uh year, and uh easy to use. It leverage eBPF feature, so really fast really fast and best performance. Obviously, it's called the the vibes that it gives is a future of

networking in Kubernetes. This is a first CNI contender. What is the problem with Cilium? Okay, first of all, Cilium has a CNI concept when allocating IP until now, okay, that is not permitting to uh live migration preserving IP. I will come to this after I will be talking a little bit about limitation. So, it reduce the complexity, yes, okay, but require require a little bit of work

and knowledge in eBPF to have best Second contender, Kube-OVN. Okay, Kube-OVN is really nice project, to be honest, really deep into networking. The point is that you are bringing what you have as a physical networking into Kubernetes. That are the concept. So, when it comes to this, it's a little bit less uh likely when when it comes to DevOps engineer, but more like a like a when

it comes to network engineer. Because, obviously, Kube-OVN has a some feature like a subnet, like a static IP, VLANs, etc. So, this helps helps some uh uh keeping some concept that you have into the virtualization environment and preserving that also in Kubernetes. To be honest, both offer when it comes to software, both offer really really high performance, okay? When it comes to Cilium software based is a

really high performance, but when it comes to bare metal, Kubevirt has the nearly the same performance, so it's really cool both. This This could could change. I said that to you that yesterday while listening to this maintainer, I noticed something more that they are planning, but up until now, this is comes until I applied this uh this uh session. >> And um and this uh slide Cilium

has a bit of of a loss when live migrate is not preserving the static IP. So, what does it this mean? If I'm moving a virtual machine from one node to the other, my virtual machine is changing the IP. This will change. They already announced it this, but up until now, this was a a little bit of an issue. While using Kubevirt, this not happen. When I'm

migrating other using Kubevirt as an CNI, this virtual machine can preserve the static IP. So, I don't need to remap maybe load balancer or stuff like this. And this is a big changing. So, conclusion a little bit, sorry. Conclusion a little bit in my experience, DevOps engineer like a more Cilium to use there network engineer, network engineer like a little bit more Kubevirt because a little bit

more deep into that concept of networking. So, please use in this case, I could that also this this makes no sense anymore because if they both have the same feature, this will be okay, choose whatever you want because you are preserving the feature, choose what you are more familiar to. Before I noticed that this feature, I was saying choose Cilium if you have less workload in virtual

machine and more in pod. Use Kubevirt if your your workload workload is fully virtual machine. So, I will come to some uh everything what you are expecting. I'm uh I believe so. So, give me some use case, okay, to this. So, let's understand which virtual machine can be migrated there. So, first use case is uh uh legacy Windows VM. Yes, they can be moved, okay, easily. Why

this? Because uh normally, this application cannot be cannot be uh made made cloud native, so cannot be converted to container. So, what happens is that this will be keep kept as a virtual machine and moved into Kubevirt. First use case, really good. Second use case, Telco. Why this? Because Telco needs a lot of fast spinning virtual machine. And you know, when it's talking about about Kubernetes, the

spinning at time is the easy win when talking about Kubernetes, so I think that the scheduling is a lot higher. Then control planes. You can run ephemeral control planes for your older Kubernetes cluster in this uh environment. So, you can have different nested Kubernetes cluster there. These are just a small use case. Now, you will say, "Okay, based on what this use case this To be honest,

this use case come from come from a real experience that I had, okay, in my current time involved with also in this project still. So, this come from a company called Schwarz in this case, a Stuckit that comes a part of group Schwarz Digit. They are deeply involved in using Kubevirt and opting this as a early adopter and trying to also help this project in uh grow.

So, everything that I showed about use case can be uh like related to this real experience, let's call it like that because in the end, we want real production experience, and this comes with a with a real one. let's wrap up a little bit the concept. Is Kubevirt the real choice, the good choice? Uh let's go with some little limitations. So, I lower a little bit the

expectation. First of all, live migration, as I said, is a little bit we say in Italian fragile, okay? Uh but you know, maybe you're familiar with this kind of writing Italian that is fragile. But okay, let's go over. So, a live migration is a little bit uh fragile, so not really strong and backed So, this needs to be improved uh still, okay? So, please be aware be

aware if you come from a maybe VMware, you will have less performance, to be honest, we need to be fair. Then to have live migration because persistent volume in Kubernetes works with the read write many or read write ones. If you want to live migrate, you need to have read write many for your volume. This is a big, big point because when I'm moving this virtual machine

from one node to the other, there will be some point where both the node are accessing the storage. So, I need read write many. This is a pure technical concept beyond the storage. And then you need This is a simple configuration that you can change. You need to reduce what is called the uh reboot time in Kubernetes, so the eviction time because you know, by default, Kubernetes

can grow up until 300 seconds for eviction. And virtual machine cannot wait 300 to notice that this is gone gone down. So, simple concept, reduce this uh this settings. Windows, not every Windows can be migrated. Depends on which kind of use case you are migrating. Normally, you know, Windows has this sysprep, okay? So, sysprep is normally already a nightmare, moving that in Kubernetes could lead to a

little bit of issue there, too, to be honest. But if you are good in uh uh like a mapping this settings, maybe it's a little bit demanding, but will be fun to do that, also. As I said, this I said already. If you are making a little bit of pass-through, please be aware that your machine will be locked on one node. So, please be aware of this

because it's not stateless anymore. It's not moving somewhere else this machine anymore. So, and then last but not least, and I have another one, too. Um limitation uh number six, this is roll your own DR. Normally, when it comes to from VMware, you have different solution that working fine like HA uh and so some something like that. If you are working Kubernetes, you need to manage your

own disaster recovery. Be wise about this. So, use whatever tool you want, Kasten, Velero, GitOps for approach, and stuff like this to be sure that your application is protected because when it happens, will be a little bit of a nightmare to recover. Also, virtual machine uh sorry, dashboard. Normally, we are adapted to having some dashboard. So, be aware that there are tons of dashboard for this. Rancher

is one, OpenShift has a dashboard, Kubevirt has Kubevirt Manager, open source dashboard for seeing this machine, too. So, this is not an issue. Could be a limitation when you install only Kubevirt, but it's not an issue because you need to install additional concept. virtual machine normally are really bounded and binded by uh what is called IP, okay? In this case, uh more more about MACs, but pods

care more about IP. So, be aware that we are changing the layer, migrating from layer two to layer three. So, be aware if you have some license binded to MAC MAC address, this is an issue in this case because you cannot do that because MAC address means nothing in Kubernetes in this This can only change when you use pass-through there for network And then, obviously, security-wise, you

need to work a lot on that because you need to make some network policy, identity, Rback solution, and stuff like this. So, be aware that this can be a little bit more demanding, also. So, is it production ready? Depends on which kind of uh uh application you are trying to move there. I wrote down some example here that I could think that they can be like good

choice or bad choice. When it comes, for example, for multicast heavy application, mm let's wait a little bit. I I will say this frankly, you cannot remove completely, I think, uh virtualization just now. You need to work on that, but there will be time, but it's still growing the project, so believe in it. Last thing I wrote my notes that I believe could be reasonable is at

least alpha half a year and a half, okay? So, at least 18 month, okay, to make this. And what I believe will be the future, it's not about running virtual machine, running container, running serverless, it's about running the application. Whatever fit best the application will be the best choice there. do you If you have any question regarding what I showed, I know that we have still two,

three minutes left. So, there is a microphone here, and you can bring your your microphone and ask the question. In the meantime, you find two QR code. One is to rate my session. The other is to grab the slide that I showed from GitHub, and thank you all because I really appreciate your attendance. >> [applause] >> Last things last, I will be around to ask if you

need any additional question, Hello. Hello. Yes. >> Can you hear me? Yeah, absolutely. >> Great. Thank you for a great presentation. So, I wanted just to say if you had met any problems running virtual machines inside of Kubernetes now because it's like nested virtualization. So, have you met some kind of problems with where we can learn from you when we would like to start running virtual machines

inside of Kubernetes? Thanks. >> Okay. This is This is a good point, but be wise, it's not always nested because this can be like Kubernetes running bare metal and you're running virtual machine into a bare metal. So, it's kind of first level virtualization, still not nested. But uh as I said, uh you need to be aware that performance-wise, if you have a lot of requests when writing

or reading, uh you need to be aware that this could like more latency, let's say like that. So, but no, normally you don't have any issue moving this virtual machine into Kubernetes. Thank you. >> you. Thank you. Yeah, I have also a question. You you mentioned that Can you Can you raise your voice, please? Uh yeah, you mentioned that we can live migrate virtual machine from one

node to another without losing the IP address, but when it's running on the pod network, it will use the headless service. And when during my test when I tried when I switch from one to another node, it will change the IP address because the node is the pod that is created get another IP address and then uh starts to If I understood correctly, you are asking me

that when I live migrate, the IP is changing. Okay, but you have load balancer in front of it, right? Yeah. Okay, this is true, but normally having a load balancer there, it No, it's not always the solution because maybe you want to arrive directly to the virtual machine to reduce also latency there, okay? And also for a configuration prospective, if I look at the application inside, if

some application is reading its IP address, okay? And when booting and I live migrated, this IP is changing, this is an issue. Currently, it's the IP still changing, but I know that for sure that Kubevirt has this not Cilium is implementing this also. It's equivalent in this case. And Calico also is implementing in the next release this static IP. I I meant really on the specific use

case when we use the pod network, it used the the like the virtual machine instance inside the pod will always have the same IP address. It's like 2. I don't know point two. Please, I I I did not I did not hear you so well, please. Like the the virtual machine that's running inside the pod That's correct, but the one of the pod that is advertised outside

so that we can contact the the virtual machine is changing when we live migrate. To be honest, uh the IP that is in the virtual machine, it's true that it will be kept, but the output IP is changing, but what we care about is more the In this case, if you don't use it services, it's more virtual machine side for application prospective in this case inside it.

But to be honest, when moving this from one node to another, it's changing also that IP of the virtual machine. In fact, this is the main issue in the in this case. Okay, so you If you want, we can talk later more deeply. Feel free to wait, Excellent session. Thank you very much. Thank you. Thank you. Really appreciate I mean you got what Kate over there, one

control plane to rule everything concept. >> What would you say are the biggest challenges to reaching this goal? To what challenge, please? Uh which are the biggest challenges to reaching the one control plane to rule all? Ah, okay. Uh this is a big big gap. I think that uh it's from a a management prospective, not from come from a technical prospective this because you know there are

some deals that come from company and company that you want to keep the deal to keep your infrastructure as it is also and because maybe you need to have this partnership also or this contract in place. So, I think that in this case, a lot can be like a management problem to migrate to a unified platform, okay? That rule everything. it's more like a a skill issue,

a people issue rather than technical. No, I don't think it's a skill issue, but it's more like a I'm not closing that because I'm accustomed to that technology and I'm scared to change also for some critical application. Yeah, yesterday I heard a cool phrase saying, "What's your worst database? I will start from that, okay? Migrating." Because if I migrate the the bad one, the worst one, I'm

easily migrating also the the easy the easy one. Okay, you you got what I mean? It's a more a scared, okay? Scared to change because I I'm good with this partnership still, okay? I'm I'm not having any issue, so why should I go out apart from we know Broadcom political about licenses stuff like this, okay? A lot [snorts] of people are trying to go out from there.

That's fine. I I'm For me, it's hard to go completely out still. This is my pure opinion. Okay. Thank you very much. >> Thank you. Thank you. Really appreciate Also, uh thank you for a good talk. Um Thank you. been doing a bit of work already with Kubevirt. I'm going to put like this. Okay, thank you. Um and also meet the problem with you mentioned the Open

vSwitch as the base. Okay. So, I have done the initial look at both the Cube OVN that you mentioned and then OVN Kubernetes. Okay. I wanted to hear if you have done some look across those two because it's like you're having the Open vSwitch selection is a bit hard when there's two so similar products even from CNCF. >> Okay, to be honest in this regard when it's

still come to the network, what I suggest to this this things about the networking that from between Cube OVN and the Cilium, what I suggest that we still follow our the project that growing because to be honest, I never tried the Calico for example in this case to to have a network in because I was never finding some hint that Calico could be a CNI with this

feature too. But yesterday, I I heard from Calico directly that they in the next release, there will be this feature. They are supporting live migration in Kubernetes without with preserving IP. So, as you can understand, it's maybe Calico is the in between between the complexity of Cube OVN sometimes and the feature that you want to achieve. Uh we are using a bit of Calico right now. The

reason I didn't mention that because then we can moving out to yet another vendor where we're trying to Okay. Have a bit more of a trusted one that we like the Cilium or one of the OVN's. Okay. Just the OVN seems so similar in some regards. Okay, so you are moving out from Calico now. Yeah, yeah. If I understood correctly and Kubevirt looks similar to >> Yes.

I understand. In this case, if you want to find some easily a CNI to manage, the only solution will be Cilium. But for this feature of preserving IP, you need to go for the enterprise one, Isovalent, because they will not publish that on the open source still. As for now, this is what Because for now, they are announcing this feature. They announced it yesterday also. This feature

of a bridge Cilium bridge network, something like this that is going as a use case to go out from VMware. That's what they they published yesterday. So, I think that Cilium could be your choice, but you need to talk with them about this feature if they give you early access, okay? Thank you so much. Really appreciate it. Uh hi. Hi. Can you speak couple of words about

management overcommit memory and CPU in Kubevirt? Okay. Uh okay, I had a big a big slide on this, but I felt that it was not fair to show this this slide because to be honest, um the overhead that Kubevirt is giving is a little bit higher compared to other virtualization still, okay? So, yes, Kubevirt is consuming a little bit of resource, okay? But you can understand it's

a application running into another platform, okay? But when you look at this implementation in bare metal one, you don't care really about how much you are if the overhead is a Yeah, sorry. Yeah, sorry. Uh we Can we move next to the maybe outside in the corridor