Cloud Native Theater | KubeVirt Summit: Achieving 10× Faster VM Migration to Kube... Ryosuke Tatsumi

This talk covers VM migration strategies from legacy platforms to Kubevirt, focusing on the increasing importance of migration throughput as enterprises rethink their virtualization approaches. The speaker highlights challenges related to long migration durations and the need for maintaining dual infrastructures during the process, which can drive up costs and complexity. To address these issues, the talk introduces a storage offload approach using XCOPY, which offloads data transfer operations to the storage array, bypassing host-based data transfer bottlenecks. The speaker details how this method improves performance, achieving up to ten times faster migration times by minimizing CPU usage and network traffic. The session concludes with insights into the broader applicability of this solution across multiple migration scenarios and its contribution to the open-source community.

Hello. Uh, good afternoon. so my name is Ryosuke Tatsumi. I'm a chief researcher at Hitachi America. I have been working on creating storage plugin solution for Kubernetes. And today, I'd like to talk about VM migration from legacy platform to Kubevirt. In recent years, uh, more and more enterprise are re-evaluating their virtualization platforms. Especially with changes in the VMware ecosystem, many organizations are actively considering migration strategies. At

the same time, they are not just looking for a replacement. So, they are also thinking about uh, application modernization. So, that is why platform like Kubevirt uh, gaining significant attention. So, they provide a path to run VM virtual machine today, while gradually transitioning to cloud-native architecture. However, one major challenge remains. So, how do we migrate large-scale VM environment effectively and at scale? Today, I will talk about

how storage offload can solve this problem. So, this is today's agenda. First, I will start with uh, background and why VM migration throughput is becoming more important today. Then, I will explain the key bottleneck in VM migration, especially from data transfer perspective. Next, I will introduce the storage offload approach using XCOPY. So, which changes how data is moved. However, there are technical challenges when applying this in

Kubernetes environment. So, after that, I will explain our implementation architecture, including how we overcome the challenge with storage operation integration. Then, I will share performance result to show the actual impact. And finally, I will touch on our contribution to the open-source community. So, let's start with the background. As you know, the the Kubevirt is a powerful platform to manage virtual machine on Kubernetes. However, when the organization

try to migrate VMs from the legacy environment, migration throughput becomes a serious bottleneck. So, this is especially critical in large enterprise environment, where they need to migrate tens of thousands of VMs and petabytes of data. So, there are two major concerns. The first one is long migration duration. According to Gartner, uh, large-scale VM migration project can take 18 months or even longer. And this is not just

a report. So, I have I have also heard similar stories from people at this KubeCon. So, the where the customer uh, spend more than a year completing their migration project. The second concern is the maintenance window. The migration operation often requires VM downtime. So, they must be completed within the limited maintenance windows. In reality, this becomes a major constraint. For example, I spoke with another person who

said they could not migrate a single 30 TB VM within their maintenance So, even though Kubevirt is powerful, the but migration itself becomes a bottleneck for adapting the new platform. So, as I mentioned, large-scale VM migration takes a longer time. And as data size increases, migration time increases as well. So, from days to months and even years. But the problem is not just about the time. So,

during this uh, long migration period, both source and the target environment must be maintained in parallel. In other words, organizations need to operate duplicate infrastructure. This directory increases license cost, hardware cost, and operational cost. >> [snorts] >> So, but there are also other impact. So, resources are fragmented between older and new environments for a long which reduces overall efficiency. And operations becomes more complex because teams must

manage uh, two different platforms simultaneously. And the longer migration takes, the higher uh, the risk of failure, the inconsistencies, or rollback issues. So, more importantly, it delays application modernization and business transformation. So, the migration throughput is not just a technical metric. It directly impact cost, risk, and speed of business So, let me briefly introduce the Forklift. The Forklift is an open-source tool that orchestrate the VM migration

from VMware It managed the entire migration workflow, including the creating target target storage, and copying the VM disk, and converting OS image to Kubevirt, and creating the target VMs. Among these steps, the VM disk copy is the most data-intensive operation. Especially, when migrating large-scale environment, the disk data copy steps uh, this data copy steps dominant the uh, dominant the overall migration time. So, Forklift makes migration easier

and more automated, but the migration time still relies on the how data is actually transferred. So, the key question is the where and how is the data copy performed? So, now let's look at the how data is actually transferred uh, in a typical Forklift uh, migration. All VM disk data is copied through our conversion pod, that is a part of the Forklift, uh, which means the data

path is uh, the host-based. The ESXi host uh, read the data from the storage array, and then it over the network to the uh, the net uh, to the Kubevirt. Then, the Kubevirt uh, write it back to the storage again. So, even if the storage and target learns are in the same storage array, so data is not copied within the storage. It always goes through the host.

This uh, introduce the multiple bottlenecks. The first CPU usage on the host, and the second network bandwidth between the ESXi and the Kubevirt. So, which becomes our shared and a limited resources. And the third, the memory and IO overhead in the both node. So, the problem is not only the data size. The real issue is that the data path is inefficient. So, here is uh, our solution

idea. The idea is actually very simple. So, we change the data path. Instead of copying the data through the host, uh, we move the data path into the storage array itself. So, on the left side, data flows uh, through the host, consuming the CPU and memory and the network. And but the on the right side, though the data is copied directly within the So, this means that

no CPU and network and memory bottleneck. So, we eliminate the bottleneck by changing uh, where the data moves. So, the how to achieve this? Uh, we leverage an existing technology called uh, XCOPY. XCOPY is a very famous function in VMware. So, the XCOPY is a standard SCSI command that offloads the data copy operations to the storage array. In other words, the data is copied inside the storage

without going through the host. In VMware, so VMDK clone operation automatically triggers Xcopy. So, this is not a new mechanism. It is already widely used well optimized and proven in production environments. And many enterprise storage systems already support Xcopy. So, technically, the capability to eliminate the bottleneck already exists But there is a technical challenge. So, the question is, so why can't we use the Xcopy in this

migration workflow? So, the answer to execute the Xcopy or even trigger a VMDK clone, ESXi must have access to both the source and the target storage. In other words, the both LUNs must be visible to ESXi. in this migration case, the target LUN is created and managed by Kubernetes. So, and Kubernetes does not control the storage visibility from ESXi. To make that target LUN visible to ESXi,

we need to perform storage level operations. For example, the LUN mapping or unmapping on the storage array. And these operations are very vendor specific. They require the direct storage API calls to each storage system. So, as a result, Forklift cannot trigger Xcopy itself. This is a gap among the Kubernetes, ESXi, and storage system. And this is exactly what we need to solve. So, to bridge this gap,

we introduce a new component into Forklift. It is called Xcopy volume populator. This component has two main parts. So, one is the remote Xcopy trigger, and the other is a storage offload plugin. The Xcopy trigger is responsible for requesting the clone operation to ESXi. And the storage offload plugin is responsible for handling storage specific operation. So, let me walk walk you through the flow. The first, Forklift

creates a target LUN. Then the storage offload plugin maps that LUN to ESXi using the storage array API. These steps make the target LUN visible After that, the copy trigger the request a VMDK clone, and the ESXi issue the Xcopy command to the storage array, and finally, the data copied entirely within the storage. So, we control the storage visibility from Forklift through this plugin But this cannot

be implemented in the generic way because the storage operations are different across storage vendors and even the across the storage models. So, it each storage system has its own API, its own behavior, and its own constraint. So, that's why we designed a pluggable architecture. So, this allows each vendor to implement their own logic while integrating into a common workflow. So, now let me show you the how

we integrate this into the existing Forklift workflow. So, on the top, this is the original workflow. And the VM disk copied and the performed inside the conversion pod using the host-based data transfer. On the bottom side, the this is our updated workflow. We added a new component Xcopy volume populator, and it performed the two things. The first, LUN mapping, and the second, it triggered Xcopy. And we

replaced only the VM disk copy part from the original one. Instead of the copying data through the host, now we offload it to the storage But importantly, we did not change the pipeline. All other steps remains almost the same. So, the target PVC creation, OS image conversion, and creation. This means we can't we can introduce storage offload without disrupting the existing migration It is minimal change, but

with a significant impact on performance. So, I will show you the performance results. we achieved the 10 times faster migration, basically. So, let's start with a VM migration example. So, using about 1 TB of the data, the traditional host-based copy take more than 3 hours. But with the storage offload, it completed in just 10 minutes. So, that is more than the 20 times faster. It is the

best scenario. So, now when we scale this up for eight VMs and a large data set and two ESXi nodes, so that gives the host-based migration more resources and parallelization. >> The migration time was reduced nearly 5 hours to about 50 minutes. So, that is still around the six time And with the high-end storage system, we expected even better performance up to around the 10 times faster

at scale. So, this is not only about the speed. Because we removed the host from the data path, we significantly reduce CPU usage, network traffic, and system load. So, which means more predictable migration time, better scalability, and lower operational cost. So, this is not just an optimization. It fundamentally changed the how migration performance scales. So, let me briefly show where this approach can be applied. multiple migration

types. So, therefore, the migration type, the storage offload work for the both the cold migration and warm migration. So, the live migration is not supported because the Forklift itself does not support live migration from VMware yet. And for the data store type, it supports all the format VMFS, from VMFS, from vVOL, and from RDM. So, overall, this approach is broadly applicable across different migration scenarios. And let

me share some practical consideration. This is important when you actually use this approach in the production. The first, about applicability. So, the most enterprise storage systems support Xcopy, but not all of them. So, also Xcopy works only within the same storage array. So, but even though those cases, ESXi have a fallback copy mechanism. So, when the Xcopy is not available, the system >> um the system fall

back to the regular input-output copy method. So, even then, it can still be faster than the host-based copy approach because [snorts] we can avoid the unnecessary host data path. So, next, about the performance. Of course, the actual performance depend on the capability of the storage system. also, if you run too many migrations at the same time, it may impact storage IO So, from an operational perspective, you

don't need to run a large number of migration in a parallel with storage offload. So, because each migration is already fast enough with storage offload. Uh running controlled continuous manner is often more The VM migration does not scale well, uh, with data side, and the host-based data path becomes the bottleneck. By changing the data path to storage offload, we achieved, uh, 10 times faster migration. And finally,

this approach brings the storage native capability into Kubernetes and enables an open pluggable ecosystem for the Kubevirt ecosystem. So, here are the QR codes to Forklift GitHub repo and demo video provided by Hitachi Vantara. Thank you.