Cloud Native Theater | Data on Kubernetes Day: KubeVirt Benchmar... Bhumitra Nagar & Dhruv Bhatnagar

This talk discusses the challenges faced by customers transitioning from VMware to Kubernetes environments and aims to address the need for a benchmarking framework to assess virtual machine performance in Kubernetes using KubeVirt. The speakers introduce War Bench, an open-source and vendor-neutral tool designed to benchmark various aspects such as VM provisioning, boot storms, live migrations, and failure recovery in Kubernetes clusters. They highlight a specific use case involving a university that aimed to scale up its infrastructure from VMware, revealing critical questions about performance, capacity, and reliability during this transition. The session includes a detailed overview of how War Bench operates, including its capabilities for running FIO benchmarks and visualizing results on a dashboard, making it easier for users to manage their Kubernetes/Virt workloads. Future enhancements to War Bench are also discussed, including easier installation and integration with Grafana.

So, today we are going to talk about a very real world use case what we encountered with our customers. Where they when the customers when they are moving from VMware to a Kubert environment or let's say if they have already moved they're already running at Kubert, they want to scale up, right? They want to scale up to thousands of VMs. So, the kind of problems that they

face, the kind of questions that they have that's the the problem that we wanted to solve and that's where we created this open source, vendor neutral, and an end-to-end benchmarking framework. We are calling it word bench. Good evening all. I'm Dhruv. I work as a senior engineer in Portworx by Pure Storage. Hey, I'm Amitra. I'm also a senior engineer at Portworx by Pure Storage, formerly known as

Pure Storage. Yeah, and together you know, our job is to make sure that Kubert and Portworx operate at scale and we fix all the and our customers don't have to face them. Right? So, having said that, let's just try to begin with a with a very real world use case. So, we were talking to a university and this university was running near about 10,000 VMs on approximately

50 ESXi's. They wanted to move away from VMware and move to a Kubert backed system. So, they had already identified Red Hat OCP as their Kubernetes platform and Portworx as their storage provider. So, the first thing the first test that they wanted to do was to run one VM and see how did it how does it go, right? So, look at look at this demo and see

how does actually one VM looks like bringing up one VM looks like in a Red Hat OCP environment. So, this is we can assume a Red Hat OpenShift environment. It's it's already set up. Kubernetes is OCP is Kubevirt is configured. Let's say we want to identify that we want to run RHEL 9, Red Hat Linux 9. What we'll do quickly is we'll go ahead and create what

namespace we want to bring up this particular VM in. And once we do that, we will say that we'll go ahead and point to the golden volume, the golden PVC, from which we want to clone and create this particular virtual machine. Once we do that and you know maybe three or four more clicks, we go ahead and configure the VM and say quick create. And as soon

as we do that, the VM comes up. As you can see, it's here in provisioning state. The VM comes up in running very quickly. Now, uh this is exactly what what the customer did, what the university did, right? They noticed that the single VM it came up in 30 seconds. But we are not talking about just one VM here, right? We're talking about more than 10,000 VMs

across 50 nodes. Uh and and this basically, you know, kind of spin up into several different new questions now. The questions were whether whether my 50 VM whether my 50 nodes, you know, they can handle my 10,000 VMs or not, right? And if I'm distributing 10,000 VMs across 50 nodes, which means near 200 VMs on a node, Let's say if that node goes down, uh how soon

can I recover my 200 VMs? Will those 200 VMs see a downtime and things like that, right? Now, this was just one customer, right? We talked to several of them and noticed that the problems uh we can actually bucket into three different categories. Uh they are performance, reliability and recovery, and the capacity or whether you can really scale up your cluster to support however your KubeVirt infra,

you know, you want you want uh to scale up your KubeVirt infra, too. So, how fast you can spin up spin up your 100 plus VMs, right? The speed is what customers always want. Uh or how quickly, if let's say we want to upgrade, we are draining a node, right? Uh uh so, how quickly can we drain a node so that my upgrades can be quick, reliable,

and smooth? Now, coming to recovery and reliability, what happens if, let's say, there's a rack failure? What happens if my entire cluster goes down, right? So, uh and, you know, what if a node crashes? Uh will my VMs recover so quickly that I still uh do not see any application downtime? I am still under my specified RTO. you know, and how many VMs actually can I run

on my particular node? This This is the most interesting question that uh everybody asks us. So, uh uh and, you know, if I'm cloning, let's say, 200 VMs from one golden image, whether whether that is scalable, whether that can actually run, uh you know, or not, right? So, uh another interesting question that that generally comes up is uh if, let's say, I want to migrate live migrate

50 VMs, right, together across a cluster. Like, is that is that even possible? Uh will it cause any storage contention? Will it cause any network contention, etc.? Right? So, the questions basically kept on expanding and there was no particular answer to these questions, right? You can basically keep on testing these questions one by one, but but there was no set tool or utility that was available that

can help the customers get these of these kind of answers at scale. This is the reason we went ahead and created War Bench. Now, if I have to describe War Bench in a sentence, War Bench can be described a benchmarking solution for Qbert, right? That's all. Now, uh what it basically does, it does these four five things what you can see on the screen. You can You

can benchmark how quickly you can clone VMs. You can benchmark how how much time it takes to live migrate. You can benchmark if let's say you are boot-storming 100 VMs or 50 VMs onto your cluster or even a particular node, right? War Bench can give you that data. Uh you can benchmark the capacity of your cluster, right? Whether whether your cluster can actually handle 50 VMs or

it can handle 500, right? You can uh War Bench can help you with that. uh War Bench can even give you uh data on if let's say you want to run any IO load inside your VMs like FIO or even every any other tool as well. Uh that can uh So, what is the throughput? What is the IOPS, etc. you are getting with 10 VMs in the

system, with 100 VMs in the system, and with 500 VMs in the system, and how the IO is behaving as as and when you are increasing the load, the number of VMs in the system. that also is one of the capabilities and features of War Bench here today. Now, it can run on any storage vendor. The back end does not matter here. Uh it can on any

Kubernetes provider, any distro. It can run on any scale. Pretty much it can run wherever KubeVirt can run. So, that's that's the uh the scope of VirtBench. Uh one thing which I really personally love about is how uh once you have created everything, once you have ran all the tests, uh you can visualize the results in a very neat dashboard. And that's something which we'll demo as

well in the next 10 minutes. And uh VirtBench can be operated using just one CLI, which is VirtBench again. So, it's very easy to run. It can be run on prem, it can be run on cloud, pretty much wherever KubeVirt runs. And the best of all, it is open source. Right? So, uh very quickly, how it works, right? What is the recipe? So, I'll just go over

a quick framework on what actually you need to do when you uh trying to install and run VirtBench. And then uh my colleague Bhumitra will go over each of those things in detail. first, as soon as we install VirtBench and try and try to run it, uh the first step is always to run the prereqs. The prereqs check whether our cluster uh is ready for VirtBench, whether

it's KubeVirt ready, uh whether all the networking and you know, all the other CSI uh infra is set up and all. quickly, uh then we will we will select what particular benchmark we want to run. Now, the benchmark could be whether we want to boot storm your cluster, whether we want to run live migration at scale, or we want to test the capacity, or we want to

just simply run FIO within the VM. Right? Once you have selected this benchmark, you will run uh uh you'll you'll configure that benchmark based on the configuration parameters, whatever you want to run. The these parameters include scale, uh parameters include what nodes you want in the picture, etc. and once you have done this configuration, you can simply run it and visualize using the dashboard. You can you

can visualize the trends, how it how it went. You can visualize uh if there was any regression from the previous runs, etc. Right, so with this, I'll now hand it over to Bhumitra who'll who'll go over everything in detail. So, the first thing that you would do like when you have a Kubevirt VM is to deploy it. So, starting with the VM provisioning test, that's one of

the capabilities from Workbench where you are essentially taking a golden template. You're cloning that golden template to multiple PVCs, and each of these PVCs which eventually attach to a VM. And once that gets attached, then your VM will boot up, OS will boot up, and that's when we can actually ping the VM to check if the OS is ready or not. So, that's when we call the

VM is actually in a running state. So, the provisioning test, we measure like the end-to-end time for a golden image to clone to like end data sources, and then for the time to VM to come up. And then once your VM set up, let's say it deployed 100 VMs. Now your VM set up, and you want to run the boot storm test. So, what we do here

is we basically power off all the VMs. And then we boot them up at the same time. Right? And then we are measuring the performance in terms of the time it takes. And uh we'll show I'll show you a quick demo on like how the boot storm test works. Like in this scenario, I have taken 10 VMs here with the range 1 to 10. And then I've

specified that I'm doing it on a single node with the scenario selected as boot storm. Right? And then I'm running the tool which runs certain scripts. And then once the VMs are powered on and they are pingable and in a ready state we can see here that we get the individual times like time taken to for the VM to reach to a running state, and when actually

the OS was actually ready and you could actually ping the VM. Right? And we get the average, max, and minimum times for each of like for the overall run here. And one of the things that this is like a you good use case for VDI. Moving on to the VM live migration, right? Like we have deployed the VMs on your node. VMs are up and running. Now,

if you want to move the VMs from one node to the other. So, that's basically live migration, right? You can do it at scale in like multiple modes. One is sequential, one is in parallel where you're migrating multiple VMs all together. And then there's the node evacuation scenario where you want to evacuate all the VMs from one node. You want to figure out like how much time

it would take for the VMs to move to the other nodes in the cluster. And with the live migration test, you can capture all the metrics per and capture them per VM and as a batch, and you can predict like the maintenance windows and how much time it would take if such a scenario was to happen. Now, you have your cluster up and running. Let's say you

deployed 10 VMs on a particular node, but you don't know how many VMs that node can sustain, right? So, with Workbench, what you can do is you can keep deploying VMs on that particular node by cloning that golden template, and you'll reach to a point where you would no longer be able to deploy any more VMs, right? It It would just say that the VM is stuck

in scheduling state. So, that's kind of like the limit testing that you can do with Workbench. And along with that, you can do certain things like um snapshots as well. So, figure out like the break breaking point for uh your cluster. And uh you can also figure out like the maximum attachments per node and like maximum VMs uh per node. So, you can do these kind of

test with Workbench. The next is the failure recovery scenario. let's say what happens when one of the node goes down in the cluster, right? You want to You want the VMs for that node to move to a different node. So, to be able to figure that out, we have that test inbuilt into workbench. Uh we have two modes here. One is with the far operator, which is

the fence agent remediation operator available on OpenShift. So, if you were to use that, and if a node goes down, then the operator kicks in and it basically moves all the VMs for that node to all the other available nodes in the cluster. And if you are not running OpenShift and you don't have a far operator or something equivalent you have or if your CSI driver or

your solution supports it, then you can use the manual failure testing and you'll be able to figure out what your recovery time would be in a scenario like this. Then you have the FIO benchmarking capability. So, basically what we are doing here is we are running FIO inside the VM with the tunable parameters such as block size, read write mix, queue depth, number of jobs, right? And

we are sort of figuring it out we are sort of aggregating all the results for the FIO tool and collecting all the metrics. And you can see the aggregated IOPS throughput and latency for all the VMs that were running the IOs. And the next is the workbench CLI, which basically powers all the tests that we are doing here. So, you pick pick workbench, pick a scenario, and

then choose your corresponding options to run the test. Once you've run all the tests, right? There there you have the collection of all the results, which we are able to show on a dashboard, right? And you'll be able to see like the version, what you run what you ran with, uh the VM count, what your configuration was, and you can compare the results across different runs and

different configurations. And you get charged for like VM creation, boost on migration, uh But before I show the demo for the dashboard, uh, I just want to talk about an interesting use case. So, one of the things that we did was to compare Portworx Enterprise with other CSI. And the test that we ran here was for uh like doing the boot storm, VM creation, live migrations, and

node failures. And we ran this test for up to 400 VMs uh with these different scenarios. And I'll show you uh how the collection of the results look like here. So, this is how the dashboard looks like, right? So, you have your version what you ran with, and then you have two different configurations of VMs. So, we ran the test with one one disk per VM, and

then we also ran another set of tests with four disks per VM. And we ran it in like four different batches. So, one test was run with 50 VMs, the others batch was 100 VMs, 200 VMs, and 400 VMs. So, you get all the aggregated results for all the VMs on this single dashboard, right? And you get it for both uh all of VM creation, boot storm,

live migration, you can choose the different tabs at the bottom to see the results for each individual run for either VM creation or live migration, and then you can also scroll down to see the results per VM what it was. Right? So, you have your aggregated results up there, and you have your individual metrics down below. And you can choose to go uh you can check the

results for like one disk versus four disks what your timing was. And if if you were to compare like two different runs, like two different solutions, then you can choose choose to do that as well. Like for example, in this case, I see that the live migration for 400 VMs was roughly under 5 minutes uh with Portworx Enterprise. And for the other CSI for the same four

disks, it took roughly under 22 minutes. This is the sort of comparison that you can do here. We also have the other dashboards for the FIO benchmarking, right? Where you can see like what the FIO configuration was when you ran the test with. Uh so, you get your IO dev, number of jobs, run time, right? Uh block size. And then you get all the aggregated results on

here uh for the read write IOPS, your bandwidth, your latency. And then below that you get all the different metrics uh for uh individual VMs which ran the FIO on. And then you can click on uh any of them to like visualize the metrics that were collected. So, Word Bench is actually coming in very handy with the customer use cases that we have. so, one of the

use cases where we put put Word Bench to use what was with uh HSBC. So, they have been uh on their So, they have been a Portworx uh customer for like with the container side. And now they have started their journey with uh virtual machines. And Word Bench is coming in handy to evaluate like how the system behaves at scale when you're running virtual machines uh with

KubeVirt. So, what's next? Like uh what are we So, what we are trying planning to do next with Word Bench is that going to make the installation and execution a bit more easier. So, uh you're going to be able to get Word Bench through pip install Word Bench. Uh and we are also going to be providing a container image. So, if you run uh with containers, you

can use that. We're building new dashboards with Grafana integration so that you can see and visualize all the results that were collected directly on Grafana dashboards. And we're also adding certain day-two operation capabilities like doing hot plug disks uh or if you want to say I want to power off 50% of the VMs on a particular node, or if I want to like start and stop IOs

on certain number of VMs. So, you're integrating those into Workbench. And along with that, we're also adding support for like different tools such as L bench tool uh where you can run like standardized IO workloads inside the VMs. And also like TPC workload support for like Windows and Linux VMs. So, all this is coming next. And this is where we are asking you to contribute to Workbench.

It's all open source. So, to get started, you can scan the QR code there. Or you can go to workbench.io. You'll get redirected to the docs. And yeah, just follow on the instructions there. You can get started in minutes. And if you have Workbench uh if you have KubeVirt in your system uh and you're running virtual machines, you feel free to try this out. Right. And we

are at there at booth 450 uh for at the Portworx booth. So, if you want to come stop by, talk to us, we'll be