Daniel Hiller: Making Running Virtual Machines in a Kubernetes Cluster a Mainstream Activity

In this talk, Daniel Hiller discusses the integration of virtual machines into Kubernetes clusters using the KubeVirt project. He explains the necessity of running virtual machines in a containerized environment, where Kubernetes excels in managing containerized applications but may not address all use cases that traditional virtual machines handle. Daniel provides a comprehensive overview of KubeVirt, including its architecture and functionalities such as live migration and snapshots, which ensure efficient resource management and operational flexibility. He also demonstrates a live demo showcasing the importing of a virtual machine image and the process of running it within a Kubernetes cluster. Throughout the session, he highlights the practical applications of KubeVirt for developers and organizations looking to leverage both containers and virtual machines effectively.

[Music] ladies and Gentlemen please welcome our next speaker Daniel Hiller presenting the topic Hubert making running virtual machines in a kubernetes cluster a mainstream activity hey hello everyone how you doing okay so um my talk will be about I guess something like a contradictory topic like um what why would you even need like virtual machines inside a kubernetes cluster probably and I'm trying to answer these questions

during this talk um also I'm going to um try to present you a live demo of some things that you can do with cubert um hopefully the demo Gods will be with me today so um I'll do a little bit of a prayer I think and I hope you can join a me in so that everything goes right okay um to my person I'm uh software engineer

at cubert um working for Red Hat I'm around five years in the project right now I've been working with databases and stuff and like um like boring Java and such things no I'm just kidding a friend of mine is sitting there um so um the last five years I've been uh maintaining the cupid C ecosystem um so like with uh my previous speaker the previous speaker talked

about like uh doing trying to do gitops on cupert and that's what we're actually doing and lots of things like ring the bell to me um there are a couple of very smart ideas I just want to pitch it because I think it should be really interesting for you probably if you're interested in devops okay um yeah you can if you want you could could look at

my uh homepage or um yeah well I forgot to update the Twitter com I have only a masteron account I don't use Twitter anymore um and my GitHub handle is De Hiller um okay what is this about again this is about like running virtual machines inside kubernetes clusters and uh why you would have the need for this um so my first uh couple of question would be

who's familiar or is using kubernetes okay okay couple of people um and who is familiar or is even using Cube word okay great so I'm hope hopefully I can convince you to use it afterwards if you have the need for that but I figure that you're here for a reason so um yeah let's try to uh help you um understand the use cases and maybe you can

come up uh in the Q&A section uh with some other use cases that I might not even have imagined um so yeah um the kubernetes is stands in contradiction with virtual machines CU kubernetes is of course like a platform where you can run your containers in and it gives you a lot of like boiler plate stuff so you don't have to handle this yourself like restarts or

whatever virtual machines on the other hand are running on a hypervisor normally on most of the time spare metal but in recent years uh there has been support for running virtual machines also on the cloud um um so um I'm just uh reading this because I'm always getting this wrong so bear with me qnet is a portable extensible open source platform for managing containerized workloads and services

that facilitates both declarative configuration automation so it tries to fix or like support you in most of the problems that are coming when you want to run like um your application on a scale and um like automatic restart and all these things virtual machines on the other hand try to solve something different they are um uh the a container for whatever software you have um but not

not like a real bare metal thing but like you have something that is emulated on the Node um where it's running so it this complete operating system that is running inside the virtual machine that is the contrast be in contrast to The Container which is uh uh doing something different and of course it's needs somehow a physical computer underneath of course but both do do that at

at some point so let's take a quick look at the diag here um so this is like um what the layers of the uh containers versus virtual machines are so you have normally an app that is containerized inside a container and is running inside its its container and is run by a container runtime on an open rating system on the underlying Hardware versus a virtual machine that

also has some apps running there most of the time this is a difference that uh that there might be several applications running on the same system at the same time um also with uh binaries and libraries and operating system and all that stuff running on a hypervisor which is then also running on an operating system and on Hardware so now let's get back to the question that

I had initially um why am I using this or why would I be needing to use this um I had my myself um a very old virtual machine for example that I had like lying around like a small like uh Windows XP if anyone knows that operating system even um and this was like lying around and I was thinking about how I could reuse that um that's

just one case I would not want to have it like on all day somewhere um but I would like want to like pull it up whenever I need it and I would and um like um give it to my kids playing those learning games on this one because it's very old and my kids are bit little bit older now so they don't need it anymore but that

was the time um this is one use case or when you have this I I figure that probably some people still have like those old machines that are running some services that they that they might be hiding in some closet and it's still running and it's still doing its job but if you are like if you are like uh needing to maintain that system or if like

someone stumbles over the cable and the power gets lost then you're then you might lose some service that might be crucial even so to get back um how can I in general um get some uh virtual machine that is completely somewhere else but uh what my containers are talking to probably um how can I get it closer to where it is used that's one of the main

ideas what we want to solve yeah let's get that I think that's an old one that you have seen already um so yeah what is CER cerd is um a platform uh basically and an extension kubernetes um and it enables you to run cluster and also it enables you to um to use the standard kubernetes objects um that uh enable it like for example to front a

virtual machine that is running inside the cluster with a service object example so um actually this is uh this is important um so this is the initial um I think the initial pit for this project so there was the need that people had virtual machines that they wanted to move inside the kubernetes cluster and actually this is exactly what what this uh project does so let's skip

that I said that um again I yeah this is also like I said so let's let's just look at the yaml file which actually is like the description of a cube virtual machine so most of the time like basic yl which you should be familiar with if you're working with CU kubernetes you should be pretty um should be pretty understandable what it's doing here so there is

also like a spec section where you have um the uh definition at this point for example is there a lighter in here I don't know you can see for example like the spec has top running fold this means that the initially when the when the virtual machine is created it's not running that's an interesting point because you might want to have this running whenever it's created then

there is the specification of the devices that are attached to the virtual machine and this is basically like um you work with other kubernetes resources you can for example like attach um like this is not shown here this is a special like a container dis which I will not go into detail um basically that container dis is just uh like an image that you can attach to

the virtual machine and that you can run then um or that is available to the virtual machine you would say um normally you could also like attach uh persistent volume claims if you know um and and anything else that you're somehow accessible through the kubernetes ecosystem that your clusters Opera ating on okay um so much for Theory so I what I'm going to show you now is

um I'm going to do a live demo where I am going to import uh virtual machine disk like a qow q file if some people probably might know that then I'm going to convert that image actually I'm not doing it the machine is doing it for me but yeah it's converted into to something that cubber can use then I'm uh uh actually creating the virtual machine with

that image um and then I'll have my virtual machine running inside the cupid cluster in the cluster um demo time not yet I just want to talk a bit about um how the demo um itself is organized so let's um somehow imagine I have two services one is an external service that is running on a virtual machine that is outside the kubernetes cluster and the other service

is something that is using that external service so that external virtual machine service and what I want to do of course is because only this other service is using the virtual machine service so I want to bring it closer into that uh kubernetes cluster and then have it like um like every other Service uh running inside that kubernetes cluster um I'm using the cubt C demo environment

which is part of the cubid project which gives me a set of cluster nodes in of a containerized kubernetes cluster that is running on my local machine uh which is I'm which I'm using for demo purposes so this is a dock rised um kubernetes cluster which um has like pre-o images so that this uh should be pretty fast um I have uh some components I have is

there is a containerized data importer that is somehow re or that is required for interacting with storage and kubernetes cubert um then rooka is a backend storage provider for local use and um normally I would also have promethus and grafana for monitoring but I must admit that I didn't get the working for that sorry so I can't show you monitoring but I think that the other things

might be pretty much um enough to see for to day um if you can uh I can I can give you pointers or you can can probably also look at the at the documentation it should be pretty easy to to get that running if you are interested to uh look that okay so let's see oh no of course my preparation has failed okay so let's see can

you by the way is that big enough can you see everything should be okay okay for yeah that's a problem with life demos right if it's not working it's not working so let's see so okay now I'm uh I'm in the step that um I'm deploying Cube bird on the local kubernetes cluster um this should give me the ability to run the virtual machine um and the

next one which I will then um do is I will import the um image but yeah it's taking a while no to myself next time be even better prepared so while we're waiting I can probably get back to um what I also want to talk about um so um I'm not I'm not I don't only want I also want to show not only the VM import but

also the live migration which you does anyone know what live migration is for virtual machines okay no hands two hands okay um so just uh just in a in a quick in a nutshell like life migration is the migration of a virtual machine from one note to the other so without switching it off like live so you are just switching the note underneath the virtual machine while

still the machine is running which is probably pretty interesting for people who are like um having to maintain nodes um that might need to get restarted um this uh comes in very handly if you're going to drain the node for example like you can configure a virtual machine automatically so that it just migrates to another node without you needing to anything um and also I'm going to

show snapshot and restore uh which is like also a virtual machine feature which most hypervisors of course like provide you can um just like from a from a created uh snapshat you can restore a virtual machine to a previous state that's also what I'm going to show it will be pretty pretty simple what I'm showing but I think it's demonstrates enough so okay let's talk about the

project um cubber is an open source project with Apache License um the latest release was in March uh the version 1.2 uh we have a release schedule that is in sync with kubernetes releases um our releases are tested on the latest three minor kubernetes releases So like um at the moment for example we are testing against 130 129 and 128 so that the next release will be

ready when kubernetes 1.30 is released so that it will be comfortable with it and working with it um as of May 2024 we have 5,100 GitHub stores 310 contributors 1,300 forks and 8,700 PRS um we are a cncf project in incubation State well working to get to the next state but there's a couple of obstacles that we need to um that we need need to overcome but

it so far it's looking pretty good that we'll reach the next stage we have contributions from obviously from redhead from IBM also from sus from Google from Nvidia arm sap apple and lots of others so it's like a highly contributed project and it's adopted by vendors and end users so just to name one of course redhead open shift has the virtualization feature which gives you like under

the hood cird with belts and whistles I would um okay so there is uh there is an Outlook to features that uh are will at some point be implemented but I can't guarantee anything so for example like the AMD people are working on secure encrypted virtualization memory encryption for virtual machines which would then also like uh work on cuer um I think there is also I'm not

that a virtual machine guy to be honest like I said I'm maintaining rather the CI system but I remember faintly that Intel has a similar feature which is also being worked on to support that um also multiarch clusters should work basically in theory I've seen the merch request merged but I'm saying merch request yeah well whatever um but it's not officially supported yet and we uh there

are people from IBM working on s39 anx support a general incomplete list of features um I'm not going to run through all this so but just so that you have seen it for examp but the notable things would for example be like being able to give to the contain or to give to the virtual machine the access of virtual of vgpus uh so that you can run

your um your graphical workloads on that one um and um yeah we have some uh like Hardware throughput or Hardware um pass through um options that are related to network um and also to USB and all these things that you um that you might need I'm pretty much um confident that there might be lots of features uh that might be supported already and if not please reach

out to the cupert echo system and uh create an issue if there is something missing that you need it is easy to install well if it works um so um in general normally you would just uh catch the latest release um by uh curl command and then deploy the cubid operator and the cbid CR and then you should wait until all cbid components are up and um

then it's up and running okay so that for my talking I hope that this condition met okay that's good oh come on oh that's interesting oh no why so what I'm seeing now is that um the CDI component itself has not been deployed which is pretty bad because like all the storage is uh backed by CDI so I would only have the option to completely redeploy the

um which I will do because I don't have any other option and then I'll um uh I'll go on right away to Q&A um while we're waiting for that and trying to recover that so wait a second I'm just going to tear down the cluster and Reen it demo okay yeah I'm not ready to thank you by the way but yeah let's let's just uh start with

Q&A if you have something for me yeah anyway we have uh at least now one question here in slid I will read it does look like the response uh to the broadcom wine Wars Hot Topic these days will it be a competitive alternative in the near future if it is there uh if it plan so I think like I said before uh we don't actually have like

plans to replace someone else uh it's not about replacing something um and especially not like be in uh uh just making someone else obsolete or something like that we are just trying to solve problems for the users and one of these problems is like that this virtual machine uh which is somewhere else needs to get moved and we are just trying to support that that case I

hope that does that do do you think that answers that question good enough I think so if you have something to compliment please raise your hand I the qu question Live question uh give me sign yeah we have here uh hello my name is vadis I would like to ask a few questions related to the C in general so first is it running like a pot on

the kubernetes and it's managed like uh in the same way actually yes I mean the virtual machine the virtual machine is running inside a p that's absolutely correct so we are using KVM qo for that so inside the inside the poort there is a qo process running the virtual machine okay so it does it mean also according to my understanding that we can run a Windows machines

if uh you have the basic setup for that one yes exactly you could also run Windows and everything else that somehow would be feasible to run on a normal virtual machines hypervisor I I'm not sure if that's too broad an answer but actually we are testing with virtual machines on Windows basis I'm asking because we had a situation where we utilize Docker for simil similar stuff not

like on the kubernetes and we need to run simultaneously windows and Linux machines and usually uh there are some issues we both of them so if this Cube V allows us to spin windows and Linux machines on kubernetes it could be very useful should work yes yeah ex but the thing is I think that of course like the license issues that you have normally with Windows um

you need to have licenses for every windows and stand that you're running but other than that I don't see any problem on that it should even support like Windows 11 with TPM um so um that that should also work and one last question also sure uh I see I saw in the presentation that you show uh that the machine was already created and you import it so

can we does it mean that we need to create this machine somewhere no no you don't have to you can also like directly create the virtual machine on the kubernetes cluster if you need that so that that's also like I think there is even uh so so part of the cubid project um has some tecton tasks for that actually for like creating uh virtual machines inside Cube

bird so that you don't have to do all the heavy lifting and I think that there might be even a tecton task that you might be interested in with which you can I think that there should at least be a demo or something somewhere that um this tecton task is actually creating a v Windows Virtual Machine and in that context is this machine able to virtualize a

GPU as well windows I I can answer that question to be honest I don't know but I would figure that it could but don't count on me on that one I think we would probably need to um uh create a GitHub issue for that so asking for that so like I said I'm I'm not oh yeah thanks um I'm not that big into virtualization itself um um

that would be something specific specific for the people who are actually like um um supporting the vgpu solutions okay thank you very much you're welcome questions are growing so still we have like time so let's go one by one okay does the cuber with has terraform open of Provider similar to kubernetes that's a good question and I'm afraid I can't answer it I don't know I think

there was a terraform support but yeah I I figure if if there is such a thing there should be a terraform repository ins under the cubid organization um that that holds it I don't think that it should be outside organization okay next one already voted one additional like like so it uh became up uh what is the estimate of performance impact uh comparing uh VMS running with

Cu uh cuber uh vs um bar hypervisor do we have such like information um the M running with CU versus bare metal plus hypervisor interesting question I don't completely get it because like we are normally like um suggesting that people are should run C Cube Bird on a bare metal kubernetes cluster to achieve maximum performance so since actually like the virtual machine runs on a hypervisor also

like it it doesn't have like no hypervisor or something it it runs like a normal virtual machine inside inside a pot um with a kbmq ammo process so um there is uh a performance project that actually measures the performance of the cube word um cubt virtual machines and I'd suggest that you look at the um cubert performance repository to get detailed metrics I'm not sure if it's

actually like compare ring um virtue machines running on standard hypervisor but at least you can get the data on that one how what what we are measuring there is like startup times um and run times under load and how many uh virtual machines you can uh you can run for example and what what the performance um um metrics are there okay next one uh already was appeared

disappeared and uh popped up again uh what does multi architecture support means is it able to run different architectures VMS in one cluster like uh x86 and arms exactly so if you have like um a mixed architecture cuetes clusters um we are basically supporting um running like uh arm workloads on arm nodes and x86 workloads on x86 node next one already two likes uh how is the

communication between VMS and other ports uh looks like via K communication uh can either be achieved by standard kubernetes um objects as I said like Services you can for example like um create a service to access a virtual machine um also there is uh the access via the console uh which you can use which I would have shown you normally um but you can also SSH into

virtual machines um directly um so that you can work on the cubid on the on the virtual machine directly and and just uh manipulate things as you as you need to do um there is also like support for VNC protocol um and also RDP is supported so you could also like do a graphical um management of the virtual machine if you need that do we have questions

live no no mics oh okay here because yeah let's spend this time uh related to this uh Services uh if I you SSH to the VM managed by kubernetes and update some kernel parameters or something which normally is ruled by the the yumland and specs how does they survive for example switching on some external file system or oh so U okay so we asking contradiction your local

setup against the definitions in cuon um so you were asking I think this is two questions but I'm let me let me try to get to the first question question at first so if you're changing like the virtual machine configuration inside the virtual machine you need to be reliant on on persistent storage of course so normally you would do that with a persistent volume claim which backs

the virtual machine storage so if you change something in there it will be persisted onto the per persistent volume claim and uh it will survive if you would then like restart the virtual machine it would have all the configuration uh um and the second one I'm not sure I completely understood could you probably for for example if I changed the the memory configuration uh the ah ah

so outside of the virtual machine you would be changing the the amount of memory that was given to the virtu machine yes so we have like support for I think I probably let me see if I can find that again um we have like um several hot plug aspects um and uh memory ballooning is one of the uh options that is uh hot plugable um I'm not

sure if it's like live um live updatable so that you can like uh on the Fly extend the memory for the virtual machine but it should be uh be at least possible to extend the memory and then uh probably restart the virtual machine I'm I'm not sure if that is the live update is supported there is features that are hot pluggable and that are live updatable but

I uh honestly I don't know whether that is one of these thank you very much you're welcome yeah and the question related to the storage from from Slider you were using uh Rock chap as the backend storage for the demo do cuber with support storage for all three types file system S3 and block storage actually cuber integrates with uh with rooka um over CDI so you can

support or choose any any Storage Shop like any live questions from the audience no hands so at least what I see the last one in slid though is the is the difference between kubernetes uh vice versa opens technology what is open VZ I have no idea sorry what is that does anyone know what openvz is okay so I'm not the only person here that's great I'm feeling

dumb okay yeah okay there's no questions and uh yeah we see or at least those who does demo yeah we know that usually it's not according to your plan happens yeah uh or still we have an opportunity to see it because we we have time yeah yeah the funny thing is now it worked of course okay okay sometimes it it works like in that way yeah it

was just wanting me to anger me I think somehow but yeah so okay so uh let's get back to the demo now that it's working um so on the right side you see uh two Cur commands that are being permanently executed bya via watch um actually it's pretty basic so um this is like um I'm uh simulating One servers that actually just call the C binary if

you know that so it puts like messages into fancy Bubbles and the other one is like the fortune service which is like um it generates a message that uh that should be entertainable somehow and by mixing these both up I just wanted to simulate that you have like the external service that is running inside the virtual machine and the um the internal service cluster internal I'm just

um doing external for virtual machine outside the cluster and internal for virtual machine inside the cluster so um the coway service is the one that's external inside the virtual machine that is currently outside the cluster so um I'm keeping these two running because I want to demonstrate that you shouldn't see any failures inside those windows while I'm doing the things that doing okay so um let's try

to put up some wait a second what's that ah okay t-o I should open the t-mo session if I okay okay so I'm first of all I'm uh doing a watch Cube cuddle get pods so that you can follow what the pods are doing and also I'm no I'm not doing that I'm just wait a second then I'm going to do a cube watch get vmis so

that you can see the virtual machine instances that are running and also interesting is the virtual machine migration objects that are going to be created at a later time when we are going to live migrate machine um and as fourth I'm not sure if I need that but I think I have the feeling that I need something else oh well so first of all like um I

have this external virtual machine running which I can s into okay I'm inside the virtual machine that is running outside the cluster currently so what I'm now showing you is that it's receiving requests from the other service from the for Fortune service and you should see like it's uh it's cing around so that you see like the requests that are going in there so let me see

I think it's hopefully it's not getting confusing to see all those windows and I hope you can follow if you have questions just just shout in um probably I'm trying to repeat those if it's okay for you um so what we also should be seeing would be um So currently we are Cube cudle Get Services we have this Fortune service that is currently running on its own

which is like uh generating The Fortunes and then calling the cay Service uh which actually like puts this into nice Bubbles and then returning that uh via HTTP response oh okay I hope I can just at least show you the migration oh no I don't want to close terminal okay so task um um wait a second l so what I'm going to do now is um I'm

going to uh in inside a task I'm going to import uh the virtual machine image from the external um what what it's actually doing is um via containerized data importer it's uh putting the data onto a PVC um and converting that to the required format for CU bird and then making that accessible um so I'm while we're doing this I'm just going to show definition of the

of the cay service that will then be internal so what you see in the upper um part of the yaml is the service that is fonting the C service and in the lower uh you are going to see the virtual machine definition which I use so what what is interesting about this is like that you for example can use node Affinity rules um if you are familiar

with kubernetes um you should uh you are you know of course that this is like a way to um to Define where the kubernetes objects are going to live oh sorry and I have to have to stop here but just one one more sentence you can see like the domain definition which uh has the hard drive that is attached to the persistent volume claim that is the

C Service vmis that I just imported sorry for my demo not working but yeah in the end okay thank you warm applauses yeah to Daniel yeah this is a small present from yeah the organizers