Cloud Native Theater | KubeVirt Summit: Bridging Islands: EVPN Overlays for Multi-C... Miguel Duarte

This talk focuses on the implementation of EVPN (Ethernet Virtual Private Network) overlays for managing multicluster workloads in Kubernetes using KubeVirt. The speaker, Miguel from Red Hat, discusses the challenges of maintaining Layer 2 adjacency for legacy applications and the need for scalability and resiliency in multicluster environments. He describes how EVPN can effectively stretch Layer 2 networks across multiple clusters, facilitate workload mobility, and reduce unnecessary network traffic. The session highlights how EVPN utilizes BGP for advertising routing information, including type two and type five routes, to support seamless communication between workloads. A demo illustrates the live migration of virtual machines between clusters while maintaining connectivity.

Uh hello everyone. My name is uh Miguel. I work for Red Hat specifically in the networking team for well downstream distribution of cubevert and well here to present to you about like uh EVPN overlays for a multicluster cube. So our agenda for today we're going to begin with the motivation like what brings us here what problem are we trying to solve with this? Then explain what goals

we have, what EVPN is, what it's used for, and well, how it is implemented. Show a brief demo and finalize with some conclusions. Okay, what brings us here? First thing, some uh type of applications like legacy applications, they totally depend on being um L2 adjacent. This means how many of you have seen VMs where its ID is pretty much its IP address. I guess this is quite

common in the virtualization world and it's not amazing. That means that you if you have to import or you want to migrate that VM off to a different platform, you need to bring the carry the network address with it like both MAC IP that entire thing. That means that everybody's stuck with the same IP address. it got probably back in 2010 and that is not good. So

you're stuck with having to maintain the uh network ID and IP addresses of the VMs. Another thing a reason for you to have multicluster is clearly uh resiliency. Your application will be more resilient. Uh it's actually quite common in the cloud for you to architect something for I don't know your availability zone to go down first. So that means like the entire cluster going down and have

a way for that application to res keep continuing working in a different cluster. Finally, you might have hybrid cloud scenarios. So you could have part of your application in onrem and another part of your application running on different clouds somewhere. In case one of these clouds go down, it can still work. in part your on-prem data center goes down, it will still keep working. So multicluster is

a thing. Now what problems do we have with that? First scalability. So what you would usually do on the virtualization world is to just plum VLANs into everything like we will solve everything with a VLAN. Thing is that's not scalable. It like 4,094 VLANs might sound like a lot, but it clearly isn't. like uh after I don't know for multi-tenency you will not go far with 4,94

uh efficiency also like you have multiple uh clusters workloads like lots of workloads in both clusters remember uh in cube vert you have pods and containers in the same network this means that you will have lots of containers some VMs probably lots of VMs you will have lots of traffic like lots of bump traffic. So lots of ARBs, lots of neighbor discovery going on. We cannot rely

on uh on this if you have many interconnected clusters. And finally network coverages like if you migrate something from a cluster to another, we want it to service the the workload to service the users in that moment. It cannot just like wait for the network to converge across multiple uh disconnected uh well sites. So our solution for that is as you kind of hint uh EVVPN which

will provide to you these four things. So stretch layer 2 networks this means that you can create like a layer 2 overlay across the multiple sites and the workloads attached to it would communicate as if they were sitting like in the same uh rack and connected to the same physical switch. We also want to have routing between networks like everything does not have to be in the

same L2 but we want it we may want to but we may also want to connect to a different network. For instance, let's say you have a service running on your on your cloud service but it's on a separate network. you still want it to be you want your VM that you migrated to a different uh type of cluster to still be able to reach that existing

uh service. As I said, we may we want to have uh efficient uh network. So we need to reduce the type of bomb traffic we have. So unknown uniccast pretty much like all those ARPS NDP uh messages we want to uh keep them down and like the cherry on top is or actually our big piece of uh deliverable is we want to have seamless workload mobility. Uh

and for that we will use EVPN which stands for EthernetVPN. It's essentially a control plane protocol uh on top of BGP. So it's it relies on multi-protocol BGP and what it essentially introduces is two different um types of messages which are passed around in the BGP update which are the uh type two routes and type five routes. Each of these uh achieves a different type of use

case. For instance, type two route will um serve to implement a stretch layer to network and to implement for instance in our scenario live migration. So with this what happens is whenever a VM comes up it would it will tell I have this MAC this IP and the fabric like a bunch of interconnected routers will know this MAC this IP will be made available via this next

hop and that's all it does. A type five route on the other hand instead of advertising a MAC and IP will advertise an entire prefix. So you will know how to reach a certain network. So either an IPv6 or IPv4 prefix. This on the other hand would uh serve to enable the routing between networks and ingress without not use case. the data plane. Well, it's not EVPN.

But uh the good thing here is EVPN is a control plane PL control plane protocol which means that you can have different types of data planes. Our implementation is using VXLAN exclusively but we have seen interest in having like SRV6 for instance and you can also use uh MLS. Now the implementation this is where things uh get a little bit trickier but we have very simple design

principles here. What we want to do is we want to integrate existing solutions. We do not want to reinvent the wheel and come up with a yet another way to do something. We also want to work with multiple CNIs. We just don't want to bake this into a particular type of CNI. We wanted to allow the user to choose what CNI to use and allow them to

interconnect to whatever features they might have. So for instance, if you rely if you want to use EVPN to interconnect your clusters, but you still want to use for instance bandwidth CNI, you can chain that on top. If you want to use multi-etric policy, you use a CNI that implements that and integrate with our solution. For that we have a set of building blocks. So Qvert the

most obvious one uh well because it's what allows you to run the VMs on. Uh we have this component open P router. This is actually the router component or where everything will be is implemented and then you can have multiple CNIs. Here we are pointing at two like ovs CNI and bridge CNI but you could use more than them uh like oven kubernetes probably cube oven we

haven't checked that out but probably it could work and I'll see uh we'll see more about that in a second. So openp router this is nothing more than a router running in each of your Kubernetes nodes. It's um an F FRR pretty much like um so it's been used by and it's used in production by uh many different companies and well just a software implementation of a

router which will run in each of your nodes. it will expose networking to the Kubernetes node via a vest pair which is left dangling on the it's left uh dangling on the physical nodes and then you can just plump it in any way you want into your VMs and or containers. So this dangling verif pair you would then attach it to a bridge any type of bridge

a Linux bridge an ovs bridge and then you can plum that into your workloads using any type of CNI. All you need is a bridge. So you could for instance create a MAC VLAN on top of the bridge management interface. You can use bridge CNI. You can use ovs CNI assuming this bridge is an ovs bridge and so forth. Like the choice is yours. It depends on

your use cases and on what you're trying to achieve. Uh what else to say here? Okay, for in you would this only right now we haven't checked how this would work with the cluster default network. So uh these are all secondary uh secondary network attachments into the VMs but we are working on uh enabling this for the cluster default network network as well. Now as I said

in the in the beginning of the implementation stage uh we are we do not want to reinvent the wheel. So we are reusing pretty much everything we can and as I've said we are using F FRR which is the router implementation. We are using Linux VRFs to encapsulate each of these stretch layer 2 networks. We are using vep pairs to integrate with a Kubernetes node. and we

are using any type of bridge either a Linux bridge or a novs bridge to plum that into your workloads. Finally, we want to use a Kubernetes native approach. So, we don't want us or the users to have to manage any of this too many things going on complex errorprone. This is a recipe for shooting your shooting yourself in the foot if you have to configure this. So,

what do we want? We want a declarative API where a user could express their intent. Just throw a YAML at the cluster saying I want this stretch del 2 and the cluster will bend over backwards to make that happen. This intent that the user expresses will then generate the FRR configuration. So the router configuration, it will configure the Linux VRF and the uh VXLAN interfaces which will

be the muscle to plum your uh package through. Now nitty-gritty details what happens and how do we advertise and know where the workloads are. So workload will come Linux FDB table will understand that when it sees like it will learn the MAC to port binding. the F FRR will understand that and see it via a net link and then it will provision a type two route which

will have that MAC address. So if you look at this here essentially what this will tell you is anything on the fabric. So the fabric could is a bunch of interconnected routers that wants to re reach this MAC address over here will need to communicate via this next hop which is the IP address of the node and that is it. So now the fabric like the whole

internet maybe will know how to reach this MAC address. Same thing happens when you want there are type two routes that have MAC and IP. So same thing workload sends an ARP request or a neighbor discovery message. F FRR realizes that via net link and provisions a type two route that has the MAC address and the IP address and it will say that to reach this IP

with this MAC you have to go through this IP which is the IP of the node. Um maybe this will make more sense with a demo and I hope it does. So you have here two things before that we have the link to the to the repository and with this you can kind of uh set up the scenario that we'll be seeing. So what happens is I'm

going to run a kind cluster actually two kind clusters and interconnect them with this and in here you have a bunch of scripts that you can use to kind of reproduce the entire thing. Now this is actually what's what we're going to be seeing. So we have as I said two kind clusters they are interconnected by a different router. We have another router here. We have a

client thing that is outside in a different network and it will pretty much just try to access into our workload using its IP address. We will make this a little bit more interesting because we'll then migrate this IP address this IP address this virtual machine to a different cluster and while the client is trying to access it and we'll see what happens uh here because I am

not brave enough to try a live demo for this and I forgot to prep the images. We have a recording which I'm sorry I think I'll have to do it like this. Okay. So I will be pausing this a lot will explain a little bit the distribution of the of the console here. So in the upper right corner we have a virtual machine running in one cluster.

You have the cube config configuration right here. So you can see it's the the configuration to reach cluster A and in the right corner upper right corner we have the configuration to uh access cluster B which is our second cluster. We have a running virtual machine on cluster A. We have a running virtual machine in cluster B and in crosscluster live migration you have to provision also

like the receiver of the migration. So you have this VM1 also waiting uh for sync. So this state right here waiting for sync provision in cluster B. Uh cluster A is going to be the source cluster. Cluster B is going to be the destination cluster. Here uh in the bottom we'll see uh it's like our script. What we're doing now is ensuring that everything is as it

should. So we have as I said VM1 on cluster A, VM2 on cluster B and we have VM receiver on cluster B. This is the thing that will receive the running virtual machine on the destination Now I'm going to run a series of steps to ensure that everything is running as it should. So I'm pinging from this external client outside the Kubernetes clusters. I'm accessing the virtual

machine. You can see here the ping command went through to both VMs. So this client can reach VMs running in the two different which is a good uh diagnostics uh step. I'm showing like uh pretty much trying to uh improve your confidence that this script is not just like invented and I'm showing like the the state of the client above and I will also run a ping

command from it from the VM apparently. Oh no, I'm going to check the IP address of this VM. So I uh use the console to access it. imaginative password fedora fedora I show the IP address you can see 1927 170.13 and I will ping from the virtual machine one which runs on cluster A will ping the VM that is running on uh the other cluster you can

see it right here the IP30 it's the IP address for VM number two. Now I'm showing those type two routes I kept mentioning. this is a meaningful part right here. You have uh MAC address, IP address of VM2 right here. Wait a minute. I'm sorry. But the point is this is our VM1, right? So you have its MAC address. So very easy uh to uh keep track

of. So 02 03 04 5 6 7 with its IP address and that VM is reachable via this next hop IP. This is very important when we see the migration part because this is how we will be tracking everything. I hope this is not what well I'm going to run through it. So I'm pinging from the client. External client can reach our virtual machine. I've run the

migration now or I will do migration has started. And here in the bottom you can see what's happening. I'm monitoring three things. the migration objects on both clusters, the VMIs on both clusters. So, and the next hop or how will I reach that running virtual machine. So, you have migration scheduling on both clusters, VM running on source cluster, VM scheduling on the destination cluster and it is

reachable via this next hop which is cluster A and I'm continuously monitoring it. I am continuously pinging the moving VM the VM that is being migrated from the client that is outside the cluster. Things are working in the background. VM has been scheduled in cluster B. And if you see right here in this step, what happens is it moved like the next hop moved from cluster A

to cluster B because it is reachable via a different next hop IP address. And we see here the ping out the pings continuously went through of a VM that was migrated. This is not 100% efficient like we lost I think we lost four packets during this but that roughly equates to 400 milliseconds of loss traffic. So this is quite good and with this we can wrap up

our session and I hope the battery survives. the last slide I have. Okay. So, what we've seen, we've seen that a uh an object we can stretch a layer 2 network over multiple clusters. We have seen also that we can different networks and reach the workloads using their direct IP addresses. does granting us like direct access into a VM using its IP address. So no, you no

longer need to use uhnat or you need to like expose a workload via a service and that this stretch layer 2 network can be used to achieve crosscluster live without losing too much traffic which is something that I guess is very useful to migrate like a running virtual machine from a different cluster across well to a different site somewhere and honestly this is all I So feel

free to It stopped responding. Well, I guess I'm over. Um any questions you might have, I will be here available. Uh Andrew told me to well welcome you and thank you all for being here. Uh I think this was the first summit in person. I hope there's more. And uh remember you can also always drop by the booth we have uh at the demo not demo pavilion

in the pavilions where like one of the last ones and we will have some goodies for you there. So, thank you for your