Multi-Network Step-by-Step: Enabling SR-IOV Support From Kubernetes... Masaharu Kanda & Lionel Jouin

This talk discusses the implementation and use of SR-IOV (Single Root I/O Virtualization) in Kubernetes, specifically focusing on the new model of the Kubernetes network driver. The speakers, Lo and Masar, explain the basics of SR-IOV and how it allows a single physical network card to present multiple logical interfaces, or virtual functions (VFs), for better resource utilization and low latency networking. They emphasize the importance of hardware-level virtualization compared to software solutions, highlighting significant use cases such as telco and HPC workloads. The talk includes a practical demonstration of a video inference workload pipeline using RDMA over the Nvidia ConnectX, illustrating SR-IOV's capabilities in real-world applications. Additionally, they introduce Kubernetes enhancements for multi-network configurations, including the Consumable Capacity and Partable Devices APIs, which help manage network bandwidth and device allocation dynamically.

Hello everyone. Um, welcome to this talk about multi network and SRV using the new model Kubernetes network driver. Um, I'm Lo. Um, I'm working on Red Hat. Uh, I'm a multi network project lead. I've been involved in uh this area for a while. I was working on network service mesh before and I've been I'm also a CNI maintainer. And here is Masar. >> Yeah, I'm Masar. I'm

from entity. I'm developing the disagregated computing infrastructure controller. So today we're talking about SRV in Kubernetes. So let's start really at the basic things the foundation. So what is SRV? So uh the device to try to represent and cover in Kubernetes are this V2 function that are provided by SRV capabilics and SRV is a PCI express standard that allows a single physical network card to expose multiple

logical function or uh interfaces called virtual function or VF. So this is not uh software virtualization it's it's a hardware level virtualization done directly by the nick. So this comes with some limitation. So the first is like um you need requirement you need the requirement or you need to to have a you need you require the hardware support itself. You also have a uh amount of for

example VF that can be configured by the next that is fixed by the hardware itself. But from the so so what's the idea with the SRV is from the operating uh system perspective is the idea is to have each VF to look like completely independent PCI uh devices with its own PCI configuration space its own MAC address and a lot of a lot of uh different configuration

but in reality a lot of this uh or or this VF will share the same underlying physical nick So for example the the bandwidths will be shared between all the VF and this becomes very relevant for what we will talk about today uh uh with the the bandwidth guarantee for example so now we understand what's quickly what a VF is why do they matter so SRV is

typically for and uh highend nicks that includes advanced capabilities like hardware lowloading Um so you can run RDMA, traffic steering, encryption offload and more. So having the ability to duplicate this uh physical nick uh into multiple sub interfaces allows each workloads to to get what they would get as a their own dedicated nick. So you can save money by having one nick and using it like multiple

times. And this is very critical for high performance and and advanced features at scale. So this um hardware level enable deterministic low latency networking. So we bypass the kernel and and we can process the the packets directly in the nit. So we don't involve the the CPU usage for example. So the typical use cases are telco use cases uh with the CNF. uh we have also HPC

with scientific computing. We have um uh large scale simula for example with the large scale simulation that require massive IO and and a IML training and inference with a large distributed uh node exchange uh and large volume of data. But now let's take a look at a real concrete example that Masaru will present. >> Yeah. Uh let me share one real use case we run in production.

Last year at the expo in Osaka, Japan, we operated our own pavilion and run video inference workload from onsite cameras. Things like for detection and crowd congestion detection. To do that, we build a video inference pipeline by connecting the pavilion to a site we call our regional edge crowd using RDMA. For the RDMA connectivity, we we used Nvidia Connect X with SROB batcher functions. One special part

of this setup is that the side to side link was provided by APN or Photonix network. So, so we connect into the connectex through an optical electrical convergence switch and that's it. And the same kind of RDMA based inference pipeline is applicable in much more common deployments even without APN. So thanks thanks Masaru for explanation of of of this concrete example that's uh of the usage of

SRV. Um now now let's take a or let's zoom out and what we are doing in kubernetes itself to support multi network configuration through the multi network sub project that is a a sig network sub project. So originally this work was captured in a cap uh 3698 uh multi network. So while this gap is now officially closed uh the objective were not abandoned and they have been

continued in fact in multiple uh different initiatives. So uh the the the parts of this gap that we're mentioning today are uh two user stories that were captured. The first one is the ver the virtual function itself uh feature and the gu the uh bandwidth guarantee feature or user story. So since as said this gap has been abandoned but the objective were not uh abandoned and it

has been continued and the key enabler here is the dynamic process allocation DRA. So DRA allows uh us in Kubernetes to manage hardware and virtual devices more flexibly including network interfaces. So several changes in GR and Kubernetes specifically targeted the networking area like cap 5075 consumable capacity another cap that we will touch about uh later that extend this this one and also the cap 4817 about um

uh uh device status for networking. So with this API we can now trade multin configuration has a first class citizen in Kubernetes. So for example, pods can request multiple network attachments to networks. U and the next step of the multin sub project is to get networks to be recognized by kubernetes in a standard way. So we can integrate with the rest of the ecosystem like services, network

policies and and more. So we have a real multi network um uh uh representation as a first class citizen in kubernetes and the next step the last step is um probably just the implementation of this kubernetes network driver that will represent the multi network. So what is this new model kubernetes network driver that I was talking about uh uh during the previous slide. So it's a new

model that's aimed to modernize um the existing Kubernetes network model by bridging CNI and device plug-in concept. So it leverage the strength of DRA for the scheduling part and NRI for the configuration. So NRI is a is a containerd or um container runtime extension where you can you can hook into some specific uh pod or container life cycle. So the go the goal here as I mentioned

is to treat networking networking as a first class uh uh resource in kubernetes. So this approach enable uh or simplify the complex use cases uh such as topology aware scheduling high performance networking and from a architecture standpoint uh candy provide just simplification so driver are independent from each other we don't have we don't no longer have this big drivers or big CNI that are doing everything at

the same time they are this the new model that we are trying to bring here is more composable so we can mix and match everything based on API instead of having EV coupling and it will make the life cycle management cleaner. The driver knows where to when to attach how to configure and where to uh release the resources uh in coordination with the pod life cycle itself.

So it will be fully integrated and Masar will now present a concrete um concretely hold this work with a concrete example. >> Yes, thank you Ryan for the background explanation. Kubernetes network driver uses D API the core of DR became G and it provides a new way to allocate devices to ports. In this talk, we highlighted two DR APIs that are especially useful nick bandwidth planning cons

capacity and partial devices. First uh let's think about how we could do SRO BBF bandwidth planning using the consumer capacity API. Consumer capacity is meant for the case where we multiple ports share a single device. near driver advertises the devices maximum capacity and then each port can request and to apply that model to SR BBFS we can treat each BF as a part of its parent PF

like this in this figure driver list the PF in the resource r and define two capacities on it the first is bandwidth which represent the available bandwidth and second is nom BF which represents how many BFs are available. In this example, we create two BFS. The driver sets maximum bandwidth to 100 GI and number of BF to two. Then when you deploy port a port the port

A request one BF by setting number of BF equals one and it requests the desired bandwidth by setting bandwidth equals 30 ZI. However, there's a key limitation with this approach. When you deploy port A, normal BF let you express that the port consumes one BF, but it doesn't let you specify which BF you get out of the two. If different BFs have different setting, you may want

to select a particular BF with the configuration you need. So in practice it's preferable it's preferable to treat each BF as an individual device and advertise them separately in the next let's look at the BF bandwidth planning with the partial devices sees this figure was originally proposed this feature was originally proposed with Nvidia GPU MIG in mind you split one physical device and each slice shows up

as a separate device that lines up nicely with what we wanted in the previous slide. Treating each BF as an individual device. A BF can be seen as a part of its parent BF and it also appears on the node as it as its own unique device. So the approach is you configure per BF bandwidth via config map then advertise each BF as an individual device in

the resource R. After that you can select the BF you want in the resource frame and allocate it to your port. In this video we have a parent VF called network and we created two BFS network Vel and B1. We assign 30 Gbps to net one B 0 and 17 Gbps to net one B1. Then network B is allocated to port A and network B1 is allocated

to port B. Now let's think about what an actual implementation looks like. How we design BF BF bandwidth using partner devices. We built the sampling sampling implementation on top of the CND driver with a config map like this. Users can configure bandwidth for each VF. We list the PF on the node and each charge BFS using their PCI addresses and then specify the percentage of bandwidth to

allocate to each BF. In this example, we create two BFS and we assign 80% of the bandwidth to one BF and 20% to the other. The CN driver reads the bandwidth assignment from the config map and then reflects them in the VF devices advertised in the resource price. With the personal devices API, we define a shared counter and set its maximum value. In our implementation, we name

that counter to include the PF's PCI addresses and we use it to represent the PF's maximum bandwidth 100 GI. Each BF references that shared counter and declar declares how much it consumes via a consumes counter. In this example, the BF net 1 B 0 is assigned a value of 80KI which correspond to about 80% of the PF. Uh it is matching what we configured in the config

map and the unit here is megabit per second that works out to roughly 80 GB per second. Now let's walk through the demo and verify the behavior. This demo is public on my GitHub. So if you are interested, please try it yourself. So uh I keep uh this figure on the side as a reference of my uh environment uh information. We have two worker nodes eagle04 with

an Intel E810 and Eagle05 with a connect X6 and I'll use the short hand V 0 means VF0 and V1 means VF1. So I'll say E0 for Vz and so on on each node we created two VFs Vz and V1. Now let's start the demo. First, let's confirm the environment. Oh. and uh we create the de config mark to configure the uh bandwidth. So e04 uh b

0 is 80% and the v1 is 20%. And e05 is uh b 0 is 40% and b1 is 60%. And I deploy the uh device class and CNI deploy deploy the CN driver. So let's confirm the driver port running on both nodes. Yeah, it succeed. So now let's inspect the resource rises starting with EU 0 for B 0. In the resource r bandwidth shows up in MPS

Mbps quantities for example 80k Mbps which correspond to 80 Gbps and and uh next here is Eagle 04 B1. You should see a 20 m 20k MPS. Next let's check the EU 05 B0. This one is 40k Mbps meaning 40 Gbps. And finally05 this one is 60k Mbps meaning 60 Gbps. So the config map is reflected correctly. Now let's deploy port A. Pod A requests an SR

VBF whose allocated bandwidth is greater than 50 Gbits per second. The key part is the cell expression in the resource claim template. We filter devices where allocated bandwidth is greater than 50,000 megabit per second. In other words, greater than 50 Gbit per second. And I defined IP address to assign BF ending in 119 for per day. So now I I apply Pday and check the resource claim

status. You can see P was assigned Eagle Z5 B1 in this case. And you can see the IP reset in the resource claim template ending in 119. And if we check inside the container with IP commands, the interface and the IP configuration match correctly. Now I deploy port B with similar request. Bandwidth is greater than 50 Gbits per second and IP is ending in one to zero

for port B. Since port already take one of the BF above above 50 Gbits per second port should run on the remaining VF that still satisfies that condition. I check the resource claim status for port B from the resource claim status. You can see port B was assigned E0 for V 0 and you can see the IP we set in the resource claim template ending in one

to zero. And again if we if we run IP command inside the container it matches. All right. Uh let's go back to the So with the personal devices API, we can treat SR VBFs as individual devices and plan bandwidths. But it's still not ideal. Cluster admins have to preconfigure every VF up front and end user can't request exactly the bandwidths they need at deploy time. They can

only pick from preconfigured VFS. What we really want is to request bandwidth dynamically when we deploy a port, express that intent in the port spec and let the system assign the right device accordingly. To make that possible, we are proposing a new DR API. Thank you Masaru for the great demo and explanation around um and the partionable devices and consumable capacity for SRV and bandwidth allocation. So

as mentioned as as master mentioned and demonstrated there's some problematic and there's this new cap that has been created uh 5941 by Sebastian here. So he's in the room if you want to ask him question after the presentation. So the idea is to extend um this cap 5075 consumable capacity so we can thread uh the V2 function use cases and share the bandwidth. So first 75 was

introduced uh as part of this consumable capacity API and it allows pods to request and consume capacities uh uh from a single device. So 5941 is built on top of that with two main enhancements. So the first one is to share the capacity. So multiple devices can share the the same underlying capacity as Mas mentioned multiple function virtual function on the single nick uh can share the

physical bandwidth of of that nick. So the user other use cases for this cap could be for example IP address management. So if we have multiple devices that are uh allocating IP addresses from the same IP pools, we can also represent this. uh I heard other use cases around CPU. So for example, we have S3 cache that is shared between different cores. So we can represent also

uh this this resources via via this um this new cap and also the memory bandwidth. So I think it will be very uh beneficial to have this new cap. So it will we can have a workload that um that consume resources that express correctly using the RS. So it allows workloads to coexist efficiently without overprovisioning hardware uh and breaking guarantees uh the bandwidth guarantees for example. So

the second improvement that we are thinking about for this cap is to have optional capacity with guarantees. So this enhancement allows capacity to be uh optional. Uh while the support guarantees uh are uh the the the gu the the while we support the guarantees and we we ensure that they are still there. So this is used in mixed um workloads environment where we have multiple tenants. So

some workloads may have guarantee guaranteed bandwidth requirements some other might not have it and work has best effort best effort basis so as of today so we don't want to mix them together and one have that works on best effort and one that consume all the bandwidth some are want to have bandwidth guarantees so they are splitted over different nodes so by combining this two improvement this

new cap 5941 provides a much flexible uh and real realistic model for resources in So to contribute to this to this um um Kubernetes network uh driver model um what do we have today? So we have different drivers that are already available. We have the um Kubernetes network drivers under Kubernetes 6 where you can already contribute to this project with new ideas. You can propose, you can

create issues, you can experiment here giving more ideas for how to improve the array, how to um expose the devices uh or resources in an efficient way that will solve your use cases. And this is mostly for this project or yeah this project is mostly made for incubating let's say incubating their drivers. We have JNET that maybe you heard about it from Google that has been donated

in DSCON uh to the community 6. So it's a pro promising option that is already running in production in some uh uh environment for handling RDMA uh nicks uh using the we have the syn driver also that Masaru demonstrated. Um so this one is is a bit complex. It was uh aimed for migration from multus to a more native or closer or putting closer multus to to

kubernetes. But there was so much complexity added on top of it that I do believe that this project might be deprecated or used for um experimentation still like Masaru did today. Um but it's good to it has been a good project for experimenting for consumable capacity the cap 4817 about device status. So if you want to contribute you are also welcome. I heard some people already used

it in production with some um improvement. Um but it it it could become very complex if we add all the features on top of it. And the last one is run by the Kubernetes network pling working group. So Sebastian uh who created the CAP um um 5941 is already working on this. He has been working on SRV CNI SROV device plug-in and all the ecosystem around it.

A lot of different CNI. So this CNI uh this D driver sorry um is a a D driver for SRVF as we similar as what we demonstrated here but with the particularity that it is um uh they might be um um a migration plan using extended uh resources uh kept in in the array. So we might have a migration path that uh that is clearly for from

the device SRV device plug-in to SRV DR Um so what are the latest changes in Kubernetes around networking area that could cover SRV? So we have uh multiple ones. So we have the three ones that uh we were mentioning about today. So the consumable capacity, the extended consumable capacity, partable devices. We I also mentioned the device status but we also have the uh device uh attribute in

donard API. So this is particularly useful I think for cube for example that needs to know PCI address uh PCI addresses of the device that is being configured in their workloads. So uh it would probably work in the same way as the rest of the donor API. So you can consume for example the attributes in environment variable or in config map or any way you declare you

declare the this configuration. So this is quite useful and if you want more information about the DR features I also put the planning of the the next feature. So there's tons of feature that might be interesting also for SRV use cases. And last being if you want to get involved in the community, get involved in the survey to fulfill your use cases, please scan this link. Uh

there is uh links towards the SIG network uh meeting uh agenda um Slack, there is the multi network sub project, there is the working group device management working on this. There is I put the plumbing group, the different caps that we have, all the DR drivers and there is the links to to this presentation. So I would like last to thank you for your attention and if

you have any question please feel free and you can also reach us uh after the presentation. Yeah we thank you for the interesting presentation. I have two questions. Uh do you need the CNI DR driver on top of another CNI or does it replace the CNI? So the the CI driver is calling CNI. So you pass the CNI config as part of uh the config in the

resource claim and you can you can call the CNI. So you just just need the CI driver and then you just need CNI uh uh together with it. So you can you can call it >> so I can then use any CNI >> right? That that's the idea at the beginning. But the problem is that the CN driver to support all the use cases because CNI is

very generic. you it's very hard what for example one of the problematic is how to expose which which device to expose in the syn driver it can be an overlay network in kubernetes it can be a vppP interface or v interface it can be a host interface and how do you represent it to cover all the use cases doesn't understand this so >> okay and the second

question in the example it looked like uh the the pots run on host network or or Where did the pot run host network or their own network? >> Uh network name specimen >> in the examples you you showed with with the cubectl xac and the IP command >> the the name of the network devices looked like this is host the host network. Is this true or is

it? >> No, it has the pod has its own network name space and then the the VF has been configured directly in this uh has been moved to the network name space. >> Okay. Because the name was not ETH0, ETH1 and so on. There were other names in the in the IP address show list >> uh >> in this sample implementation. So the interface name is the

same setting. So just move the move Nick to the content name space. The the interface name is keeping keep kept. >> Okay. Thank you. >> Hey uh good talk. I have a question maybe uh regarding the user interface of the of the new CAP uh resource. I'm curious how how it works when the node for example gets scheduled for maintenance and is drained. What can you explain

a little bit how how the attachment of the of the VF works when the like pod gets rescheduled to another node? And we've seen in the previous examples that I need to know the PCI address of the of the nick which uh changes based on where the which which nick the node has. If it's an Intel or Mananox, do you have an idea how this could work

like in a dyn more dynamic >> way or >> Yeah, I'm I'm not super aware of this. I know that there is I think there are some other deer improvement regarding uh no drain and and maintenance, but I'm not super aware of this. Um so I don't really have so much answer, but yeah, Sebastian might have more answer around So um I wanted actually to know if

um I need like to inject um then like the outcome of it. So basically the multiple interfaces um for a single pot. So do I have like necessary to use mult or does it like offer something out of the box >> which I can use. So in this example um we don't use moltus um the CNI config is directly in the resource flame or in the vice

class and we don't need multus at all. Um gearet is not using CNI at all. Uh SRD driver there's two mode I think you can talk to Sebastian just here. Um I think he used multus in one of the mode and uh it really depends on the implementation of this of the D driver. >> Okay. Um another smaller question maybe you know this out of your head.

So um how it's possible like to know how many v um VFS I would be able to create for one uh network interface um um this is uh this is based on the hardware. So uh you go do you know do you remember where it is you know to go in SIS class uh and then check it somewhere you can say SRV number of EF and you

have max number of VF where you can check this thank you very much >> okay um you focused on capacity and bandwidth uh config configurability but I have a question about hardware cues so are there any plans or is there any way to configure number of uh hardware cues in the resource claims? >> Um so the cues are shared between uh the VF right. So using this

new cap I do believe that it will be possible and if it's possible to configuring to configure it when you specify the VF then yes it probably will be possible and there is also uh the idea to support sub functions uh from SRV capabernics. So I maybe I don't know if Sebastian also might have answered or M do you have any more answer? Yeah. Uh the the

bandwidth is just example. So you can uh apply the various uh capacity uh things about about this with this new cap I think. >> Thank you for the presentation. Uh which tool uh did Masaharu used for the demo for typing? >> Typing demo. >> Yeah. Yeah. Uh, >> I'm pretty sure you didn't type that quick one. >> Did I use Aska? >> Yeah. And so demo magic.

Ask human with >> Okay. Please. >> Thank you very much. >> Right. Um, yeah. One last question. >> Yeah. Sorry, just a quick one. We're over time. >> So the end game for you is it for these resources to be managed like memory and CPU where people just request on the pod or do you want to keep this model? what the network team is going to expose

what's available and then people are going to request it. >> The idea is to that the pod is being able to request the VF that they want with the bandwidth that they want and as Master mentioned it's not possible today but with this new cap uh we we plan to to have this kind of >> so you're not going to keep this ID like right now we

have in storage where there's a separate object for a different team to announce what's actually available overall. No, but when you do this resource claim uh there will do some magic so nobody will take this >> Okay, cool. Thank you. >> Thank you. And yeah, thank you everybody. And if you have any more question, please feel free to reach.