Kubernetes Network Driver Unpacked: Modularity, Tra... Lionel J, Sebastian S, Antonio O & Sunyanan C

This talk discusses Kubernetes network drivers with a focus on Device Resource Allocation (DRA) and its impact on various networking needs. The speakers, representing major companies such as Google, Red Hat, and IBM, share their experiences and explore how DRA addresses the limitations of Kubernetes in managing high-performance networking, particularly for telco and AI workloads. They explain the benefits of integrating DRA into the scheduling processes, enabling better resource allocation and management across heterogeneous environments. The session also touches on the importance of a composable approach to network drivers, emphasizing the need for flexibility and interoperability among different networking constructs within the Kubernetes ecosystem.

Hello. Well, it's super hard to see with the lights. Well, nice to meet you. Hello. We are here to talk about Kubernetes network drivers. And let's introduce ourselves. My name is Antonio here. I work in in Google and I'm signal board and testing TL. I'm part of the Kubernetes steering committee. And I'm Jonas and I'm working at Red Hat. I'm working I'm a senior maintainer and working

on the machine and trucks and products within the uh signal board signal working Kubernetes and I've contributed a little bit to to the area and I think it's a great project. And hello, my name is Sebastian. I work in at Reddit and I'm a maintainer for the Kubernetes network planning working group and the sub projects like multius, SRV, and things like that. Hello, I'm Sunayana and from

IBM Research. I'm a maintainer of one of the CNI that we use in on our cloud and also I proposed one gap related to the DRA for networks and implement some DRA drivers. That's all of that. Okay. So, let's get going. Do you see that we have a lot of networking people here, right? From different areas. And you saw that the title of the of the talk

is Kubernetes network driver, but we are mentioning DRA, right? And one of the origins of DRA for networking was that all these communities needed some more capabilities in core Kubernetes, right? So, we have people working in in different areas of the project, especially on the accelerator stuff, and they were doing DRA. And that's when networking people went to DRA. And I want our co-speakers to explain a

bit more what problem is solving the DRA that could wasn't possible to solve before, right? It's why why did you need to use DRA? What are the benefits? Yeah, so I think I will start from my side. First of all, on the Telco area, if I'm looking, one of the core requirement for Telco is high performance networking that need, for example, SR-IOV devices and also low latency

with using CPU pinning and memory. So, for that, you can allocate with the extended resource that exist today using device plugin. And you can request using the kubelet feature from device manager, CPU you can request a single NUMA. But the problem is that happened on the node itself. So, on the allocation in kubelet, and you can get to a point where you still have available devices, for

example, in the node, but none of them is capable of doing a single NUMA. So, they are from different NUMA, and what will happen, your pod will get allocated there and get an an affinity error. So, with DRA, we can move that topology into the scheduler, and it's able to see the full cluster and get topology awareness, but also PCI route, for example, and other capabilities. Yeah,

for for my use cases that in our cloud, we want to run the AI trainings. I think everyone here I most of you here is on to run the AI trainings on on the cluster, and we are trying to um handles the secondaries. I mean, like um adding the secondary networks to increase the bandwidth so that it can do more like transfers. The problem we have is

on the cost optimizations of like the NIC selection stuff. So, what we do is we develop our CNI ourselves, and we do the policies based NIC selection in our CNI to like to optimize the cost of like which NIC is good for this GPU, for example. And also like uh we can do like dynamic metrics uh cost optimizations as well. But it mean that we put their

thing that scheduler should do in our CNI. Then there we know about the DRA comes and then we see the opportunity is to put the right thing in the right place. Yeah, on my side I will continue on what Peng and and Sebastian was saying. Uh that's was one of the objective of the Multus CNI project that's to have the network as a first class citizen in

Kubernetes. And one of the overlapping that was that was noted or marked during the during the first proposal was the overlapping with DRA. So DRA was a great opportunity for networking to be represented in in Kubernetes. So we can have bandwidth allocations as Peng mentioned. Uh so this is something that is not possible without scheduling integration. And the current solution or the past solution were fully or

mostly based on annotation or on something that Kubernetes cannot recognize. So this this new DRA API is is just a great opportunity for for multi network in general for in the in this case. Okay, so we were talking about different functionality, right? Different use cases and different platforms. And now how do you envision this future? Are we going to have an ecosystem with multiple drivers? Are we

going to have a like a super mega driver that is going to implement everything? You were working on this. You were developing drivers. What is your opinion on on this topic? So I I can start on this because I tried to do that with the CNI DRA driver. So it's something that was a bit what Multus was doing. Just we were having integration with the scheduler. But

there was a lot of complexity added on top of it because CNI is not is quite a old API. It's more than 10 years old. And it doesn't support all these new features and all the all the Kube is not is not something that Kubernetes recognize. For example, it doesn't support scheduling. It doesn't have any integration. It doesn't have any verb like add or delete to to

expose what devices it's it should support. the one of the main challenge with the CNI device driver was first, okay, what device do expose? We have a very large range of different CNIs. We have SRV CNI. We have Maglev CNI. We have all the overlay CNI that's that's now are running Kubernetes the primary network like Cilium or even Kubernetes Calico or any other. And then how to

bring them together was a real challenge. So it's it's it's not In my opinion, it's not really a good idea to have a single monolith CNI driver that will cover all of all all of these different CNIs. There were also other features that were not possible because CNI has said it's quite a old API. So they were not supporting validation. We were adding a lot of complexity

on top of it that makes the product over complicated compared to what we could do probably with a constellation of different CNI drivers. I will say from the other side on the constellation of multiple drivers you are going from some time you if you have today if you are familiar with CNI for secondary networking, you can have uh your CNI, but you can also have for example

your IPAM and that can be reused. You can use Whereabouts. You can use whatever IPAM you want and it's not completely connected to the CNI that moves the interface inside the pod. That can be a different one. So if you will have different drivers, then all of them needs to implement the same IPAM, the same routing, CC TL, and all the options that today you are just

let's call it chaining or putting one after the other with your primary CNI. That's one thing, but the problem with multiple of them is, for example, if you have a MacVLAN or IPVLAN DRA driver, and you have a host CNI, for example, or host DRA driver, what's happened when that one moved the full interface into the pod? Now, the other ones need to update their resource lines,

for example, because the device doesn't exist anymore. So, that can be or you can have running containers that now their primary NIC is somewhere else. Yeah, I think so on this topic we discuss a lot even like 1 hour before this thing we still discuss about this like how we going to do about what their I think the reason is because it doesn't like their Nvidia GPU

driver which have like one vendors doing things like tree wheels, not not tree wheels, like direct things that they know what to do with this device and no one going to get the Nvidia GPU driver and do something else from the from the drivers, but the network is different. We put the fancy thing on the overlay everyone have a different way on configuring the network and also

it depend on the infrastructure as well. So, like I I do agree with Lionel's and Sebastian and I think Lionel's also have the same agreement that we are it's it's too hard to have a make us DRA driver for doing all the things. But, the next challenge that we still discuss is how we are going to synchronize multiple driver if it's co-exist in the same cluster. Whether

like we should have the meta's meta driver like a multi's do like connect everything to this one or it could be like changing some API to have like I think we can touch this a little bit later for more details. So, we have a couple of the discussions going on like how we are going to do with the multi-driver synchronizations. So, if you are interested, we can

go over it. It's a long story. I don't want to touch it here. No, it's it's it's good. Because people come to KubeCon and okay, we are talking about all these things, but the main question that we get after this and people when reach out to you in during the hallways, okay, that's really cool, but when can I use it? What can I use it? So, let's

let's dig more into this. This what people can use today with with the array and all these new functionality we have in in this slide, right? What? I I moved to this slide? No, it's this slide. Yeah. You want to start? Oh, yeah, yeah. I I can start with the Kev stuff because like it's related to the one that I proposed to the community. It's like as

I told you like I work on the multi-NIC scenario and I want to move to the DRA and then I check out like what is already in the DRA at that time. The network is something that I want it to be shared with multiple pods. So, I check the share features inside the DRA and I found the partitionable device and trying first trying to use it first.

And then then I found the use case that it doesn't cover that like the like I know has already touched about it. It's the bandwidth allocations where we don't partition the bandwidth in advance. We just request the bandwidth at the time of the of of of the like the pod creations. So, that's why I started the Kev called consumable capacity to make like the capacity of the

network device can be consumable in the in the run time like in the request time. Yeah, and then I came and see that this one is not good enough for my use case. how we say, something was missing on this one. So, from my side as a maintainer for the SR-IOV project, we also are interesting on the bandwidth allocation, but our problem is a bit different. So,

I can't put a bandwidth allocation on every virtual functions. That will not be successful and will not be right on the amount of allocations. So, there is a new cap here that still proposed for shared consumable capacity and this way I will be able to model the capacity as the PF source and you can see the example that we put here. For example, that I have one

PF with 100 gig NICs and then when we request virtual functions, I don't care if I have 100 virtual functions available on the system if to allocate 50 gigs of bandwidth, that's all I can allocate from that device. And maybe I can add that this problematic with the modeling of of the resources in the area is probably one of the largest challenge that we have. We keep

talking about that with paying and we keep adding new ideas and and we are thinking about new caps about it, but it's it's it's very complicated. They are multiple tenant as in a cluster has been mentioned. They have multiple different use cases. There are the overlay networks from from the different CNI and we have for example the the telco Sebastian is representing with the SR-IOV and probably

with the RNA is also something that's is problematic. If you you don't want to overlap the resources that that the different driver would expose. Yeah, that's indeed a a complex problem to solve, right? Oh, well. Let's move on. So, right now I I there is some people that say, "Okay, I'm super crazy this week. I'm going to install these things. I'm going to prove everything." But unfortunately,

most of the people work for some companies and the things are already running, right? how can we tell them, "Okay, don't worry about you. We are thinking in in you and this is not something that you just need to throw everything that you have. This is something that you can start using since this version because we have this migration or you can do this step or the

other." Yes, I think I can touch on this one. So, what one of the idea is using the feature in DRA that allow you to expose DRA devices as extended resource So, extended resources via the DRA API. In this way, for example, again in the SR-IOV area for Telco, I would say, you if you are using the SR-IOV operator or you are just using the SR-IOV device

plugin itself, you can switch and replace the device plugin with the D- SR-IOV DRA device plugin and your existing workload will continue to use the same resources like stan- extended resource under the standard API and they will continue to use. So, you don't need to go to your application or deployment and say, "Hey, now you need to start using resource claims, resource claim template, and all that

stuff." You will still use the regular one and it will continue to work. But at the same time, for new projects that want to use, for example, colocation between GPUs and high-performance devices, RDMA devices, and all that stuff, you can use the resource claim to have the constraint and the connection between the devices. So, I think it's a good bridge between what you currently have and it's

working to the new API and both of them lives in the cluster at the same time. No. Okay. less I think that we cover a bit of what is DRA, what we been doing, but I think that we were very technical. So, let's try to to give a bit more of examples to the people of what we are doing and what can they expect for DRA, what

bigger problems are we going to solve within Sig Network and and the community with this. I will start with DRA net. I don't know people that were in Atlanta. We created DRA net that is a network driver for AI workloads especially for RDMA interfaces. Um the project is is is working in production. It's now having support for Azure. It's going to have support for Amazon, too. So,

I think that we are growing as small but consistent community there. But we also have another groups that are investing on DRA and and planning to release new features and projects. So, please tell tell us a bit more about Right. So, so this this is the project that I was touching before, the multi-network subproject that's had a cap previously in 2022 and it's a feature that we

were waiting for a very long time and that that has a lot of background around. So, I think it started maybe 2017 or 16 we had and then we had the the plumbing group that solved the problem in their own way that was a bit separated from the the community, I would say. And then now as I said, we see a new opportunity with DRA to to

solve this use case. So, the multi-network community is trying to now to propose this new API and network class or network kind now, I think we will call it. Um that will have a route that will provide a user or cluster admin to represent um secondary network secondary networks in That's a very good step for integration with the rest of the ecosystem and to grow the API

around CNI networks. So, we can have for example, we can touch around network policy. We can include services and this is quite important use case for example for Telco that's is currently solved for example with the plumbing group. Maybe Sebastian, if you want to touch about this. Yeah, so I think from the plumbing working group in project that was started a few years ago with one of

the key component that was multus allowing you like to run to connect multiple interfaces inside the container. So, it was a bit on the side of CNI network and everything is going there and what at least me as a maintainer for that organization is trying to do is getting closer again to the CNI network, see what we can contribute, what we can move again close to the

CNI network to benefit the whole community. If it's the API, if it's the way we are connecting things and the features that we have many user requesting like chaining, connecting different type of interfaces having agnostic way of how you bring your own IPAM, your own way to connect. You can call it CNI, you DRA network in the future. The way you are connecting it and have a

community around that API. And and this is quite important part because that's what we're talking about with Antonio is the portability of the API. So, we want the user to be able to deploy or to orchestrate their their workloads on over CNI networks in a consistent consistent way. We don't want them to go on different distribution have I don't know multus on one distribution and something else

in the distribution. That makes life quite hard I do believe. Yeah, that's the whole GitOps thing, right? The GitOps work because they just deploy your YAML and it works everywhere. That's the secret sauce of Kubernetes portability. and with this I think that this panel come to a conclusion. You can check on the QR code. We have all the working groups, all the links, all the material. And

just if you have any questions, suggestions, or you want to participate, just reach out to us in these channels. And if you have any questions, feel free to to go to the mic in the center and ask any question to to us that we can help you. No questions? Come on, networking is not easy. So, uh you mentioned it uh a few different drivers and like you

you you talked about the caps and when the caps were went into are going into beta this release, but like what drivers are available today that you could deploy and make use of? And and what are are they are they vendor specific? Are they based on just the Linux kernel features? Like what what's available today? right now we have different driver, right? The SRIOV is the one

that we created at Google and was donated and is is being more stable. And but it's planning to to get all these partitionable devices and and consumable capacity, uh especially for creating IPVLAN devices. It's it's a use case that is coming. But I heard from Sebastian that in SRIOV they are trying to create something with bandwidth allocation or something like that. Yeah, so from the SRIOV there

is a there is a project called SR-IOV device plugin that is running in production in a lot of places, especially in telco customers I know they are running. It can be a standalone or can be a on top of the SR-IOV network operator and and that's orchestrate everything. And what we are doing right now, we have the DRA SR-IOV driver and our planning for the near future

is to do the replacement of the SR-IOV device plugin with the DRA together with the extended resource via DRA. So, this way it will be able to support existing customers and new customers with it. Um maybe I can add the two other drivers. There's There's the CNI driver. Uh so, this one was more an ambition to have evolution of, for example, Multus with integration of DRS, so

it's closer to um to Kubernetes. But, as I said before, it's quite complex to make to to to to develop. And I don't believe that this is the final way. It's more having having more composable device or DRA drivers that can probably interact and that are more independent. Might be better solution than this single DRA driver. There is also another report that Antonio created, this Kubernetes network

drivers. So, there you can experiment, you can propose new things and new ideas, and we'll be very happy to to hear about about these new ideas. It will really help the community in making this evolution happen. Thanks, Jared. Um thanks very much for the talk. Uh it seems a lot of the community, especially the hyperscalers, are kind of standardizing on the Cilium CNI. How does that interact

with the DRA approach that you're talking about or it's kind of just two separate things in different directions or That's a good question and that's that's precisely where where if you remember the secondary lies we talked about the monolithic versus constellation thing. So, think about CNI Kubernetes network is is relatively simple and then you have a pod, you have an IP, everybody can connect to everybody. That's

the CNI model, right? And then in 2023 or 2024 all these multi-network AI or secondary networks came back to Kubernetes again, right? the problem is that because we couldn't break the existing model, we created a new model that allowed this flexibility. So, all the CNI stuff is the same as always. That's is not going to regress. It's another it's another track. This DRA what what it does

is it enables this new functionality in a way that is additive. So, we don't break any functionality. The people that want all these special functions have this standardization based on DRA. And from there we're going to grow to AI workloads, to telco workloads, to more complex SR-IOV bare metal workloads. So, are two separate tracks. Okay, great. Thanks very much. One thing that I may add about that

maybe we have to be aware between like having the CNI and the DRA driver at the same time as well. I mean, we're talking about multiple driver pointing to the same resource, but at the same time we have the CNI with another word pointing to the same device as well. So, yeah. I I think this integration is more for very long-term feature, so. So, please don't use

CNI at the same time as the Is there time for one more question or we are we out? Yeah, I don't know. All right. So, so talking about decomposing the the drivers like are you thinking so so one way you could do that, right? Is like okay, your driver does some IPAM request and then record the IP address in the status like we've talked about but you

could also treat an IP address as a separate resource and have its own IP driver. Have you thought about decomposing it in that way where you have separate drivers and and I don't know how those all get composed together in the end but um that's another approach you can think about. >> I think I think we've also about the capacity. I think it was Mike Zappa that

demonstrated this in the SIG network meeting. So he was exposing uh the subnet as uh or the the number of of IP that can be allocated has a consumable capacity. So you could say, "Okay, I have 255 IP addresses uh possible and then GRI would help to not schedule a pod could not get an IP allocated at the end. So this is something that's uh the both

kept kept that bank created and kept that Sebastian created could help with I believe. Yes. Um our Yeah, but I just would like to add that there at at the current point we don't have the the concept having the IP and the array driver separating from the from the to host our device the array drivers for now but it's very interesting idea as well. We can think

about it. Yeah, the Mike Zappa demo was I think was conceptual so yeah. Uh okay, Yeah, I I I think I I I took on that one on one of the cons about like splitting too many DRA drivers if you want to you will need to also implement same IP again and again on the same technology. So I think it's a it's one of the ideas to

have it as outside and then the DRA that does the connection may be able to connect via GRPC or something like that to a global IP driver that shouldn't be DRA. It can It's a good idea to have it at at least as a consumable capacity key because we know how much devices we still can allocate to I think it's a good idea. I think we have

time for a last question. >> Yeah. A quick one. Uh I'm a newbie in this context of the drivers, but we usually uh when you talk about SR-IOV being able to attach like the VF to the pod, can this work also in in a virtualized environment where you have like you know you know when when I have the bare metal, the server, and I I have something

like OpenStack, I attach the VF to my OpenStack VM, which kind of kind of like is my Kubernetes node. Can the whole thing work to get the VF to the pod when there is this layer in between? Yeah, so we already support this one with the SR-IOV device plugin in this case, but also with the DRA SR-IOV we can support this one. And how it works, it's

a bit different. So, the thing is you will allocate the virtual functions, and you will move all of them into your virtual machine as many as you want for the workload, and it will look like regular devices. They will not have like any special, I will say, functionality from physical function point of view. There virtual functions, and you just going to see them as available devices in

the array, and you can just allocate them. So, that shouldn't be a problem. Well, we are out of time. Thank you very much, any question whatever we can answer to you. Thank you.