SIG Network: The State of Networking for AI on Ku... David M, Haiyan M, Bowei D & Kellen S & Nadia P

This talk introduces the Special Interest Group (SIG) Network's efforts in Kubernetes, particularly focusing on the integration and management of networking functions relevant to AI and Machine Learning (ML). The panelists from various tech companies including Google, Red Hat, and Nvidia discuss their roles and the importance of networking in supporting AI workloads. They elaborate on specific networking projects such as DRA net, a Kubernetes network driver designed to optimize RDMA networking for high-performance workloads. Additionally, they present the Gateway API and its inference extension, showcasing how it facilitates inference aware scheduling and flow control for AI applications. The discussion also highlights the development of the Cube Agent networking project, aimed at standardizing communication for AI agents within Kubernetes. Throughout the session, the panelists address the unique challenges AI workloads present to traditional networking models and the evolving requirements of observability in this context.

Okay, let's get started. Welcome to the SIG Network update. I know this is a very late session, so thank you for coming. First we will introduce our panelists. So, maybe introduce yourself from that end to here. Hey all, I'm Kellen. I'm filling in for Etai. I work for Google on the GKE inference team. Hey, I'm David. I work at Red Hat on MCP gateway. Hey, I'm Nadia.

I work at Nvidia on Kubernetes networking. Hello, I'm Haiyan. I'm from Google and I'm working on the Cube AG and the networking project. And hi, I'm Bowie. I'm work at Google and I'm the SIG Network chair. So, this session will be basically an introduction to SIG Network, kind of the scope that we cover. And then for particularly for this session, we want to kind of focus on

what is happening in AI/ML. I don't know if you know, it's kind of a big thing on everyone's mind. And then we want to reserve a lot of time for Q&A, so probably like 15 minutes at the end for Q&A. First we have some canned questions, but not too many. And then you will basically be able to ask your own. So, first introduction to SIG Network. So,

what is SIG Network? SIG Network manages the components, interfaces, and APIs for networking Kubernetes. It maintains core code such as Q proxy and also the service API and those things. And it defines APIs for many networking areas. You know, key among them ingress gateway API. And we have a variety of sub projects just cuz the scope is so big. So, we have a sub project in network

policy, multi network, and gateway API. And of course the AI related sub projects as well. And we work with lots of SIGs cuz you know networking kind of connects lots of things together. So, we work with SIG Node, SIG Scheduling, and Multi Cluster. And we participate in a lot of working groups, you know, AI gateway working group, AI conformance working group. So, kind of in terms of

AI/ML, thinking about it, it's like what are the kind of basic problems and then how do they map to categories of, say, working groups or areas in SIG Network? And just being very broad here, right, in AI we have, you know, how do you get the hardware, like the GPUs, accelerators, and so forth. How do you integrate it into your training? How do you serve inference workloads?

And then finally, we have this new category of workloads that's pretty special, agentic workloads. Like how do you integrate that into your Kubernetes cluster? So, if you look at how it maps to SIG Network, we have the Gateway API project and the Gateway API inference project, which is talking about agentic and inference. And then we have, you know, agentic networking, which is like looking specifically at how

to do networking for agentic. And we have things like DRA, which talk about how to kind of associate your hardware with your networking. So, first, um we have Nadia to talk about DRA net. Right. So, we'll start with the more traditional, potentially boring part of that, which is the actual networking interfaces and how to attach and configure them on your pods. Um if you're familiar with that

part of the networking, it used to be mostly covered by group outside of Kubernetes, which is called network plumbing, but now it is moving slightly more into the SIG Network realm and it all started with the introduction of DRA net, which is a Kubernetes network driver written by our good friend Antonio, uh which uses dynamic resource allocation and it helps you deliver high-performance networking. it doesn't cover

all the use cases possible, but it works pretty well for a set of use cases, basically to enable RDMA kind of networking in just one plugin without installing a thousand different things. Uh so, some of the things that are usually very important when you do that is first of all, it's resource discovery. When you have a node, it has some networking interfaces, you want to be able

to see them, uh see what kinds of properties they have. So, that's a quick example I have here, which is the result of the array network. It just lists you all the devices you have with all kinds of different attributes. And most important ones would be like RDMA related stuff and PCI addresses and attributes like that. So, it already does that. Now, for the RDMA handling, there

are different types of it, different details that you don't really need to know very deeply, but what I want to say here is that DRA net handles different options for RDMA and well, can cover quite a lot of use cases with that. Um and another very important part that it introduces and that also leverages the uh DRA part of it is that it helps with scheduling and

more precisely with topology awareness. So, what it lets you do right now is to have the node-wide topology awareness, where you can actually match attributes that have been standardized now like the PCI root. You may know that it may be very important for the GPU and um unique to be on the same PCI root to actually have the best performance, which is very important for AI and

ML kind of workloads. So, that's a short example of how you can do that by using the attributes exposed by the DRA net. And well, as long as you're using also the supported GPU operator that also has the right DRA attributes on it. as I said, the within the node DRA net kind of solves the problem with this topological awareness, but there is also some extra work

on the cross-node networking constraints because you know that AI/ML jobs can be pretty large and sometimes you really need to ensure optimal scheduling across multiple nodes. And it's actually work in progress that we are doing together with six scheduling and it's a part of the current running like port group scheduling effort. So, just a very high-level um view on that is what we basically want to do

is make sure that the uh optimal placement of different devices is not only available on the same node, but also across nodes with the regular core scheduler that Kubernetes has. You might have probably heard of all the other different schedulers that exist on top of Kubernetes and that basically replace the default one. As like Q, Kai, uh Slurm has some of that for to schedule AI workloads.

That does a lot of that. So, a part of this work is also moving now, well, into the core Kubernetes. That's basically it. All right. Talk about gateway API. The Kubernetes gateway API is a collection of resources that a model service networking. It include a gateway classes, gateway HTTP routes, and TLS routes. It's a role oriented, it's portable, and it's expressive. Uh it's very extensible. Um for

the extensible part, there were several AI related extension. Um on the left side, you can see the gateway API inference extension. It extended the gateway API to optimize the self-hosting generative models on Kubernetes. It introduced the two CRD, inference pool and the inference objective. Um currently, uh there's also a AI gateway work group. And there they have a proposal for up streaming one of the CRD they

introduced called ingress back backends. They try to up stream this into the core Gateway API. And another effort called Cube agent take networking project. Um, they are currently they have two CRD ongoing. One is access policy, another is telemetry policy. And in the future some of these CRDs may also will be merged into the core Gateway API if they target a broader audience. Thank you. Cool. So,

I'm going to talk about the Gateway API inference extension. I often refer to it as inference Gateway. It's the same thing. So, we are why inference pool exists and how it ties into the the Gateway. One of the reasons we did this was because with AI workloads it's very very different from standard web serving. So, we took a took a look at this and we found that

looking at the body, which would typically be a networking no-no, actually helps quite a bit with inference serving. So, we created a new way to handle that. So, that is that is why we exist. The claim to fame so so to speak is our inference aware scheduling, the prefix cache aware routing. That's one that's commonly used. If you use PEFT like Laura, that's another way to get

some efficiencies. But, some new things that we're working on this year that I think will be fascinating is flow control. Flow control will essentially create a buffer on the inference pool or the EPP to buffer requests, which allows us to both manage the pool so it's not overloaded as well as give you rich metrics like the the fullness of your pool, which you can use for auto

scaling, which was typically a little more challenging with inference workloads. Additionally, the the thing I think I'm most excited about is being able to mix priorities. So, when your production traffic is low, you don't want to downscale, but you can fill it with offline batch or other low priority requests that you can then evict, which we're actually working on implementing now. We just got that figured out

yesterday, which is exciting. Uh so, then we'll be able to utilize try to keep it at capacity as long as possible because I'm sure we all know these are expensive, so using it as much as you can is a good thing. And then, as was mentioned, we absolutely integrate with the broader ecosystem. The inference pool directly ties into gateway API as a back-end ref. And then, we

have the inference objective and the EPP is completely customizable to your workload. So, that's inference gateway. Is the Cube Agent to Agent Networking Project, uh it aims to provide a standardized API for secure governed communication for the next generation of AI agents. Um the communication could be between agent and agent, agent to tools, and agent to LLMs across the Kubernetes cluster boundaries, including ingress, egress, and east-west

traffic. Um regarding the goal of the project, we want the the API to be standardized, also be adaptive. So, we are trying to design it around the user-facing goals instead of specific API protocol. And regarding the security and the governance part, we want to it include authorization policy, so we can control exactly which agent access which tools. And we want it to have AI safety integration, covering

external authentication and authorization, and the policy enforcement. We also want to to achieve a durable traffic management including rate limiting and the traffic auditing. And the API should also be seamlessly integrated with the gateway API and the major AI protocols and the API should be extensible so it can be implemented according to different needs like a service mesh gateway and a future architecture. Uh so staying on

the topic of agent networking, um what we're focusing on is a few user journeys around these personas. Um initially anyways. So we have like the AI engineer. Um they might want to assign identities to agents, uh audit agent actions, or understand why an agent gets denials um on tool calls. Uh for the platform engineer, uh think of this maybe like an SRE as well. Um they might

want to uh define like a default deny agent traffic policy and then leave it to the AI engineers to be more specific. Uh control access to tools, to MCP servers, observe platform-wide failures or denials. So they want that kind of higher-level view rather than specific tools or servers. And then the security engineer. Uh so this this is looking a little bit more forward but can we do

like uh pre-request filtering uh to avoid attacks like prompt injection? So we're talking guardrails here. Or post response filtering to prevent data data breaches, data exfiltration. and then uh just to say about the ongoing work and future work. Uh so at the moment uh we have this we're proposing this new back-end resource. The X there by the way is just to kind of say uh to show

that hey this is really alpha alpha zero V0 alpha zero something like that. So don't use it yet. We're putting together a a implementation as a demo. Um there's a talk on that. I'll say more about that in a sec. But we have like an X back-end resource. And then it's uh XX access policy resource, which you can target a gateway or a back-end. And that's where

we're saying, "Hey, these agents are allowed to access these tools or call these tools." We're doing some stuff around observability as well. So that's the main area I'm focused on. Um so what kind of standards can we bring in here, existing standards? And can we standardize on some sort of telemetry policy for saying, "These are the signals we want to get out of the system, and they

need to be kind of they need to have semantic value because we're talking about agentic workloads." Um so future work, can we uh can we support some sort of deny list as well, not just allow lists? Um authorization policies uh for communication between agents and agents or agents and LLMs. So as I Han said, initially we're just focused on agent tools, but the rest will kind of

come in time. Uh governance of ingress AI traffic, uh rate limiting is another interesting one, particularly if you're talking about like token-based rate limiting, cuz the number of tokens that are being used, you have to do the inference first and then see, "Oh, I'm I'm over my limit." Do you send the response? Hmm, who knows? Uh and then guardrails as well. I mentioned that. So doing the

pre-request, post-request guardrails. Um so this is the talk that we're going to more detail and have a demo on some of the uh access policy work. That's on tomorrow at 5:30. So Han and Evelyn, who I think is in the audience, will be presenting that. Um so that's the agentic networking. the working group, the AI gateway working group, um so just to say about the difference between

the working group and the agentic networking project, the project will produce some reference implementation, uh some code, but the working group is more about the proposals, uh, and saying, "Hey, here's an API." Um, so this working group is sponsored by SIG Network and SIG MultiCluster. Um, so seeing how can we standardize how gateways handle AI traffic, so this kind of feeds into your AI traffic networking. So

we want to build on that Gateway API specification as much as possible, not kind of reinvent everything from scratch. Um, there's two main proposals there at the moment. The first one is, uh, payload processing, so that's pretty much looking inside the body. How do we do that in some sort of standard way? And that enables then the prompt guardrails, semantic routing, caching, uh, rag augmentation, or the

the, uh, kind of tool po- tool policy stuff we talked about. Uh, the other proposal is egress gateways. So routing traffic to external inference services. Um, so yeah, maybe I want to use some cloud provider, uh, for for for model lease. So how can yeah, so with that Okay, let's move on to discussion. By the way, you should use it, just don't use it in production. Actually,

the reference implementation is like the perfect time, you know, the X's in front of everything, to give really good feedback on like how well it works, whether or not it addresses your issues, whether we should be looking at something else. So please actually go use the implementation, just don't file Well, you can file bugs, but don't put your production on it. All right, so we're going to

start off, um, the discussion. And I get to ask the questions. So just just a quickie, so, you know, what are the biggest challenges you see regarding networking in AI? And other question similar is like, why networking for AI? How is it different than just the standard Kubernetes networking? I can take that one. Um, like mentioned earlier, we do have to look into the body a lot

more than you would in web serving. By a lot more, I mean every request versus none. additionally, the order of magnitude of request volume is or excuse me, inference workloads should have an order of magnitude less request volume even more most likely. So, some of the things that we need to do in web serving, we may not need to do an inference and vice versa. So, I

think treating it as distinct will give us a different angle to look at the problem and hopefully provide optimizations there. Yeah, well for a different type of AI workload, which would be more focused on training, you also have very different performance requirements to what the regular Kubernetes networking has. Also, it doesn't use most of the Kubernetes networking features because Kubernetes networking was like so service web service

focused and AI doesn't really need any of that. So, Kubernetes services are not that popular on that side. Like even DNS resolution is not always needed. All they want is just give me my interface with as little virtualization in between as possible. So, that brings all these extra special ways of multi-networking and secondary interfaces and lots of AI workloads actually will have the normal primary Kubernetes networking,

but will almost never use it or use it very minimally. So. Okay. Um the second one is how opinionated and exposed the user should these functions be? And also a similar question is so, you're just introducing new CRDs. Like how is this different than kind of the standard mechanisms and how do you balance that trade-off between like, you know, some's like, "Hey, you just tacked the word

AI on every single CRD." Like isn't this just a normal thing? So, you know, what where is that balance, right? I can take some of that. Yeah. Um so, in ter- terms of yeah, are we are we just slapping on AI onto these? So, I don't think that's the intention. The intention is let's let's look into let's kind of layer on top of the existing CRDs as

much as possible. So, gateways HP routes, see how far we can get. And when we can't, okay, that then let's introduce a new API. It may not make sense to push these upstream back to gateway API afterwards because they may not be generically like usable outside of AI. That that's perfectly fine. But in the case of say backend that that's needed here if we want to do

say in the egress use case because how do you how do you define an external an external host with a service right now? That's tricky. You have to you have the external host name and there's other trickiness as well. But with a with a backend resource that the intention is to make that a lot easier and backend resource can be referenced in the HP route. So, we're

not we're not inventing a new resource. We're seeing okay, we're saying use the backend ref in the HP route. Um, the access policy that's that's an an interesting one as a new resource because like what's the alternative? Do you define your access policy rules directly in the gateway or in the HP route closer to where you define your routes or listeners? Um, so I think having a

separate resource there makes a lot of sense. Though, it does follow the policy attachment model which has its own kind of caveats. Do you want to add anything, Calvin? Um, I think you covered it well. Okay. All right, final question before the audience questions. Please prepare your hard questions. So, what do you think the impact is on um observability? Like all these AI workloads, does it impose

kind of new requirements or like what's special about that? I'll jump in here again. I don't know if anyone else wants to. Yeah, so observability of AI it's it's it's different than traditional observability of say HTTP requests. In HTTP land you're worried about latency, throughput, that kind of thing. So you can have like your SLIs and SLOs defined around those things. When we're talking about a generic

workloads, the semantic meaning is is a lot more important. So how can we bubble that up through our observability signals? But in order to do that we need to look into the request body. There's no avoiding that. Pull out that semantic meaning and then expose that as observability signals. So in in the Agentry networking project we have a proposal around observability that has landed. And that's more

or less saying though we're not saying we're not building something new here. We're saying use the existing open telemetry spans, trace, etc. uh rather than something completely new. And also when it comes to propagating that those signals throughout your system. So from agents to tools to models leverage the W3C standards for trace propagation or the baggage standard as well for propagating like the user ID, agent ID.

And by doing that on the back end then when you're correlating all these traces, you should in theory have like a single trace for what an agent is doing. Like what you really want to answer is okay, this agent last night it made like 1,200 calls to this tool. Why did that happen? You need to get the whole picture of that. Okay, now Quick >> Oh, yeah.

Quick observability from the accelerated networking world because it is actually very different too from the normal networking observability, which most of the tools are now well, basically observing the traffic on the host EBPF based kind of things. And none of that actually works with the accelerated networking. So, a set of observability tools and what you can actually do is also very different That's true. It's on a

separate network. Now, we open it up for There has to be at least one. Anything? Could be on anything. Okay, well, please use the mic. Thank you for the talk. Uh the question regarding about the DRAIN net. Uh do you have any plans to uh change the motors with it? Or not? No. Or any plans in in terms of CNI CNI V2 >> Also also no. That's

good that you're asking these questions because we have actually established the answers to those. We are not trying to replace them. We're kind of working in parallel and it's going to take time and we are not even sure if it's possible to reach the total parity in between like the whole network plumbing world and like what DRAIN net and maybe other plugins that are going to be

developed will be doing. So, we are not replacing anyone and not trying to do that now, but we are actually now trying to work together a bit more. You may I don't know if you've been to the uh presentation just two slots before, I think, uh where at that part was discussed also about the future of the networking and we now have network plumbing group representatives talking

with us together and working a bit more. So, there is a hopefully a chance that these two kind of ecosystems are going to grow a bit closer together. But, we don't want to like replace anyone and say, "Okay, this is V2. That's how you do it now." That's not planned. Okay, thank you. Any other questions? No? Go ahead. Go ahead. Yeah, please use Hi, thanks for the

talk. I have a question about the AI inference. So, for uh the for the pre- pre-fill and the decode the disaggregation, what would be the challenge for the network? Yeah, um it depends on implementation. Uh that's that's uh topic of hot discussion, I would say. It's a great question. Uh currently, we we would call it instantaneous decode scheduling. So, when we schedule pre-fill and decode, we do

that at the same time. Uh we're finding that we don't want to do that specifically with YDP. Uh with YDP, you really want to avoid uh overfilling the HBM on your decode. So, you want to do deferred decode. Uh so, the what does that mean? It means we might need to take a look at the way we currently use XProc uh because XProc does not allow us

to do deferred decode. So, we might either have to do some clever things or some additional components to allow us to do that deferred decode. So, that's the thought. Anyone else? You can ask about anything, not just AI. Anything you would like the SIG to be looking at that you don't think uh we're looking at right now. I'll throw that since no one else is. so, you

say you're going to be looking at the body. is there metadata already in that body? Is that already structured or is there going to be another agent to look at the to get observability or whatever? I can take that, but first I'll make sure. Um so, specifically with routing, we care about the prompt. Uh, we can't know how to what the KB cache is going to look

like and how to match that without looking at the prompt. Uh, originally we also looked at the model name, but that's when you can put in a header, uh, but the the prompt is is very, very unlikely to come out of the body, uh, because what would be left. But, uh, that's going to be true, I think, as well with multi-modality. Uh, we're going to have the

different assets and their encodings, whatever that's represented as. So, we'll need to take a look at that as well. Um, that's from the inference routing side. Yeah, I can take a talk about the access policy part. Yeah. For the Cube Agent and Network Working project, like we are trying to uh, have a policy to control which agent is allowed to access which tool. So, we do need

to get the tool name from the body. That's where like you know, when we do the reference implementation, currently like we're just using Envoy. So, they have some HTTP filter, which can look into the body and get all the tool name out and put into the dynamic metadata. And then, later you can have like our back policy or external auth to leverage this information, so you can

to do the authorization part. That's uh, reading the structured data, isn't it? JSON-RPC or it's it's not just like a regex, you tell Envoy it's like, "Hey, I understand this JSON-RPC message." Yeah. Okay. So, yeah, it's structured enough. It's time for one last question, and they can't run away. All right, please. Um, my question is that to support the agent workload and AI workload, right now I've

been discussing more about change on the application layer. Do you foresee that we may need it to do change on other layers of network stack like transport layer to better improve the support of this kind of workloads. I'll try to take that one. I think the question was around do we need to change the network stack specifically for agentic workloads? Is that Is that a fair summary?

Yes. Cool. So, I know that we have people doing work here to support that. As far as like just inference, um there's things that we can do, but I where inference pool is positioned, we get to treat agents as just another request uh because we're not concerned about the tool calling. That's That's outside of our purview. Uh so, from our perspective, there are things you can do

like maybe after end tool calls, you'll you'll ratchet up the priority because you don't want this request to run too long, so then it can take some priority, but that's just a That's workload management and not really anything specific to networking. I I can say something vague. Um so, agentic workloads you can look at like the specific uh you know, we're serving agents and so forth, but

also looking more broadly at like I need to run agentic workloads and what does that look like, right? Like we know that, you know, people run like a ton of agents, they have a ton of sandboxes, and it's like a very dynamic environment. And I think that will lead to some new requirements for basic networking like for example, like how to do policy and those things in

a very dynamic environment that might lead to some different conclusions uh for networking. So, yes, it absolutely there's a ton of stuff at the application layer, but like just looking at broadly speaking how the workload works, like you might have new requirements for the lower stacks. Um with that, I think we are out of time, but we are actually here to answer more questions, just not on

So, thank you for coming.