AI'm at the Gate! Introducing the AI Gateway Working G... Morgan Foster, Nir Rozenbaum & Shachar Tal

This talk introduces a new working group focused on developing an AI gateway, which is designed to manage AI traffic more intelligently compared to traditional gateways. The speaker elaborates on the capabilities of the AI gateway, explaining its role in routing requests based on payload data, including model specifications and prompts, rather than just headers. Key features include support for policies like token-based rate limiting and semantic routing. The working group aims to establish standard APIs for handling AI workloads in Kubernetes, emphasizing the need for a declarative architecture that can handle complex payload processing. Various examples illustrate the proposed request and response flow, highlighting the necessity for an application layer egress gateway that responds to specific AI use cases. The session concludes with a call for community involvement in shaping the architecture and features of the AI gateway.

So, welcome everybody to our talk. In this talk, we are introducing a new working group, a new ish working group, uh where we are trying to build an AI gateway. It's a very provocative name in these days when everything's called AI something. So, what the heck is an AI gateway? And today we're going to tell you specifically what we mean by that and how we are trying

to achieve supporting those use cases. But first, you should tell everybody who you are. Yes, yes. I'm getting to it. Okay. Uh so, uh first of all, I want to say we were supposed to have three speakers with us. Uh one of our co-chairs, Nir Rosenbaum, and also one of our uh contributors, Shahar Tal. Uh they are stuck uh in the Middle East. They cannot travel because

of the conflict that's happening there. Um it's uh really >> It's very annoying and sad. Yes. Uh and so, um instead, they provided us with videos. Uh and I will turn off my Slack at some point. They provided us with videos uh with their sections of the talk, which is great. Nir is um amazing in every possible way. If I told you how amazing he was, you

would it would sound like cliches. So, instead, I'll say he worked on the gateway inference extension heavily, and he knows all about optimizing inference at scale, which is a very big-brained thing to know about. And Shahar is a distinguished engineer. I think they should give you a monocle when you become a distinguished engineer at Palo Alto Networks. It's a little difficult to wear a monocle if you

already have glasses, though. That might be why they >> Uh you glue them on, maybe. Ew. >> I don't know. We'll solve this problem. Uh We'll ask an AI about it, at least. And so, uh Shahar is uh has authored uh a great deal of our payload processing proposal, which he'll talk about. He's a very precise thinker, a lovely person to talk to. Every time I talk

to Shahar, I feel a little smarter than before I talked to him. And of course, we have Flynn here. Uh Uh I was going to say you're kind of stuck with me because Shahar and Nir couldn't be here. Um I am also one of the chairs of the AI AI gateway working group. There are ways I think I got invited to do that because I have not

drunk the AI Kool-Aid as much as some other people have. So it's very, very interesting. And of course we have Morgan, who is also our newest chair, in fact. Hello. How long have you? Couple months, maybe? Uh yeah, I started working on the group I think in something like October of last year. Uh so I've I've >> time the group got formed. Yes. And I worked on

a large portion of our egress proposal, which we'll talk about. Uh and so yes, I work at Red Hat. Work on AI stuff, like so many people, blah, blah, blah. But let's get on to the good stuff. So first we're going to have Nir Rosenbaum uh describing to us what is an AI gateway in his own words. Thanks for the intro, Morgan. So hi, everyone. I'm Nir.

And today I'm going to talk about the question of what working group AI gateway is. More specifically, we're going to touch the mission and the scope of this working group, including some typical use cases that we aim to solve by introducing new primitives that could be used to achieve our goals. Before diving into what working group AI gateway is, maybe we should start with a simpler what

is an AI gateway in this context. So when you think about traditional gateways, we usually think about being able to route requests based on headers or path matching. But in AI inference use cases, this is not always the case as the request payload contains information that might actually affect the routing decision, like the model, the prompt, or other fields. In our context, AI gateway refers to a

traditional gateway with extended capabilities for AI use cases, such as being able to apply policies and routing decisions based on the payload for both the request and the response. Some examples for use cases could be token-based rate limiting, fine-grained access control, payload inspection, which enables intelligent decisions such as semantic caching, semantic routing, or guardrails, and more AI-specific protocols and routing patterns. I also want to touch the

point of what is not within the scope of this working So, we are not building new gateways implementation, and we are also not defining the model serving behavior. So, what is within the scope of this working group? We are aiming to standardize AI traffic management at the gateway layer to support all those use cases. More specifically, we aim to develop new declarative APIs and standards for how

to handle AI workload networking in Kubernetes. We also want to make sure the proposed architecture is composable, pluggable, customizable, and allows ordered execution of payload processing pipelines. Not less important, we want to leverage existing standards where possible and to collaborate with the community to build consensus around best practices for AI infrastructure. To emphasize some of those use cases I've been talking about, I'd like to walk you

through an example that might represent a typical inference request flow. In this diagram, we see an example of a user who is sending an OpenAI chat completions request to the AI gateway. The request might need to go first through authentication and authorization to make sure the user is authenticated and that he is allowed to access the model that was specified in the request payload. Then, we also

want to make sure the user has quota to perform his inference request based on token-based rate limiting. Assuming the user is authorized and he has quota, the next stage of the request flow could be triggering multiple payload processing, a new concept we're working on in our working group that essentially allows to do header or body mutations to the request and response, as well as to return immediate

responses. Some examples for such payload processors could be guardrails, semantic hashing, semantic routing, API translation, API key injection, and more. Once all payload processors have been invoked successfully, the request comes back to the gateway, which can then decide whether the request should be sent to an in-cluster endpoint or to an out-of-cluster model for serving. When thinking about the response flow, it looks very similar to this diagram

just with the opposite direction. We need to get the response back from the serving model, go through the response payload processors like guardrails, update the token-based rate limit based on the response usage, and then back to the user. In the next slides, we'll discuss more about payload processing and external model serving, and walk you through our active proposals of the working group. Okay, fantastic. Didn't I say

Near was great? He's so He's got it all in his head. So, basically what we're saying is we need an application layer egress gateway that supports policies based on the content of a request. Okay, so next we're going to talk about Shahar is going to talk about what it looks like to get access to the contents of that request and make policy around it because it has

some interesting problems. Interesting. Interesting. While she's queuing up that video, payload processing is processing itself is not a new idea. We used to do a lot of this decades ago. Standardizing it in Kubernetes is a new idea because we haven't needed to do it for a long time. Now we do and yeah, there are a lot of really interesting things about it. Indeed. So, I'll let Shahar

take it away. Hey, everyone. My name is Shahar. Today I wanted to introduce payload processing for AI gateway. As Near mentioned, Gateway API currently operates on request metadata, headers, and URLs. It does not operate on the body. For growing class of workloads, the body is where the signal is. Consider an AI inference request. The headers carry a content type, maybe an API key. The body carries the

prompt, the model parameters, the context. Effectively, everything that determines what the request actually means. This proposal defines a standard declarative mechanism for adding processing steps that operate on the full HTTP payload, headers and body, request and responses within the Gateway API resource model. Moving on. Here are the key properties we're targeting for this design. Declarative. Payload processors are configured as part of our definitions, just like filters

today. There is no imperative clue. You declare what processing you want, and the gateway makes it happen. Ordered. Multiple processors execute in configured order. Importantly, processors that only validate, inspecting the payload without modifying it, can execute concurrently, reducing added latency. Mutating processors, on the other hand, must run in sequence, since each mutation changes the input to the next processor. Scalable. A route might need many processors. The

API needs to support potentially large numbers of payload processors without becoming unwieldy. Routing capable. Some processors may trigger routing decisions. A semantic cache processor might short-circuit the entire request and return a cached response. A semantic router might redirect to a different back end. These routing decisions are conditional. They only fire when the processor determines they should. Failure modes. Each processor is configured with a failure mode. If

your PII scanner is down, do you fail open or fail closed? How long do you want to wait for a processor verdict? That's a per processor policy decisions the operator must be able to express, and the API needs to support that. Bidirectional. Processors operate on requests, responses, or both. Scanning a prompt pre-inference, and scanning a response pre-delivery are equally valid and often both required. And finally, rejection.

Processors can reject requests and responses, blocking a disallowed prompt on the request side, or blocking PII leakage on the response side. This is a first-class capability. So, without further ado, let's look at an example. As a first step, we're proposing this to take on the form of a filter type within HTTP route tools. The inline form is shown on the slide. I would say it follows a

standard gateway API pattern. A filter on a rule pointing at yet another backend ref. And please note, this is where the processor is hosted, which is distinct from the upstream server backend reference. This filter has a timeout value, a failure mode, and optional context passed to the processor. For use across multiple routes, we also propose a reusable form where the processor chain is listed as part of

a payload processing pipeline CRD, not shown here, which is then referenced by one or more HTTP route or gateways. The processor can run on the same cluster as a gateway, of course, or it can be a service hosted outside the cluster. Moving on to open questions. I want to stress that this is a work in progress, and there are design decisions where we want community input. Protocol.

Processors need a wire protocol for communication with the gateway, receiving payload chunks, signaling approval or rejection, requesting mutations, etc. We're looking at GRPC as a transport to carry this, but specifics are still open. Interactions with existing filters. There's overlap between filters and payload processors. Do processors run before or after filters? Can they eventually subsume some filter functionality? Do we keep them cleanly separated? Payload processors as a

concept is ostensibly a generalized form that could implement every single filter that is currently supported by Gateway API, but it doesn't mean we should drop native filter support. There are real architectural trade-offs here. Gateway level versus route level We want both. When both exist on a request path, how do we determine execution order and priority? Processing loops. If a mutating processor changes the payload, and we reevaluate

out matching against the new content, do we even allow that? Is there a use case that requires it, or can we prohibit it without losing expressiveness? These are questions that benefit from perspectives across implementers, platform operators, and security practitioners. We're at the stage where community input has the most leverage. Thank you. All right. Nice, easy, open questions. If anybody here has used XProc, you'll probably be familiar

with some of these problems. Failure semantics, yes, but retries are very important. When we do a callout, if you're familiar with XProc, and then a failure happens, during our retry, do we do the callout again? I don't know. This is a problem that the end end point picker and the Gateway inference extension has to deal with. Even more fun, it's probably a problem where the right answer

depends on the context and depends on your use case. So, we probably don't get to just dictate one or the other. Let's get back to these slides. boy, oh boy, we've gone through a lot of We're almost to this. So, now we're to my section of the talk. We've talked about what we're trying to solve, which is the problem of L7 egress with body processing. We've talked

about the ways we're thinking about allowing access to the body for uh for our policies. And now I want to talk to you a little bit about that L7 egress gateway part. Why do we need an L7 egress gateway? Well, because in Gateway API today, we have built primitives around supporting ingress. And all of these primitives are also built from the perspective of a producer and not

a consumer. From a server and not a client. And this causes a lot of problems. So, for example, today in Gateway API, if you want to do egress with no no outside resources, what do you do? You make a service. You make an external name. External name maps a CNAME to your You want to add TLS to it, you make a back end TLS policy. All of

that is doing what? It's you controlling a service that is pretending to be the thing that you're connecting That's a little weird. Because that's a little weird, it causes lots of tricky problems around DNS, around TLS policy. And to get around these tricky problems, we have vendors who implement their own resources to describe egress. Notably, service entry by Istio. Uh but others as well. It's okay for

you to bash on Istio, but I have to confess that Linkerd did their own resource, too. Yes, Linkerd also did >> We have a thing called an egress network that hooks into all this, so. Yes, indeed. Um there's multiple vendor imp- implementations. Istio and Linkerd are um among the most used. This is a problem. We have fragmentation across vendors trying to solve a problem that is a

signal to us that we need new primitives in Gateway API to support these use cases. And there's another part to this problem when I'm talking about producer versus consumer focused primitives, that problem is one of the resolution that our policies work at. And so I also want to talk about how we can offer a sort of resolution that will be very useful for our application layer egress

policies. Quick question for the audience. Raise your hand if you know what we mean when we say producer and consumer in this context. Yeah, okay, that's about what I thought. Um, these are terms of art within Gateway API. the producer for a given workload resource service, whatever, is the entity that created that. The consumer is the entity that is making use of it. To put that into

more specific contexts, if you're an application developer and you have added a deployment and a service to a Kubernetes cluster, you are the producer for that particular workload and service. A consumer would be anybody who calls you. Gateway API runs into some really interesting things with this distinction because there are cases where you really want the consumer to be able to control something, but you also probably

really want the producers to be able to control it as A good example there might be a timeout where a producer might want to specify a default timeout, but a consumer might decide that the default timeout is too long and want to override it with something shorter. We have to support both of those things. This is a complex problem. It only gets worse when we're talking about

some of these AI loads where not only do we have to support both producers and consumers, but we also have to recognize the fact that the infrastructure cannot make decisions based solely on the headers. We have to look into the body and do all this crazy really quickly as well, Gateway API, one of the common questions we get is along the lines of, "Why don't you just

use filters for all of this stuff? And the answer that is is that in Gateway API, you can't talk about ordering on So, that's a problem from payload processing's point of view. Right, exactly. And like Shahar mentioned, if there's mutation, you need to be careful about the ordering of your filters. For example, in semantic routing, we might choose our model in one filter, and then we need

to add a something like a jailbreak detection guardrail that depends on the model, because jailbreak prompts are different between model families and models. And so, you have to define that sequence. Moving on here, I also want to talk about the class of policies where per route is too fine-grained and per gateway is too coarse-grained. And what do I mean by that? Let's imagine that we want to

support credential injection for OpenAI's API, okay? Do we want to define that credential injection across lots of routes? Yes. Yes, we do. Sometimes. Sometimes, yes. So, let's just imagine the trivial case where we make an HTTP route and we say, "I'm going to make the path forward slash." And now, everything is going to hit that route, and I'm going to trigger my policies so I can put

uh credential injection there, right? Well, then we make a new route to represent a specific a more specific path in the API, and that overrides that default that default matching. And so, we have to copy our policy over. But, we have a solution for this. We might say policy attachments. Maybe we'll attach a policy to the gateway itself. But now, we are applying that policy to too

many routes. We want credential injection only for that particular back end, and only for some potential subset, maybe a large subset of routes to that back end. And at every level you start talking about this, you get to ask the question of maybe it's just maybe it's more than one HTTP route that wants to override things. Maybe it's more than one for the default. Maybe the gateway

has things attached to it that have nothing to do with AI workloads at all. And so it's not appropriate to do injection there. And these are fun questions because you need the specificity, but you also would like to arrange it so that people don't have to constantly be copying and pasting YAML all over the place. Yep. So TLDR, there's no good way to have default policy for

a back end today unless you use something that's vendor specific. All right. So what's our solution to We want a way of representing a back end. That something where we can have lots of different routes that can target that and that we can hang some policy off of. So that becomes our default policy when you have a more specific it it's policy can operate in conjunction with

this policy. So it becomes a sort of default. So for stuff like circuit breakers, for stuff like credential injection, this is ideal. another thing that we can do here is we in our examples so far inline TLS policy. And this is also useful because we're thinking of this back end resource as being from the perspective of the consumer, the way that Flynn described So this allows us

to let me put it this way. Whenever you configure TLS policy for what's called a synthetic service, that's that service I told you about that's pretending to be an external back end, your TLS policy is not saying this is how I want to connect to you. It's saying this is how I want things to connect to me, which is wrong. That causes some tricky problems sometimes um

around uh things like host name and SNI being conflated, which we won't go into today, but needless to say, this solves problems for defining our TLS policy from that consumer perspective, and it decreases the number of resources we have to carry around. Now, we don't have to have back end TLS policy and external name and a service name. And likewise, you can attach policy here, like I

mentioned. So, something like credential injection. Also, I want to to note that the agentic networking subproject also has their own back end resource, and they are using their backs Oh, go ahead. Go ahead. Their back end resource has another utility that I didn't mention yet, which having a representation of back end as a concept allows us to produce back ends for internal and external back ends. Why

is this useful? Let's say you have your own pool of inference. Okay? And so, you run out of capacity in your your own, you know, self-hosted large language models, for example, and then you want whatever depends on those inference endpoints to fall back to an external inference provider. You might want a single surface area to work from where you can attach the same sorts of policies to

both that internal and that external We have open questions as well. I showed you that we have inline TLS policy, but there have been proposals to take back end TLS policy and provide consumer-focused semantics for it. Uh for so that implementers can can sort of flip the meaning of things in certain scenarios to get around some of those edge cases I mentioned. This is the interesting characteristic

that it would be giving a single Gateway API resource multiple wildly different semantics. Correct. >> Which is a big part of the reason why we are leaning towards not doing that. Yes, it creates ambiguity. Um it's it creates context dependence whenever we are configuring our resource. Is it What are we trying to do with this? And we have to signal that somehow, and if we don't, it

can cause weird bugs. Scoping for our backends, we're leaning towards namespace scoping, but maybe you want a backend that is global. You want all of the services in all of your namespaces to be able to reuse policies or TLS configuration. That's something we're exploring. I know uh Linkerd has an interesting solution to that. Um that I've tried to borrow from. And then finally, I mentioned how we

could use backend to represent internal and external backends. Uh but how do we do that? You could uh reuse service for those internal backends, but then you run around run back around into the same sorts of problems I was mentioning before. Service has baggage. It wants to help you uh it wants to help you reach your service via DNS, for example. And it turns out that gets

in our way in some use cases, and in fact, uh in the Gateway Ingress extension, they created the Ingress pool, which solves that problem by essentially ripping that stuff out from a service. And they actually created the Ingress pool so that they would not have to deal with some of the stuff that service does helpfully for you. Uh which is of course helpful until you don't want

it to do that, and then it is not helpful. Exactly. Uh because service was built to host a service, not to do the things we're trying to do now. And so, we're looking at um further creating a new resource called endpoint selector, which allows us to just interact with endpoint slices directly. So, that we can side-step all of that baggage that comes with service, which is really

helpful when you're standing up an API server that you manage, but is not helpful at all when represent some sort of internal resource like like uh like a a an LLM that you've stood up. So, finally, we're talking about whether or not we should reuse gateway or create a new egress gateway resource. Flynn already touched on the idea of uh context dependence whenever we're using a resource.

Gateway is really built for the ingress use case. If we reuse it for egress, then we are relying on the people who use it to keep in mind always whether it's doing egress or ingress. I have opinions on that, which are probably already evident. Uh before we jump into this, since it doesn't look like you have stuff talking about this, some of you may also be looking

at the back-end resource and going, "Oh, this seems like they're just sort of grabbing things out of service and trying to do things so that we can add our own policies without having to mess with the service resource and all that." Yes, yes, it does because one of the big problems we keep running into is not only does service have all this extra baggage of cramming together

what, six or more different functions in a way that you can't decompose, but we also can't change service. A huge part of the reason why Gateway API has get 713 policy attachment at all is because we cannot modify service to add things to it on any realistic time frame. we've been kind of dancing around this idea of maybe there is a path towards some of this decomposition

by introducing new new resources. And I think we're pretty much finally at the point of actually trying the experiment and seeing how far we get. So, everybody cross your fingers. It'll be an interesting ride. And uh yeah, community feedback community feedback is really helpful with stuff like this. Yep. If you remember the the uh Ingress resource, in fact, Gateway API itself was meant to resolve some of

the problems you run into when you try to cram too much into a single resource. Gateway API gave us separation of concerns. If you remember in in Ingress, we were using what? Annotations to And everybody loves annotations. >> Yes, using annotations to try to handle all these edge cases, which really um can be handled much more cleanly when you just create multiple resources that each have their

own domain to operate in. >> Yeah. This isn't really a Gateway API talk, but I will point out that one of the other challenges with Ingress was because the timeline for making changes to core Kubernetes is so long, the Ingress folks pretty much had to move forward with an API without the opportunity for community feedback. And we would really like to not make that mistake again. If

you look at the timestamps here, this is from last night, very late. You can see I was a up late in local time. >> You you don't have to admit that. It's fine. No, it is a it's a real demo running on my laptop, but I captured it all into a fake shell script. Um so, anyways, here you can see uh a prototype of the Egress Gateway

resource I mentioned. It's just a gateway. Under the hood, it's literally just that Gateway resource that I mentioned. Then you can see a back end that's been defined. This is for HTTP bin. You can see we've defined it by its destination, its host name. Then you can see we have inline TLS policy. Lovely. So, now how do we get this connected to our gateway? We make an

HTTP route. Remember, the cool thing here is we can have lots of HTTP routes targeting this back end. We have this HTTP route. Whoop, let's go we can see its parent is the gateway and it's targeting our back end. And then we can run curl from our cluster. This is an explicit proxy. We are not trying to intercept traffic. We're not trying to implement a transparent proxy,

so we have set an HTTP proxy environment variable. Uh and sure enough, we also do TLS origination in this demo. So, we're sending an HTTP request, but then the gateway itself initiates an HTTPS connection for us. So, you can see that And voila, there you have it. A back end resource that you can target with lots of routes, that you can do inline TLS policy for, and

as one more knock-on effect, in case just in case anybody had dealt with it, it's a pain in the tuchus to export TLS policy across name spaces right now. And so, I want to show you that it's a little bit simpler to export TLS policy using this back end with its inline TLS policy across name spaces. So, I'm creating some name spaces, a couple of reference grants

here. multiple name spaces, I'm reusing my TLS Which, if you've ever dealt with dealt with that problem, is kind of a blessing. we'll get to our future plans. Overtime. So, if you want to talk to us about future plans, you should come up and talk to us about future plans, but they all boil down to, for the love of all that is holy, get involved, talk to

us. We wants to know what you have to have to say about this before we go ahead and build everything without listening to what you have to say. Thank you very much. Thank you.