Cloud Native Theater | EnvoyCon: The Future of AI Traffic: What's New in Envoy AI Tra... Xiaolin Lin

This talk focuses on the future of AI traffic management through the development of an AI gateway by the speaker's team at Boomber. The speaker discusses the specific challenges associated with traditional AI API gateways, including the need for special handling of traffic and the management of costs related to token usage. They emphasize the importance of integrating multiple AI service providers and ensuring operational ease for platform owners. The speaker highlights key features of the AI gateway, such as unified API layers, automatic failover mechanisms, and a policy framework for managing rate limiting across different tenants. They also touch upon advancements in cost management strategies, including quota-aware routing and batch processing capabilities for achieving operational efficiency and minimizing expenses. Furthermore, the speaker details the community collaboration behind the project and its ongoing enhancements to support various cloud APIs and improve observability.

Good afternoon everyone. Thank you for joining the talk. Uh my name is Shins for Boomber. Uh we are the I'm from the AI traffic team at Boomber. We manage all the infants. Uh and then the GI traffic at Boomer. Um so I'm going to talk about what's the future look like of AI traffic. Uh what's the news coming out in the on AI gateway. So before I

get into there, so I want to highlight what kind of problem we trying to solve uh with this on AI gateway. uh previous two talk speakers talk about the onoy envoy gateway. So we pretty much kind of on top of that and to dealing with the AI uh specific uh processing. So uh I want to highlight that the traditional AI API gateways for sure because that the

error traffic requires special handling like you extract the model names from the uh payload instead of like looking at a header and also organizations looking to like uh control the cost uh about the token usage. Uh and also like sometimes the model fade over uh doesn't work that way because like the a lot of model is not like cloud provider have their issues and then we also

you in order to provide the enterprise grade we need to do a lot of um fail over and then smart failover as well. Um and also um as I mentioned that we need to have like multi-provider integration. Uh different provider have their own specific API uh the way to authenticate and then the way to handle the uh specific of the the traffic as well. Uh like also

the capacity type. Um and the other uh challenge or objective is that we want to make that uh operational uh easy for the platform owners and also building the uh governance into that to monitor this observability and token usage cost uh as well and uh the other part is that um when we building the onway AI gateway we try to look out outside is there existing solutions

that is open and collaborate we didn't realize there's a one and then we start to talking with the uh on community and then we start building this uh gateway uh with the different company and different developer and contributors from all over the world. Um uh so on gateway is essentially is a uh a project on built on top of and on gateway uh from to handling the

AI specific uh workloads. So the key object uh I think I'm kind of a luing into the array uh we try to provide a unifi uh API layer uh to for routing and manage the uh error and AI traffic and also supporting the uh automatic failover mechanism to share that the reliability uh as well as like end to end security like authentications authorizations uh etc. We've also

uh implement the uh policy framework to support that uh different rate limitings uh for the your different tenants uh use case and we aiming to foster open source community to address the ji uh traffic and the quality service. So we are not kind of looking to the vendor logging or sing single or two vendors uh solutions. Um just a little bit overview about how the project uh

progress uh in the late 2024 uh we start the proposal meet the community uh community and then uh by the uh February 2025 we do the initial release with all the core features we have like uh unifi APIs uh upstream au authentications with different provider like azures uh AWS uh and also we implement the token based raining which is like different from the traditional request based uh

rainings uh because that you were looking into the both uh input path and output path. So input you have a a prom token output you have a different kind of tokens coming out like reasoning token eter and thinking and also we build in the traffic managements like uh uh rating as like part of it and uh by the end of uh uh 2025 we make it uh

production ready a lot of company using the AI gateway as a uh enterprise solutions uh we also extend the provider support as well like including the GCP uh and the uh anthrop uh APIs and adding the observability and as well as the MCP support. Uh and uh early this year uh well now uh the main uh the thing is that we focus on the cost managements. We

will going to talk going a little bit deep into the different uh like quota routing and then quota aware routings and prong cachings and batch processing. Those are the uh kind of a pillars to help you to uh further optimize your cost and then be efficiency. Um and uh I going to briefly go through like what we left in the 25. This is a thousand foot view

uh in terms of a control plane and the data planes. Essentially you have a we have like a two uh controller AI gateway controller uh on gateway controller uh AI gateway controller basically do the uh fine-tune the configuration through the SDX and then you talk to the data planes on gateways um AI gateways as processing and then also dealing with the communicate with the envoy and as

well as the meaning um the current adopters uh there's a few companies out there like LM uh softmap uh and then the NPR boomber uh ly corporations and nutanics uh tensson cloud and then tetrix uh being using the in the productions um and we have this kind of core features that unifi API I'm going to be into the uh what does it mean in terms of unifi

API so we using the open uh AI uh format as taking as a input and then internally we transform uh depends on the cloud uh depends on the model we will transfer uh transform to the specific provider specific API format and on the way back uh we also convert it back to open AI uh compatible format as well. Uh but sometimes that not necessarily fitting to all

the provider use case. So we provide a flexibility to uh to have a cloud provider specific field. All right as well uh we also make these uh uh open uh response uh recently I think the openi rolled out the response API and then later they kind of confound the open response. So we also be uh NY Gway has this uh uh open response compatible API. Um we

simplify the authentications including the API keys uh open ID uh connect uh Microsoft Azure ID and then also GCP workload federations uh and the token based limitings uh cross provider and region for is important uh because there often times I think imagine you have like model called uh GBT OS like 120BS and you want to hold you want to be more like reliable so you will first

choose like open AI but Open AI can be down sometime you have a different format mechanism. Uh on a G allow you to set this for fallback and then stitching together. Uh you can forat from the open AI Azure and all to your cell host on GPU as well. Um and same similarly uh you could have like a cloud model uh which is running on the GCP

vertex as well as you another one on the AWS bar. So they provide more resilience to your uh to a client or your internal use case. Um and uh the last but not the least is the uh observability which is a key factor to drive the business decisions uh on how much token being used per request per models per provider and per application and whatnot. uh and

tracing the spans and through the uh streaming and non-streaming case because like uh streaming case you want to look into like intra token latency to improve your uh user experience and uh and then non-streaming cases is more kind of you can think about like traditional request and response path. Uh yeah, we also need to expect like how that performs uh when we talking to the uh cloud

providers and with the inference uh to answer your client's question or or maybe your uh your team and then your application teams and then your managers question to justify and explain how um how it works and also show the proof. Um with that we also very exciting to share that uh our API CR as uh V1 beta beta uh stable uh many companies shower uh previously is

running this in production uh workloads. Uh so if you're looking for enterprise level uh production solutions on Gway should be one of uh top solution um or choice. Uh let me move into a little bit into like the uh 2026 uh which is like uh cost optimization which is the things we have. I sure there's a lot of uh companies who look into that. Um first one

is the quota aware routing. Uh so in the quota aware routing uh in the context of a quota there's often times in the cloud provider side have like a provision throughput which is handling the predictable uh you reserve the capacity and then most of the time you have a guarantee SLA you also can putting your predictable workload on this capacity. Uh also they offer the ondemand as

a kind of pay as you go. Um so this more kind of a less uh well definitely not a SA like PT uh provision through book uh guarantee but uh you have like a spy traffic you can can predict sometimes you can route into there uh and so the way we do that we're going to do the uh kod for uh routing to like highest priority back

end with available kod if all the provision throughput back end is sitting uh we will route into the uh od uh depends on your use case and how do you want to uh manage the fallback mechanism. So uh we are going to adding the koda policy. So to tracking the back end uh which is like you can imagine the cloud prov cloud provider size on your on

primer uh koda policy in terms of uh capacity type or size. Uh this is kind of a brief uh CD YAML file. Essentially on the right you'll see the koda policies and you can put in the per models and then you have different type of uh in this case is a cloud uh cloud sonet 4 uh you have like reserve tier and then uh also as a

default tier you can apply that uh policy on top of a AI service back end which is actually the uh last two steps to talking to the communicate with the cloud provider for this specific models. Um and the way how it work is that uh this is simplified version. uh basically the request coming in the AI gateway external pro we're going to extract the model uh and

then when we check the per backend quota and then populate the quota mode uh validation informations that in the dynamic metadata uh and then let pass through to the uh ono routers and then excluding the the back end that fail the quota check per metadata values and then routing to highest priority uh back end with the available quota Uh another big uh another area we can optimize

as well is in the context caching. Uh I think a lot of company offering like implicit uh contact caching. They basically cache for your prom already. And then some of other basic contact caching like Gemini they offering the prewarming offering the API allow you to prewarming the C contacts uh uh ahead of time so that you can reduce your cost uh effectively. So a little bit into

that is that the client sending the check completion request with the cache control which is like we follow the enthropy sematic uh and uh the oni gateway going to post that to the we're going to adding the endpoint to resolve this cache uh depends on what cache uh provider you have we will talk that and then we cach the content and the cache service returning back the

location reference in the Gemini cases uh and then we'll be talking to the actual uh AI's uh inference and then with the cache content so that in the vertex size then you already know where to look at your uh contents uh prompt contents etc and then response back will naturally follow back and then with in the open air format um and uh coming to the batch processings

often times the real time a lot of use case can be like real time and also not like time sensitive and cloud provider providing like uh huge discounts on like 50% on the batch processing uh depends on your workload right some for example you want to look into the evaluation of your jobs or your uh particular benchmarking something you bash will be your ideal uh appro approach

to handle that you don't need to imping the uh actual production workload so you can wait a bit and then to handle that so you handle the large scale workloads and also the higher uh thooks uh offering the cost discount the main API is the uh uh this is for the uh open uh API specific API. Uh we have we're going to be implement the v1 files

uh and then the bash API. Uh we want files basically allow you to pre-upload your files and then you calling the batch to uh request the cloud provider to processing your files and then also to uh fetching your batch and then the status uh and including the son of aer uh uh usage as well. Um shifting a little bit uh we will extending we'll continue developing the

uh on AI gateway on the API part as well. Uh one thing is the response API uh what is matter because uh it providing stable uh APIs and then design for the agentic workflow. So in the agentive workflow you don't need to like on the client side you don't need to just like keeping your state appending all your user message or system message or assistant message as

well on your client side like adding the burden on the client side. So the server side already handle that you can just like reference to the previous conversation and uh it's perfect for the agentic workflow and then uh reduce the amount of engineering work on the client size and it providing the context managements and then we make it as a open uh response API compatible and uh

in the uh rare use case retrieval uh generations we providing the two APIs uh we want embeddings and reams. Uh this is like embedding tasks such as like a semantic search and classification and clustering uh providing more accurate context aware results uh than a keyword based uh approach. Uh building this uh ra system is common use case in the AI product. uh embedding played a key role

and in a significant enhancing the uh model outputs and with a improved factual accuracy and coherence uh as well as like a contextual richness uh multimodal models and then we offering the two uh key APIs the test to B speech and then also the image generation think about that in your uh when you generate the contents on your whatever client mobile so uh web so you can

just like integrate using this two API using the test to speech API converting to the audios and then to fitting your needs. Um I would like to highlight the performance uh in terms of onward AI gateway. We measure that in the on the data plane uh levels that we seen that is a millisecond levels performance uh as well as in the control plane we can easily scale

into like 2,00 uh AI gateway routes and with a zero routing failures. Uh this is also uh just like require minimum resource uh increase from like a default gpc message size to 4 megabyte to the 25 megabytes. Um if you're interesting to uh more into the gateway uh I would recommend to join our community. Uh this is our website and then we have a weekly meetup uh

as well and uh feel free to uh join and then raise any issues or uh contribute as well. Uh with that, uh thank you for attending for listening. >> All right, now it's time for Q&A. Any questions for Shaoling? >> Hello. Uh so I have a question not necessarily related to AI but uh is listener sets uh planned for release soon in a in envoy gateway >>

can you repeat again sorry >> listener sets >> listen in the air gway >> yeah in gway >> I'm not aware of that but I I think we can look into that uh so do you have a specific uh use case for >> yeah we have a specific use case so currently we have engineix controller in each name space so we have strict big network uh requirements

will be a bank >> and uh we want to replace it with with envoy gateway but we'll need the listener set for that. I was wondering when it will be released. >> Oh uh yeah I think we have a maintainer in here uh and them and errands and we can look into that. Uh we would like to yeah uh perfect feel free to like we can yeah

okay the issue and then we will walk into that. Okay. >> Awesome. Thank you.