BoF | Beyond Nginx Ingress: Higress as the K8s Gateway for the AI Era

This talk focuses on Higress, a new ingress solution for Kubernetes that addresses the impending retirement of NGINX ingress. The speaker discusses the challenges faced with NGINX, such as traffic jitter for long-lived connections, uneven load balancing for gRPC, and scalability issues as configurations grow. Higress, built on Envoy, aims to provide zero downtime during configuration reloads and offers features like WASM plugin support, dynamic certificate management, and enhanced security measures. The speaker highlights how Higress can serve as both a traffic and microservice gateway, offering significant improvements in performance and ease of migration from traditional ingress setups. A case study demonstrates the effectiveness of Higress in real-world applications, particularly for AI traffic in organizations. Finally, the talk touches on the project’s recent addition to the CNCF sandbox and outlines its roadmap for future developments.

My name is Hushing. Uh thank you for also being here and uh I'm from Alibaba cloud and uh today I'm gonna talk about the beyond in ingress hyres as the kubernetes gateway for the AI era. Uh just a quick note that the hyres just joins the CNCF sandbox a couple of days before writing the keynote has it has been announced. So it's a new project in CSF

sandbox. Okay, let's let me first uh talk about the retirement of engines in ingress. So this is a very special moment that the you know all know that kubernetes the s network and security response community has announced that the retirement of engine ingress. uh so right at the end of this months it will be uh retired and uh so that means a couple of days later we

will get no further release no bug fix or security patches. So this is the fact but another fact is that there according to an estimation that 50% of the global cloud native uh environment still use engines uh some sort of engines infrastructure ingress infrastructure. So this is a crucial uh uh issue for us. We need to address that. And then there's another uh uh critical issue that

this engines allows us to write uh allocation based code snippets which allows to write uh any form of um engines config. Uh this is very flexible and powerful but uh since the retirement there were going to be very critical issues critical vulnerabilities. So that's one thing we need to address. So we actually we are using at Alibaba column we using used the engine based ingress before and

we have faced several challenges. I I'll list some of them. The first one is the traffic jitter for long lived connections. So as you will know that when we are updating engine synress the config uh every config will go through a new uh worker process to uh update the config but so this which will cause the longlived connection like gRPC web socket to terminate the connect connections

and reestablished so this uh is very uh painful for our users. So they were got suffered from the uh this kind of uh update issues. So another uh issue is the JRPC load balancing issues. Uh engine has uh designed for the shortlived connections and the for the uh long live multiplex especially multiplex uh http2 connections we suffer from um uneven load balancing issues. So even enginex has

provides support for the GPC load balancing. But if we are using a non connection and send request multiple uh multiplexing request over one single connections it will uh cause some um load balancing issues and resource saturations. The third one is the uh scalability issues. So when the number of the configuration has grown uh the the the time it will take for the reload will grow uh dramatically.

So that will um will not be a a good practice when we are scaling to thousands of uh maybe tens of thousands of configurations in one single kubernetes cluster. So so we have decide to like uh facing this issue. So we decide to uh re reimplement a new architecture of ingress. So I'll briefly introduce the brief big milestones before the hyres has been open sourced and the

so we are we we decide to uh embrace the and envoy uh architecture. So, Higris actually is built on ingress uh east and envoy but we firstly in 2020 we uh firstly uh launched the highress in our company and uh is quickly expanded to uh multiple services uh business units and has been proven uh stable and uh in 111 uh festival of Alibaba and uh which handles

a huge amount of data and it's proven production ready and in 2021 we have launched a service based on this product and launch the cloud service uh for the customers and the when com we combine the traffic gateway highress as a traffic gateway plus uh microser gateway so that's a twoin one uh gateway solutions which will uh cost us uh we have uh bring five 50% %

cost reduction from our side compared to the pre previous approach and from since the 2020 2022. So we have officially open sourced the highress and we as the user come in and uh we added more features like security and uh uh a lot of plugins and uh protocol translations and make the guide gateway as a traffic and microser and security threein one gateway. So that is a

brief uh evolution of this um project. So uh what is the architecture of the hygi is built on envoy and but uh um uh not only using the open source architecture but also we add some enhancement to them to so on the control plane side we add some features like uh dynamic certificate management that can automatically renew your certificate once the uh it is expired and we

also add the customer CDs to like do service discovery extra service discovery for especially microservices it's only not only discover uh services from kubernetes but also discover uh services uh based on the popular service registry open sourced like kongu knuckles and zookeeper and in uh in order to help the user migrate from engines ingress to highress. We also provide support for the duro stack support which support

both the engine singress annotations and then the gateway API and the new gateway API. So on the data plane side we added some uh more features like protocol translations we can convert HTTP request into an RPC uh popular framework called Apach double in China. And this is the RPC framework. We can add automatically protocol translation to that. We also bringing a new feature called WASAM plug-in which

is I will I will describe it in the next slides. And for the features that the W wasam cannot handle, we added the native support for the HTTP filter in Golan and is natively running in the envoy proxy and to support things like MCP or other long n long live connection issues and uh we also add add more dimensional uh telemetry data for uh what's the core

features of hygiore features here and the first is the uh zero downtime configuration reload compared to the enginex ingress thanks to the uh and data automatically handle that for us and the dimen the configurations can be automatically reloaded and will not cause any uh downtime for us. The second one is the wasam uh extensibility which I think is the coolest one of the coolest feature that Higress

has. And if we are going to add some extension to our envoy proxy, we either would like to add native filters to the envoy that requires uh a reload restart of the process. You need to compile the and and restart the gateway proxy. Or you could have a external process. You like you you you you like the you let the envoy proxy talk to the external process,

send a gRPC request and get get a response back. Uh but we think that the uh we we need a more flexible and more stable way to do that. So we add support for the WASAM plugins. So you you have you you can write easily write any wasam plugins and hot reload on the envoy proxy. So this makes us very easy to implement new features without restarting

the gateway. And then the the another feature that was brings us is the isolation. So uh any code running wasam sandbox cannot break the the the master the the the envoy proxy process. it will not crash and which is safe a safe protection for the custom code and we based on that we pro provide more kind of uh a lot of development uh language support for develop

uh was some plugins you can use golang rust C++ and JavaScript to write your own was some plugins and another thing that you need rest is the the the wasome plug-in that you built is distribute in an OCI compatible images which allows you to like use your existing docker registry to like to distribute your bot plugins. So hress supports the natively support over 100 plugins out of

box. So you can uh which I some of them will be introduced later but I cannot explain all. So the third one is the security uh features that Higris has natively integr integrated the W uh things of features like bot detection replay protection IP restriction and etc and as well as provide some uh rich authentication strategies like you can use GWT API key HMAC ODC or uh

quite a bunch of uh strategies that has been supported uh out of the box and for the guard reels u we we also provide things for guard reels like u API detection and etc. This is very useful for especially the AI traffic which is will be described later. So this is the core three main feature I think so that highress has the three main features. uh how

do we migrate from enginex ingress to highress? This is a very cool uh core issue that we want to address. When we use user has migrate from uh unjinx to highress we want user to uh sim simly migration without any code of modifications. So in order to do that we support the most common engine ingress annotations so that you can uh use the existing annotations but you

you don't need to do any change and the reason behind is the the the we have translate this kind of uh engine annotations into uh ECU and avoid proxies they can so that they can be configured uh to the invoid proxies. So this is not the whole picture I gonna I'm not going to go through all the features but it actually we support the most of the

common an scenarios like canary rewrite cur retry and read limiting or security. This the this these annotations are list are already supported but some of them may not be uh easily translated into the is still uh config configurations. So we add new uh support with the wasam plug-in or the powerful the powerful plug-in we can write same among some plugins to support the same features. So people

are also want to embrace the gateway API. So HRS the there's another feature that hress supports both the engine ingress annotations and the at the same time. So that separates the users concern. So you users can first without uh change any configurations because the enginex has been retired right then the user can uh gradually migrate their old configurations annotations into the new gateway APIs. So this when

when the both of the application uh configuration exists the gateway API configuration will take the precedence. So this allows us to uh easily migrate from the previously previous uh So there's the case study from COS. Uh Higress is not only uh adopted by Alibaba, but there's a COS is a cloud provider that provides uh nearly a service to nearly over uh 200,000 users. And before using hygi

they they suffered from issues like uh configuration reload peak time up to minutes when they are when they are trying to scale from over 10 10 uh 10,000 they they cannot scale anymore because of the the feature issues that they they found. So when they migrate from ingress to uh the peak the peak reload time has been dramatically reduced from 30 minutes to less than 5 seconds.

So that's is quite a improvement and also the per performance has been the memory consumption has been dropped uh about 10 times less than the the previous one. So this is a very uh good good case study for for the migrations and there's a actually a blog post talking about this migration on on their on their official uh website. So uh hress is not only used by

Alibaba but also used by many uh companies like DJI quo pay and other ones. So this is a production ready uh solutions for a lot of companies. So let's let me talk about the from the traditional API gateway to the AI native uh gateway. So as we know that there's a quite a parame shift from the uh gateway because of they are kind of traffic are very

different the the the traditional API gateway they are serving some shortlived HTTP request response while in in the AI traffic there are some long long runninging connections with server send events streams web socket they are quite different and the payload may not be quite the same because uh we we for AI traffic we need to understand the payload of the body to know that the uh which

am we we we need to uh route it to and we need to do a lot of stuffs uh by a by by being aware of the payload. So we need to do things like that and for the uh connection life cycle with the connection will for AI traffic that will leave for minutes to hours for the long running AI agent tasks. So HR to to address

these issues higher can be uh act as two kind of roles. First, haggress can be an LM gateway which can handle your traffic to the backend LM models. So, it provides features like model multimodel routing, multimodel fail over, token rate limiting and semantic caching and observabilities and uh also hress can act as a MCP gateway because AI agent use uh quite a lot of tools. they can

call to through MCP servers. So, hypers can act as a MCP gateway and the co the core feature that it can provide an automatically uh conversion transformation from the existing open APIs that you call into uh MCP server uh MCP servers. So that in that progress this is not you cannot you you you should you don't write need to write any kind of code. Okay so when

hress act as gateway there's some core features that I' like to address. First is the model fallback features. So uh you know that the AI agent will call quite a lot of models but when when the primary models have fails when we don't want the user uh feels about that so hygien models once the it reaches some limit it will automatically fall back to the the back

back the fallback models and this will uh make sure the users get uh FE feedback rather than uh sending some errors. The next is the token aware rate limiting. So the hygi can provide uh different uh to different consumers that cause the backend error models for different usually we divide it in in a company we usually divide it into a different business unit and each unit has

its token kas. So one of if one of the units has reached the kota hress will automatically apply rate limiting to that but it will not also will it will not affect the other business unit. So this is a very good use cases for the uh token aware rate limit. Third one is the semantic caching. So we don't want the similar questions sent to the back end

every time. So if the hypers can utilize the uh AI cache plugins to extract the semantic com uh semantic content from the user's query or request or response and stored in a radius cache and for the next query or uh next request coming if they are similar to the previous one and they can be use the cached result. this which can prevent the all every request will

be sent to the preventing to send to the back end and saving cost. So for observability um Haggress provide quite a bunch of metrics like TTFT, TPOT and reasoning tokens, cache token which is very important for the uh cost control and for debugging issues and it provides tracing features. It can send be sent to quite a lot of back end using a different uh uh for protocol

and it can also be configured to log any columns that you want dynamically. If you are for example you can add a session ID reasoning content and the tool calls to the logs so that you canize it later. All of them are are written uh in WASAM plugins and can be configured So for the API MCP gateway uh Higress provides the ability to automatically confer an existing

open API to an MCP server. So for the AI agent they don't want to don't don't want to uh look at a very complicated uh JSON that your open API returns but al uh but we highress can uh easily extract informations from the uh return the JSON and only have the extract the necessary informations to the AI agent and discard discard the unnecessary data to uh and

uh this is uh can be configured in a very simple YAML configuration and that will be automatically converted for you. And so in this approach we can uh dramatically save a lot bunch of the return data from uh sending it to LM and also we can reduce the hallucination for LM because uh the LM are not very you use uh quite useful to uh complicated JSON data.

If you have a nested JSON with a lot of reference to each other because uh normally the open API specs will allows us to do that. It's not very friendly for AI to read. So we can like we can convert automatically from a complicated JSON data into a markdown format which is very easy for AI to read. So this is only or RB can be configured uh

Okay. This is a case study for two companies from is a online travel provider in China and it utilized highress uh as this AI gateway to achieve uh cross business unit token allocations and zero downtime failures. and Ant group is a a payment online payment service in China and uh it's sofa AI gateway is also powered by so Higress to reduce the integration complexity and significantly speed

up the development of the AI agent. So these are these are two case studies from these companies and then it can also be uh found a lot of quite of more companies can be found in our website. So okay this is just uh a few hours ago the CNCF blog post has posted the new blog post that officially joins CNCF sandbox. So if you are interested in

this project, you can check out the website and the GitHub and the soon will be transfer transferred to an CNCF repository and if you are interested in contributing, you're more than welcome to contribute. Yeah. Finally, this is the gate road map of this project. Currently Higress is the serves as a ingress gateway and it it will continues working on the support of the uh recent gate AP

gateway API inference extension and actually there is a initial support on that but it's not enabled by default. you can try it out and um we are working on supporting more uh awesome plugins to the gateway API infer inference extensions. So uh from next we want to address the real time application uh support for the real time open API real open AI realtime APIs for the web

RTC support. So, Higress is a a vital mature and production ready battle tested project and uh you can definitely you should check it out and uh with that and this is my talk and thank you for joining. Yeah, that's all.