Project Lightning Talk: Evolving KServe: The Unified Model Inference Platform For Both... Yuan Tang

This talk covers KServe, a scalable model inference platform designed for both predictive and generative AI workloads on Kubernetes. The speaker, Yuan Tang, outlines KServe's support for diverse serving runtimes, including VOM, XGBoost, and scikit-learn, as well as its integration with cloud-native technologies like Knative and KEDA for auto-scaling. KServe also provides enhanced features such as metric-based auto scaling tailored for large language models, prompt caching to optimize throughput, and support for traffic management and logging. As an incubating project within the CNCF, KServe has gained significant community traction with numerous contributors and maintainers offering support for users and developers alike.

Hello everyone. Can you hear me okay? Okay. Cool. Before I start, I have one simple question. How many of you are running model inference in production today? What about predict predictive AI workloads? Okay, large language models? Okay. Interesting to know. So Okay, my name is Yuan Tang. I'm a senior principal software engineer at Red Hat. I'm also a project lead for KServe. So today I'm going to

talk about KServe. So what is KServe? KServe is a standardized, distributed, and scalable model inference platform on Kubernetes for both predictive AI and generative AI. It supports a wide range of serving runtimes such as VOM for generative AI and XGBoost, scikit-learn for predictive It also supports um integrates well with generative AI solutions such as RMD, Envoy AI Gateway, and RMCache. It's also integrated with cloud-native technologies such

as Knative and KEDA for auto scaling. And it supports gateway API from Kubernetes so that it's compatible with Istio and Envoy. And everything's based on Kubernetes so you can run it wherever Kubernetes can be run. So we joined the CNCF as an incubating project in September last year and it's amazing to see so many community adopters and organizations adopting and contributing to KServe. We have 19 amazing

maintainers so far and over 300 contributors accumulated over the years. So if you've been following KServe's history for a while, we started as a model serving project focused on predictive AI. We still have the features there in case you have predictive AI workloads. So, things like scaling to or from zero, and request patching, and security through auth N and auth Z, distributed tracing, and auto scaling based

on request on GPU and CPU usage. Logging of Oh. Logging of requests and response, traffic management, and out-of-box metric support. And today we've been We've been actually working on generative AI features for a long for a while ever since it got the traction from the the community. So, it we In order to scale better for large language models, we have metric-based auto scaling. So, for example, you

can use KEDA auto pull basically auto scale based on metrics from your serving runtimes. And we also support additional like GenAI runtimes such as vLLM, TensorRT-LLM, and OpenMMLab to improve the efficiency of model serving. And we also support prompt caching to reduce the cost and also improve the throughput through features like intelligent routing and traffic management. It's worth mentioning that we leverage the OpenMMLab project model inference

optimizations for large language models. The OpenMMLab project has just been accepted as a sandbox project in CNCF. We We have an amazing lineup of from different organizations Uh as shown on the slide. yeah, join our community. Here are the links for the repo and website. And we host bi-weekly community meetings on Thursdays. And there are two dedicated channels for both end users in the for the in

the slack case of channel. And there's also a contributors channel in order to if you are ever have deeper questions regarding technical side of things. And you can find us this week. We have a maintenance session on Thursday. And we also have a project booth throughout the week on Tuesday, Wednesday, and Thursday. Mostly in the morning, but also they can extend to early afternoon as well. That's

it from me. Yeah. For us, thank you very much.