Evolving KServe: The Unified Model Inference Platform for Both Predictive and... F. Spolti & J. Lee

This talk covers the evolution of KServe, a unified model inference platform designed for both predictive and generative AI, presented by Filippe Spolti and Jooho Lee from Red Hat. As generative AI reshapes application development, the session addresses the critical need for scalable, flexible, and interoperable model serving infrastructure. The speakers highlight key challenges in deploying large language models, such as inference efficiency, distributed execution, and cost optimization. They introduce the latest advancements in KServe, including a new Custom Resource Definition tailored for large language model serving and enhanced integration with the open source Envoy AI Gateway, marking a significant step forward in cloud-native model serving capabilities.