Sponsored Keynote: Inference and Sovereign AI: Scaling Cloud-Nat... Karena Angell & Vincent Caldeira

This talk addresses the integration of generative AI into production systems using Kubernetes, highlighting the challenges of scaling AI models without compromising data sovereignty. The speaker emphasizes the need for a Kubernetes-native AI infrastructure that supports complex, memory-intensive AI workloads and offers interoperability to avoid vendor lock-in. They propose an open blueprint for AI that enables deployment across various environments, including cloud and edge locations, while maintaining regulatory compliance like the EU AI Act. To achieve this, the speaker introduces an open-source AI inference stack utilizing technologies like KServe and LLMD, which optimize Kubernetes for AI demands, enabling advanced load balancing and efficient resource management.

Good morning KubeCon. Hi. We have reached a tipping point. Generative AI is finally getting out of the labs into a production system. But we now run into a massive roadblock. How do we scale up without surrendering control of our data? The industry has spoken. 79% of organization already view sovereign AI as a top strategic priority. And we heard yesterday that 60% of organization are already using Kubernetes

to scale their AI workload. So the challenge we have now is how do we bridge these two worlds? How do we bring the scale of Kubernetes to support our AI model and our data sovereignty requirements? Scaling GenAI is nothing like scaling web application. Traditional web app are lightweight. They are predictable. AI models, they are heavy. They are memory-hungry. They need to serve really non-uniform workloads. Because of

that the traditional way of routing traffic and load balancing is broken for AI. The result painful delays, massive hardware inefficiency and cloud bills just jumping through the roof. If we want to run GenAI at scale we can't rely on yesterday's plumbing. We need a platform that is fully AI-aware. Built from the ground up to handle the sheer width of model and their complexity. So we really need

an open blueprint for AI that is cloud-native. But first let's clarify that true sovereignty doesn't mean isolation. It actually means interoperability and the ability to avoid vendor lock-in. To do this we are building a Kubernetes-native AI factory. By using an open-source foundation, we retain the technical and operational control over the entire stack. This modular blueprint has a huge advantage. It allows you to deploy your artificial intelligence

anywhere. In your cloud at the edge into your sovereign cloud as well. Crucially this open approach allows you to handle regulatory compliance at scale in particular mandates such as the EU AI Act. And to make this AI factory a reality, we have been the leading developers of the new open-source AI and firm inference stack based on VLLM and LLMD. And migrate and integrating it natively into Kubernetes.

To serve models seamlessly, we first leverage KServe. It acts as our standard abstraction layer providing production-ready auto and hiding the underlying complexity. Next, we use Kubernetes to scale dynamically. We are bringing enterprise-grade orchestration to AI with the new API inference gateway. Or the gateway API inference extension, pardon me. Uh to enable true AI-aware load balancing. And finally we introduce LLMD to optimize distributed resources. LLMD acts as

the orchestrator delivering highly cost-effective distributed inference through intelligent scheduling. Together, this entire stack turns a standard cluster into a highly tuned, sovereign inference engine. To achieve this optimization LLMD is fundamentally upgrading Kubernetes with AI-aware routing fabric. Instead of basic HTTP routing the inference gateway looks at the KB cache. It uses prefix-aware routing to intelligently direct prompts to the exact node that holds that context in memory. This

slashes your time to first token by offloading the need for GPU processing. LLMD also handles prefill and decode disaggregation. It dynamically separates the compute-heavy prompt reading phase from the memory-bound token generation phase across the cluster. We're moving far beyond simple round robin scaling to multi-dimensional state-aware scheduling that squeezes every ounce of performance out of your expensive GPUs.