Sponsored Keynote: Inference and Sovereign AI: Scaling Cloud-Native... K. Angell & V. Caldeira (ASL)

This talk addresses the challenges of scaling generative AI systems within production environments while maintaining control over data sovereignty. The speaker highlights that 79% of organizations consider sovereign AI crucial, especially as 60% are using Kubernetes for their AI workloads. Traditional methods of scaling web applications do not apply to the heavy and complex nature of AI models, necessitating new strategies for load balancing and resource management. The proposed solution involves a Kubernetes native AI factory built on open-source tools to ensure interoperability and avoid vendor lock-in. Critical components such as KServe and the new inference gateway optimize resource allocation and improve performance, enabling organizations to effectively deploy AI models across different environments while adhering to regulations like the EU AI Act.

Good morning, KubeCon. Hey. We have reached a tipping point. Generative AI is finally getting out of the labs into a production system. But we now run into a massive roadblock. How do we scale up without surrendering control of our data? The industry has spoken. 79% of organizations already view sovereign AI as a top strategic priority. And we heard yesterday that 60% of organizations are already using Kubernetes

to scale their AI workload. So, the challenge we have now is how do we bridge these two worlds? How do we bring the scale of Kubernetes to support our AI model and our data sovereignty requirements? Scaling GenAI is nothing like scaling web application. Traditional web apps are lightweight. They are predictable. AI models, they are heavy, they are memory hungry, and they need to serve really non-uniform workloads.

Because of that, the traditional way of routing traffic and load balancing is broken for AI. The result, painful delays, massive hardware inefficiency, and cloud bills just jumping through the roof. If we want to run GenAI at scale, we can't rely on yesterday's plumbing. We need a platform that is fully AI aware, built from the ground up to handle the sheer width of model and their complexity. So,

we really need an open blueprint for AI that is cloud native. But first, let's clarify that true sovereignty doesn't mean isolation. It actually means interoperability and the ability to avoid vendor locking. To do this, we are building a Kubernetes native AI factory. By using an open-source foundation, we retain the technical and operational control over the entire stack. This modular blueprint has a huge advantage. It allows you

to deploy your artificial intelligence anywhere, in your cloud, at the edge, into your sovereign cloud, as well. Crucially, this open approach allows you handle regulatory compliance at scale, in particular mandates such as the EU AI Act. And to make this AI factory a reality, we have been the leading developers of the new open-source AI inference stack based on vLLM and LLMD. And migrate and integrating it natively

into Kubernetes. To serve models seamlessly, we first leverage KServe. It acts as our standard abstraction layer, providing production-ready auto scaling, and hiding the underlying complexity. Next, we use Kubernetes to scale dynamically. We are bringing enterprise-grade orchestration to AI with the new API inference gateway, or the gateway API inference extension, pardon me, uh to enable true AI-aware load balancing. And finally, we introduce LLMD to optimize distributed resources.

LLMD acts as the orchestrator, delivering highly cost-effective distributed inference through intelligent scheduling. Together, this entire stack turns a standard cluster into a highly tuned, sovereign inference engine. To achieve this optimization, LLMD is fundamentally upgrading Kubernetes with AI-aware routing fabric. Instead of basic HTTP routing, the inference gateway looks at the KB cache. It uses prefix-aware routing to intelligently direct prompts to the exact node that holds that context

in memory. This slashes your time to first token by offloading the need for GPU processing. LLMD also handles prefill and decode disaggregation. It dynamically separates the compute-heavy prompt reading phase from the memory-bound token generation phase across the cluster. We're moving far beyond simple round robin scaling to multi-dimensional, state-aware scheduling that squeezes every ounce of performance out of your expensive GPUs.