Towards Building an Open Source AI Reference Stack for EU Sov... Madhav Bhargava & Sanjay Chatterjee

This talk discusses the creation of an open-source AI reference stack for the EU sovereign cloud, presented by Sanjeev Chatterjee from Nvidia and Madhav Bhargav from SAP. They highlight the necessity of GDPR compliance and data sovereignty in building AI infrastructures. The speaker outlines the evolution of AI workloads from traditional models to more complex disaggregated architectures and the challenges Kubernetes faces in orchestrating these systems. They introduce the New European Stack Foundation, funded by the EU, and describe the various layers of the AI reference stack, including data center resources, metal automation, and Nvidia components. Additionally, the presentation covers the integration of tools like Dynamo and Grove for managing AI inference tasks efficiently.

Welcome to our talk towards building an open-source AI reference stack on EU sovereign cloud. I'm Sanjeev Chatterjee from Nvidia. And my colleague Hi, I'm Madhav Bhargav from SAP. Let's get started. We are entering the era of sovereign AI. And open-source software is the foundation of EU sovereign infrastructure. Because it is built trust through transparency. What it really means for the EU sovereign cloud is that this infrastructure

needs to be rooted with European values such as GDPR compliance, data sovereignty, uh, you know, data being processed locally. So, when you build the AI reference stack on top of this infrastructure, it'll include sovereign models that have been trained with sovereign data. Um, and data being processed by LM chatbots or AI uh, agents will need to be localized within EU sovereign boundaries. we want to understand the

AI landscape a little bit today. So, we can see that AI workloads are rapidly evolving. For example, the traditional single-node serving model is slowly shifting towards complex multi-node, multi-component systems. For example, disaggregated inference architectures are typically composed of a mix of different kinds of tasks like compute-bound tasks which are memory-bound, some some are latency-bound, and they all can run on, you know, not just homogeneous architectures but

also heterogeneous architectures with CPUs, GPUs, LPUs, DPUs, et cetera. So, now Kubernetes has a new challenge. How to orchestrate multi-node, multi-component AI with disaggregated architectures on heterogeneous compute at scale. And this enforces quite a few requirements. For example, hierarchical gang scheduling, multi-level auto scaling, topology aware placement, and startup dependencies between these tasks. So, in this In this talk, we are going to explore how to build the

an AI reference stack on Kubernetes that satisfies all of the above requirements in the context of the EU sovereign Thanks, Sanjay. So, we start we stand on the shoulders CNCF and and and Kubernetes. So, SAP has created a foundation called New European Stack Foundation under the which is funded through the European Union's initiative called the important project of common European interest or IPCEI. This is public money.

And it unlocks {quote} {unquote} somewhere around 8 billion euros of investment. So, with the goal of building a multi-cloud multi-provider cloud-to-edge continuum. So, the participating companies and organizations are collaborating to power cloud native in a fungible fashion for completely sovereign purposes. Some of these projects are net new ideas. And the others have been used in crucial production environments for some time. So, now now let's dive deeper

into the lower layers and explain how these projects are entering into joint EU AI cloud initiative. So, you you have on the So, you have a Kubernetes layer in the middle, then you have all these layers in the bottom, then these components come on the top and on the sides. So, now let's dive deep into the AI reference stack a little bit more and zoom in on

it. And let's start from bottom up. So, we have data center resources like compute, network, storage. And on top of that, we have a data center management layer. In In So, what we have is something called as a metal automation or a metal API, which has several parts, for example, metal operator, boot operator, networking subcomponents. And they all The interesting part about this is that actually uses

the K 2's declarative APIs to actually manage all these resources. On top of this we have an IAS layer which has two variants. We offer one in as Cobalt Core which is OpenStack based and offers imperative APIs. And then we have another variant which is Iron Core which essentially offers again a declarative K 2's APIs. It has an extension API server which handles natively a lot of

these resources and offers it to the end consumers. On top of that we have K 2's services Gardner provides K 2's managed K 2's offering and on the left hand side you see Garden Linux which essentially is a foundation OS that we use all the way from bare metal to to the containers as well. On top you see Nvidia components that we bring in now which is

Nvidia GPU operator and Nvidia network operator which essentially enhances the OS level and also the K 2's node level to enable RDMA Rocky and other network accelerations that are required for all these high performance AI workloads. And on top of this Kubernetes layer you essentially have I only we have only shown these two but there are many other components that are there. So you have Dynamo which

is Nvidia Dynamo which is an inferencing and serving engine and you have Nvidia Grove that is developed collaboratively SAP and Nvidia which provides a K 2's orchestration layer for hierarchical AI Now let's dive uh zoom in on the IAS So what you see in front of you is Iron and at the top you have Iron Core which provides Iron Core runtime interface. So there's two categories of

interfaces that it provides. One of them is meant for end user facing which is to manage compute, storage, IPAM, network, etc. And the other is for providers who are who will actually provide pluggable providers. So, by default you have SEF as an provider for block and object store. You have virtualization provider which is libvirt which uses QEMU and KVM. And you have metal net plus DP service

which uses DPDK. to essentially provide accelerated packet processing by bypassing the kernel and enabling enabling direct user space access to the NICs. So, IronCore also includes subprojects effectively and that deals with bare metal and network switch and routers. So, which we also talked about and which is highlighted in the in the previous slide in terms of bare metal API. Which we don't cover in this part of

the slides. Now, let's look at another component called Gard Linux. Gard Linux is a very lean, hardened, container focused, Debian based purpose optimized Linux distribution. So, we chose Debian because you you can support for Debian from your preferred partner anywhere in the world. It is not really locked into a single vendor distribution. Gard Linux runs the latest LTS kernel. It has a very mature build process which

means the builds are repeatable and auditable. It is purely system D based. And from security perspective it has the built-in security modules which are SE Linux which are very popular in US. You have AppArmor which is popular in Europe. And you have a system D which is moving to landlock. So, we also support that. For security features we have fully signed kernel module and boot path and

there you can get your own keys and get it signed with your own keys. So, we support secure boot, trusted boot. Gard Linux also offers you capability to build flavors of it by choosing feature sets. And then you can build flavors of Gard And you can see now how we have used such features to actually also enable AI capabilities on top. Garden Linux runs on servers, switches,

smart NICs, and GPU hosts. So, as part of the AI reference stack implementation, we also have enabled and as part of a feature what we talked about in the last slide, we have actually developed a feature which provides a Garden Linux flavor which has capabilities to provide support out of the box for RDMA, GPU direct RDMA, Rocky, and from Nvidia we have NVLink support. For that, now

what we include as part of the core kernel modules as part of Garden Linux, some of them are DMA DMA buff, you have RDMA specific kernel modules, you have Mellanox kernel modules. Then you have Nvidia specific kernel some of which are Nvidia and Nvidia UVM. And these are proprietary kernel modules at this point of time, but we are planning to move to open source Nvidia kernel modules

and because those those support are actually from architectures which are Turing and new generations. For kernel parameters and system D settings, we have there are certain specific requirements from RDMA to work properly and so IOMMU settings, memlock settings have been have been changed as well. For the network configuration, we are using jumbo frames and because that is also a requirement from RDMA which means MTU has to

be anything beyond 9000 plus. Then we use the Nvidia GPU operator to set up the net the Nvidia fabric manager which essentially configures routing among NV switch ports and monitors fabric for NVLink and NV switch errors. We also use Nvidia network operator to set up SR-IOV which provides direct hardware pass-through for lightweight lightweight virtual functions for MPI and libraries which are sitting top on top of it

like Nickel can actually use true zero copy kernel bypass GPU to GPU communication at line rate, full line rate. Now, let's focus on the next layer, which is essentially a Gardener, which is a hosted control plane offering Now, Gardener is a KaaS service product that is even shipped on premise to all the other cloud providers and that is run by the new and reformed members and others

and beyond. The enterprise characteristics are the life cycle management and upgrades are part of the product. So, adopters do not need to worry about fiddling with you know, DevOps to secure SLAs. So, a self-hosted Gardener offering is used for fully air-gapped. So, we we also have the self-hosted Gardener offering, which can be used for fully air-gapped and highly secure environments. on the right-hand side, what you see

is a shoot cluster, which is the end user-facing cluster, where you can actually deploy your workloads. And the control plane of these shoot clusters are now hosted and bin packed into another KaaS cluster, which is a seed cluster. What makes this KaaS cluster into a seed cluster is a component called Gardenerlet, which is very similar to what you have if you want to draw a parallel. So,

you have a kubelet, which makes a VM into a node. Here, the Gardenerlet makes this cluster into a seed cluster. So, the the benefit of having this bin packing this shoot control planes into a seed is that you don't have to you can optimize the resource usages, which means you don't have to start dedicated VMs for hosting control planes for each of these shoot clusters. You essentially

have the you leverage the capability of self-healing capabilities offered out of Kubernetes if you carefully configure your liveness and readiness probes because all these shoot control plane components are actually started in a namespace, a dedicated namespace, and which are running as deployments, stateful sets, and so on and so forth. On the right-hand side on the left-hand side, can actually see the garden cluster, which essentially is the

heart of the entire hierarchy of these clusters. Here is when where you have the capability. So, there are two sets of APIs that are offered out of garden cluster. It runs an extension server. And one is one set is a consumer facing APIs which you will interact with to create projects, to create shoots, which you see on the right hand side on and which you will operate

on. And the second set of APIs are for extending gardener. So, gardener offers extensibility at multiple layers. So, as a user, you will use the standard coop cuddle tools to connect to the garden API server and the exposed coop API server for the shoot control planes to access the shoot clusters and manage your workloads. Now, let's look at the garden extensibility. So, gardener core is we have

kept the go gardener core as quite minimal, which means that we now have an extension framework which provides capability to extend gardener at multiple layers. So, you can have gardener can can be hosted on any provider. So, AWS, Azure, GCP, Ancor, OpenStack, etc. and so on. So, you can add your own infrastructure provider. Similarly, OS, run container runtimes, network, etc. For the purpose of the AI reference

stack implementation, we use these mechanisms to actually extend gardener to introduce the NVIDIA AI acceleration which then automatically installs a lot of NVIDIA specific components like the GPU operator, the network operator, Dynamo, Grooves, Kai scheduler, and and so on and so Now, for zooming let's zoom in on the AI acceleration layer now and I'll hand it over to Sanjay to zoom in further. Thank you, Madhav. Okay.

So, now above the Kubernetes layer, in this talk we'll focus on the Dynamo and Grove as the components for the reference AI uh uh, inference framework uh for this particular talk. So, let's take a typical disaggregated inference workload in Dynamo. It It contains modules like, uh, task like prefill, decode, and a module called front end. Now, these prefill tasks are essentially something that generates the context of

an inference request, builds the KV cache. Now, this KV cache needs to be transferred over to, uh, a decode component which will read the KV cache and generate tokens. So, what happens is we need to also understand that this KV cache transfer taking place, meaning that the prefill and the decode components need to be topology optimized and packed. So, Mix of libraries are packaged into the image

to ensure, uh, to optimize the KV cache transfer over high-speed interconnects, uh, like NVLink or any kind of RDMA fabric. And the front end module that I talked about basically acts as the API server. It also, uh, has, uh, KV-aware router which will essentially try to maximize the KV cache, uh, reuse. And then all of this, the workload, finally gets orchestrated through Grove. And then Grove works

with Kai scheduler to ensure that the gang scheduling constraints, the topology-aware constraints, and all of that gets satisfied, uh, and the and the, uh, application gets deployed on on the on the most optimal set of hardware. And finally, there's a another key component in Dynamo called planner. Uh, it integrates into Grove as as an SLA-based horizontal auto scaler, and it can auto scale, uh, the workloads at

different levels. And, uh, it it basically reacts to real-time metrics from the Dynamo application. we I was talking about Grove as, uh, the API, the Kubernetes API, and that can do hierarchy hierarchical gang scheduling. So, let's dive a little bit deeper into this API and understand how it does it. So, essentially Grove provides a single unified API to the user. It's called a pod click set. And

this allows the user to hierarchically compose the entire AI workload using a primitive called as a pod click, which is a pod click is a collection of pods of the same template. with these pod clicks, they can be either used individually within an AI workload or it could be composed of multiple pod clicks as a scaling group. And the scaling group essentially allows you to do something

like you can group together multiple pod clicks and then scale them proportionately. So, with this composition of pod clicks and pod click scaling groups, you create a pod click set. And the user essentially once that it creates the pod click set, then Grove basically takes over and you know, ensures that its scheduling constraints are satisfied. So, it is essentially a system where you can actually set your

scheduling intent using the pod click set and and Grove works with Kai scheduler to make sure that these are Okay, so now let's look under the hood at what is happening, right? So, when the user creates a pod click set, the Grove operator will create pod clicks and pod click scaling groups as sub resources and they will have scale sub resources which allow independent scaling of these

of these objects. And then it also creates additional resources such as service account secrets, a headless service for discovery for example. And all of this essentially gets created and managed by Grove under the hood. For multi-node link like multi-node NVLink architectures like GB200, Grove can will automatically generate the compute domains and manage, you know, IMX domains and channels for the user. So, that is all abstracted from

the user. And then finally, Grove will generate the pods and and put all the scheduling constraints into something called as a pod gang. Now, this pod gang essentially captures all of your scheduling constraints for the back-end scheduler. And Grove currently has like a back-end scheduler framework, which will basically you can you know, target any kind of scheduler, for example, Kai scheduler, Volcano coordinator, or or or just

use the default Cube scheduler. So, with all of this together, Grove manages to ensure that it it provides the capabilities of hierarchical gang scheduling, topology aware optimized placement, and so on. I I want to hand it over to Madhav, and we are going to talk a little bit about the demo. So, for the demo, we have a lab setup. And if you see here, we have four

B200 Nvidia B200 nodes, which have two sets of NICs. One of them is the BlueField-3 DPU, which is essentially used for It's been set up primarily for north and south traffic, which has 2x 100 GbE bandwidth. That essentially forms the data center network fabric. And then we have 8x Nvidia ConnectX-7 NICs, which essentially are 400 GbE each bandwidth, and they are set up to provide um Rocky,

which is RDMA over Converged Ethernet. What we have is a single switch, which uses um which we use as Nvidia Spectrum-X 4, which has 800 GbE Ethernet. It's a It's It's a 800 GbE Ethernet switch. So, that's a very simplistic setup. Of course, for a more complicated setup, we will probably move to rail-based optimized network fabric, but we have not done that as part of this lab

setup. So, let's move to the demo. So, we are trying to do a live demo on a let's hope everything works. Um Oh. It's not visible. Can we mirror the screen? Oh, okay. Where is it? There you go. Oh, there you go. Oh, now I can't see it. It's going to be difficult to type like this. I thought it was mirror. All right, so PowerPoint is on.

Okay. So, I think we can see now. >> Yeah. Now? Yeah. Oh, yeah. Oh, okay. All right, all right. All right, all right. Thank you so much. Thank you. on the left-hand side, what we have is So, what you see here is a com- is a terminal that's connected to the garden runtime cluster. So, the part that you see on the left-hand side. And just to show

you the extension. wait for it. And there you go. So, you have the uh we have created a new extension called the extension AI acceleration. And let's get the extension YAML file just to show Come on. And it's slow. Unable to connect. Well, let's see. Disconnected from the right cluster. Let's try it again. Maybe we can play the demo. Yeah, let's go to the demo then. So,

what I wanted to show you was the the AI acceleration extension will have it's set up um with a policy called on demand, which means that um when you as a when you create a shoot cluster, you essentially define an extension configuration. And if you go to uh one of the shoot cluster definitions, which you see here, this is the demo cluster that we have defined and

the provider is Ion Core. That's the ERS part. And if you see the YAML, and it takes time. Okay, in the meantime, you have this So, the policy is on demand, so as you can see, which which effectively means that uh when um there is at least one shoot cluster in the seed that that request for this particular extension, then on demand there'll be an operator sitting

in the garden runtime cluster, which will actually provision the controllers for this extension in the seed cluster. And then the seed cluster, the controllers are already there, which will create resources that are going to be provisioned eventually on the shoot cluster. So, let's directly go to the seed cluster. And if you can see here, you get a new name space for this particular extension. I hope it

comes up. Yeah. And you have the extension AI acceleration here, which essentially is running the controllers for this uh So, from the purpose of this demo, let's set up Grafana. Let's also the shoot cluster now. Let's start K and S. So, that we can easy for you to view. Everything takes time. So, is it visible? Yes. All right. So, what you see here is let's start with

something called as a Dynamo graph deployment. And for that, let's explain what we are what we having as a reference application. So, we have Dynamo graph deployment, which has three services. One is decode, another is front end, and then you have prefill. If you go to decode, then you will actually see that this particular application is requesting eight GPUs. So, as we said in the setup, we

are using Nvidia V200s, and each of these nodes are having eight GPUs. So, which means this will fit into one Similarly, you have prefill, which also is requesting eight And as you can see on the right-hand side, this is the layout of the pod click set. So, it's a very simple example that we are taking because in the lab setup we only have four V200s. So, we

have one pod click, so which we represent the front end the router server as one pod click, which has replicas one. Then you have another pod click, which is the prefill worker. And in a distributed aggregate this aggregate setup you will actually have this prefill divided into prefill leader and a prefill worker. Similarly, but for this for this example we are only having one. Uh and then

you're having then you have a decode worker, which is also has replica one, which also request eight Now, let's look at the the dynamograph deployment, which we call in short DGD. And here you see that this is actually being reflected here. So, it's a pretty large CR. And now what it creates behind the scenes is actually which you see here as Mistral large NVFBO FB4 dis agg.

So, the Dynamo has a component called as a Dynamo operator, which integrates with Grove and and creates the pod click set for for Grove. >> So, once a pod click set has been created, now here on Grove now will become active and it will reconcile. And it will create something called as the pod clicks. The connection went down. Okay. So, we're back. So, it creates pod clicks

and each of these pod clicks borrows has a specific naming convention. So, it starts the prefix starts with the pod click set name followed by the replica index of the pod click set followed by the name of the pod click. So, here the role of this pod click is decode. Then you have front end and then then you have then you have a prefill. Now, if you

go to the scheduler API that actually it created for this uh reference application. Essentially, you will see that there are three pod groups. Each pod group is now mapped one is to one to a pod click. So, you have one and it has pod references which actually gets updated run at run time whenever pod goes down and a new pod comes up and replaces it. automatically refreshes

this particular CR all the time. So, this is then used as a construct which is then used by the backing schedulers to actually gang schedule all of these pod groups together as a as a single gang. Now, let's go since we have already deployed this application which is Mistral large NPFB4 because it takes quite a while for this application to come because at startup, the inner containers

loads a lot of weights. So, it takes somewhere around 5 to 7 minutes sometimes. So, we have actually deployed this applications here. we also have let's start a Grafana dashboard. Let's port forward. Second. Okay, so I have port forward the Grafana Grafana dashboard. I have a chat app ready. Let's go and open the Grafana dashboard. Let's just localhost. And let's look at the dashboards here. All right,

let's move it to the right. All right, not. And then what we start is we have uh we use hey to actually load uh this particular application with a lot of post requests to the chat endpoint. Let's start that. Okay, connection refused. Let's >> It's It's not loading. Okay. Then this here application is not loading right now. So, let's start um the chat app. At least this

will work. So, let's open the chat app. >> So, we created a simple chat app to connect and talk to this inference And let's ask it a long question, which So, let's explain it in great detail the theory of general relativity. And let's hope it answers. And unfortunately, nothing. It'll take some time. Let's look at the dashboard and see if it comes up. Okay. All ready. No,

it's not going to come. And in these situations, we can only >> I I think we have a also a demo recording that we can actually show during our uh booth as well. Sorry about this. So, just to show you a quick um demo. And what happens in the end the end is important. So, essentially, you can see that as you can see that the front-end requests

per second are increasing. You have the average time to first token. And it essentially stabilizes somewhere around 30 milliseconds. Our SLI to overall ratios are also uh reflected properly. Uh so, this is what uh the These are Grafana dashboards which are provided by Dynatrace. And this we also have uh loaded. And this were actually working, but unfortunately, there's no connectivity right now and as part of a

live demo. Well, lab setup. So, that's all. If you have any questions, then please feel free to ask.