Optimizing LLM Inference for the Rest of Us - Abdel Sghiouar, Google

This talk explores the optimization of running large language models (LLMs) on Kubernetes, particularly for those who do not have easy access to extensive GPU resources. The speaker introduces various concepts crucial for deploying LLMs, such as inference, serving engines, and quantization. He discusses the challenges presented by increasing model sizes and highlights best practices for multi-host and multi-accelerator setups. The talk emphasizes the importance of efficient model management and infrastructure optimization, along with new Kubernetes features like dynamic resource allocation and the gateway API inference extension for intelligent load balancing. The speaker also shares resources and techniques that can enhance performance in production environments, focusing on network management and storage solutions. Overall, the talk serves as a comprehensive guide for developers aiming to deploy LLMs more effectively on Kubernetes.

Hi everyone, thank you for coming. Um, I was supposed to be the going this talk with my colleague Mofe, but unfortunately he's not here. Um, we did something similar at the last CubeCon at London, where we talked about basically running large language models on Kubernetes in general. And then we realized very quickly that we work for a big cloud provider, and we have virtually unlimited access to

GPUs, which is not everybody's case. So, we came up with this idea of like, what what do you do if you're not the big cloud providers, right? So, how do you optimize Kubernetes for running large language models? Hence the title, running open LLMs for the rest of us. My name is Abdel, I'm a developer advocate at Google. I work on Kubernetes, I co-host a podcast called Kubernetes

podcast, and I live up north in the cold country of Sweden. I want you to raise your hand if you have used either Ollama before. Docker model runner. Or Hugging Face to run a model. Oh, so that's most of you. So, you can just leave, thank you. I'm just kidding. what you have probably seen over time is that basically models are getting more and more sophisticated. And

by sophisticated, they're just getting bigger and bigger. Both in terms of how many parameters they can actually support, which is how many neural networks we inside the model, but also the size of the model themselves, right? And at some point, you will get to a physical limitation of how many actually GPUs you will need to single to run a single model. And you will have to start

thinking about running models across multiple GPUs, or sometimes even running them across multiple hosts. So, yeah, we're talking about cases where you're running DeepSeek at the 6.71 billion parameters, which is a 1. roughly 1.3 terabytes model in the size, if you have a full quantization, which is like the full precision of the model itself. Before I get going, I just want to spend like super quick time

just doing some very basic explanations for concepts. I guess everybody knows what large language models are, so I'm not going to spend time on that. Inference is kind of this weird term that we use to describe a little bit of everything. Inference is essentially what happens when you give models data for the first time. So, inference is what happens after training, right? So, um inference is actually

quite complex quite more complex than people think about it because a lot of times when you run a model on Ollama on your laptop and you just curl it, you think that that's inference. That's just you making a request. Inference is what happens inside the model. It's an entire pipeline which invokes decoding, encoding the tokens, etc. etc. I'm not going to go into the details, but it's

quite like a complex process. Serving or serving engines is essentially the piece of software that allows you to do inference on a model. Um the very basic way I explain serving or a serve serving engine is it's what a web server is for web applications. If you have a web app which is HTML, JavaScript, and CSS, it's useless if you don't have a web a web server

to serve it to you. So, that's what serving engines do. Accelerators, we all know what they are. Quantization is the model precision. So, the model precision is a value which which allows you to tweak how precise you want the model to be and that's has a direct correlation to how much memory the model actually need to run. The lower the precision, the less memory footprint the model

um requires. The higher the precision, the more memory footprint. The weights are the numerical values that defines the strength of the neural networks. Those are extra parameters that usually you have to download next to the model itself to be able to run it. Context window is actually quite important so I'm going to talk about it in a bit. Context window is basically how many tokens the model

can get or can process as one input. Why does it important? Because a lot of times when you are using models to build intelligent applications, you need to have a large context window because models are stateless by default. They don't remember things, right? So, if I'm talking to a human and I ask you, "What's the capital of the Netherlands?" You say, I'm assuming Amsterdam, but I might

be wrong. And then, Sorry? It is? Okay, it it's good. Thank you. and then if I ask you how many people live there, you know from context that I'm talking about Amsterdam, but a model wouldn't. So, each time you are sending a new question or a new follow-up question, you have to send back the previous questions and the previous answers. So, in a chat case scenario, that

windows get like bigger and bigger and bigger, right? So, that's something you have to keep in mind. From your perspective as a platform administrator, network administrator, that basically means each time you have a new turn around or new turn back to the model, the size of the request gets bigger. And then it gets even more complex when you are talking about multimodal LLMs, which supports text, image,

videos. The request can be in the order of like uh megabytes or gigabytes sometimes. And then when you are dealing with large language models, you might be in one of these three scenarios. Either you're running a single host, single accelerator, a single host, multi-accelerator, and at some point you're going to run out of enough PCI Express ports on that server server to have accelerators, so you have

to split it across multiple hosts, multiple accelerators. running large language models, you don't really have that many options to do them in production. You basically can do either a virtual machine or a bare metal, whatever you prefer. You can do Kubernetes or you can use some sort of fully managed uh fully managed um platform, Bedrock, Azure, Vertex AI, whatever. Um if you are doing the bare metal

stuff, I don't think you should be at KubeCon. If you are I'm just kidding. If you are doing the the the automated stuff, you probably wouldn't care. The only thing I would say probably is that if you are on the right side, the challenge you might be facing is the fact that most of these platforms does not scale to zero. You deploy a model, it's running there

all the time. At least I know it's for us for our case. Kubernetes is somewhere in the middle, where you have kind of the benefits of the tweaking of your of your deployments however you want, and um you get all the the benefits of like, you know, securing, optimizing, etc., Um so, why would you want to run your models locally? Well, regulation is one of them. You

don't trust a cloud provider. Latency is an important one. Actually, this is it's very interesting for me that we have to talk about this quite often with with developers to tell them that you have to deploy your application as close as possible to your large language model cuz latency matters. It feels to me like 15 years ago we were telling people you have to put your database

close to your application cuz latency matter. Um, so it's the same discussion all over the place. You have to run it close to your So, you can't Like, if you run your application on prime and your model is on the cloud, you know, you have you are at the risk of running into latency. Special use cases where the model itself that you are using cannot actually fit

your use case, you need to tweak it tweak it you need to tune it. Uh, privacy is another one. You just don't trust cloud. Cost. Uh, cloud hosted models are usually super expensive per token and you want to be able to like take the most benefit out of that that particular model or any combination of these three or these five reasons, right? So, naturally, since we're at

KubeCon, Kubernetes is one of the platforms to be able to do that. And Kubernetes, I mean, historically have been really designed to do web applications or back-end applications. But the community have been actually working super hard over the years to try to accommodate more and more large language models and AI use cases. And I'm going to talk about few initiatives within Kubernetes projects that have been uh,

introduced to achieve that. Why do we want to use Kubernetes? Well, I think we all know why. Automation, scalability. The The two most most interesting things I think are the advanced device management. So, today when you are using Kubernetes, you basically have to tag your deployment with GPU one and then Kubernetes provision all of that stuff under the hood for you. You don't have to install drivers

for your accelerators. You don't You don't have to do anything any extra wiring. Kubernetes handles all of that stuff for you and it's part of the automation. And then of course, it's multi-cloud so you can virtually deploy it anywhere you want. The thing with large language models is that they are kind of a special type of workloads. They are not web applications and they are not back

ends. They are special due to the due to due to the nature of their size, um the way we need to handle them, the fact that they need specialized hardware like accelerators, and also the fact that the traffic going in and out of those models um is not your typical web traffic. Very large requests, very long um connections. So, I'm going to cover this in kind of

a couple of layers, how do we go about optimizing large language models when you deploy them in Kubernetes? So, the layers of optimization that we came up with with my colleague are these three. Optimizing the model itself, so that's things where you have to edit or change the quantization of the model. Um or um yeah, quantization um or you might need to tweak some parameters in the

model itself. Then you have the serving engine optimization, so that's where you have to choose the right serving engine that will give you the maximum amount of output or throughput for that particular model that you're trying to use. And then at the bottom is optimizing the the infrastructure in general, but very precisely optimizing the accelerators themselves. And then on the side, kind of the horizontal optimization, are

things like um pre-warming and caching and pre-loading. I'm going to talk about a couple of examples there. Uh auto-scaling, which is quite important. You know, we have been basically as an industry going toward making containers smaller and smaller and smaller because they they load faster, they auto-scale faster, and then now you have to run Ollama with like Gemma 3 and the model with the container is like

6 GB, right? So, good luck auto-scaling that very quickly. um I talked about what serving engines are or model servers are. There are tons of them on the market. These are just a couple of examples that you have probably came across. vLLM is probably one of the most popular one. Um there is Hugging Face, they have their own called TGI, text generation interface. Nvidia NIM is a

very special uh um uh model server from Nvidia. They also have two other ones. One is called Dynamo and the other one is called Triton. Um Ray, which is the Python framework, has its own server engine called Ray Serve. JetStream and Ollama that you probably have all seen. Now, why I put this on the screen is very simple. Um for web applications, you can virtually use any

web server to run any sort of websites because it's all standards. But we are selling a space where these things are not standards. So, a lot of times, depending on which model you're trying to run, you will have to go to the documentation of these server engines and figure out is that model compatible with that particular server engine and is that serving engine compatible with that particular

type of accelerators you need to run on. Right? Most of them support CUDA drivers, so you can run them on GPUs, but most of them doesn't support TPUs, which is the Google accelerators. So, you can't actually run them on TPUs. So, you have to kind of figure out this compatibility metric between or compatibility matrix between the model, the engine, and the type of accelerator you're trying to

run on. And so, when it comes into figuring out what to optimize for, um again, back to the example of web applications, you're going to optimize for latency, uh the tail latency, you know, how many requests you can handle, etc., etc. For models, it's a little bit different. There are a couple of extra metrics you have to pay attention to. The most important one or couple of

them is the time to the first token. So, how long it takes the model to return the first token and that's a factor of how fast the model is running, what kind of accelerator it's running on, and what kind of caching you are using to get back the tokens very quickly. The normalized time per output token, uh the the the request latency, etc., etc. Then for the

throughputs, so that's all for the latency and then for the throughputs, how many tokens are you inputting per second, how many tokens are you outputting per second, and how many total tokens are you doing per second. And these are all metrics that you need to somehow orchestrate across your model deployments using any of the monitoring tools that exist in the market to be able to see this

this this metrics and be able to trace them and be able to optimize for them. So, I'm going to cover basically three main topics. Storage, which has to do with how big the models are or how how how big the the the containers or yeah, the models themselves are. I'm going to talk about accelerator management and then I'm going to talk about about networking. So, starting with

storage. This is just a quick table that we put together a while ago, which basically indicates for each of these model servers that are very popular, what's the size of the image and on average how long it takes us to deploy it to Kubernetes. We are using GKE in this case, but these numbers might differ depending on which Kubernetes installation you're trying to be on. So, I'm

going to just quickly uh take your attention to let's say line number two. So, that's vLLM. Uh vLLM OpenAI is a variation of the vLLM container that has a OpenAI compatible interface. So, OpenAI ChatGPT, they have a an OpenAI spec and most actually not all of them, but most of these support an OpenAI compatible spec. Ollama is one of them. So, that's basically when vLLM loads the

model into the memory and gives you an open API um um interface, right? That you can interact with. So, that's like 5.24 GB. This is just the inference server. There is no model inside this thing. Just the inference server alone, right? Um so, you can see on the right how long it takes on average. So, as you can imagine, if you are trying to do auto scaling

on paddle, a 5 minutes to download the container image for the model is not the best thing you want to have from an auto scaling event. But then, we look at the size of the models So, then things start getting a little bit more complicated. So, this is how much the So, the model size in GB depending on which model and what quantization. And so, uh if

we look at the DeepC, which is all the way at the bottom, at full quantization, which is 16 bits, it's roughly 1.37 terabytes. One Yeah, 1.37 terabytes. So, if I want to run that particular let's say uh vLLM, so that's like 5.24 GB for the inference server. Adding a 1.3 terabytes, so then in this case, 5 GB is irrelevant cuz it's peanuts. So, downloading that every single

time you want to run the model, you know, becomes a problem if you have an auto-scaling event. So, how do we solve this? One of my favorite projects is this project called Spiegel. It's an open-source project that basically creates a mirroring environment within your own Kubernetes. So, the way Spiegel works is that it turns every node into a mirror for the images, which means that if a

node already has the image, then the adjacent nodes in the cluster will download the image first from the node that has it cached. Um this works on almost all Kubernetes installations except ours, and I am trying to solve that. Uh it has to do with container deconfiguration. There is a bug open about it. We're well aware, and we're trying to solve it. But, this is actually quite

cool because if the If you're using If you're packaging the model itself inside the container, so let's say for example, you take the vLLM container, you you use it as a base image, and then you download the model inside, and then you have this like super big container, then you just need to download it once to one of the nodes, and then downloading it across the nodes

hopefully faster. And I say hopefully faster because obviously you are um fairly limited by the size of the link between the the nodes themselves, right? So, so it's something to have to keep in mind. Another thing caveat with this is that Kubernetes itself doesn't have a concept of quality of service when it comes into network traffic. You can't configure Kubernetes to prioritize certain traffic versus another. So,

it's something to, you know, take with a pinch of salt. Another cool thing that actually can be paired with Spiegel is using Docker containers as volumes. This is available in Kubernetes since 1.35 in beta, where you could basically load a volume and the volume point you point it to an OCI artifact, so an OCI container image. So, if that OCI container image that you are using as

a volume is available cached on one of the nodes at Spiegel, then it should also be faster to download the the container image. These two projects, the reason I'm mentioning them here is because these projects, features, whatever you want to call them, are basically things that anybody can use. They are open source, right? Um obviously, if you're running on cloud, you might have something like image streaming,

which should make model container or container image download faster for your particular use case, but not everybody can can be on cloud, as I started my talk with. Um so, these are something that hopefully everybody can use. Couple of more techniques you could use is basically you could preload or cache the the model itself on the node. So, that could be done with things like extra volumes.

You basically just have um a volume and hopefully based on SSD, so it's fast, and then you can just mount it to every single node in your cluster. Sort of like read once or um no, write once read many. So, you only write the image to it once and then you mount the same volume on every single node in your cluster, and then whenever the server or

the inference server starts, you just point it to the folder that contains the the model itself. So, it's it's faster. It doesn't take time to download it over the network and then load it into the into the memory of the GPU. Now, accelerators. Um you all probably familiar with something like this, uh where you are basically mentioning a pod and then in the pod you have just

uh limits that says I want one or three GPUs. This is This have existed in Kubernetes for quite a while and this have been all possible through what we call device plugin, which is a feature in Kubernetes I believe since 1.25 or something. So, the way this works is that basically you have a controller that runs on the cluster that whenever a node spins up, it will

go and detect which GPUs are attached to that particular cluster, and then um register them with Kubernetes. So, when the scheduler gets a pod like this, it goes, "Oh, I need three GPUs. I know that this node has three free GPUs. So, I'll assign it to that particular node." Device plugin have served us super well so far, but it does have a couple of limitations. One of

them is actually you are just saying I want three GPUs. You're not saying which GPUs you want. Right? So, it is actually normally up to you as a platform administrator to provision the nodes properly with the right type of accelerators, so the the the correct model before you can actually use the device plugin. And then it's up to the people who are deploying on your Kubernetes cluster

to figure out to use usually something like either node selectors or taints or tolerations to say, "Okay, for this particular model, I need to run on this like L4 or T100 or whatever kind of GPUs you want to run on, right?" And then over time, the community introduced something called DRA, dynamic resource allocation. So, the easiest way to explain DRA to people is to say DRA is

an API that is trying to replicate the same behavior of storage in Kubernetes. The storage uh CSI or the storage uh the container storage interface have had this concept of um you as a platform admin introduce this concept of storage classes. The storage classes basically indicates what type of storage is under, SSD, not SSD, network attached, not network attached, etc. And then your developers use a claim

to basically claim the particular storage they want, persistent volume claim, right? Everybody is familiar with this. DRA is trying to introduce the same concept, but in a more generic way, which can actually you be used for any sort of hardware resource. It's not limited to Kubernetes. So, the way it works is that you create what we call a resource slice. The resource slice describe this particular type

of hardware which is available in the cluster. So, I have an Nvidia or this particular Nvidia GPU. This is the model ID. It has 40 GB of memory and it has um X number of cores. Then um uh pod will request a resource claim. The resource claim is what allows the scheduler to match what's available in the cluster, which is the slices, to what the container or

what the workload is requesting. And what's cool about actually about the array is that we can if the accelerator support resource sharing, we can actually use the array to express that. So, in this particular case, I'm saying that my workload needs a GPU an Nvidia GPU with at least 30 GB of memory. And since I am not specifying a specific model, then this particular claim will land

on that particular slice because the slice has 40 GB the claim request 30. Right? As long as the GPU itself supports um sharing or like virtualization, this should work. It's actually more expressive than this. This is a very simple example. But you can do things like um one of the my favorite examples is actually being able to do network claims. So, you could actually run a pod

and say I need this pod to have two network interfaces, one in this particular network and one in this other particular network, right? Um and and the DRA API itself have been extended or is being extended to be able to support more of these use cases. So, right now it's focusing on accelerators. Uh by the way, just this week uh Nvidia have announced that they are um

they they open source their GPU drivers because you know, the API is the API within Kubernetes, but it still had me require requires drivers to be able to interact with the different type of hardware. So, Nvidia have um uh open source their GPU drivers. We are open sourcing our TPU drivers. And then um there have already been uh an implementation of network drivers that have been open

sourced by Google a while ago. But the area is very cool API. I really highly recommend you look at it. I think, in my opinion, if I have to speculate, this is probably the future hardware claims within Kubernetes in general will be done in the future. And now we come down to the everybody's favorite topic, networking. One of the interesting things about large language models, when it

comes when it comes to like network patterns, is that they are unlike web applications. Web applications have basically fast, uniform, cheap, and small requests that happens very often, right? So, you send a request, you get a response, you send a request, you get a response, blah blah blah. Which means we can't actually use traditional microservices type load balancers to just round robin across multiple um backends. And

we can make sure that all the backends will have unified utilization because we're just load balancing the same type of requests across all of them. But with large language models, the requests can be slow because either they are big, they contain an image or a video, or they have long connections, right? You have a web socket connecting the client back to the to the backend. They can

be non-uniform. Some will be big, some will be small. And they can actually be super expensive to process. Which means that you might actually want to do things like for this particular kind of for like uh image processing, I want to send them to this particular model. For, you know, um text processing, I want to send them to this um other type of model, etc. So, in

this case, traditional load balancers doesn't work because what you might end up with is one of your backends will be just vastly more busy than others because the load balancer doesn't know. It's just trying to kind of load balance across all the backends available. Also, most load balancers doesn't actually have any way to express extra metrics except CPU and memory, right? Um and like if you look

at things like HPA, horizontal pod auto scaling inside Kubernetes with custom metrics, most of the time it supports like HTTP-based metrics or some sort of like CPU memory metrics for for being able to like auto scale the backends, right? Yeah, media generation might require longer connections, especially if you're doing something like a diffusion model, you're basically getting the tokens back to the client as they are generated

from the from the backend. Um um and actually another thing you might want to do is you might want to prioritize traffic based on how important the traffic is to your business. Imagine you're an airline and then there are airline cancel like airplane flight cancellations and you might want to be able to process more chatbot people talking to the chatbot to get their flights arranged uh rather

than other type of processing that that you might be doing, right? And yeah, there is a bunch of other uh reasons why you want to take care about Um so, the Gateway API, which you probably have all familiar with, have been trying to solve this via something called the inference extension. The inference extension is cool, it tries to solve one particular problem which I'm going to cover

now. But the way the Gateway API does that is it introduced actually two new objects. If you have seen the Gateway API before, um the Gateway is how you define the load balancer, the HTTP route is how you map the load balancer to the service or to the backend. There have been two new objects introduced, inference pool and inference objective. So, the inference pool defines a pool

of um large language models basically or model servers. And then the objective, that that's how you can do uh sort of like um threshold-based balancing or threshold-based auto scaling or something like that. One of the cool things about this is if you have used actually um any sort of server that supports OpenAI as an interface, uh inference server, you probably have had to run the model first

and then curl it and when you curl it, you have to send a request to these endpoints v1 chat completions and then in the body of the request, you have to mention the the ID. And if you don't, then you get a response that I don't know which model you're talking about. So, traditional type of load balancers or traditional load balancers cannot actually load balance based on

the core or the body of the request. They can only load balance most of the time based based on the path. Right? So, imagine you're in the situation where you have multiple models running behind the scene and you have a single endpoint or a single load balancer and you want to be able to send requests coming to model A to model A, model B to model B,

right? So, what the inference extension introduce is this thing called the endpoint picker extension. It has one particular implementation, but it's very extensible. So, the endpoint picker extension is basically a pod that will get the request from the gateway from the load balancer and then return back to the load balancer which pool to send the request to based on the body of that particular request. Right? So,

the the basic example, if you go to the inference extension page would what we call body-based routing, which is looking at the model ID and then forwarding to the model ID. But, you can extend that to look at things like KV cache, the utilization of the model, the memory of the of the of the GPUs, etc., etc. Because you basically have a decision point that the extension

the EPP the extension the endpoint endpoint picker extension EPE, sorry. Um is the thing that basically returns back to the gateway to tell it, "Hey, go this go to this particular inference pool." Right? And so, based on one specific implementation or one specific use case, there have been kind of new project trying to solve extra use cases like, for example, KV cache being able to load balance

based on how busy a backend is. And that's essentially what LLMD as a project have been trying to do or is trying to do. LLMD have also been donated this week to the CNCF. And what LLMD is trying to do essentially is introducing what we call a will lit path. So, LLMD itself is not code. It's just a bunch of existing projects that we are packaging in

a in a particular way to solve a very particular pattern, right? And so, if you look at the the architecture of LLM D, it basically leverages the inference gateway. The proxy itself that the inference gateway runs is almost irrelevant. In this particular case, it runs Envoy, but you could also run any other proxy you want there. I think there is one implementation with K agents, if I'm

not mistaken, or agent gateway, one of them. Um it has a a body-based router, and that is has also something called an inference scheduler. So, this inference scheduler is is an implementation of that extension point picker, the EPP I talked about earlier, which uses basically things like how loaded is the back end, how the KV caching is, etc. etc. And it will be able to route to

the back end that that the request can have to go to. And then in the back end, it just leverages vLLM. And then it has also the possibility to leverage existing solutions for doing caching distribution. So, I didn't want to go too much into the details, but in a very natural way, you can think about it as in some particular scenarios, you might want for the same

exact request coming from a user to use some sort of accelerators to do tokenization, and then other type of accelerators to do the actual decoding, the actual responses, right? Because some are more optimized for some tasks than others. So, what you might want to do is you might just want to send the request to the prefill pool first, that will prefill the cache, and then send the

request back to the decode pool, which then sends process request, sends it back to the user. I'm not going to go too much too much into details because I don't have time, but I really highly recommend you check out the LLM D project. It's pretty cool. So, we have this QR code if you want to take a look at it. This is a very basic implementation, where

we just basically deploy like 10 large language models into Kubernetes and we built an interface that allows you to chat with all of them at the same time. So, if you have a weekend to spare and a lot of GPUs, you can actually try to deploy it. It does require quite a lot of them, but it does actually have a lot of very cool very cool It

does have a lot of YAML files that can be reused. So, in a nutshell, optimizing large language models on Kubernetes is a multi-layer problem. It's a multi-dimensional problem. You shouldn't think about it as a single place. It's not how fast you can auto scale. It's not how small your container is. It's basically a sum or a combination of all of these things. I hope this was useful.

Thank you so much. That's my QR code if you want to find me on social media and I have three minutes for your questions. There is a microphone here if you want to ask questions so that it can be captured on the video. Well, it's either that was very clear or no one had an idea what I was talking about for 3 minutes. Can you come to

the microphone if you don't mind? Okay, tell me. I can repeat the question. The question is if the picker extension has delay. It's very fast, make microseconds. It's quite fast. Oh. Yeah, it's quite fast. Yeah. Right? Yes. He's coming in the back. All right. So, um I I've been trying to to reduce the loads the cold start on my on my Kubernetes cluster for like images that

are like close to 30 GB in size. And yeah, I've been try Have you tried with something like BottleRocket? I know it's like an open source project and Google doesn't have a variant yet, but I'm just curious if you have and then what do you recommend? I haven't tried with BottleRocket specifically, so I cannot recommend anything for that particular use case. Um but I would recommend to

look at something where you can actually pre-load the image into the node itself. Somehow like pre-cache the image into the node. That could be through a second volume. You can just attach a second volume to the node and have the the volume attached. Are you running on cloud by any chance? Sorry? Are you running on cloud? Yeah. Okay. Do they have image streaming? I don't know. Maybe

I heard about it today for the first time. So I guess it's EKS, so maybe I think they do, yeah. If you have image streaming on it does exist on EKS, so you can just turn it on and see if that will help. Okay. It usually helps on the second pull. The first pull is usually slower. But the second pull is usually faster. Okay. Yeah. Yeah, I

have a question. Yeah. What with the pre-loading of the images and everything? Uh how do we count with the fragmentation complexity when we have certain sizes of these LLM jobs or workloads? And we have a big cluster with various capacities and so on. And I cannot fit it every time to the any node, so is the solution just to load it everywhere? >> Or to load it

to a single volume that you can mount everywhere. Okay. Right, so you don't have to load it to every node. You can just have a single volume and this will attach And then you can load that to every But what what about the actual mapping? I mean what mapping those jobs to those specific nodes? Uh it's an an NP problem, so Yeah, then you will have to

use the Kubernetes standard mechanisms to make sure that the job lands on the node that might have the image pre-loaded on it. >> It's It's It's a little bit complex. Yeah. Yeah, that's why I'm asking. Thank All right, thank you so much.