To Swap or Not To Swap: Memory Management Design Patterns for AI Workloads in Kuber... Nic Vermande

This talk focuses on managing memory in Kubernetes when running large language models (LLMs) and the implications of using swap. The speaker addresses the challenges posed by memory thrashing and out-of-memory (OOM) kills when enabling swap on Kubernetes nodes, especially in contexts with GPUs and LLM workloads. He presents findings from tests on latency degradation and memory management techniques, highlighting the trade-offs between using unlimited swap and limited swap in Kubernetes. Nick explains scenarios where swap can optimize resource management effectively and under what conditions it may lead to bottlenecks. He emphasizes the importance of correctly sizing resource requests and configuring quality of service settings to enhance performance during resource spikes.

Okay. Hi everyone. Hopefully you had a good lunch. Very happy to be here. Second day for almost all, first day for some of you, second day for other who were already there yesterday the KubeCon event. So hope you guys enjoying your time. So let's get started. Eight times latency degradation every inference request can be eight times slower, 1.6 GB of memory thrashing between RAM and disk on

a single misconfigured Kubernetes nodes. So quick show of hands, uh who here has been told by your ops team or platform team that they should never enable swap on a Kubernetes nodes? Yeah, or yourself. All right. What about the rest? Like you guys think you can enable swap and everything is going to be magic or All right. No, it's not going to be magic. Uh I would

say like now who, you know, keep your hands up for some people but not enough of you have their hands up. So I'm just going to tell you something. You're going to face this problem if you're going to enable enable swap and GPU in one to run LLM workloads in Kubernetes. Obviously at some point to save you some bad time in terms of OOM kills, the first

thing that you would want to do is increase the number of nodes or increase GPU resources to manage LLM workloads. Why is that? We're going to see it, but at some point the biggest risk is OOM kill. Not necessarily OOM kill in terms of setting a limit, but also just filling up a whole node. So some people they say, "Okay, now I'll just have, you know, more

node so it's, you know, I will have more space. Therefore wasting money." So the goal of today is really about that. How can swap enable in a way that is more, you know, smarter. And we're going to go through different use case when it's good to do it, when it's bad to do it, all those kind of things. swap versus LLM workloads, it's essentially the contradiction I

spent the last couple of month, I would say, investigating. So my name is Nick. I'm a DevRel at ScaleOps. So quickly ScaleOps may help you with a lot of things we're going to see today. We are do doing resource optimization at Scale. We can help you with GPU, CPU. For some of the use cases I'm going to mention how it would do like without us and quick

the value we can add to make your life easier, right? That's essentially the idea here. Um so I uh it's mainly a lot of benchmarks, a lot of number because usually I like to talk about thing I test myself for a long period of time. So this is essentially the results of what I've been testing. Uh I think it's pretty accurate. I I didn't run everything at

super large large scale, but when it's not the case, I will have, you know, I will tell you, "Okay, this is small scale most small scale. At large scale, this is what you will see, right?" So I run control experiments with swap on GP nodes, different workload types, different storage backends, different QoS configuration. We'll We'll see that QoS configuration is very important for swap. Um so I'm

going to show you three main scenario and give you a decision framework when swap swap helps, when it destroys performance, um and when actually bad sizing swap is worse than having no swap at all, right? So So let's me start with what happens when you get swap wrong um because basically this is what motivated this whole investigation. I was exploring the the feature which has been there

for a while and noticing that no one was using that. And I was it was confirmed with no one raising their hands. So let's go go ahead, right? So I set I set up a Kubeadm 1.29 cluster deliberately, right? So even though I say 1.35, 1.29 because that's the last version where unlimited swap was available. Uh I needed to show you what we escaped from, what was

before and what is now, you know, after 1.34. Uh so everything's everything else I'll show you today including limited swap is also verified on what 1.30, right? So the first test I the first test I run here, um I run a Phi-2 model on CPU only inference. So the idea is like no GPU, everything on CPU. With constrained pod memory and unlimited swap enabled, which is very

specific to you know, before um I mean 1.29. So the baseline P50 latency was 40 seconds per request and under swap pressure we moved to 116 seconds. So that's basically an eight times degradation. The kernel meanwhile also swapped 1.6 gig um to disk. For that was around 100,000 page ins and around 400 page outs. So what that means is that every inference step was waiting on disk

IO instead of computing. So no GPU, fully on CPU. So the second test the second test I did, so the same idea but this time on GPU with Mistral 7B. So again constrained the pod memory. We will I mean I'll show you why I on purpose deliberately put less request. Uh there's a formula that is involved in that, math formula. Um so the constraint the idea was

to for the the pod to have request below what LLM needs. So the kernel swapped 1.6 gig to the disk. The GPU utilization meanwhile dropped from 94% utilization to 73.5, which is 21 21 percentage points collapses essentially, right? And at the same time the latency gets got 26% worse. So the GPU was waiting on the CPU and the CPU was waiting on disk. This is why we

had this this number. So now here's the contrast. For the same pod so same spec, same traffic, the swap disabled the pod OOM killed immediately. Crash and back off. So total outage, zero requests served. That's the difference. Degraded versus OOM killed instantly. So now the question becomes should I use swap? It's it shouldn't be should I use swap. It should be which failure mode do I prefer?

Degraded but alive or completely dead, right? That's essentially that. And more importantly, can I configure swap so it helps without the performance collapse at all, right? So that's why that's what limited swap is designed to solve. So limited swap, let's take a look at what changed over time. So this is the timeline here. So KEP 2400 introduced swap support as alpha in that was 1.22 back in

2021. It took four years to reach GA. That's a lot. Four years to reach GA. That timeline tells you how carefully this was designed by the team. And the critical change was in 1.30 where the Kubernetes team removed unlimit unlimited swap entirely. The reason was pretty straightforward. So unlimited swap let any container use any amount of swap with no limits. So what that means a single pod

could fill the swap device and entirely crash the node, right? You don't want that. So in my testing I reproduced this consistently with one misbehaving best effort pod could exhaust swap in under 3 minutes on a 60 gig swap device. So one misconfiguration, a whole node That's the risk here. So what survived basically is limited uh which is now stable as of 1.34 back in August uh

2025. So limited limited swap is container aware and works within cgroup v2 boundaries. Uh and here's what it gives you. So three quality of service classes, the one that's everyone hopefully is familiar with, right? Um so for three different swap swap behavior. First, guaranteed pods. We're going to talk about when we should use it this way towards the end. So guaranteed pod where request equal limits get

essentially memory swap max set to zero. So no swap uh available for those type of workloads. So burstable pods get proportional swap based on their memory request. So this is where the link with constrained will be important because it's proportional to that. The less request, well, the less one. So, in my um tested setup, best effort pods had memory swap max uh zero under limited swap. So,

no request means no swap allocation. So, burstable proportional best effort because, you know, you don't have any requests, it's essentially zero. So, the mechanism the the mechanism the mechanism here can be seen on the left. That's the formula. The kubelet sets memory swap in each container CGroup. Uh so, for burstable pod, this is the the the formula. Swap equal uh request divided by capacity times node swap.

So, I've just verified this on the you know, even I at smaller scale. So, on GKE, for example, if you put 120 um meg request on a 16 gig node with four um gig of swap, that give you roughly uh 32 meg of swap, which is not enough. I probably it's very small. So, be careful about the request here. Uh so, the kernel enforces this use um

through CGroup V2, right? So, this quality of service swap relationship is really the foundation for everything that we are going to go through. So, this is basically the foundation you have to to keep in mind. So, why does it matter really for uh AI workloads? So, now we have how it works from, you know, technical perspective currently in the latest version. So, why does it matter for

AI workloads compared to others? So, traditional web services have relatively predictable memory patterns. You set requests, and of course, the problem with requests is that they are static. If you need something a bit more intelligence, I come to see me at the scale of boost. I'm going to tell you how we we do a dynamic resource uh monitoring and allocation. That's one But, um AI workloads are

fundamentally different. So, an inference server like vLLM loads the model weight entirely into GPU VRAM. uh if you take the Mistral 7B, uh so, 7 billion parameters, FP16, so two bytes multiplied by two, that gives you 14 gig of uh memory um required for the parameters. So, that's outside the CGroup completely into GPU memory. But, at the same time, vLLM, right, runs So, on the CPU side

runtime, you have things like tokenizer, scheduler, output buffers, and the CPU swap pool for evicted KV cache blocks. So, the context, let's say you have a chat, you you're talking for a long time, at some point for the the the context need to be compressed or eventually with vLLM because it's optimized for that, it can be placed into a dedicated pool, and this is what uh that

option gives you, right? But, that's part of the vLLM runtime. vLLM runtime running on CPU mean it goes into CPU memory. If you don't have enough memory, then you can go to swap. So, that's why, you know, all the evicted KV cache blocks, they needs about, you know, 1 GB of CPU RAM uh taking into account tokenizer and all those parameters. So, we have all the model

weight into GPU VRAM, and we still need about 1 gig um of of RAM for for um for for the vLLM runtime. So, that CPU memory is inside the CGroup. And during a traffic spike, what's happening is that, as I said, so the KV cache eviction push more data from VRAM, right, into the CPU RAM. And the total CPU memory demand can exceed your CGroup limits, basically.

So, it's the same thing. So, that's for inference. And with workload with PyTorch, have similar dynamics. So, it's more like uh gradients accumulation phases consume memory proportional to the batch size. So, dynamic batch size means memory usage is inherently variable. So, you also have checkpoints for the model. So, the checkpoint saves the checkpoint uh saves temporarily required enough memory to hold both the model state and the

serialized checkpoint. So, all that adds up. So, for both, you need extra memory, right? So, this is why if you go over that, you need some swap. Otherwise, what is the risk, as we've seen before, So, the question is now provision for peak, Waste expensive GPU node capacity most of the time or provision for average and accept OOM kills during spikes. That's the two solutions you have.

But, we have a third one um with with swap if you current configured correctly. But, to understand where swap fits, you need to see the full memory hierarchy and how it works. So, when I say, you know, I if you want to oversize your environment, so probably you will need more GPU nodes. So, swap when I say swap saves can save you money for example, uh A100

nodes on GKE cost roughly $2,700 per month, right? So, if you add a second node purely for memory headroom, then you're doubling that, right? And if you have issue with, you know, knowing what GPU um utilization you're currently using, scale up scale up also it helps you. Go talk to me at the booth. the two types of swap. This is the key insight that most discussions Kubernetes

swap miss. There are two types of swap in an AI inference stack, right? And they operate at completely different level level. So, you can think about it if you're familiar with, you know, how CPU are working, uh like the CPU memory hierarchy. You have L1 cache, L2 cache, L3 cache. Each level is larger, but slower. So, imagine that GPU VRAM is your L1 connected to compute units

via, so uh HBM, high bandwidth memory at around 2 3 TB per second bandwidth, right? This is where the, as I said, the model weights and active KV cache entries live during inference. Then you have the vLLM CPU memory pool is your L2. So, again, that's the flag when you run uh vLLM that allows you to use um the node memory as an L2 cache for your

LLM workloads. So, the vLLM page attention system manages this. So, as I said earlier, just repeating again so everyone understand, it tracks the KV cache blocks at uh at page granularity and evicts uh intelligently. So, when the VRAM is full, the page attention mechanism move less recent used block of your context when you, you know, having a conversation with Claude or everyone or any other, sorry, any

other uh GPT, this is what's going to happen, right? They are going to move the some of the pages into from the VRAM transferred And the speed here is roughly, so it's PCI limited, 60 64 uh gig per second or more for the very very recent one. Uh and the swap space flag control the size of the pool. And then we have our L3. So, that's Kubernetes

swap. So, when the CPU RAM is exhausted with the second mechanism, this is the kernel pages memory to disk. And NVMe gives you roughly 3 to 5 gig per second. Uh hard drive, so spinning drives gives you 100 to 100 to 200 MB, right? So, that's obviously the design choice here is very easy. Never use spinning disk, always use NVMe. So, uh the critical design principle here

is always prefer higher level of swap. So, we have vLLM vLLM's application level memory management is designed for transformer workloads. That's why. Uh it understand which KV cache blocks are important application level. So, it does intelligent eviction. And at the opposite of the spectrum, we have Kubernetes swap. So, it's a kernel level mechanism. It doesn't distinguish between between uh critical attention key and a temporary, you know,

temporary tensor. Doesn't have notion of that. It's the last you know, the last resource the last resource effort. But, when you have the vLLM CPU pool that is also full, Kubernetes swap is the difference at that stage when you are at that level between an OOM kill and a degraded but alive service. So, the idea is that application level memory management handles the common case. Kubernetes swap

handles the failure case. That's the idea. But, you need both layers. So, um we're going to go through three scenario that demonstrate where each layer matters, starting with multi mode multi model serving. So, this is our first scenario. Multi model serving, this is a use case where swap provide the clearest uh benefits. So, I I set it uh this up using KServe on the GPU node. So,

with three, as you see on the left here, uh three inference services resources. So, I've used a text classification model, uh translation model, and a summarization model. So, the idea was to make the total base memory approach the node's limit. So, with the the node was like 40, and I think we had uh around 35, right? So, if you're familiar with KServe, it manages the routing and

scaling for each model independently. So, each model has a specific dynamic memory overhead for inference, batch buffers, as I mentioned, KV cache for the transformer models, and uh you know, temporary tensor. But, when two or three models spikes simultaneously, the total memory demand exceeds the node capacity, right? And here we are at the extreme. So, the important observation here is that average utilization stays well below the

node's capacity. But, that the spikes that trigger OM can last 2-3 minutes, no more than that, and you'll OM kill. So, you're paying for a GPU node that's running at high average utilization, but you're still getting killed during brief windows, right? So, again, right sizing GPU, But, what's the cost of the alternative, right? Add a second node purely for memory headroom. Remember the 20 $2,700 per month

for a single A uh 100 node on GKE? Uh that's the gap swap can address here. So, how do we solve this? So, the configuration is here. Burstable quality of service, remember? This is where we have uh the request set to the base model weights. So, remember Mistral 7B, that's FP16, 14 gig. The kubelet config is four lines. The node swap gate to true, um swap behavior

limited swap NVMe swap space. So, the configuration super easy. Uh then, you can verify per container on the right side. Again, single command, just take a look at the at the C group. this is the memory swap max in the C group for a 15-gig model on a 64-gig So, with 16 gig of swap, the limits that has been calculated is 3.75. Remember the formula we had

before, requests based on the request, the node's um uh total memory, and the swap. This is the the calculation. So, it's enforced again at the kernel level. So, why NVMe? Again, 3 to 5 microsecond versus 10 to 15 milliseconds for a spinning disk. It's a 1,000 time So, NVMe means moderate latency increase when you're running into trouble. HDD means like cascading timeouts. So, this is why you

don't want to run to to use that. So, over the test window I've used on both GKE and kubeadm cluster, so again, kubeadm was because of 1.29. GKE doesn't work with 20 1.29. It's not supported. With correctly sized entitlements, zero OM kills, right? That's the main difference. So, the swap activating during spikes and drain within minutes. So, the latency increased moderately within acceptable bounds, you know, for

batch and and I would say chatbot SLAs, So, that's scenario one, uh swap as cost optimization. But, what about sudden spikes? So, that's scenario two. Here, think about viral, you know, social mention, marketing email landing, or uh peak business hours on a customer-facing AI assistants, what a lot of companies are are building in So, I've tested it using a VLM serving uh a chatbot workload. when traffic

is normal, moderate concurrent request here, the the KV cache fit in the configuration VLM swap space and the GPU VRAM. So, then I ramped the traffic from normal to 3x in minutes. Nothing, you know, really exotic, just the kind of spike you can get when um you have like a campaign of email that lands. Um and in that particular case, the KV cache exhausted both VRAM and

VLM CPU swap pool. This is where you get into trouble. So, in our memory hierarchy, remember L1, L2, L3? L1 is full now with this spike. L2 is full as well. And without L3, again, you're out of option. So, without Kubernetes enable, again, the result is OM kill. So, that means the chatbot goes down at the exact moment you have the most users, right? You have the

spike, but you can't serve that. So, yeah, your product succeeded, like your AI assistant is great, uh but your AI infrastructure turned your um success into into an outage. So, this is where Kubernetes swap as a safety net, not at cost optimization this time, but as a safety net comes in. Um it's really like the last line of defense in that particular case. So, 5 minutes. Okay,

so let me show you the data. So, that's uh four different scenario uh for that particular use case. So, two dead, two alive. C and D both had the same swap entitlement, about 196 megabytes because they share the same memory requests. Here, the difference is the C group Um so, on the one side, um the difference between C and D is that one had the limit that

was wrongly configured, and one that was rightly configured. So, again, it's not only depending on your you know, on your uh or on your the the the swap allocation that you end up with. It's you need to have the right CPU limit configured on the on the workload. It needs to be right sized. So, for a chatbot API with a relaxed SLA, swap keeps you alive during

For real-time API with sub-100 milliseconds, even with a 26 latency increase, you know, that's sort of acceptable. So, does swap always help? That's scenario three. So, scenario three here is when swap destroy performance. So, one is latency sensitive sensitive inference. So, if you need sub-millisecond SLAs, speech recognition, real-time translation, all of that won't work. All right. So, this is where swap will not solve the problem. And

the second is training workloads with sequential memory access. So, I've tested with a LoRA fine-tuner fine fine-tune uh under constrained So, again here, um the impact at scale was only 1.01 uh x, but that was at small scale. At large scale, it's a lot worse. Again, it's not meant to be used uh with swap for that particular particular case. So, the pattern here is your workload if

your workload is latency sensitive on eliminate swap with guaranteed quality of service, right? This is where you want instead of burstable, go guaranteed, and your swap space is disabled, but it's guaranteed. Like, if you're training, NVMe swap may be acceptable, but measure your actual workload before committing, right? Okay, so how do you protect all of that? So, two things. The first one is, as I said, equal

uh request equal to limit for guaranteed quality of service. And second, three Prometheus metrics. Container memory swap for for per container um swap bytes. Uh above 50% is probably the right metric here, and the limit for 5 The node memory swap free for remaining um and alert below 20%. And container swap IO count for thrashing. So, on kubeadm cluster, I've measured around 200K page for in for

VLM, um 82 for Whispers for Whisper, and 372 page out during training. And this is important because it does have um you know, a sustain rate above your device IOPS capacity. So, you have to make this coincide with your IOPS capacity. This This is another um things you have to because it's related to IOPS. So, the decision framework. Here, you got the model. This is the picture

if you want to take. That's That's one. Explain to you when to use limited swap for multimodal serving, burst traffic, everything that is burstable. Real-time inference, no swap. Training, eventually, but NVMe only. Size uh the in the right way for uh for that. Be sure to don't commit until you have uh you know uh your IOPS for that, the number paging page out. And finally, for database,

etcd control plane, uh again, you should not use any swap for that because you need consistency. So, this need the sizing. Again, it's dynamic. Talk to me at the booth, but that's basically the idea, So, quick actions for you. 2 minutes left. So, things things. One, audit your GP nodes. Uh run kubectl describe nodes and compare memory capacity to what's actually requested. Um if you provisioning for

peak to avoid OM kills, measure the gap between average and peak utilization. The gap is where um you know, swap provides the value. Uh second one is classified by uh latency sensitivity. Sub-millisecond SLA, no swap. Relaxed SLA, swap is a you know, viable safety net. In between, I would say benchmark with your specific workload before deciding. Three, enable uh limited swap in non-production cluster. So, as I

said, kubelet is just four lines of configuration. Uh verify the cgroup v2 settings. Uh and then, four, set quality of service classes intentionally. Request equal limits for latency sensitive workloads. Guaranteed. Request less than limits for workload that benefit from um the swap safety net. If you remove request, again, with the formula, it will be zero for the swap. So, always set the request for that. Five, deploy

the monitoring stack. Uh Prometheus metrics I've mentioned, the Grafana dashboard. Set alerts before uh you need them. And then, for everything else, if you want to use uh swap, then of course, NVMe over uh All right, just a quick summary on uh the benchmark. So, we had three test campaigns. PyTorch on CPU with swap show eight time latency degradation. Mistral, GPU, 21% GPU utilization drop. So, it

stayed alive. No swap killed the pod. Unlimited swap kept it alive, but degraded. But now you we have access to that configuration anymore. So, limited swap feel a 512 uh limit failed. So, not enough cgroup headroom. So, again, if you have not enough limits, then you will reach the OM kill before you can even use the swap. So, be careful with that. Limited swap with a 2

gig limit, then it survived because it could um it could um host the full VLLM um run time memory requirements. All right, so GKE correctly sized swap survived every burst level. I mean, mean-sized swap was 60% worse than no swap at all. All right, so at this stage, it's better um to to have no swap. So, on the Kubernium cluster, uh Whisper shows 503 uh meg of

swap, and training show 1.4 gig, which was only 1.01x, but again, smaller scale. Bigger scale would have been a lot different. All right, so that's the last one. One principle to take away from this talk, if you have to remember about one thing, uh so application level memory management first. So, VLLM pay attention. We talk about PyTorch memory allocator uh should always be first line of defense,

L1 memory. Then you we have these systems are designed for their specific workload, and they make intelligent decision about well, how to how to evict and when. Kubernetes swap is the last line of defense. It catches the cases, the corner cases, should I say, that application level management can handle. So, we've seen the unexpected spike, um multi So, simultaneous multi-mode model pressure, traffic burst that exceed your

capacity planning. But it only works. This is like, yeah, but there's an if. If you size it correctly. You fiddle swap entitlements, and you get the worst of both worlds. This is the 60%, right? Swap I/O overhead followed by an OM kill. So, it's better to OM kill right now than wait 60% increased latency, and then OM kill, right? So, you have to size the request uh

in the right way. So, configure both layer, monitor both and understand which layer is active at any given moment. So, swapping Kubernetes is no longer a question of should we, uh it's a where, how much, and for which You now have the data to answer that. So, thank you. My name is Nick, and if you want to talk more, let's meet at the ScaleOps booth after.