Ctrl-X, Ctrl-V Your Pods: WG Checkpoint Restore in Kub... Peter H, Adrian R, Radostin S & Viktória S

This talk introduces the Checkpoint Restore Working Group, focusing on the checkpoint restore functionality relevant to containerized applications. The speakers provide a historical context, detailing how this feature has evolved since its inception around 2005, and discuss its motivations within Kubernetes, particularly in the context of resource management and fault tolerance. They highlight three main use cases for checkpoint restore: accelerating application startup, providing fault tolerance for long-running workloads, and optimizing resource utilization. Challenges and complexities of implementing checkpoint restore within Kubernetes are discussed, including the need for API standardization and security considerations. The session emphasizes community involvement in refining the functionality and encourages contributions to the ongoing development.

Hello everyone. Um my name is Razin and uh this in this session we are going to present the checkpoint restore working group. Um to introduce myself I'm a PhD student at University of Oxford and we have been working on the checkpoint restore functionality for several years with Adrian. Uh unfortunately Adrian was not able to come today but uh we are going to uh cover essentially the background

of the history of the project and how everything started and um Peter do you want to introduce yourself? >> Yeah. Hey everyone um my name is Peter Hunt. I'm a uh chair of Sign Node. I work at Red Hat principal software engineer and um also chair of the working group uh Checkpoint Restore. Thank you for joining. and Vicki. >> Hi everyone, I'm Victoria. I am from Maseric

University in Czech Republic. And I will be also starting the whole presentation. So some of you might be asking what is even checkpoint restore? Why do I need it? Is it useful? Have I ever seen it? Maybe you haven't seen it in the context of like containers, compute, workloads or anything like that. But I believe many of you have ever encountered these menus and you know what

they're capable of. And uh usually you would use them to just like save the game and later reload the save state. Why would you do that? Well, maybe you just like uh won a hard battle or you passed some quests or you are before a hard battle or you have been playing too much and want to save the state and go to sleep and maybe like resume

next day or in a couple of days and these are all use cases that are quite applicable to other workloads as well and in the context of containers. So now moving from the games to the containers. Um checkpoint restore is a quite old concept and the origins of its development and implementation date back somewhere to maybe 2005. Then over next couple of years the implementation evolved and

changed and I would say that around 2010 people started to pick up on this exciting new feature that is huge. as someone said in the discussion to the one post and they started to come up with the use cases. Um the older use cases are really interesting. Someone suggested to like turn out of memory killer into out of memory dumper. Someone suggested that checkpoint restore could be

really useful for like uh faster kernel security kernel upgrades. Some people said that it's exciting for debuggers or tracers or profilers. But the use cases changed a little over time. Then next year in 2012, there was a follow up followup post about uh Creo and the out title it are we there yet? It's a really good blog post. I quite recommend reading it and the out concludes

with u with this uh quote. Um although many groups are interested in having checkpoint restore functionality, an implementation that works with the mainline kernel has taken a long time in coming. When one looks into the details and realize how complex the task is, it is perhaps unsurprising that it has taken so long and this also applies to Kubernetes. It is a really complex task and it has

taken a couple of years to come here to even like establish a working group. Radostian will talk more about the current state of the checkpoint restore in Kubernetes and its history but now we are here to just you know integrate it there and have it there. So what are our motivations for these changes? We already see that many companies are creating their own software stacks around checkpoint

restore which obviously marks the value of it overall and within these software stacks there is a focus on GPU resources and GPU checkpoint restore. This is quite understandable because GPUs are expensive. They are scarce. They are in very high demand with the current computational landscape. And so we think that it is really important to enable upstream Kubernetes native support for checkpoint restore because it will be beneficial

for maybe a lot of users of upstream Kubernetes. Currently we are working with three use cases. So the first use case is accelerating application startup. This is really useful for containers that have a long startup time. And with checkpoint restore, it is possible to transparently save initialized state for future starts and later we may restart one or more containers from this checkpoint without the need to reinitialize.

This is already used in combination with AI agents, persistent memory systems and Java applications. But the AI systems are obviously market drivers. Um below on the slide are some more resources. where you could learn more especially about combination of checkpoint restore with inference serving. The second use case is fault tolerance. So checkpoint restore can be used to provide periodic checkpointing for longunning workloads to simply avoid loss

of already done work and also avoid future computations in case of any failure. Um this is important because not all applications implement some recovery mechanisms or interim state saving especially not all stateful applications or HPC lag applications. So all sort of workloads would benefit from infrastructure provided fall tolerance and uh this use case can be also paired with um other events that tend to happen in infrastructure

operations like plant maintenance or advanced failure detection or also uh with spot instances. The third use case is optimizing resource utilization. um inefficient CPU and GPU utilization is a persistent issue of cloud infrastructures and checkpoint restore could be used as a primitive for load balancing between nodes or for non-disruptive preeemptions of lower priority workloads. So instead of killbased preeemption we would do checkpoint based preemption or we

could also use this for resource rec for resource reclamation from idle stateful workloads and this can be further integrated with cluster autoscalier solutions and scheduler and now to >> thank you Vicki. Um so as Victor mentioned there are several use cases for checkpoint restore and this also creates a challenge for us because every use case has different requirements and um we have been working in the CRU

community for a long time in integrating the checkpointment restore functionality with container engines first with Docker then with Podman and um around 2019 Adrian and I were talking at the Lonforms conference And uh this was just after uh Google announced like the year prior that they have integrated the checkpoint restore functionality with bork and for us this was uh showing confidence in the implementation showing that it's

at stable state and was considering how we can essentially enable this functionality in Kubernetes. so today we have already achieved container level checkpointing but I want to go through the history and what led to this and then I will cover what we're going to do next. Um the first um approach that Arian took was to essentially just implement the whole thing and open a pull request and

see what happens. And the feedback was hey you need to create a cap first and then he go back and talk with the community get some feedback there were some questions and essentially uh this led to a few implementation changes reducing the scope and um we eventually focused on a single use case the forensic container checkpointing and this is mainly to simplify what we have what changes

we have to introduce in Kubernetes but also to um essentially have a specific use case that people can use once it's merged. So um for us it was important to have the restore functionality to be able to restore from a checkpoint and we were trying to figure out how to do that having only the checkpointing functionality and we sort of looked at containerd which was already building

um OCI images from checkpoints and we decided to well we can probably implement the same thing in podmon and we tried and it was accepted from the community. We sort of simplified how the images are built. We just copy the checkpoint files inside. But essentially we try to use the same annotation pattern across cryo containerd and podman just uh to be able to recognize what is a

checkpoint image. Um and eventually this uh transitioned to a beta version enabled by default in Kubernetes 130. And since then we had a few uh interesting use cases how we can essentially coordinate the checkpoint restore mechanism across multiple nodes which is very important for Kubernetes compared to for example container engines like Podman and Docker. But also how we can use this with machine learning. This became very

useful once we enabled GPU checkpointing and later uh today we have another talk focusing on fall tolerance and how we can use this with training jobs. Um so this is a brief background how we essentially last year after our cubecon talk we we received feedback from many people and there were many similar questions and we decided to kind of it would be the best thing to do

would be to create a space where we can essentially combine the efforts of different people and um focus on solving the problem. But this will also help us to standardize essentially the checkpoint restore APIs. So we talked with Victoria and Adrian and we decided to propose the working group and um once we got to the essentially state where we filled in everything that we had to do

and we we talked with Peter and he said well the first thing we should do is ble checkpointing because this is the natural flow of um you know we can checkpoint containers but um in Kubernetes everything is the smallest deployable unit is a pod. So we need to um this is the first step to be to make it useful in Kubernetes. And so Adrian uh we went

back to Adrian's initial pull request for checkpoint uh drain option essentially to create checkpoints when you drain pots from uh from Kubernetes and then we also looked at how well essentially how we could potentially extend the API server and we kind of defined this as initial proposal for for the working group to work on. Subsequently, we had a few iterations on what should be in scope, what

should be out of scope and how how we can do port level checkpointing. And one thing that Peter suggested is we can essentially enable checkpoint restore at CRI level and at cublet level. But we need to somehow tell Kubernetes that their store pot now exists and uh we decided to extend the scope further. But we still want to minimize it in a sense that life cycle management

storage backends, shared resources, IP migration are left out of scope and Peter will talk more about this. Uh so far we have two essentially proof concepts. One is built in into Kubernetes just modifying the controller manager and the other one is an external snapshot and we're trying to figure out what is the best way to um essentially implement this and the consensus so far is uh integrating

it with the core Kubernetes API would be probably the best solution here and we created a fork here in the checkpoint restore uh organizations where we can do the development further. Um essentially what our initial state of the proposal the provisional checkpoint object um is uh just getting a request. So the user will create a pot checkpoint object which is the request to create a checkpoint. Then

we have a controller that will essentially accept this request and it will create the checkpoint. So this is pretty straightforward. We kind of I think everyone agrees what the design should be here. Um the restore functionality is a bit more complicated. So we had a few iterations on this but the best thing we have so far I guess I should cover the sort of like the why

we made these decisions. Um our first idea was to just modify the pod specification with the new field that specifies the checkpoint that we want to restore from. But we went to the sik API machinery meeting and the feedback we got was um if you have a different life cycle for pods and checkpoints and for example you restore from a checkpoint and then you delete this checkpoint

the reference essentially becomes invalid and there's also security permissions like for example you want to have security permission to create a pod but not to restore from a checkpoint because this will export expose sensitive data. So creating a separate uh essentially object for requesting the restore functionality to restore from a checkpoint is kind of what we agreed for now. And our next step will be to create

a proof concept and get feedback on that and how it's implemented. And I have a short demo here but it's it's really simple. It's just counter application and showing that this actually works and have also links to the um architecture and design documents. And because we are short on time, I'll just to transition to Peter. >> Okay. >> Thank you, Edison. Um, so yeah, I Victoria sort

of covered this a little bit at the beginning and um, you know, we've been hinting at it for a little while, but like you know, I think a big difficulty of doing something so large into Kubernetes is you have to really sort of like narrow down on the use cases that you're trying to complete and, you know, make sure we're honing in on that when we're proposing

an API that's going to be pretty complicated and have a lot of integrations. So, I'm going to talk a little bit about some of the cases that we're thinking about and then um some of the, you know, proposed piece like a sort of vague sequence of what we're thinking about doing and then throw a whole bunch of open questions at you. And I think part of what

we want you to take away from this is that this is very much a work in progress, but also very much something that we want participation from the community and end users from to make sure that we're making something that's useful for you and we're not missing any cases. So, um, we've sort of distilled down the different, uh, into three different use cases that we're sort of

thinking about. Um, the and they're pretty similar semantically because obviously you're going to be checkpointing and restoring, but it sort of depends on what the pod is up to while you're in the midst of doing that work. So, the warm start case is you have like a single, you know, some pod, you know, maybe a Java application or something like that or and then you want to

recreate it multiple times. Um, and so this is going to be each of the new newly created pods are going to be a brand new pod. We're not going to do IPA address migration or anything like that. Um, and then the second case we're sort of thinking about is like a sort of suspend and resume. So you have an existing pod and you after you checkpoint it,

you stop it. Um, and then you spin it back up after and it retains its identity. Um something that uh like a guiding sort of principle that I'm thinking about when thinking about this case is like how could we make it look as close as possible as the pod itself just got CPU throttled for that little period of time rather than actually like fully stopped and started

again. And then the final case is like a live migration case. Um and this one is probably one of the more complex ones that we want. um you know how do we sort of like transition uh the the pod state somewhere else while keeping the old one running but then you know once the new one is up stop the old one um so this is more like

as we're designing the API we want to be able to be flexible enough to cover these cases but this is not really an initial implementation thing so you know as we think uh so here's as a warning this is some ideas that I've had and we've talked about this as the leads a little bit but like I don't I think we've gotten consensus either with the Kubernetes

community or even working in group checkpoint restore. So take this all with a grain of uh salt. Uh it's just something that I'm thinking about in this process. But um in terms of our tier list of like what we want to support, like what we want to do as the working group, the first thing is like try to get as minimum of an implementation as possible because

the larger the scope, the more difficult it will be to get it in. Um, unfortunately that does include a whole bunch of different things including an entire at least one maybe two new APIs, an entire potentially entry controller, hopefully in tree controller, um, CRI implementation changes, cublet changes, and then, you know, different uh, cluster ro things to make sure that we uh, can protect users from uh,

anything malicious. So um, personally I want to sort of target the warm start use case. that seems the simplest to me because having uh not needing to retain pod identity between checkpoint and restore seems a little bit simpler. Um so that would make the the second stage of it sort of in like making making the first thing a little bit more useful. So um things like namespace

migration which we would have to think the rback case a little bit like should someone who uh is allowed to write to one namespace but not the other one should you be able to checkpoint or restore between them probably not. So these things like these things we want to think through thoroughly and figure out how we can make sure that that's a safe thing to do and

a secure thing to do. Um basic pod edits like some fields are probably safe to use uh to change but probably not other ones. So like um how do we audit which ones would be safe to change? And then um IP address saving that's I think a piece that's going to be kind of tricky to do but in the ideal world we would be able to do

it so that we can do the suspend resume case while you know holding open the IP address so that the pod when it wakes up it doesn't suddenly have something different. Um then the the the you know P2 the second uh the third stage of this work I think is when it's going to start becoming really materially useful for a lot of people which is when we

get ecosystem integration. So we think of things like you know what does it look like to checkpoint a deployment. What does it look like to checkpoint like a running job? What does it look like to have Q think about um preempting a job with a checkpoint rather than with you know by just stopping it? or what does it look like to have like an eviction manager in

the cubelet be checkpointing rather than stopping. So these sort of cases, they're sort of more uh up in the air right now, but they're things that I think start to really answer um the open questions that or the things that people want out of this feature. And then there's a number of things that we have sort of way off in the distance. Um the biggest one being

like node migration which I think is going to be really tricky actually because of how strict CRU is with um you know kernel requirements and dynamic libraries and things like so we have to figure out like what like who is the one who decides whether a pod is able to be restored on a new node. So these things I think and also what would the API look

like uh for that. So this thing is quite far away but something that we're thinking about again as we sort of design this feature we want to make sure that we have all of these cases in our mind and also like as um everyone here is listening to this and thinking about the ways that this could solve use cases uh in their clusters like we want to

hear those cases as well to make sure that we're not ignoring them in the design process. So now um I've hinted a little bit about all of the open questions and there are a lot. I mean you know in 136 we didn't we didn't even get uh enhancement merged. So we're still very much in the beginning stages pre-alpha of this feature. We're hoping to get some work

done in 137 concretely in the cube community. But a number of the open questions that we have um sort of are a culmination of all of the different communities that are participating in this. you know, as a working group in Kubernetes, there's a whole bunch of different uh entities that are have a stake in this process and we want to make sure that we in the design

process uh have something that can please everyone who is involved. So, um some of the open questions that we think about uh you know, so I'm like break broke it down the open questions vaguely by SIG that is participating. So like you know something a sig node uh lead might have a question about is like how closely should a checkpoint be related to the pod? Do we

have a field in the podsp spec or do we have an extra API uh for restore like um Rison was talking about? How do we cover multiple checkpointing implementations? So right now we know about Cryu but also G Visor um uh the container runtime by Google is also like an implementation that can do this today. How do we make sure that the API is flexible enough to

be able to handle these multiple implementations but not also be like way too generic or know too many things about the end case? Um, and then are there future integrations with evictions? Like do we want there to be a world where a cublet instead of stopping a pod uh checkpoints the pod? How will that sort of affect notability and things like that? We want to be playing

with these things but we don't want you know to block the initial implementations on this. Um a big uh set of open questions comes from the security posture of this like you know checkpointing a pod. Uh if you checkpoint a pod and you're able to inspect the checkpoint archive you can see the entirety of the memory space. You can see all of the you know all the

basically everything that the pod was up to. Um how do we make sure that you know a cluster admin can enable this feature and not worry about you know a malicious user who has access to the cluster stealing information uh secrets or things like that. So we ask you know who is allowed to checkpoint a pod is it just the pod author or is it like you

know is it cluster role that we're granting we're sort of trying to figure this out in the same way how who may restore a pod because if you're like I hinted at the namespace migration piece before but if you're able to move a pod from one namespace to another even though namespaces are pretty poor isolation boundary they are sort of like what the community has sort of

backed as an isolation boundary in kubernetes and multi-tenant kubernetes so how do we make sure that uh we make it pos safe to restore a checkpoint and then like what granularity of pol like should policy engines have access to. So like should you should we have local like both the opportunity to do local namespace uh checkpointing or global checkpointing? Uh same thing with restore like like what

would be useful to people to make sure that this is you know something that they would want to enable by default rather than like be very cautious of or only do it on special clusters. Um lots of questions about you know where are these checkpoint archives going? I mean especially if we're checkpointing GPU memory like that's a huge amount of uh work to do. It's like a

very especially for you know larger pods or just even like thinking about a job if you're checking checkpointing 10,000 pods like where are we going to put that? Um so and especi like thinking about like where how do we move checkpoints between nodes like what is the entity that checkpoints uh that moves checkpoints between nodes. I think part of the advantage of having a inry controller is

maybe that should be the entity that takes care of that. I don't think the cublet should do it, but like if we didn't have a controller, who else would? Um, and how do we make sure we store we can store these in different storage mediums? I mean, especially as we think about um incluster like between node migration, but even just generally like you want to be able

to put it in a GCS bucket or so like a S3 storage, but like how do we make that um API sort of clear and um possible. Um couple of questions from sig gaps and this is like thinking about in the future of like you know the P2 level of like how do we checkpoint higher level objects. Do we have a different API for each of these

different objects or is there a way we can express that without that? Um and then also how do we handle uh objects with like workloads with special checkpointing behavior. If you think about cubvert like you could imagine a VM would be much easier to checkpoint and restore. there already exists that uh possibility but like how do we make it so that the API uh can be used

the same way for cubevert or like a pietorch process which has a special workload level checkpointing while still allowing for there to be a fallback in cryu or g visor or something like that and then you know here's we only we didn't have that many other open there's a number of other open questions but these are still the remaining big ones like how are we going to

do IP address migration how are we going to save that IP address and be able to you know use Um and then um how do we make sure like who is the entity that makes sure a restore will fit on a migrated node. So um you know and this is obviously much much in the future for the migration node migration but this is something we want to

be designing and thinking about. So, this is uh with all of that, this is a call to action. If you want to get involved or if you have any answers to any of the questions or any sort of suggestions or thoughts or new questions that we haven't uh mentioned, um we would love it if you got involved um and joined us uh at the working group meeting.

Um it's currently running weekly and uh maybe we'll dial down to bi-weekly. We'll see. Um, also if you're interested in learning more or seeing it more practically done uh in a couple of hours, you know, as mentioned earlier, there's another talk um about this and also here, which is cool. So, you don't even have to leave unless you want lunch. Um, here's a session form for feedback.

Um, and we'd love it if uh to take any questions. If you have any questions, there's mics around. You need to go and stand at the mic to ask us so everyone can hear. But thank you. Hello. Uh quick question. I wanted to like ask you what real case scenarios do you already like see that are like implemented proprietary because you mentioned like a lot of directions

and I really like wonder like what somebody else is having because of course most of us don't have Borg in the backyard. >> Yeah. Uh I mean there are several implementations most of them are like mod using for example um container dim to implement the checkment restore functionality and trying to find a way to extend Kubernetes and the container runtime to to make it work and we

are essentially talking to different people and uh seeing what uh challenges they face and figuring out how to solve them. Like for example using the same IP address after migration is one of the key challenge that everyone has and for example with Calico with the CNI plugin you can just specify a static IP address and this seems to work but then how do you manage this for

different workloads and different use cases is is important and then GPU migration GPU checkpointing is also important like you have uh different driver versions you have different types of GPUs and you know uh there are many oh Essentially we try to focus on specific use cases and then define the requirements and from there figure out what is the minimal scope that we can essentially propose to the

upstream community. Um that did that answer your question? >> Uh yeah I guess it did. Thank you very much. >> And one other thing um just there we've had actually a handful of um of use case calls. I forget what we called it. um examples of like proprietary op like uh uh proprietary examples of this on our earlier calls of the checkpoint restore working group. So if

you look on YouTube, you can actually find some some use cases that people have said and these are things that we're using to like guide the implementation a little bit. >> Perfect. Yeah to be honest we created the working group so that we can combine the input from different people and have an open discussion about it and yeah >> um maybe it would be any somewhere in

the docs issues whatever uh have you thought about use cases along the line of like um freezing uh pots stful sets deployments for backup purposes basically to allow at least the storage side to be crash consistent while the backup solution takes a snapshot. >> Yeah, I mean the that was one of the examples that we sort of talked about there is the CSI notion of the snapshot.

>> We're not really in the business of freezing so much but while the checkpoint is happening we do freeze the pod. Yeah, this what I was from. So, we actually discussed during the working group meetings that we sort of need pow state for a pod. Essentially, when you're check when you're creating a checkpoint, the pod still appears as running, but it's not actually running. It's being frozen.

And uh this is also quite useful for another use case we have called hot swapping. When you have like a limited amount of GPUs and you want to switch essentially to move the GPU state into host memory and then reload different workloads. So this pausing the pod and then uh checkpointing on the GPU memory is one of the use cases we have in export. But yeah, essentially

having pause state functionality in Kubernetes is it's kind of similar to what Docker and Podman already support but it's missing in Kubernetes. Yeah. Okay. >> Thanks. Thank you. >> Hi. So when you're taking a checkpoint at the pod level, uh there might be more containers in the pod. you probably don't want to checkpoint uh any init containers for warmstar uh possibly not even other sidecars. Any thoughts

on >> Yeah, we we actually received this back uh feedback earlier on and uh for now we decided to uh sort of focus only on running containers. So we assume that the init containers have already finished and uh if you have any stopped containers not in a running state, we don't checkpoint them. we we can save that the state that they're not running but uh essentially we

kind of try to limit the scope for now and uh for side cars we for now they're out of scope uh for the enhancement proposal but we we have to define the use case how it's going to be used and then figure out what is the best way to to support it >> yes thank you >> okay thanks >> uh thank you for all these efforts really

amazing um I have a question so we run a lot of shortlived Python workloads like patch jobs and we lose a lot of time on Python startup because it's cold start every time. Have you considered the use case when it's possible for me to checkpoint like I don't need actually the entire pod or container I just need to checkpoint the warm-up stage and then change the inputs

that go for it but I don't need a web server so that kind of use case will it be possible with something like this? >> Yeah, this actually quite popular with serverless platforms. There are a few companies out there that already have functionality and use this in production. Um we have a talk later today is talking about PyTorch jobs and batch jobs and how we can use

this and I we have some interesting results. If you want you can join but to answer your question uh last no in I think in 2024 there was a talk uh I think it was >> called the party you must go on but essentially it was exactly the same use case but uh they essentially install Creo inside of the container and then they create the checkpoint inside

of the container then during restore they just invoke Creo and this this works but you need uh permissions so preu has requires root access so you need to create the privilege containers to do that. >> Okay, thank you so much. >> Yeah, no problem. >> I will say probably for this the initial iteration of the ke, we probably will be checkpointing the entirety of the pod. Um,

but I think keeping in mind partial checkpoints is definitely relevant. >> All right. Uh, so how close do you see the checkpoint restore um um functionality to application consistent backups which are maybe already existing? Is it a clear boundary for you that uh you say okay this is something completely different or are there is is are the fronts less clearly defined and that you say okay we

allow already existing application consistent backup strategies to flow into this checkpoint and restoring mechanisms >> yeah this actually very good question I I think we should discuss it in the working group meetings if you want you can join we have a meeting every Thursday um but yeah it's if this application specific we have to figure out what is the application how does it work and then what

is the best way to integrate it? Um we have for example the same problem with distributed checkpoints where we have to synchronize to coordinate the checkpoint across multiple nodes across multiple pots and make sure that it happens at the same time and the pots are frozen at the same time. So it's kind of u you know there are many questions but um yeah essentially join one of

the meetings and we can discuss. >> All right perfect. Thanks a lot. Thank