Snapshots Gone Wild: Taming Multi-PVC Chaos with VolumeGroupSna... Shubham Pampattiwar & Scott Seago

This talk covers the challenges of backing up multi-volume stateful applications in Kubernetes and how volume group snapshots can address these issues. The speakers, Shubham Pawar and Scott Siegel, both principal software engineers at Red Hat, explain how traditional backup methods can result in inconsistent states due to different timing of individual snapshots. They introduce the volume group snapshot API, designed to create atomic and crash-consistent snapshots of multiple volumes simultaneously. The session delves into the integration of this feature into Velero, a popular open-source backup and recovery tool for Kubernetes, and discusses practical implementation, testing experiences with various CSI drivers, and the benchmarks of performance improvements. The importance of simplifying backup strategies and ensuring data integrity through proper configurations and practices is emphasized.

Hello everyone. How many of you have backed up a multi-volume stateful application and prayed that the restore would work? Yeah, so us too and that's what this talk is about. Today we're going to show you why those prayers were necessary and how volume group snapshots solves it. A little bit about us. I'm Shubham Pawar, principal software engineer at Red Hat. I work on OpenShift engineering team, part

of the OpenShift API for data product and I've been a Valero maintainer since 2022. And I'm Scott Siegel, also principal software engineer at Red Hat, also a Valero maintainer and a and a member of the OpenShift team. Okay, so a quick show of hands, how many of you have used Valero? Okay, great. For those of who haven't, Valero is an open source tool for backup, restore, and

disaster recovery on Kubernetes. It handles everything, cluster artifacts, resources, persistent volumes, schedule backups, you name it. It supports multiple volume backup methods, CSI snapshots, file system backups, native cloud provider snapshots. But today we are focused on CSI snapshot path because that's where volume group snapshots plugin. One quick announcement, Valero's CNCF sandbox application was just approved by the CNCF technical oversight committee last month. So this is a

great moment for us in the project. Thank you. Yeah. And before starting, just like to say that we like to think backups should be simple, reliable, and ideally so boring that they never make headlines. So let's get into it. So this is going to be our roadmap for today. We'll start with why multi-volume backups break, then Scott will walk you through volume group snapshot API, the community's

answer that solves this. Uh then I'll show you how we brought volume group snapshots into Velero and the design decisions that we took and the implementation aspects. Uh then Scott will share some war stories from real CSI drivers testing, like how things broke in ways we didn't expect. And we'll close with some results, benchmarks, and a blueprint for adopting volume group snapshots in your environment. So, the

problem. Uh think about a database application uh like PostgreSQL, MongoDB, Cassandra, anything. So, they typically spread data across volumes. Like you have got your data volume with table and indices, uh you have a write-ahead log volume, uh you have a config volume with certificates and settings. So, in this example, there are three PVCs, uh three PVs, and this is all a one whole logical application. So, this

is the norm for um any serious stateful workload deployed on Kubernetes today. Like uh databases, it could be analytics pipelines or message queues. So, what happens when you when you try to back this up, right? So, here is what happens today. Uh with individual snapshots, uh your backup tool uh snapshots the data PVC at T equal to zero, let's say. But, the application is still running. And

writes are still happening on the disk. So, a second later, the backup tool snapshots the write-ahead log PVC at T equal to one. But, here is the key. Between those two snapshots, let's say 10 writes happened. So, your data PVC has a state before those 10 writes, and your write-ahead log PVC has a state after those two 10 writes. Then more writes happen. The backup tool takes

the snapshot of config PVC at T equal to two. So, all in all, three snapshots at three points in time. So, your backup now contains a state that never existed in your application in reality, right? Ever. So, the data volume thinks that end rights never happened, but the write-ahead log volume thinks they did. So, when you restore this, your database most likely will get corrupted. And this

isn't theoretical. Like, here is what actually happens in different scenarios when you restore such apps. database with write-ahead logs out of sync, um you'll see corruption. The write-ahead log references data that doesn't exist. Uh in case of Kafka, you might see lost messages or duplicate delivery. Uh in case of Cassandra, you might see split-brain behavior. Like, nodes come up in inconsistent state. App config doesn't match with

the data, you might see the app crash at startup. but here is a deeper issue. Your backup reported success. Everything looked green, right? It's the restore that fails. So, individual snapshots in such scenarios give you a false sense of security. So, what do we actually need? So, look at the left-hand side. This is what we just talked about. Uh three snapshots, three different points in time, inconsistency.

Look at the right-hand side. One API call, create volume group snapshot. The storage then storage layer captures a point-in-time snapshot of all the volumes in the group atomically. All three volumes are captured at the same exact time. Data PVC, write-ahead log PVC, or config PVC. Same point in time. Atomic and crash consistent just via one command. Now, Scott will take us through the volume group snapshots API.

Okay, so the volume group snapshots were introduced via kept 3476. Um they were for it's a V1 beta one and Kubernetes 132 uh beta two and 134 and they're uh currently slated for GA in 136. Um, basically following the existing pattern, volume group snapshot class similar to volume snapshot class, volume group snapshots um from volume snapshots and the volume group snapshot content from the volume snapshot content.

Um, basically the volume group snapshot class will define the driver and the deletion policy. Um, the volume group snapshot itself, that's a user request um which identifies the included volumes via label selector. So, that's that's the user saying, "I want to these volumes to be snapshotted together and in one atomic operation." Um, volume group snapshot content is then the storage way group snapshot which in turn uh

contains the volume snapshot and volume snapshot class. So, here we see the workflow under the hood. Uh the user or an application such as Velero uh will create the volume Then the snapshot controller will create the volume group snapshot content and the CSI sidecar handles the provisioning. Uh it basically makes a create volume group snapshot call to the CSI driver, which creates an atomic group snapshot. Um,

and then the group and individual snapshot IDs get returned, and the status gets updated uh for the volume group snapshot content, and we create the volume snapshot and Let's step back a minute and talk about backup consistency. This can mean different things depending on context. Volume group snapshot aims to solve the problem of crash consistency. Uh in this case, all volumes are captured at the same instant.

This is equivalent to pulling the power plug on a local machine. Um, recovery relies on the app's own crash recovery scenario, uh write-ahead log replay, journal replay. Uh it's automatic, no app cooperation is needed. However, at a higher level, we have the notion of application consistency. This is an additional level of consistency which requires coordination with the backup tool and the application. Uh and basically the application

closes writes first, flushes buffers, closes transactions, then we create the volume group snapshot the uh the snapshots to get cut, state, and then um everything gets restarted. So, this requires app-specific hooks, coordination between the backup tool and the applications. Best practices here is that the more frequent frequent backups, say nightly, mainly need crash consistency, no need to shut applications down even for a few seconds, and that

can no freezing is needed. But maybe once a week you have you want application-specific backups, that requires some small downtime, and you can combine that with the pre post hooks with your application to get application consistency. The volume group snapshot handles the storage consistency, and the hooks connected to your your backup application will handle the application Oh, thanks for that. So, we had a challenge, right? Velero

already had a mature CSI snapshot pipeline. PVC to volume snapshot, volume snapshot to volume snapshot content to storage snapshot. So, it works. People depend on it, and users have been using it extensively. The question that we had to answer was, how do we add group snapshot support without breaking anything, like for existing users? We followed a simple design principle. We kept volume group snapshots opt-in. No label

on your PVC, nothing changes, everything works exactly as before. And if you have a label for grouping, you get atomic group snapshots across the volume group. And same restore flow either way, and the key inside here that I would like to highlight is we use volume group snapshots as an orchestration mechanism in the backup workflow, not as a new backup format. So, here is how Velero decides

what to do. Like when a PVC comes up for backup, Velero checks two things. One, does it have a volume group snapshot label? And is data mover enabled? That gives you four modes. VGS, that is volume group snapshots with data mover, you get atomic snapshots across the volume group, and they're moved to object storage, and this is useful generally for cross-cloud disaster recovery. And if volume group

snapshots without data mover, in that workflow, you get atomic snapshots on local storage. They are mostly used for fast recovery. If you don't have volume group snapshot label, you get individual snapshots with or without data mover. Same as today, legacy workflow. No changes there. Velero picks the right mode automatically, and users don't need to think about it. So, here's the backup workflow at a glance for volume

group snapshots in Velero. You start with labeling the PVCs, and the step one that is involved here is grouping. So, Velero groups them by label value, and creates item blocks. Then, step two is snapshotting. The CSI plugin, Velero CSI plugin, creates a volume group snapshot and waits for individual volume snapshots to be ready. And step three is cleanup. That is we delete the volume group snapshot artifacts

and keep the individual volume snapshots. Uh the CLI stays the same, same as always. Velero create backup create backup name included namespace namespace name. No new flags, no new configs, just a label on your PVCs. So, let's go over each step individually. So, grouping, uh look at the YAML on the right side. Uh we just added one label. That's it. velero.io/volume group db1. All PVCs with the

same label go into the same group. Velero puts them into an item block so they're processed together. And if you already have labels on your PVCs that group them, you don't even need to add Velero specific label. You can configure any custom label uh per backup or server wide. The app blocks PVC here in the example has no label, so it gets an individual snapshot, and that's

business as usual. Same label value equals same group. That's all the configuration like you need to know for volume groups. Step two is the step where magic happens. The item block with all the three PVCs goes to the Velero CSI plugin. The plugin creates single volume wraps volume group snapshot. Just one CSI call to the driver. The driver talks to the storage layer. All three volumes are

snapshotted atomically. And the outcome is three volume three individual volume snapshots. DB data, DB writer head lock, DB config for our example. All pointing to the same instant in in time. Just one CSI call. Um step three is my favorite design decision in the whole feature. Uh after the snapshots are ready, we delete the volume group snapshot and volume group snapshot content. Gone. On purpose. So, you

might ask why. So, three reasons. First is uh restore portability. Your target cluster doesn't need volume group snapshot CRDs installed. You can restore to any cluster that has standard volume snapshot support. Second is life cycle management. Uh the individual volume snapshots follow Velero's existing deletion flow. So, no new cleanup logic was needed there. And thirdly, uh no orphans. Like, if you kept the volume group snapshot around,

accidentally deleting it would cascade delete all the child snapshots. Uh we don't want to do that. So, the key insight here is volume group snapshots in Velero's backup workflow are transient by design. It does its job during the backup and then gets out of the way. A quick note on how Velero picks the volume group snapshot class. Uh three-tier mechanism. Uh PVC annotation overrides everything, then followed

by that backup annotation, and then default uh Velero label. You can label your volume group snapshot class that you want to use for your backup. Uh you only need the tiers for advanced multi-driver setups if you have those in your cluster. there might be a scenario like what happens when your app has PVCs on different storage backends, right? Say you have DB data on EBS CSI and

DB data at log on EBS CSI, but your config PVC is on NFS. And all of them have the same volume group snapshot label. Without volume policies, the backup Velero sees multiple CSI drivers in the group and stops. With volume policies, the NFS PVC gets filtered out before grouping happens. So, volume group snapshot proceeds and just with the EBS PVCs. The key thing to remember here is

volume policies are evaluated before grouping. That's it, and this was added in Velero 1.18. So, we're now going to talk about the war stories, basically things we've come across in testing that surprised us. Sometimes you run into unexpected surprises when you start testing a new environment with a new driver, new cluster. We deployed VGS backup to a Ceph RBD cluster. We initially done our testing with CephFS.

Everything worked fine with CephFS and this RBD cluster. The backup was fine, but then we tried to restore. We got this error, volume group snapshot volume snapshot content reconciliation failed. Volume group snapshot handle is set, but no matching volume group snapshot content found. Here was the problem. Ceph's RBD driver populates volume group snapshot handle on every volume snapshot content that was created via the group snapshot. However,

we didn't retain those between backups as you might have mentioned earlier. We cleaned those out because in most cases they're not needed. In addition, if this was for a restore to a new cluster, it wouldn't even be here if we had retained it. So, during restore, the CSI snapshot controller sees this field, requires the to exist. We deleted it, so it failed. The solution, we needed to

introduce some stub VGSCS. So, in this case during backup, the Velero volume snapshot basically captures this volume group snapshot handle as an annotation if it's present. So, if it's there, we track the annotation, so now we know on restore time if it's needed. Uh we now have a restore action which detects this annotation. If it finds it, create this stub VGSC matching the group snapshot handle, individual

snapshot handles, the retain policy, and basically and the restore name, which is a an internal label label for cleanup. Um at this point, after the restore, we wait for all the VSCs to become ready to use, then we delete the stub. We don't need it anymore. End result here is Ceph RBD restore works. Stub VGSCs are temporary scaffolding. They exist just long enough for the killer to

reconcile, and then they're cleaned up. Um one thing of to note, um Shimada just mentioned we don't necessarily need these CRDs defined in the cluster. If you're using Ceph RBD, you can need them. Um this is again one of the one of these issues where every driver is different. You got to test. Um and then and then and that's that's the point here. We learned a few

lessons here. Um drivers are not uniform. There could be subtle or not so subtle differences in behavior from one driver to another. The volume group snapshot API defines what to do. However, each driver decides how to do it, and they might do it in different ways um that might be externally different from each other. Um some drivers populate the volume group snapshot handle, some don't. Some require

the VGSC to to be there during reconciliation, some don't. Some return the snapshot handles immediately, some are async. Um so test across multiple CSI drivers, the API spec is necessary but not sufficient for determining whether everything's going to work for you. Um thanks, Ward. Uh so we have seen the problem, the API, the implementation, and what can go wrong, right? Like so let's talk about results. So

here's the comparison uh side by Uh individual snapshots give you per volume consistency only. Volume group snapshots give you crash consistency across the entire group of volumes. Individual snapshots make N separate CSI calls, VGS makes only one. Individual snapshots fail per volume, you might get a partial backup, right? But VGS is all or nothing. It either works or not. The trade-off is that for volume group snapshots,

you need a CSI driver that supports volume group snapshots. And you need to add a label to the PVC for grouping. That's it. Um I'll just follow a simple thumb rule here. If you have complex multi-volume stateful apps, as we saw the example, use volume group snapshots. But if you have a single volume or independent volume app, yeah, you can go ahead with individual snapshots as usual.

So, here are some real numbers from our testing. Uh so, we used uh Kubernetes 1.32 uh in the environment for testing. We used Kubernetes 1.32 and the storage used was CephFS. Uh the PVCs were 1 gigs of size and Velero 1.17 was installed with volume group snapshot support. So, at three PVCs, volume group snapshot is 1.3 times faster. Modest. At 10 PVCs, it's 1.5 times faster. At

20 PVCs, it's 1.6 times faster. The speed-up is nice, but that's not the point. Let's take a step back. Look at the consistency story here. At 50 PVCs, individual snapshot took 266 seconds. That's 4 minutes and 26 seconds where rights are happening between the snapshots. Right? 4 minutes of inconsistency risk. That's a lot. Volume group snapshots, one CSI call, zero inconsistency window. All 50 volumes captured at

the same instant, same point in time. See, the performance win is just a bonus, I think. The consistency win is where you you should use volume group snapshots for. Um we just have a short demo. Let me just So, this is like uh how we implemented uh volume group snapshots in Valero, right? Just a short demo for that. Uh pre-recorded, of course. So, we have a little

monitoring tool just to visualize things um properly. Uh you can see the backup name that we're going to use is demo VGS backup and the namespace is demo database. Uh whatever steps we discussed, we are going to go through them step by step. Uh the volume group snapshot class used is SFS for this demo and we have labeled it uh using the Valero label. the namespace has

three PVCs. Uh they're labeled for grouping uh with the value as DB cluster one. You can see the PVCs are bound, DB config, DB data, DB write-ahead log. Uh we are just waiting. The monitoring tool is just waiting for the backup to be triggered. Now, let's kick off the As I mentioned earlier, simple same old Valero backup command, Valero backup create, backup name, and included namespace, data

demo database. Uh once kicked off, let's monitor. So, you can see the backup is in progress and we are at the grouping state right now. Now, next is creation of volume group snapshot. You can see the volume group snapshots highlighted in yellow. Uh they got created. Now, individual snapshots will get created based on And you can see all three volume snapshots were created in the same Now,

once they're ready, uh we delete during cleanup and the individual volume snapshots, right? Those get cleaned up during the finalizing phase uh in Valero. After that, the backup backup is marked as complete. You can see the volume group snapshot artifacts highlighted in red were deleted. And you can see the backup completed in 14 seconds. Uh three PVCs were involved, one volume group snapshot, and three volume snapshots.

That's it. that was a quick demo just to see how things work in Valero right now. continuing again. Yep. So, let's go over the blueprint on how to adopt volume group snapshots today. Uh so, you want to try this, right? Like, so here's the checklist. Five prerequisites. Uh most are one one-time setup. You need Kubernetes 1.22 and later. That's when volume group snapshots went beta. You need

the volume group snapshots CRDs installed, of course. And your CSI driver needs to support Best way is to read up the documentation of the driver, whether it supports or not. Uh or you could just check your cluster by fetching the volume group you need a default volume snapshot class. Uh that's what we observed, and this is one of the most gotcha common gotchas uh that the snapshot

controller requires. And obviously, you need Valero 1.17 or later. Uh five steps, that's it. Uh this took about a year from design proposal to production. Uh started with a design doc in March 2025. Worked on PVC grouping, then core workflow implementation, documentation, followed by that volume policy integration. Uh then in February 26, we hit the safe RBD restore issue. And just last month, Valero's sandbox application was

approved. Yeah, every step uncovered new challenges for us. That's what I would like to say and that's we tried that's what we tried fixing. Uh, what's next? Volume Group Snapshots is production ready but still evolving. The Volume Group Snapshot API itself is moving towards GA in the future Kubernetes release, most likely 1.36, I we're working on validating more CSI drivers beyond safe. More storage backends we test,

the more edge cases we uncover. try Volume Group Snapshots in your environment, file issues and that's how every fix in this talk happened. takeaways, five things to take away from this talk. Number one, if you have multi-volume apps, need atomic snapshots, individual snapshots create inconsistent backups that fail on Number two, Volume Group Snapshot is the answer. Crash consistent atomic snapshots via the Kubernetes API. Uh, number three,

Valero makes it simple, just label your PVC. Same CLI, same restore flow. Number four, VGS uh straight transient by design, atomic snapshots during backup, standard Valero life cycle after. And number five, test across CSI drivers. The API spec is necessary but not sufficient. Uh, if you remember two things from this talk, just remember four and five, I would say. Uh, we're around for the rest of the

conference. Come find us if you want to talk about data protection, disaster recovery on Kubernetes, Volume Group Snapshots, Valero, or if you want to share some war stories from your backup stories. That's it. Let us know if you have any questions. Thank you.