Kubernetes Data Protection WG Intro & Deep Dive - Dave Smith-Uchida, Veeam

This talk provides a deep dive into the Kubernetes Data Protection Working Group, led by technical leaders from Veeam and VMware. The session covers the significance of data protection in Kubernetes, particularly for stateful workloads, detailing the operations needed for backing up both Kubernetes metadata and persistent volumes. The speakers explain existing solutions like COSI for backup repositories and introduce concepts such as volume group snapshots and change block tracking to enhance data protection practices. Additionally, they address challenges faced by operators in managing application states and dependencies during backup and restoration processes. The session emphasizes the need for standards in operator quiescing and best practices in backup strategies.

Uh, thanks for coming to our session here for the Kubernetes data protection working group deep dive. I'm Dave Smith Uchida. I'm a technical leader at Veeam where I work on Kubernetes backup and recovery. And uh, this is Xing Yang. Hi everyone. My name is uh, Xing Yang. I work at VMware by Broadcom. I'm also co-chair of Kubernetes six storage and the data protection working group. Here's today's

agenda. First, we'll talk about why we established this working group, who are involved, and some of the projects we are working on, and the white paper that we are working on, and finally, how to get involved. In Kubernetes, day one operations for stateful workloads are well supported. We have persistent volume claims, persistent volumes for the volume operations. We have workload APIs such as stateful set with deployment

for the declarative management of your stateful workloads. More and more stateful workloads are moving to Kubernetes. On the other hand, day two operations for stateful workloads such as data protection are still limited. That's why we formed this working group to find a better solution. Here are the companies supporting this initiative listed here. This diagram shows the backup workflow with the existing and missing building blocks in Kubernetes.

The blue color shows the process. The green color shows existing Kubernetes components. And yellow means it is work in progress. Orange means it's a missing Kubernetes component. To backup a application in Kubernetes, we need to backup two pieces of data. We need to backup Kubernetes metadata. We need to backup the data stored in persistent volumes. There are two ways to backup the data stored in the volumes.

You could use native data dump such as MySQL dump, or you could use controller coordinated approach while a volume snapshot is created. To ensure application consistency, you should quiesce the application before taking the snapshot and unquiesce afterwards. Both the Kubernetes metadata and data will be exported to a backup repository. A backup repository is a location or repo where you can store your data and COSI is a

project that can be used to support a backup repository such as an object store. I will talk more about COSI and a few other projects shown here later. This diagram shows the restore workflow To restore application, we first need to import backup from the backup We need to restore Kubernetes metadata, and we need to restore PVC and PVs. If the PVC was uh, backed up natively, we

need to restore from the native data dump. Otherwise, we need to rehydrate the PVC from the volume snapshot or a volume backup. Now, let me talk about the backup repository. As mentioned earlier, COSI can be used to support backup COSI container object storage interface provides standard APIs so that a user can provision an object bucket just like how a PVC is provisioned. There are several COSI components.

We have a COSI controller that manages the life cycle of a bucket. It binds COSI created buckets to bucket claims. There's a COSI sidecar that watches COSI Kubernetes API objects, and it calls the COSI driver to provision the buckets. There is a COSI driver that communicates with object storage backends, and perform bucket related operations. COSI is moving to v1 alpha 2 in Kubernetes 1.26 release. There are

two sets of Kubernetes APIs. The relationship between bucket, bucket claim, and bucket class is very similar PV, PVC, and the storage class. We also have a set of APIs that allow a pod to access a bucket. COSI also has gRPC interfaces for object storage vendors to create or delete a There are some notable v1 alpha 2 changes. Bucket access can now request access to multiple bucket claims.

Previously, it can only request access to one. There are also changes to the bucket access modes. There are three types of accesses. Uh, read-write, read only, and read only. There are also three types of data, object data, object metadata, and bucket metadata. So, you have you can have different combinations of access modes to different types of data. As shown here, a bucket claim is a user's request

to a bucket. A bucket represents a physical bucket on the storage backend. The bucket class is defined by the admin uh, to specify what kind of a bucket can be provisioned. The bucket claim and the bucket have a one-to-one relationship. They are bound to each other when they are provisioned. As shown here in the bucket access class, you can specify the authentication type. And here we specify

that using this access class, you can have access to multiple buckets. Also, here we specify that the read-write and read only access mode are disallowed for bucket metadata. So, using this access class, you can only have read only bucket uh, you can only have read only mode to access the bucket metadata, but you have no restrictions to access object data or object metadata. In the bucket access,

you specify the bucket access class name and the number of bucket claims, and also a service account name. And uh, to use the bucket, user creates a pod with a projected volume pointing to a secret in the bucket access. We mentioned earlier that to ensure application consistency, you need to quiesce the application before taking a But what if you cannot quiesce the application, or quiesce the application

is so expensive, so you want to do it less frequently, but you still want to be able to take a crash consistent snapshot more frequently? Also, some applications may require user to take a snapshot of multiple volumes at the same point in time. There's also a performance element here. That's why we added support for consistent group snapshot. This feature is tagged in GA in We have a

set of Kubernetes APIs to support volume group snapshot. So, you can take a snapshot of multiple volumes at the same point in time to ensure write order consistency. There are also CSI spec changes. We have a new group controller service and a set of new gRPC interfaces to support volume group snapshot operations. A volume group snapshot is a user's request for a group snapshot. A volume group

snapshot content represents a physical group snapshot on the storage system. A volume group snapshot class is defined by the admin to specify the kind of group snapshot to be provisioned. Uh, to create a volume group snapshot, first, you need to identify the PVCs that you want to be snapshotted together. Then you apply a label to those PVCs. In the volume group snapshot, you specify the volume group

snapshot class name and the label selector, and make sure that you use the same label that you applied to the PVCs. And when the the volume group snapshot and volume group snapshot content have a one-to-one relationship, they are bound to each other after they are created. As a result of creating a volume group snapshot, you will also get individual volume snapshots. Each is corresponding to its source

persistent volume, and you can use the volume snapshot to create a PVC from it. Now, let me talk about change block tracking. Change block tracking allows you to identify change blocks between two snapshots of the same persistent volume to enable incremental volume backups. Now, without this feature, a backup vendor will have to call each vendor's API for each storage system, which is highly inefficient. Now, we have

vendor agnostic Kubernetes So, a backup vendor can decouple their backup system from the storage system and enable efficient backups. To support this feature, we also added a new snapshot metadata service and the new CSI RPCs. We also added Kubernetes APIs to support this feature, so backup vendor can retrieve the snapshot metadata. This feature is targeting beta in Kubernetes 1.36 release. Now, let me hand it over to

Dave, who will talk about the white paper. Thanks, Xing. Um so, we had develop we've done the first white paper for uh the data protection working group, which pretty much lays out why you need backup for Kubernetes and when you would need it. And now we're working on a second white paper, which where we go through what are going to be the best practices both for uh

admins, developers, and so, what we try to do here is go through all of the mechanisms that we have. So, in Kubernetes, we have a lot of different mechanisms available and different strategies available for protecting your data, but you need to understand what the right strategy and what the right what your needs are. So, first thing you need to do is understand the resiliency needs of your

application, and then you need to select the right strategy, and then maybe you may even have to make modifications to the application to handle that. So, these are things that we're going to be covering in the white paper, and today though we dive into some of the challenges that operators pose and what operator developers uh can do to help things out. So, what's an operator? So, my

definition of an operator is something that manages an application based on custom resources. So, these are controllers, and when you write a custom app custom resource, for example, for a database, it would allocate all of the resources necessary to run that database, a stateful set, a um whatever else it would need, and then actually make sure that the application is up and running properly. The operator often

handles upgrades of the software, and uh sometimes they may even manage the backup and restore or replication of the of the application. So, what kind of challenges does this give us? Well, uh I'm going to go into each of these in more depth, but one of our first problems is figuring out what the order of creation of resources on restore is. We need and in order to

do that, we need to figure out the relationship of managed resources with the controlling CR. Uh quiescing the operator is something that's often useful, and uh being able to use the built-in data protection operations of the operator is another area. Um as we see more advanced operators, we have operators calling other operators, which makes things even more fun. And then the operator itself may need to be

backed up, restored, or at least um brought back to a useful state after a disaster. So, why do we need to know the order of creation on restore? Why can't we just create all of the resources and let the chips fall where they may? Well, the reason for this is that the operator is going to create things in a top-down manner. So, if you put your database

uh custom resource there, it's going to go and look and reconcile and see, "Hey, do I have all the pieces that I need to support this database?" If they're not there already, it goes ahead and starts creating them. And that winds up with a race condition with the restore operation. So, you can wind up with a fresh empty database instead of a database containing the data you

expected. In order to get around that, we can restore the uh the resources from the bottom up. So, we might create the PV first, then the PVC, then the stateful set, and then finally the resource that actually would map to those, so the the database resource itself. But, we need to to figure out what's the hierarchy of these resources. So, this is um kind of a challenge

in Kubernetes because things are basically present uh presented in a flat manner inside of Kubernetes. One thing that does help is if we get uh owner refs back from the various resources back to the custom resource itself. And uh this is important both for if we're restoring directly, where we might create all of the resources from the backup utility, or even if we're using the data protection

operations, we need to understand that the operator is responsible for doing the backup and restore of the data, and that these resources don't or shouldn't be protected, and they certainly shouldn't be restored individually. They should be we should use the data the operator's data protection operation to create and restore the data in there. So, not every not all of operators set these correctly, and so we need

lots of heuristics and special cases for figuring this out. Um another thing that can help out is if we can actually quiesce the operator during a backup or restore phase, since the operator may decide that it needs to tune something or change something during uh the backup, and that would cause the the backup to be inconsistent. And then during restore, you know, if we can just if

we can shut the operator down, then we the order of restore actually doesn't matter as much. So, right now, some some operators have quiesce operations. Uh it's not standardized, so you wind up writing custom scripts that have to be executed. So, one thing we would ask and prescribe is that we start moving towards standard ways to quiesce operators. Now, again, many um many operators also provide data

protection operations. For example, the Postgres operator has things like you can you can tell it to take a backup, but there's no standard mechanism at the moment for specifying that. So, each operator has a different way to do it, usually with the custom resource, but they're all different custom resources. And then um again, we would need to figure out which resources the backup utility shouldn't protect because

the operator is doing that as part of the data protection operation. So, it comes back to the original problem of figuring out the what the resource dependencies are. Then we have um another problem, which is where are these backups being stored, and how do we manage them as part of the life cycle of the overall backup? So, if you have a database and then say you have

Postgres and then you have something else um that's using a bunch of files, and we back all of them up together, on restore, you know, how do we get them to restore together? If we're uh deleting a backup, how do we get rid of the backup? Some operators provide uh mechanisms that will just let you delete individual backups, other ones will set retention intervals. So, these are

some areas where we're going to work on uh prescribing what would be good ways for operators to expose things. And then you get more complicated uh that you actually get operators calling other operators. So, one example of this was the Harbor operator, and Harbor consists of uh the Harbor pod itself, a Postgres database, uh a Redis cache, and um a bunch of stuff in object storage on

a file system. And the Harbor operator would actually use the Postgres and Redis operators to stand up the Postgres and Redis services that it needs to work on top of. again, now when we are in the uh the mode of trying to backup and restore this, we need to understand this entire hierarchy. We need to understand who is responsible for doing the backup of what parts, and

then we need to be able to orchestrate the recovery of this. Then we come to the actual backup and restore of the operator itself. So, most operators don't keep a lot of state, but they do have things like Kubernetes uh where they keep track of things. And often an operator is installed in one namespace and then provides services in all of the namespaces. So, it's a cluster

resource, not a namespace resource. So, you need to understand And on restore, we need to understand what we're going to uh bring back, if we bring back the operator first, or if that's a problem left to the user, etc. Uh the dependency graph inverts when we have the operator handling backup operations, because the operator needs to be up before we can ask it to do something. And

it also can't be quiesced. So, we need to understand that and And then when these multiple operator scenarios, we need to have all of the operators back up before anything can be restored, plus we'd need to handle things in a uh bottom-up fashion. Like, for example, in the the Harbor instance, maybe we're using the Postgres native backup and restore, but we would want to restore that first

before uh gets triggered with the custom resource for the harbor repository. Um and then we have things like custom resource definition mismatches, etc. and version mismatches. So, uh those are just some of the challenges. Uh what we're going to work on is outlining what the best practices for avoiding these problems or mitigating these problems is. Uh we'd like to work harder with operator developers to consider the

backup restore replication scenarios. And then on the data protection application side, we need to work harder to handle them better. So, uh data protection working group, uh we meet every 2 weeks on Wednesdays. Uh there's a link here for it. Uh and the agenda and we have the mailing list, etc. And we encourage anybody who has any interest in data protection, whether you're an application developer, admin,

you know, Kubernetes uh internals person to come join us. And we're definitely looking for more input into the next white paper and all the other projects that are going on. And at this point, we'll take any questions if you like, if you use the microphone. Stunned silence, that's always fun. All right. Well, thank you all for coming. Um and please, you know, contact us if you have

any questions. And uh we'll see you next time.