Kubernetes SIG Storage: Intro & Deep Dive - Xing Yang, VMware by Broadcom & Jan Šafránek, Red Hat

This talk provides an in-depth exploration of the Six Storage project within Kubernetes, highlighting features introduced in the recent 1.35 release and upcoming developments in 1.36. The speakers discuss their roles in managing Kubernetes APIs related to storage, such as persistent volumes, snapshots, and volume plugins. In version 1.35, they introduce features including beta support for service account tokens in CSI drivers, mutable volume attach limits, and enhanced node affinity for persistent volumes. The upcoming version, 1.36, will graduate several features to general availability and includes new innovations such as volume group snapshots for efficient backups and an easier method for tracking changes between snapshots. The speakers encourage community involvement in ongoing discussions and development efforts.

Hello everyone. Thank you for coming to the Kubernetes six storage intro and deep dive session. My name is Shinang. I work at the VMwell by Borcon. I'm also a co-chair of six storage. >> I am Yan Shafa Franek. I work at Redhead and I am co- tech lead of six storage. >> Here's today's agenda. We will talk about Y6 storage and we'll talk about what we did

in 1.35 release and what we're working on in 1.36 and the features we are designing and prototyping and finally how to get involved in six storage sad and myself are co-chairs Michelle and young tech leads other than the leads there are also many other contributors as shown What we do in six storage is defined in our charter. We maintain Kubernetes APIs related to storage like persistent volumes,

persistent volume claims, storage classes, volume snapshots and so on. We also maintain all the code around those objects like dynamic provisioning of volumes and snapshots. We maintain Kubernetes volume plugins like the NFS at icegazi plug-in. We co-own projected volumes like secrets and config maps with sik node. We designed the container storage interface so that a storage vendor can write the driver and have their storage system consumed

by containers running in kubernetes. We maintain CSI set cars. CSSI cars are helper containers that watch Kubernetes API objects and trigger waring operations against a CSI driver. Most CSI drivers are owned by Sik cloud provider and other communities. We also designed container object storage interface so that a user can provision a object bucket in Kubernetes just like how PVC is provisioned. I will talk about what we

did in Kubernetes 1.35 release. In 1.35, we introduced a feature that allows a CSI driver to opt in for service account tokens where the secrets field. This feature actually moved to beta directly due to some security concerns. Without this feature, there is already a token request field in CSI driver. If that is specified, Cublet will be creating service account tokens based on audience and expiration set in

token requests. Cublet then adds that to the volume context um and then pass that to CSA driver through the node publish volume request call. uh passing the sensitive service account token information together with nonsensitive information through ving context is not a good practice. There are security issues. There is already a dedicated secrets field in the node publishing request in the CSS back. So uh we introduced this

feature. We added this service account token in secrets field in CSS driver. If that is set to true that means CSS driver opt in to use this field and want to store secrets in the secrets field. Um so to support this feature CS driver need to uh first support this uh new field. It still needs to handle the existing token request field for backward compatibility reason. In

1.35, we also enabled mutable volume attach limits by default while keeping it in beta. Volume attach limits specify how many volumes can be attached to a node. Without this feature, a mismatch between the reported and actual attach capacity could result in permanent scheduling failures and stuck workloads. This could happen if you have other devices like networking interfaces or GPU or other hardware devices get attached to the

node after CSI initial initialization has happened. So we introduced this feature. We made allocatable count field in CSI note driver mutable. We also allow CSI driver to specify how often this field should be updated. This field will also be automatically updated if a failure is detected while attaching a volume to a node due to insufficient attachment capacity. In 1.35 we also introduced an alpha feature u mutable

persistent volume node affinity PV node affinity specifi it specifies the which nodes can have access to the volume. Uh right now without this feature uh when a PV is created node affinity field is set at that time and after that you cannot change it. However, a user may want to modify a volume after the volume is created uh to change the volume attributes um for example to

get better IOPS and this kind of uh volume modification might result in a volume migration from one storage to another. Uh so like in this example uh initially the volume is accessible to the nodes in this one zone only and after migration it is actually accessible to all the nodes in the entire region. So if you do not change the node affinity uh this information is outdated

and scheduler uh will be making scheduling decisions based on outdated information. It could lead to pause get scheduled on the nodes that do not have access to the volume or it get rejected on the nodes. Well, it actually have access to the volume. Uh that's why we introduced this feature. Uh so now this node affinity field is mutable. Uh it can actually be changed. We didn't make

changes in the CSI driver to allow CSI driver to report the new accessibility requirements uh from a controller modify volume. We also did not make any changes to the core scheduler. Uh we are getting feedback from the community and depending on what we are getting we will uh come back to this feature and see what other change we need to make to better support this use case.

So if you are interested in this feature, please come to talk to us. In 1.35, we have another alpha feature. Uh this is co-owned with sik auto scaling. We uh integrates the CSI volume attached limits with the cluster autoscaler. Uh without this feature, cluster autoscaler has no knowledge of the CSI volume attached limits. So when it tries to decide how many nodes to spin up to satis

s satisfy the pending parts uh it may make wrong decisions and create insufficient number of nodes for the pending parts. Uh so we made changes in cluster autoscaler. Um we now uh make cluster autoscaler aware of the volume attached limits. There are two different cases. So in the node group maybe you do not have any nodes to start with. So that is the scale from zero case

or maybe in the note group you already have one or more nodes. Now you want to scale to more nodes. So we need to uh fix for those two cases. We also made changes in theuler. So now we only schedule parts to notes while CSR driver is installed. If clust admin opt in uh through this prevent pod placement without driver flag. Now with this enhancement cluster uh

understands this CSI volume attach limits. When it does the simulation to decide how many nodes needs to be uh created to satisfy the pending parts it can make more intelligent decision. Now let me hand it over to Yan to talk about what we are working on in 1.36. >> Thank you. So uh these are four four features that Shink already covered and in 136 we are graduating

SGA the service echo to tokens for secrets and mutable volume attach limits and the two remaining uh features stay in alpha in 1.36. Uh we are graduating volume group snapshots as GA and that's a feature that allows users or backup software to take a snapshot of multiple persistent volume claims at once and the snapshot is is going to be crash consistent. The API closely follows existing volume

snapshot API. The objects have the same purpose as in volume snapshots. Uh we also added corresponding cores in the CI so the storage back end can actually take the group snapshot and uh it has been in development for quite some time. Uh the biggest uh change happened between v1 beta 1 and v v1 beta 2. Uh the final API v1 is going to be the same as

v1 beta 2 basically without any uh conversion. How does it look uh from the user perspective? If user wants to take a group snapshot of multiple persistent volume claims, they create a volume group snapshot object. It is a namespaced object. Uh and in that object they specify what do they want to take uh snapshot of and uh they do it via uh label selector of PVC. So

in this case uh the snapshot controller will go through all the PVCs in the same name space and uh take a snapshot of all the PVCs that have uh label app Kubernetes io/name uh using appropriate uh volume group snapshots class. Uh once the snapshot is ready, the snapshot controller uh lets user know using the status field ready to use and also the snapshot controller creates volume snapshot

objects for the individual volumes and they have corresponding fields to find to which volume group snapshot this snapshot belongs and also which uh persistent volume claim is source of this snapshot. So you can identify which snapshot is what. Uh another feature that uh graduates SGA at least partly is SEO labeling with mount options. Uh out of curioity who here uses SC Linux in their cluster a couple

of hands and which of who uses Open Shift? Almost the same. Okay. Uh so uh until now uh when the container runtime uh starts a pot with a volume then the container runtime goes through every single file on that volume and applies the right asynous label. If you have millions billions of files of the volume it can take some time. I have seen minutes I have seen

even hours and that happens every single time you start the bot. And this is highly inefficient and we want to speed it up and there is a way how to apply a synops label as a mount option. However, it breaks uh backwards compatibility. We are going to break some applications. I'm sorry. And even if you don't use Linux, this could be exercise for you how to break

compatibility in Kubernetes. uh what we are going to break is uh when a privileged port and unprivileged port together simultaneously access the same volume. This was possible until now and when it gets GA hopefully 137 this will not be possible anymore if you think about it like it's not a very common use case sharing a volume between privileged and unprivileged spots. uh my employ employer as a

uh Kubernetes vendor we uh see in our telemetry that only about 2% of the clusters would be affected 90 98% of the clusters will be just fine there will not see any change for these two class 2% of clusters uh in 1.36 we graduate all the feature gates that are safe to use nothing breaks yet. And we are also preparing uh some metrics and events so you

can see what could break when it when 137 uh is GA and you upgrade to it. So in 136 nothing is broken. You can list what could break using uh a sinus change policy feature gate which is enabled by default. It is G. And you can either rework your your pots. Maybe you don't need to share a unprivileged ports, but if you still do, we have opt

out. You in 1.36 you can opt out using a pot spec security context as change policy. And you can see that you all your ports are healthy. You opted out from everything you could and you you can safely upgrade to 1.37. So this is how uh Kubernetes can break backwards compatibility. We can warn we warn users in one.36. We give them option to opt out. These 2%

of clusters they can opt out and upgrade build safe. Uh as beta we are graduating change block tracking and change block tracking is a feature of many storage backends. where the storage back end can can tell you what changed between two snapshots. So you have a snapshot of a volume at snapshot at a time t1 and you take another snapshot of the same volume and at time

t2 and the change block tracking in the storage back end can tell you that only few blocks changed and it can give you numbers of those blocks. In this case it would be 26 89. And the primary use case is here incremental backup. So you can do just incremental backup of the field of the things that changed or it could be a synchronous replication. And we want

to make this feature available through CSI and through Kubernetes primitives. And for that we introduce new uh sidecar on the control plane of the CSI drivers external snapshot metadata sidecar and it will expose uh gRPC endpoint as a service in Kubernetes where the backup software can query uh for the change block between two snapshots. Uh why do we have a GP gRPC service is because uh even

though the change block tracking gives you just numbers of the blocks if you have a huge volume with billions of uh we can't stream that huge amount of blocks of the numbers uh through API server so we have a new uh service this external sension metadata sidecar does does authentication authorization encryption and the CSA driver vendors can just stream the data directly from the drivers easily. Uh

we are also removing git repo in three volume plug-in. It's one of the oldest volume plugins we had in Kubernetes. Uh what it does it basically clones uh g repository configured in the potspec and gives directory v clone to a pot. This volume plugin uh is really old. It predates init containers that could do the same thing. Today we had a volume plug-in for that and unfortunately

uh this uh volume plug-in was target of few CVEes and after last one in 2024 we decided it's doesn't make sense to maintain it if there are better alternatives and we are going to remove So in 1.36 uh the volume plug-in is disabled and I think you cannot even enable it anymore. So if you have any ports that use this ancient volume plugin, please use in containers

or use get sync sidecar. They will clone the volume for you. You don't need a special volume plug-in for that anymore. And we are also evolving a container object storage interface coy which tries to provide API for object storage like Amazon S3 or OpenStack Swift uh in similar fashion as persistent volumes persistent volume claims. So it will provide buckets, bucket claims, bucket classes and a lot of

authentic authorization uh primitives around. It is moving to v1 alpha 2 and it adds support for multiple buckets to bucket access and uh simplifies the arbback rules that all the components need. And last but not least uh is a very simple feature uh where we report report when a PVC persistent volume claim was last used by a pot and the use case is just better UI better

UX. So users can list their persistent volume claims and they can see I haven't used this PVC for half a year maybe I can delete it. It's going to manifest in PVCs as a status as a condition in the status with the time stamp when it was last used by a pot or if it is used by a pot right now. So those were the features that

actually changed any status in Kubernetes or we developed something in Kubernetes and you can try uh in the design and prototyping. We have a couple of other features together with sik apps. We are still debating how to expand volumes in stateful set. So you could edit a stateful set and the volume person volume claim template inside the stateful set and it would automatically edit all the PPCs.

However, it is harder than it seems. Uh we are also trying to combine all the CSI sidecars into one repository. Uh right now we have about eightish uh git repositories for each sitecar. Well each sidecar has its own repository and it is tedious to maintain them to bump the libraries fix cvees release do the releases make images. So we would like to reduce that but as a

secondary objective we would like to combine them into single binary and save some CPU and memory in the runtime when the sidec cars run. And we have huge debates around volume health which is currently in alpha. It can report volume health as a metric and people can alert on it. However, we want to do something better and provide some API and API fields about volume health and

uh have some programmatic action uh based on those fields because reacting to metrics is pretty hard. uh how to get involved with six storage. Uh the best place is to start with uh our landing page which describe what six storage is, what do we do and it also lists all the information all the other information on this slide. Uh we have bi-weekly six storage meeting every Thursday

6 p.m. European time. The next one is going to be next week and where we talk about the features for the next release. Uh we discuss design, we discuss pull request, we discuss anything else that's necessary to discuss on a six storage level. And each sub sub project uh has their own meetings. We have meetings for uh the object storage interface. We have object uh we have

a call for CSI twice a week actually. And we have a call for data protection I think every week. We have a mailing list which is not very used but it we use it for announcements but the most of the discussion happens on the slick ch select channels. This is just few of them. We have more uh check the land uh six storage landing page if you

are interested in more slack And finally, this is just the last slide uh where you can uh see uh some basic material uh about six storage and uh our software and CSI and things like that. It's going to be available in the slides that have you uploaded the slides. I >> I will I will later. >> Okay. Sorry. It will be there in few minutes. And with

that uh thanks for the attention. Do you have any questions? There is a microphone over there that you can use. Great. Thank you.